Cluster Hardware Specs

Optimization Strategy

Waiting for input calculation...

Executors

--

Cores / Ex

--

RAM / Ex

--

Parallelism

--

Spark Submit Command 
Adjust specs and click calculate...

Tradeoffs in Executor Sizing

Choosing the right executor memory and cores is crucial. The optimal number of executor cores typically falls between 3 to 7 (with 5 being the sweet spot). Here's why "Tiny" or "Fat" approaches fail, and how a "Balanced" approach maximizes throughput.

Tiny Executors

E.g., 1 Core per Executor = 16 Executors per Node

  • • No Task Sharing: Multiple tasks in the same executor JVM can't be run concurrently.
  • • High Overhead: Huge number of small JVMs creates massive start and kill overhead.
  • • Memory Waste: Shared/cached variables (broadcast variables, accumulators) replicate in each executor (16 times per node!).
  • • Daemon Starvation: Does not leave enough memory overhead for Hadoop/Yarn daemon processes and ApplicationManager.
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
1C
16 Small JVMs / High Overhead

Fat Executors

E.g., 16 Cores per Executor = 1 Executor per Node

  • • Daemon Ignored: With all 16 cores assigned to one executor, ApplicationMaster and OS daemon processes are not counted, risking stability.
  • • HDFS Bottleneck: HDFS client has trouble handling too many concurrent threads. Throughput will hurt significantly.
  • • GC Pauses: Managing a massive JVM heap will result in excessive Garbage Collection (GC) execution, leading to severe performance degradation.
1 EXECUTOR
16 CORES
ALL RAM
Massive JVM Bottleneck
Optimal

Balanced Strategy

E.g., 5 Cores per Exec = 3 Executors per Node

  • ✓ Best of Both Worlds: Finds the right balance. Achieves the parallelism of a fat executor and the best throughputs of a tiny executor.
  • ✓ Optimal HDFS I/O: --executor-cores=5 is the established sweet spot for maintaining excellent HDFS throughput.
  • ✓ Safe Overhead: Leaves 1 core and dedicated RAM strictly for OS daemons, NodeManager, and the ApplicationMaster.
5C
5C
5C
1 CORE RESERVED FOR OS
Maximum Throughput

Step-by-Step Calculation Formula

SCENARIO
Cluster of 10 Worker Nodes | 16 Cores per Node | 64GB RAM per Node

1 Compute Available Cores

We must reserve 1 core per node for background processes (OS, DataNode, NodeManager daemons).

Num cores per node = 16 - 1 = 15 cores
Total cluster cores = 15 × 10 = 150 cores

2 Distribute Executors

Assign 5 cores per executor. Then reserve 1 total executor for the ApplicationMaster.

Executors available = 150 / 5 = 30 executors
Minus 1 for AM = 30 - 1 = 29 total executors
Execs per node = 30 / 10 = 3 executors/node

3 Calculate Memory & Off-Heap

Divide RAM among executors, then deduct ~15% for off-heap and YARN overhead.

RAM per exec = 64GB / 3 = 21GB
Off-heap overhead = 21GB × 15% = ~3GB
Final Exec Memory = 21 - 3 = 18GB

Final Result: --num-executors 29 | --executor-memory 18G | --executor-cores 5