QBoard » Big Data » Big Data - Spark » What are workers, executors, cores in Spark Standalone cluster?
User Dashboard

What are workers, executors, cores in Spark Standalone cluster?

Back To Topics
Tags : apache_spark distributed_computing
  • 24

    I read Cluster Mode Overview and I still can't understand the different processes in the Spark Standalone cluster and the parallelism.

    Is the worker a JVM process or not? I ran the bin\start-slave.sh and found that it spawned the worker, which is actually a JVM.

    As per the above link, an executor is a process launched for an application on a worker node that runs tasks. An executor is also a JVM.

    These are my questions:

    1. Executors are per application. Then what is the role of a worker? Does it co-ordinate with the executor and communicate the result back to the driver? Or does the driver directly talks to the executor? If so, what is the worker's purpose then?

    2. How to control the number of executors for an application?

    3. Can the tasks be made to run in parallel inside the executor? If so, how to configure the number of threads for an executor?

    4. What is the relation between a worker, executors and executor cores ( --total-executor-cores)?

    5. What does it mean to have more workers per node?

    Updated

    Let's take examples to understand better.

    Example 1: A standalone cluster with 5 worker nodes (each node having 8 cores) When I start an application with default settings.

    Example 2 Same cluster config as example 1, but I run an application with the following settings --executor-cores 10 --total-executor-cores 10.

    Example 3 Same cluster config as example 1, but I run an application with the following settings --executor-cores 10 --total-executor-cores 50.

    Example 4 Same cluster config as example 1, but I run an application with the following settings --executor-cores 50 --total-executor-cores 50.

    Example 5 Same cluster config as example 1, but I run an application with the following settings --executor-cores 50 --total-executor-cores 10.

    In each of these examples, How many executors? How many threads per executor? How many cores? How is the number of executors decided per application? Is it always the same as the number of workers?

      December 29, 2020 12:29 PM IST
    0
  • 319 1
    Spark has a master-slave architecture, where it has got one central coordinator,i.e. Driver, that communicates with many distributed workers nodes(executors).

    DRIVER

    The driver is basically a process where the main method runs. It converts the user program into tasks and just after that it schedules the tasks on the executors.

    WORKERS

    Workers (slaves) are running Spark instances where executors live to execute tasks. They are the compute nodes in Spark.

    A worker receives serialized tasks that it runs in a thread pool.

    It hosts a local Block Manager that serves blocks to other workers in a Spark cluster. Workers communicate among themselves using their Block Manager instances.

    When you create SparkContext, each worker starts an executor. This is a separate process (JVM), and it loads your jars, too. The executors connects back to your driver program and now the driver can send them commands, like flatMap, map and reduceByKey. When the driver quits, the executors shuts down.

    EXECUTORS

    Executor is a distributed agent that is responsible for executing tasks. It typically runs for the entire lifetime of a Spark application which is called static allocation of executors.

    Executors reports to HeartbeatReceiver RPC Endpoint on the driver by sending heartbeat and partial metrics for active tasks. It runs multiple tasks over its lifetime(parallel and sequentially). They track running tasks (by their task ids in runningTasks internal registry). They are also used to provide in-memory storage for RDDs that are cached by user programs through Block Manager.

    Answering your questions

    1. As soon as the executors are started they quickly get registered with the driver and right from that point of time they communicate directly. The workers job is to communicate with the cluster manager for the availability of their resources.

    2. In a YARN cluster you can do that with --num-executors. In a standalone cluster you will be provided with one executor per worker unless you work with spark.executor.cores and a worker has enough cores to hold more than one executor.

    https://github.com/apache/spark/commit/16b6d18613e150c7038c613992d80a7828413e66

    style="font-size: 10pt; font-family: arial, helvetica, sans-serif;">3. You can assign the number of cores per executor with --executor-cores

    4. --total-executor-cores is the max number of executor cores per application

    5. there's not a good reason to run more than one worker per machine. You would have many JVM sitting in one machine for instance.

    EXAMPLE 1: Since no. of cores and executors acquired by the Spark is directly proportional to the offering made by the scheduler, Spark will acquire cores and executors accordingly. So in the end you will get 5 executors with 8 cores each.

    EXAMPLE 2 to 5: No executors will be launched, Since Spark won't be able to allocate as many cores as requested in a single worker.
      October 18, 2021 2:05 PM IST
    0
  • 259
     

    enter image description here

    Spark uses a master/slave architecture. As you can see in the figure, it has one central coordinator (Driver) that communicates with many distributed workers (executors). The driver and each of the executors run in their own Java processes.

    DRIVER

    The driver is the process where the main method runs. First it converts the user program into tasks and after that it schedules the tasks on the executors.

    EXECUTORS

    Executors are worker nodes' processes in charge of running individual tasks in a given Spark job. They are launched at the beginning of a Spark application and typically run for the entire lifetime of an application. Once they have run the task they send the results to the driver. They also provide in-memory storage for RDDs that are cached by user programs through Block Manager.

    APPLICATION EXECUTION FLOW

    With this in mind, when you submit an application to the cluster with spark-submit this is what happens internally:

    1. A standalone application starts and instantiates a SparkContext instance (and it is only then when you can call the application a driver).
    2. The driver program ask for resources to the cluster manager to launch executors.
    3. The cluster manager launches executors.
    4. The driver process runs through the user application. Depending on the actions and transformations over RDDs task are sent to executors.
    5. Executors run the tasks and save the results.
    6. If any worker crashes, its tasks will be sent to different executors to be processed again. In the book "Learning Spark: Lightning-Fast Big Data Analysis" they talk about Spark and Fault Tolerance:

    Spark automatically deals with failed or slow machines by re-executing failed or slow tasks. For example, if the node running a partition of a map() operation crashes, Spark will rerun it on another node; and even if the node does not crash but is simply much slower than other nodes, Spark can preemptively launch a “speculative” copy of the task on another node, and take its result if that finishes.

    1. With SparkContext.stop() from the driver or if the main method exits/crashes all the executors will be terminated and the cluster resources will be released by the cluster manager.

    YOUR QUESTIONS

    1. When executors are started they register themselves with the driver and from so on they communicate directly. The workers are in charge of communicating the cluster manager the availability of their resources.

    2. In a YARN cluster you can do that with --num-executors. In a standalone cluster you will get one executor per worker unless you play with spark.executor.cores and a worker has enough cores to hold more than one executor. (As @JacekLaskowski pointed out, --num-executors is no longer in use in YARN https://github.com/apache/spark/commit/16b6d18613e150c7038c613992d80a7828413e66)

    3. You can assign the number of cores per executor with --executor-cores

    4. --total-executor-cores is the max number of executor cores per application

    5. As Sean Owen said in this thread: "there's not a good reason to run more than one worker per machine". You would have many JVM sitting in one machine for instance.

    UPDATE

    I haven't been able to test this scenarios, but according to documentation:

    EXAMPLE 1: Spark will greedily acquire as many cores and executors as are offered by the scheduler. So in the end you will get 5 executors with 8 cores each.

    EXAMPLE 2 to 5: Spark won't be able to allocate as many cores as requested in a single worker, hence no executors will be launch.

      August 11, 2021 1:24 PM IST
    0
  • 7 1
    enter image description here
      December 29, 2020 4:09 PM IST
    0