Configuring Apache Spark™ nodes

DataStax recommends using the default values of Spark environment variables unless you need to increase the memory settings due to an OutOfMemoryError condition or garbage collection taking too long. Use the Spark memory configuration options in the dse.yaml and spark-env.sh files.

You can set a user-specific SPARK_HOME directory if you also set ALLOW_SPARK_HOME=true in your environment before starting DSE.

For example, on Debian or Ubuntu using a package installation:

To configure worker cleanup, modify the SPARK_WORKER_OPTS environment variable and add the cleanup properties. The SPARK_WORKER_OPTS environment variable can be set in the user environment or in spark-env.sh.

For example, the following enables worker cleanup, sets the cleanup interval to 30 minutes (i.e. 1800 seconds), and sets the length of time application worker directories will be retained to 7 days (i.e. 604800 seconds).

After you start up a Spark cluster, DataStax Enterprise creates a Spark work directory for each Spark Worker on worker nodes. A worker node can have more than one worker, configured by the SPARK_WORKER_INSTANCES option in spark-env.sh. If SPARK_WORKER_INSTANCES is undefined, a single worker is started. The work directory contains the standard output and standard error of executors and other application specific data stored by Spark Worker and executors; the directory is writable only by the DSE user.

By default, the Spark parent work directory is located in /var/lib/spark/work, with each worker in a subdirectory named worker-number, where the number starts at 0. To change the parent worker directory, configure SPARK_WORKER_DIR in the spark-env.sh file.

The Spark RDD directory is the directory where RDDs are placed when executors decide to spill them to disk. This directory might contain the data from the database or the results of running Spark applications. If the data in the directory is confidential, prevent access by unauthorized users. The RDD directory might contain a significant amount of data, so configure its location on a fast disk. The directory is writable only by the cassandra user. The default location of the Spark RDD directory is /var/lib/spark/rdd. The directory should be located on a fast disk. To change the RDD directory, configure SPARK_LOCAL_DIRS in the spark-env.sh file.

Before starting up nodes on a tarball installation, you need permission to access the default Spark directory locations: /var/lib/spark and /var/log/spark. Change ownership of these directories as follows:

In multiple datacenter clusters, use a virtual datacenter to isolate Spark jobs. Running Spark jobs consume resources that can affect latency and throughput.

DataStax Enterprise supports the use of virtual nodes (vnodes) with Spark.

There are two kinds of authorization permissions which apply to Spark. Work pool permissions control the ability to submit a Spark application to DSE. Submission permissions control the ability to manage a particular application. All the following instructions assume you are issuing the CQL commands as a database superuser.

Use GRANT CREATE ON ANY WORKPOOL TO role to grant permission to submit a Spark application to any Analytics datacenter.

Use GRANT CREATE ON WORKPOOL datacenter_name TO role to grant permission to submit a Spark application to a particular Analytics datacenter.

There are similar revoke commands:

When an application is submitted, the user who submits that application is automatically granted permission to manage and remove the application. You may also grant the ability to manage the application to another user or role.

Use GRANT MODIFY ON ANY SUBMISSION TO role to grant permission to manage any submission in any work pool to the specified role.

Use GRANT MODIFY ON ANY SUBMISSION IN WORKPOOL datacenter_name TO role to grant permission to manage any submission in a specified datacenter.

Use GRANT MODIFY ON SUBMISSION id IN WORKPOOL datacenter_name TO role to grant permission to manage a submission identified by the provided id in a given datacenter.

There are similar revoke commands:

In order to issue these commands as a regular database user, the user needs to have permission to use the DSE resource manager RPC:

Each DSE Analytics user needs to have permission to use the client tools RPC:

Spark memory options affect different components of the Spark ecosystem:

You can configure additional Java options that are applied by the worker when spawning an executor for the application. Use the spark.executor.extraJavaOptions property, described in Spark 2.0.2 Available Properties. For example: spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

You can manage the number of cores by configuring these options.

See the Spark documentation for details about memory and core allocation.

DataStax Enterprise can control the memory and cores offered by particular Spark Workers in semi-automatic fashion. The initial_spark_worker_resources parameter in the dse.yaml file specifies the fraction of system resources that are made available to the Spark Worker. The available resources are calculated in the following way:

The lowest values you can assign to Spark Worker memory and cores are 64 MB and 1 core, respectively. If the results are lower, no exception is thrown and the values are automatically limited. The range of the initial_spark_worker_resources value is 0.01 to 1. If the range is not specified, the default value 0.7 is used.

This mechanism is used by default to set the Spark Worker memory and cores. To override the default, uncomment and edit one or both SPARK_WORKER_MEMORY and SPARK_WORKER_CORES options in the spark-env.sh file.

If you are using a cloud infrastructure provider like Amazon EC2, you must explicitly open the ports for publicly routable IP addresses in your cluster. If you do not, the Spark workers will not be able to find the Spark Master.

One work-around is to set the prefer_local setting in your cassandra-rackdc.properties snitch setup file to true:

This tells the cluster to communicate only on private IP addresses within the datacenter rather than the public routable IP addresses.

When Spark fetches configuration settings from DSE, it does not fail immediately if it cannot retrieve the configuration data. Spark retries 5 times by default with increasing delay between retries. The number of retries can be set in the Spark configuration, by modifying the spark.dse.configuration.fetch.retries configuration property when calling the dse spark command, or in spark-defaults.conf.

Continuous paging streams bulk amounts of records from DSE to the Java driver used by DSE Spark. By default, continuous paging in queries is disabled. To enable it, set the spark.dse.continuous_paging_enabled setting to true when starting the Spark SQL shell or in spark-defaults.conf. For example:

By default the Spark web UI runs on port 7080. To change the port number, do the following:

Users can add third party JARs to Spark nodes by adding them to the Spark lib directory on each node and restart the cluster. Add the Graphite Metrics JARs to this directory to enable metrics in DSE Spark.

To add the Graphite JARs to Spark in a package installation, copy them to the Spark lib directory:

The spark-daemon-defaults.conf file configures DSE Spark Masters and Workers.

By default Spark executor logs, which log the majority of your Spark Application output, are redirected to standard output. The output is managed by Spark Workers. Configure logging by adding spark.executor.logs.rolling.* properties to spark-daemon-defaults.conf file.