Spark server configuration
The spark-daemon-defaults.conf file configures DSE Spark Masters and Workers.
| Option | Default value | Description |
|---|---|---|
|
30 |
The duration in seconds after which the application will be considered dead if no heartbeat is received. |
|
7447 |
The port number on which a shuffle service for SASL secured applications is started.
Bound to the |
|
7437 |
The port number on which a shuffle service for unsecured applications is started.
Bound to the |
By default Spark executor logs, which log the majority of your Spark Application output, are redirected to standard output.
The output is managed by Spark Workers.
Configure logging by adding spark.executor.logs.rolling.* properties to spark-daemon-defaults.conf file.
spark.executor.logs.rolling.maxRetainedFiles 3 spark.executor.logs.rolling.strategy size spark.executor.logs.rolling.maxSize 50000
Additional Spark properties that affect the master and driver can be added to spark-daemon-defaults.conf.
For example, to enable Spark’s commons-crypto encryption library:
spark.network.crypto.enabled true