Spark server configuration

The spark-daemon-defaults.conf file configures DSE Spark Masters and Workers.

Spark server configuration properties
Option Default value Description

dse.spark.application.timeout

30

The duration in seconds after which the application will be considered dead if no heartbeat is received.

spark.dseShuffle.sasl.port

7447

The port number on which a shuffle service for SASL secured applications is started. Bound to the listen_address in cassandra.yaml.

spark.dseShuffle.noSasl.port

7437

The port number on which a shuffle service for unsecured applications is started. Bound to the listen_address in cassandra.yaml.

By default Spark executor logs, which log the majority of your Spark Application output, are redirected to standard output. The output is managed by Spark Workers. Configure logging by adding spark.executor.logs.rolling.* properties to spark-daemon-defaults.conf file.

spark.executor.logs.rolling.maxRetainedFiles 3
spark.executor.logs.rolling.strategy size
spark.executor.logs.rolling.maxSize 50000

Additional Spark properties that affect the master and driver can be added to spark-daemon-defaults.conf. For example, to enable Spark’s commons-crypto encryption library:

spark.network.crypto.enabled true

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com