Securing Spark connections
Communication between Spark applications and transactional nodes, masters and workers, and intercommunication between Spark drivers and executors can be encrypted.
Communication between Spark applications and transactional nodes, masters and workers, and intercommunication between Spark drivers and executors can be encrypted. You must configure encryption on each node in your cluster.
Encryption between the Spark driver and DSE is configured by enabling client encryption in cassandra.yaml.
Encryption between Spark nodes, including between the Spark master and worker, is configured by enabling Spark security in dse.yaml.
Encryption between the Spark driver and executors in client applications is configured by enabling Spark security in the application configuration properties, or by default in spark-defaults.conf in the Spark configuration directory.
- Package installations and Installer-Services: /etc/dse/spark/
- Tarball installations and Installer-No Services: installation_location/resources/spark/conf
dse.yaml
The location of the dse.yaml file depends on the type of installation:
Package installations |
/etc/dse/dse.yaml |
Tarball installations |
installation_location/resources/dse/conf/dse.yaml |
cassandra.yaml
The location of the cassandra.yaml file depends on the type of installation:
Package installations |
/etc/dse/cassandra/cassandra.yaml |
Tarball installations |
installation_location/resources/cassandra/conf/cassandra.yaml |
Procedure
-
Enable mutual authentication and encryption between Spark master and worker
nodes in dse.yaml.
Tip: In DSE 5.1.15 and later, when DSE authentication is enabled with authentication_options in dse.yaml, Spark security is enabled regardless of these settings.
-
Enable mutual authentication by setting
spark_security_enabled
totrue
. -
Enable encryption by setting
spark_security_encryption_enabled
totrue
.
spark_security_enabled: true spark_security_encryption_enabled: true
Note: To enable encryption, you must also enable mutual authentication. -
Enable mutual authentication by setting
- To encrypt communication between the Spark driver and master, DSE inherits the client to cluster connection encryption options.
-
The Spark web UI by default uses
client-to-cluster encryption settings to enable SSL security in the web
interface. To enable SSL security separately from DSE client-to-cluster
encryption, change the settings in dse.yaml
under
spark_ui_options
. Setencryption
tocustom
, then set the keystore settings inencryption_options
.The Spark web UI server automatically discards all cipher algorithms that end with a suffix SHA, SHA1 or MD5. If you manually specify cipher suites for encryption, make sure the specified cipher suites are not ignored by the server and are supported by the web browsers used to access the Spark web UI. For example, two 256 bit cipher suites that are supported by both the server and current web browsers areTLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
andTLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
. - The Spark SQL Thrift server can optionally use SSL to encrypt client connections.
- Restart the node to recognize the changes.
-
For each Spark application, set the following client encryption options to true
to enable encryption between the Spark driver and executors. To enable
encryption by default for all Spark applications modify the options in the
spark-defaults.conf file in the Spark configuration
directory.
Option Description spark.authenticate
Enables or disables mutual authentication using a secret key. The default is false. spark.authenticate.enableSaslEncryption
Enables or disables SASL encryption between a Spark driver and its executors. The default is false.