Secure Spark connections
Communication between Spark applications and transactional nodes, masters and workers, and intercommunication between Spark drivers and executors can be encrypted. Encryption must be configured individually on each node in the cluster.
The location of the cassandra.yaml file depends on your installation type.
-
Package installations
-
Tarball installations
/etc/hcd/cassandra/cassandra.yamlINSTALLATION_LOCATION/resources/cassandra/conf/cassandra.yamlReplace INSTALLATION_LOCATION with the path where you extracted the HCD tarball.
Encryption between Spark applications, including between the Spark master and worker nodes, is configured by enabling Spark security in the hcd.yaml file.
Encryption between the Spark driver and executors in client applications is configured by enabling Spark security in the application configuration properties, or by default in spark-defaults.conf and spark-daemon-defaults.conf in the Spark configuration directory.
The default location of the Spark configuration files depends on your installation type.
-
Package installations
-
Tarball installations
/etc/hcd/spark/INSTALLATION_LOCATION/resources/spark/confReplace INSTALLATION_LOCATION with the path where you extracted the HCD tarball.
Use the hcd.yaml configuration file to enable Spark security
-
Locate the
hcd.yamlconfiguration file. The location of this file depends on your installation type.-
Package installations
-
Tarball installations
/etc/hcd/hcd.yamlINSTALLATION_LOCATION/resources/hcd/conf/hcd.yamlReplace INSTALLATION_LOCATION with the path where you extracted the HCD tarball.
-
-
Enable mutual authentication and encryption between Spark master and worker nodes in the
hcd.yamlfile.spark_security_enabled: true spark_security_encryption_enabled: trueTo enable encryption, you must also enable mutual authentication.
-
To encrypt communication between the Spark application and master, HCD inherits the
client to cluster connection encryptionoptions. To enable SSL security separately from HCD client-to-cluster encryption, change the settings in thecassandra.yamlfile underspark_ui_options. Setencryptiontocustom, then set the keystore settings inencryption_options.The Spark web UI server automatically discards all cipher algorithms that end with a suffix
SHA,SHA1orMD5. If you manually specify cipher suites for encryption, make sure the specified cipher suites are not ignored by the server and are supported by the web browsers used to access the Spark web UI. For example, two 256 bit cipher suites that are supported by both the server and current web browsers areTLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384andTLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384.
-
For each Spark application, set the following client encryption options to true in the
spark-defaults.conffile to enable encryption between the Spark driver and executors.Option Description spark.authenticateEnables or disables mutual authentication using a secret key. The default is
false.spark.authenticate.enableSaslEncryptionEnables or disables SASL encryption between a Spark driver and its executors. The default is
false.spark.io.encryption.enabledEnables or disables encryption for data stored by drivers directly on disk.
To enable encryption by default for all Spark applications, modify the options in the
spark-defaults.conffile in the Spark configuration directory. To encrypt data stored on the server by default, modify the options in thespark-daemon-defaults.conffile.Option Description spark.io.encryption.enabledEnables or disables encryption for data stored by executors and shuffle services directly on disk.
