Connecting to DataStax Enterprise using the Spark shell on an external Spark cluster
Use the generated byos.properties configuration file and the byos-version.jar from a DataStax Enterprise node to connect to the DataStax Enterprise cluster from the Spark shell on an external Spark cluster.
Where is the clients directory?
The default location of the clients directory depends on the type of installation:
| Installation Type | Location |
|---|---|
Package installations + Installer-Services installations |
|
Tarball installations + Installer-No Services installations |
|
Prerequisites
You must generate the byos.properties on a node in your DataStax Enterprise cluster.
Procedure
-
Copy the
byos.propertiesfile you previously generated from the DataStax Enterprise node to the local Spark node.scp user@dsenode1.example.com:~/byos.properties .If you are using Kerberos authentication, specify the
--generate-tokenand--token-renewer <username>options when generatingbyos.properties, as described indse client-tool configuration byos-export. -
Copy the
byos-version.jarfile from the clients directory from a node in your DataStax Enterprise cluster to the local Spark node.The
byos-version.jarfile location depends on the type of installation.scp user@dsenode1.example.com:/usr/share/dse/clients/dse-byos_2.10-5.0.1-5.0.0-all.jar byos-5.0.jar -
Merge external Spark properties into
byos.properties.cat ${SPARK_HOME}/conf/spark-defaults.conf >> byos.properties -
(Optional) If you are using Kerberos authentication, set up a CRON job or other task scheduler to periodically call
dse client-tool cassandra renew-token <token>where<token>is the encoded token string inbyos.properties. -
Start the Spark shell using the
byos.propertiesandbyos-version.jarfile.spark-shell --jars byos-5.0.jar --properties-file byos.properties
