Connecting to DataStax Enterprise using the Spark shell on an external Spark cluster
Use the generated byos.properties configuration file and the byos-version.jar from a DataStax Enterprise node to connect to the DataStax Enterprise cluster from the Spark shell on an external Spark cluster.
-
Generate the
byos.propertieson a node in your DataStax Enterprise cluster. -
Copy the
byos.propertiesfile you previously generated from the DataStax Enterprise node to the local Spark node.scp user@dsenode1.example.com:~/byos.properties .If you are using Kerberos authentication, specify the
--generate-tokenand--token-renewer <username>options when generatingbyos.properties, as described indse client-tool configuration byos-export. -
Copy the
byos-version.jarfile from theclientsdirectory from a node in your DataStax Enterprise cluster to the local Spark node.The default location of the
clientsdirectory depends on the type of installation:-
Package installations and Installer-Services installations:
/usr/share/dse/clients -
Tarball installations and Installer-No Services installations:
<installation_location>/clients
The
byos-version.jarfile location depends on the type of installation.scp user@dsenode1.example.com:/usr/share/dse/clients/dse-byos_2.10-5.0.1-5.0.0-all.jar byos-5.0.jar -
-
Merge external Spark properties into
byos.properties.cat ${SPARK_HOME}/conf/spark-defaults.conf >> byos.properties -
(Optional) If you are using Kerberos authentication, set up a CRON job or other task scheduler to periodically call
dse client-tool cassandra renew-token <token>where<token>is the encoded token string inbyos.properties. -
Start the Spark shell using the
byos.propertiesandbyos-version.jarfile.spark-shell --jars byos-5.0.jar --properties-file byos.properties
