Running Apache Spark™ commands against a remote cluster
To run Spark commands against a remote cluster, you must export the DSE configuration from one of the remote nodes to the local client machine.
To run a driver application remotely, there must be full public network communication between the remote nodes and the client machine.
Prerequisites
The local client requires Spark driver ports on the client to be accessible by the remote DSE cluster nodes. This might require configuring the firewall on the client machine and the remote DSE cluster nodes to allow communication between the machines.
Spark dynamically selects ports for internal communication unless the ports are manually set. To use dynamically chosen ports, the firewall needs to allow all port access from the remote cluster.
To set the ports manually, set the ports in the respective properties in spark-defaults.conf as shown in this example:
spark.blockManager.port 38000
spark.broadcast.port 38001
spark.driver.port 38002
spark.executor.port 38003
spark.fileserver.port 38004
spark.replClassServer.port 38005For a full list of ports used by DSE, see Securing DataStax Enterprise ports.
Where is the spark-defaults.conf file?
The location of the spark-defaults.conf file depends on the type of installation:
| Installation Type | Location |
|---|---|
Package installations + Installer-Services installations |
|
Tarball installations + Installer-No Services installations |
|
Procedure
-
Export the DataStax Enterprise client configuration from the remote node to the client node:
-
On the remote node:
dse client-tool configuration export dse-config.jar -
Copy the exported JAR to the client nodes.
scp dse-config.jar user@clientnode1.example.com: -
On the client node:
dse client-tool configuration import dse-config.jar
-
-
Run the Spark command against the remote node.
dse spark-submit submit options myApplication.jarTo set the driver host to a publicly accessible IP address, pass in the
spark.driver.hostoption.dse spark-submit --conf spark.driver.host=IP address myApplication.jar
