Running Spark commands against a remote cluster

To run Spark commands against a remote cluster, you must export the DSE configuration from one of the remote nodes to the local client machine.

To run a driver application remotely, there must be full public network communication between the remote nodes and the client machine.

spark-defaults.conf

The default location of the spark-defaults.conf file depends on the type of installation:

Package installations	/etc/dse/spark/spark-defaults.conf
Tarball installations	`installation_location`/resources/spark/conf/spark-defaults.conf

Prerequisites

The local client requires Spark driver ports on the client to be accessible by the remote DSE cluster nodes. This might require configuring the firewall on the client machine and the remote DSE cluster nodes to allow communication between the machines.

Spark dynamically selects ports for internal communication unless the ports are manually set. To use dynamically chosen ports, the firewall needs to allow all port access from the remote cluster.

To set the ports manually, set the ports in the respective properties in spark-defaults.conf as shown in this example:

spark.blockManager.port 38000
spark.broadcast.port 38001
spark.driver.port 38002
spark.executor.port 38003
spark.fileserver.port 38004
spark.replClassServer.port 38005

For a full list of ports used by DSE, see .

Procedure

Export the DataStax Enterprise client configuration from the remote node to the client node:
1. On the remote node:
```
dse client-tool configuration export dse-config.jar
```
2. Copy the exported JAR to the client nodes.
```
scp dse-config.jar user@clientnode1.example.com:
```
3. On the client node:
```
dse client-tool configuration import dse-config.jar
```
Run the Spark command against the remote node.
```
dse spark-submit submit options myApplication.jar
```
To set the driver host to a publicly accessible IP address, pass in the spark.driver.host option.
```
dse spark-submit --conf spark.driver.host=IP address myApplication.jar
```