dse spark-submit

Launches applications on a cluster to enable use of Apache Spark™ cluster managers through a uniform interface.

This command is supported only on nodes with analytics workloads.

Synopsis

dse spark-submit \
  --class <class_name> <jar_file> <other_options>|
  --status|--kill <driver_id> [--master <master_ip_address>]
Syntax legend
Syntax conventions Description

Italic, bold, or < >

Syntax diagrams and code samples use one or more of these styles to mark placeholders for variable values. Replace placeholders with a valid option or your own user-defined value.

In CQL statements, it is a requirement to use angle brackets to enclose data types in a set, list, map, or tuple. Separate the data types with a comma. For example: '<<datatype1>,<datatype2>>'

In Search CQL statements, use angle brackets to identify the entity and literal value to use when overwriting the XML element in the schema and solrconfig files, such as @<xml_entity>='<xml_entity_type>'.

[ ]

Square brackets surround optional command arguments. Do not type the square brackets.

( )

Parentheses identify a group to choose from. Do not type the parentheses.

|

A pipe separates alternative elements. Type any one of the elements. Do not type the pipe.

...

Indicates that you can repeat the syntax element as often as required.

'

Use single quotation marks to surround literal strings in CQL statements. Use single quotation marks to preserve uppercase.

For Search CQL only: Single quotation marks surround an entire XML schema declaration, such as '<<schema> ... </schema>>'

{ }

Map collection. Curly braces enclose maps ({ <key_datatype>:<value_datatype> }) or key value pairs ({ <key>:<value> }). A colon separates the key and the value.

;

Ends a CQL statement.

--

Separate command line options from command arguments with two hyphens. This syntax is useful when arguments might be mistaken for command line options.

Options

--class

The entry point for the application. Specified as the fully qualified name of the main class, the path to the JAR file containing the main class, and any other options required by the application.

The class JAR file can be located in a DSEFS directory. If the DSEFS cluster is secured, provide authentication credentials as described in DSEFS authentication.

--status

Get the status of a Spark application running in the DSE cluster, specified by the driver ID.

For DSE deployments, --status doesn’t require --master.

--kill

Kill a Spark application running in the DSE cluster, specified by the driver ID.

For DSE deployments, --kill doesn’t require --master.

--master

The IP address of the cluster’s Spark master node.

Examples

These examples use the HTTP response demo program that is in included in the demos directory with your DSE installation. The -d option specifies the number of Spark nodes to use for the HTTP response demo program.

Start the HTTP response demo program

Package installations:

dse spark-submit --class com.datastax.HttpSparkStream target/HttpSparkStream.jar -d $NUM_SPARK_NODES

Tarball installations:

INSTALL_DIRECTORY/bin/dse spark-submit --class com.datastax.HttpSparkStream target/HttpSparkStream.jar -d $NUM_SPARK_NODES
Submit an application with cluster mode and use the supervise option to restart in case of failure
dse spark-submit --deploy-mode cluster --supervise --class com.datastax.HttpSparkStream target/HttpSparkStream.jar -d $NUM_SPARK_NODES
Submit an application with cluster mode when TLS is enabled

Pass the SSL configuration with standard Spark commands to use secure HTTPS on port 4440.

dse spark-submit \
--conf spark.ssl.ui.enabled=true \
--conf spark.ssl.ui.keyPassword=keystore password \
--conf spark.ssl.ui.keyStore=path to keystore \
myApplication.jar
Set the driver host to a publicly accessible IP address
dse spark-submit --conf spark.driver.host=203.0.113.0 myApplication.jar
Get the status of a driver

The Spark master IP address isn’t required.

dse spark-submit --status driver-20180726160353-0019

If the driver exists, the output is similar to the following:

Driver driver-20180726160353-0019 found: state=<state>, worker=<workerId> (<workerHostPort>)
Kill a driver

The Spark master IP address isn’t required.

dse spark-submit --kill driver-20180726160353-0019

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM