dse spark-submit
Launches applications on a cluster to enable use of Apache Spark™ cluster managers through a uniform interface.
This command is supported only on nodes with analytics workloads.
Synopsis
dse spark-submit \
--class <class_name> <jar_file> <other_options>|
--status|--kill <driver_id> [--master <master_ip_address>]
| Syntax conventions | Description |
|---|---|
Italic, bold, or |
Syntax diagrams and code samples use one or more of these styles to mark placeholders for variable values. Replace placeholders with a valid option or your own user-defined value. In CQL statements, it is a requirement to use angle brackets to enclose data types in a set, list, map, or tuple.
Separate the data types with a comma.
For example: In Search CQL statements, use angle brackets to identify the entity and literal value to use when
overwriting the XML element in the schema and |
|
Square brackets surround optional command arguments. Do not type the square brackets. |
|
Parentheses identify a group to choose from. Do not type the parentheses. |
|
A pipe separates alternative elements. Type any one of the elements. Do not type the pipe. |
|
Indicates that you can repeat the syntax element as often as required. |
|
Use single quotation marks to surround literal strings in CQL statements. Use single quotation marks to preserve uppercase. For Search CQL only: Single quotation marks surround an entire XML schema declaration, such as |
|
Map collection.
Curly braces enclose maps ( |
|
Ends a CQL statement. |
|
Separate command line options from command arguments with two hyphens. This syntax is useful when arguments might be mistaken for command line options. |
Options
This command supports the same options as the Apache Spark spark-submit command.
- --class
-
The entry point for the application. Specified as the fully qualified name of the main class, the path to the JAR file containing the main class, and any other options required by the application.
The class JAR file can be located in a DSEFS directory. If the DSEFS cluster is secured, provide authentication credentials as described in DSEFS authentication.
- --status
-
Get the status of a Spark application running in the DSE cluster, specified by the driver ID.
For DSE deployments,
--statusdoesn’t require--master. - --kill
-
Kill a Spark application running in the DSE cluster, specified by the driver ID.
For DSE deployments,
--killdoesn’t require--master. - --master
-
The IP address of the cluster’s Spark master node.
Examples
These examples use the HTTP response demo program that is in included in the demos directory with your DSE installation.
The -d option specifies the number of Spark nodes to use for the HTTP response demo program.
- Start the HTTP response demo program
-
Package installations:
dse spark-submit --class com.datastax.HttpSparkStream target/HttpSparkStream.jar -d $NUM_SPARK_NODESTarball installations:
INSTALL_DIRECTORY/bin/dse spark-submit --class com.datastax.HttpSparkStream target/HttpSparkStream.jar -d $NUM_SPARK_NODES - Submit an application with cluster mode and use the supervise option to restart in case of failure
-
dse spark-submit --deploy-mode cluster --supervise --class com.datastax.HttpSparkStream target/HttpSparkStream.jar -d $NUM_SPARK_NODES - Submit an application with cluster mode when TLS is enabled
-
Pass the SSL configuration with standard Spark commands to use secure HTTPS on port 4440.
dse spark-submit \ --conf spark.ssl.ui.enabled=true \ --conf spark.ssl.ui.keyPassword=keystore password \ --conf spark.ssl.ui.keyStore=path to keystore \ myApplication.jar - Set the driver host to a publicly accessible IP address
-
dse spark-submit --conf spark.driver.host=203.0.113.0 myApplication.jar - Get the status of a driver
-
The Spark master IP address isn’t required.
dse spark-submit --status driver-20180726160353-0019If the driver exists, the output is similar to the following:
Driver driver-20180726160353-0019 found: state=<state>, worker=<workerId> (<workerHostPort>) - Kill a driver
-
The Spark master IP address isn’t required.
dse spark-submit --kill driver-20180726160353-0019
See also
For details on using Apache Spark with DSE, see: