Running spark-submit job with internal authentication

Example of running a spark-submit job with internal authentication.

This example shows how to run a spark-submit job with internal authentication.

When you use dse spark-submit to submit a Spark job, the Spark Master URL and the Spark database connection URL are set automatically. Use the Spark session builder API to set the application name. For example:

SparkSession spark = SparkSession
      .builder()
      .appName("Datastax Java example")
      .getOrCreate();
  1. Clone the example source files from github.
    git clone https://github.com/datastax/SparkBuildExamples.git
  2. Select you preferred language and build system. For example for Java and Maven:
    cd SparkBuildExamples/java/maven/dse
  3. Build the package with Maven:
    mvn package
  4. Create your authentication credentials.

    Authentication credentials can be provided in several ways, see Providing credentials from DSE tools.

  5. Use spark-submit to run the application. The following example assumes you've set your authentication credentials in an environment variable or config file.
    dse spark-submit --class com.datastax.spark.example.WriteRead ./target/writeRead-0.1.jar