Running spark-submit job with internal authentication

Example of running a spark-submit job with internal authentication.

This example shows how to run a spark-submit job with internal authentication.

When you use dse spark-submit to submit a Spark job, the Spark Master URL and the Spark database connection URL are set automatically. Use the Spark session builder API to set the application name. For example:

SparkSession spark = SparkSession
      .builder()
      .appName("Datastax Java example")
      .getOrCreate();
  1. Clone the example source files from github.

    git clone https://github.com/datastax/SparkBuildExamples.git
  2. Select you preferred language and build system. For example for Java and Maven:

    cd SparkBuildExamples/java/maven/dse
  3. Build the package with Maven:

    mvn package
  4. Create your authentication credentials.

    Authentication credentials can be provided in several ways, see Providing credentials from DSE tools.

  5. Use spark-submit to run the application. The following example assumes you’ve set your authentication credentials in an environment variable or config file.

    dse spark-submit --class com.datastax.spark.example.WriteRead ./target/writeRead-0.1.jar

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com