Running spark-submit job with internal authentication
Example of running a spark-submit job with internal authentication.
This example shows how to run a spark-submit
job with internal
authentication.
When you use dse spark-submit
to submit a Spark job, the Spark Master
URL and the Spark database connection URL are set automatically. Use the Spark session
builder API to set the application name. For example:
SparkSession spark = SparkSession
.builder()
.appName("Datastax Java example")
.getOrCreate();
- Clone the example source files from github.
git clone https://github.com/datastax/SparkBuildExamples.git
- Select you preferred language and build system. For example for Java and
Maven:
cd SparkBuildExamples/java/maven/dse
- Build the package with
Maven:
mvn package
- Create your authentication credentials.
Authentication credentials can be provided in several ways, see .
- Use
spark-submit
to run the application. The following example assumes you've set your authentication credentials in an environment variable or config file.dse spark-submit --class com.datastax.spark.example.WriteRead ./target/writeRead-0.1.jar