Creating a Spark Structured Streaming sink using DSE

Spark Structured Streaming is a high-level API for streaming applications. DSE supports Structured Streaming for storing data into DSE.

The following Scala example shows how to store data from a streaming source to DSE using the cassandraFormat method.

val query = source.writeStream
  .option("checkpointLocation", checkpointDir.toString)
  .cassandraFormat("<table name>", "<keyspace name>")
  .outputMode(OutputMode.Update)
  .start()

This example sets the OutputMode to Update, described in the Spark API documentation.

The cassandraFormat method is equivalent to calling the format method and in org.apache.spark.sql.cassandra.

val query = source.writeStream
  .option("checkpointLocation", checkpointDir.toString)
  .format("org.apache.spark.sql.cassandra")
    .option("keyspace", ks)
    .option("table", "kv")
  .outputMode(OutputMode.Update)
  .start()

Creating a Spark Structured Streaming sink using DSE

Was this helpful?

Give Feedback