Creating a Spark Structured Streaming sink using DSE

A Spark Structured Streaming sink pulls data into DSE.

Spark Structured Streaming is a high-level API for streaming applications. DSE supports Structured Streaming for storing data into DSE.

The following Scala example shows how to store data from a streaming source to DSE using the cassandraFormat method.

val query = source.writeStream
  .option("checkpointLocation", checkpointDir.toString)
  .cassandraFormat("table name", "keyspace name")
  .outputMode(OutputMode.Update)
  .start()

This example sets the OutputMode to Update, described in the Spark API documentation.

The cassandraFormat method is equivalent to calling the format method and in org.apache.spark.sql.cassandra.

val query = source.writeStream
  .option("checkpointLocation", checkpointDir.toString)
  .format("org.apache.spark.sql.cassandra")
    .option("keyspace", ks)
    .option("table", "kv")
  .outputMode(OutputMode.Update)
  .start()