Creating a Spark Structured Streaming sink using DSE
Spark Structured Streaming is a high-level API for streaming applications. DSE supports Structured Streaming for storing data into DSE.
The following Scala example shows how to store data from a streaming source to DSE using the cassandraFormat method.
val query = source.writeStream
.option("checkpointLocation", checkpointDir.toString)
.cassandraFormat("<table name>", "<keyspace name>")
.outputMode(OutputMode.Update)
.start()This example sets the OutputMode to Update, described in the Spark API documentation.
The cassandraFormat method is equivalent to calling the format method and in org.apache.spark.sql.cassandra.
val query = source.writeStream
.option("checkpointLocation", checkpointDir.toString)
.format("org.apache.spark.sql.cassandra")
.option("keyspace", ks)
.option("table", "kv")
.outputMode(OutputMode.Update)
.start()