Getting started with Spark Streaming

Spark Streaming allows you to consume live data streams from sources, including Akka, Kafka, and Twitter. This data can then be analyzed by Spark applications, and the data can be stored in the database.

You use Spark Streaming by creating an org.apache.spark.streaming.StreamingContext instance based on your Spark configuration. You then create a DStream instance, or a discretionized stream, an object that represents an input stream. DStream objects are created by calling one of the methods of StreamingContext, or using a utility class from external libraries to connect to other sources like Twitter.

The data you consume and analyze is saved to the database by calling one of the saveToCassandra methods on the stream object, passing in the keyspace name, the table name, and optionally the column names and batch size.

Spark Streaming applications require synchronized clocks to operate correctly. See Synchronize clocks.


The following Scala example demonstrates how to connect to a text input stream at a particular IP address and port, count the words in the stream, and save the results to the database.

  1. Import the streaming context objects.

    import org.apache.spark.streaming._
  2. Create a new StreamingContext object based on an existing SparkConf configuration object, specifying the interval in which streaming data will be divided into batches by passing in a batch duration.

    val sparkConf = ....
    val ssc = new StreamingContext(sc, Seconds(1)) // Uses the context automatically created by the spark shell

    Spark allows you to specify the batch duration in milliseconds, seconds, and minutes.

  3. Import the database-specific functions for StreamingContext, DStream, and RDD objects.

    import com.datastax.spark.connector.streaming._
  4. Create the DStream object that will connect to the IP and port of the service providing the data stream.

    val lines = ssc.socketTextStream(<server IP address>, <server port number>)
  5. Count the words in each batch and save the data to the table.

    val words = lines.flatMap(_.split(" "))
    val pairs = => (word, 1))
    val wordCounts = pairs.reduceByKey(_ + _)
        .saveToCassandra("streaming_test", "words_table", SomeColumns("word", "count"))
  6. Start the computation.



In the following example, you start a service using the nc utility that repeats strings, then consume the output of that service using Spark Streaming.

Using cqlsh, start by creating a target keyspace and table for streaming to write into.

WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1 };

CREATE TABLE IF NOT EXISTS streaming_test.words_table

In a terminal window, enter the following command to start the service:

nc -lk 9999
one two two three three three four four four four someword

In a different terminal start a Spark shell.

dse spark

In the Spark shell enter the following:

import org.apache.spark.streaming._
import com.datastax.spark.connector.streaming._

val ssc =  new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream( "localhost", 9999)
val words = lines.flatMap(_.split( " "))
val pairs = => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.saveToCassandra( "streaming_test",  "words_table", SomeColumns( "word",  "count"))

Using cqlsh connect to the streaming_test keyspace and run a query to show the results.

cqlsh -k streaming_test
select * from words_table;
word    | count
   three |     3
     one |     1
     two |     2
    four |     4
someword |     1

What’s next

Run the http_receiver demo. See the Spark Streaming Programming Guide for more information, API documentation, and examples on Spark Streaming.

Creating a Spark Structured Streaming sink using DSE

A Spark Structured Streaming sink pulls data into DSE.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000,