Using DSE geometric types in Spark

DSE geometric types can be used in Spark.

About this task

DSE geometric types can be used in DSE Spark. This example shows how to create a table containing geometric types using CQL, and then read and write to this table in the DSE Spark shell.

Procedure

Start cqlsh.
```
cqlsh
```

Create the test keyspace for the geometric data.

CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

Create the table to store the geometric data.

CREATE TABLE IF NOT EXISTS test.geo (
    k INT PRIMARY KEY,
    pnt 'PointType',
    line 'LineStringType',
    poly 'PolygonType');

Insert a test row.

INSERT INTO test.geo (k, pnt, line, poly) VALUES
    (1,
    'POINT (1.1 2.2)',
    'LINESTRING (30 10, 10 30, 40 40)',
    'POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))');

Exit cqlsh.
```
QUIT;
```
Start a Spark shell.
```
dse spark
```

Import the geometric types, type converters, and save modes.

import com.datastax.driver.dse.geometry._
import com.datastax.spark.connector.types.DseTypeConverter.{LineStringConverter, PointConverter, PolygonConverter}
import org.apache.spark.sql.SaveMode

Read the Point value from the test row in test.geo.

val results = spark.read.cassandraFormat("geo","test").load().select("pnt").collect()
val point1 = Point.fromWellKnownText(results(0).getString(0))

Write a Polygon value to a new row in test.geo.

val polygon1 = new Polygon(
    new Point(30, 10),
    new Point(40, 40),
    new Point(20, 40),
    new Point(10, 20),
    new Point(30, 10))
val df = spark.createDataFrame(Seq((2, polygon1.toString))).select(col("_1") as "k", col("_2") as "poly")
df.write.mode(SaveMode.Append).cassandraFormat("geo", "test").save()

Check that the value was written to the test.geo table.

spark.read.cassandraFormat("geo","test").load().show()

+---+--------------------+---------------+--------------------+
|  k|                line|            pnt|                poly|
+---+--------------------+---------------+--------------------+
|  1|LINESTRING (30 10...|POINT (1.1 2.2)|POLYGON ((30 10, ...|
|  2|                null|           null|POLYGON ((30 10, ...|
+---+--------------------+---------------+--------------------+

Using DSE geometric types in Spark

About this task

Procedure

Was this helpful?

Give Feedback