Use DSE geometric types in Apache Spark

About this task

DataStax Enterprise (DSE) geometric types can be used in DSE Spark. This example shows how to create a table containing geometric types using CQL, and then read and write to this table in the DSE Spark shell.

Procedure

Start cqlsh.
language-bash
```
cqlsh
```

Create the test keyspace for the geometric data.

language-cql

CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

Create the table to store the geometric data.

language-cql

CREATE TABLE IF NOT EXISTS test.geo (
    k INT PRIMARY KEY,
    pnt 'PointType',
    line 'LineStringType',
    poly 'PolygonType');

Insert a test row.

language-cql

INSERT INTO test.geo (k, pnt, line, poly) VALUES
    (1,
    'POINT (1.1 2.2)',
    'LINESTRING (30 10, 10 30, 40 40)',
    'POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))');

Exit cqlsh.
language-cql
```
QUIT;
```
Start a Spark shell.
language-bash
```
dse spark
```

Import the geometric types, type converters, and save modes.

language-scala

import com.datastax.driver.dse.geometry._
import com.datastax.spark.connector.types.DseTypeConverter.{LineStringConverter, PointConverter, PolygonConverter}
import org.apache.spark.sql.SaveMode

Read the Point value from the test row in test.geo.

language-scala

val results = spark.read.cassandraFormat("geo","test").load().select("pnt").collect()
val point1 = Point.fromWellKnownText(results(0).getString(0))

Write a Polygon value to a new row in test.geo.

language-scala

val polygon1 = new Polygon(
    new Point(30, 10),
    new Point(40, 40),
    new Point(20, 40),
    new Point(10, 20),
    new Point(30, 10))
val df = spark.createDataFrame(Seq((2, polygon1.toString))).select(col("_1") as "k", col("_2") as "poly")
df.write.mode(SaveMode.Append).cassandraFormat("geo", "test").save()

Check that the value was written to the test.geo table.

language-scala

spark.read.cassandraFormat("geo","test").load().show()

results

+---+--------------------+---------------+--------------------+
|  k|                line|            pnt|                poly|
+---+--------------------+---------------+--------------------+
|  1|LINESTRING (30 10...|POINT (1.1 2.2)|POLYGON ((30 10, ...|
|  2|                null|           null|POLYGON ((30 10, ...|
+---+--------------------+---------------+--------------------+

Use DSE geometric types in Apache Spark

About this task

Procedure

Was this helpful?

Give Feedback