Mapping rules for DseGraphFrame

DseGraphFrame uses mapping rules for column names and types.

Column mapping rules

DataFrame column names are the same as graph property names except in the following cases.

  • Conflict with column names reserved by GraphFrame will result in an underscore (_) added to the property name. For example, the id column will result in a property named _id.

Reserved column name
Reserved column name

id

src

dst

new_id

new_src

new_dst

graphx_attr

  • DseGraphFrame and Spark SQL are case insensitive by default. Column names that differ only in case will result in conflicts. Set the Spark property spark.sql.caseSensitive=true to avoid case conflicts.

    $ dse spark --conf spark.sql.caseSensitive=true

Type mapping rules

DseGraphFrame and Spark SQL have a limited set of supported types. A vertex is represented by a Row instance.

If the vertex has multiple properties, each property will be represented as a Spark SQL array with property values. If a property has meta-properties it will be represented as StructType. The value field of the struct contains the property value. All other fields will represent the meta-properties.

DSE Graph to Spark SQL and DseGraphFrame type mapping
DSE Graph type Spark SQL type Conversion rules

boolean

BooleanType

smallint

ShortType

int

IntegerType

bigint

LongType

float

FloatType

double

DoubleType

decimal

DecimalType(38, 18)

varint

DecimalType(38, 0)

timestamp

TimestampType

date

DateType

time

LongType

The number of nanoseconds from the beginning of the day.

text

StringType

uuid

StringType

The UUID.toString() and UUID.fromString() methods are used to convert the value.

inet

StringType

The toString and InetAddress.getByName() methods are used to convert the value.

blob

BinaryType

'PointType'

StringType

The toWellKnownText() and fromWellKnownText() methods are used to convert the value.

'LineStringType'

StringType

The toWellKnownText() and fromWellKnownText() methods are used to convert the value.

'PolygonType'

StringType

The toWellKnownText() and fromWellKnownText() methods are used to convert the value.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com