Mapping rules for DseGraphFrame

DseGraphFrame uses mapping rules for column names and types.

DseGraphFrame uses mapping rules for column names and types.

Column mapping rules

DataFrame column names are the same as graph property names except in the following cases.

  • Conflict with column names reserved by GraphFrame will result in an underscore (_) added to the property name. For example, the id column will result in a property named _id.
    Table 1. Reserved column names in GraphFrame
    Reserved column name
    id
    src
    dst
    new_id
    new_src
    new_dst
    graphx_attr
  • DseGraphFrame and Spark SQL are case insensitive by default. Column names that differ only in case will result in conflicts. Set the Spark property spark.sql.caseSensitive=true to avoid case conflicts.
    dse spark --conf spark.sql.caseSensitive=true

Type mapping rules

DseGraphFrame and Spark SQL have a limited set of supported types. A vertex is represented by a Row instance.

If the vertex has multiple properties, each property will be represented as a Spark SQL array with property values. If a property has meta-properties it will be represented as StructType. The value field of the struct contains the property value. All other fields will represent the meta-properties.

Table 2. DataStax Graph to Spark SQL and DseGraphFrame type mapping
DataStax Graph type Spark SQL type Conversion rules
boolean BooleanType
smallint ShortType
int IntegerType
bigint LongType
float FloatType
double DoubleType
decimal DecimalType(38, 18)
varint DecimalType(38, 0)
timestamp TimestampType
date DateType
time LongType The number of nanoseconds from the beginning of the day.
text StringType
uuid StringType The UUID.toString() and UUID.fromString() methods are used to convert the value.
inet StringType The toString and InetAddress.getByName() methods are used to convert the value.
blob BinaryType
'PointType' StringType The toWellKnownText() and fromWellKnownText() methods are used to convert the value.
'LineStringType' StringType The toWellKnownText() and fromWellKnownText() methods are used to convert the value.
'PolygonType' StringType The toWellKnownText() and fromWellKnownText() methods are used to convert the value.