Mapping rules for DseGraphFrame

DseGraphFrame uses mapping rules for column names and types.

Column mapping rules

DataFrame column names are the same as graph property names except in the following cases.

Conflict with column names reserved by GraphFrame will result in an underscore (_) added to the property name. For example, the id column will result in a property named _id.

Reserved column name

id

src

dst

new_id

new_src

new_dst

graphx_attr
DseGraphFrame and Spark SQL are case insensitive by default. Column names that differ only in case will result in conflicts. Set the Spark property spark.sql.caseSensitive=true to avoid case conflicts.
```
dse spark --conf spark.sql.caseSensitive=true
```

Reserved column name
`id`
`src`
`dst`
`new_id`
`new_src`
`new_dst`
`graphx_attr`

Type mapping rules

DseGraphFrame and Spark SQL have a limited set of supported types. A vertex is represented by a Row instance.

If the vertex has multiple properties, each property will be represented as a Spark SQL array with property values. If a property has meta-properties it will be represented as StructType. The value field of the struct contains the property value. All other fields will represent the meta-properties.

DataStax Graph type Spark SQL type Conversion rules

DataStax Graph type	Spark SQL type	Conversion rules
`boolean`	`BooleanType`
`smallint`	`ShortType`
`int`	`IntegerType`
`bigint`	`LongType`
`float`	`FloatType`
`double`	`DoubleType`
`decimal`	`DecimalType(38, 18)`
`varint`	`DecimalType(38, 0)`
`timestamp`	`TimestampType`
`date`	`DateType`
`time`	`LongType`	The number of nanoseconds from the beginning of the day.
`text`	`StringType`
`uuid`	`StringType`	The `UUID.toString()` and `UUID.fromString()` methods are used to convert the value.
`inet`	`StringType`	The `toString` and `InetAddress.getByName()` methods are used to convert the value.
`blob`	`BinaryType`
`'PointType'`	`StringType`	The `toWellKnownText()` and `fromWellKnownText()` methods are used to convert the value.
`'LineStringType'`	`StringType`	The `toWellKnownText()` and `fromWellKnownText()` methods are used to convert the value.
`'PolygonType'`	`StringType`	The `toWellKnownText()` and `fromWellKnownText()` methods are used to convert the value.

boolean

BooleanType

smallint

ShortType

int

IntegerType

bigint

LongType

float

FloatType

double

DoubleType

decimal

DecimalType(38, 18)

varint

DecimalType(38, 0)

timestamp

TimestampType

date

DateType

time

LongType

The number of nanoseconds from the beginning of the day.

text

StringType

uuid

StringType

The UUID.toString() and UUID.fromString() methods are used to convert the value.

inet

StringType

The toString and InetAddress.getByName() methods are used to convert the value.

blob

BinaryType

'PointType'

StringType

The toWellKnownText() and fromWellKnownText() methods are used to convert the value.

'LineStringType'

StringType

The toWellKnownText() and fromWellKnownText() methods are used to convert the value.

'PolygonType'

StringType

The toWellKnownText() and fromWellKnownText() methods are used to convert the value.

Mapping rules for DseGraphFrame

Column mapping rules

Type mapping rules

Was this helpful?

Give Feedback