DSE Graph Loader reference

The three stages of load processing are:

Preparation

Reads entire input data. This stage either ensures that the data conforms to the graph schema, or the stage updates the graph schema according to the provided data (if enabled). At the end of this stage, statistical estimates are provided on how much data will be added to the graph but no data is loaded. Set

-dryrun true

to abort the loading process after the preparation stage and before any changes are made. Inspect the output and verify that it matches your expectations. For large datasets, doing a dry run is important for spotting errors.

Vertex Loading

The second stage adds or retrieves all of the vertices in the input data and caches them locally to speed up the subsequent edge loading.

Edge and Property Loading

Adds all edges and properties from the input data to the graph.

To get the listing of possible options, use -help.

graphloader -help

This example will use the loading script mymapscript.groovy to read data from a file /tmp/recipe/all.dat into the graph test that is running on the localhost. Dry run is specified to test the loading without inserting the data.

graphloader mymapscript.groovy -filename /tmp/recipe/all.dat -graph test -address localhost -dryrun true 

This example will use the loading script csv2Vertex.groovy to read data from a file MyUsers.csv into the graph csvTest that is running on the localhost. The -label option specifies that the vertex label will be User, rather than the filename MyUsers.

graphloader ./scripts/csv2Vertex.groovy -filename MyUsers.csv -graph csvTest -label User -address 127.0.0.1

The configuration settings can also be specified in the loading script. A fragment of a loading script is shown here that sets create_schema to true and load_vertex_threads to 3.

// CONFIGURATION
// Configures the data loader to create the schema and set load_vertex_threads to 3
config load_new: true, load_vertex_threads: 3

By default, the graphloader logs debug information to the file loader.log in the directory from which graphloader is run. The location of the log can be specified with -load_failure_log:

graphloader mymapscript.groovy -graph test -address localhost -load_failure_log /tmp/dgl.log

If log4j modifications are desired to log information differently, a configuration file can be created, and used in conjunction with the -load_failure_log. Here is a sample configuration file:

# Set root logger level to the designated level and its appenders to F1 and stdout
log4j.rootLogger=INFO, WARN, A1, stdout
#/dev/stdout
# Log INFO messages to A1.  A1 is set to be a ConsoleAppender.
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.Target=System.out
log4j.appender.A1.Threshold=INFO
# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
# Direct INFO log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.Threshold=INFO
# stdout uses PatternLayout.
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

and a sample graphloadercommand:

java -Dlog4j.configuration=file:./lib/log4j.properties -jar graphLoaderJar mymapscript.groovy -graph test -address localhost -load_failure_log /dev/stdout

that will write the log information to stdout.

To use authentication, configure graphloader with -user and -password:

graphloader mymapscript.groovy -graph test -address localhost -username myName -password myPasswd

To configure graphloader with SSL encryption and using Kerberos:

java -Djavax.net.ssl.trustStore=<TRUSTSTORE_PATH> -Djavax.net.ssl.trustStorePassword=<PASSWORD> -Djavax.net.ssl.keyStore=<KEYSTORE_PATH> \
-Djavax.net.ssl.keyStorePassword=<PASSWORD> -jar dse-graph-loader-5.0.3-uberjar.jar -kerberos true -sasl dsename -graph new -address localhost mymapscript.groovy

If the truststore and keystore java options are set in cassandra-env.sh, the command is simplified:

java -jar dse-graph-loader.jar -kerberos true -sasl dsename -graph new -address localhost mymapscript.groovy

If a large data set is loaded, configure the heap space to cache all vertices. This command runs Java and calls the jar file for DSE Graph Loader. For example:

java -Xmx10g -jar dse-graph-loader.jar

Vertex caching uses a temporary directory to store data during loading. If the temporary directory is not large enough, loading is blocked. To change the location of the temporary directory, use a runtime variable LOADER_TMP_DIR:

LOADER_TMP_DIR=/home/user ./graphloader -graph new -address localhost mymapscript.groovy 

When graphloader has successfully loaded the data specified, notification of the results are logged to /var/lib/cassandra/system.log:

2017-02-09 23:27:22 INFO Reporter:97 - Current total additions: 1155735 vertices 1982536 edges 6583940 properties 0 anonymous