Configuring DSE Graph Loader
Configuring DSE Graph Loader files.
Before loading data using any of the methods detailed in the next topics, decide which configuration items to include in the mapping script.
The configuration settings can be applied in the command line using a "-" command,
like -read_threads
, or the settings can be included in the mapping
script. All configuration settings are shown in the DSE Graph Loader reference including security options.
Procedure
-
The
dryrun
setting will run the DSE Graph Loader with a mapping script, and output the results, but will not execute the loading process. It is useful for spotting potential errors in the mapping script or graphloader command.
This command may be more useful to use as a command line option, since it is not common to leave in after checking a mapping script:config dryrun: true
graphloader map.groovy -graph food -address localhost -dryrun true
Note: This configuration option discovers schema and suggests missing schema without executing any changes. In DSE 6.0, this option is deprecated and may possibly be removed in a future release. -
The
preparation
setting is a validity checking mechanism. Ifpreparation
is true, then a sample of the data is analyzed for whether or not the schema is valid. This setting is used in conjunction withcreate_schema
. Ifcreate_schema
andpreparation
are both true, then the data is analyzed, compared to the schema, and new schema is created if found missing.
See the table below for all permutations./* CONFIGURATION */ /* Configures the data loader to analyze the schema */ config preparation: true
Note: This configuration option validates and creates schema if used in conjunction withcreate_schema
. The default will be set tofalse
, and this option is deprecated with DSE 6.0. In a future release, it may be removed. -
This example sets
create_schema
to true, so that schema is created from the data. Settingcreate_schema
to true is a good method of inputting new data, to get feedback on what schema may be required for the data. It is not recommended for Production data loading./* CONFIGURATION */ /* Configures the data loader to create the schema */ config create_schema: true
Note: It is strongly recommended that schema is created prior to data loading, so that the correct data types are enforced and indexes created. Settingcreate_schema
to true is recommended only for testing. In DSE 6.0, this configuration option is deprecated and will be removed in a future release.preparation
andcreate_schema
must be considered togetherpreparation create_schema Results true true Data is analyzed, and if schema is found missing, it is created; loading succeeds. true false Data is analyzed, and if schema is found missing, it is not created; loading fails. If schema is previously loaded manually, including indexes on the vertices loaded, loading succeeds. false true Data is not analyzed, and if schema is missing, it is not created; loading fails. false false Data is not analyzed, and if schema is found missing, it is not created; loading fails. -
The
load_new
setting is used if vertex records do not yet exist in the graph at the beginning of the loading process, such as for a new graph. Configuringload_new
can significantly speed up the loading process. However, it is important that the user guarantee that the vertex records are indeed new, or duplicate vertices can be created in the graph. Edges that are created in the same script will use the newly created vertices for the outgoing vertexoutV
and incoming vertexinV
.config load_new: true
Warning: Duplicate vertices will be created ifload_new
is set tofalse
and the data being loaded contain any vertex that already exists in the graph. -
Setting the number of threads used for loading vertices or edges uses
load_vertex_threads
andload_edge_threads
, respectively; the default is 0, which will setload_vertex_threads
to the number of cores divided by 2, andload_edge_threads
to the number of nodes in the datacenter multiplied by six .config load_vertex_threads: 3 load_edge_threads: 0
-
Multiple configuration settings can be listed together.
config load_new: true, dryrun: true, schema_output: '/tmp/loader_output.txt'