Configuring DSE Graph Loader
Before loading data using any of the methods detailed in the next topics, decide which configuration items to include in the mapping script.
The configuration settings can be applied in the command line using a "-" command, like
-read_threads, or the settings can be included in the mapping script.
All configuration settings are shown in the DSE Graph Loader reference including security options.
dryrunsetting will run the DSE Graph Loader with a mapping script, and output the results, but will not execute the loading process. It is useful for spotting potential errors in the mapping script or graphloader command.
config dryrun: true
This command may be more useful to use as a command line option, since it is not common to leave in after checking a mapping script:
graphloader map.groovy -graph food -address localhost -dryrun true
This configuration option discovers schema and suggests missing schema without executing any changes. In DSE 6.0, this option is deprecated and may possibly be removed in a future release.
preparationsetting is a validity checking mechanism. If
preparationis true, then a sample of the data is analyzed for whether or not the schema is valid. This setting is used in conjunction with
preparationare both true, then the data is analyzed, compared to the schema, and new schema is created if found missing.
/* CONFIGURATION */ /* Configures the data loader to analyze the schema */ config preparation: true
See the table below for all permutations.
This configuration option validates and creates schema if used in conjunction with
create_schema. The default will be set to
false, and this option is deprecated with DSE 6.0. In a future release, it may be removed.
This example sets
create_schemato true, so that schema is created from the data. Setting
create_schemato true is a good method of inputting new data, to get feedback on what schema may be required for the data. It is not recommended for Production data loading.
/* CONFIGURATION */ /* Configures the data loader to create the schema */ config create_schema: true
It is strongly recommended that schema is created prior to data loading, so that the correct data types are enforced and indexes created. Setting
create_schemato true is recommended only for testing. In DSE 6.0, this configuration option is deprecated and will be removed in a future release.
create_schemamust be considered together
Table 1. Schema preperation, creation, and results preparation
Data is analyzed, and if schema is found missing, it is created; loading succeeds.
Data is analyzed, and if schema is found missing, it is not created; loading fails. If schema is previously loaded manually, including indexes on the vertices loaded, loading succeeds.
Data is not analyzed, and if schema is missing, it is not created; loading fails.
load_newsetting is used if vertex records do not yet exist in the graph at the beginning of the loading process, such as for a new graph. Configuring
load_newcan significantly speed up the loading process. However, it is important that the user guarantee that the vertex records are indeed new, or duplicate vertices can be created in the graph. Edges that are created in the same script will use the newly created vertices for the outgoing vertex
outVand incoming vertex
config load_new: true
Duplicate vertices will be created if
load_newis set to
falseand the data being loaded contain any vertex that already exists in the graph.
Setting the number of threads used for loading vertices or edges uses
load_edge_threads, respectively; the default is 0, which will set
load_vertex_threadsto the number of cores divided by 2, and
load_edge_threadsto the number of nodes in the datacenter multiplied by six.
config load_vertex_threads: 3 load_edge_threads: 0
Multiple configuration settings can be listed together.
config load_new: true, dryrun: true, schema_output: '/tmp/loader_output.txt'