Loading CSV data
How to use the DSE Graph Loader to load CSV data.
A common file format for loading graph data is CSV (comma-delimited data). An input
CSV file generally identifies the property keys in the first line of the file with a
header line. However, the mapping script can also identify the property keys to be
read with header()
in the data input line. If more flexibility is
desired, such as manipulation of the vertex labels using labelField, use
Loading TEXT data.
Mapping several different CSV files
Mapping several different CSV files with DSE Graph Loader.
SAMPLE INPUT
// For the author.csv file:
name|gender
Julia Child|F
// For the book.csv file:
name|year|ISBN
Simca's Cuisine: 100 Classic French Recipes for Every Occasion|1972|0-394-40152-2
// For the authorBook.csv file:
bname|aname
Simca's Cuisine: 100 Classic French Recipes for Every Occasion|Simone Beck
Because
the property key name
is used for both vertex labels
author
and book
, in the
authorBook
file, variables aname
and
bname
are used for author name and book name, respectively.
These variables are used in the mapping logic used to create the edges between
author
and book
vertices.Procedure
Example
/* SAMPLE INPUT
author: Julia Child|F
book: Simca's Cuisine: 100 Classic French Recipes for Every Occasion|1972|0-394-40152-2
authorBook: Simca's Cuisine: 100 Classic French Recipes for Every Occasion|Simone Beck
*/
// CONFIGURATION
// Configures the data loader to create the schema
config create_schema: true, load_new: true, load_vertex_threads: 3
// DATA INPUT
// Define the data input source (a file which can be specified via command line arguments)
// inputfiledir is the directory for the input files
inputfiledir = '/tmp/CSV/'
authorInput = File.csv(inputfiledir + "author.csv").delimiter('|')
bookInput = File.csv(inputfiledir + "book.csv").delimiter('|')
authorBookInput = File.csv(inputfiledir + "authorBook.csv").delimiter('|')
//Specifies what data source to load using which mapper (as defined inline)
load(authorInput).asVertices {
label "author"
key "name"
}
load(bookInput).asVertices {
label "book"
key "name"
}
load(authorBookInput).asEdges {
label "authored"
outV "aname", {
label "author"
key "name"
}
inV "bname", {
label "book"
key "name"
}
}
Mapping several files with same format from a directory
Mapping several same format CSV files with DSE Graph Loader.
SAMPLE INPUT
// For the author.csv file:
name|gender
Julia Child|F
Simone Beck|F
// For the knows.csv file:
aname|bname
Julia Child|James Beard
A
number of files with the same format exist in a directory. If the files differ, the
graphloader will issue an error and
stop:java.lang.IllegalArgumentException: /tmp/dirSource/data has more than 1 input type.
Procedure
Mapping files from a directory using a file pattern
Mapping several same format CSV files with DSE Graph Loader.
$ ls data badOne.csv person1.csv person2.csvA number of files with the same format exist in a directory. If the files differ, DSE Graph Loader will only load the files that match the pattern in the map script.
Pattern | Description | Example |
---|---|---|
* | Matches zero or more characters. While matching, it will not cross directory boundaries. | *.csv will match all CSV files ending in csv. |
** | Same as * but will cross directory boundaries. | CSV files in more than one directory. |
? | Matches only one character. | person?.csv will match all CSV files, named as person1.csv or personA.csv, but not person11.csv. |
\ | Avoid characters being interpreted as special characters, e.g. \\ to get a single \ . | |
[ ] | Matches a set of designated characters, though only as ingle character is matched. | [efg] matches "e", "f", or "g", so person[efg] matches persone, personf or persong. [1-9] matches any one number; person[1-9] will get files person1.csv through person9.csv. |
{ } | Matches group of sub-patterns. | {csv,json} will match all CSV and JSON files in the directory. |
Procedure
Mapping using *
Mapping using [ ]
Mapping using { } with multiple patterns