Use configuration files for DataStax Bulk Loader

You can pass all DataStax Bulk Loader options as command line arguments or in configuration files. Using one or more configuration files is often easier than passing all configuration options on the command line.

Options set on the command line override options set in configuration files.

Load configuration files from the default location

By default, configuration values are loaded from the conf directory of your dsbulk installation.

This directory includes empty application.conf and driver.conf files that you can customize for your environment. application.conf is the main configuration file for DSBulk options. It includes a reference to driver.conf, which is used to configure the Java driver options.

For commented example files, see application.template.conf and driver.template.conf.

Load configuration files from a custom location

To load a custom configuration file from another location, use the -f option to specify the path to the file on the command line:

dsbulk count -f /path/to/custom_config.conf

Configuration file syntax

Use long form option names

When setting options in configuration files, you must use the long form for the option names, which includes a prefix.

For most options the prefix is dsbulk., such as dsbulk.connector.csv.recursive = true.

For the driver options, the prefix is datastax-java-driver, such as datastax-java-driver.basic.request.consistency = "QUORUM".

Configuration files won’t work with short forms or standard forms without the proper prefix.

Use HOCON syntax

The structure and contents of DSBulk configuration files must comply with HOCON syntax.

This syntax is flexible, and it allows sections to be grouped together with blocks, dot notation, or both. For example:

Nested block notation

dsbulk {
  connector {
    name = "csv"
      csv {
        url = "C:\\Users\\My Folder"
        delimiter = "\t"
      }
   }
 }

Dot notation

dsbulk.connector.name = "csv"
dsbulk.connector.csv.url = "C:\\Users\\My Folder"
dsbulk.connector.csv.delimiter = "\t"

Combined block and dot notation

dsbulk {
   connector.name = "csv"
   connector.csv.delimiter = "|"
   schema.keyspace = "myKeyspace"
   schema.table = "myTable"
   schema.mapping = "0=name, 1=age, 2=email"
}

For information about HOCON syntax for configuration values, see Escape and quote command line arguments.

Use file inclusions for multiple configuration files

If you split your configuration across multiple files, then you must use HOCON-formatted file inclusions in the primary configuration file. For example:

include classpath("other-conf-file.conf")

Prompt for passwords

You can use the --runner.promptForPasswords (--dsbulk.runner.promptForPasswords) option to control whether DSBulk prompts for passwords that are missing from configuration files:

true (default): If a username is present in the configuration, but the corresponding password is missing, DSBulk prompts the user to enter the password on the command line.

Prompting for passwords requires an interactive shell. If the standard input isn’t connected to a terminal, DSBulk doesn’t prompt for passwords, even if this option is enabled.

Because DSBulk bypasses password prompting when it’s impossible, only disable this option if DSBulk assumes incorrectly that it is running in an interactive shell.
false: DSBulk never prompts for missing passwords.

Use configuration files for DataStax Bulk Loader

Load configuration files from the default location

Load configuration files from a custom location

Configuration file syntax

Prompt for passwords

Was this helpful?

Give Feedback