DataStax Bulk Loader release notes

'--log.checkpoint.enabled'

'--log.checkpoint.file'

'--log.checkpoint.replayStrategy'

At the conclusion of a dsbulk run, results are printed from the Count workflow even if failures occur. In general, when querying billions of rows, some failures are expected. However, reporting these failures may indicate whether there is need to retry the dsbulk run command. (BULK-18)

Upgraded driver to 4.14.0.

When issuing a BATCH query to load data, DSBulk unwraps the statements of this query and incorporates them into its own batching function, creating a consolidated BATCH message. This avoids nesting BATCH statements inside protocol-level BATCH messages, which is forbidden. A consolidated BATCH message also greatly improves performance when loading with timestamp and TTL preservation enabled. (BULK-23)

Added support for Prometheus. (BULK-26)

When unloading data using the options -timestamp or -ttl (to automatically preserve cell timestamp and time-to-live (TTL)), the operation fails if the table being unloaded contains collections. Excluded unsupported types from an automatic timestamp and TTL unload, and log a warning explaining that some timestamps and TTLs may be lost. (BULK-25)

DSBulk distribution archives are now uploaded to Maven Central. (BULK-24)

Removed the check for a primary key column in a record that is empty. This allows the server to handle BLOB types with empty buffers in any primary key column, as well as to handle a composite partition key that accepts empty blobs or strings. (BULK-28)

Added support for nested functions in DSBULK mappings. (BULK-29)

Added support for literal strings in mappings outside of function arguments. (BULK-30)

Support for Well-known Binary (WKB) geometry data:

Added options to automatically preserve Time-To-Live (TTL) and timestamps:

Breaking changes:

Fixed issues:

Introduced a new setting --log.sources, --dsbulk.log.sources boolean that determines whether to print record sources in debug files, and enable "bad files" with load operations. See --dsbulk.log.sources.

Introduced a new setting --monitoring.console, --dsbulk.monitoring.console boolean to enable or disable console reporting. See --dsbulk.monitoring.console.

A clarification regarding writing empty strings as quoted empty fields: To insert an empty string, with the intention to override a value that existed previously in the given column, you can insert an empty quoted field in the data file, as in this CSV example:

foo,"",bar

Fixed null pointer exception when reading from Stdin.

Starting with version 1.6.0, DataStax Bulk Loader is available under the Apache-2.0 license as open-source software (OSS). This change and enhancement makes it possible for the open-source community of developers to contribute features that enable loading and unloading CSV/JSON data to and from Cassandra, DataStax Enterprise (DSE), and Astra DB databases.

The product’s official name is now DataStax Bulk Loader.

The public GitHub repo is https://github.com/datastax/dsbulk.

Some features are specific to DSE environments, including parameters associated with DSE Graph. In topics such as Schema options, Graph-only features are highlighted with an icon.

Ability to specify a list of allowed or denied hosts in the driver’s load balancing policy with the datastax-java-driver.basic.load-balancing-policy.filter.allow and datastax-java-driver.basic.load-balancing-policy.filter.deny options. Load balancing policies aren’t applicable to Astra DB because the load balancing policy is set in the Secure Connect Bundle (SCB).

Introduced a new setting in Engine Options dsbulk.engine.maxConcurrentQueries that regulates the number of queries executed in parallel. It applies to all types of dsbulk operations (load, unload, count). The setting requires a valid integer value greater than zero. The NC notation is also possible, for example, 2C means twice the number of cores.

The setting executor.continuousPaging.maxConcurrentQueries is deprecated. Instead use --dsbulk.engine.maxConcurrentQueries.

In prior releases, the connector.csv.maxConcurrentFiles and connector.json.maxConcurrentFiles settings were for dsbulk unload only. In 1.6.0, you can also use them with dsbulk load, which sets the maximum number of files that can be read in parallel. For important considerations, see connector.csv|json.maxConcurrentFiles.

DataStax Bulk Loader accepts binary input in the following formats: BASE64 or HEX. For dsbulk unload only, you can choose the format when converting binary data to strings. See codec.binary in Codec options.

Raised the driver default timeouts to 5 minutes for the following:

Added support for multi-character delimiters. For example, to load a CSV file with '||', add a -delim '\|\|' parameter on the dsbulk load command. The resulting CSV format is as follows:

Foo||bar||qix

When providing a custom query to dsbulk count, only global is accepted for stats.modes. The query is executed "as is." See Count options.

The ORDER BY, GROUP BY and LIMIT clauses now cause the query to be executed "as is," without parallelization. See Schema options.

Fixed an issue where a zero-length array was exported as "" (empty string). The issue was that an empty string, when reimported to a blob column, was interpreted as null, instead of as a zero-length array.

Per-file limits, maxRecords and skipRecords for CSV and JSON data, were not applied when there was more than one file to read.

Previous DataStax Bulk Loader releases included support for loading and unloading graph data using prior settings and workflows. Starting with this DataStax Bulk Loader 1.5.0 release, the product provides an improved user experience to set related DSE Graph (DSG) properties, plus enhanced validation of requested DSG operations. The changes include new DataStax Bulk Loader schema settings that are specific to DSG operations. The new options are:

With the enhanced support for DSG features, DataStax Bulk Loader displays metrics for graph data in vertices per second or edges per second. For non-graph data, the metrics continue to be displayed in rows per second. The metrics display type occurs automatically based on how the table DDL was created and whether DataStax Bulk Loader schema options include graph settings; that is, you do not need to configure graph-specific monitoring options for the metrics.

DataStax Bulk Loader 1.5.0 adds support for release 4.5.0 of the DataStax Java driver.

Addressed an issue with the deserialization of untrusted data by upgrading to the latest 2.9.10 release of the FasterXML jackson-databind library, although DataStax Bulk Loader was not directly affected.

DataStax Bulk Loader 1.4.1 adds support for using the dsbulk load command to write CSV/JSON data to Cassandra 2.1 and later database tables. Previously, you could only use dsbulk unload and dsbulk count commands with Cassandra.

When exporting data, the \u0000 null character is now enclosed in quotes, so that the exported data can be loaded subsequently with the same DataStax Bulk Loader settings. By default, the null character is used as the comment character.

DataStax Bulk Loader 1.4.0 has been upgraded to use the latest 2.x version of the DataStax Java driver, and many new driver options are available directly with dsbulk commands via the datastax-java-driver prefix.

You can connect DataStax Bulk Loader to an Astra DB database by including the path to the database’s Secure Connect Bundle and providing an application token for the password. For examples, see dsbulk load.

You can use DataStax Bulk Loader to load/unload your table data from/to compressed CSV or JSON files. For details, see the --connector.csv|json.compression option.

The DataStax Bulk Loader Help provides an entry for --version.

Improved error message provided when a row fails to decode. In the DataStax Bulk Loader logging options, the format is: -maxErrors,--log.maxErrors ( number | "N%" ). Where number | N% is the maximum number of errors to allow before aborting the entire operation. This setting may be expressed as an absolute number of errors (an integer greater than or equal to zero) or a percentage of the total rows processed so far (a string of the form "N%", where N is a decimal number between 0 and 100 exclusive). For example, -maxErrors "20%". Setting this value to any negative integer disables the feature, which is not recommended.

When a table contains static columns, it is possible that some partitions only contain static data. In this case, that data is exported as a pseudo row where all clustering columns and regular columns are null. Example:

create table t1 (pk int, cs int static, cc int, v int, primary key (pk, cc));

insert into t1 (pk, cs) values (1,1);

select * from t1;

 pk | cc   | cs | v
----+------+----+------
  1 | null |  1 | null

INSERT INTO t1 (pk, cs) values (:pk, :cs);

Operation LOAD_20190412-134352-912437 failed: Missing required primary key column conversation_id
from schema.mapping or schema.query.

You can use the CQL date and time types with UNITS_SINCE_EPOCH, in addition to timestamp. Previously, you could only use the CQL timestamp type. On the dsbulk command, you can use codec.unit and codec.epoch to convert integers to, or from, these types. Refer to --codec.unit, --dsbulk.codec.unit and --codec.epoch, --dsbulk.codec.epoch.

You can use a new monitoring setting, monitoring.trackBytes, to enable or disable monitoring of DataStax Bulk Loader operations in bytes per second. Because this type of monitoring can consume excessive allocation resources, and in some cases excessive CPU cycles, the setting is disabled by default. If you want monitoring in bytes per second, you must enable it with monitoring.trackBytes. Test and compare the setting in your development environment. Disabling this setting may improve the allocation rate. If leaving the setting disabled improves throughput, consider disabling it (or keeping it disabled) in production. Or enable monitoring of bytes per second on an as-needed basis.

The default output file name format, defined by the --connector.csv|json.fileNameFormat string option, no longer includes the thousands separator. The prior default output file name format was:

In load operations, you can pass the URL of a CSV or JSON data file. In cases where you have multiple URLs, DataStax Bulk Loader 1.3.4 makes this task easier by providing the following command-line options. You can point to the file that contains all the URLs for the data files:

When using REPLICA_SET batch mode, the server may issue query warnings if the number of statements in a single batch exceeds unlogged_batch_across_partitions_warn_threshold. To avoid reporting excessive warning messages in stdout, DataStax Bulk Loader logs only one warning at the beginning of the operation.

DataStax Bulk Loader should reject CSV files containing invalid headers, such as headers that are empty or contain duplicate fields.

Logging option -maxErrors 0 does not abort the operation.

DataStax Bulk Loader should reject invalid execution IDs. An execution ID is used to create MBean names. DataStax Bulk Loader now validates user-provided IDs to ensure, for example, that an ID does not contain a comma.

Previously, exports of varchar column containing JSON could truncate data. Columns of type varchar that contain JSON are now exported "as is," meaning DataStax Bulk Loader does not attempt to parse the JSON payload.

Print basic information about the cluster.

Unload timestamps as units since an epoch.

Provide better documentation on how to choose the best batching strategy.

Implement unload and count for materialized views.

Calculate batch size dynamically with adaptive batch sizing. The new setting batch.maxSizeInBytes defaults to -1 (unlimited).

batch.bufferSize should be a multiple of batch.maxBatchStatements. By default batch.bufferSize is set to 4 times batch.maxBatchStatements if its value is less than or equal to 0.

Improve support for lightweight transactions. DataStax Bulk Loader can detect write failures due to a failed LWT write. Records that could not be inserted will appear in two new files:

Extend DataStax Bulk Loader rate limiting capability to reads. Previously, the rate limiter used by DataStax Bulk Loader, and adjustable via the --executor.maxPerSecond setting, only applied to writes. DataStax Bulk Loader extends the functionality to reads by making it consider the number of received rows instead of the number of requests sent.

Expose settings to control how to interpret empty fields in CSV files. There are two new settings for the CSV connector: nullValue and emptyValue. Previously when reading a CSV file, the connector would emit an empty string when a field was empty and non-quoted. By default, starting with DataStax Bulk Loader 1.3.2, the CSV connector will return a null value in such situations, which may not make a difference in most cases. The only noticeable difference will be for columns of type VARCHAR or ASCII; the resulting stored value will be null instead of an empty string.

Allow functions to appear in mapping variables. Previously for loads only, a mapping entry could contain a function on the right side of the assignment. This functionality has been extended to unloads. For example, loads may continue to use now() = column1, and the result of the now() function is inserted into column1 for every row. For unloads, you can export the result of now() as fieldA for every row read, such as fieldA = now().

Detect writetime variable when unloading. You can specify a writetime function in a mapping definition when unloading. For example, if fieldA = column1, fieldB = writetime(column1), because the data type is detected, fieldB is exported as a timestamp, not as an integer.

Relax constraints on queries for the Count workflow. The schema.query setting can contain any SELECT clause when counting rows.

Automatically add token range restriction to WHERE clauses. When a custom query is provided with --schema.query, to enable read parallelization, it is no longer necessary to provide a WHERE clause using the form WHERE token(pk) > :start AND token(pk) ⇐ :end. If the query does not contain a WHERE clause, DataStax Bulk Loader automatically generates that WHERE clause. However, if the query contains a WHERE clause, DataStax Bulk Loader is not be able to parallelize the read operations.

Allow JSON array mapping with UDTs. Previously, when loading User Defined Types (UDTs) it was required that the input be a JSON object to allow for field-by-field mapping. Starting with DataStax Bulk Loader 1.3.2, a JSON array can also be mapped to UDTs, in which case the mapping is based on field order.

Improve WHERE clause token range restriction detection. When you provide a custom query for unloading, the token range restriction variables can have any name, not only start and end. For example, the following is valid: SELECT * FROM table1 WHERE token(pk) > :foo AND token(pk) ⇐ :bar

Remove record location URI. DataStax Bulk Loader previously provided a record’s URI to uniquely identify the record. However, the URI was very long and difficult to read. You can instead identify a failed record by looking into the record’s source statement or row.

Allow columns and fields to be mapped more than once. It is possible to map a field/column more than once. The following rules apply:

UDT and tuple codecs should respect allowExtraFields and allowMissingFields.

Add support for DSE 4.8 and lower. All protocol versions are supported. Some features might not be available depending on the protocol version and server version. The schema.splits (default: 8C) setting was added to compensate for the absence of paging in C* 1.2. The token ring is split into small chunks and is controlled by this setting. For example:

bin/dsbulk unload -url myData.csv --driver.pooling.local.connections 8 \
  --driver.pooling.local.requests 128 --driver.pooling.remote.requests 128 \
  --schema.splits 0.5C -k test -t test

Add support for keyspace-qualified UDFs in mappings. If needed, you can qualify a user-defined function (UDF) with a keyspace name. For example: fieldA = ks1.func1(column1, column2).

Allow fields to appear as function parameters on the left side of mapping entries. When loading, a mapping entry can contain a function on the left side that references fields of the dataset. For example, consider the case where:

Improve handling of search queries. You can supply a DSE search predicate using the solr_query mechanism. For example, assume you create a search index on the dsbulkblog.iris_with_id table:

cqlsh -e "CREATE SEARCH INDEX IF NOT EXISTS ON dsbulkblog.iris_with_id"

dsbulk unload -query "SELECT id, petal_length, petal_width, \
 sepal_length, sepal_width, species FROM dsbulkblog.iris_with_id \
  WHERE solr_query = '{\\\"q\\\": \\\"species:Iris-setosa\\\"}'"

Ability to hard-limit the number of concurrent continuous paging sessions. DataStax Bulk Loader adds a new setting: executor.continuousPaging.maxConcurrentQueries (Default: 60). It sets the maximum number of concurrent continuous paging queries that should be carried in parallel. Set this number to a value equal to or less than the value configured server-side for continuous_paging.max_concurrent_sessions in the cassandra.yaml configuration file, which is also 60 by default. Otherwise some requests may be rejected. You can disable executor.continuousPaging.maxConcurrentQueries by assigning any negative value or 0.

Ability to skip unloading or loading the solr_query column. DataStax Bulk Loader will skip the solr_query column when loading and unloading.

Setting executor.maxInFlight to a negative value triggers fatal error.

Murmur3TokenRangeSplitter should allow long overflows when splitting ranges.

CSV connector trims trailing white space when reading data.

Avoid overflows in CodecUtils.numberToInstant.

Call to ArrayBackedRow.toString() causes fatal NPE.

Improve range split algorithm in multi-DC and vnodes environments.

Support simplified notation for JSON arrays and objects in collection fields.

CSVWriter trims leading/trailing whitespace in values.

CSV connector fails when the number of columns in a record is greater than 512.

DSBulk fails when mapping contains a primary key column mapped to a function.

Combine batch.mode and batch.enabled into a single setting: batch.mode. If you are using the batch.enabled setting in scripts, change to batch.mode with value DISABLED.

Improve handling of Univocity exceptions.

Logging improvements:

New count workflow:

Counter tables are supported for load and unload.

Improve validation to include user-supplied queries and mappings.

The codec.timestamp CQL_DATE_TIME setting is renamed to CQL_TIMESTAMP. Adjust scripts to use the new setting.

Generated query does not contain all token ranges when a range wraps around the ring.

Empty map values do not work when loading using dsbulk.

DSBulk cannot handle columns of type list<timestamp>.

Generated queries do not respect indexed mapping order.

DSBulk fails to start with Java 10+.

DataStax Bulk Loader 1.0.2 is bundled with DSE 6.0.1.

Configure whether to use ANSI colors and other escape sequences in log messages printed to standard output and standard error.

DataStax Bulk Loader version 1.0.1 is automatically installed with DataStax Enterprise (DSE), and can also be installed as a standalone tool. DataStax Bulk Loader 1.0.1 is supported for use with DSE 5.0 and later.

Support to manage special characters on the command line and in the configuration file.

Improve error messages for incorrect mapping.

Improved monitoring options.

Detect console width on Windows.

Null words are supported by all connectors. The schema.nullStrings is changed to codec.nullWords. Renamed the convertTo and convertFrom methods. See Codec options and Schema options.

Use Logback to improve filtering to make stack traces more readable and useful. On ANSI-compatible terminals, the date prints in green, the hour in cyan, the level is blue (INFO) or red (WARN), and the message prints in black.

Improved messaging for completion with errors.

Settings schema.allowExtraFields and schema.allowMissingFields are added to reference.conf.

Support is dropped for using :port to specify the port to connect to. Specify the port for all hosts only with driver.port.

Numeric overflows should display the original input that caused the overflow.

Null words are not supported by all connectors.

Addresses might not be properly translated when the cluster has a custom native port.