DataStax Bulk Loader Release Notes

Added the ability to resume a failed operation by using checkpoint files. Three new options are available to use; '--log.checkpoint.enabled', '--log.checkpoint.file', and '--log.checkpoint.replayStrategy'.

Adjusted the default throughput from DataStax Bulk Loader to Astra DataBase to a more conservative setting in order to avoid triggering a rate limit exception.

1.9.0 Changes and enhancements

At the conclusion of a dsbulk run, results are printed from the Count workflow even if failures occur. In general, when querying billions of rows, some failures are expected. However, reporting these failures may indicate whether there is need to retry the dsbulk run command. BULK-18

Upgraded driver to 4.14.0.

When issuing a BATCH query to load data, DSBulk unwraps the statements of this query and incorporates them into its own batching function, creating a consolidated BATCH message. This avoids nesting BATCH statements inside protocol-level BATCH messages, which is forbidden. A consolidated BATCH message also greatly improves performance when loading with timestamp and TTL preservation enabled. BULK-23

Added support for Prometheus. BULK-26

When unloading data using the options -timestamp or -ttl (to automatically preserve cell timestamp and time-to-live (TTL)), the operation fails if the table being unloaded contains collections. Excluded unsupported types from an automatic timestamp and TTL unload, and log a warning explaining that some timestamps and TTLs may be lost. BULK-25

DSBulk distribution archives are now uploaded to Maven Central, allowing DSBulk distros to be downloaded from there as well from the DataStax downloads site). BULK-24

Removed the check for a primary key column in a record that is empty. This allows the server to handle BLOB types with empty buffers in any primary key column, as well as to handle a composite partition key that accepts empty blobs or strings. BULK-28

Added support for nested functions in DSBULK mappings. BULK-29

Added support for literal strings in mappings outside of function arguments. BULK-3

1.8.0 Changes and enhancements

1.8.0 Resolved issues

New support in DataStax Bulk Loader 1.8.0 for accepting Well-known Binary (WKB) geometry data, along with a new Codec setting.

Added options to automatically preserve Time-To-Live (TTL) and timestamps.

Fixed issues identified with Geo data. An example: "Could not deserialize column s_geo of type Set(Custom(org.apache.cassandra.db.marshal.PointType), not frozen) as java.lang.String" was returned while unloading/loading a table with PointType. See the related 1.8.0 implementation of a new setting that addressed the issue, --codec.geo, which is also summarized.

Corrected several occurrences of a documentation typo that previously showed --stats.mode. The correct option for the dsbulk count command is --stats.modes or --dsbulk.stats.modes. Meaning, the plural form is correct. See Count options.

A new setting: --log.sources, --dsbulk.log.sources boolean

A new setting: --monitoring.console, --dsbulk.monitoring.console boolean

A clarification regarding writing empty strings as quoted empty fields. To insert an empty string, with the intention to override a value that existed previously in the given column, you can insert an empty quoted field in the data file, as in this CSV example:

foo,"",bar

Fixed null pointer exception when reading from Stdin.

Starting with version 1.6.0, DataStax Bulk Loader is available under the Apache-2.0 license as open-source software (OSS). This change and enhancement makes it possible for the open-source community of developers to contribute features that enable loading and unloading CSV/JSON data to and from Apache Cassandra® , DataStax Enterprise (DSE), and DataStax Astra databases.

The product’s official name is now DataStax Bulk Loader®.

The public GitHub repo is https://github.com/datastax/dsbulk.

Some features are specific to DSE environments, including parameters associated with DataStax Graph. In topics such as Schema options, Graph-only features are highlighted with an icon.

Ability to specify a list of allowed or denied hosts. Use the following settings:

A new setting in Engine Options: dsbulk.engine.maxConcurrentQueries.

The setting executor.continuousPaging.maxConcurrentQueries is deprecated. Instead use --dsbulk.engine.maxConcurrentQueries.

In prior releases, the following settings were for dsbulk unload only:

DataStax Bulk Loader accepts binary input in the following formats: BASE64 or HEX. For dsbulk unload only, you can choose the format when converting binary data to strings. See codec.binary in Codec options.

Raised the driver default timeouts to 5 minutes for the following:

Added support for multi-character delimiters.

When providing a custom query to dsbulk count, only the stats.modes global is now accepted. The query is executed "as is." See Count options.

The ORDER BY, GROUP BY and LIMIT clauses now cause the query to be executed "as is," without parallelization. See Schema options.

Fixed an issue where a zero-length array was exported as "" (empty string). The issue was that an empty string, when reimported to a blob column, was interpreted as null, instead of as a zero-length array.

Per-file limits, maxRecords and skipRecords for CSV and JSON data, were not applied when there was more than one file to read.

Previous DataStax Bulk Loader releases included support for loading and unloading graph data using prior settings and workflows. Starting with this DataStax Bulk Loader 1.5.0 release, the product provides an improved user experience to set related DataStax Graph (DSG) properties, plus enhanced validation of requested DSG operations. The changes include new DataStax Bulk Loader schema settings that are specific to DSG operations. The new options are:

With the enhanced support for DSG features, DataStax Bulk Loader displays metrics for graph data in vertices per second or edges per second. For non-graph data, the metrics continue to be displayed in rows per second. The metrics display type occurs automatically based on how the table DDL was created and whether DataStax Bulk Loader schema options include graph settings; that is, you do not need to configure graph-specific monitoring options for the metrics.

DataStax Bulk Loader 1.5.0 adds support for release 4.5.0 of the DataStax Java driver.

DataStax Enterprise (DSE) 5.1 and 6.8 databases

DataStax Astra DB databases

Loading data examples

Unloading data examples

Count options

Many new driver options are available directly with dsbulk commands via the datastax-java-driver prefix.

You can connect DataStax Bulk Loader to a cloud-based DataStax Astra database by including the path to the secure connect bundle, and by specifying the username and password entered when the database was created.

You can use DataStax Bulk Loader to load/unload your table data from/to compressed CSV or JSON files.

The DataStax Bulk Loader Help provides an entry for --version.

Improved error message provided when a row fails to decode.

When a table contains static columns, it is possible that some partitions only contain static data. In this case, that data is exported as a pseudo row where all clustering columns and regular columns are null. Example:

create table t1 (pk int, cs int static, cc int, v int, primary key (pk, cc));

insert into t1 (pk, cs) values (1,1);

select * from t1;

 pk | cc   | cs | v
----+------+----+------
  1 | null |  1 | null

INSERT INTO t1 (pk, cs) values (:pk, :cs);

Operation LOAD_20190412-134352-912437 failed: Missing required primary key column conversation_id
from schema.mapping or schema.query.

You can use the CQL date and time types with UNITS_SINCE_EPOCH, in addition to timestamp. Previously, you could only use the CQL timestamp type. On the dsbulk command, you can use codec.unit and codec.epoch to convert integers to, or from, these types. Refer to --codec.unit, --dsbulk.codec.unit and --codec.epoch, --dsbulk.codec.epoch.

You can use a new monitoring setting, monitoring.trackBytes, to enable or disable monitoring of DataStax Bulk Loader operations in bytes per second. Because this type of monitoring can consume excessive allocation resources, and in some cases excessive CPU cycles, the setting is disabled by default. If you want monitoring in bytes per second, you must enable it with monitoring.trackBytes. Test and compare the setting in your development environment. Disabling this setting may improve the allocation rate. If leaving the setting disabled improves throughput, consider disabling it (or keeping it disabled) in production. Or enable monitoring of bytes per second on an as-needed basis.

The default output file name format, defined by the --connector.(csv|json).fileNameFormat string option, no longer includes the thousands separator. The prior default output file name format was:

In load operations, you can pass the URL of a CSV or JSON data file. In cases where you have multiple URLs, DataStax Bulk Loader 1.3.4 makes this task easier by providing the following command-line options. You can point to the file that contains all the URLs for the data files:

When using REPLICA_SET batch mode, the server may issue query warnings if the number of statements in a single batch exceeds unlogged_batch_across_partitions_warn_threshold. To avoid reporting excessive warning messages in stdout, DataStax Bulk Loader logs only one warning at the beginning of the operation.

DataStax Bulk Loader should reject CSV files containing invalid headers, such as headers that are empty or contain duplicate fields.

Logging option -maxErrors 0 does not abort the operation.

DataStax Bulk Loader should reject invalid execution IDs. An execution ID is used to create MBean names. DataStax Bulk Loader now validates user-provided IDs to ensure, for example, that an ID does not contain a comma.

Print basic information about the cluster.

Unload timestamps as units since an epoch.

Provide better documentation on how to choose the best batching strategy.

Implement unload and count for materialized views.

Calculate batch size dynamically - Adaptive Batch Sizing.

batch.bufferSize should be a multiple of batch.maxBatchStatements.

Improve support for lightweight transactions.

Extend DataStax Bulk Loader rate limiting capability to reads.

Expose settings to control how to interpret empty fields in CSV files.

Allow functions to appear in mapping variables.

now() = column1

fieldA = now()

Detect writetime variable when unloading.

fieldA = column1, fieldB = writetime(column1)

Relax constraints on queries for the Count workflow.

Automatically add token range restriction to WHERE clauses.

WHERE token(pk) > :start AND token(pk) <= :end

Should allow JSON array mapping with UDTs.

Improve WHERE clause token range restriction detection.

SELECT * FROM table1 WHERE token(pk) > :foo AND token(pk) <= :bar

Remove record location URI.

Allow columns and fields to be mapped more than once.

UDT and tuple codecs should respect allowExtraFields and allowMissingFields.

Add support for DataStax Enterprise 4.8 and lower.

bin/dsbulk unload -url myData.csv --driver.pooling.local.connections 8 \
  --driver.pooling.local.requests 128 --driver.pooling.remote.requests 128 \
  --schema.splits 0.5C -k test -t test

Add support for keyspace-qualified UDFs in mappings.

Allow fields to appear as function parameters on the left side of mapping entries.

fieldA = colA, fieldB = colB, sum(fieldA,fieldB)=colSum

Improve handling of search queries.

cqlsh -e "CREATE SEARCH INDEX IF NOT EXISTS ON dsbulkblog.iris_with_id"

dsbulk unload -query "SELECT id, petal_length, petal_width, \
 sepal_length, sepal_width, species FROM dsbulkblog.iris_with_id \
  WHERE solr_query = '{\\\"q\\\": \\\"species:Iris-setosa\\\"}'"

Ability to hard-limit the number of concurrent continuous paging sessions.

Ability to skip unloading or loading the solr_query column.

Setting executor.maxInFlight to a negative value triggers fatal error.

Murmur3TokenRangeSplitter should allow long overflows when splitting ranges.

CSV connector trims trailing white space when reading data.

Avoid overflows in CodecUtils.numberToInstant.

Call to ArrayBackedRow.toString() causes fatal NPE.

Improve range split algorithm in multi-DC and vnodes environments.

Support simplified notation for JSON arrays and objects in collection fields.

CSVWriter trims leading/trailing whitespace in values.

CSV connector fails when the number of columns in a record is greater than 512.

Bulk Loader fails when mapping contains a primary key column mapped to a function.

Combine batch.mode and batch.enabled into a single setting: batch.mode. If you are using the batch.enabled setting in scripts, change to batch.mode with value DISABLED.

Improve handling of Univocity exceptions.

Logging improvements.

New count workflow.

Counter tables are supported for load and unload.

Improve validation to include user-supplied queries and mappings.

The codec.timestamp CQL_DATE_TIME setting is renamed to CQL_TIMESTAMP. Adjust scripts to use the new setting.

Generated query does not contain all token ranges when a range wraps around the ring.

Empty map values do not work when loading using dsbulk.

DSBulk cannot handle columns of type list<timestamp>.

Generated queries do not respect indexed mapping order.

DSBulk fails to start with Java 10+.

DataStax Bulk Loader 1.0.2 is bundled with DSE 6.0.1.

Configure whether to use ANSI colors and other escape sequences in log messages printed to standard output and standard error.

1.0.1 Changes and enhancements

1.0.1 Resolved issues

DataStax Bulk Loader (dsbulk) version 1.0.1 is automatically installed with DataStax Enterprise, and can also be installed as a standalone tool. DataStax Bulk Loader 1.0.1 is supported for use with DSE 5.0 and later.

Support to manage special characters on the command line and in the configuration file.

Improve error messages for incorrect mapping.

Improved monitoring options.

Detect console width on Windows.

Null words are supported by all connectors. The schema.nullStrings is changed to codec.nullWords. Renamed the convertTo and convertFrom methods. See Codec options and Schema options.

Use Logback to improve filtering to make stack traces more readable and useful. On ANSI-compatible terminals, the date prints in green, the hour in cyan, the level is blue (INFO) or red (WARN), and the message prints in black.

Improved messaging for completion with errors.

Settings schema.allowExtraFields and schema.allowMissingFields are added to reference.conf.

Support is dropped for using :port to specify the port to connect to. Specify the port for all hosts only with driver.port.

Numeric overflows should display the original input that caused the overflow.

Null words are not supported by all connectors.

Addresses might not be properly translated when the cluster has a custom native port.