Using DSE Advanced Replication

Operations including starting, stopping, and configuring DSE Advanced Replication.

dse.yaml

The location of the dse.yaml file depends on the type of installation:
Package installations /etc/dse/dse.yaml
Tarball installations installation_location/resources/dse/conf/dse.yaml

cassandra.yaml

The location of the cassandra.yaml file depends on the type of installation:
Package installations /etc/dse/cassandra/cassandra.yaml
Tarball installations installation_location/resources/cassandra/conf/cassandra.yaml

Operations including starting, stopping, and configuring DSE Advanced Replication.

  1. Starting DSE Advanced Replication
  2. Stopping DSE Advanced Replication
  3. Configuring global configuration settings
  4. Configuring destination settings
  5. Configuring channel settings
  6. Security
  7. Data insert methods
  8. Monitoring operations

Starting DSE Advanced Replication

Attention: Prerequisite: If you are using Advanced Replication V1 from DSE 5.0, you must upgrade to DSE 5.1 and migrate to Advanced Replication V2.

Before you can start and use DSE Advanced Replication, you must create the user keyspaces and tables on the source cluster and the destination cluster.

On all nodes in the source cluster:
  1. Enable replication in the dse.yaml file.
    Uncomment all advanced_replication_options entries, set enabled: true, and specify a directory to hold advanced replication log files with advanced_replication_directory:
    # Advanced Replication configuration settings
    advanced_replication_options:
      enabled: true
      advanced_replication_directory: /var/lib/cassandra/advrep
  2. Enable Capture-Data-Change (CDC) in the cassandra.yaml file on a per-node basis for each source:
    cdc_enabled: true
    cdc_raw_directory: /var/lib/cassandra/cdc_raw
    Note: Advanced Replication will not start if CDC is not enabled. Either use the default directory or change it to a preferred location.
  3. Consider increasing the default CDC disk space, depending on the load (default: 4096 MB or 1/8 of the total space where cdc_raw_directory resides):
    cdc_total_space_in_mb: 16384
  4. Commitlog compression is turned off by default. To avoid problems with advanced replication, this option should NOT be used:
    # commitlog_compression:
    #   - class_name: LZ4Compressor
  5. Do a rolling restart: restart the nodes in the source cluster one at a time while the other nodes continue to operate online.

Disabling DSE Advanced Replication

When replication is not enabled, data is not written to the replication log. On all nodes in the source cluster:
  1. To disable replication, edit the dse.yaml file.
    In the advanced_replication_options section, set enabled: false.
    # Advanced Replication configuration settings
    advanced_replication_options:
      enabled: false
  2. Do a rolling restart: restart the nodes in the source cluster one at a time while the other nodes continue to operate online.
  3. To clean out the data that was used for DSE Advanced Replication, use cqlsh to remove these keyspaces:
    DROP TABLE dse_system.advrep_source_config;
    DROP TABLE dse_system.advrep_destination_config;
    DROP TABLE dse_system.advrep_repl_channel_config;
    DROP KEYSPACE dse_advrep;

Configuring global configuration settings

Global settings apply to the entire source cluster. These global settings are stored in the CQL table dse_system.advrep_source_config that is automatically created.

Change global settings by using the dse advrep command line tool with this syntax:
dse advrep conf ...
To view the source node configuration settings:
dse advrep conf list
The result is:
-----------------------------------
|name             |value          |
-----------------------------------
|audit_log_file   |/tmp/myaudit.gz|
-----------------------------------
|audit_log_enabled|true           |
-----------------------------------

The following table describes the configuration keys, their default values, and identifies when a restart of the source node is required for the change to be recognized.

The dse advrep command line tool uses these configuration keys as command arguments to the dse advrep command line tool.
Configuration key Default value Description Restart required
permits 30,000 Maximum number of messages that can be replicated in parallel over all destinations. No
source-id N/A Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur. No
collection-expire-after-write N/A
collection-time-slice-count 5 The number of files which are open in the ingestor simultaneously. Yes
collection-time-slice-width 60 seconds The time period in seconds for each data block ingested. Smaller time widths => more files. Larger timer widths => larger files but more data to resend on CRC mismatches. Yes
invalid-message-log SYSTEM_LOG Select one of these logging strategies to adopt when an invalid message is discarded:

SYSTEM_LOG: Log the CQL query and the error message in the system log on the destination.

CHANNEL_LOG: Store the CQL query and the error message in files in /var/lib/cassandra/advrep/invalid_queries on the destination.

NONE: Perform no logging.

See Managing invalid messages. Requires node restart.
No
audit-log-enable false Specifies whether to store the audit log. Yes
audit-log-file /tmp/advrep_rl_audit.log Specifies the file name prefix template for the audit log file. The file name is appended with .gz if compressed using gzip. Yes
audit-log-max-life-span-mins 0
Specifies the maximum lifetime of audit log files. Periodically, when log files are rotated, audit log files are purged when they:
  • Match the audit log file template
  • And they have not been written to for more than the specified maximum lifespan minutes
To disable purging, set to 0.
Yes
audit-log-rotate-time-mins 60 Specifies the time interval to rotate the audit log file. On rotation, the rotated file is appended with the log counter .[logcounter], incrementing from [0]. To disable rotation, set to 0. Yes

Configuring destination settings

A destination is a location to which source data will be written. Destinations are stored in the CQL table dse_system.advrep_destination_config that is automatically created.

Change destination settings by using the dse advrep command line tool with this syntax:
dse advrep destination ...
You can verify the channel configuration before you change it. For example:
dse advrep destination list-conf
The result is:
--------------------------------------------------------------------------------------------
|destination|name                                |value                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_enabled                  |false                                    |
--------------------------------------------------------------------------------------------
|mydest     |addresses                           |10.200.182.251                           |
--------------------------------------------------------------------------------------------
|mydest     |driver_read_timeout                 |15000                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_connections_max              |8                                        |
--------------------------------------------------------------------------------------------
|mydest     |source_id_column                    |source_id                                |
--------------------------------------------------------------------------------------------
|mydest     |driver_connect_timeout              |15000                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_protocol                 |TLS                                      |
--------------------------------------------------------------------------------------------
|mydest     |driver_consistency_level            |QUORUM                                   |
--------------------------------------------------------------------------------------------
|mydest     |driver_used_hosts_per_remote_dc     |0                                        |
--------------------------------------------------------------------------------------------
|mydest     |driver_allow_remote_dcs_for_local_cl|false                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_compression                  |lz4                                      |
--------------------------------------------------------------------------------------------
|mydest     |driver_connections                  |1                                        |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_cipher_suites            |[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,|
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_GCM_SHA384,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_GCM_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,   |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_3DES_EDE_CBC_SHA,           |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_RC4_128_SHA,          |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_SHA,                |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_RC4_128_SHA,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_RC4_128_SHA,           |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_MD5,                |
|           |                                    |,                                        |
|           |                                    |TLS_EMPTY_RENEGOTIATION_INFO_SCSV]       |
--------------------------------------------------------------------------------------------
|mydest     |source_id                           |source1                                  |
--------------------------------------------------------------------------------------------
|mydest     |transmission_enabled                |true                                     |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_enabled                  |false                                    |
--------------------------------------------------------------------------------------------
|llpdest    |addresses                           |10.200.177.184                           |
--------------------------------------------------------------------------------------------
|llpdest    |driver_read_timeout                 |15000                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connections_max              |8                                        |
--------------------------------------------------------------------------------------------
|llpdest    |source_id_column                    |source_id                                |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connect_timeout              |15000                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_protocol                 |TLS                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_consistency_level            |ONE                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_used_hosts_per_remote_dc     |0                                        |
--------------------------------------------------------------------------------------------
|llpdest    |driver_allow_remote_dcs_for_local_cl|false                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_compression                  |lz4                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connections                  |1                                        |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_cipher_suites            |[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,|
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_GCM_SHA384,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_GCM_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,   |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_3DES_EDE_CBC_SHA,           |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_RC4_128_SHA,          |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_SHA,                |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_RC4_128_SHA,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_RC4_128_SHA,           |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_MD5,                |
|           |                                    |,                                        |
|           |                                    |TLS_EMPTY_RENEGOTIATION_INFO_SCSV]       |
--------------------------------------------------------------------------------------------
|llpdest    |source_id                           |source1                                  |
--------------------------------------------------------------------------------------------
|llpdest    |transmission_enabled                |false                                    |
--------------------------------------------------------------------------------------------

The following table describes the configuration keys, their default values, and identifies when a restart of the source node is required for the change to be recognized.

Configuration key Default value Description Restart required
separator N/A Field separator. No
name N/A Name for destination (required). No
addresses none REQUIRED. A comma separated list of IP addresses that are used to connect to the destination cluster using the DataStax Java driver. No
driver-allow-remote-dcs-for-local-cl false Set to true to enable automatic failover for destination clusters with multiple datacenters. The value of the driver-consistency-level parameter must be LOCAL_ONE or LOCAL_QUORUM. Yes
driver-compression lz4 The compression algorithm the DataStax Java driver uses to send data from the source to the destination. Supported values are lz4 and snappy. Yes
driver-connect-timeout 15000 Time in milliseconds the DataStax Java driver waits to connect to a server. No
driver-connections 32 The number of connections the DataStax Java driver will create. Yes
driver-connections-max 256 The maximum number of connections the DataStax Java driver will create. Yes
driver-max-requests-per-connection 1024 The maximum number of requests per connection the DataStax Java driver will create.
driver-consistency-level ONE The consistency level used by the DataStax Java driver when executing statements for replicating data to the destination. Specify a valid DSE level: ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, SERIAL, LOCAL_SERIAL, or LOCAL_ONE. No
driver-local-dc N/A For destination clusters with multiple datacenters, you can explicitly define the name of the datacenter that you consider local. Typically, this is the datacenter that is closest to the source cluster. This value is used only for clusters with multiple data enters. Yes
driver-pwd none Driver password if the destination requires a user and password to connect. Changing the driver-pwd value for connection to a destination will automatically connect, but with a slight delay.
Note: By default, driver user names and passwords are plain text. DataStax recommends encrypting the driver passwords before you add them to the CQL table.
Yes
driver-read-timeout 15000 Time in milliseconds the DataStax Java driver waits to read responses from a server. No
driver-ssl-enabled false Whether SSL is enabled for connection to the destination. Yes
driver-ssl-disabled Disable SSL for connection to the destination.
driver_ssl_keystore_path none The path to the keystore for connection to DSE when SSL client authentication is enabled. Yes
driver_ssl_keystore_password none The keystore password for connection to DSE when SSL client authentication is enabled. Yes
driver_ssl_keystore_type none The keystore type for connection to DSE when SSL client authentication is enabled. Yes
driver_ssl_truststore_path none The path to the truststore for connection to DSE when SSL is enabled. Yes
driver-ssl-truststore-password none The truststore password for connection to DSE when SSL is enabled. Yes
driver-ssl-truststore-type none The keystore type for connection to DSE when SSL client authentication is enabled. Yes
driver-ssl-protocol TLS The SSL protocol for connection to DSE when SSL is enabled. Yes
driver-ssl-cipher-suites none A comma-separated list of SSL cipher suites for connection to DSE when SSL is enabled. Cipher suites must be supported by the source machine. Yes
driver-used-hosts-per-remote-dc 0 To use automatic failover for destination clusters with multiple datacenters, you must define the number of hosts per remote datacenter that the datacenter aware round robin policy (DCAwareRoundRobinPolicy) considers available. Yes
driver-user none Driver username if the destination requires a user and password to connect. Changing the driver-user value for connection to a destination will automatically connect, but with a slight delay. Yes
source-id N/A Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur. No
source-id-column source-id The column to use on remote tables to insert the source id as part of the update. If this column is not present on the table that is being updated, the source id value is ignored. No
transmission-enabled false Specify if data collector for the table should be replicated to the destination using boolean value. No

Configuring channel settings

A replication channel is a defined channel of change data between source clusters and destination clusters.

A replication channel is defined by the source cluster, source keyspace, source table name, destination cluster, destination keyspace, and destination table name. Replications for each channel (unique keyspace and table) are stored in the CQL table dse_system.advrep_repl_channel_config that is automatically created.

Change the settings using the dse advrep command line tool with this syntax:
dse advrep channel ...
You can verify the channel configuration before you change it. For example:
dse advrep channel status
The result is:
------------------------------------------------------------------------------------------------------------------------------------------------------
|dc       |keyspace|table          |collecting|transmitting|replication order|priority|dest ks|dest table     |src id |src id col|dest  |dest enabled|
------------------------------------------------------------------------------------------------------------------------------------------------------
|Cassandra|foo     |bar            |true      |true        |FIFO             |2       |foo   |bar             |source1|source_id |mydest|true        |
------------------------------------------------------------------------------------------------------------------------------------------------------
Properties are continuously read from the metadata, so a restart is not required after configuration changes are made. The following table describes the configuration settings.
Column name Description
separator Field separator.
keyspace The keyspace on the source for the table to replicate.
table The table name on the source to replicate.
source-id Placeholder to override the source-id that is defined in the advrep_conf metadata
source-id-column Placeholder to override the source-id-column that is defined in advrep_conf metadata.
enabled If true, replication will start for this table. If false, no more messages from this table will be saved to the replication log.
data-center-id Datacenter this replication channel is meant for, if none specified the replication will happen in all specified dc1.
destination Destination to which data is written.
destination-keyspace The keyspace on the destination for the replicated table.
destination-table The table name on the destination for the replicated table.
priority Messages are marked by priority in descending order (DESC).
transmission-enabled Specify if the data collector for the table should be replicated to the destination.
fifo-order Specify if the channel should be replicated in FIFO order (default).
lifo-order Specify if the channel should be replicated in LIFO order.

Security

Authentication credentials can be provided in several ways, see .

The user who is doing the replicating with DSE Advanced Replication requires table and keyspace level authorization. If the same user access is required, then ensure that the authorization is the same on the source and destination clusters.

Advanced Replication also supports setting row-level permissions on the destination cluster. The user which connects to the destination cluster must have permission to write to the specified destination table at the row level replicated from the source, according to the RLAC restrictions. The user is specified with the --driver-user destination setting. Row-level access control (RLAC) on the source cluster does not impact Advanced Replication. Because Advanced Replication reads the source data at the raw CDC file layer, it essentially reads as a superuser and has access to all configured data tables.

Advanced Replication supports encrypting the driver passwords. Driver passwords are stored in a CQL table. By default, driver passwords are plain text. DataStax recommends encrypting the driver passwords before you add them to the CQL table. Create a global encryption key, called a system_key for SSTable encryption. Each node in the source cluster must have the same system key. The destination does not require this key.
  1. In the dse.yaml file:
    • Verify that the config_encryption_active property is false:
      config_encryption_active: false
    • Enable driver password encryption with the conf_driver_password_encryption_enabled property:
      conf_driver_password_encryption_enabled: true
    • Define where system keys are stored on disk. The location of the key is specified on the command line with the -d option or with system_key_directory in dse.yaml. The default filepath is /etc/dse/conf.
    • To configure the filename of the generated encryption key, set the config_encryption_key_name option in dse.yaml. The default name is system_key.
  2. Generate a system key:
    On-server:
    dsetool createsystemkey cipher_algorithm strength system_key_file
    Off-server
    dsetool createsystemkey cipher_algorithm strength system_key_file -kmip=kmip_groupname
    For example:
    dsetool createsystemkey 'AES/ECB/PKCS5Padding' 128 system_key_file
    where system_key_file is a unique file name for the generated system key file. See createsystemkey.

    Result: Configure transparent data encryption (TDE) on a per table basis. You can configure encryption with or without compression. You can create a global encryption key in the location that is specified by system_key_directory in the dse.yaml file. This default global encryption key is used when the system_key_file subproperty is not specified.

  3. Copy the returned value.
  4. On any node in the source cluster, use the dse command to set the encrypted password in the DSE Advanced Replication environment:
    dse advrep destination --driver-pwd "Sa9xOVaym7bddjXUT/eeOQ==" --driver-user "username"
  5. Start dse.

SSL configuration and ports

For details about SSL configuration with DSE Advanced Replication, refer to .
Note: Enabling client encryption will encrypt all traffic on the native_transport_port (default: 9042). If both encrypted and unencrypted traffic is required, an additional cassandra.yaml setting must be enabled. The native_transport_port_ssl (default: 9142) sets an additional dedicated port to carry encrypted transmissions, while native_transport_port carries unencrypted transmissions.

Data insert methods

There are several ways to get data into a DataStax Enterprise cluster. Any normal paths used will result in data replication using DSE Advanced Replication.

Supported data insert methods:
  • CQL insert, including cqlsh and applications that use the standard DSE drivers
  • from a CSV file
  • Solr HTTP or CQL
  • Spark saveToCassandra
Unsupported data insert methods:

Monitoring operations

Advanced replication can be monitored with JMX metrics. The outgoing replication queue size is a key factor to watch. See Metrics for more details.