Using DSE Advanced Replication

Operations including starting, stopping, and configuring DSE Advanced Replication.

Starting DSE Advanced Replication
Stopping DSE Advanced Replication
Configuring global configuration settings
Configuring destination settings
Configuring channel settings
Security
Data insert methods
Monitoring operations

Prerequisite: If you are using Advanced Replication V1 from DSE 5.0, you must upgrade to DSE 5.1 and migrate to Advanced Replication V2.

Starting DSE Advanced Replication

Before you can start and use DSE Advanced Replication, you must create the user keyspaces and tables on the source cluster and the destination cluster.

On all nodes in the source cluster:

Enable replication in the dse.yaml file.

The location of the dse.yaml file depends on the type of installation:

Where is the dse.yaml file?

The location of the dse.yaml file depends on the type of installation:

Installation Type Location

Installation Type	Location
Package installations + Installer-Services installations	`/etc/dse/dse.yaml`
Tarball installations + Installer-No Services installations	`<installation_location>/resources/dse/conf/dse.yaml`

Package installations + Installer-Services installations

/etc/dse/dse.yaml

Tarball installations + Installer-No Services installations

<installation_location>/resources/dse/conf/dse.yaml

# Advanced Replication configuration settings
advanced_replication_options:
  enabled: true
  advanced_replication_directory: /var/lib/cassandra/advrep

Enable Capture-Data-Change (CDC) in the cassandra.yaml file on a per-node basis for each source:

At the end of the file, uncomment all advanced_replication_options entries, set enabled: true, and specify a directory to hold advanced replication log files with advanced_replication_directory:

cdc_enabled: true
cdc_raw_directory: /var/lib/cassandra/cdc_raw

Advanced Replication cannot start if CDC is not enabled. Either use the default directory or change it to a preferred location.

Where is the cassandra.yaml file?

The location of the cassandra.yaml file depends on the type of installation:

Installation Type Location

Installation Type	Location
Package installations + Installer-Services installations	`/etc/dse/cassandra/cassandra.yaml`
Tarball installations + Installer-No Services installations	`<installation_location>/resources/cassandra/conf/cassandra.yaml`

Package installations + Installer-Services installations

/etc/dse/cassandra/cassandra.yaml

Tarball installations + Installer-No Services installations

<installation_location>/resources/cassandra/conf/cassandra.yaml

Consider increasing the default CDC disk space, depending on the load (default: 4096 MB or 1/8 of the total space where cdc_raw_directory resides):
```
cdc_total_space_in_mb: 16384
```
Commitlog compression is turned off by default. To avoid problems with advanced replication, this option should NOT be used:
```
# commitlog_compression:
#   - class_name: LZ4Compressor
```
Do a rolling restart: restart the nodes in the source cluster one at a time while the other nodes continue to operate online.

Disabling DSE Advanced Replication

When replication is not enabled, data is not written to the replication log. On all nodes in the source cluster:

To disable replication, edit the dse.yaml file.

In the advanced_replication_options section, set enabled: false.
```
# Advanced Replication configuration settings
advanced_replication_options:
  enabled: false
```
Do a rolling restart: restart the nodes in the source cluster one at a time while the other nodes continue to operate online.

To clean out the data that was used for DSE Advanced Replication, use cqlsh to remove these keyspaces:

DROP TABLE dse_system.advrep_source_config;
DROP TABLE dse_system.advrep_destination_config;
DROP TABLE dse_system.advrep_repl_channel_config;
DROP KEYSPACE dse_advrep;

Configuring global configuration settings

Global settings apply to the entire source cluster. These global settings are stored in the CQL table dse_system.advrep_source_config that is automatically created.

Change global settings by using the dse advrep command line tool with this syntax:

dse advrep conf ...

To view the source node configuration settings:

dse advrep conf list

The result is:

-----------------------------------
|name             |value          |
-----------------------------------
|audit_log_file   |/tmp/myaudit.gz|
-----------------------------------
|audit_log_enabled|true           |
-----------------------------------

The following table describes the configuration keys, their default values, and identifies when a restart of the source node is required for the change to be recognized.

The dse advrep command line tool uses these configuration keys as command arguments to the dse advrep command line tool.

Configuration key Default value Description Restart required

Configuration key	Default value	Description	Restart required
permits	30,000	Maximum number of messages that can be replicated in parallel over all destinations.	No
source-id	N/A	Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur.	No
collection-expire-after-write	N/A
collection-time-slice-count	5	The number of files which are open in the ingestor simultaneously.	Yes
collection-time-slice-width	60 seconds	The time period in seconds for each data block ingested. Smaller time widths => more files. Larger timer widths => larger files but more data to resend on CRC mismatches.	Yes
invalid-message-log	`SYSTEM_LOG`	Select one of these logging strategies to adopt when an invalid message is discarded: `SYSTEM_LOG`: Log the CQL query and the error message in the system log on the destination. `CHANNEL_LOG`: Store the CQL query and the error message in files in `/var/lib/cassandra/advrep/invalid_queries` on the destination. `NONE`: Perform no logging. See Managing invalid messages. Requires node restart.	No
audit-log-enable	false	Specifies whether to store the audit log.	Yes
audit-log-file	`/tmp/advrep_rl_audit.log`	Specifies the file name prefix template for the audit log file. The file name is appended with `.gz` if compressed using gzip.	Yes
audit-log-max-life-span-mins	0	Specifies the maximum lifetime of audit log files. Periodically, when log files are rotated, audit log files are purged when they: Match the audit log file template And they have not been written to for more than the specified maximum lifespan minutes To disable purging, set to 0.	Yes
audit-log-rotate-time-mins	60	Specifies the time interval to rotate the audit log file. On rotation, the rotated file is appended with the log counter `.[logcounter]`, incrementing from [`0`]. To disable rotation, set to 0.	Yes

permits

30,000

Maximum number of messages that can be replicated in parallel over all destinations.

source-id

N/A

Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur.

collection-expire-after-write

N/A

collection-time-slice-count

The number of files which are open in the ingestor simultaneously.

Yes

collection-time-slice-width

60 seconds

The time period in seconds for each data block ingested. Smaller time widths => more files. Larger timer widths => larger files but more data to resend on CRC mismatches.

Yes

invalid-message-log

SYSTEM_LOG

Select one of these logging strategies to adopt when an invalid message is discarded: SYSTEM_LOG: Log the CQL query and the error message in the system log on the destination.

CHANNEL_LOG: Store the CQL query and the error message in files in /var/lib/cassandra/advrep/invalid_queries on the destination.

NONE: Perform no logging.

See Managing invalid messages. Requires node restart.

audit-log-enable

false

Specifies whether to store the audit log.

Yes

audit-log-file

/tmp/advrep_rl_audit.log

Specifies the file name prefix template for the audit log file. The file name is appended with .gz if compressed using gzip.

Yes

audit-log-max-life-span-mins

Specifies the maximum lifetime of audit log files. Periodically, when log files are rotated, audit log files are purged when they:

Match the audit log file template
And they have not been written to for more than the specified maximum lifespan minutes

To disable purging, set to 0.

Yes

audit-log-rotate-time-mins

Specifies the time interval to rotate the audit log file. On rotation, the rotated file is appended with the log counter .[logcounter], incrementing from [0]. To disable rotation, set to 0.

Yes

Configuring destination settings

A destination is a location to which source data is written. Destinations are stored in the CQL table dse_system.advrep_destination_config that is automatically created.

Change destination settings by using the dse advrep command line tool with this syntax:

$ dse advrep destination ...

You can verify the channel configuration before you change it. For example:

$ dse advrep destination list-conf

The result is:

--------------------------------------------------------------------------------------------
|destination|name                                |value                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_enabled                  |false                                    |
--------------------------------------------------------------------------------------------
|mydest     |addresses                           |10.200.182.251                           |
--------------------------------------------------------------------------------------------
|mydest     |driver_read_timeout                 |15000                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_connections_max              |8                                        |
--------------------------------------------------------------------------------------------
|mydest     |source_id_column                    |source_id                                |
--------------------------------------------------------------------------------------------
|mydest     |driver_connect_timeout              |15000                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_protocol                 |TLS                                      |
--------------------------------------------------------------------------------------------
|mydest     |driver_consistency_level            |QUORUM                                   |
--------------------------------------------------------------------------------------------
|mydest     |driver_used_hosts_per_remote_dc     |0                                        |
--------------------------------------------------------------------------------------------
|mydest     |driver_allow_remote_dcs_for_local_cl|false                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_compression                  |lz4                                      |
--------------------------------------------------------------------------------------------
|mydest     |driver_connections                  |1                                        |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_cipher_suites            |[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,|
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_GCM_SHA384,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_GCM_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,   |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_3DES_EDE_CBC_SHA,           |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_RC4_128_SHA,          |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_SHA,                |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_RC4_128_SHA,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_RC4_128_SHA,           |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_MD5,                |
|           |                                    |,                                        |
|           |                                    |TLS_EMPTY_RENEGOTIATION_INFO_SCSV]       |
--------------------------------------------------------------------------------------------
|mydest     |source_id                           |source1                                  |
--------------------------------------------------------------------------------------------
|mydest     |transmission_enabled                |true                                     |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_enabled                  |false                                    |
--------------------------------------------------------------------------------------------
|llpdest    |addresses                           |10.200.177.184                           |
--------------------------------------------------------------------------------------------
|llpdest    |driver_read_timeout                 |15000                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connections_max              |8                                        |
--------------------------------------------------------------------------------------------
|llpdest    |source_id_column                    |source_id                                |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connect_timeout              |15000                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_protocol                 |TLS                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_consistency_level            |ONE                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_used_hosts_per_remote_dc     |0                                        |
--------------------------------------------------------------------------------------------
|llpdest    |driver_allow_remote_dcs_for_local_cl|false                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_compression                  |lz4                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connections                  |1                                        |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_cipher_suites            |[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,|
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_GCM_SHA384,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_GCM_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,   |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_3DES_EDE_CBC_SHA,           |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_RC4_128_SHA,          |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_SHA,                |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_RC4_128_SHA,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_RC4_128_SHA,           |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_MD5,                |
|           |                                    |,                                        |
|           |                                    |TLS_EMPTY_RENEGOTIATION_INFO_SCSV]       |
--------------------------------------------------------------------------------------------
|llpdest    |source_id                           |source1                                  |
--------------------------------------------------------------------------------------------
|llpdest    |transmission_enabled                |false                                    |
--------------------------------------------------------------------------------------------

The following table describes the configuration keys, their default values, and identifies when a restart of the source node is required for the change to be recognized.

Configuration key Default value Description Restart required

separator

N/A

Field separator.

name

N/A

Name for destination (required).

addresses

none

REQUIRED. A comma separated list of IP addresses that are used to connect to the destination cluster using the DataStax Java driver.

driver-allow-remote-dcs-for-local-cl

false

Set to true to enable automatic failover for destination clusters with multiple datacenters. The value of the driver-consistency-level parameter must be LOCAL_ONE or LOCAL_QUORUM.

Yes

driver-compression

lz4

The compression algorithm the DataStax Java driver uses to send data from the source to the destination. Supported values are lz4 and snappy.

Yes

driver-connect-timeout

15000

Time in milliseconds the DataStax Java driver waits to connect to a server.

driver-connections

The number of connections the DataStax Java driver creates.

Yes

driver-connections-max

256

The maximum number of connections the DataStax Java driver creates.

Yes

driver-max-requests-per-connection

1024

The maximum number of requests per connection the DataStax Java driver creates.

driver-consistency-level

ONE

The consistency level used by the DataStax Java driver when executing statements for replicating data to the destination. Specify a valid DSE consistency level: ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, SERIAL, LOCAL_SERIAL, or LOCAL_ONE.

driver-local-dc

N/A

For destination clusters with multiple datacenters, you can explicitly define the name of the datacenter that you consider local. Typically, this is the datacenter that is closest to the source cluster. This value is used only for clusters with multiple data enters.

Yes

driver-pwd

none

Driver password if the destination requires a user and password to connect. Changing the driver-pwd value for connection to a destination automatically connects, but with a slight delay.

By default, driver user names and passwords are plain text. DataStax recommends encrypting the driver passwords before you add them to the CQL table.

Yes

driver-read-timeout

15000

Time in milliseconds the DataStax Java driver waits to read responses from a server.

driver-ssl-enabled

false

Whether SSL is enabled for connection to the destination.

Yes

driver-ssl-disabled

Disable SSL for connection to the destination.

driver_ssl_keystore_path

none

The path to the keystore for connection to DSE when SSL client authentication is enabled.

Yes

driver_ssl_keystore_password

none

The keystore password for connection to DSE when SSL client authentication is enabled.

Yes

driver_ssl_keystore_type

none

The keystore type for connection to DSE when SSL client authentication is enabled.

Yes

driver_ssl_truststore_path

none

The path to the truststore for connection to DSE when SSL is enabled.

Yes

driver-ssl-truststore-password

none

The truststore password for connection to DSE when SSL is enabled.

Yes

driver-ssl-truststore-type

none

The keystore type for connection to DSE when SSL client authentication is enabled.

Yes

driver-ssl-protocol

TLS

The SSL protocol for connection to DSE when SSL is enabled.

Yes

driver-ssl-cipher-suites

none

A comma-separated list of SSL cipher suites for connection to DSE when SSL is enabled. Cipher suites must be supported by the source machine.

Yes

driver-used-hosts-per-remote-dc

To use automatic failover for destination clusters with multiple datacenters, you must define the number of hosts per remote datacenter that the datacenter aware round robin policy (DCAwareRoundRobinPolicy) considers available.

Yes

driver-user

none

Driver username if the destination requires a user and password to connect. Changing the driver-user value for connection to a destination automatically connects, but with a slight delay.

Yes

source-id

N/A

Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur.

source-id-column

source-id

The column to use on remote tables to insert the source id as part of the update. If this column is not present on the table that is being updated, the source id value is ignored.

transmission-enabled

false

Specify if data collector for the table should be replicated to the destination using boolean value.

Configuring channel settings

A replication channel is a defined channel of change data between source clusters and destination clusters. A replication channel is defined by the source cluster, source keyspace, source table name, destination cluster, destination keyspace, and destination table name. Replications for each channel (unique keyspace and table) are stored in the CQL table dse_system.advrep_repl_channel_config that is automatically created.

Change the settings using the dse advrep command line tool with this syntax:

$ dse advrep channel ...

You can verify the channel configuration before you change it. For example:

$ dse advrep channel status

The result is:

------------------------------------------------------------------------------------------------------------------------------------------------------
|dc       |keyspace|table          |collecting|transmitting|replication order|priority|dest ks|dest table     |src id |src id col|dest  |dest enabled|
------------------------------------------------------------------------------------------------------------------------------------------------------
|Cassandra|foo     |bar            |true      |true        |FIFO             |2       |foo    |bar            |source1|source_id |mydest|true        |
------------------------------------------------------------------------------------------------------------------------------------------------------

Properties are continuously read from the metadata, so a restart is not required after configuration changes are made. The following table describes the configuration settings.

Column name Description

Column name	Description
`separator`	Field separator.
`keyspace`	The keyspace on the source for the table to replicate.
`table`	The table name on the source to replicate.
`source-id`	Placeholder to override the `source-id` that is defined in the `advrep_conf` metadata
`source-id-column`	Placeholder to override the `source-id-column` that is defined in `advrep_conf` metadata.
`enabled`	If true, replication starts for this table. If false, no more messages from this table are saved to the replication log.
`data-center-id`	The datacenter this replication channel is meant for. If none is specified, the replication happens in all specified `dc1`.
`destination`	Destination to which data is written.
`destination-keyspace`	The keyspace on the destination for the replicated table.
`destination-table`	The table name on the destination for the replicated table.
`priority`	Messages are marked by priority in descending order (DESC).
`transmission-enabled`	Specify if the data collector for the table should be replicated to the destination.
`fifo-order`	Specify if the channel should be replicated in FIFO order (default).
`lifo-order`	Specify if the channel should be replicated in LIFO order.

separator

Field separator.

keyspace

The keyspace on the source for the table to replicate.

table

The table name on the source to replicate.

source-id

Placeholder to override the source-id that is defined in the advrep_conf metadata

source-id-column

Placeholder to override the source-id-column that is defined in advrep_conf metadata.

enabled

If true, replication starts for this table. If false, no more messages from this table are saved to the replication log.

data-center-id

The datacenter this replication channel is meant for. If none is specified, the replication happens in all specified dc1.

destination

Destination to which data is written.

destination-keyspace

The keyspace on the destination for the replicated table.

destination-table

The table name on the destination for the replicated table.

priority

Messages are marked by priority in descending order (DESC).

transmission-enabled

Specify if the data collector for the table should be replicated to the destination.

fifo-order

Specify if the channel should be replicated in FIFO order (default).

lifo-order

Specify if the channel should be replicated in LIFO order.

Security

Authentication credentials can be provided in several ways, see Connecting to authentication enabled clusters. The user who is doing the replicating with DSE Advanced Replication requires table and keyspace level authorization. If the same user access is required, then ensure that the authorization is the same on the source and destination clusters.

Advanced Replication also supports setting row-level permissions on the destination cluster. The user that connects to the destination cluster must have permission to write to the specified destination table at the row level replicated from the source, according to the RLAC restrictions. The user is specified with the --driver-user destination setting. Row-level access control (RLAC) on the source cluster does not impact Advanced Replication. Because Advanced Replication reads the source data at the raw CDC file layer, it essentially reads as a superuser and has access to all configured data tables.

Advanced Replication supports encrypting the driver passwords. Driver passwords are stored in a CQL table. By default, driver passwords are plain text. DataStax recommends encrypting the driver passwords before you add them to the CQL table. Create a global encryption key, called a system_key for SSTable encryption. Each node in the source cluster must have the same system key. The destination does not require this key.

In the dse.yaml file:
- Verify that the config_encryption_active property is false:
  config_encryption_active: false
- Enable driver password encryption with the conf_driver_password_encryption_enabled property:
  conf_driver_password_encryption_enabled: true
- Define where system keys are stored on disk with the system_key_directory property:
  system_key_directory: /etc/dse/conf
  The default value is /etc/dse/conf.
- Specify that encryption keys are generated as system keys with the config_encryption_key_name property:
  config_encryption_key_name: system_key
Generate a system key:

On-server:
```
dsetool createsystemkey cipher_algorithm strength system_key_file
```
Off-server
```
dsetool createsystemkey cipher_algorithm strength system_key_file -kmip=kmip_groupname
```
For example:
```
dsetool createsystemkey 'AES/ECB/PKCS5Padding' 128 system_key_file
```
where system_key_file is a unique file name for the generated system key file. See createsystemkey.

Result: Configure transparent data encryption (TDE) on a per table basis. You can configure encryption with or without compression. You can create a global encryption key in the location that is specified by system_key_directory in the dse.yaml file. This default global encryption key is used when the system_key_file subproperty is not specified.
Copy the returned value.
On any node in the source cluster, use the dse command to set the encrypted password in the DSE Advanced Replication environment:
```
$ dse advrep destination --driver-pwd "Sa9xOVaym7bddjXUT/eeOQ==" --driver-user "username"
```
Start dse.

SSL configuration and ports

For details about SSL configuration with DSE Advanced Replication, refer to Setting up SSL for nodetool, dsetool, and dse advrep.

Enabling client encryption encrypts all traffic on the native_transport_port (default: 9042). If both encrypted and unencrypted traffic is required, an additional cassandra.yaml setting must be enabled. The native_transport_port_ssl (default: 9142) sets an additional dedicated port to carry encrypted transmissions, while native_transport_port carries unencrypted transmissions.

Data insert methods

There are several ways to get data into a DataStax Enterprise cluster. Any normal paths used result in data replication using DSE Advanced Replication.

Supported data insert methods:

CQL insert, including cqlsh and applications that use the standard DSE drivers
Copy from a CSV file
Solr HTTP or CQL
Spark saveToCassandra

Unsupported data insert methods:

Tables that are defined for compact storage
sstableloader (Cassandra bulk loader)
OpsCenter restore from backup
Spark bulkSaveToCassandra

Monitoring operations

Advanced replication can be monitored with JMX metrics. The outgoing replication queue size is a key factor to watch. See Metrics for more details.