Using DSE Advanced Replication

Starting DSE Advanced Replication

If you are using Advanced Replication V1 from DSE 5.0, you must upgrade to DSE 5.1 and migrate to Advanced Replication V2.

Before you can start and use DSE Advanced Replication, you must create the user keyspaces and tables on the source cluster and the destination cluster.

On all nodes in the source cluster:

  1. Enable replication in the dse.yaml file.

    Uncomment all advanced_replication_options entries, set enabled: true, and specify a directory to hold advanced replication log files with advanced_replication_directory:

    # Advanced Replication configuration settings
advanced_replication_options:
  enabled: true
  advanced_replication_directory: /var/lib/cassandra/advrep

  2. Enable Capture-Data-Change (CDC) in the cassandra.yaml file on a per-node basis for each source:

    cdc_enabled: true
cdc_raw_directory: /var/lib/cassandra/cdc_raw

    Advanced Replication does not start if CDC is not enabled. Either use the default directory or change it to a preferred location.

  3. Consider increasing the default CDC disk space, depending on the load (default: 4096 MB or 1/8 of the total space where cdc_raw_directory resides):

    cdc_total_space_in_mb: 16384

  4. Commitlog compression is turned off by default. To avoid problems with advanced replication, this option should NOT be used:

    # commitlog_compression:
#   - class_name: LZ4Compressor

  5. Do a rolling restart: restart the nodes in the source cluster one at a time while the other nodes continue to operate online.

Disabling DSE Advanced Replication

When replication is not enabled, data is not written to the replication log. On all nodes in the source cluster:

  1. To disable replication, edit the dse.yaml file.

    In the advanced_replication_options section, set enabled: false.

    # Advanced Replication configuration settings
advanced_replication_options:
  enabled: false

  2. Do a rolling restart: restart the nodes in the source cluster one at a time while the other nodes continue to operate online.

  3. To clean out the data that was used for DSE Advanced Replication, use cqlsh to remove these keyspaces:

    DROP TABLE dse_system.advrep_source_config;
DROP TABLE dse_system.advrep_destination_config;
DROP TABLE dse_system.advrep_repl_channel_config;
DROP KEYSPACE dse_advrep;

Configuring global configuration settings

Global settings apply to the entire source cluster. These global settings are stored in the CQL table dse_system.advrep_source_config that is automatically created.

Change global settings by using the dse advrep command line tool with this syntax:

dse advrep conf ...

To view the source node configuration settings:

  • command

  • Result

dse advrep conf list
-----------------------------------
|name             |value          |
-----------------------------------
|audit_log_file   |/tmp/myaudit.gz|
-----------------------------------
|audit_log_enabled|true           |
-----------------------------------

The following table describes the configuration keys, their default values, and identifies when a restart of the source node is required for the change to be recognized.

The dse advrep command line tool uses these configuration keys as command arguments to the dse advrep command line tool.

Configuration keys for restarting source nodes part one
Configuration key Default value Description Restart required

permits

30,000

Maximum number of messages that can be replicated in parallel over all destinations.

No

source-id

N/A

Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur.

No

collection-expire-after-write

N/A

collection-time-slice-count

5

The number of files which are open in the ingestor simultaneously.

Yes

collection-time-slice-width

60 seconds

The time period in seconds for each data block ingested. Smaller time widths => more files. Larger timer widths => larger files but more data to resend on CRC mismatches.

Yes

invalid-message-log

SYSTEM_LOG

Select one of these logging strategies to adopt when an invalid message is discarded:

* SYSTEM_LOG: Log the CQL query and the error message in the system log on the destination.

* CHANNEL_LOG: Store the CQL query and the error message in files in /var/lib/cassandra/advrep/invalid_queries on the destination. This is the default value.

* NONE: Perform no logging.

See Managing invalid messages. Requires node restart.

No

audit-log-enable

false

Specifies whether to store the audit log.

Yes

audit-log-file

/tmp/advrep_rl_audit.log

Specifies the file name prefix template for the audit log file. The file name is appended with .gz if compressed using gzip.

Yes

audit-log-max-life-span-mins

0

Specifies the maximum lifetime of audit log files. Periodically, when log files are rotated, audit log files are purged when they:

* Match the audit log file template * And they have not been written to for more than the specified maximum lifespan minutes

To disable purging, set to 0.

Yes

audit-log-rotate-time-mins

60

Specifies the time interval to rotate the audit log file. On rotation, the rotated file is appended with the log counter .[<logcounter>], incrementing from [0]. To disable rotation, set to 0.

Yes

Configuring destination settings

A destination is a location to which source data will be written. Destinations are stored in the CQL table dse_system.advrep_destination_config that is automatically created.

Change destination settings by using the dse advrep command line tool with this syntax:

dse advrep destination ...

You can verify the channel configuration before you change it. For example:

  • command

  • Result

dse advrep destination list-conf
--------------------------------------------------------------------------------------------
|destination|name                                |value                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_enabled                  |false                                    |
--------------------------------------------------------------------------------------------
|mydest     |addresses                           |10.200.182.251                           |
--------------------------------------------------------------------------------------------
|mydest     |driver_read_timeout                 |15000                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_connections_max              |8                                        |
--------------------------------------------------------------------------------------------
|mydest     |source_id_column                    |source_id                                |
--------------------------------------------------------------------------------------------
|mydest     |driver_connect_timeout              |15000                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_protocol                 |TLS                                      |
--------------------------------------------------------------------------------------------
|mydest     |driver_consistency_level            |QUORUM                                   |
--------------------------------------------------------------------------------------------
|mydest     |driver_used_hosts_per_remote_dc     |0                                        |
--------------------------------------------------------------------------------------------
|mydest     |driver_allow_remote_dcs_for_local_cl|false                                    |
--------------------------------------------------------------------------------------------
|mydest     |driver_compression                  |lz4                                      |
--------------------------------------------------------------------------------------------
|mydest     |driver_connections                  |1                                        |
--------------------------------------------------------------------------------------------
|mydest     |driver_ssl_cipher_suites            |[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,|
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_GCM_SHA384,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_GCM_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,   |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_3DES_EDE_CBC_SHA,           |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_RC4_128_SHA,          |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_SHA,                |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_RC4_128_SHA,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_RC4_128_SHA,           |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_MD5,                |
|           |                                    |,                                        |
|           |                                    |TLS_EMPTY_RENEGOTIATION_INFO_SCSV]       |
--------------------------------------------------------------------------------------------
|mydest     |source_id                           |source1                                  |
--------------------------------------------------------------------------------------------
|mydest     |transmission_enabled                |true                                     |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_enabled                  |false                                    |
--------------------------------------------------------------------------------------------
|llpdest    |addresses                           |10.200.177.184                           |
--------------------------------------------------------------------------------------------
|llpdest    |driver_read_timeout                 |15000                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connections_max              |8                                        |
--------------------------------------------------------------------------------------------
|llpdest    |source_id_column                    |source_id                                |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connect_timeout              |15000                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_protocol                 |TLS                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_consistency_level            |ONE                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_used_hosts_per_remote_dc     |0                                        |
--------------------------------------------------------------------------------------------
|llpdest    |driver_allow_remote_dcs_for_local_cl|false                                    |
--------------------------------------------------------------------------------------------
|llpdest    |driver_compression                  |lz4                                      |
--------------------------------------------------------------------------------------------
|llpdest    |driver_connections                  |1                                        |
--------------------------------------------------------------------------------------------
|llpdest    |driver_ssl_cipher_suites            |[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,|
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_CBC_SHA,            |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_CBC_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_256_GCM_SHA384,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,   |
|           |                                    |,                                        |
|           |                                    |TLS_RSA_WITH_AES_128_GCM_SHA256,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,  |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,    |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,     |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,   |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,     |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_3DES_EDE_CBC_SHA,           |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,    |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,      |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,       |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,        |
|           |                                    |,                                        |
|           |                                    |TLS_ECDHE_RSA_WITH_RC4_128_SHA,          |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_SHA,                |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_ECDSA_WITH_RC4_128_SHA,         |
|           |                                    |,                                        |
|           |                                    |TLS_ECDH_RSA_WITH_RC4_128_SHA,           |
|           |                                    |,                                        |
|           |                                    |SSL_RSA_WITH_RC4_128_MD5,                |
|           |                                    |,                                        |
|           |                                    |TLS_EMPTY_RENEGOTIATION_INFO_SCSV]       |
--------------------------------------------------------------------------------------------
|llpdest    |source_id                           |source1                                  |
--------------------------------------------------------------------------------------------
|llpdest    |transmission_enabled                |false                                    |
--------------------------------------------------------------------------------------------

The following table describes the configuration keys, their default values, and identifies when a restart of the source node is required for the change to be recognized.

Configuration keys for restarting source nodes part two
Configuration key Default value Description Restart required

separator

N/A

Field separator.

No

name

N/A

Name for destination (required).

No

addresses

none

REQUIRED. A comma separated list of IP addresses that are used to connect to the destination cluster using the DataStax Java driver.

No

driver-allow-remote-dcs-for-local-cl

false

Set to true to enable automatic failover for destination clusters with multiple datacenters. The value of the driver-consistency-level parameter must be LOCAL_ONE or LOCAL_QUORUM.

Yes

driver-compression

lz4

The compression algorithm the DataStax Java driver uses to send data from the source to the destination. Supported values are lz4 and snappy.

Yes

driver-connect-timeout

15000

Time in milliseconds the DataStax Java driver waits to connect to a server.

No

driver-connections

32

The number of connections the DataStax Java driver creates.

Yes

driver-connections-max

256

The maximum number of connections the DataStax Java driver creates.

Yes

driver-max-requests-per-connection

1024

The maximum number of requests per connection the DataStax Java driver creates.

driver-consistency-level

ONE

The consistency level used by the DataStax Java driver when executing statements for replicating data to the destination. Specify a valid DSE CONSISTENCY level: ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, SERIAL, LOCAL_SERIAL, or LOCAL_ONE.

No

driver-local-dc

N/A

For destination clusters with multiple datacenters, you can explicitly define the name of the datacenter that you consider local. Typically, this is the datacenter that is closest to the source cluster. This value is used only for clusters with multiple data enters.

Yes

driver-pwd

none

Driver password if the destination requires a user and password to connect. Changing the driver-pwd value for connection to a destination automatically connects, but with a slight delay.

By default, driver user names and passwords are plain text. DataStax recommends encrypting the driver passwords before you add them to the CQL table.

Yes

driver-read-timeout

15000

Time in milliseconds the DataStax Java driver waits to read responses from a server.

No

driver-ssl-enabled

false

Whether SSL is enabled for connection to the destination.

Yes

driver-ssl-disabled

Disable SSL for connection to the destination.

driver_ssl_keystore_path

none

The path to the keystore for connection to DSE when SSL client authentication is enabled.

Yes

driver_ssl_keystore_password

none

The keystore password for connection to DSE when SSL client authentication is enabled.

Yes

driver_ssl_keystore_type

none

The keystore type for connection to DSE when SSL client authentication is enabled.

Yes

driver_ssl_truststore_path

none

The path to the truststore for connection to DSE when SSL is enabled.

Yes

driver-ssl-truststore-password

none

The truststore password for connection to DSE when SSL is enabled.

Yes

driver-ssl-truststore-type

none

The keystore type for connection to DSE when SSL client authentication is enabled.

Yes

driver-ssl-protocol

TLS

The SSL protocol for connection to DSE when SSL is enabled.

Yes

driver-ssl-cipher-suites

none

A comma-separated list of SSL cipher suites for connection to DSE when SSL is enabled. Cipher suites must be supported by the source machine.

Yes

driver-used-hosts-per-remote-dc

0

To use automatic failover for destination clusters with multiple datacenters, you must define the number of hosts per remote datacenter that the datacenter aware round robin policy (DCAwareRoundRobinPolicy) considers available.

Yes

driver-user

none

Driver username if the destination requires a user and password to connect. Changing the driver-user value for connection to a destination automatically connects, but with a slight delay.

Yes

source-id

N/A

Identifies this source cluster and all inserts from this cluster. The source-id must also exist in the primary key on the destination for population of the source-id to occur.

No

source-id-column

source-id

The column to use on remote tables to insert the source id as part of the update. If this column is not present on the table that is being updated, the source id value is ignored.

No

transmission-enabled

false

Specify if data collector for the table should be replicated to the destination using boolean value.

No

Configuring channel settings

A replication channel is a defined channel of change data between source clusters and destination clusters.

A replication channel is defined by the source cluster, source keyspace, source table name, destination cluster, destination keyspace, and destination table name. Replications for each channel (unique keyspace and table) are stored in the CQL table dse_system.advrep_repl_channel_config that is automatically created.

Change the settings using the dse advrep command line tool with this syntax:

dse advrep channel ...

You can verify the channel configuration before you change it. For example:

  • command

  • Result

dse advrep channel status
------------------------------------------------------------------------------------------------------------------------------------------------------
|dc       |keyspace|table          |collecting|transmitting|replication order|priority|dest ks|dest table     |src id |src id col|dest  |dest enabled|
------------------------------------------------------------------------------------------------------------------------------------------------------
|Cassandra|foo     |bar            |true      |true        |FIFO             |2       |foo    |bar            |source1|source_id |mydest|true        |
------------------------------------------------------------------------------------------------------------------------------------------------------

Properties are continuously read from the metadata, so a restart is not required after configuration changes are made. The following table describes the configuration settings.

Configuration settings
Column name Description

separator

Field separator.

keyspace

The keyspace on the source for the table to replicate.

table

The table name on the source to replicate.

source-id

Placeholder to override the source-id that is defined in the advrep_conf metadata

source-id-column

Placeholder to override the source-id-column that is defined in advrep_conf metadata.

enabled

If true, replication starts for this table. If false, no more messages from this table are saved to the replication log.

data-center-id

Datacenter this replication channel is meant for, if none specified the replication happens in all specified dc1.

destination

Destination to which data is written.

destination-keyspace

The keyspace on the destination for the replicated table.

destination-table

The table name on the destination for the replicated table.

priority

Messages are marked by priority in descending order (DESC).

transmission-enabled

Specify if the data collector for the table should be replicated to the destination.

fifo-order

Specify if the channel should be replicated in FIFO order (default).

lifo-order

Specify if the channel should be replicated in LIFO order.

Security

Authentication credentials can be provided in several ways, see Providing credentials from DSE tools.

The user who is doing the replicating with DSE Advanced Replication requires table and keyspace level authorization. If the same user access is required, then ensure that the authorization is the same on the source and destination clusters.

Advanced Replication also supports setting row-level permissions on the destination cluster. The user which connects to the destination cluster must have permission to write to the specified destination table at the row level replicated from the source, according to the RLAC restrictions. The user is specified with the --driver-user destination setting. Row-level access control (RLAC) on the source cluster does not impact Advanced Replication. Because Advanced Replication reads the source data at the raw CDC file layer, it essentially reads as a superuser and has access to all configured data tables.

Advanced Replication supports encrypting the driver passwords. Driver passwords are stored in a CQL table. By default, driver passwords are plain text. DataStax recommends encrypting the driver passwords before you add them to the CQL table. Create a global encryption key, called a system_key for SSTable encryption. Each node in the source cluster must have the same system key. The destination does not require this key.

  1. In the dse.yaml file:

    • Verify that the config_encryption_active property is false:

      config_encryption_active: false

    • Enable driver password encryption with the conf_driver_password_encryption_enabled property:

      conf_driver_password_encryption_enabled: true

    • Define where system keys are stored on disk. The location of the key is specified on the command line with the -d option or with system_key_directory in dse.yaml. The default filepath is /etc/dse/conf.

    • To configure the filename of the generated encryption key, set the config_encryption_key_name option in dse.yaml. The default name is system_key.

  2. Generate a system key:

    On-server:

    dsetool createsystemkey <cipher_algorithm> <strength> <system_key_file>

    Off-server

    dsetool createsystemkey <cipher_algorithm> <strength> <system_key_file> -kmip=<kmip_groupname>

    For example:

    dsetool createsystemkey 'AES/ECB/PKCS5Padding' 128 <system_key_file>

    where <system_key_file> is a unique file name for the generated system key file. See createsystemkey.

    Result: Configure transparent data encryption (TDE) on a per table basis. You can configure encryption with or without compression. You can create a global encryption key in the location that is specified by system_key_directory in the dse.yaml file. This default global encryption key is used when the system_key_file subproperty is not specified.

  3. Copy the returned value.

  4. On any node in the source cluster, use the dse command to set the encrypted password in the DSE Advanced Replication environment:

    dse advrep destination --driver-pwd "Sa9xOVaym7bddjXUT/eeOQ==" --driver-user "username"

  5. Start dse.

SSL configuration and ports

For details about SSL configuration with DSE Advanced Replication, refer to Configuring SSL for nodetool, nodesync, dsetool, and Advanced Replication.

Enabling client encryption will encrypt all traffic on the native_transport_port (default: 9042). If both encrypted and unencrypted traffic is required, an additional cassandra.yaml setting must be enabled. The native_transport_port_ssl (default: 9142) sets an additional dedicated port to carry encrypted transmissions, while native_transport_port carries unencrypted transmissions.

Data insert methods

There are several ways to get data into a DataStax Enterprise cluster. Any normal paths used result in data replication using DSE Advanced Replication.

Supported data insert methods:

  • CQL insert, including cqlsh and applications that use the standard DSE drivers

  • COPY TO from a CSV file

  • Solr HTTP or CQL

  • Spark saveToCassandra

Unsupported data insert methods:

Monitoring operations

Advanced replication can be monitored with JMX metrics. The outgoing replication queue size is a key factor to watch. See Metrics for more details.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com