CDC for Cassandra FAQs

The CDC for Cassandra is a an open-source product from DataStax.

With CDC for Cassandra, updates to data in Apache Cassandra are put into a Pulsar topic, which in turn can write the data to external targets such as Elasticsearch, Snowflake, and other platforms. The DataStax Cassandra Source Connector for Apache Pulsar™ component has a one-to-one correspondence between a Cassandra table and a single Pulsar topic.

Yes, CDC for Cassandra is open source software under the Apache 2.0 license. You can find the source code on the DataStax CDC for Cassandra GitHub repository.

In effect, the CDC for Cassandra implements the reverse of Apache Pulsar or DataStax Cassandra Sink Connector. With those sink connectors, data is taken from a Pulsar topic and put into Cassandra. With CDC for Cassandra, updates to a Cassandra table are converted into events and put into a data topic. From there, the data can be published to external platforms like Elasticsearch, Snowflake, and other platforms.

Follow the installation instructions.

See the installation instructions.

See Deploy multiple Cassandra datacenters.

For each CDC-enabled Cassandra table, Cassandra needs extra processing cycles and storage to process the CDC commit logs. The impact for dealing with a single CDC-enabled table is small, but when there are a large number of Cassandra tables with CDC enabled, the impact within Cassandra increases. The performance impact occurs within Cassandra itself, not the Cassandra CDC solution with Pulsar.

The Change Agent for Cassandra is started as a JVM agent of the Cassandra process and it shares the same hardware resource of the same Cassandra node. However, the only job that the Change Agent for Cassandra does is to scan the CDC commit log directory on a regular basis and send messages to the Pulsar cluster. This is a lightweight process when launched on a single thread, but the Change Agent for Cassandra can be launched with multiple threads. As more threads are launched, more resources are consumed.

For each Cassandra write operation (one detected change-event), the Pulsar CSC connector performs a primary key-based Cassandra read to get the most complete, up-to-date information of that particular Cassandra row.

In a worst-case scenario, where a CDC-enabled Cassandra has 100% write workload, the CDC solution would double the workload by adding the same amount of read workload to Cassandra table. Since the Cassandra read is primary key-based, it is efficient.

See CDC for Cassandra limitations.

If the Pulsar cluster is down, the Change Agent for Cassandra on each Cassandra node attempts to send the mutations periodically, and it keeps the CDC commitlog segments on disk until the data sending is successful.

The Change Agent for Cassandra keeps track of the CDC commitlog segment offsets, so the Change Agent for Cassandra knows where to resume sending the mutation messages when the Pulsar cluster is back online.

DataStax recommends active monitoring of the disk space of the Cassandra nodes. If the Pulsar cluster is down, the change agent continues trying to send messages, and the CDC commitlog files accumulate on the Cassandra node. If the maximum CDC directory disk space is reached, future Cassandra writes to the CDC-enabled table will fail.

When the disk space of the cdc_raw directory reaches your cdc_total_space_in_mb Cassandra setting (less than 4 GB by default), writes to CDC-enabled tables fail with a CDCWriteException. The following warning message is included in Cassandra logs:

To avoid or recover from this situation, increase the cdc_total_space_in_mb and restart the node. To prevent hitting this new limit, increase the write throughput to your Apache Pulsar cluster, or decrease the write throughput to your node.

Increasing the write throughput can involve tuning one or more of the following:

As a last resort, if losing data is acceptable in your CDC pipeline, remove commitlog files from the cdc_raw directory. Restarting the node isn’t needed in this case.

You can check the CDC status of a table by running the following CQL query:

There are three possible statuses:

You can check the status of the Change Agent for Cassandra by running the following CQL query:

There are three possible statuses:

Unacknowledged messages mean the Change Agent for Cassandra couldn’t produce the event message in Pulsar. In this case, the table row mutation fails. The Cassandra client handles this as an exception. The data is committed to Cassandra and no event is created.

Another scenario might be the Pulsar broker is too busy to process messages and a backlog has been created. In this case, Pulsar’s backlog policies take effect and event messages are handled accordingly. The data is committed to Cassandra but there might be some additional latency to the event message creation.

The design of CDC in Cassandra assumed that when table changes are synchronized to the raw_cdc log, another process is draining that log. There is a max log size setting that disables writes to the table when the set threshold is reached. If a connection to the Pulsar cluster is needed for the log to be drained, and it isn’t responsive, then the log begins to fill, which can impact a table’s write availability.

For more, see the Scaling up your CDC configuration.

A dead letter topic is used when a message cannot be delivered to a consumer. Possible causes include the following:

The CSC for Pulsar creates a consumer to receive new event messages from the Change Agent for Cassandra, but doesn’t configure a dead letter topic. It is assumed that parallel instances, broker compute, and function worker compute are sized to handle the workload.

There are 3 areas of scalability to focus on:

Transformation functions are a great way to manipulate messages on CDC data (with no code required). Put them inline to watch the data topic and write to a different topic. Name the topic something memorable, like filtered-data topic.

One of the requirements for CDC is that both the Cassandra and Pulsar clusters must be in the same cloud region or on-premise data center. If you are using geo-replication, you need the change data to be replicated across multiple clusters. The most manageable way to handle this is to use Pulsar’s Cassandra sink to "watch" the CDC data topic and write the change to a different Cassandra table in another organization.

The Cassandra sink requires the following provisions:

Migrating data between tables solves quite a few different challenges. The basic approach is to use a Cassandra sink to watch the Cassandra source and write to another table while mapping columns appropriately. As the original table is phased out, the number of messages decreases to none, while consumers are watching the new table’s CDC data topic. For more information, see How do I configure multi-region CDC using the Cassandra sink?.