Performance tuning for the Apache Kafka® integration

Use these tools and guidelines to maintain the Kafka integration and understand how it behaves in production. For setup instructions, see Integrate Apache Kafka® with Hyper-Converged Database (HCD). For monitoring, see Monitor the Apache Kafka® integration.

Multi-datacenter replication

ReplicatorMutatorInterceptor intercepts all write paths described in Integrate Apache Kafka® with Hyper-Converged Database (HCD). The cassandra.kafka.processing.datacenter_aware_replication property controls how the HCD cluster routes those mutations to Kafka in single and multi-datacenter deployments.

Datacenter-aware replication (default)

When cassandra.kafka.processing.datacenter_aware_replication=true (the default), each HCD datacenter replicates exclusively to its local Kafka cluster.

The coordinator node in DC1 writes exactly once to its local Kafka cluster. HCD then replicates the mutation to every replica node in DC2, and each of those replicas independently writes to the DC2 Kafka cluster, resulting in N writes, where N equals the replication factor for DC2. There are no cross-datacenter Kafka writes.

Single-cluster replication

When cassandra.kafka.processing.datacenter_aware_replication=false, all HCD nodes share a single Kafka cluster. Only the coordinator node writes to Kafka (one write total). Replica nodes in all datacenters do not publish the mutation.

Deduplication

If your application cannot use the rowVersion and late headers to handle version ordering, enable deduplication. HCD then publishes only the current state of each row to the destination topic.

Set deduplicationEnabled=true on the index to enable deduplication. For a full list of related options, see Index options.

Standard replication versus deduplication

Standard replication, deduplicationEnabled=false, is the default. The coordinator reads the row back at QUORUM consistency and publishes it directly to the destination topic. HCD does not create a Kafka consumer.

With deduplication, deduplicationEnabled=true, the replicator separates mutation capture from row materialization using an intermediate events topic and background Kafka consumers in a shared consumer group.

Instead of reading the row back on the mutation path, the coordinator extracts only the primary key (partition key and clustering columns) and publishes a compact binary event to the events topic. A background EventsConsumerWorker consumes batches of events from this topic and does the following:

Collapses duplicate primary keys within each batch into a single read.
Reads the row from HCD at LOCAL_QUORUM.
Publishes the materialized row, or a tombstone for a deleted row, to the destination topic.

Multi-instance consumer group topology

With a multi-node HCD cluster, all EventsConsumerWorker instances for a table share the consumer group that eventsConsumerGroupId specifies. Kafka balances the partitions of the events topic across the active workers.

If a node stops or restarts, Kafka rebalances the events topic partitions among the surviving workers in the consumer group, with no data loss.

Events topic message format

Each mutation produces a lightweight binary primary key event in the events topic (eventsTopic) in the following format:

Key: The serialized primary key (partition key and clustering components), as raw bytes.
Value: Empty array.
Headers: None. The EventsConsumerWorker generates and attaches headers such as rowVersion and late when it publishes to the destination topic.

Kafka partitions events by primary key using standard Kafka hashing. All mutations for the same row go to the same partition, and the assigned worker consumes them sequentially.

The key uses the following big-endian binary layout:

pkLen: int32
pkBytes: pkLen bytes (serialized partition key)
ckCount: int32 (number of clustering columns)
Repeated compLen, compBytes: int32 length (-1 if null), followed by the component bytes

Write timeouts and eventual consistency

Because HCD is eventually consistent, there is a narrow window where the replicator can publish a message to Kafka that does not reflect the final state observed by HCD clients.

The most common scenario involves a write timeout on a standard mutation (not an LWT or a LOGGED batch). When a write times out, fewer than a quorum of replicas might have acknowledged the mutation. The replicator reads the row back at quorum consistency immediately after the coordinator processes the write. If the slow replicas apply the mutation after that read-back, the Kafka message reflects the earlier state, not the final committed value.

To prevent data divergence, retry any write that times out. This approach is simple, effective, and low-cost.

If your application already uses LWTs or LOGGED batches, consistency is automatically guaranteed even when timeouts occur. The replicator captures the same recovery mechanisms as Cassandra and inherits its guarantees.

Consistency level upgrade for indexed tables

To keep HCD and Kafka consistent, the integration automatically upgrades certain write consistency levels to QUORUM for tables with Kafka indexes. This prevents hinted handoff from publishing stale data.

The integration upgrades the following consistency levels to QUORUM: ANY, ONE, and LOCAL_ONE.

When a write succeeds at ANY, ONE, or LOCAL_ONE, the coordinator can acknowledge it before a quorum of replicas receives it. The integration then performs a read-back at QUORUM to fetch the complete row for publishing. Without the consistency level upgrade, the read-back might return stale or missing data.

Upgrading to QUORUM ensures consistency between HCD and Kafka, but it has consequences for workloads that deliberately chose a weaker consistency level, including the following:

Latency: Clients experience QUORUM latency instead of ONE/ANY latency for writes to indexed tables.
Availability: Writes might time out or return unavailable errors where ONE/ANY would have succeeded.
Scope: This change affects only writes to tables with Kafka indexes.

The following consistency levels already provide sufficient guarantees and remain unchanged: QUORUM, LOCAL_QUORUM, EACH_QUORUM, ALL, TWO, THREE, SERIAL, and LOCAL_SERIAL.

Node decommission safety

nodetool decommission blocks until the node’s ReplicatorStorage has sent all pending mutations to Kafka. A decommissioned node does not leave unpublished writes behind.

If Kafka is unreachable, decommission blocks indefinitely. To unblock it, do one of the following:

Use the acknowledgeAll JMX operation on the relevant PendingMutationsStorage MBean to release all pending items. For command examples, see Inspect and manage MBeans with nodetool sjk.
Set -Dcassandra.replicator.register_decommission_hook=false to disable the hook entirely.

Rebuild an index

Rebuild an index after topology changes or to recover from errors:

nodetool rebuild_index KEYSPACE TABLE INDEX_NAME

Replace the following:

KEYSPACE: The keyspace that contains the table.
TABLE: The table name.
INDEX_NAME: The name of the Kafka index to rebuild.