Monitor the Apache Kafka® integration
Use the following commands and JMX operations to monitor the Kafka integration on a running node. For setup instructions, see Integrate Apache Kafka® with Hyper-Converged Database (HCD).
Index status
Check index status using nodetool:
nodetool describecluster | grep INDEX_STATUSCheck HCD logs for Kafka integration messages:
grep -i "kafka" /var/log/cassandra/system.logKey log prefixes:
-
KafkaReplicator: Index lifecycle events. -
ReplicatorMutatorInterceptor: Mutation interception. -
InProcessReplicatorService: Replication pipeline. -
ContentSummary: Detailed write operation tracking.
Monitor with JMX
Use JConsole, VisualVM, or any JMX monitoring tool to connect to HCD and view the metrics under:
org.apache.cassandra.metrics:type=Table,* (filter by name starting with Kafka)Kafka metrics
The Kafka integration exposes metrics using JMX for index lifecycle, replication performance, and storage state.
All metrics use the JMX path org.apache.cassandra.metrics:type=Table,keyspace=<ks>,scope=<table>,name=<metric>.
When you query metrics using the Prometheus-compatible endpoint provided by the HCD Management API agent, the agent rewrites metric names to lowercase with underscores and prefixes them with org_apache_cassandra_metrics_table_kafka_.
For example, KafkaMutations becomes org_apache_cassandra_metrics_table_kafka_mutations.
Index lifecycle metrics
The index lifecycle counters, which are monotonic totals:
| Metric | Description |
|---|---|
|
Total initial builds started. |
|
Successful initial builds. |
|
Failed initial builds. |
|
Total rebuilds started. |
|
Successful rebuilds. |
|
Failed rebuilds. |
|
Index drops and invalidations. |
|
Total rows indexed. |
|
Rows skipped because this node is not the primary replica. |
|
Rows indexed from a remote datacenter. |
|
Remote rows skipped because this node is not the primary replica. |
|
Total batches submitted. |
|
Total items across all batches. |
|
Indexer operation errors. |
|
Primary replica check errors. |
The index lifecycle histograms, which show distributions:
| Metric | Description |
|---|---|
|
Initial build duration distribution. |
|
Rebuild duration distribution. |
|
Batch size distribution (rows per batch). |
The index lifecycle gauges, which show current state:
| Metric | Description |
|---|---|
|
Current index status: |
|
Whether a build is in progress ( |
|
Timestamp of last build completion (milliseconds since epoch). |
|
Timestamp of last error (milliseconds since epoch). |
The index lifecycle rate meters:
| Metric | Description |
|---|---|
|
Rate of rows indexed (per second). |
|
Rate of batches submitted (per second). |
|
Rate of errors (per second). |
Replication performance metrics
The replication performance counters, which are monotonic totals:
| Metric | Description |
|---|---|
|
Total mutations processed. |
|
Failed mutations. |
|
Mutations rejected due to storage backpressure. |
|
Out-of-order (late) mutations received. |
|
Mutations recovered from a logged batch. |
The replication performance histograms:
| Metric | Description |
|---|---|
|
Coordinator write latency distribution (microseconds). |
The replication performance gauges, which show current state:
| Metric | Description |
|---|---|
|
Currently coordinating mutations. |
|
Operations pending in storage. |
|
Available storage capacity. |
Pending mutations storage metrics
The storage counters, which are monotonic totals:
| Metric | Description |
|---|---|
|
Total mutation STARTs accepted and appended to the commit log. |
|
Total mutation COMPLETEs appended (pending entries cleared). |
|
Total mutations rejected because the store was full (sum of all rejection reasons). |
|
Mutations rejected due to per-table or shared disk budget exhaustion (subset of |
|
Memtable spills to data segments. |
|
Commit-log segment rolls (new |
|
Group-commit fsyncs issued ( |
|
Checkpoints completed (log compacted and truncated). |
|
Mutations replayed from the commit log at store open. |
The storage histograms and gauges:
| Metric | Description |
|---|---|
|
Distribution of records spilled per memtable flush. |
|
Current number of pending mutations in the store. |
|
Approximate bytes of in-memtable payloads currently held in heap. |
|
Bytes currently held on disk (commit logs and data segments). |
Deduplication pipeline metrics
When deduplicationEnabled=true, the replicator exports metrics for the background EventsConsumerWorker and for the two Kafka sinks.
The primary sink writes to the intermediate events topic, and the worker destination sink writes to the destination topic.
The events consumer worker metrics:
| Metric | Type | Description |
|---|---|---|
|
Counter |
Total primary-key events processed from the events topic and sent to the destination topic. |
|
Counter |
Total redundant records dropped by within-batch primary-key deduplication. |
|
Counter |
Total errors encountered during per-primary-key processing and retries. |
|
Counter |
Total Kafka batches processed and committed. |
|
Counter |
Total Kafka batches that failed during processing or commit. |
|
Counter |
Total partition assignment changes (assign, revoke, and close). |
|
Gauge |
Current number of events topic partitions assigned to this worker. |
|
Gauge |
Number of unique primary keys currently being processed in the worker pool. |
|
Histogram |
Distribution of unique primary keys per committed batch. |
|
Histogram |
Latency distribution of |
|
Histogram |
Latency distribution of destination Kafka writes (microseconds). |
The sink metrics use the prefix Sink for the destination-topic sink or EventTopicSink for the events-topic sink.
For example, SinkOperationsSubmitted and EventTopicSinkOperationsSubmitted.
The sink metrics:
| Metric | Type | Description |
|---|---|---|
|
Counter |
Total operations submitted to the sink. |
|
Counter |
Total operations acknowledged by Kafka. |
|
Counter |
Total operations that failed to publish to Kafka. |
|
Counter |
Total row insert and update messages produced (non-null value). |
|
Counter |
Total tombstone messages produced (null value). |
|
Histogram |
Latency distribution of Kafka producer acknowledgments (microseconds). |
Inspect and manage MBeans with nodetool sjk
Use the JMX beans that the replicator exports to investigate the internal status of the replicator, query pending mutations, and perform administrative operations.
You can use nodetool sjk to query and manage these MBeans directly from the command line:
BEAN='com.datastax.replicator.index:type=PendingMutationsStorage,keyspace=KEYSPACE,table=TABLE'
# Get the number of pending mutations
nodetool sjk mx -mg -b "$BEAN" -f PendingCount
# Read the pending-mutations cap
nodetool sjk mx -mg -b "$BEAN" -f MaxPendingCount
# Change the pending-mutations cap at runtime (does not persist across restarts)
nodetool sjk mx -ms -b "$BEAN" -f MaxPendingCount -v 200000
# List pending operations (LSN + primary key)
nodetool sjk mx -mc -b "$BEAN" -op listPendingMutations
# Acknowledge a single LSN
nodetool sjk mx -mc -b "$BEAN" -op acknowledge -a 7:42
# Acknowledge multiple LSNs (repeat -a for each element)
nodetool sjk mx -mc -b "$BEAN" -op acknowledge -a 7:42 -a 7:43 -a 8:0
# Acknowledge all pending mutations (clear all pending items)
nodetool sjk mx -mc -b "$BEAN" -op acknowledgeAll
# Force a checkpoint
nodetool sjk mx -mc -b "$BEAN" -op checkpoint
# Describe the MBean (attributes and operations)
nodetool sjk mx -mi -b "$BEAN"