Monitor the OpenSearch integration

Use these tools to monitor the OpenSearch integration on a running node, including metrics and options for Grafana dashboards. For setup instructions, see Integrate OpenSearch with Hyper-Converged Database (HCD).

Index status

Use nodetool to check index status. Example:

nodetool describecluster | grep INDEX_STATUS

Check the HCD logs for OpenSearch integration messages. Example:

grep -i "opensearch" /var/log/cassandra/system.log

Key log prefixes:

OpenSearchIndex: Index lifecycle events
OpenSearchMutatorInterceptor: Write capture
InProcessReplicatorService: Replication pipeline
ContentSummary: Detailed write operation tracking

OpenSearch metrics

The OpenSearch integration provides metrics to help you monitor index lifecycle, replication performance, and storage state.

The metrics are categorized as:

Index lifecycle metrics
Replication performance metrics
Pending mutations storage metrics

All metrics use the following JMX path, where <metric> is the metric name prefixed with OS:

ObjectName: org.apache.cassandra.metrics:type=Table,keyspace=<ks>,scope=<table>,name=<metric>

The metrics are exposed through JMX, so any JMX-based tool can be used to monitor. Use JConsole, VisualVM, or any JMX monitoring tool to connect to HCD and view the metrics under:

org.apache.cassandra.metrics:type=OpenSearchIndex,*
org.apache.cassandra.metrics:type=OpenSearchReplicator,*
org.apache.cassandra.metrics:type=OpenSearchStorage,*

When you query metrics using the Prometheus-compatible endpoint provided by the HCD Management API agent, the agent rewrites metric names to lowercase with underscores. For example, OSMutations becomes org_apache_cassandra_metrics_table_os_mutations.

Grafana dashboards

HCD bundles Grafana dashboard definitions for the OpenSearch metrics under src/main/grafana/. Import them into a Grafana instance pointed at a Prometheus endpoint that scrapes the HCD Management API metrics endpoint to visualize index lifecycle, replication throughput, and storage state.

Index lifecycle metrics

ObjectName: org.apache.cassandra.metrics:type=Table,keyspace=<ks>,scope=<table>,name=<metric>

The index lifecycle counters, which are monotonic totals, for the given keyspace and table:

Metric Description

Metric	Description
`OSIndexerInitialBuilds`	Total initial builds started.
`OSIndexerInitialBuildSuccesses`	Successful initial builds.
`OSIndexerInitialBuildFailures`	Failed initial builds.
`OSIndexerRebuilds`	Total rebuilds started.
`OSIndexerRebuildSuccesses`	Successful rebuilds.
`OSIndexerRebuildFailures`	Failed rebuilds.
`OSIndexerInvalidations`	Index drops and invalidations.
`OSIndexerRowsIndexed`	Total rows indexed.
`OSIndexerRowsSkippedNonPrimary`	Rows skipped because this node is not the primary replica.
`OSIndexerRemoteRowsIndexed`	Rows indexed from a remote datacenter.
`OSIndexerRemoteRowsSkippedNonPrimary`	Remote rows skipped because this node is not the primary replica.
`OSIndexerBatchesSubmitted`	Total batches submitted.
`OSIndexerBatchItemsTotal`	Total items across all batches.
`OSIndexerErrors`	Indexer operation errors.
`OSIndexerPrimaryReplicaCheckErrors`	Primary replica check errors.

OSIndexerInitialBuilds

Total initial builds started.

OSIndexerInitialBuildSuccesses

Successful initial builds.

OSIndexerInitialBuildFailures

Failed initial builds.

OSIndexerRebuilds

Total rebuilds started.

OSIndexerRebuildSuccesses

Successful rebuilds.

OSIndexerRebuildFailures

Failed rebuilds.

OSIndexerInvalidations

Index drops and invalidations.

OSIndexerRowsIndexed

Total rows indexed.

OSIndexerRowsSkippedNonPrimary

Rows skipped because this node is not the primary replica.

OSIndexerRemoteRowsIndexed

Rows indexed from a remote datacenter.

OSIndexerRemoteRowsSkippedNonPrimary

Remote rows skipped because this node is not the primary replica.

OSIndexerBatchesSubmitted

Total batches submitted.

OSIndexerBatchItemsTotal

Total items across all batches.

OSIndexerErrors

Indexer operation errors.

OSIndexerPrimaryReplicaCheckErrors

Primary replica check errors.

The index lifecycle histograms, which show distributions, for the given keyspace and table:

Metric Description

Metric	Description
`OSIndexerInitialBuildDurationMillis`	Initial build duration distribution.
`OSIndexerRebuildDurationMillis`	Rebuild duration distribution.
`OSIndexerBatchSize`	Batch size distribution (rows per batch).

OSIndexerInitialBuildDurationMillis

Initial build duration distribution.

OSIndexerRebuildDurationMillis

Rebuild duration distribution.

OSIndexerBatchSize

Batch size distribution (rows per batch).

The index lifecycle gauges, which show current state, for the given keyspace and table:

Metric Description

Metric	Description
`OSIndexerCurrentStatus`	Current index status: `0` = UNKNOWN, `1` = BUILD_STARTED, `2` = BUILD_SUCCEEDED, `3` = BUILD_FAILED, `4` = DROPPED.
`OSIndexerIsBuilding`	Whether a build is in progress (`0`=false, `1`=true).
`OSIndexerLastBuildTimestamp`	Timestamp of last build completion (milliseconds since epoch).
`OSIndexerLastErrorTimestamp`	Timestamp of last error (milliseconds since epoch).

OSIndexerCurrentStatus

Current index status: 0 = UNKNOWN, 1 = BUILD_STARTED, 2 = BUILD_SUCCEEDED, 3 = BUILD_FAILED, 4 = DROPPED.

OSIndexerIsBuilding

Whether a build is in progress (0=false, 1=true).

OSIndexerLastBuildTimestamp

Timestamp of last build completion (milliseconds since epoch).

OSIndexerLastErrorTimestamp

Timestamp of last error (milliseconds since epoch).

The index lifecycle rate meters for the given keyspace and table:

Metric Description

Metric	Description
`OSIndexerRowsIndexedRate`	Rate of rows indexed (per second).
`OSIndexerBatchesSubmittedRate`	Rate of batches submitted (per second).
`OSIndexerErrorsRate`	Rate of errors (per second).

OSIndexerRowsIndexedRate

Rate of rows indexed (per second).

OSIndexerBatchesSubmittedRate

Rate of batches submitted (per second).

OSIndexerErrorsRate

Rate of errors (per second).

Replication performance metrics

ObjectName: org.apache.cassandra.metrics:type=Table,keyspace=<ks>,scope=<table>,name=<metric>

The replication performance counters, which are monotonic totals, for the given keyspace and table:

Metric Description

Metric	Description
`OSMutations`	Total mutations processed.
`OSMutationFailures`	Failed mutations.
`OSBackpressureErrors`	Mutations rejected due to storage backpressure.
`OSLateMutations`	Out-of-order (late) mutations received.
`OSMutationsRecoveredFromLoggedBatch`	Mutations recovered from a logged batch.

OSMutations

Total mutations processed.

OSMutationFailures

Failed mutations.

OSBackpressureErrors

Mutations rejected due to storage backpressure.

OSLateMutations

Out-of-order (late) mutations received.

OSMutationsRecoveredFromLoggedBatch

Mutations recovered from a logged batch.

The replication performance histograms for the given keyspace and table:

Metric Description

Metric	Description
`OSMutationLatencyMicros`	Coordinator write latency distribution (microseconds).

OSMutationLatencyMicros

Coordinator write latency distribution (microseconds).

The replication performance gauges, which show current state, for the given keyspace and table:

Metric Description

Metric	Description
`OSInFlightMutations`	Currently coordinating mutations.
`OSPendingOperations`	Operations pending in storage.
`OSAllowedMutations`	Available storage capacity.

OSInFlightMutations

Currently coordinating mutations.

OSPendingOperations

Operations pending in storage.

OSAllowedMutations

Available storage capacity.

Pending mutations storage metrics

ObjectName: org.apache.cassandra.metrics:type=Table,keyspace=<ks>,scope=<table>,name=<metric>

The pending mutations storage counters, which are monotonic totals, for the given keyspace and table:

Metric Description

Metric	Description
`OSStorageMutationsStarted`	Total mutation STARTs accepted and appended to the commit log.
`OSStorageMutationsCompleted`	Total mutation COMPLETEs appended (pending entries cleared).
`OSStorageMutationsRejectedFull`	Total mutations rejected because the store was full (sum of all rejection reasons).
`OSStorageMutationsRejectedDiskFull`	Mutations rejected due to per-table or shared disk budget exhaustion (subset of `OSStorageMutationsRejectedFull`).
`OSStorageFlushes`	Memtable spills to data segments.
`OSStorageWalSegmentsRolled`	Commit-log segment rolls (new `cl-*.log` files opened).
`OSStorageWalForces`	Group-commit fsyncs issued (`fdatasync` calls on the commit log).
`OSStorageCheckpoints`	Checkpoints completed (log compacted and truncated).
`OSStorageMutationsRecovered`	Mutations replayed from the commit log at store open.

OSStorageMutationsStarted

Total mutation STARTs accepted and appended to the commit log.

OSStorageMutationsCompleted

Total mutation COMPLETEs appended (pending entries cleared).

OSStorageMutationsRejectedFull

Total mutations rejected because the store was full (sum of all rejection reasons).

OSStorageMutationsRejectedDiskFull

Mutations rejected due to per-table or shared disk budget exhaustion (subset of OSStorageMutationsRejectedFull).

OSStorageFlushes

Memtable spills to data segments.

OSStorageWalSegmentsRolled

Commit-log segment rolls (new cl-*.log files opened).

OSStorageWalForces

Group-commit fsyncs issued (fdatasync calls on the commit log).

OSStorageCheckpoints

Checkpoints completed (log compacted and truncated).

OSStorageMutationsRecovered

Mutations replayed from the commit log at store open.

The pending mutations storage histograms and gauges for the given keyspace and table:

Metric Description

Metric	Description
`OSStorageRecordsPerFlush` (histogram)	Distribution of records spilled per memtable flush.
`OSStoragePendingCount`	Current number of pending mutations in the store.
`OSStorageMemtableBytes`	Approximate bytes of in-memtable payloads currently held in heap.
`OSStorageDiskBytes`	Bytes currently held on disk (commit logs and data segments).

OSStorageRecordsPerFlush (histogram)

Distribution of records spilled per memtable flush.

OSStoragePendingCount

Current number of pending mutations in the store.

OSStorageMemtableBytes

Approximate bytes of in-memtable payloads currently held in heap.

OSStorageDiskBytes

Bytes currently held on disk (commit logs and data segments).

Inspect and manage MBeans with nodetool sjk

Use the JMX beans exported by the replicator to investigate the internal status of the replicator, query pending mutations, and perform administrative operations. You can use nodetool sjk to query and manage these MBeans directly from the command line:

BEAN='com.datastax.opensearch.index:type=PendingMutationsStorage,keyspace=KEYSPACE,table=TABLE'

# Get the number of pending mutations
nodetool sjk mx -mg -b "$BEAN" -f PendingCount

# Read the pending-mutations cap
nodetool sjk mx -mg -b "$BEAN" -f MaxPendingCount

# Change the pending-mutations cap at runtime (not persisted — lost on restart)
nodetool sjk mx -ms -b "$BEAN" -f MaxPendingCount -v 200000

# List pending operations (LSN + primary key)
nodetool sjk mx -mc -b "$BEAN" -op listPendingMutations

# Acknowledge a single LSN
nodetool sjk mx -mc -b "$BEAN" -op acknowledge -a 7:42

# Acknowledge multiple LSNs (repeat -a for each element)
nodetool sjk mx -mc -b "$BEAN" -op acknowledge -a 7:42 -a 7:43 -a 8:0

# Acknowledge all pending mutations (clear all pending items)
nodetool sjk mx -mc -b "$BEAN" -op acknowledgeAll

# Force a checkpoint
nodetool sjk mx -mc -b "$BEAN" -op checkpoint

# Describe the MBean (attributes and operations)
nodetool sjk mx -mi -b "$BEAN"

Monitor the OpenSearch integration

Index status

OpenSearch metrics

Grafana dashboards

Index lifecycle metrics

Replication performance metrics

Pending mutations storage metrics

Inspect and manage MBeans with nodetool sjk

Was this helpful?

Give Feedback