Performance tuning for the OpenSearch integration

To keep HCD and OpenSearch consistent, the integration automatically upgrades certain write consistency levels to QUORUM for tables with OpenSearch indexes. This prevents hinted handoff from indexing stale data.

The integration upgrades the following consistency levels to QUORUM: ANY, ONE, and LOCAL_ONE.

When a write succeeds at ANY, ONE, or LOCAL_ONE, the coordinator can acknowledge it before a quorum of replicas receives it. The integration then performs a read-back at QUORUM to fetch the complete row for indexing. Without the consistency level upgrade, the read-back can return stale or missing data and index the wrong values in OpenSearch.

Upgrading to QUORUM ensures consistency between HCD and OpenSearch, but it has consequences for workloads that deliberately chose a weaker consistency level, including the following:

The following consistency levels already provide sufficient guarantees, so the integration does not modify them: QUORUM, LOCAL_QUORUM, EACH_QUORUM, ALL, TWO, THREE, SERIAL, and LOCAL_SERIAL.

The replicator captures most writes that coordinators process and forwards them to OpenSearch. However, the eventually consistent design of HCD means that capturing all coordinator writes is not always sufficient to guarantee consistency with OpenSearch.

The most important case is a write timeout on a standard mutation (not LWT, not a LOGGED batch). When an application receives a timeout or loses contact with the coordinator, fewer than a quorum of replicas might hold the mutation because some replicas were slow to respond. If the slow replicas eventually apply the mutation locally, the row value in OpenSearch can diverge from what HCD clients observe, because the replicator read the row before the slow replicas caught up.

Processing mutations on every replica instead of only the coordinator would avoid this, but is cost-prohibitive.

To prevent data divergence, retry any write that times out. This approach is simple, effective, and low-cost.

If your application already uses LWTs or LOGGED batches, consistency is automatically guaranteed even when timeouts occur. The replicator captures the same recovery mechanisms as Cassandra and inherits its guarantees.

nodetool decommission blocks until the node’s ReplicatorStorage has sent all pending mutations to OpenSearch. A decommissioned node does not leave unindexed writes behind.

If OpenSearch is unreachable, decommission blocks indefinitely. To unblock it, do one of the following:

Rebuild an index after topology changes or to recover from errors:

Replace the following: