OpenSearch integration reference
Use these options and properties to configure the OpenSearch integration, including index options, restrictions, index examples, and configuration settings. For setup instructions, see Integrate OpenSearch with Hyper-Converged Database (HCD).
Supported versions
The HCD OpenSearch integration supports OpenSearch versions 3.7.x, 3.8.x, and 3.9.x.
Index options
Pass all options as string values inside WITH OPTIONS = { … }.
Naming and lifecycle
The options that control the name and initial build behavior of the OpenSearch index:
| Option | Default | Description |
|---|---|---|
|
|
Name of the OpenSearch index to create or use.
Must be all lowercase, must not start with |
|
|
When |
|
|
When |
Column projection
The options that control which columns are replicated to OpenSearch:
| Option | Default | Description |
|---|---|---|
|
(none) |
Columns specified between the parentheses in |
|
(all columns) |
Comma-separated list of non-primary-key columns to replicate to OpenSearch. Primary key columns are always included regardless of this setting. Omit to index every column. |
JSON field expansion
Use these options for tables that store a complete JSON document verbatim in a single text column alongside auxiliary query columns. They let HCD expand that JSON document into individual top-level OpenSearch fields so every property is searchable and mappable. The options that control JSON field expansion:
| Option | Default | Description |
|---|---|---|
|
(none) |
Comma-separated list of column names whose values contain a serialized JSON object.
HCD removes each listed column from the document and merges its top-level properties at the root of the OpenSearch document.
HCD validates column names against the table schema at When specified, set |
|
(all properties) |
Comma-separated list of JSON property names to keep after expansion.
When specified, HCD merges only the listed properties into the root document and discards the rest.
When empty (the default), HCD includes every property from the expanded JSON.
Only meaningful when |
Sharding and replication
The options that control the number of primary and replica shards HCD creates for the index on OpenSearch, when createIndexIfNotExists is true:
| Option | Default | Description |
|---|---|---|
|
|
The number of primary shards for the OpenSearch index. Increase this value for large indexes where a single shard becomes a bottleneck. You cannot change this value after you create the index. Must be a positive integer. |
|
|
The number of replica shards per primary shard.
Use on single-node OpenSearch deployments.
Set to |
Non-primary replica indexing
The option that controls whether non-primary HCD replicas participate in replication:
| Option | Default | Description |
|---|---|---|
|
|
When |
Field mapping (schema)
The options that control how column types are mapped to OpenSearch field types, which apply only when createIndexIfNotExists is true:
| Option | Default | Description |
|---|---|---|
|
|
When |
|
|
When |
|
(none) |
A JSON string containing a complete OpenSearch type mapping object.
The value of the |
The three mapping modes are mutually exclusive and evaluated in priority order:
| Priority | Condition | Behavior |
|---|---|---|
1 |
|
HCD sends the supplied JSON mapping verbatim. |
2 |
|
HCD derives the mapping from the CQL table schema. |
3 |
Both false |
HCD sends no mapping. OpenSearch applies a matching index template if one exists. |
Tables that cannot be indexed
CREATE CUSTOM INDEX raises InvalidRequestException and leaves the schema untouched for the following table types:
-
A table that already has an
OpenSearchIndex. Only one OpenSearch index is allowed per table. -
Tables with static columns. A static column applies to every row in a partition, which is incompatible with the per-row replication model.
-
Tables with counter columns. Counter mutations use a different write path that the replicator does not capture.
-
Tables with dynamic data maskings (DDM) applied to columns. Unless you set
-Dcassandra.replicator.allow_exporting_masked_columns=true(defaultfalse), HCD rejects index creation on tables with masked columns. HCD applies masking only at the CQL result-set layer, so an index on a masked table exports raw column values. HCD also rejectsALTER TABLE … MASKED WITHon an already-indexed table. Drop the index before masking a column, or drop the masks before creating the index.If a race condition adds masked columns to an already-indexed table, or if you set
allow_exporting_masked_columnstofalseafter you create the index, HCD suspends processing for that table until you drop the masks.Querying an indexed table that has masked columns using
expr(…)also requires theUNMASKpermission, because the search service evaluates predicates and computes aggregations over unmasked data.
Index examples
The following examples show common CREATE CUSTOM INDEX configurations for different deployment scenarios:
High-availability production index
Three shards spread search load; one replica copy per shard provides redundancy. Index only the columns needed for search to reduce storage overhead.
CREATE CUSTOM INDEX IF NOT EXISTS articles_search
ON blog.articles ()
USING 'OpenSearchIndex'
WITH OPTIONS = {
'indexName': 'articles-search',
'createIndexIfNotExists': 'true',
'numShards': '8',
'numReplicas': '1',
'columnsToIndex': 'title,body,tags'
};This configuration requires at least two OpenSearch nodes (one primary plus one replica per shard).
Custom field mappings
Override the CQL-derived schema with a custom OpenSearch mapping. This is useful when you need specific analyzers, field types, or sub-fields that the automatic mapping does not produce.
CREATE CUSTOM INDEX IF NOT EXISTS products_search
ON store.products ()
USING 'OpenSearchIndex'
WITH OPTIONS = {
'indexName': 'products-search',
'applyCustomSchema': 'true',
'customMappingsJson': '{
"properties": {
"id": { "type": "keyword" },
"name": { "type": "text", "analyzer": "english" },
"price": { "type": "double" },
"description": { "type": "text" }
}
}'
};The customMappingsJson value must be a single-line JSON string inside a CQL string literal.
Escape inner single quotes by doubling them ('').
JSON field unpacking (Data API schema)
Tables that the DataStax Data API manages store the full JSON document in a doc_json text column alongside auxiliary shredded columns.
Use unpackJsonFields to expand the document’s properties into individual OpenSearch fields, and propertiesFromJsonFields to index only the subset you need:
CREATE CUSTOM INDEX IF NOT EXISTS products_search
ON store.products (doc_json)
USING 'OpenSearchIndex'
WITH OPTIONS = {
'indexName': 'products-search',
'createIndexIfNotExists': 'true',
'applyCustomSchema': 'true',
'customMappingsJson': '{
"properties": {
"name": { "type": "text", "analyzer": "english" },
"price": { "type": "double" }
}
}',
'unpackJsonFields': 'doc_json',
'propertiesFromJsonFields': 'name,price'
};For a row with doc_json = '{"name":"Widget","price":9.99,"internal_id":"x42"}', HCD removes the doc_json column, expands it, keeps only name and price, and writes the OpenSearch document {"key": {…}, "name": "Widget", "price": 9.99}.
Omit propertiesFromJsonFields to merge every property from the JSON.
Skip initial backfill
Index only writes made after CREATE INDEX, leaving pre-existing rows out of OpenSearch until you run a separate backfill:
CREATE CUSTOM INDEX IF NOT EXISTS events_search
ON analytics.events ()
USING 'OpenSearchIndex'
WITH OPTIONS = {
'indexName': 'events-search',
'skipBuildIfTableNotEmpty': 'true',
'numShards': '5',
'numReplicas': '0'
};Pre-provisioned index
When you manage the OpenSearch index externally, tell HCD not to create it:
CREATE CUSTOM INDEX IF NOT EXISTS orders_search
ON commerce.orders ()
USING 'OpenSearchIndex'
WITH OPTIONS = {
'indexName': 'orders-search',
'createIndexIfNotExists': 'false'
};Defer mapping to an OpenSearch index template
Set applyDefaultSchema=false and omit applyCustomSchema and customMappingsJson to send the Create Index request with no mappings body.
OpenSearch then applies any composable index template whose index_patterns matches the index name.
Use this approach when:
-
You manage the mapping outside HCD. For example, an operations team or a shared OpenSearch template library manages it.
-
You need analyzers, token filters, or index settings that HCD’s automatic mapping does not support.
-
The same template should apply consistently across multiple environments.
-
Create the template in OpenSearch (once, before any
CREATE INDEX):PUT _index_template/articles-template { "index_patterns": ["articles-search"], "template": { "settings": { "number_of_shards": 3, "number_of_replicas": 1 }, "mappings": { "properties": { "title": { "type": "text", "analyzer": "english" }, "body": { "type": "text", "analyzer": "english" }, "tags": { "type": "keyword" }, "published_at": { "type": "date" } } } } } -
Create the CQL index with both schema flags off. HCD creates the index without a mappings body and OpenSearch applies the template automatically:
CREATE CUSTOM INDEX IF NOT EXISTS articles_search ON blog.articles () USING 'OpenSearchIndex' WITH OPTIONS = { 'indexName': 'articles-search', 'applyDefaultSchema': 'false' };
-
If you also set numShards or numReplicas in WITH OPTIONS, HCD includes them in the Create Index settings block and they override any matching template values.
Omit them to let the template control all settings.
Configuration
You can provide settings as JVM system properties, environment variables, or cassandra.yaml keys.
When HCD is started with HCDConfigurationLoader (the default), you can place all OpenSearch settings in the hcd: top-level section of cassandra.yaml.
For an annotated cassandra.yaml example covering every available setting, see OpenSearch settings in cassandra.yaml.
System properties take precedence over environment variables, which take precedence over cassandra.yaml values, which take precedence over built-in defaults.
OpenSearch connection
The properties that configure the connection to the OpenSearch cluster:
| Property | Environment variable | Default | Description |
|---|---|---|---|
|
|
Comma-separated list of OpenSearch server URLs. |
|
|
|
|
Enable GZIP compression on outgoing requests (request body and |
|
|
|
Maximum retry attempts for index operations.
Set to |
|
|
|
Initial retry delay. |
|
|
|
Maximum retry delay (exponential backoff cap). |
Mutation processing
The properties that tune the internal mutation processing pipeline:
| Property | Environment variable | Default | Description |
|---|---|---|---|
|
|
|
Timeout for mutation processing. |
|
|
|
Number of threads for mutation processing. |
|
|
|
Number of threads for OpenSearch bulk indexing. |
|
|
|
Number of documents per bulk request. |
|
|
|
Number of rows buffered per indexer build batch. |
|
|
|
Number of hits resolved per HCD read batch during |
|
|
|
Maximum wait time before flushing a partial batch. |
|
|
|
Timeout for remediation operations. |
|
|
|
Maximum retry attempts for bulk operations. |
|
|
|
Initial retry delay for bulk operations. |
|
|
|
Maximum retry delay for bulk operations. |
|
|
|
Number of threads used to execute |
|
|
unbounded |
Maximum retry attempts for general processing operations (separate from |
|
|
|
Initial retry delay for general processing operations. |
|
|
|
Maximum retry delay for general processing operations. |
Storage
The replicator supports two storage modes for pending mutations:
-
memory: Holds pending mutations in heap memory only. Faster, but the node loses any mutations not yet forwarded to OpenSearch if it crashes. -
disk(default): Writes pending mutations to a durable write-ahead log on disk. Survives crashes and node restarts.
The storage properties:
| Property | Environment variable | Default | Description |
|---|---|---|---|
|
|
|
Storage mode for pending mutations.
Accepted values: |
|
|
(auto-discovered) |
Directory for persistent storage (disk mode only).
When empty, HCD auto-discovers the directory from HCD’s data directory configuration (typically |
|
|
|
Maximum number of pending mutations before HCD applies flow control and slows incoming writes. |
The disk storage properties:
| Property | Environment variable | Default | Description |
|---|---|---|---|
|
|
|
Per-table memtable size limit. |
|
|
|
Maximum commit log file size. |
|
|
|
Checkpoint trigger size. |
|
|
|
Fsync batching window. |
|
|
|
Flush immediately when the queue is empty (low-latency mode). |
|
|
|
Shared memory budget across all tables. |
|
|
|
Complete |
|
|
|
Flush interval in milliseconds for completed mutations. |
The replicator maintains persistent state in the following directory structure, where <storage.directory> is the configured storage directory, <keyspace> and <table> identify the HCD table being replicated, and <id> is a file identifier the replicator assigns automatically:
<storage.directory>/
├── <keyspace>/
│ └── <table>/
│ ├── manifest.json # Current checkpoint state
│ ├── cl-<id>.log # Commit log files (write-ahead log)
│ └── seg-<id>.dat # Data segment files (flushed memtables)-
Commit log: Rolling log files (
cl-*.log) that record all pending mutations for crash recovery. -
Segments: Immutable data files (
seg-*.dat) containing data written from memory to disk. -
Manifest: JSON file tracking current checkpoint state and active files.
Other system properties
The properties that control miscellaneous integration behaviors:
| Property | Environment variable | Default | Description |
|---|---|---|---|
|
|
|
Allow creating or keeping an OpenSearch index on a table with dynamically masked columns. For more information, see Tables that cannot be indexed. |
|
|
|
When |
|
|
|
Attach the |
|
|
|
Maximum size of a single aggregation’s JSON before HCD returns it as a result column. |
|
|
|
Maximum size of the raw OpenSearch response for a |
