Integrate OpenSearch with Hyper-Converged Database (HCD)

You can use OpenSearch to search and analyze data that you store in Hyper-Converged Database (HCD). HCD captures writes at the coordinator level and forwards them to OpenSearch through a durable write-ahead log and a batched processing pipeline. The integration uses OpenSearchIndex, which implements the database’s secondary index API for index lifecycle management.

With the OpenSearch integration, you can do the following:

Perform full-text searches on HCD data.
Search nested data structures (UDTs, collections, and tuples).
Run aggregation queries.
Build analytics dashboards.

OpenSearchIndex captures all write types in HCD, including the following:

Standard writes.
LOGGED batches (atomic across multiple partitions) and UNLOGGED batches (atomic only if they target a single partition). For more information, see BATCH.
Lightweight transactions (LWT/CAS) using IF NOT EXISTS and IF ... conditions, for both Paxos v1 and v2 engines. For more information, see Linearizable consistency.

This includes recovery of mutations committed by a background Paxos repair, SERIAL reads, LOGGED batch recovery, and any other automatic recovery from failure modes in HCD.

Prerequisites

HCD 2.0.8 or later.
Host-level access to each HCD node that you want to replicate from, so that you can set environment variables and JVM system properties and restart the node. For more information, see Set JVM system properties at startup.
An OpenSearch cluster running OpenSearch 3.7.x, 3.8.x, or 3.9.x that is reachable from every HCD node that you want to replicate from.
An OpenSearch account that HCD can use to connect to your OpenSearch cluster. The account must have permission to create indexes in OpenSearch, unless you pre-provision the index. For more information, see Security and Pre-provisioned index.
A keyspace and table in HCD that you want to replicate.

The table cannot have static columns, counter columns, or dynamically masked columns. For all restrictions, see Tables that cannot be indexed.

Set up the OpenSearch integration

To set up the OpenSearch integration, do the following:

Enable the replicator and configure the connection to your OpenSearch cluster on each HCD node that you want to replicate from. The OpenSearch replicator is bundled with HCD, but you must enable it explicitly to capture HCD write operations. Set the replicator and the connection properties together, and then start or restart the node once.

In the examples, replace OPENSEARCH_URL_1,OPENSEARCH_URL_2 with the URLs of your OpenSearch servers, such as http://localhost:9200. The default is http://localhost:9200. To connect to a secured OpenSearch cluster, see Security. For all connection properties, see Configuration.

The method depends on your installation type:
Tarball installation
Pass --search (or -s) and the connection properties to hcd cassandra:
# Optional: enable the Management API to export metrics export USE_MGMT_API=true HCD_HOME/hcd cassandra --search \ -Dcassandra.opensearch.addresses=OPENSEARCH_URL_1,OPENSEARCH_URL_2 \ -Dcassandra.opensearch.index_retry.max_retries=3 \ -Dcassandra.opensearch.index_retry.initial_interval=1s \ -Dcassandra.opensearch.index_retry.max_interval=30s
When you use --search, HCD exports HCD_SEARCH_ENABLED=1 and appends the required JVM system properties to JVM_OPTS before launching.
Debian or RHEL package installation
Add the connection properties to jvm-server.options:
-Dcassandra.opensearch.addresses=OPENSEARCH_URL_1,OPENSEARCH_URL_2 -Dcassandra.opensearch.index_retry.max_retries=3 -Dcassandra.opensearch.index_retry.initial_interval=1s -Dcassandra.opensearch.index_retry.max_interval=30s
Then set HCD_SEARCH_ENABLED=1 in the service environment, and restart the service:
export HCD_SEARCH_ENABLED=1
Create a custom index on the table you want to replicate, using the OpenSearchIndex class:
```
CREATE CUSTOM INDEX IF NOT EXISTS INDEX_NAME ON KEYSPACE_NAME.TABLE_NAME ()
USING 'OpenSearchIndex'
WITH OPTIONS = {
    'indexName': 'OPENSEARCH_INDEX',
    'createIndexIfNotExists': 'true',
    'columnsToIndex': 'COLUMN_1,COLUMN_2'
};
```
Replace the following:
- INDEX_NAME: A name for the custom index in HCD
- KEYSPACE_NAME.TABLE_NAME: The keyspace and table that you want to replicate
- OPENSEARCH_INDEX: The name of the OpenSearch index to create or use
- COLUMN_1,COLUMN_2: The non-primary-key columns to replicate
  
  You can also specify the columns to index in the parentheses after the table name:
  CREATE CUSTOM INDEX IF NOT EXISTS INDEX_NAME ON KEYSPACE_NAME.TABLE_NAME (COLUMN_1, COLUMN_2) USING 'OpenSearchIndex'
  HCD performs additional validation on column types. Some types, such as non-frozen UDTs, might not be allowed in the column list, but OpenSearch can still index them using the columnsToIndex option.
  
  For all index options, see Index options.

The following example creates an articles table and an OpenSearchIndex that replicates the title and body columns to an OpenSearch index named articles-search:

CREATE TABLE blog.articles (
    id int PRIMARY KEY,
    title text,
    body text
);

CREATE CUSTOM INDEX IF NOT EXISTS articles_search
ON blog.articles ()
USING 'OpenSearchIndex'
WITH OPTIONS = {
    'indexName': 'articles-search',
    'createIndexIfNotExists': 'true',
    'columnsToIndex': 'title,body'
};

The CQL shell (cqlsh) DESCRIBE INDEX command does not work for OpenSearch indexes. To view the index definition, use DESCRIBE TABLE or DESCRIBE KEYSPACE instead.

For more examples, including custom field mappings, high-availability indexes, and index templates, see Index examples.

Test the integration

Write a row to the replicated table. For the previous example, run the following:

INSERT INTO blog.articles (id, title, body) VALUES (1, 'Search with OpenSearch', 'Full-text search for Cassandra data');

Query the document directly in OpenSearch:
```
curl "OPENSEARCH_URL/articles-search/_search?pretty"
```
Replace OPENSEARCH_URL with the URL of an OpenSearch server. The response includes the document for the row that you wrote.
Query the database with a CQL statement using the search index:
```
SELECT id, title FROM blog.articles WHERE expr(articles_search, '{"match": {"title": "search"}}');
```
For more query types, see Filter CQL queries with OpenSearch indexes. For index configuration examples, see Index examples.

If the document doesn’t appear, see Troubleshoot.

Security

The HCD OpenSearch integration supports basic authentication, API key authentication, mutual TLS, and TLS encryption for connections to your OpenSearch cluster.

Use the following properties to enable security and select an authentication type.

Store all sensitive values (passwords and API keys) in files and reference them using the *_file properties. Do not embed sensitive values directly in configuration files or JVM arguments.

Enable security and set the authentication type:

cassandra.opensearch.security.enabled=true
cassandra.opensearch.security.auth.type=BASIC

The auth.type property accepts: NONE, BASIC, API_KEY, CLIENT_CERT.

Basic authentication

Use basic authentication to connect with a username and password file:

cassandra.opensearch.security.auth.basic.username=USERNAME
cassandra.opensearch.security.auth.basic.password_file=PASSWORD_FILE_PATH

Replace the following:

USERNAME: The username for the OpenSearch connection
PASSWORD_FILE_PATH: The path to a file containing the password

API key authentication

Use API key authentication to connect with an API key ID and key file:

cassandra.opensearch.security.auth.apikey.id=API_KEY_ID
cassandra.opensearch.security.auth.apikey.key_file=KEY_FILE_PATH

Replace the following:

API_KEY_ID: Your OpenSearch API key ID
KEY_FILE_PATH: The path to a file containing the API key

Client certificate authentication (mutual TLS)

Use client certificate authentication to connect with a PKCS12 keystore:

cassandra.opensearch.security.auth.client_cert.keystore_path=KEYSTORE_FILE_PATH
cassandra.opensearch.security.auth.client_cert.keystore_password_file=KEYSTORE_PASSWORD_FILE_PATH
cassandra.opensearch.security.auth.client_cert.keystore_type=PKCS12

# Optional: separate key password
cassandra.opensearch.security.auth.client_cert.key_password_file=KEY_PASSWORD_FILE_PATH

Replace the following:

KEYSTORE_FILE_PATH: The path to your PKCS12 keystore file
KEYSTORE_PASSWORD_FILE_PATH: The path to a file containing the keystore password
KEY_PASSWORD_FILE_PATH: The path to a file containing the key password (optional)

TLS

Use the following properties to configure TLS encryption for connections to OpenSearch:

cassandra.opensearch.security.tls.enabled=true
cassandra.opensearch.security.tls.verify_hostnames=true
cassandra.opensearch.security.tls.truststore_path=TRUSTSTORE_FILE_PATH
cassandra.opensearch.security.tls.truststore_password_file=TRUSTSTORE_PASSWORD_FILE_PATH
cassandra.opensearch.security.tls.truststore_type=PKCS12
cassandra.opensearch.security.tls.protocols=TLSv1.3,TLSv1.2

# Optional: comma-separated list of cipher suites. Leave empty to use JVM defaults.
cassandra.opensearch.security.tls.ciphers=

Replace the following:

TRUSTSTORE_FILE_PATH: The path to your PKCS12 truststore file
TRUSTSTORE_PASSWORD_FILE_PATH: The path to a file containing the truststore password

Transparent Data Encryption

The replicator automatically applies the Transparent Data Encryption (TDE) settings from HCD to all files it writes to local storage for a given table.

After a key rotation, complete the following steps in order:

Run the key-rotation step (upgradesstables -a) described in the TDE documentation.
Reopen the store (node restart or index reload) so it picks up the new key. The store binds its cipher at open time; reopen it before the next step.
Run the checkpoint JMX operation. The replicator rewrites every pending record into fresh files under the current encryption configuration and deletes the older files.
Retire the previous key only after checkpoint() returns.

For more information, see About Transparent Data Encryption (TDE).

Troubleshoot

Use these troubleshooting tips to resolve common issues with the OpenSearch integration. For descriptions of the metrics that these tips refer to, see OpenSearch metrics.

If mutations don’t appear in OpenSearch, do the following:

Verify that the node is the primary replica for the partition.
Check the OpenSearch connection configuration.
Review logs for InProcessReplicatorService errors.
Check the OSMutationFailures and OSIndexerErrors metrics.
Verify that external versioning is not causing document rejections in OpenSearch.

If you experience index build failures, do the following:

Check OpenSearch cluster health.
Verify that HCD has index creation permissions in OpenSearch.
Review logs for IndexManager errors.
Check the OSIndexerInitialBuildFailures and OSIndexerRebuildFailures metrics.

If you reach write throughput and storage capacity, do the following:

Increase the OpenSearch cluster capacity.
Tune the bulk_batch_size and flush_interval settings.
Increase max_pending_count in the storage configuration.
Monitor the OSPendingOperations and OSAllowedMutations metrics.
Consider setting cassandra.opensearch.storage.mode=disk for higher pending-mutation capacity.

For more information, see Storage.

If you experience high memory usage, do the following:

Set cassandra.opensearch.storage.mode=disk to offload pending mutations to disk.

For more information, see Storage.
Reduce the max_pending_count configuration.
Increase the OpenSearch indexing throughput.
Monitor the OSPendingOperations gauge.

Integrate OpenSearch with Hyper-Converged Database (HCD)

Prerequisites

Set up the OpenSearch integration

Test the integration

Security

Basic authentication

API key authentication

Client certificate authentication (mutual TLS)

TLS

Transparent Data Encryption

Troubleshoot

See also

Was this helpful?

Give Feedback