CREATE INDEX (Apache Kafka)

A Kafka index is a custom index that uses the KafkaReplicator class to replicate row mutations from a HCD table to a Apache Kafka® topic in real time.

Unlike SAI and 2i indexes, Kafka indexes don’t store index data with the table’s SSTables, and you can’t query the table through them. Instead, the index intercepts row mutations at the coordinator and publishes them to your Kafka cluster, where downstream consumers read them.

KafkaReplicator captures all write types, including standard writes, BATCH writes, and lightweight transactions (LWT). For more information, see Linearizable consistency.

Before you create a Kafka index, you must enable the Kafka integration and configure the connection to your Kafka cluster on each node that you want to replicate from. Kafka indexes require HCD 2.0.8 or later.

For prerequisites and setup instructions, see Integrate Apache Kafka® with Hyper-Converged Database (HCD).

Create a Kafka index

To create a Kafka index, define the table name, the optional columns to replicate, and the KafkaReplicator index class.

If the table already contains data, the existing data is replicated to the Kafka topic during the execution of this statement, unless you set skipBuildIfTableNotEmpty to true. After the index has been created, HCD publishes every subsequent change to the table.

This index type cannot be altered. If you need to modify an index, you must drop the index, and then create a new index.

 
CREATE CUSTOM INDEX [ IF NOT EXISTS ] <index_name>
  ON [ <keyspace_name>.]<table_name> ( [ <column_name> [, <column_name> ... ] ] )
  USING 'KafkaReplicator'
  [ WITH OPTIONS = { <option_map> } ] ;
Legend
Syntax conventions Description

UPPERCASE

Literal keyword.

Lowercase

Not literal.

< >

Variable value. Replace with a user-defined value.

[]

Optional. Square brackets ([]) surround optional command arguments. Do not type the square brackets.

( )

Group. Parentheses ( ( ) ) identify a group to choose from. Do not type the parentheses.

|

Or. A vertical bar (|) separates alternative elements. Type any one of the elements. Do not type the vertical bar.

...

Repeatable. An ellipsis ( ... ) indicates that you can repeat the syntax element as often as required.

'<Literal string>'

Single quotation (') marks must surround literal strings in CQL statements. Use single quotation marks to preserve upper case.

{ <key> : <value> }

Map collection. Braces ({ }) enclose map collections or key value pairs. A colon separates the key and the value.

<<datatype1>,<datatype2>>

Set, list, map, or tuple. Angle brackets ( < > ) enclose data types in a set, list, map, or tuple. Separate the data types with a comma.

<cql_statement>;

End CQL statement. A semicolon (;) terminates all CQL statements.

[--]

Separate the command line options from the command arguments with two hyphens ( -- ). This syntax is useful when arguments might be mistaken for command line options.

' <<schema\> ... </schema\>> '

Search CQL only: Single quotation marks (') surround an entire XML schema declaration.

@<xml_entity>='<xml_entity_type>'

Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrConfig files.

Options

IF NOT EXISTS

Allows silent skip by checking for existing indexes with the same name. If a match is found, the operation silently does nothing (no error is thrown). If no match is found, the operation creates the index.

index_name

Unique identifier for the index.

Enclose in quotes to preserve capitalization.

Must be unique across all indexes in the keyspace. Although indexes apply to a specific table, they are created at the keyspace level.

keyspace_name

Optional. The name of the keyspace that contains the table to replicate. If no name is specified, the current keyspace is used.

table_name

The name of the table to replicate.

You can’t create a Kafka index on a table that has static columns or counter columns. For all restrictions, see Tables that cannot be indexed.

column_name

Optional. The non-primary-key columns to include in the message value. Leave the parentheses empty to include every non-primary-key column.

Primary key columns are always included in the message key.

You can also select columns with the columnsToIndex option.

WITH OPTIONS = { <option_map> }

The <option_map> is a map of options and their values defined in JSON simple format. Pass all values as strings.

Options aren’t required. For the supported options, see Index options.

Index options

The following options are specific to Kafka indexes. For the complete reference, see Index options.

Option Default Description

topicName

hcd_<keyspace>_<table>

The Kafka topic to publish messages to. Must be non-blank if supplied.

createTopicIfNotExists

false

When true, HCD creates the Kafka topic on CREATE INDEX if it doesn’t already exist. When deduplication is enabled, HCD also creates the events topic.

partitions

1

Number of partitions when HCD creates the topic. Also the default for eventsTopicPartitions.

replicationFactor

1

Replication factor when HCD creates the topic. Also the default for eventsTopicReplicationFactor.

format

json

Serialization format for Kafka messages. Accepted values: json, avro. avro requires a configured Confluent Schema Registry URL.

columnsToIndex

All non-primary-key columns

Comma-separated list of non-primary-key columns to include in the message value.

deduplicationEnabled

false

When true, HCD publishes through an intermediate events topic so that only the latest version of each row reaches the destination topic. For more information, see Deduplication.

eventsTopic

<topicName>-events

Name of the intermediate events topic. Used only when deduplicationEnabled is true.

eventsTopicPartitions

Value of partitions

Number of partitions for the events topic when HCD creates it.

eventsTopicReplicationFactor

Value of replicationFactor

Replication factor for the events topic when HCD creates it.

eventsConsumerGroupId

hcd-dedup-<keyspace>-<table>

Kafka consumer group ID shared by all deduplication workers for this table across the cluster.

skipBuildIfTableNotEmpty

false

When true, HCD skips the initial SSTable scan even when the table already contains data. HCD replicates only writes made after CREATE INDEX.

indexNonPrimaryReplicas

true

When true, every replica independently processes each mutation. Set to false so that only the primary replica processes rows, which avoids duplicate processing. With false, HCD doesn’t replicate rows held exclusively by a non-primary replica until a rebuild or until that replica becomes primary.

Replicated messages

HCD publishes one message for each row mutation. The message key contains the primary key columns, and the message value contains the replicated columns. A delete publishes a message with a null value (a tombstone).

For example, for a table with a primary key column id and the columns name and price:

-- INSERT: key: {"id":42}  value: {"name":"Widget","price":9.99}
INSERT INTO ks.products (id, name, price) VALUES (42, 'Widget', 9.99);

-- DELETE: key: {"id":42}  value: null  (tombstone)
DELETE FROM ks.products WHERE id = 42;

For the complete message format, including Avro schema mapping, see Message format.

Kafka index examples

The examples use the following table. Each example is an alternative configuration for the table, so don’t run them in sequence.

CREATE TABLE ks.products (
    id    int PRIMARY KEY,
    name  text,
    price float
);
Basic index

Create an index with an explicit topic name. This example assumes that you enabled the Kafka integration and configured the broker connection.

CREATE CUSTOM INDEX products_kafka
  ON ks.products (name, price)
  USING 'KafkaReplicator'
  WITH OPTIONS = {'topicName': 'ks.products'};
Create the topic automatically

Create the Kafka topic at index creation time, with custom partition and replication settings.

CREATE CUSTOM INDEX products_kafka
  ON ks.products (name, price)
  USING 'KafkaReplicator'
  WITH OPTIONS = {
    'topicName':              'ks.products',
    'createTopicIfNotExists': 'true',
    'partitions':             '6',
    'replicationFactor':      '3'
  };
Avro format

Use Avro serialization to derive schemas and register them in the Confluent Schema Registry. This example requires the cassandra.kafka.schema.registry.url system property. For more information, see Configuration.

CREATE CUSTOM INDEX products_avro
  ON ks.products (name, price)
  USING 'KafkaReplicator'
  WITH OPTIONS = {
    'topicName': 'ks.products',
    'format':    'avro'
  };
Replicate selected columns

List only the columns that you want in the message value to reduce message size.

CREATE CUSTOM INDEX products_kafka
  ON ks.products (price)
  USING 'KafkaReplicator'
  WITH OPTIONS = {'topicName': 'ks.products'};
Deduplication

Publish only the latest version of each row to the destination topic. HCD creates the destination topic and the events topic if they don’t exist.

CREATE CUSTOM INDEX products_dedup
  ON ks.products (name, price)
  USING 'KafkaReplicator'
  WITH OPTIONS = {
    'topicName':              'ks.products',
    'deduplicationEnabled':   'true',
    'eventsTopic':            'ks.products-events',
    'eventsTopicPartitions':  '6',
    'eventsConsumerGroupId':  'hcd-dedup-ks-products',
    'createTopicIfNotExists': 'true'
  };
Skip the initial backfill

Replicate only writes made after CREATE INDEX, and leave existing rows out of Kafka until you run a separate backfill.

CREATE CUSTOM INDEX products_kafka
  ON ks.products (name, price)
  USING 'KafkaReplicator'
  WITH OPTIONS = {
    'topicName':                'ks.products',
    'skipBuildIfTableNotEmpty': 'true'
  };

See also

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM