CREATE INDEX (SAI)
|
For performance and maintenance reasons, don’t create superfluous indexes, and don’t create multiple types of indexes on the same column. Identify the columns that are required for your query patterns, and then choose the one index type that best fits your use case. If you need to index the same column multiple times, consider table-per-query and denormalization. |
Creates a storage-attached indexing (SAI) index on a column. A table can have multiple SAI indexes, but each column can have only one index. If a table has multiple SAI indexes, you can use one or more indexes in the same query.
If the column already contains data, the existing data is indexed when you create the index. After creating an index on a column, updates to that column automatically trigger reindexing.
|
This index type cannot be altered. If you need to modify an index, you must drop the index, and then create a new index. |
If you’re new to SAI, see the SAI quickstart.
Syntax
CREATE CUSTOM INDEX [ IF NOT EXISTS ] [ <index_name> ]
ON [<keyspace_name>.]<table_name>
([ KEYS | VALUES | ENTRIES | FULL ] <column_name>)
USING StorageAttachedIndex
[ WITH OPTIONS = { <options> } ] ;
| Syntax conventions | Description |
|---|---|
UPPERCASE |
Literal keyword. |
Lowercase |
Not literal. |
|
Variable value. Replace with a user-defined value. |
|
Optional.
Square brackets ( |
|
Group.
Parentheses ( |
|
Or.
A vertical bar ( |
|
Repeatable.
An ellipsis ( |
|
Single quotation ( |
|
Map collection.
Braces ( |
|
Set, list, map, or tuple.
Angle brackets ( |
|
End CQL statement.
A semicolon ( |
|
Separate the command line options from the command arguments with two hyphens ( |
|
Search CQL only: Single quotation marks ( |
|
Search CQL only: Identify the entity and literal value to overwrite the XML element in the schema and solrConfig files. |
Options
At minimum, you must specify the table name, column name, and index type when creating an SAI index.
- CREATE CUSTOM INDEX … USING StorageAttachedIndex
-
To create an SAI index, use the
CREATE CUSTOM INDEXcommand and specify the SAI class withUSING StorageAttachedIndex. For example:CREATE CUSTOM INDEX index_name ON keyspace_name.table_name (column_name) USING 'StorageAttachedIndex' WITH OPTIONS = { ... } ; - IF NOT EXISTS
-
Allows silent skip by checking for existing indexes with the same name. If a match is found, the operation silently does nothing (no error is thrown). If no match is found, the operation creates the index.
- index_name
-
Unique identifier for the index.
Enclose in quotes to preserve capitalization.
Must be unique across all indexes in the keyspace. Although indexes apply to a specific table and column, they are created at the keyspace level.
If no name is specified, the default is
<table_name>_<column_name>_idx. - keyspace_name
-
Optional name of the keyspace that contains the table to index. If no name is specified, the current keyspace is used.
- table_name (column_name)
-
The name of the table and column being indexed. For
mapcollections, a function is required. See Column specification. - WITH OPTIONS
-
Additional index options in JSON simple format. See WITH OPTIONS.
Column specification
An index definition must contain the name of the column to index.
For all data types except map collections, specify the column name in the index definition.
For example:
CREATE CUSTOM INDEX IF NOT EXISTS idx_name
ON keyspace_name.table_name (column_name)
USING 'StorageAttachedIndex';
For map collections only, use the FULL(), KEYS(), VALUES(), or ENTRIES() functions to specify the dimension to index:
-
FULL(<column_name>): Index the full map. -
KEYS(<column_name>): Index only the keys. -
VALUES(<column_name>): Index only the values. -
ENTRIES(<column_name>): Index the entries (key-value pairs).
For example:
CREATE CUSTOM INDEX IF NOT EXISTS map_entries_idx
ON keyspace_name.table_name (ENTRIES(map_column_name))
USING 'StorageAttachedIndex';
|
You can create multiple SAI indexes on the same |
Supported data types
All data types are supported for SAI indexes except counter, geospatial types (PointType, LineStringType, PolygonType), and non-frozen UDTs.
Tables with static columns are compatible with SAI.
Primary key columns
You can create an SAI index on composite partition key columns and clustering columns with the exception of analyzed and filtered indexes.
A composite partition key is a partition key with two or more columns.
To query a composite partition key without SAI, you must specify all partition key columns in a WHERE clause, or enable ALLOW FILTERING (not recommended).
However, with SAI, you can index one partition key column, and then query on that indexed column.
If you index all partition key columns, you can only query on one column at a time.
A clustering column sorts the rows within a partition.
You can create an SAI index on a clustering column to enable queries on that column without requiring other columns or ALLOW FILTERING.
DataStax recommends limiting queries to one partition.
Queries that use an indexed clustering column with a partition key filter will have better performance.
A single-column partition key is already covered by the table’s primary key index. If you attempt to create an SAI index on a single-column partition key, the request fails with the following error:
InvalidRequest: Error from server: code=2200 [Invalid query] message=
"Cannot create secondary index on the only partition key column id"
WITH OPTIONS
Use the WITH OPTIONS clause to define any of the following additional index options in JSON simple format.
Analyzer options
SAI uses the Apache Lucene™ Java Analyzer API to transform text columns into tokens for indexing and querying which can use built-in or custom analyzers.
The analyzer determines how a column’s values will be analyzed before indexing occurs, and then the analyzed index stores values derived from the raw column values. The stored values are dependent on the analyzer configuration options, which include tokenization, filtering, and charFiltering. When the index is used in a query, the analyzer is applied to the query term.
To configure an analyzed index, you must specify the index_analyzer option in the WITH OPTIONS clause.
The value of index_analyzer is either:
-
A single string that selects a built-in analyzer, which is a preconfigured set of tokenizers and filters. For example:
{ "index_analyzer": "standard" } -
A JSON object that defines a custom analyzer with a
tokenizer, optionalfiltersandcharFilters, and any other fields required to configure the analyzer. For example:{ "index_analyzer": { "tokenizer": { "name": "ngram", "args": { "minGramSize": "2", "maxGramSize": "3" } }, "filters": [ { "name": "lowercase" } ] } }
There are many analyzers, tokenizers, and filters from the Apache Lucene™ project (version 9.8.0) that you can use in your index_analyzer configuration.
| Type | Usage | Values |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
For analyzer examples, see SAI examples.
Non-tokenizing filters
Non-tokenizing filters are similar to analyzers, but they cannot be combined with analyzers in the same index. However, you can create an index with multiple non-tokenizing filters. When combined, the filters are applied in the order they are specified.
Non-tokenizing filters are specified as top level children of the WITH OPTIONS clause.
For example:
CREATE CUSTOM INDEX comment_case_insensitive_idx ON cycling.comments_vs (comment)
USING 'StorageAttachedIndex'
WITH OPTIONS = { 'case_sensitive': false };
| Option | Description | Default |
|---|---|---|
|
Whether indexed and query values are case-sensitive. |
|
|
Whether indexed and query values are normalized for Unicode characters. SAI supports Normalization Form C (NFC) Unicode. When set to When implementations keep strings in a normalized form, equivalent strings have a unique binary representation. See Unicode Standard Annex #15, Unicode Normalization Forms. |
|
|
Whether to convert ASCII characters to their associated UTF-8 values in the same way as the Apache Lucene™ ASCIIFoldingFilter. When set to |
|
For non-tokenizing filter examples, see SAI examples.
Vector search options
Vector search options are only for SAI indexes on vector columns used in queries with ORDER BY … ANN OF.
|
To change the similarity function, you must drop and re-create the index. |
Option |
Description |
Default |
|
Vector search relies on computing the similarity between vectors to identify relevant matches. The similarity function is used for this calculation. It is important that you use the appropriate similarity function for your model and use case. Choices are |
|
|
Automatically configures If |
|
|
For vector search tuning in specific use cases. Misconfiguration can cause severe performance degradation. Test in isolation before applying to production databases. Controls the maximum number of connections per node in the graph.
The actual graph degree is twice this value.
Higher values increase graph quality but also increase storage and query costs.
Must be an integer from |
|
|
For vector search tuning in specific use cases. Misconfiguration can cause severe performance degradation. Test in isolation before applying to production databases. Controls how many candidates to evaluate during graph construction.
Higher values increase graph quality but also increase build time.
Must be an integer from |
|
|
For vector search tuning in specific use cases. Misconfiguration can cause severe performance degradation. Test in isolation before applying to production databases. Whether to enable hierarchical graph construction. |
|
|
For vector search tuning in specific use cases. Misconfiguration can cause severe performance degradation. Test in isolation before applying to production databases. Controls graph pruning during construction.
Higher values result in denser graphs.
Explicitly setting this value overrides both memtable and SSTable defaults.
Must be greater than |
|
|
For vector search tuning in specific use cases. Misconfiguration can cause severe performance degradation. Test in isolation before applying to production databases. Controls how aggressively to explore the graph during search.
Higher values increase recall at the cost of latency.
Must be greater than |
|
For vector search examples with SAI, see Query with SAI.
Examples: SAI indexes on primary key columns
These examples create and query SAI indexes on primary key columns using the following table:
CREATE TABLE IF NOT EXISTS cycling.rank_by_year_and_name (
race_year int,
race_name text,
cyclist_name text,
rank int,
PRIMARY KEY ((race_year, race_name), rank)
);
Partition key columns
-
Create SAI index indexes on the composite partition key columns
race_yearandrace_name:CREATE CUSTOM INDEX IF NOT EXISTS race_year_idx ON cycling.rank_by_year_and_name (race_year) USING 'StorageAttachedIndex'; CREATE CUSTOM INDEX IF NOT EXISTS race_name_idx ON cycling.rank_by_year_and_name (race_name) USING 'StorageAttachedIndex'; -
Query either or both partition key columns in
SELECTstatements.With SAI, you don’t need to index or specify all columns in the composite partition key to query the table, and SAI indexes do not require
ALLOW FILTERINGwhen querying on a column that is not the full partition key.-
Get all rows that match a specified
race_yearvalue:SELECT * FROM cycling.rank_by_year_and_name WHERE race_year = 2014;race_year | race_name | rank | cyclist_name -----------+--------------------------------------------+------+---------------------- 2014 | 4th Tour of Beijing | 1 | Phillippe GILBERT 2014 | 4th Tour of Beijing | 2 | Daniel MARTIN 2014 | 4th Tour of Beijing | 3 | Johan Esteban CHAVES 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 1 | Daniel MARTIN 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 2 | Johan Esteban CHAVES 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 3 | Benjamin PRADES (6 rows)
-
-
Use the
ANDoperator to query on both indexes:SELECT * FROM cycling.rank_by_year_and_name WHERE race_year = 2014 AND race_name = 'Tour of Japan - Stage 4 - Minami > Shinshu';The results are narrowed to the specific year and race name:
race_year | race_name | rank | cyclist_name -----------+--------------------------------------------+------+---------------------- 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 1 | Daniel MARTIN 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 2 | Johan Esteban CHAVES 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 3 | Benjamin PRADES (3 rows)
Clustering columns
-
Index the
rankclustering column:CREATE CUSTOM INDEX IF NOT EXISTS rank_idx ON cycling.rank_by_year_and_name (rank) USING 'StorageAttachedIndex'; -
Query only the
rankcolumn:SELECT * FROM cycling.rank_by_year_and_name WHERE rank = 3;race_year | race_name | rank | cyclist_name -----------+--------------------------------------------+------+---------------------- 2014 | 4th Tour of Beijing | 3 | Johan Esteban CHAVES 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 3 | Benjamin PRADES 2015 | Tour of Japan - Stage 4 - Minami > Shinshu | 3 | Thomas LEBAS (3 rows) -
Query the
race_year(partition key) andrankcolumns.Limiting clustering column queries with a partition key condition is recommended for better performance.
SELECT * FROM cycling.rank_by_year_and_name WHERE race_year = 2014 AND rank = 3 ALLOW FILTERING;race_year | race_name | rank | cyclist_name -----------+--------------------------------------------+------+---------------------- 2014 | 4th Tour of Beijing | 3 | Johan Esteban CHAVES 2014 | Tour of Japan - Stage 4 - Minami > Shinshu | 3 | Benjamin PRADES (2 rows)
Examples: SAI indexes on non-primary key columns
The following examples demonstrate how to create and query SAI indexes on non-primary key columns.
Text and date columns
This example creates an SAI index on a date column and a non-analyzed SAI index on a text column to demonstrate how you can use one or more SAI indexes in queries, including multiple matches with logical operators and inequality ranges.
For more powerful text queries, use analyzed SAI indexes.
-
Create a table called
cycling.alt_statswith atextcolumn callednationalityand adatecolumn calledbirthday:CREATE TABLE IF NOT EXISTS cycling.cyclist_alt_stats ( id UUID PRIMARY KEY, lastname text, birthday date, nationality text, weight float, w_units text, height float, first_race date, last_race date ); -
Index the
nationalitycolumn:CREATE CUSTOM INDEX nationality_idx ON cycling.cyclist_alt_stats (nationality) USING 'StorageAttachedIndex'; -
Query only the
nationalitycolumn:SELECT first_race, last_race, birthday FROM cycling.cyclist_alt_stats WHERE nationality = 'France';first_race | last_race | birthday ------------+------------+------------ 2006-03-15 | 2017-04-16 | 1990-05-27 (1 rows) -
Index the
birthdaycolumn:CREATE CUSTOM INDEX birthday_idx ON cycling.cyclist_alt_stats (birthday) USING 'StorageAttachedIndex'; -
Query only the
birthdaycolumn:SELECT lastname, birthday, nationality FROM cycling.cyclist_alt_stats WHERE birthday = '1981-03-29';lastname | birthday | nationality -----------+------------+------------- VEIKKANEN | 1981-03-29 | Finland (1 rows) -
Query on both the
nationalityandbirthdaycolumns:SELECT lastname, birthday, nationality FROM cycling.cyclist_alt_stats WHERE birthday = '1991-08-25' AND nationality = 'Ethiopia'lastname | birthday | nationality ----------+------------+------------- GRMAY | 1991-08-25 | Ethiopia (1 rows) -
Use the
INoperator to query a list of values for thenationalitycolumn:SELECT * FROM cycling.cyclist_alt_stats WHERE nationality IN ('Finland', 'France');id | birthday | first_race | height | last_race | lastname | nationality | w_units | weight --------------------------------------+------------+------------+--------+------------+-----------+-------------+---------+-------- c09e9451-50da-483d-8108-e6bea2e827b3 | 1981-03-29 | 1996-05-21 | 1.78 | 2012-10-02 | VEIKKANEN | Finland | kg | 66 4ceb495c-55ab-4f71-83b9-81117252bb13 | 1990-05-27 | 2006-03-15 | null | 2017-04-16 | DUVAL | France | null | null (2 rows) -
Use an inequality range to query the
birthdaycolumn:SELECT lastname, birthday, nationality FROM cycling.cyclist_alt_stats WHERE birthday >'1981-03-29';lastname | birthday | nationality -----------+------------+------------- BRUTT | 1982-01-29 | Russia ISAYCHEV | 1986-04-21 | Russia TSATEVICH | 1989-07-05 | Russia PAULINHO | 1990-05-27 | Portugal GRMAY | 1991-08-25 | Ethiopia BELKOV | 1985-01-09 | Russia DUVAL | 1990-05-27 | France (7 rows) -
Use the
ORoperator to find matches on either column:SELECT lastname, birthday, nationality FROM cycling.cyclist_alt_stats WHERE birthday = '1991-08-25' OR nationality = 'Ethiopia';lastname | birthday | nationality ----------+------------+------------- GRMAY | 1991-08-25 | Ethiopia (1 rows)
Timestamp
-
Create a table called
cycling.commentswithtimestampcolumn calledcreated_atif it doesn’t already exist.CREATE TABLE IF NOT EXISTS cycling.comments ( record_id timeuuid, id uuid, commenter text, comment text, created_at timestamp, PRIMARY KEY (id, created_at) ) WITH CLUSTERING ORDER BY (created_at DESC); -
Index the
created_atcolumn:CREATE CUSTOM INDEX created_at_idx ON cycling.comments (created_at) USING 'StorageAttachedIndex'; -
Query on the
created_atcolumn using numeric operators, such as equality, inequality, and range:- Equality
-
SELECT commenter, comment, created_at FROM cycling.comments WHERE created_at = '2024-06-07 03:13:40.268';commenter | comment | created_at -----------+----------------------+--------------------------------- Amy | Ride in any weather! | 2024-06-07 03:13:40.268000+0000 (1 rows) - Inequality range
-
SELECT commenter, comment, created_at FROM cycling.comments WHERE created_at < '2024-06-07 03:13:40.268';commenter | comment | created_at -----------+----------------------------------------+--------------------------------- Alex | Raining too hard should have postponed | 2023-04-01 11:21:59.001000+0000 (1 rows)
Float
Float columns can be indexed and searched:
-
Create a table called
cycling.cyclist_alt_statswithfloatcolumn calledweightif it doesn’t already exist.CREATE TABLE IF NOT EXISTS cycling.cyclist_alt_stats ( id UUID PRIMARY KEY, lastname text, birthday date, nationality text, weight float, w_units text, height float, first_race date, last_race date ); -
Index the
weightcolumn:CREATE CUSTOM INDEX weight_idx ON cycling.cyclist_alt_stats (weight) USING 'StorageAttachedIndex'; -
Query on the
weightcolumn using numeric operators, such as equality, inequality, and range:- Equality
-
SELECT lastname, nationality, weight FROM cycling.cyclist_alt_stats WHERE weight = 66;lastname | nationality | weight -----------+-------------+-------- VEIKKANEN | Finland | 66 (1 rows) - Inequality range
-
SELECT lastname, nationality, weight FROM cycling.cyclist_alt_stats WHERE weight > 70;lastname | nationality | weight ----------+-------------+-------- ISAYCHEV | Russia | 80 BELKOV | Russia | 71 (2 rows)
Tuple
Tuples columns can be indexed and searched.
Like list and set collections, indexes on tuple columns include the entire tuple.
You cannot select individual elements of a tuple for indexing.
Unlike collections, tuples don’t support the CONTAINS operator.
You must query on the full tuple value.
-
Create a table called
cycling.nation_rankwithtuplecolumn calledinfoif it doesn’t already exist.CREATE TABLE IF NOT EXISTS cycling.nation_rank ( nation text PRIMARY KEY, info tuple<int, text, int> ); -
Index the
infocolumn:CREATE CUSTOM INDEX info_idx ON cycling.nation_rank (info) USING 'StorageAttachedIndex'; -
Query the table using the
infocolumn with an equality operator on the full tuple value.Tuples can only be filtered using the full tuple value.
SELECT * FROM cycling.nation_rank WHERE info = (3, 'Phillippe GILBERT', 6222);nation | info ---------+-------------------------------- Belgium | (3, 'Phillippe GILBERT', 6222) (1 rows)
User-defined type (UDT)
SAI can index user-defined type (UDT) columns and list columns containing UDT elements.
This example shows how to index a list of UDTs:
-
Create a UDT named
racein thecyclingkeyspace:CREATE TYPE IF NOT EXISTS cycling.race ( race_title text, race_date timestamp, race_time text ); -
Create a table named
cyclist_raceswith alistcolumn namedracescontaining UDT elements:CREATE TABLE IF NOT EXISTS cycling.cyclist_races ( id UUID PRIMARY KEY, lastname text, firstname text, races list<FROZEN <race>> ); -
Create an SAI index on the
racescolumn:CREATE CUSTOM INDEX IF NOT EXISTS races_idx ON cycling.cyclist_races (races) USING 'StorageAttachedIndex'; -
Query the
racescolumn.Because
racesis alist, you can use theCONTAINSoperator.SELECT * FROM cycling.cyclist_races WHERE races CONTAINS { race_title:'Rabobank 7-Dorpenomloop Aalburg', race_date:'2015-05-09', race_time:'02:58:33'};Because
racesis a list of UDTs, theracesvalues can be long:id | firstname | lastname | races --------------------------------------+-----------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 5b6962dd-3f90-4c93-8f61-eabfa4a803e2 | Marianne | VOS | [race(race_title='Rabobank 7-Dorpenomloop Aalburg', race_date=datetime.datetime(2015, 5, 9, 0, 0), race_time='02:58:33'), race(race_title='Ronde van Gelderland', race_date=datetime.datetime(2015, 4, 19, 0, 0), race_time='03:22:23')] (1 rows)