Indexes in collections
By default, when you add or modify data within a collection, all properties in the added or modified documents are indexed. If you don’t want to index all properties, you can use the Data API to configure selective indexing.
|
Selective indexing is not recommended for all collections. Consider the advantages and disadvantages of selective indexing before applying it to any collection. DataStax recommends that you test your application in a development environment before applying selective indexing in production. |
Indexes enable Data API queries that need to filter or sort data based on indexed properties.
There are index limits for collections and databases. Furthermore, the index limit informs the collection limit. However, do not use selective indexing exclusively to bypass the collection limit. In most cases, selective indexing does not change a database’s collection limit due to the minimum required indexes for collections in Serverless (Vector) databases.
Carefully consider the advantages and disadvantages of selective indexing before applying it to your collections.
Considerations for selective indexing
The primary disadvantage of selective indexing is that sort and filter clauses can only use indexed fields. This means that you can’t perform these types of queries on fields that you do not index.
If you apply selective indexing to a collection, consider which properties might be important in queries that rely on sort and filter clauses, and make sure that you index those fields.
Potential advantages to selective indexing include the following:
-
Read/write performance: Selective indexing can increase write-time performance by reducing the amount of content that needs to be indexed. If certain properties are irrelevant to your application, you can save time by not indexing them.
-
Data capacity: Indexed properties are bound by lower maximum size limits to ensure efficient and performant read operations through the index. By comparison, non-indexed properties can support larger quantities of data, such as the body content of blog posts.
These outcomes are not guaranteed. The results of selective indexing depend on the specific characteristics and use of your applications and data.
DataStax recommends testing your application’s performance, under average and peak demand, in a non-production environment before deploying selective indexing to production. Make adjustments as necessary to optimize your application’s performance.
Configure indexing for collections
You set the indexing behavior when you create a collection. The configuration applies to all data that you load into the collection.
Collections that you create directly in the Astra Portal use default indexing and index all fields. You can’t change the indexing behavior for these collections.
Collections that you create with the Data API can use the optional indexing clause on the createCollection command to set the indexing behavior.
To maintain the default behavior and index all properties, omit the indexing clause from createCollection.
To apply selective indexing, include the indexing clause and either an allow or deny array that determines the fields to index.
|
If you apply selective indexing, make sure that your indexed fields support your application’s needs and query requirements. Evaluate the value of each property in your collection’s documents before you create your collection and decide which fields to index. |
-
Allow array
-
Deny array
To use the allow array in the indexing clause, specify the fields that you want to index.
For example, the following curl command creates a collection where the index includes only the values of the property1 and property2 fields:
curl -sS -L -X POST ${ASTRA_DB_API_ENDPOINT}/api/json/v1/${ASTRA_DB_KEYSPACE} \
--header "Token: ${ASTRA_DB_APPLICATION_TOKEN}" \
--header "Content-Type: application/json" \
--data '{
"createCollection": {
"name": "some_collection",
"options": {
"vector": {
"dimension": 5,
"metric": "cosine"
},
"indexing": {
"allow": [
"property1",
"property2"
]
}
}
}
}' | jqIf you add data to the collection that includes additional properties that weren’t present when you first created the collection, the index remains limited to property1 and property2.
When you use an 'allow' array for selective indexing, subsequent Data API queries can perform sort and filter clauses only on property1, property2, or both.
Attempting to perform these operations on any other fields returns an error.
If you use a wildcard (*) for the allow array, all properties are indexed.
This is equivalent to the default indexing behavior.
{
"indexing": {
"allow": [ "*" ]
}
}To use the deny array in the indexing clause, specify the fields that you do not want to index.
For example, the following curl command creates a collection where the index includes the values of all fields except property1, property3, property5.prop5b, and any sub-properties of property1 and property3:
curl -sS -L -X POST ${ASTRA_DB_API_ENDPOINT}/api/json/v1/${ASTRA_DB_KEYSPACE} \
--header "Token: ${ASTRA_DB_APPLICATION_TOKEN}" \
--header "Content-Type: application/json" \
--data '{
"createCollection": {
"name": "some_collection",
"options": {
"vector": {
"dimension": 5,
"metric": "cosine"
},
"indexing": {
"deny": [
"property1",
"property3",
"property5.prop5b"
]
}
}
}
}' | jqIf a property in the deny array has any sub-properties, those sub-properties are also inherently excluded from indexing.
For example, if property3 has two sub-properties (property3.prop3a and property3.prop3b), those sub-properties are also excluded from indexing because the deny array includes only the parent property3.
If you want to exclude a parent property and some of its sub-properties, you must specify both the parent and the specific sub-properties that you want to exclude.
For example, if you deny property3 and property3.prop3a, then property3.prop3b is still indexed.
To exclude specific sub-properties, but not the parent, you must specify those sub-properties in the deny array, as was done for property5.prop5b.
Furthermore, if you add data to the collection that includes additional properties or sub-properties that weren’t present when you first created the collection, those new properties are indexed if they are not named in the deny array, either explicitly or by inheritance.
When you use the deny array for selective indexing, subsequent Data API queries can perform sort and filter clauses on any field except the denied (non-indexed) fields.
Attempting to perform these operations on denied fields returns an error.
If you use a wildcard (*) for the deny array, no properties are indexed, not even $vector.
However, the collection can still create a small number of indexes for minimal functionality.
{
"indexing": {
"deny": [ "*" ]
}
}Non-indexed field error
The Data API returns an error if you attempt to sort or filter by a non-indexed property. For example:
UNINDEXED_FILTER_PATH("Unindexed filter path: The filter path ('FILTER') is not indexed")
UNINDEXED_SORT_PATH("Unindexed sort path")
ID_NOT_INDEXED("_id is not indexed")