Updating the index after data expires (TTL)
Time-To-Live (TTL) set on a CQL field also applies to the indexed values.
The DSE Search engine searches the index for expired documents and then deletes them from the index in batches of a configurable size, as defined by the ttl_index_rebuild_options in the dse.yaml file.
The default interval is 300 seconds (5 minutes).
Where is the dse.yaml file?
The location of the dse.yaml file depends on the type of installation:
| Installation Type | Location |
|---|---|
Package installations + Installer-Services installations |
|
Tarball installations + Installer-No Services installations |
|
Setting data expiration in CQL
-
Using CQL
INSERTorUPDATE, set the TTL property.For example, insert a row with a life of 60 seconds into the
health_data demo:INSERT INTO demo.health_data (id, age, gender) VALUES (9999,88,'female') USING TTL 60; -
To force the index to update with the new data:
COMMIT SEARCH INDEX ON demo.health_data ; -
In this configuration example, after 60 seconds, the row is removed from the CQL table and the search index.
Configuring expiration scope
You can configure the solrconfig.xml to include the TTL per-document or per-field on data added to the search index or the DSE database.
You construct a Solr HTTP API query to query the search index using a ttl component.
Depending on the configuration, TTL then applies to the entire document or just to a named field.
Managing expired columns
After a column is expired using the time-to-live (TTL) mechanism, DSE Search can still find the expired column. The column data remains in the index until one of the following conditions is met:
-
Reindexing occurs due to a DSE Search TTL rebuild timeout.
Set the ttl rebuild timeout properties in the
dse.yamlfile. -
All columns in a row expire due to the time-to-live (TTL) mechanism, triggering removal of the entire row from the index.
Setting the TTL timeout properties is the recommended method for managing expired columns.
