Updating the index after data expires (TTL)

Configure purge of expired and deleted data.

Time-To-Live (TTL) set on a CQL field also applies to the indexed values. The DSE Search engine searches the index for expired documents and then deletes them from the index in batches of a configurable size, as defined by the ttl_index_rebuild_options in the dse.yaml file. The default interval is 300 seconds (5 minutes).

Setting data expiration in CQL

  1. Using CQL INSERT or UPDATE, set the TTL property.
    For example, insert a row with a life of 60 seconds into the health_data demo:
    INSERT INTO demo.health_data (id, age, gender) 
                VALUES (9999,88,'female') USING TTL 60;
  2. To force the index to update with the new data:
    COMMIT SEARCH INDEX ON demo.health_data ;
  3. In this configuration example, after 60 seconds, the row is removed from the CQL table and the search index.

Configuring expiration scope

You can configure the solrconfig.xml to include the TTL per-document or per-field on data added to the search index or the DSE database. You construct a Solr HTTP API query to query the search index using a ttl component. Depending on the configuration, TTL then applies to the entire document or just to a named field.

Managing expired columns

After a column is expired using the time-to-live (TTL) mechanism, DSE Search can still find the expired column. The column data remains in the index until one of the following conditions is met:

Setting the TTL timeout properties is the recommended method for managing expired columns.