Configure data caches

DataStax Enterprise (DSE) includes integrated caching for table data that improves read and write performance by storing frequently accessed data in memory. The database doesn’t need to perform a full read from disk when the requested data is already in the cache.

Data caches primarily improve read performance. They only improve write performance for operations that require a read-before-write, such as counter updates.

Use caching in conjunction with other techniques for improving read and write performance, such as compaction, compression, optimized query patterns, and denormalized tables.

Caching isn’t recommended for tables with rarely accessed data or very wide partitions. For example, archive tables aren’t read often enough to benefit from caching.

You must restart DSE to apply cache configuration changes.

Cache durability

The database distributes cache data across replicas. When a node goes down, cached data can be read from another replica.

This architecture also simplifies troubleshooting because there is no separate caching tier, and cached data matches what is in the database exactly (at the time that the cache was populated).

The integrated cache can alleviate cold starts by saving certain types of cache keys to disk periodically. After a restart, the database repopulates the cache from the saved keys. This is configurable and can be disabled if you prefer to start with an empty cache or your applications run preemptive queries to warm up the cache.

Cache keys are saved to the location specified by saved_caches_directory in cassandra.yaml.

The key cache file name format includes the keyspace name, table name, a table UUID, and a cache UUID:

KEYSPACE_NAME-TABLE_NAME-TABLE_UUID-KeyCache-b.dbCACHE_UUID.tmp

OS page cache

The OS page cache is a cache maintained by the operating system. In DSE 5.1, properly tuning the OS page cache is recommended over row caching, if possible. If you need to tune the page cache, see the documentation for the node’s OS.

Partition key cache

The partition key cache is enabled by default. It stores a partition index cache in off-heap memory. For more information about how this cache is used, see Read operations.

Enable either the partition key cache or the row cache for a table. Don’t enable both on the same table.

To disable the partition key cache, set the keys table property to NONE with CREATE TABLE or ALTER TABLE. The default if unspecified is ALL (enabled).

CREATE TABLE users (
  userid text PRIMARY KEY,
  first_name text,
  last_name text,
)
WITH caching = { 'keys' : 'NONE' };

In cassandra.yaml, you can set global partition key cache settings, including the maximum size of the key cache in memory for all tables (key_cache_size_in_mb), and how often the database saves partition key cache keys to disk (key_cache_save_period).

cassandra.yaml includes parameters prefixed with key_cache_ for both the partition key cache and authentication key cache. Make sure you edit the correct parameters.

Row cache

The row cache is recommended only for read-heavy workloads (95 percent of requests are reads) because writes invalidate the cache. However, DataStax recommends tuning the OS page cache instead of using the row cache.

To enable the row cache, you must set global row cache settings in cassandra.yaml and enable row caching in the table definition.

In cassandra.yaml you can set global (node-wide) row cache settings. The database uses these settings to automatically distribute memory for each table on the node based on the overall workload and specific table usage.

row_cache_size_in_mb: Configure the maximum size of the rows cache in memory for all tables. Must be greater than 0 to enable the row cache. To determine an appropriate row cache size, you must run performance tests to benchmark your workloads.
row_cache_class_name: The OHCProvider class is recommended. Use SerializingCacheProvider for backwards compatibility.
row_cache_save_period and row_cache_keys_to_save: Configure how often row cache keys are saved to disk for cache durability and warm start.

Use CREATE TABLE or ALTER TABLE to set the rows_per_partition table property. Possible values are ALL, NONE (default), or an integer.

CREATE TABLE users (
  userid text PRIMARY KEY,
  first_name text,
  last_name text,
)
WITH caching = { 'keys' : 'NONE', 'rows_per_partition' : '120' };

When using the row cache, keys must be set to NONE to disable the partition key cache. Don’t use both caches on the same table.

Counter cache

The counter cache is exclusively for writes to counter columns. This cache stores recently updated counters in memory, reducing the need for a read-before-write when updating the same counter again. If the replication factor is 1, a counter cache hit causes the database to skip read-before-write entirely before updating a counter. If the replication factor is greater than 1, a counter cache hit can reduce the duration of the lock hold (most beneficial for frequently updated counters), but it doesn’t completely skip read-before-write because the database still needs to ensure consistency across replicas.

The counter cache doesn’t require as much memory as other caches because only the local (clock, count) tuple of a counter cell is kept in memory, not the whole counter.

In cassandra.yaml you can set global (node-wide) counter cache settings. The database uses these settings to automatically distribute memory for each table on the node based on the overall workload and specific table usage.

counter_cache_size_in_mb: Configure the maximum size of the counter cache in memory for all tables. By default, this is automatically calculated as the smaller of 2.5 percent of heap or 50 MB.

Set to 0 to disable the counter cache.

The counter cache is enabled by default and recommended for almost all counter workloads. The only exception are workloads with counter deletes that rely on a low tombstone grace period.
counter_cache_save_period and counter_cache_keys_to_save: Configure how often counter cache keys are saved to disk for cache durability and warm start. By default, all keys are saved after two hours.

Monitor caching

You can monitor caching at the node level and the query level.

`nodetool info`

Check cache usage and monitor cache performance with nodetool info. If cache is full and the hit ratio is below 90%, you might need to tune the cache configuration.

Run this command after the node is warm and receiving traffic. After a restart, when the cache is cold, the output isn’t useful. For example:

ID               : 387d15ba-7103-491b-9327-1a691dbb504a
Gossip active    : true
Thrift active    : true
Native Transport active: true
Load             : 65.87 KB
Generation No    : 1400189757
Uptime (seconds) : 148760
Heap Memory (MB) : 392.82 / 1996.81
datacenter      : datacenter1
Rack             : rack1
Exceptions       : 0
Key Cache        : entries 10, size 728 (bytes), capacity 103809024 (bytes), 93 hits, 102 requests, 0.912 recent hit rate, 14400 save period in seconds
Row Cache        : entries 0, size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache    : entries 0, size 0 (bytes), capacity 51380224 (bytes), 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token            : -9223372036854775808

Query tracing

You can use query tracing in CQL shell and Cassandra drivers to determine if the requested data was read from the cache or from disk. The following examples use the row cache.

Tracing outputs a cache miss if the row is not cached:

Row cache miss [ReadStage:41]

Tracing outputs a cache hit if the row was read from the cache:

Row cache hit [ReadStage:55]

Tracing outputs Ignoring row cache if the query couldn’t be fulfilled from the cache, requiring a read from disk. This can happen if the query required cached and uncached rows, exceeded the global limit, or requested rows that weren’t at the beginning of the partition.

Ignoring row cache as cached value could not satisfy query [ReadStage:89]

If your query could have skipped rows from the beginning of a partition, remove the constraints that might cause it to do so, and then retry the query.

If your query is too broad, try narrowing the query or limiting returned rows.

If the cache is frequently missed or ignored, you might need to increase the cache size or restructure the table to position frequently accessed rows at the start of partitions.