Tuning Java resources

The DataStax Distribution of Apache Cassandra™ (DDAC) uses the garbage first collector (G1) by default. G1 is recommended for the following reasons:

• Heap sizes from 16 GB to 64 GB.

  G1 performs better than CMS (concurrent-mark-sweep) for larger heaps because it scans the regions of the heap containing the most garbage objects first, and compacts the heap on-the-go, while CMS stops the application when performing garbage collection.

• The workload is variable, that is, the cluster is performing the different processes all the time.
• For future proofing, as CMS will be deprecated in Java 9.
• G1 is easier to configure.
• G1 is self tuning.
• You only need to set MAX_HEAP_SIZE.

However, G1 incurs some latency due to profiling.

CMS is recommended only in the following circumstances:

• You have the time and expertise to manually tune and test garbage collection.

  Be aware that allocating more memory to the heap can result in diminishing performance as the garbage collection facility increases the amount of database metadata in heap memory.

• Heap sizes are smaller than 16 GB.
• The workload is fixed, that is, the cluster performs the same processes all the time.
• The environment requires the lowest latency possible.

The database automatically calculates the maximum heap size (MAX_HEAP_SIZE) based on this formula:

max(min(1/2 ram, 1024 megabytes), min(1/4 ram, 32765 megabytes))

For production use, use these guidelines to adjust heap size for your environment:

• Heap size is usually between ¼ and ½ of system memory.
• Do not devote all memory to heap because it is also used for offheap cache and file system cache.
• Always enable GC logging when adjusting GC.
• Adjust settings gradually and test each incremental change.
• Enable parallel processing for GC, particularly when using DSE Search.
• The GCInspector class logs information about any garbage collection that takes longer than 200 ms. Garbage collections that occur frequently and take a moderate length of time (seconds) to complete, indicate excessive garbage collection pressure on the JVM. In addition to adjusting the garbage collection options, other remedies include adding nodes, and lowering cache sizes.
• For a node using G1, DataStax recommends a MAX_HEAP_SIZE as large as possible, up to 64 GB.

The recommended maximum heap size depends on which GC is used:

Hardware setup	Recommended MAX_HEAP_SIZE
G1 for newer computers (8+ cores) with up to 256 GB RAM	16 GB to 32765 MB. See Java performance tuning.
CMS for newer computers (8+ cores) with up to 256 GB RAM	No more 16 GB.
Older computers	Typically 8 GB.

The easiest way to determine the optimum heap size for your environment is:

1. Set the maximum heap size in the jvm.options file to a high arbitrary value on a single node. For example, when using G1:

   -Xms48G
   -Xmx48G

   Set the min (-Xms) and max (-Xmx) heap sizes to the same value to avoid stop-the-world GC pauses during resize, and to lock the heap in memory on startup which prevents any of it from being swapped out.

2. Enable GC logging.
3. Check the logs to view the heap used by that node and use that value for setting the heap size in the cluster:

For CMS, you may also need to adjust HEAP_NEWSIZE. This setting determines the amount of heap memory allocated to newer objects or young generation. The database calculates the default value for this property in megabytes (MB) as the lesser of:

• 100 times the number of cores
• ¼ of MAX_HEAP_SIZE

As a starting point, set HEAP_NEWSIZE to 100 MB per physical CPU core. For example, for a modern 8-core+ machine:

-Xmn800M

The database performs the following major operations within JVM heap:

• To perform reads, the database maintains the following components in heap memory:
  • Bloom filters
  • Partition summary
  • Partition key cache
  • Compression offsets
  • SSTable index summary

  This metadata resides in memory and is proportional to total data. Some of the components grow proportionally to the size of total memory.

• The database gathers replicas for a read or for anti-entropy repair and compares the replicas in heap memory.
• Data written to the database is first stored in memtables in heap memory. Memtables are flushed to SSTables on disk.

To improve performance, the database also uses off-heap memory as follows:

• Page cache. The database uses additional memory as page cache when reading files on disk.
• The Bloom filter and compression offset maps reside off-heap.
• The database can store cached rows in native memory, outside the Java heap. This reduces JVM heap requirements, which helps keep the heap size in the sweet spot for JVM garbage collection performance.