Tuning Java resources

Tuning the Java Virtual Machine (JVM) can improve performance or reduce high memory consumption.

Tuning the Java Virtual Machine (JVM) can improve performance or reduce high memory consumption.

Topics in this page:

About garbage collection

Garbage collection is the process by which Java removes data that is no longer needed from memory. To achieve the best performance, it is important to find the right heap size settings.

One situation that you definitely want to minimize is a garbage collection pause, also known as a stop-the-world event. A pause occurs when a region of memory is full and the JVM needs to make space to continue. During a pause all operations are suspended. Because a pause affects networking, the node can appear as down to other nodes in the cluster. Additionally, any Select and Insert statements will wait, which increases read and write latencies. Any pause of more than a second, or multiple pauses within a second that add to a large fraction of that second, should be avoided. The basic cause of the problem is the rate of data stored in memory outpaces the rate at which data can be removed. For specific symptoms and causes, contact DataStax Support.

Determining the heap size

You might be tempted to set the Java heap to consume the majority of the computer's RAM. However, this can interfere with the operation of the OS page cache. Recent operating systems maintain the OS page cache for frequently accessed data and are very good at keeping this data in memory. Properly tuning the OS page cache usually results in better performance than increasing the Cassandra row cache.

Cassandra automatically calculates the maximum heap size (MAX_HEAP_SIZE) based on this formula: 
max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB)
For production use, you may wish to adjust heap size for your environment using the following guidelines:
  • Heap size is usually between ¼ and ½ of system memory.
  • Do not devote all memory to heap because it is also used for offheap cache and file system cache.
  • Always enable GC logging when adjusting GC.
  • Adjust settings gradually and test each incremental change.
  • Enable parallel processing for GC.
  • Cassandra's GCInspector class logs information about any garbage collection that takes longer than 200 ms. Garbage collections that occur frequently and take a moderate length of time (seconds) to complete, indicate excessive garbage collection pressure on the JVM. In addition to adjusting the garbage collection options, other remedies include adding nodes, and lowering cache sizes.
Note: For more tuning tips, see Secret HotSpot option improving GC pauses on large heaps.

MAX_HEAP_SIZE

The recommended maximum heap size depends on which GC is used:
Hardware setup Recommended MAX_HEAP_SIZE
Older computers Typically 8 GB.
CMS for newer computers (8+ cores) with up to 256 GB RAM No more 14 GB.
The easiest way to determine the optimum heap size for your environment is:
  1. Set the maximum heap size in the cassandra-env.sh file to a high arbitrary value on a single node. For example:
    MAX_HEAP_SIZE="12G"
  2. Enable GC logging.
  3. Check the logs to view the heap used by that node and use that value for setting the heap size in the cluster:
Note: This method decreases performance for the test node, but generally does not significantly reduce cluster performance.

If you don't see improved performance, contact the DataStax Services team for additional help.

HEAP_NEWSIZE

This setting determines the amount of heap memory allocated to newer objects or young generation. Cassandra calculates the default value for this property (in MB) as the lesser of:
  • 100 times the number of cores
  • ¼ of MAX_HEAP_SIZE
As a starting point, set HEAP_NEWSIZE to 100 MB per physical CPU core. For example, for a modern 8-core+ machine: 
HEAP_NEWSIZE="800M"

A larger HEAP_NEWSIZE leads to longer GC pause times. For a smaller HEAP_NEWSIZE, GC pauses are shorter but usually more expensive.

How Cassandra uses memory

Cassandra performs the following major operations within JVM heap:
  • To perform reads, Cassandra maintains the following components in heap memory:
    • Bloom filters
    • Partition summary
    • Partition key cache
    • Compression offsets
    • SSTable index summary

    This metadata resides in memory and is proportional to total data

  • Cassandra gathers replicas for a read or for anti-entropy repair and compares the replicas in heap memory.
  • Data written to Cassandra is first stored in memtables in heap memory. Memtables are flushed to SSTables on disk.
To improve performance, Cassandra also uses off-heap memory as follows:
  • Page cache. Cassandra uses additional memory as page cache when reading files on disk.
  • The Bloom filter and compression offset maps reside off-heap.
  • Cassandra can store cached rows in native memory, outside the Java heap. This reduces JVM heap requirements, which helps keep the heap size in the sweet spot for JVM garbage collection performance.

Adjusting JVM parameters for other Cassandra services

  • Solr: Some Solr users have reported that increasing the stack size improves performance under Tomcat.
    To increase the stack size, uncomment and modify the default setting in the cassandra-env.sh file. 
    # Per-thread stack size.
JVM_OPTS="$JVM_OPTS -Xss256k"
    Also, decreasing the memtable space to make room for Solr caches can improve performance. Modify the memtable space by changing the memtable_heap_space_in_mb and memtable_offheap_space_in_mb properties in the cassandra.yaml file.

  • MapReduce: Because MapReduce runs outside the JVM, changes to the JVM do not affect Analytics/Hadoop operations directly.

Other JMX options

Cassandra exposes other statistics and management operations via Java Management Extensions (JMX). JConsole and the nodetool utility are JMX-compliant management tools.

Configure Cassandra for JMX management by editing these properties in cassandra-env.sh.

  • com.sun.management.jmxremote.port: sets the port on which Cassandra listens from JMX connections.
  • com.sun.management.jmxremote.ssl: enables or disables SSL for JMX.
  • com.sun.management.jmxremote.authenticate: enables or disables remote authentication for JMX.
  • -Djava.rmi.server.hostname: sets the interface hostname or IP that JMX should use to connect. Uncomment and set if you are having trouble connecting.
Note: By default, you can interact with Cassandra using JMX on port 7199 without authentication.
The location of the cassandra.yaml file depends on the type of installation:
Package installations /etc/cassandra/cassandra.yaml
Tarball installations install_location/resources/cassandra/conf/cassandra.yaml
The location of the cassandra-env.sh file depends on the type of installation:
Package installations /etc/cassandra/cassandra-env.sh
Tarball installations install_location/conf/cassandra-env.sh