Tuning Java resources
Tuning the Java Virtual Machine (JVM) can improve performance or reduce high memory consumption.
These options are set in several files:
The location of the cassandra.yaml file depends on the type of installation:
-
Package installations and Installer-Services installations:
/etc/dse/cassandra/cassandra.yaml -
Tarball installations and Installer-No Services installations:
<installation_location>/resources/cassandra/conf/cassandra.yaml
The location of the cassandra-env.sh file depends on the type of installation:
-
Package installations and installer-services installations:
/etc/dse/cassandra/cassandra-env.sh -
Tarball installations and installer-no services installations:
INSTALLATION_LOCATION/resources/cassandra/conf/cassandra-env.sh
The location of the jvm.options file depends on the type of installation:
-
Package installations and Installer-Services installations:
/etc/dse/cassandra/jvm.options -
Tarball installations and Installer-No Services installations:
<installation_location>/resources/cassandra/conf/jvm.options
About garbage collection
Garbage collection (GC) is the process by which Java removes data that is no longer needed from memory. To achieve the best performance, it is important to select the right garbage collector and heap size settings.
One situation that you definitely want to minimize is a garbage collection pause, also known as a stop-the-world event.
A pause occurs when a region of memory is full and the JVM needs to make space to continue.
During a pause all operations are suspended.
Because a pause affects networking, the node can appear as down to other nodes in the cluster.
Additionally, any SELECT and INSERT statements wait, which increases read and write latencies.
Avoid a pause of more than a second, or multiple pauses within a second that add to a large fraction of that second.
The basic cause of the problem is the rate of data stored in memory outpaces the rate at which data can be removed.
For specific symptoms and causes, see Garbage collection pauses.
|
DataStax recommends that you enable parallel processing for GC, particularly when using DSE Search. |
Choosing a Java garbage collector
DataStax Enterprise (DSE) 5.1 uses the garbage first collector (G1) by default. G1 is recommended for the following reasons:
-
Supports heap sizes from 16 GB to 64 GB.
G1 performs better than concurrent-mark-sweep (CMS) for larger heaps because it scans the regions of the heap containing the most garbage objects first, and compacts the heap on-the-go, while CMS stops the application when performing garbage collection.
With G1, it is safe to set the heap as large as possible, with respect for other system resources.
-
The workload is variable, that is, the cluster is performing the different processes all the time.
-
G1 is easier to configure.
-
G1 is self tuning. You only need to set
MAX_HEAP_SIZE.
However, G1 incurs some latency due to profiling.
Setting CMS as the Java garbage collector
CMS is deprecated in Java 9.
CMS is recommended only in the following circumstances:
-
You have the time and expertise to manually tune and test garbage collection. Be aware that allocating more memory to the heap can result in diminishing performance as the garbage collection facility increases the amount of database metadata in heap memory.
-
Heap sizes are smaller than 16 GB.
-
The workload is fixed, that is, the cluster performs the same processes all the time.
-
The environment requires the lowest latency possible.
To use CMS, see Using the Continuous Mark Sweep (CMS) garbage collector.
Ideal heap size
If the heap size is too large, it can interfere with the OS page cache. Instead, properly tuning the OS page cache usually results in better performance than increasing the row cache.
By default, DSE automatically calculates heap size. Depending on your environment, manually setting heap can improve performance. If manually set, it is important to use an appropriate value to avoid causing new performance issues. For more information, see Changing heap size parameters and Capacity planning: Memory and heap.
MAX_HEAP_SIZE
The recommended maximum heap size depends on which GC is used:
| Hardware setup | Recommended MAX_HEAP_SIZE |
|---|---|
G1 for newer computers (8+ cores) with up to 256 GB RAM |
16 GB to 32765 MB. See Java performance tuning. |
CMS for newer computers (8+ cores) with up to 256 GB RAM |
No more 16 GB. |
Older computers |
Typically 8 GB. |
The easiest way to determine the optimum heap size for your environment is:
-
Set the maximum heap size in the
jvm.optionsfile to a high arbitrary value on a single node. For example, when using G1:-Xms48G -Xmx48GSet the min (
-Xms) and max (-Xmx) heap sizes to the same value to avoid stop-the-world GC pauses during resize, and to lock the heap in memory on startup which prevents any of it from being swapped out. -
Enable GC logging.
-
Check the logs to view the heap used by that node and use that value for setting the heap size in the cluster:
|
This method decreases performance for the test node, but generally does not significantly reduce cluster performance. |
If you don’t see improved performance, contact IBM Support for additional help.
HEAP_NEWSIZE
For CMS, you may also need to adjust HEAP_NEWSIZE.
This setting determines the amount of heap memory allocated to newer objects or young generation.
The database calculates the default value for this property in megabytes (MB) as the lesser of:
-
100 times the number of cores
-
¼ of
MAX_HEAP_SIZE
As a starting point, set HEAP_NEWSIZE to 100 MB per physical CPU core.
For example, for a modern 8-core+ machine:
-Xmn800MA larger HEAP_NEWSIZE leads to longer GC pause times.
For a smaller HEAP_NEWSIZE, GC pauses are shorter but usually more expensive.
How DSE uses memory
The database performs the following major operations within JVM heap:
-
To perform reads, the database maintains the following components in heap memory:
-
Bloom filters
-
Partition summary
-
Partition key cache
-
Compression offsets
-
SSTable index summary
-
This metadata resides in memory and is proportional to total data. Some of the components grow proportionally to the size of total memory.
-
The database gathers replicas for a read or for anti-entropy repair and compares the replicas in heap memory.
-
Data written to the database is first stored in memtables in heap memory. Memtables are flushed to SSTables on disk.
To improve performance, the database also uses off-heap memory as follows:
-
Page cache. The database uses additional memory as page cache when reading files on disk.
-
The Bloom filter and compression offset maps reside off-heap.
-
The database can store cached rows in native memory, outside the Java heap. This reduces JVM heap requirements, which helps balance heap size for JVM garbage collection performance.
Adjusting JVM parameters for other DataStax Enterprise services
-
DSE Search: Some users have reported that increasing the stack size improves performance under Apache Tomcat®.
To increase the stack size, uncomment and modify the default setting in the
cassandra-env.shfile.# Per-thread stack size. JVM_OPTS="$JVM_OPTS -Xss256k"Also, decreasing the memtable space to make room for search caches can improve performance. Modify the memtable space by changing the
memtable_heap_space_in_mbandmemtable_offheap_space_in_mbproperties in thecassandra.yamlfile. -
MapReduce: BecauseMapReduceruns outside the JVM, changes to the JVM do not affect Analytics/Hadoop operations directly.
Other JMX options
DataStax Enterprise (DSE) exposes metrics and management operations through Java Management Extensions (JMX). For more information, see Monitor DataStax Enterprise (DSE) clusters.
The following system properties can be set in cassandra-env.sh to configure JMX:
-
com.sun.management.jmxremote.port: sets the port on which the database listens from JMX connections. -
com.sun.management.jmxremote.ssl: enables or disables SSL for JMX. -
com.sun.management.jmxremote.authenticate: enables or disables remote authentication for JMX. -
-Djava.rmi.server.hostname: sets the interface hostname or IP that JMX should use to connect. Uncomment and set if you are having trouble connecting.
|
By default, you can interact with DSE using JMX on port 7199 without authentication. |
- Configuring the garbage collector
-
Select and configure a garbage collector (GC) to remove data from memory that is no longer in use.
