Recommended production settings

DataStax recommends the following settings for DataStax Enterprise (DSE) deployments in production environments.

These are general recommendations for optimal performance in common use cases. You might need to adjust these settings according to your environment and workloads.

It is important that these settings persist through restart.

Persistence depends on your environment. Check with your system administrator to ensure these settings are viable for your environment.

Chunk cache

The amount of native memory used by the DSE process is significant. This is primarily due to the chunk cache, also known as the file cache.

The chunk cache was originally designed to cache small parts (chunks) of SSTable files to make read operations faster. In earlier versions of Apache Cassandra, the chunk cache had a secondary role, and it was limited to 512 MB. In DSE 6.8, the chunk cache is a central component of the asynchronous thread-per-core (TPC) architecture, and it fully replaces the OS page cache for database read operations.

The chunk cache is similar to an OS page cache, but they have significantly different impacts on system memory:

The OS page cache is sized dynamically by the operating system, and it scales automatically based on available server memory. In contrast, the chunk cache is sized statically.

If the chunk cache is too small, the system cannot make full use of available server memory. For servers with large amounts of memory (50 GB or more), this is wasted memory. However, if the chunk cache is too large, it can consume too much of the available memory, prompting the OS to kill the DSE process to avoid out of memory errors.
The OS page cache is cleared only after a server restart, not after restarting a process. In contrast, the chunk cache is cleared every time the DSE process restarts.

When a cache is cleared, all previously cached file data is removed, and subsequent read operations must repopulate the cache. This can slow performance temporarily until the cache is repopulated.
The OS treats OS page cache memory as buffer memory. In contrast, the OS treats chunk cache memory as user memory.

This is because the memory used by the file cache is part of the DSE process memory, and, therefore, attributed to the user running the DSE process.
The chunk cache uses mostly NIO direct memory by storing file chunks into NIO byte buffers. However, NIO consumes on-heap memory.

Chunk cache configuration

You must restart the DSE process to apply chunk cache configuration changes.

By default, the chunk cache size is half of max direct memory, and chunk cache operations use the following percentages of max direct memory:

50 percent of the max direct memory for the DSE process
25 percent of the max direct memory for tools

By default, max direct memory is calculated as 50 percent of system memory minus JVM heap size:

Max direct memory = (system memory - JVM heap size) / 2

To explicitly configure the max direct memory size, set the JVM parameter -XX:MaxDirectMemorySize.

To explicitly configure the chunk cache size, and override auto-sizing based on max direct memory, set the file_cache_size_in_mb parameter in cassandra.yaml. For example, if the cache hit rate is too low and there is still available memory on the server, you can explicitly set the file cache size parameter to as much as 90 percent of MaxDirectMemorySize (depending on your workloads).

Recommended chunk cache memory

The ideal memory for the chunk cache depends on your workloads. DataStax recommends the following when configuring max direct memory and chunk cache size for DSE nodes:

Total server memory size
Adequate memory for the OS and other applications
Adequate memory for the Java heap size
Adequate memory for native raw memory (such as bloom filters and off-heap memtables)

For 64 GB servers, the default settings are typically adequate. Larger servers might benefit from increased -XX:MaxDirectMemorySize as long as 15 to 20 percent of memory is available for the OS and other in-memory structures.

For DSE advanced workloads, consider the following additional recommendations:

DSE Search nodes: Enable Asynchronous I/O (AIO) and use the default value for file_cache_size_in_mb.
DSE Analytics nodes: Memory is critical for DSE Analytics performance. Because Apache Spark™ effectively manages its own memory through the Spark application settings, you must determine how much memory the Spark application receives. Then, you can decide how much memory to allocate to the chunk cache and Spark applications.
DSE Graph nodes: Because DSE Graph relies on multiple workload types, consider all previous recommendations, depending on the node’s combination of transactional and advanced workloads.

Install the Java Virtual Machine (JVM)

DSE 6.8 is built to run on Java 8 primarily. DSE 6.8.22 and later can use Java 11 for core (transactional) functionality only. Advanced workloads (such as DSE Search, Analytics, and Graph) require Java 8 in all DSE 6.8 patch releases.

Your deployment must use a compatible Java runtime, such as the following:

Java version Java runtime Minimum build version Notes

Java version	Java runtime	Minimum build version	Notes
Java 8	OpenJDK 8	`1.8.0_242`	Recommended.
Java 8	Oracle Java 8 JRE/JDK	`1.8.0_242`	Supported but DataStax does more extensive testing on OpenJDK 8 due to the end of public updates for the Oracle Java 8 JRE/JDK.
Java 11	OpenJDK 11	`11.0.19`	Recommended runtime if using Java 11 for core functionality in DSE 6.8.22 or later. If using advanced workloads, a Java 8 runtime is also required.
Java 11	Oracle Java SE 11 JDK	`11.0.18`	Supported runtime if using Java 11 for core functionality in DSE 6.8.22 or later. If using advanced workloads, a Java 8 runtime is also required.

Java 8

OpenJDK 8

1.8.0_242

Recommended.

Java 8

Oracle Java 8 JRE/JDK

1.8.0_242

Supported but DataStax does more extensive testing on OpenJDK 8 due to the end of public updates for the Oracle Java 8 JRE/JDK.

Java 11

OpenJDK 11

11.0.19

Recommended runtime if using Java 11 for core functionality in DSE 6.8.22 or later. If using advanced workloads, a Java 8 runtime is also required.

Java 11

Oracle Java SE 11 JDK

11.0.18

Supported runtime if using Java 11 for core functionality in DSE 6.8.22 or later. If using advanced workloads, a Java 8 runtime is also required.

If you install multiple Java versions, you must set $JAVA_HOME to the latest DSE-supported version that you have installed (Java 11 or Java 8). If you install both Java 11 and Java 8, you must set $JAVA_HOME to Java 11.

Check your system’s Java version to verify that a supported Java runtime is installed:

java -version

Make sure the output shows a supported JDK. In the following example, OpenJDK 8 is installed:

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b09)
OpenJDK 64-Bit Server VM (build 25.242-b09, mixed mode)

If the installed version isn’t a supported version, install or upgrade your Java runtime. The following steps summarize the process to install OpenJDK 8 or Oracle JDK 8 on RHEL and Debian platforms. For more detailed installation guides for your specific build and platform, see the Oracle or OpenJDK documentation.

Install OpenJDK 8 on RHEL

Update the repositories:
```
sudo yum update
```
Install OpenJDK:
```
sudo yum install java-1.8.0-openjdk
```
Check the version:
```
java -version
```
Make sure the result indicates that OpenJDK 8 is being used, such as openjdk version "1.8.0_242".

If the correct version of the JDK is not being used, use the alternatives command to switch it, and then check the version again:
```
sudo alternatives --config java
```
After installing the JDK, you might need to set JAVA_HOME to your profile:
- For shell or bash: export JAVA_HOME=path_to_java_home
- For csh (C shell): setenv JAVA_HOME=path_to_java_home

Install OpenJDK 8 on Debian

Update the repositories:
```
sudo apt-get update
```
Install OpenJDK:
```
sudo apt-get install openjdk-8-jdk
```
Check the version:
```
java -version
```
Make sure the result indicates that OpenJDK 8 is being used, such as openjdk version "1.8.0_242".

If the correct version of the JDK is not being used, use the update-alternatives command to switch it, and then check the version again:
```
sudo update-alternatives --set java /usr/lib/jvm/jdk1.8.0_version/bin/java
```
After installing the JDK, you might need to set JAVA_HOME to your profile:
- For shell or bash: export JAVA_HOME=path_to_java_home
- For csh (C shell): setenv JAVA_HOME=path_to_java_home

Install Oracle JDK on RHEL

If necessary, go to Oracle Java SE Downloads, accept the license agreement, and download the installer for your distribution. The Oracle JDK License has changed for releases starting April 16, 2019. See Java SE Development Kit 8 Downloads.

If installing the Oracle JDK in a cloud environment, download the installer to your local client, and then use scp (secure copy) to transfer the file to your cloud machines.

Update the repositories:
```
sudo yum update
```
From the directory where you downloaded the package, run the installer:
```
sudo rpm -ivh jdk-8uversion-linux-x64.rpm
```
The RPM installs the JDK into the /usr/java/ directory.
Use the alternatives command to add a symbolic link to the Oracle JDK installation so that your system uses the Oracle JDK instead of the OpenJDK:
```
sudo alternatives --install /usr/bin/java java /usr/java/jdk1.8.0_version/bin/java 200000
```
If you have any problems, set the PATH and JAVA_HOME variables:
```
export PATH="$PATH:/usr/java/latest/bin"
set JAVA_HOME=/usr/java/latest
```
Verify the version of the JRE or JDK:
```
java -version
```
Make sure the result indicates that Oracle Java 8 is being used:
```
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)
```
If the OpenJDK is still being used, use the alternatives command to switch it, and then check the version again:
```
sudo alternatives --config java
```
After installing the JDK, you might need to set JAVA_HOME to your profile:
- For shell or bash: export JAVA_HOME=path_to_java_home
- For csh (C shell): setenv JAVA_HOME=path_to_java_home

Install Oracle JDK on Debian

If installing the Oracle JDK in a cloud environment, download the installer to your local client, and then use scp (secure copy) to transfer the file to your cloud machines.

Make a directory for the JDK:
```
sudo mkdir -p /usr/lib/jvm
```
Extract the tarball and install the JDK:
```
sudo tar zxvf jdk-version-linux-x64.tar.gz -C /usr/lib/jvm
```
The JDK files are installed into a directory named /usr/lib/jvm/jdk-8u_VERSION.
Tell the system that there is a new Java version available:
```
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_version/bin/java" 1
```
If updating from a previous version that was removed manually, you must execute the above command twice because you’ll get an error message the first time.

Set the new JDK as the default:

sudo update-alternatives --set java /usr/lib/jvm/jdk1.8.0_version/bin/java

Verify the version of the JRE or JDK:

java -version

Make sure the result indicates that Oracle Java 8 is being used:

java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

After installing the JDK, you might need to set JAVA_HOME to your profile:
- For shell or bash: export JAVA_HOME=path_to_java_home
- For csh (C shell): setenv JAVA_HOME=path_to_java_home

Synchronize clocks

To prevent clock-drift, use Network Time Protocol (NTP) to synchronize the clocks on all nodes and all application servers that interact with the cluster, such as Cassandra drivers.

Synchronizing clocks is required because DSE overwrites a column only if there is another version whose timestamp is more recent, which can happen when machines are in different locations.

DSE timestamps are encoded as microseconds because UNIX Epoch time does not include timezone information. The timestamp for all writes in DSE is Universal Time Coordinated (UTC). DataStax recommends converting to local time only when generating human-readable output.

If available, set system clocks to TSC:

cat /sys/devices/system/clocksource/clocksource0/available_clocksource
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
echo tsc | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Install NTP for your operating system:
Debian
sudo apt-get install ntpdate
RHEL
sudo yum install ntpdate
On RHEL 7 and later, chrony is the default network time protocol daemon. The chrony configuration file is located at /etc/chrony.conf.
Start the NTP service on all nodes:
```
sudo service ntp start -x
```
Run the ntpdate command to synchronize clocks:
```
sudo ntpdate 1.ro.pool.ntp.org
```
Verify that your NTP configuration is working:
```
ntpstat
```

Set kernel parameters

Configure kernel parameters for TCP settings and user resource limits.

You can run sysctl -a to get your current Linux kernel settings:

sudo sysctl -a

TCP settings

You must configure network kernel parameters to support thousands of concurrent connections, and avoid closing connections to nodes unexpectedly.

Configure TCP keepalive timeout parameters to prevent unexpected node connection closure during low traffic periods.

During low traffic intervals, a firewall configured with an idle connection timeout can close connections to local nodes and nodes in other datacenters. To prevent connections between nodes from timing out, set the following parameters:

Keepalive parameters
Parameter	Description	Recommended value
`net.ipv4.tcp_keepalive_time`	Duration of inactivity before a keepalive request is sent to the connection	`60`
`net.ipv4.tcp_keepalive_probes`	Number of keepalive requests to attempt before closing the connection	`3`
`net.ipv4.tcp_keepalive_intvl`	Interval between keepalive requests after the first	`10`

sudo sysctl -w \
net.ipv4.tcp_keepalive_time=60 \
net.ipv4.tcp_keepalive_probes=3 \
net.ipv4.tcp_keepalive_intvl=10

These values set the TCP keepalive timeout to 60 seconds with 3 probes and a 10-second gap between each probe. This configuration detects dead TCP connections after 90 seconds: 60 + (3 x 10) = 90. Additional traffic is negligible, and it is safe to persist these TCP keepalive timeout settings.

As an alternative to system TCP settings, you can prevent reset connections during streaming by tuning the streaming_keep_alive_period_in_secs parameter in cassandra.yaml.

Change the following settings to handle thousands of concurrent connections used by the database:

sudo sysctl -w \
net.core.rmem_max=16777216 \
net.core.wmem_max=16777216 \
net.core.rmem_default=16777216 \
net.core.wmem_default=16777216 \
net.core.optmem_max=40960 \
net.ipv4.tcp_rmem='4096 87380 16777216' \
net.ipv4.tcp_wmem='4096 65536 16777216'

To persist the configuration, add the modified network kernel parameters to the /etc/sysctl.conf file:

net.ipv4.tcp_keepalive_time=60
net.ipv4.tcp_keepalive_probes=3
net.ipv4.tcp_keepalive_intvl=10
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.core.optmem_max=40960
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216

To apply the changes, run one of the following commands:

sudo sysctl -p /etc/sysctl.conf

sudo sysctl -p /etc/sysctl.d/*.conf

Set user resource limits

Use the ulimit -a command to view the current limits.

Although limits can also be temporarily set with ulimit command, DataStax recommends setting user resource limits permanent in /etc/security/limits.conf or a related file.

When you install DSE, it creates a cassandra superuser role in the database, and DSE runs as this user. Don’t use the default cassandra role in production because it is a security risk. Instead, create a new superuser role for running DSE.

Make the following changes for your specific platform, as well as the changes for all platforms, and then apply the changes to the node:

All platforms: add the following line to /etc/sysctl.conf:
```
vm.max_map_count = 1048575
```
All platforms: In the DSE security limits configuration file, configure limits for the system user that runs the DSE (cassandra) process.

These options configure the maximum number of open files, the maximum number of processes, and the memory lock limit:
- memlock: Set maximum locked-in-memory address space to unlimited.
- nofile: Allow at least 100,000 open files.
- nproc: Allow at least 32,000 processes.
- as: Set the address space to unlimited.
Don’t use the default cassandra user that is automatically created when you install DSE.
Package installations

Set the following limits in /etc/security/limits.d/cassandra.conf:

DATABASE_USER - memlock unlimited DATABASE_USER - nofile 1048576 DATABASE_USER - nproc 32768 DATABASE_USER - as unlimited

Tarball installations (standard)

For standard tarball installations where DSE runs as a standalone process, set the following limits in /etc/security/limits.conf:

DATABASE_USER - memlock unlimited DATABASE_USER - nofile 1048576 DATABASE_USER - nproc 32768 DATABASE_USER - as unlimited

Tarball installations (service)

For non-standard tarball installations where DSE is configured to start and stop as a service, specify the following limits in the relevant .service file. This file is named after the service and located at /etc/systemd/system/SERVICE_FILE_NAME.service.

[Service] LimitMEMLOCK=unlimited LimitNOFILE=1048576 LimitNPROC=32768 LimitAS=unlimited

RHEL-based systems only: Set the nproc limits to 32768 in the /etc/security/limits.d/90-nproc.conf configuration file:

DATABASE_USER - nproc 32768

Replace DATABASE_USER with the name of your system user that runs the DSE (cassandra) process. Don’t use the default cassandra user that is automatically created when you install DSE.

Be aware of the CGroup version your RHEL version uses and how it can affect resource limits.

By default, RHEL 9 uses Control Group (CGroup) V2, and previous versions use CGroup V1. This change can impact how settings like nproc and vm.max_map_count are limited and interpreted at system, slice, and user levels. For example, in RHEL 9, 'systemd' limits override traditional Privileged Access Management (PAM) limits in /etc/security/limits.d.

When upgrading to RHEL 9 from an earlier RHEL version, see Migrating from CGroups V1 to CGroups V2 in Red Hat Enterprise Linux.

Debian-based systems only: In the /etc/pam.d/su file, uncomment the following line to enable the pam_limits.so module.

This ensures that the system reads the files in the /etc/security/limits.d directory.
```
session    required   pam_limits.so
```
Debian-based systems only: If you run DSE as root, some Linux distributions (such as Ubuntu) require setting the following limits for the root user explicitly:
```
root - memlock unlimited
root - nofile 1048576
root - nproc 32768
root - as unlimited
```
All platforms: Apply changes to user resource limits by rebooting the server or running the following command:
```
sudo sysctl -p
```
All platforms: To confirm the user limits were applied to the DSE process, run the following command:
```
cat /proc/PID/limits
```
Replace PID with the process ID of the currently running DSE process.

Disable settings that throttle performance

Unless you have a reason to enable the following settings, DataStax recommends that you disable these settings because they can cause performance issues.

Disable CPU frequency scaling

Recent Linux systems include a feature called CPU frequency scaling or CPU speed scaling. This feature allows a server’s clock speed to be dynamically adjusted so that the server can run at lower clock speeds when the demand or load is low. This change reduces the server’s power consumption and heat output, which significantly impacts cooling costs. Unfortunately, this behavior has a detrimental effect on servers running DSE because throughput can be capped at a lower rate.

On most Linux systems, a CPUfreq governor manages the scaling of frequencies based on defined rules. The default ondemand governor switches to the maximum clock frequency when demand is high, and it switches to the lowest frequency when the system is idle.

In contrast, the performance governor doesn’t switch frequencies, which means that power savings are bypassed to always run at maximum throughput. This governor is recommended for DSE.

Don’t use governors that lower the CPU frequency. To ensure optimal performance, reconfigure all CPUs to use the performance governor, which locks the frequency at maximum.

On most systems, you can run the following command to set the performance governor:

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
do
    [ -f $CPUFREQ ] || continue
    echo -n performance > $CPUFREQ
done

If this directory doesn’t exist on your system, see the following:

Debian: cpufreq-set
RHEL: CPUfreq

Disable `zone_reclaim_mode` on NUMA systems

The Linux kernel can be inconsistent in enabling and disabling zone_reclaim_mode, which can cause performance problems.

To ensure that zone_reclaim_mode is disabled, run the following command:

echo 0 > /proc/sys/vm/zone_reclaim_mode

Disable swap

Severe performance degradation can occur if swap isn’t disabled completely.

Because the database has multiple replicas and transparent failover, it is preferable for a replica to be killed immediately, rather than go into swap, when memory is low. This allows traffic to be immediately redirected to a functioning replica instead of continuing to hit the replica that has high latency due to swapping.

Even with lots of DRAM, swapping still lowers performance significantly because the OS swaps out executable code to free up DRAM for caching disks.

To disable swap completely, do the following:

Run the following command:
```
sudo swapoff --all
```
To make the change permanent, remove all swap file entries from /etc/fstab.

If your organizational policies or unresolvable infrastructure limitations require swap, set vm.swappiness=1. This allows the kernel to swap out the least used memory.

Optimize disk settings

The default disk configurations on most Linux distributions are not optimal. Use these guidelines to optimize settings for your SSDs or spinning disks.

For information about disk choices for your deployment, see Capacity planning: Storage subsystem.

Based on your cluster’s hardware, only apply optimization settings for SSDs or spinning disks. Don’t apply both configurations.

Optimize SSDs

If your deployment uses SSDs, apply the following optimizations:

Ensure that the SysFS rotational flag is set to false (zero).

This overrides any detection by the operating system to ensure the drive is considered an SSD.
Apply the same rotational flag setting for any block devices created from SSD storage, such as mdarrays.
Run lsblk to get your current block devices:
```
lsblk
```
In the next steps, you must specify the name of each device that you want to configure. In the following example, the device names are sda1 and sda2:
```
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  32G  0 disk
|
|-sda1 253:1    0   8M  0 part
|-sda2 253:2    0  32G  0 part /
```
For each listed device, set the I/O scheduler to either mq-deadline or none:
- mq-deadline: This scheduler optimizes requests to minimize I/O latency. If you don’t know which scheduler to use, mq-deadline is a safe choice for all devices.
  echo mq-deadline > /sys/block/DEVICE_NAME/queue/scheduler
- none: Use this scheduler for block devices that are an array of SSDs behind a high-end I/O controller that performs I/O optimization.
  echo none > /sys/block/DEVICE_NAME/queue/scheduler
For each device, set the nr_requests value to the maximum number of read and write requests that can be queued.

The recommended queue size setting for large machines is 128, and the recommended setting for small machines is 32.
Temporary setting
This setting is temporary and resets after a reboot. For production environments, set the value using the permanent method.
echo 128 > /sys/block/DEVICE_NAME/queue/nr_requests
Permanent setting
To set the nr_requests value permanently, create a file that executes commands on the system before any service starts:

cat << EOF > /etc/udev/rules.d/FILE_NAME.rules ACTION=="add|change", KERNEL=="KERNEL_VALUE", SUBSYSTEM=="block", RUN+="/bin/sh -c 'echo 128 > /sys/block/%k/queue/nr_requests'" EOF

Replace or add the following:

FILE_NAME: A name for the .rules file.

KERNEL_VALUE: A partial match (with * for globbing) or an exact match for the target disk that you want to change.

For NVMe, use KERNEL=="nvme*". To target a specific disk, use an exact match like KERNEL=="nvme0n1". To target a specific partition, use an exact match like KERNEL=="nvme0n1p1".

DataStax recommends that you match the KERNEL parameter to the mount point of your transactional (Cassandra) data disk.

Additional rules: To add additional rules, add extra lines before the second EOF in the command.

To add rules after creating the file, edit your .rules file using a Linux file editor.

Activate the rules:

/sbin/udevadm control --reload-rules

/sbin/udevadm trigger --type=devices --action=change

Verify the setting by checking a device’s configuration:

cat /sys/block/DEVICE_NAME/queue/nr_requests
Set the readahead value for the block device to 8 KB.

This setting tells the operating system not to read extra bytes, which can increase I/O time and pollute the cache with bytes that weren’t requested by the user.

The recommended readahead setting is the same for RAID and non-RAID SSDs.

Add the following lines to /etc/rc.local to set readahead on startup:
```
touch /var/lock/subsys/local
echo 0 > /sys/class/block/sda/queue/rotational
echo 8 > /sys/class/block/sda/queue/read_ahead_kb
```

Optimize spinning disks

DataStax recommends a 128 KB read ahead for spinning disks.

Read ahead is an operating system feature that attempts to keep as much data as possible loaded in the page cache. Spinning disks can have long seek times that cause high latency. Therefore, additional throughput on reads using page cache can improve performance.

Setting read ahead with the blockdev tool doesn’t persist after reboot. DataStax recommends that you use a run-level init.d script to set read ahead during the boot process.
Read ahead is never useful for solid state drives (SSDs) or small partitions, such as tables with a single partition key. Misconfigured read ahead can increase disk usage, decrease throughput, and increase latency as much as five times. Read-heavy, key-value tables with small (under 1 KB) rows are especially prone to this problem.

Make sure the current readahead value isn’t 65536:
```
sudo blockdev --report /dev/SPINNING_DISK_NAME
```
Set readahead to 256, which equates to 128 KB:
```
sudo blockdev --setra 256 /dev/SPINNING_DISK_NAME
```
blockdev sets the read ahead in 512-byte sectors. Therefore, --setra 256 equates to a 128 KB read ahead: (256 * 512 B) / 1024 = 128 KB.

Set the heap size for Java garbage collection

Heap is important for efficient garbage collection (GC) operations.

By default, DSE allocates 25 to 50 percent of memory to the JVM heap, up to a maximum of 8GB. You can explicitly set heap size, as explained in Tuning Java Virtual Machine and Changing heap size parameters.

The default JVM garbage collector (GC) for DSE is G1 GC, which is is more performant when heap size is set larger than the default value. For heap recommendations, see Capacity planning: Memory and heap.

Don’t devote all memory to heap because it is also used for off-heap cache and file system cache.

To determine the heap size for your environment, monitor GC heap usage on a single node:

Set the MAX_HEAP_SIZE in the jvm-server.options file to a high arbitrary value on a single node.

This is a temporary change that allows the node to use as much heap as it needs for GC. This configuration decreases performance for the test node, but it doesn’t significantly reduce cluster performance in most cases.
View the heap used by that node.

For example, the GCInspector class logs information about any garbage collection that takes longer than 200 ms. Garbage collections that occur frequently and take a moderate length of time (seconds) to complete, indicate excessive garbage collection pressure on the JVM.
Use that value to set the heap size in the cluster.

For assistance with GC and JVM tuning, contact IBM Support.

Check Java Hugepages settings

Many modern Linux distributions ship with the Transparent Hugepages (THP) feature enabled by default. When Linux uses THP, the kernel tries to allocate memory in large chunks (usually 2 MB), rather than 4K. This allocation can improve performance by reducing the number of pages the CPU must track. However, some applications still allocate memory based on 4K pages, which can cause noticeable performance problems when Linux tries to defragment 2 MB pages.

DataStax recommends disabling defrag for THP:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

Recommended production settings

Chunk cache

Chunk cache configuration

Recommended chunk cache memory

Install the Java Virtual Machine (JVM)

Install OpenJDK 8 on RHEL

Install OpenJDK 8 on Debian

Install Oracle JDK on RHEL

Install Oracle JDK on Debian

Synchronize clocks

Set kernel parameters

TCP settings

Set user resource limits

Disable settings that throttle performance

Disable CPU frequency scaling

Disable `zone_reclaim_mode` on NUMA systems

Disable swap

Optimize disk settings

Optimize SSDs

Optimize spinning disks

Set the heap size for Java garbage collection

Check Java Hugepages settings

See also

Was this helpful?

Give Feedback