Recommended production settings

Use Network Time Protocol (NTP) to synchronize the clocks on all nodes and application servers.

Synchronizing clocks is required because DataStax Enterprise (DSE) overwrites a column only if there is another version whose timestamp is more recent, which can happen when machines are in different locations.

DSE timestamps are encoded as microseconds because UNIX Epoch time does not include timezone information. The timestamp for all writes in DSE is Universal Time Coordinated (UTC). DataStax recommends converting to local time only when generating output to be read by humans.

1. Install NTP for your operating system:

   Operating system	Command
   Debian-based system	sudo apt-get install ntpdate
   RHEL-based system1	sudo yum install ntpdate
   1On RHEL 7 and later, chrony is the default network time protocol daemon. The configuration file for chrony is located in /etc/chrony.conf on these systems.

2. Start the NTP service on all nodes:

   sudo service ntp start -x

3. Run the ntupdate command to synchronize clocks:

   sudo ntpdate 1.ro.pool.ntp.org

4. Verify that your NTP configuration is working:

   ntpstat

TCP settings

During low traffic intervals, a firewall configured with an idle connection timeout can close connections to local nodes and nodes in other data centers. To prevent connections between nodes from timing out, set the following network kernel settings:

1. Set the following TCP keepalive timeout values:

   sudo sysctl -w \
   net.ipv4.tcp_keepalive_time=60 \
   net.ipv4.tcp_keepalive_probes=3 \
   net.ipv4.tcp_keepalive_intvl=10

   These values set the TCP keepalive timeout to 60 seconds with 3 probes, 10 seconds gap between each. The settings detect dead TCP connections after 90 seconds (60 + 10 + 10 + 10). The additional traffic is negligible, and permanently leaving these settings is not an issue. See Firewall idle connection timeout causes nodes to lose communication during low traffic times on Linux .

2. Change the following settings to handle thousands of concurrent connections used by the database:

   sudo sysctl -w \
   net.core.rmem_max=16777216 \
   net.core.wmem_max=16777216 \
   net.core.rmem_default=16777216 \
   net.core.wmem_default=16777216 \
   net.core.optmem_max=40960 \
   net.ipv4.tcp_rmem='4096 87380 16777216' \
   net.ipv4.tcp_wmem='4096 65536 16777216'

Set user resource limits

Use the ulimit -a command to view the current limits. Although limits can also be temporarily set using this command, DataStax recommends making the changes permanent.

For more information, see Recommended production settings.

1. Edit the /etc/pam.d/su file and uncomment the following line to enable the pam_limits.so module:

   session    required   pam_limits.so

   This change to the PAM configuration file ensures that the system reads the files in the /etc/security/limits.d directory.

2. If you run DSE as root, some Linux distributions (such as Ubuntu), require setting the limits for the root user explicitly instead of using cassandra_user:

   root - memlock unlimited
   root - nofile 1048576
   root - nproc 32768
   root - as unlimited

1. Set the nproc limits to 32768 in the /etc/security/limits.d/90-nproc.conf configuration file:

   cassandra_user - nproc 32768

1. Add the following line to /etc/sysctl.conf:

   vm.max_map_count = 1048575

2. Open the configuration file for your installation type:

   Installation type	Configuration file
   Tarball installation	/etc/security/limits.conf
   Package installation	/etc/security/limits.d/cassandra.conf

3. Configure the following settings for the <cassandra_user> in the configuration file:

   <cassandra_user> - memlock unlimited
   <cassandra_user> - nofile 1048576
   <cassandra_user> - nproc 32768
   <cassandra_user> - as unlimited

4. Reboot the server or run the following command to make all changes take effect:

   sudo sysctl -p

Persist updated settings

Disable CPU frequency scaling

Recent Linux systems include a feature called CPU frequency scaling or CPU speed scaling. This feature allows a server's clock speed to be dynamically adjusted so that the server can run at lower clock speeds when the demand or load is low. This change reduces the server's power consumption and heat output, which significantly impacts cooling costs. Unfortunately, this behavior has a detrimental effect on servers running DSE, because throughput can be capped at a lower rate.

On most Linux systems, a CPUfreq governor manages the scaling of frequencies based on defined rules. The default ondemand governor switches the clock frequency to maximum when demand is high, and switches to the lowest frequency when the system is idle.

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
do
    [ -f $CPUFREQ ] || continue
    echo -n performance > $CPUFREQ
done

For more information, see High server load and latency when CPU frequency scaling is enabled in the DataStax Help Center.

Disable zone_reclaim_mode on NUMA systems

The Linux kernel can be inconsistent in enabling/disabling zone_reclaim_mode, which can result in odd performance problems.

echo 0 > /proc/sys/vm/zone_reclaim_mode

For more information, see Peculiar Linux kernel performance problem on NUMA systems.

Disable swap

Failure to disable swap entirely can severely lower performance. Because the database has multiple replicas and transparent failover, it is preferable for a replica to be killed immediately when memory is low rather than go into swap. This allows traffic to be immediately redirected to a functioning replica instead of continuing to hit the replica that has high latency due to swapping. If your system has a lot of DRAM, swapping still lowers performance significantly because the OS swaps out executable code so that more DRAM is available for caching disks.

If you insist on using swap, you can set vm.swappiness=1. This allows the kernel swap out the absolute least used parts.

sudo swapoff --all

To make this change permanent, remove all swap file entries from /etc/fstab.

For more information, see Nodes seem to freeze after some period of time.

Optimize SSDs

Complete the following steps to ensure the best settings for SSDs.

1. Ensure that the SysFS rotational flag is set to false (zero).

   This overrides any detection by the operating system to ensure the drive is considered an SSD.

2. Apply the same rotational flag setting for any block devices created from SSD storage, such as mdarrays.
3. Determine your devices by running lsblk:

   lsblk

   NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
   vda    253:0    0  32G  0 disk
   |
   |-sda1 253:1    0   8M  0 part
   |-sda2 253:2    0  32G  0 part /

   In this example, the current devices are sda1 and sda2.

4. Set the IO scheduler to either deadline or noop for each of the listed devices:
   For example:

   echo deadline > /sys/block/device_name/queue/scheduler

   where device_name is the name of the device you want to apply settings for.

   • The deadline scheduler optimizes requests to minimize IO latency. If in doubt, use the deadline scheduler.

     echo deadline > /sys/block/device_name/queue/scheduler

   • The noop scheduler is the right choice when the target block device is an array of SSDs behind a high-end IO controller that performs IO optimization.

     echo noop > /sys/block/device_name/queue/scheduler

5. Set the nr_requests value to indicate the maximum number of read and write requests that can be queued:

   Machine size	Value
   Large machines	echo 128 sys/block/device_name/queue/nr_requests
   Small machines	echo 32 sys/block/device_name/queue/nr_requests

6. Set the readahead value for the block device to 8 KB.

   This setting tells the operating system not to read extra bytes, which can increase IO time and pollute the cache with bytes that weren’t requested by the user.

   Note: The recommended readahead setting for RAID on SSDs is the same as that for SSDs that are not being used in a RAID installation.
   1. Open /etc/rc.local for editing.
   2. Add the following lines to set the readahead on startup:

      touch /var/lock/subsys/local
      echo 0 > /sys/class/block/sda/queue/rotational
      echo 8 > /sys/class/block/sda/queue/read_ahead_kb

   3. Save and close /etc/rc.local.

Optimize spinning disks

1. Check to ensure read-ahead value is not set to 65536:

   sudo blockdev --report /dev/spinning_disk

2. Set the readahead to 128, which is the recommended value:

   sudo blockdev --setra 128 /dev/spinning_disk

Heap size is usually between ¼ and ½ of system memory. Do not devote all memory to heap because it is also used for offheap cache and file system cache.

See Tuning Java Virtual Machine for more information on tuning the Java Virtual Machine (JVM).

If you don't see improved performance, contact the DataStax Services team for additional help in tuning the JVM.

Many modern Linux distributions ship with the Transparent Hugepages feature enabled by default. When Linux uses Transparent Hugepages, the kernel tries to allocate memory in large chunks (usually 2MB), rather than 4K. This allocation can improve performance by reducing the number of pages the CPU must track. However, some applications still allocate memory based on 4K pages, which can cause noticeable performance problems when Linux tries to defragment 2MB pages.

For more information, see the Cassandra Java Huge Pages blog and this RedHat bug report.

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

For more information, including a temporary fix, see No DSE processing but high CPU usage.