DataStax Enterprise recommended settings for Docker

General guidance

Software versions

The official DataStax images include the latest DataStax Agent version at the time of official image build. If you require a version of the DataStax Agent that differs from the one included with the official image, you must build an image that includes the required versions.

Hardware settings

The optimum readahead setting for RAID on SSDs (in Amazon EC2) is 8 KB, the same as it is for non-RAID SSDs. For details, see Optimizing SSDs.

sudo blockdev --report /dev/spinning_disk

sudo blockdev --setra 128 /dev/spinning_disk

System settings

Because time is not namespaced in the Linux kernel, containers share the clock with the Docker host machine. Ensure that clocks are synchronized on the host machines and containers by configuring NTP or other methods on the host machines.

Swapping must be disabled for performance and node stability. Run the following command on the Docker host to disable swap. The Docker host passes this setting to the container.

See Disabling swap for DSE 6.7 | DSE 6.0 | DSE 5.1 | DDAC.

sudo swapoff --all

• To disable swap per container, see Preventing a container from using SWAP in the Docker documentation.
• To make this change permanent, remove all swap file entries from /etc/fstab.

To ensure optimal performance, do not use governors that lower the CPU frequency. Instead, reconfigure all CPUs to use the performance governor on the Docker hosts.

See Disable CPU frequency scaling for DSE 6.7 | DSE 6.0 | DSE 5.1 | DDAC.

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
do
    [ -f $CPUFREQ ] || continue
    echo -n performance > $CPUFREQ
done

THP can cause performance issues in DSE or DDAC when it defragments 4 K chunks into 2 MB chunks. To disable defrag, run the following command on the Docker host:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

See Check Java Hugepages settings for DSE 6.7 | DSE 6.0 | DSE 5.1 | DDAC.

All containers by default inherit user limits from the Docker daemon. In production environments, DSE expects the following changes to ulimit:

ulimit -n 100000 # nofile: max number of open files
ulimit -l unlimited # memlock: maximum locked-in-memory address space

1. Run the following command to check the Docker daemon defaults for ulimits:

   docker run --rm ubuntu /bin/bash -c 'ulimit -a'

2. To set ulimit for Docker containers, run the docker run command with the following ulimit options:

   --ulimit nofile=100000:100000 --ulimit nproc=32768 --ulimit memlock=-1:-1

DSE tries to lock memory using mlock. When running in Docker, that capability is disabled. To enable mlock, add the following option to the docker run command:

--cap-add=IPC_LOCK

On the Docker host, check the value of vm.max_map_count, which should be set to 1048575.

cat /proc/sys/vm/max_map_count

To set the value of vm.max_map_count, add the following line to /etc/sysctl.conf, and then run sysctl -p to propagate the changes.

vi /etc/sysctl.conf

vm.max_map_count = 1048575

sudo sysctl -p

See Set user resource limits for DSE 6.7 | DSE 6.0 | DSE 5.1 | DDAC.

For each container in production environments, explicitly set the JVM heap size using the JVM_EXTRA_OPTS environment variable with the docker run command.

For example, to use 16 GB for the JVM heap, run the docker run command with the following option:

docker run -e JVM_EXTRA_OPTS="-Xms16g -Xmx16g"

Storage and resource requirements

For advanced configuration management, DataStax provides a mechanism for modifying configurations without replacing or customizing DataStax Docker containers. When any of the approved configuration files are mounted to a host volume, the files are mapped automatically within the container. See Using the DSE configuration volume.

The DSE Docker container writes all node-specific data in the directories under /var/lib/cassandra/ by default. To persist this data, map the data directories inside the container to a directory on the host file system using the -v option with the docker run command, or by using a volume driver.

docker run -v /dse/data:/var/lib/cassandra

Hosting the /var/lib/cassandra directory outside the container with the -v option allows the container to be deleted and recreated without losing data. See Persisting data.

If using the Docker devicemapper storage driver, do not use the default loop-lvm mode, which is only appropriate for testing. Instead, configure docker-engine to use direct-lvm mode, which is suitable for production environments.

See the DataStax Developer Blog Running DSE on Microsoft Windows Using Docker. When running Docker for Windows, the default resources allocated to the Linux VM running docker are 2 GB RAM and 2 CPUs. Adjust these resources as appropriate to meet the requirements for your containers. See Getting Started in Docker Desktop for Windows.

Network considerations

Because the default network settings in Docker (via Linux bridge) slows networking considerably, do not use these network settings in production environments. Instead, use docker host networking by adding the --network host option to the docker run command, or use a plugin that can manage IP ranges across clusters of hosts. The host networking limits the number of nodes per Docker host to one, which is the recommended configuration to use in production.

docker run -d --network host --name container_name