Test your Apache Cassandra cluster

Before deploying to production, verify your configuration and test your cluster with simulated production workloads in isolation and at scale.

Testing helps you find and resolve issues before applications and configuration changes go live. Metrics collected during testing help you set measurable benchmarks, such as throughput and latency, that you can use to monitor performance in production and project future capacity needs.

After deploying to production, continue to monitor, reconfigure, and test your cluster as needed.

Monitoring and testing are universal best practices for application development. This guide provides specific advice for testing Cassandra clusters with a focus on new deployments. It doesn’t cover all aspects of synthetic testing and full-stack observability.

Prerequisites

Run system checks

You can run system-level checks when planning a Cassandra deployment or investigating cluster performance issues. Most checks use basic OS tooling, although some checks use more advanced tools or scripts. To get started, see Diving Deep, Use External Tools with Apache Cassandra.

Run these checks on each node in a cluster, or use diagnostic scripts to gather this information in bulk. DataStax recommends using diagnostic scripts for large clusters with many nodes and multi-server deployments where it can be tedious to run individual checks on each node.

Some checks can be completed while the nodes are idle. Others require activity on the node or network traffic flowing to and from the node. You might need to revisit these checks during your simulated workload tests.

The following list summarizes some checks you can perform or inspect to verify your cluster configuration. For a comprehensive explanation of system checks, see Diving Deep, Use External Tools with Apache Cassandra.

Startup checks

Cassandra 3.0 and later perform some configuration checks on system start. Potential misconfigurations are output in the startup logs. For example:

WARN [main] 2020-02-13 11:00:47,676 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
WARN [main] 2020-02-13 11:00:47,677 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
Recommended settings

Review and check all recommended production settings according to your environment and workloads.

Network-related errors

  • Use ifconfig to check the number of packets with errors for RX and TX rows. Ideally, this should be between 0 and 1 percent of the total packets. Errors in more than 1 percent of the packets is a strong indicator of network issues that must be remediated.

    • errors: Total number of packets received with errors, including CRC, overruns, and others. This indicates malfunctioning network hardware in the server or elsewhere in the network.

    • overruns: Number of received packets that experienced FIFO overruns, which occur when a buffer is full and the kernel is unable to empty. You might need to investigate and tune the Linux kernel’s TCP parameters, as described in Operating system messages.

    • carrier (TX only): Number of packets that have experienced loss of carriers. This can indicate a flapping link.

  • Check network latency and throughput for node-to-node connections and client-to-cluster connections. For simple checks, use ping. For more advanced checks, use iperf, mtr, iftop, or other tools. Inside the datacenter network, latency should not be more than 1 ms.

  • Make sure required ports are open and firewalls aren’t blocking communication. Use nc -zv host port to check for existing connections to specific ports.

  • Check that the Linux kernel is configured with the recommended production settings for TCP and networking.

Disks
CPU

  • Use lscpu or check /proc/cpuinfo to determine the CPU model, number of cores, frequency, and other information.

  • On the systems with multiple physical CPUs, use lstopo or numactl to ensure that all PCI components are in NUMA mode.

  • Check /proc/interrupts for interrupts between the different CPUs, and that interrupts are not assigned to the same CPU. This can lead to overloading of a specific CPU. If necessary, tune the SMP affinity for specific interrupt requests.

  • Check CPU stats reported by Linux tools such as iostat and vmstat. Large amounts of steal time or CPU-ready time can indicate that your virtual machine is over-allocated, you have noisy neighbors (other services that are running on the same server as your virtual machine), or both.

Operating system messages

Run dmesg -T to get system messages with a human-readable timestamp, and then search for any entries about out-of-memory (OOM) killer activity, segfaults, disk problems, network issues (typically TCP SYN flood), and other related issues. For example:

  • For OOM killer, search for messages formatted like Out of memory: Killed process NNN, and then check if the errors were caused by a Cassandra process.

  • For process segfaults, search for messages formatted like name[proc_id]: segfault at, and then check if the errors were caused by a Cassandra process.

  • For TCP SYN flood, search for messages formatted like TCP: Possible SYN flooding on port NNN, and then refer to the documentation for your specific Linux distribution to resolve the issue. For example, see kernel: Possible SYN flooding on port. Pay particular attention to messages with ports belonging to Cassandra.

  • For disk problems, search for messages formatted like disk_id: failed command or disk_id: exception where disk_id is a device identifier like ata7.00, depending on the disk type.

/tmp directory permissions

The /tmp directory must be executable because it is used for extraction and loading of the native code.

If the /tmp directory is not executable, then add the following two statements to the JVM options file:

-Dio.netty.native.workdir=PATH/TO/TMP
-Djna.tmpdir=PATH/TO/TMP

Define SLAs

For any application, you must calculate and set Service Level Agreements (SLAs) that define acceptable performance thresholds and failure tolerance.

This guide focuses on SLAs for Cassandra clusters. A full explanation of SLAs is beyond the scope of this document.

SLAs are measured in percentiles. However, this doesn’t mean that the highest percentiles are the best for evaluating SLAs.

For example, when measuring request latency, use the following guidelines for SLA percentile targets:

Latency percentiles for requests
Percentile Recommendation

100

Not recommended because it isn’t meaningful in practice. This percentile is often irrelevant to testing and based on factors external to the database.

99.9

An aggressive target that is best for latency-sensitive workloads only. Typically requires high-performance hardware and careful tuning.

99

The most common target for SLAs in practice. It represents a reasonable balance between performance and hardware/tuning for most workloads.

95

A relaxed target that is best for workloads where latency isn’t noticed or less critical. Not recommended for latency-sensitive workloads, such as high-traffic, user-facing interfaces.

Your cluster’s performance statistics and metrics that inform SLAs, such as latency and throughput, are influenced by the underlying hardware and configurations, in addition to other factors like data modeling and application logic. For example, the following metrics are based on clusters with more data on disk than RAM. Measurement was in the 99th percentile using nodetool tablehistograms.

Impact of hardware on latency
Latency (99p) Hardware

Less than 10ms

Premium, high-performance CPU nodes with NVM flash storage

Less than 150ms.

Common SATA based SSDs

Over 500ms

Common SATA based SSDs

Latency metrics vary significantly due to factors like controller, contention, and data model. Tuning can shift this metric by an order of magnitude. This can result in some hardware approaching NVM specifications.

Enable monitoring and collect metrics

Monitoring Cassandra clusters is essential for detecting, investigating, and resolving issues in your clusters.

Cassandra exposes many keyspace, table, and subsystem metrics for observation and analysis.

For all metrics, see Monitoring Apache Cassandra.

Metrics retrieval and observability tools

Cassandra uses Java Management Extensions (JMX) tools for cluster monitoring and some management tasks. You can access JMX MBeans directly or through other tools like nodetool, iostat, and dstat. Some tools are for debugging or temporary inspections of individual metrics, whereas other tools are more robust for ongoing observability.

Java Monitoring and Management Console (JConsole)

JConsole is a GUI tool included in a Java distribution, such as JConsole in OpenJDK. JConsole allows for easy browsing of the metrics and inspection of their values. It also provides the possibility to graph them over time.

This tool provides access to individual metrics at a given moment on a specific node. This tool is best for debugging and limited metrics queries because it isn’t designed to be a dedicated monitoring solution.

To access metrics, JMX must be running on the server (usually requiring installation of the GUI libraries), or it must be accessible through the network (which exposes JMX externally and might have a security impact).

jmxterm

jmxterm is a JMX command-line tool that can connect to a local or remote node (when JMX is exposed externally). For example:

open localhost:7199

_#Connection to localhost:7199 is opened_

Then, you can access specific metrics with commands such as:

get -b org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Size Value

#mbean = org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Size:
Value = 0;

In addition to obtaining values for attributes, you can set values for attributes (if they are settable), or call functions, which you can use to temporarily modify the behavior of Cassandra (in the same way that nodetool commands are used).

This tool is best for debugging and one-time metrics queries because it isn’t designed to be a dedicated monitoring solution. It can retrieve specific metrics from a node at a given moment, but it doesn’t continuously monitor or aggregate metrics over time.

nodetool sjk

In Cassandra 4.0 and later, you can use nodetool sjk, which is a wrapper for the Swiss Java Knife (SJK) library.

This subcommand is convenient because you don’t need to specify the Cassandra PID or other options that would be required when running SJK directly. Instead, you provide only the options to get the metrics you want. For example, to get a hit rate of key cache, use the following command, where the -b option specifies the name of the bean and the -f option specifies the field to retrieve:

nodetool sjk mx -b "org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=Size" -mg -f Value

Similar to jmxterm, you can use this command to set values (when settable) and call functions. Additionally, the scope of the SJK library is not limited to JMX. You can use it to get the thread dump, information about threads, and other functionality. However, it isn’t a replacement for a full monitoring solution.

Metrics Collector for Apache Cassandra

This tool exports JMX metrics for formal observability with Prometheus and Grafana (with predefined dashboards). For more information, see the Metrics Collector for Apache Cassandra repository.

External tools for integration with monitoring systems

Other monitoring tools are compatible with Cassandra, such as JMX Exporter for Prometheus. The complexity of the initial setup varies by tool.

Notable metrics

To help you get started with cluster performance monitoring, the following list introduces some useful metrics categories. Most categories include some related MBeans, recommended thresholds, and other guidance for investigating potential performance issues.

These are general recommendations. You might need to change thresholds or monitor different metrics depending on your environment and workloads.

All JMX objects are found at the MBean path org.apache.cassandra.metrics. For example, you can retrieve global write latency at org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency.

For a detailed metrics reference, see Monitoring Apache Cassandra.
Client requests

Measures how the system performs from the point of view of the client application. Important metrics include:

  • Coordinator-level latency for read operations, especially 95th and 99th percentiles.

  • Coordinator-level latency for write operations, especially 95th and 99th percentiles.

  • Pending (queued) writes.

  • Number of client connections.

Threadpools (TP)

Measures data processing and task throughput, such as compaction and flushing of data. Important metrics include:

  • Number of blocked threads, such as memtable flush writer and memtable pool allocations.

  • Number of aborted threads, such as aborted compactions.

  • Number of pending threads, such as pending compactions and pending flushes.

Individual tables

At minimum, monitor performance for your most important tables. Monitoring every metric for all tables can be noisy and reduce visibility of alerts on important tables. Important metrics include:

  • Partition size.

  • Number of SSTables overall.

  • Number of SSTables read per request.

  • Number of tombstones scanned during read request.

  • Coordinator-level read and write latencies.

  • Hit ratio for key cache (Cassandra 3.0 and earlier only).

Inter-cluster communication

Measures how data exchange happens in the cluster, including replication and hinted handoff. Important metrics include:

  • Number of dropped mutations and other messages.

  • Total number of timeouts, and timeouts per host.

  • Cross-datacenter latency.

  • System hints (number of hints on disk).

  • Hint replay (number of failed and timed-out hint messages).

Java Virtual Machine (JVM)

Measures memory usage and task execution in the JVM. Important metrics include:

  • Amount of memory used.

  • Duration of garbage collection pauses.

If heap usage shows a pattern of huge spikes with long pauses, adjust the heap.

Operating system and hardware

Measures the utilization and availability of system resources on the node. Important metrics include:

  • CPU usage on the node.

  • Amount of disk space available.

I/O wait metrics greater than 5 indicate that the CPU is waiting too long on I/O, and this is a symptom of being I/O-bound.

For CPUs that use hyper-threading, keep in mind that for each physical processor core, the operating system addresses two virtual (logical) cores, and then shares the workload between them. This means that if half your threads are at 100% CPU usage, then the CPU capacity is at maximum, even if the tool reports 50% usage. To see the actual thread usage, use the ttop command.

Log warnings and errors

Either through metrics or log files, monitor for the following strings that you might need to investigate:

  • SlabPoolCleaner

  • WARN

  • ERROR

  • GCInspector

  • hints

Other SLA-related metrics

In addition to the previously mentioned metrics, make sure you monitor all metrics related to your SLAs.

Simulate production workloads

The hardware used for load testing should mimic the parameters and number of nodes in the production environment. Performing load testing on a single node, or in environments where the number of nodes is equal to the replication factor, results in incorrect measurements.

In a test environment, run simulated production workloads, evaluating your cluster’s performance at each stage. If your cluster performs poorly or reports errors, investigate the issues, reconfigure your cluster, and then rerun the tests until your cluster meets your desired performance targets.

Make sure your tests cover a range of scenarios:

  • Reads and writes: Common queries, uncommon queries, prepared statements, BATCH, LTWs, and known slow queries, such as SELECT *.

  • Database operations: nodetool commands, memtable flushes, compactions.

  • Uncommon or undesirable events: Outages, bootstrapping new and replacement nodes, repairs, disaster recovery, and failures.

There are many ways to run these tests. For example, you can create test scripts that use a Cassandra driver or cqlsh to run read/write operations against the cluster. Compared with production applications, you might need to include extended logging, tracing, or tailing to capture more detailed information about the cluster’s behavior during the tests.

For each of the following scenarios, questions and key metrics are provided to help you evaluate cluster performance:

  1. Simulate average read/write traffic.

    Some level of I/O must continue throughout all tests to adequately measure node performance under baseline I/O stress in addition to each test scenario. Maximum throughput under ideal circumstances isn’t a realistic measurement of cluster performance. As you perform the remaining tests, you can understand how each scenario impacts performance. Then, you can adjust your performance targets or reconfigure your cluster and repeat the tests until your cluster performs as desired.

  2. Simulate production outages:

    1. For small clusters, take one or two nodes offline. For large clusters, take enough nodes offline that the outage is impactful and measurable in testing.

      • Do your queries start failing?

      • Do the nodes start working without any change when they come back online?

    2. Take a datacenter offline while processing reads and writes.

      How long can the cluster be offline before system.hints build up to unmanageable levels?

  3. Simulate operations and queries that can stress the cluster and overload nodes, potentially taking nodes offline or occupying them for long periods of time.

    These tests help you understand your clusters resilience to stress, malformed or suboptimal queries, and other workload spikes.

    1. Run resource-intensive tasks that use CPU on multiple nodes, such as cluster-wide compactions, schema changes, or memtable flushes.

    2. Run resource-intensive range queries, such as SELECT * and other known slow, inefficient, or long-running queries.

      For example, the following query reads up to 100,000,000 rows from a given table without paging:

      PAGING OFF;
SELECT * FROM keyspace1.table1 LIMIT 100000000;

  4. Simulate production operations during average read/write load:

    1. Run repair to ensure that you have enough overhead to complete a repair.

    2. Bootstrap a new node into the cluster.

      Note how long it takes to bootstrap the new node.

    3. Add a datacenter.

    4. Change the schema by adding new tables, removing tables, and altering existing tables.

      If you tested schema changes in the previous steps, make sure you test different types of changes. For example, altering one column might not have the same impact as removing an entire table.

    5. Replace a failed node using the simulated outage scenarios described in the previous steps.

      Note how long it takes to bootstrap the replacement node.

    6. Repeat these tests during heavy read/write load to simulate high traffic periods.

    7. If you haven’t done so already, repeat these tests during the outage and overload scenarios in the previous steps.

    These tests ensure that your cluster can tolerate normal and abnormal database operations under normal and heavy I/O load. This is also a good opportunity to practice these procedures and events in a non-production environment.

  5. Establish transactions per second (TPS) targets.

    Determine how many TPS you expect your cluster to handle. Ideally, align this number with the node count in your test cluster. Alternately, test with a ratio of TPS to nodes to evaluate how the cluster behaves.

    If the cluster does not meet your SLA requirements, adjust your targets or look for the root cause. For example, does the cluster meet SLAs if the TPS level is lower?

  6. Run a real replication factor on at least 5 nodes.

    If you plan to run more than 5 nodes in production, run the same number of nodes that you expect to run in production.

    If you plan to run 3 nodes in production, test with 5 nodes. You might find that some queries that were faster with 3 nodes are slower with 5 nodes. For example, secondary indexes are often slower at scale.

    Use the same replication strategy that you plan to use in production.

  7. Simulate a production datacenter setup.

    Each datacenter adds additional write costs. If your production cluster has more datacenters than your test cluster, your writes might be greater than you anticipate. For example, if your replication factor is 3, and your test cluster has 2 datacenters, each write goes to 4 nodes in 2 datacenters. If your production cluster uses a replication factor of 3 with 5 datacenters, each write goes to 7 nodes with 5 datacenters. This means production writes are almost twice as expensive on the coordinator node because, in each remote datacenter, the forwarding coordinator sends the data to the other replicas in its datacenter.

  8. Simulate your network topology with the same latency and pipe.

    Test with your expected TPS.

    Be aware that remote links can be a limiting factor for many use cases and expected bandwidth.

  9. Simulate expected node density.

    Once you’ve determined your target per-node density, add data to the test cluster until the target capacity is surpassed. Observe the following as you approach, reach, and surpass the target density:

    • What happens to your SLAs when approaching the target density?

    • What happens when surpassing the target density?

    • How long does bootstrap take with maximum or excess density?

    • How long does repair take with maximum or excess density?

Use the NoSQLBench tool for testing

NoSQLBench is an open-source testing tool that emulates real application workloads in your development environment. This means that you can run performance, sizing, and data model tests without writing your own testing harness or requiring live production workloads.

With NoSQLBench, you can:

  • Run common testing workloads directly from the command line.

  • Generate virtual datasets of arbitrary size, with deterministic data and statistically shaped values.

  • Design custom workloads that emulate your application, contained in a single file, based on statement templates with no IDE or coding required.

  • Plot your results in a Docker and Grafana stack on Linux with a single command line option.

  • Optionally customize the runtime behavior of NoSQLBench for advanced testing, including the use of a JavaScript environment.

For more information, see the following:

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM