Prerequisites for upgrade

This page describes the prerequisites for upgrading an Apache Cassandra® cluster to a new version.

Some of the prerequisites are considered hard requirements for upgrading to a new version of Cassandra, while others are recommended best practices that are meant to prevent instability and errors during the upgrade process.

Operating system compatibility

If your cluster runs on bare metal or virtual machines, the host operating system (OS) must be compatible with the version of Cassandra you’re upgrading to. Furthermore, the host OS must also be compatible with Cassandra’s dependencies (see Java and Python below).

For more information about OS and dependency requirements, refer to Cassandra’s installation documentation:

Disk space requirements

Ensure that each node has at least 50% free disk space available to Cassandra.

Command
Result

df --output=pcent,target /var/lib/cassandra/data /var/lib/cassandra/commitlog

Use% Mounted on
 27% /
 27% /

Compactions generally require additional disk space when upgrading SSTables. You can still perform the upgrade on nodes that are using up to 65-70% disk space in most cases, so long as you’re following best practice and storing snapshots on a different disk than where you’re storing Cassandra data.

Java requirements

Nodes in your cluster must be running a version of Java that’s both compatible with your current version of Cassandra as well as the version of Cassandra you’re upgrading to.

For example, if you plan to upgrade a cluster from Cassandra 3.x to 4.x, all cluster nodes must be running the latest release of Java 8 before proceeding with the upgrade. Although Cassandra 4.x supports both Java 8 and Java 11, Cassandra 3.x only supports Java 8. Therefore, nodes shouldn’t be updated to Java 11 until after the cluster has been successfully upgraded to Cassandra 4.x.

Java versions required for upgrading Cassandra
Current Cassandra version	Supported Java version(s)	Java version required for upgrade
Cassandra 2.x	Java 7, 8	All nodes must be running Java 8 before upgrading to Cassandra 3.x.
Cassandra 3.x	Java 8	All nodes must be running Java 8 before upgrading to Cassandra 4.x.

Note that even if your cluster nodes are already running the necessary major version of Java, it’s a recommended best practice to update Java installations to the latest patch release as it can improve the stability and security of the Cassandra upgrade process.

For more information about Cassandra’s Java requirements, refer to Installing Cassandra.

Updating Java

Update Java on a canary node first. Drain it using the nodetool drain command, which will flush all memtables from the node to disk, stop accepting writes, and remove it from the ring without changing any token ownership. Once drained, stop the Cassandra service, then update the Java version.

Debian/Ubuntu (APT)
CentOS/RHEL (YUM)

Update the package index from sources.
```
sudo apt-get update
```
Install the new version of Java.
```
sudo apt install openjdk-11-jdk
```

Verify that the updated version of Java is now in use.

Command
Result

java -version

openjdk version "11.0.19" 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7)
OpenJDK 64-Bit Server VM (build 11.0.19+7, mixed mode)

If the updated version of Java is not set as default, you can set it using:
```
sudo update-alternatives --config java
```
Select the appropriate version from the prompt.

Update the package index from sources.
```
sudo yum update
```
Install the new version of Java.
```
sudo yum install java-11-openjdk
```

Verify that the updated version of Java is now in use.

Command
Result

java -version

openjdk version "11.0.19" 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7)
OpenJDK 64-Bit Server VM (build 11.0.19+7, mixed mode)

If the updated version of Java is not set as default, you can set it using:
```
sudo update-alternatives --config java
```
Select the appropriate version from the prompt.

After confirming the updated version of Java and restarting Cassandra, make sure to check that Cassandra is in an UP and NORMAL state:

Command
Result

nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load        Tokens  Owns (effective)  Host ID                               Rack
UN  10.166.73.33   360.5 KiB   256     28.2%             7370b2ef-c9c3-4c83-bb44-0879cef0f6c8  rack1
UN  10.166.76.162  301.28 KiB  256     34.3%             e8f291db-b3ec-47fa-8bee-27e033c9655f  rack1
UN  10.166.77.78   132 KiB     256     37.4%             baae5ba3-a842-4b23-a70b-069364dac689  rack1

Once you’ve confirmed that the canary node is functioning as expected after updating Java, proceed with updating Java in the same way on the other nodes, one node at a time.

JDK Flight Recorder

The JDK Flight Recorder (JFR) is a framework included in distributions of Java 8 and later that can be useful when diagnosing and reporting resource degradation issues. For more information about JFR and how to use it, refer to the following RedHat article: Get started with JDK Flight Recorder in OpenJDK 8u.

Python requirements

Cassandra native tools like cqlsh, as well as other external tools like Reaper and Medusa, utilize the Cassandra Python driver to communicate with Cassandra clusters. Therefore, clients running these tools must also run a version of Python that’s compatible with your current version of Cassandra as well as the version of Cassandra you’re upgrading to.

For example, if you plan to upgrade a cluster from Cassandra 3.x to 4.x, clients running cqlsh should be running the latest release of Python 2.7 before proceeding with the upgrade. Although Cassandra 4.x supports both Python 2.7 and Python 3.6, Cassandra 3.x only supports Python 2.7. Therefore, clients running cqlsh shouldn’t be updated to a newer version of Python until after the cluster has been successfully upgraded to Cassandra 4.x.

For more information about Cassandra’s Python requirements, refer to Installing Cassandra.

Client driver compatibility

If your applications make use of client drivers to interact with your Cassandra database, you must ensure that the drivers are compatible with both your current version of Cassandra as well as the version of Cassandra you’re upgrading to. Check your driver’s documentation for any known issues and incompatibilities, and update your driver accordingly. Depending on the driver version, you may need to recompile your client application code.

DataStax provides specific compatibility information for all of their Cassandra drivers.

For increased stability and security, you should upgrade to your driver to the latest patch release that supports both your current version of Cassandra and the version of Cassandra you’re upgrading to.

Setting the native protocol version

If you’re using an older driver such as Java Driver 3.x, then you’ll need to explicitly set your desired native protocol version in all of your client drivers, and also check that the version is compatible with both your current version of Cassandra as well as the version of Cassandra you’re upgrading to. Otherwise, during a cluster upgrade, you may experience driver-specific impact when the cluster is in a partially upgraded state.

For example, if you plan to upgrade a cluster from Cassandra 3.x to 4.x, the native protocol must be set to either V3 or V4 in all of your client drivers prior to upgrading the cluster. (You can update to V5 or higher once the cluster has been successfully upgraded to Cassandra 4.x.)

The reason why it’s recommended to explicitly set the native protocol version is because when ProtocolVersion is unset, the native protocol version is negotiated with the first host to which the driver connects (though newer drivers, such as Java Driver 4.x, automatically select a protocol version that works across nodes). If the client negotiates with an upgraded node that defaults to a higher native protocol version, then all yet-to-be upgraded nodes using a lower protocol version will be considered down and unavailable to the client.

An alternative to setting the ProtocolVersion is to ensure that the list of initial contact points contains only hosts with the oldest Cassandra version or protocol version. However, this isn’t recommended.

SSTable compatibility

Cassandra can only read SSTables in the current major format, as well as last major format immediately preceding it. Therefore, before upgrading to a new major version of Cassandra, you must ensure that all SSTables on all nodes are using the latest format introduced by your current version of Cassandra.

For example, if you plan to upgrade a cluster from Cassandra 3.x to 4.x, you must ensure that all SSTables have been upgraded to the latest Cassandra 3.x format before upgrading to Cassandra 4.x, as Cassandra 4.x cannot read SSTables in any pre-3.x format.

Failure to upgrade SSTables when required can result in a significant impact to performance, increased disk usage, and possible data loss.

Upgrading SSTables

The following steps describe how to use the nodetool upgradesstables command to upgrade the SSTables on an online cluster. Offline SSTable upgrades utilizing the sstableupgrade tool are outside the scope of this guide.

Run the following command to upgrade the SSTables on a node:

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken.

You can use the --jobs option to set the number of SSTables that upgrade simultaneously. For example:

nodetool upgradesstables --jobs 4

The default setting is 2, which minimizes impact on the cluster. Setting the number of jobs to 0 will use all available compaction threads. It’s important to note, however, that the number of jobs cannot exceed the concurrent_compactors configured in cassandra.yaml.

Monitor the SSTable upgrade process.
1. The upgradesstables command operates much like a single-table compaction that rewrites the same SSTable using the new format. Because SSTables are stored in sorted order, CPU usage and disk I/O should be relatively low. However, you should monitor Cassandra and application latency metrics to ensure that concurrent executions of the upgradesstables command don’t overwhelm the cluster.
2. The upgradesstables command relies on the compaction thread pool for orchestration. You can monitor progress with the following command:
  Command
  
  Result
  
  watch -d "nodetool compactionstats -H"
  
  Every 2.0s: nodetool compactionstats -H pending tasks: 0
It’s normal behavior for the pending task count to stay above 0. However, the number of pending tasks should drastically reduce after the upgradesstables tasks have completed.
Repeat the above steps on each node in the cluster, one at a time, until all SSTables have been upgraded.

Node repair considerations

You should repair nodes regularly before upgrading. Node repair ensures that data on a replica is consistent with data on other nodes.

Cassandra can only repair across nodes when all SSTables are in the current file format (see SSTable compatibility).

A repair requires that all replicas for a token range be available before it can start. Once the repair is started, restarting a node will cause the process to fail.

For this reason, it’s recommended that you disable automated node repairs before beginning the upgrade process. The upgrade process entails a rolling restart of the cluster nodes and any scheduled repairs will most likely fail. Failing repairs on each node will generate a large amount of noise in the logs and cause unpredictable pressure during the upgrade operation.

Availability considerations

Ensure that areas of your applications that require strong consistency are using the LOCAL_QUORUM consistency level and that relevant keyspaces have a replication factor of 3.

When LOCAL_QUORUM is used with a replication factor below 3, all nodes must be available for requests to start. A rolling restart using this configuration will result in full or partial unavailability while a node is down.

Prevent schema changes

No schema changes should be performed during the upgrade process.

Propagation of schema changes between mixed version nodes might have unexpected results. Therefore, you should take action to prevent schema changes for the duration of the upgrade process.

Configuration management

Production version binaries for rollback

Ensure you have a copy of the installation packages for the Cassandra version currently running in production. If required, the packages should be downloaded. Keep these binaries readily available in the event they are required to roll back the canary node.

The following links to the different package types.

Tarballs: https://archive.apache.org/dist/cassandra/
RedHat Repo (and rpm files): https://redhat.cassandra.apache.org/311x/
Debian Repo: https://debian.cassandra.apache.org
Debian deb files: https://debian.cassandra.apache.org/pool/main/c/cassandra/
New version binaries for upgrade

Ensure you also have a copy of the installation packages for the new Cassandra version you’re upgrading to. If required, the packages should be downloaded.

Custom Patches

If the existing production version of Cassandra was patched, make sure patches you made are compatible with the new version of Cassandra. If not, you’ll need to update the patches to be compatible with the new version of Cassandra.

Docker considerations

The upgrade process requires additional action when the cluster is running in Docker.

Upgrading Cassandra on traditional dedicated servers involves updating the packages and configuration files on the server. In contrast, a Docker-based cluster starts up new containers with the updated packages and configurations.

Challenges with Docker:

New Docker containers usually result in new IP addresses.
Cassandra data must reside on Docker volumes.
Seed node IP addresses are hardcoded in cassandra.yaml files.

The recommended approach in addressing these challenges is to use the docker --ip flag to prevent the node’s IP address from changing from one container to the next, and to have all the node’s data on one or more Docker volumes that can be reattached from one container to the next. Note, the --ip flag can only be used if Docker is using a non-default network bridge.

Seed nodes

There are a few extra steps to upgrade a seed node container. Before starting the replacement seed node, the IP address of the existing seed node container will need to be removed from the seed_provider list in the cassandra.yaml file of every other node container. As it’s recommended to always have three seed nodes listed for each datacenter, another node in the cluster may need to be added to the list. This needs to be done for all of the nodes in the cluster. A rolling restart will need to be performed for the changes to take effect. Once the change is complete the replacement node can be started.

Making sure persisting such changes and performing a rolling restart is cumbersome without the use of the --ip flag. As each update to the configuration in the image would require a restart of the Docker container that would in turn change the IP address and require another configuration change. If the Cassandra node can be stopped in the container (without the container stopping, as is typically the case) it’s then possible to edit the cassandra.yaml file inside the existing container, and where in addition the updated configuration in the new images would contain (and have to maintain) both old and new seed IP addresses. Because this process is cumbersome, it’s recommended to pin the IP address.

Docker volumes

It’s recommended that nodes have their data_file_directories, hints_directory, and commitlog_directory directories on docker volumes. This simplifies upgrades as new containers using new images re-use existing Docker volumes.

A Cassandra node that starts up with existing data_file_directories in place is a continuation of that node in the cluster. Even if the new container has a new IP address, it will identify this and update itself and notify other nodes of the IP change. The handling of such IP address changes does not solve the problem with the configuration of seed nodes in the previous section, so the recommendation is still to pin the IP addresses.

If reattaching a volume cannot be done to a new container that maintains its IP address, then the node must be repaired immediately after it’s started. It’s critical that the node is repaired and holds a complete replica for all token ranges it’s responsible for (not just primary ranges) before any other node is upgraded or container restarted.

If reattaching a volume cannot be done to a new container that also cannot maintain its IP address, the new node/container must be started with the cassandra.replace_address_first_boot flag, as described in the next section. An alternative approach is upgrading the cluster via datacenter migration. However, a datacenter migration is beyond the scope of this document.

An empty `data_file_directory` and a changing IP address

If a Docker container representing a previous node is started with a new IP address and without the data from the previous node present in an attached Docker volume, then it must know about what that node’s previous IP address is. That is, the upgraded container must be started with the following JVM option: -Dcassandra.replace_address_first_boot=<IP_ADDRESS>. The easiest way to accomplish this is by adding the option to the jvm.options file. For more information on using this JVM flag, see the following blog post: Bootstrapping Apache Cassandra Nodes However, it’s recommended that you avoid using the cassandra.replace_address_first_boot flag by either re-using the node’s IP address and/or re-using the node’s data_file_directory.

It’s critical that the node is repaired and holds a complete replica for all token ranges it’s responsible for (not just primary ranges) before any other node is upgraded or container restarted.