Phase 3: Upgrade the first seed node

With the cluster ready, you can now begin upgrading to the new version of Cassandra. This procedure uses a rolling upgrade process so that the cluster does not incur downtime.

This phase should be executed on the seed node in the cluster. Once the upgrade is confirmed to be successful on this node, you can move on to Phase 4: Upgrade the remaining nodes.

Step 1: Ensure all nodes are up

Ensure all nodes in the cluster are reporting status UN (Up and Normal).

Run the following command to check for any nodes in the cluster that are currently reporting a status other than UN:

Command
Result

nodetool status | grep -v UN

If all nodes are in an Up and Normal state, then there should be no nodes listed in the output:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack

Step 2: Shut down Cassandra

Once you’ve confirmed that all nodes are up and in a consistent state, stop the running Cassandra service on the seed node.

Drain the node.

Command
Result

nodetool drain

The nodetool drain command doesn’t return any output.

You can monitor drain progress by checking the Cassandra system.log file for messages similar to the following:

INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:37,442 StorageService.java:1660 - DRAINING: starting drain process
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:37,443 HintsService.java:210 - Paused hints dispatch
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:37,449 Server.java:179 - Stop listening for CQL clients
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:37,449 Gossiper.java:1720 - Announcing shutdown
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:37,465 StorageService.java:2585 - Node /10.166.73.33 state jump to shutdown
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:39,469 MessagingService.java:985 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/10.166.72.33] 2023-06-01 03:59:39,470 MessagingService.java:1346 - MessagingService has terminated the accept() thread
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:39,794 HintsService.java:210 - Paused hints dispatch
INFO  [RMI TCP Connection(4)-127.0.0.1] 2023-06-01 03:59:39,806 StorageService.java:1660 - DRAINED

You can also confirm the status of the drain by running the nodetool netstats command and checking for Mode: DRAINED in the output.

Command
Result

nodetool netstats

The drain was successful if you see Mode: DRAINED in the output.

Mode: DRAINED
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed   Dropped
Large messages                  n/a         2              0         0
Small messages                  n/a         2              5         0
Gossip messages                 n/a         2            122         0

When you run the nodetool drain command, Cassandra stops listening for connections from clients and other nodes (no more data is written to the node) and all memtables are flushed to SSTables on disk. This ensures that all the data on the node is safely stored on disk before beginning the upgrade.

After running nodetool drain, the node will not be able to service client reads or writes until Cassandra is restarted.

Stop Cassandra.
- Package installations
- Tarball installations
To stop the Cassandra service for packaged installations:
sudo service cassandra stop
To stop the Cassandra process for tarball installations:
sudo kill $(ps auwx | grep cassandra | grep -v "grep" | tr -s ' ' | cut -d' ' -f2)
Ensure that no Cassandra process is running on the server.
- Command
- Result
ps auwx | grep CassandraDaemon
root 27921 0.0 0.0 3304 656 pts/0 S+ 07:38 0:00 grep --color=auto CassandraDaemon
Note that Cassandra might take some time to shut down, especially if it’s currently handling requests. If the service continues to run, then kill the process using the following command:
```
sudo kill -9 $(ps auwx | grep CassandraDaemon | grep -v "grep" | tr -s ' ' | cut -d' ' -f2)
```

Step 3: Back up Cassandra configuration files

It’s important to create a backup of Cassandra configuration files, as they may be overwritten with default values during installation of the new version.

Create a backup of the contents remaining in the configuration directory.

Package installations
Tarball installations

For package-based installations, the following command creates a backup of the Cassandra configuration files and places it in the user’s home directory:

Command
Result

cd /etc/cassandra && tar czfv ~/cassandra-config-backup.tgz .

./
./cassandra-topology.properties
./logback.xml
./cassandra-rackdc.properties
./hotspot_compiler
./triggers/
./triggers/README.txt
./cassandra.yaml
./cassandra-env.sh
./commitlog_archiving.properties
./logback-tools.xml
./jvm.options

For tarball-based installations, the following command creates a backup of the Cassandra configuration files and places it in the user’s home directory:

Command
Result

cd <install-location>/conf && tar czfv ~/cassandra-config-backup.tgz .

./
./cassandra-topology.properties
./metrics-reporter-config-sample.yaml
./cassandra-env.sh
./jvm.options
./logback-tools.xml
./cassandra.yaml
./commitlog_archiving.properties
./README.txt
./cqlshrc.sample
./hotspot_compiler
./cassandra-env.ps1
./logback.xml
./cassandra-rackdc.properties
./triggers/
./triggers/README.txt
./cassandra-jaas.config

If you’re using Docker, you may need to perform this step before stopping the container and ensure the backup resides on a Docker volume that’s persisted. An alternative is to ensure that the container can be stopped and still reused if needed during a rollback.

Depending on your specific needs, you may consider backing up Cassandra log files.

Step 4: Install the new Cassandra package

It’s now time to install the new version of Cassandra. If you’re using a configuration management system, you should now "converge" the node with the new version of the manifest, cookbook, etc. If no configuration management is used, then you’ll need to replace the Cassandra binary or install the new package according to the instructions below.

If you’re using Docker, make sure to review the Docker considerations. This step is replaced by deploying the new Docker image with the upgraded version of Cassandra installed and configured for your environment.

You must use the same installation type — package or tarball — that was previously used on the node.

Packaged installations (APT and YUM) require root permissions and they install the binaries and configuration files as the cassandra OS user.

Debian/Ubuntu (APT)
CentOS/RHEL (YUM)
Tarball

Update the Apache Cassandra repository information in the cassandra.sources.list file.

The latest version is 4.1.2 and the corresponding distribution name is 41x (with an "x" as the suffix). To update the repository for version 4.1.2 (41x):
- Command
- Result
echo "deb https://debian.cassandra.apache.org 41x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
deb https://debian.cassandra.apache.org 41x main

Add the Apache Cassandra repository keys to the list of trusted keys on the server:

cURL
Wget
Result

curl https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -

wget https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  278k  100  278k    0     0   168k      0  0:00:01  0:00:01 --:--:--  168k
OK

Update the package index from sources:
```
sudo apt-get update
```
Upgrade Cassandra to the new version:
```
sudo apt-get install cassandra=4.1.2
```
If the new version of Cassandra contains updated configuration files that conflict with those that are currently installed, then you’ll receive a prompt for each conflicting file, asking you how you’d like to reconcile the differences.
```
Configuration file '/etc/cassandra/cassandra.yaml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** cassandra.yaml (Y/I/N/O/D/Z) [default=N] ?
```
It’s recommended that you install the new default configuration files rather than keep your currently-installed ones (especially when upgrading to a new major version of Cassandra). This ensures that all new configuration options are included and any deprecated options are removed.

Type Y and press Return to install the new configuration files.

In the next step, you’ll manually reconcile the differences between the new configuration files and your previous ones (the ones you backed up in Step 3: Back up Cassandra configuration files).

Update the Apache Cassandra repository information in the /etc/yum.repos.d/cassandra.repo file (as the root user).

The latest version is 4.1.2 and the corresponding distribution name is 41x (with an "x" as the suffix). To update the repository for version 4.1.2 (41x), make sure the content of cassandra.repo matches the following:

[cassandra]
name=Apache Cassandra
baseurl=https://redhat.cassandra.apache.org/41x/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://downloads.apache.org/cassandra/KEYS

If you haven’t yet created /etc/yum.repos.d/cassandra.repo, you can do so with the following command:

sudo bash -c "echo -e '[cassandra]\nname=Apache Cassandra\nbaseurl=https://redhat.cassandra.apache.org/41x/\ngpgcheck=1\nrepo_gpgcheck=1\ngpgkey=https://downloads.apache.org/cassandra/KEYS' > /etc/yum.repos.d/cassandra.repo"

Update the package index.

Command
Result

sudo yum update

Apache Cassandra                                817  B/s | 833  B     00:01
Apache Cassandra                                199 kB/s | 275 kB     00:01
Importing GPG key 0xF2833C93:
 Userid     : "Eric Evans <eevans@sym-link.com>"
 Fingerprint: CEC8 6BB4 A0BA 9D0F 9039 7CAE F835 8FA2 F283 3C93
 From       : https://downloads.apache.org/cassandra/KEYS
Is this ok [y/N]:

Type Y and press Return to import each of the GPG keys.

Upgrade Cassandra to the new version:

Command
Result

sudo yum install cassandra-4.1.2-1

================================================================================
 Package                      Arch     Version                 Repository  Size
================================================================================
Installing:
 cassandra                    noarch   4.1.2-1                 cassandra   48 M
Installing dependencies:
 ibus-gtk2                    aarch64  1.5.26-9.fc36           updates     27 k
 java-1.8.0-openjdk           aarch64  1:1.8.0.362.b09-1.fc36  updates    274 k
 java-1.8.0-openjdk-headless  aarch64  1:1.8.0.362.b09-1.fc36  updates     33 M
 mkfontscale                  aarch64  1.2.1-4.fc36            fedora      31 k
 ttmkfdir                     aarch64  3.0.9-65.fc36           fedora      52 k
 xorg-x11-fonts-Type1         noarch   7.5-33.fc36             fedora     500 k
Installing weak dependencies:
 adwaita-gtk2-theme           aarch64  3.28-14.fc36            fedora     136 k
 gtk2                         aarch64  2.24.33-8.fc36          fedora     3.4 M
 libcanberra-gtk2             aarch64  0.30-28.fc36            fedora      25 k

Transaction Summary
================================================================================
Install  10 Packages

Total download size: 85 M
Installed size: 193 M
Is this ok [y/N]:

Type Y and press Return to begin the installation.

After the packages have downloaded, you’ll be asked to import GPG keys. Type Y and press Return to import each of the GPG keys, after which the installation will continue until completion.

If you haven’t already, start by downloading the Cassandra binary tarball. For example, to download Cassandra 4.1.2:
- cURL
- Wget
curl -OL https://archive.apache.org/dist/cassandra/4.1.2/apache-cassandra-4.1.2-bin.tar.gz
wget https://archive.apache.org/dist/cassandra/4.1.2/apache-cassandra-4.1.2-bin.tar.gz
To download a different version of Cassandra, visit the Apache Archives.

(Optional) Verify the integrity of the downloaded tarball using one of the methods here.

For example, to verify the SHA256 hash of the downloaded file using GPG:

Command
Result

gpg --print-md SHA256 apache-cassandra-4.1.2-bin.tar.gz

apache-cassandra-4.1.2-bin.tar.gz: 7CE3103A 76B8AF76 FFD8488D 6BF484E1 F1751196
                                   17F3205A E0526C71 D816C6F7

Compare the output with the contents of the SHA256 file:

cURL
Wget
Result

curl -L https://archive.apache.org/dist/cassandra/4.1.2/apache-cassandra-4.1.2-bin.tar.gz.sha256

wget --quiet -O - https://archive.apache.org/dist/cassandra/4.1.2/apache-cassandra-4.1.2-bin.tar.gz.sha256

7ce3103a76b8af76ffd8488d6bf484e1f175119617f3205ae0526c71d816c6f7

Unpack the tarball:
```
tar xzf apache-cassandra-4.1.2-bin.tar.gz
```
The files will be extracted to the apache-cassandra-4.1.2 directory. This is the tarball installation location.

Located in the tarball installation location are the directories for the scripts, binaries, utilities, configuration, data, and log files. For a description of those directories, see Installing Cassandra.
Move the apache-cassandra-4.1.2 directory to the same location as your current installation of Cassandra. For example:
```
mv apache-cassandra-4.1.2 /usr/local/cassandra-4
```

Update your PATH and environment variables to point to the new installation. For example:

export PATH="/usr/bin:/usr/local/cassandra-4/bin:/usr/local/cassandra-4/tools/bin:$PATH"

Delete the tarball.
```
rm apache-cassandra-4.1.2-bin.tar.gz
```

Step 5: Update Cassandra configuration files

Before starting the newly-installed version of Cassandra, you need to update Cassandra’s configuration files, such as cassandra.yaml, to ensure your desired configuration is persisted.

The location of Cassandra’s configuration files varies, depending on the type of installation:

Installation type	Location
Package installations (APT, YUM, etc.)	/etc/cassandra
Tarball installations	<installation_location>/conf
Docker installations	/etc/cassandra

Installation type

Location

Package installations (APT, YUM, etc.)

/etc/cassandra

Tarball installations

<installation_location>/conf

Docker installations

/etc/cassandra

A reliable approach for updating Cassandra’s configuration files is to diff-compare the default configuration files installed by the new version of Cassandra against your previous configuration files (the ones you backed up in Step 3: Back up Cassandra configuration files).

General guidelines for diff-comparing the configuration files:

Start with the new default configuration files.

By starting with the new default configuration files, you ensure that all new and deprecated configuration options get included in your new configuration. This is especially important when upgrading Cassandra to a new major version, as there will likely be new configuration options and/or significant changes to existing options.
Diff-compare the new default configuration files against your previous configuration files.

This will reveal any new settings that were added in the upgraded version of Cassandra, and also reveal the settings that you’ve modified from their defaults in your previous configuration files.

See About configuration changes in Cassandra 4.x for details about configuration changes specific to Cassandra 4.x.
Update the new configurations files.
1. Review any new configuration options, especially if the options are enabled by default. It’s important that you have a firm understanding of how new configuration options may affect your Cassandra environment before continuing with the upgrade.
2. Migrate any settings or values that you modified in the previous configuration files to the new ones.
3. If settings you previously relied on have changed or have been removed in the new version Cassandra, you must alter your configuration accordingly.

About configuration changes in Cassandra 4.x

Several configuration files and parameters changed in Cassandra 4.x. Understanding the changes to the following files, in particular, is critical when upgrading to 4.x:

cassandra.yaml
cassandra-env.sh
jvm.options
- The jvm.options file is replaced in Cassandra 4.x by three files: jvm-server.options, jvm8-server.options, and jvm11-server.options.

For a full description of important configuration changes made in Cassandra 4.x, refer to Important configuration changes in Cassandra 4.x.

Server encryption considerations

If your cluster utilizes server-to-server internode encryption, you must set legacy_ssl_storage_port_enabled to true in the server_encryption_options section of cassandra.yaml. Otherwise, Cassandra 4.x will use the default port 7000 for both encrypted and unencrypted messages, which can cause communication issues with un-upgraded Cassandra 3.x nodes.

This parameter can be removed after all cluster nodes have been upgraded to Cassandra 4.x.

Step 6: Start the upgraded Cassandra service

Start the new Cassandra service on the node using the following command (or an equivalent on your system):

Package installations
Tarball installations

To start the Cassandra service for packaged installations:

sudo service cassandra start

To start the Cassandra process for tarball installations:

<install-location>/bin/cassandra

After a few seconds, the internal Cassandra processes will come online.

Step 7: Confirm Cassandra and application status

After starting the upgraded Cassandra service, check the status of the node while also continually monitoring your applications for signs of degraded performance and functionality.

Monitor /var/log/cassandra/system.log to confirm there are no ERROR or WARN statements.

Command
Result

sudo tail -n 50 -f /var/log/cassandra/system.log

Messages similar to the following will appear in system.log when the Cassandra process starts:

INFO  [main] 2023-06-01 04:44:45,897 SystemKeyspace.java:1729 - Detected version upgrade from 3.11.15 to 4.1.2, snapshotting system keyspaces
...
INFO  [main] 2023-06-01 05:06:00,630 StorageService.java:864 - Cassandra version: 4.1.2

Confirm that the node is reporting status UN (Up and Normal).

Command
Result

nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load        Tokens  Owns (effective)  Host ID                               Rack
UN  10.166.73.33   360.5 KiB   256     28.2%             7370b2ef-c9c3-4c83-bb44-0879cef0f6c8  rack1
UN  10.166.76.162  301.28 KiB  256     34.3%             e8f291db-b3ec-47fa-8bee-27e033c9655f  rack1
UN  10.166.77.78   132 KiB     256     37.4%             baae5ba3-a842-4b23-a70b-069364dac689  rack1

It’s important to check that the other nodes see the node as UP, and also that the node sees all other nodes as UP. Therefore, you should run nodetool status both on the node and on at least one other node. The output should be the same on both nodes.

Confirm that the node is using the intended version of Cassandra.
- Command
- Result
nodetool version
ReleaseVersion: 4.1.2

Confirm that the node is processing read and write traffic as well as requests from clients. You can confirm this by watching for Completed tasks in the thread pool stats:

Command
Result

watch -d nodetool tpstats

Every 2.0s: nodetool tpstats

Pool Name                    Active Pending Completed Blocked All time blocked
RequestResponseStage         0      0       3         0       0
ReadStage                    0      0       3         0       0
CompactionExecutor           0      0       6422      0       0
MemtableReclaimMemory        0      0       44        0       0
PendingRangeCalculator       0      0       8         0       0
GossipStage                  0      0       36429     0       0
SecondaryIndexManagement     0      0       1         0       0
HintsDispatcher              0      0       0         0       0
MigrationStage               0      0       34        0       0
MemtablePostFlush            0      0       60        0       0
PerDiskMemtableFlushWriter_0 0      0       33        0       0
ValidationExecutor           0      0       0         0       0
Sampler                      0      0       0         0       0
ViewBuildExecutor            0      0       0         0       0
MemtableFlushWriter          0      0       44        0       0
CacheCleanupExecutor         0      0       0         0       0
Native-Transport-Requests    0      0       0         0       0

Latencies waiting in queue (micros) per dropped message types
Message type                      Dropped     50%      95%      99%      Max
READ_RSP                          0           0.0      0.0      0.0      0.0
RANGE_REQ                         0           0.0      0.0      0.0      0.0
PING_REQ                          0           0.0      0.0      0.0      0.0
PAXOS2_COMMIT_REMOTE_RSP          0           0.0      0.0      0.0      0.0
PAXOS2_COMMIT_AND_PREPARE_RSP     0           0.0      0.0      0.0      0.0

Specifically, you should confirm that counters for the following tasks are increasing:

ReadStage: local read tasks
MutationStage: local writes tasks
RequestResponseStage: tasks that process the responses from replicas when acting as a coordinator

If all of these counters are increasing, it indicates that the node is successfully processing traffic and communicating properly with the rest of the cluster. There should be no pending, blocked, or dropped messages.

End of phase

At the end of this phase:

The first seed node is upgraded to the new version of Cassandra and is functioning normally.
Any applications connecting to the cluster are functioning normally.