Phase 2: Prepare the cluster
|
The upgrade process requires additional action when the cluster is running in Docker. If you’re using Docker, please review Docker considerations before beginning this phase. |
Step 1: Prepare configuration files and RPMs
The next step is to update your packaging system and configurations for the new version of Cassandra.
If you use a configuration management system (e.g. chef, puppet, ansible), ensure that the environment properties and settings have all been backed up and updated to provision the new version of Cassandra.
For more information, see Configuration management.
Step 2: Disable automated backups
Ensure that all automated backups have been disabled.
Automated backups may interfere with, or be interfered by, the upgrade process.
Therefore, you should disable tools like Medusa, as well as any scripts that call nodetool snapshot, until the upgrade is complete.
|
You’ll create a manual backup in Step 4: Create a pre-upgrade snapshot backup. |
Step 3: Disable automated node repairs
Ensure that all automated repairs have been disabled.
This includes disabling tools like Reaper, crontab, and any scripts that call nodetool repair.
A repair requires that all replicas for a token range be available before it can start. Once the repair is started, restarting a node will cause the process to fail. Automated repairs should be disabled because the upgrade requires a rolling restart and any scheduled repairs will most likely fail. Failing repairs on each node will generate a large amount of noise in the logs and cause unpredictable pressure during the upgrade process.
|
It’s important to repair nodes before upgrading Cassandra. See Node repair considerations for more information. |
Step 4: Create a pre-upgrade snapshot backup
On each node in the cluster, create a snapshot backup for all tables in all keyspaces. These snapshots are incredibly important should you later need to roll back the upgrade.
If you’re not using a tool like Reaper, you’ll need to find a method of taking a global snapshot of all nodes, such as running the nodetool snapshot command using a parallel SSH utility like pssh.
-
Command
-
Result
nodetool snapshot --tag pre-upgradeRequested creating snapshot(s) for [all keyspaces] with snapshot name [pre-upgrade] and options {skipFlush=false}
Snapshot directory: pre-upgradeThe above command creates a snapshot named pre-upgrade that includes all keyspaces on the node.
If you don’t want to include all keyspaces in a single snapshot, you can instead specify a list of specific keyspaces:
-
Command
-
Result
nodetool snapshot --tag pre-upgrade keyspace1 keyspace2 keyspace3Requested creating snapshot(s) for [keyspace1, keyspace2, keyspace3] with snapshot name [pre-upgrade] and options {skipFlush=false}
Snapshot directory: pre-upgradeBy default, the nodetool snapshot command flushes all in-memory writes to disk, then makes a hard link of the SSTable files for each keyspace.
If a keyspace contains no data, empty directories are not created.
|
After creating the snapshot, check to make sure that the node has at least 50% free disk space available. |
End of phase
At the end of this phase:
-
The package system on all nodes point to the correct binaries/package and configuration for the new version of Cassandra.
-
All automated backups have been disabled.
-
All automated repairs have been disabled.
-
All nodes have a pre-upgrade snapshot to serve as a rollback point.
