Upgrade to DSE 5.1 using the DataStax Installer

Improper upgrades can result in downtime, data loss, and other errors.

To minimize risk for an upgrade, review the entire upgrade guide before you begin, including warnings, restrictions, release notes, and all pre-upgrade, upgrade, and post-upgrade steps.

You must prepare all nodes in the cluster before upgrading any nodes. Failure to prepare all nodes can result in mismatched configurations, repair states, and schemas that can disrupt the upgrade process.

Follow these steps to upgrade an earlier versions of DataStax Enterprise (DSE) to DSE 5.1 using the DataStax Installer.

Upgrade path to avoid data loss

Support for the DataStax Installer ended with DSE 5.1.10. Due to several bugs in earlier versions of DSE 5.1, DataStax discourages use of the discontinued Installer for all upgrades to DSE 5.1.

Current version Issue Upgrade path

Current version	Issue	Upgrade path
DSE 5.1.10 or later	Installer stops at 5.1.10.	Convert to a package or tarball installation. Upgrade to the latest DSE 5.1 patch release.
DSE 5.1.7 to 5.1.9	DSE Tiered Storage data loss	A bug in versions 5.1.0 to 5.1.15, causes data loss if you don’t remove DSE Tiered Storage `txn_compaction` log files before starting DSE. Because the DataStax Installer doesn’t support upgrades past 5.1.10, this bug applies to all Installer-based upgrades to DSE 5.1. To avoid this issue and upgrade directly to an unaffected patch release of DSE 5.1: Convert to a package or tarball installation. Upgrade to the latest DSE 5.1 patch release.
DSE 5.1.0 to 5.1.6	Year 2038 problem and Tiered Storage data loss	TTL expiration timestamps are susceptible to the year 2038 problem. If the TTL value is long and an expiration date that is greater than the maximum threshold of `2038-01-19T03:14:06+00:00`, the data is immediately expired and purged on the next compaction. Although you can use the Installer to upgrade to DSE 5.1.7 through 5.1.10, these versions are also affected by the Tiered Storage data loss bug described in the previous row. To avoid data loss caused by both of these issues: Convert to a package or tarball installation. Upgrade to the latest DSE 5.1 patch release. Recover and reinsert rows with overflowed timestamps.
DSE 4.8 or 5.0 (any patch)	Year 2038 problem, Tiered Storage data loss, and limited Installer support	Several versions of DSE 4.8 and 5.0 are affected by the year 2023 problem. However, using the Installer to upgrade to the maximum supported DSE patch release (5.1.10) introduces the DSE Tiered Storage bug. To upgrade directly to an unaffected patch release of DSE 5.1: Convert to a package or tarball installation. Upgrade from DSE 4.8 to 5.0. Upgrade from DSE 5.0 to 5.1. The DSE 5.0 and 5.1 upgrade guides include troubleshooting for the Year 2038 problem if you are currently on an affected version.

DSE 5.1.10 or later

Installer stops at 5.1.10.

DSE 5.1.7 to 5.1.9

DSE Tiered Storage data loss

A bug in versions 5.1.0 to 5.1.15, causes data loss if you don’t remove DSE Tiered Storage txn_compaction log files before starting DSE. Because the DataStax Installer doesn’t support upgrades past 5.1.10, this bug applies to all Installer-based upgrades to DSE 5.1. To avoid this issue and upgrade directly to an unaffected patch release of DSE 5.1:

DSE 5.1.0 to 5.1.6

Year 2038 problem and Tiered Storage data loss

TTL expiration timestamps are susceptible to the year 2038 problem. If the TTL value is long and an expiration date that is greater than the maximum threshold of 2038-01-19T03:14:06+00:00, the data is immediately expired and purged on the next compaction.

Although you can use the Installer to upgrade to DSE 5.1.7 through 5.1.10, these versions are also affected by the Tiered Storage data loss bug described in the previous row. To avoid data loss caused by both of these issues:

DSE 4.8 or 5.0 (any patch)

Year 2038 problem, Tiered Storage data loss, and limited Installer support

Several versions of DSE 4.8 and 5.0 are affected by the year 2023 problem. However, using the Installer to upgrade to the maximum supported DSE patch release (5.1.10) introduces the DSE Tiered Storage bug. To upgrade directly to an unaffected patch release of DSE 5.1:

The DSE 5.0 and 5.1 upgrade guides include troubleshooting for the Year 2038 problem if you are currently on an affected version.

Use the steps in this guide only if you choose to use the discontinued Installer for your upgrade and you understand the risks and limitations associated with this upgrade path.

Avoid certain operations during the upgrade

The upgrade process for DSE can be completed with minimal downtime (ideally zero) by upgrading nodes in a rolling fashion: upgrade and restart one node at a time while the other nodes continue to operate.

The moment that you install the new version on one node in a cluster, the cluster enters a partially upgraded state. In this state, the cluster continues to operate as though all nodes are running the earlier, pre-upgraded version. The cluster remains in this state until you have upgraded all nodes in the cluster.

You must avoid or block the following operations during the upgrade to maintain cluster stability and prevent errors:

Don’t enable new features.

This restriction includes net new features and existing features that you weren’t using previously. For example, don’t enable Change Data Capture (CDC) for the first time during an upgrade. Upgrade the entire cluster, and then enable the feature.
Don’t change credentials, permissions, or any other security settings unless explicitly instructed to do so.

To preserve your existing configuration, you will transfer your existing security settings from your current configuration files to the new configuration files during the upgrade process. The new files might use different settings due to enhancements, deprecations, and removals between versions. This mapping is required for a successful upgrade, and it isn’t the same as an extraneous change to the security configuration.

Examples of security changes to avoid include modifications to authenticators, authorizers, CQL roles, and CQL permissions. Postpone all such changes until after the upgrade.

If you aren’t already using Kerberos, don’t set up Kerberos authentication immediately before upgrading. First upgrade the cluster, and then set up Kerberos. If you are already using Kerberos, you can continue to use it during the upgrade.
Don’t run repairs manually or automatically:
- Disable all automated and scheduled repairs, including Reaper, crontab, and scripts that run nodetool repair.
- Don’t manually run commands like nodetool repair.
- Don’t repair SSTables while the cluster is in a partially upgraded state. Only repair SSTables before and after upgrading the entire cluster.
  
  If enabled, stop the DSE OpsCenter Repair Service.
You must complete the cluster-wide upgrade before the expiration of gc_grace_seconds (approximately 10 days by default) to ensure any post-upgrade repairs complete successfully.

Attempting to run repairs on partially upgraded clusters can cause performance issues, data inconsistencies, and repair failures. If you absolutely must run repairs on a partially upgraded cluster, determine which SSTable formats and streaming protocols are supported on each node, and then thoroughly test your intended repair operation on an isolated non-production cluster. Small incremental repairs are less risky but they aren’t risk free.
Don’t add (bootstrap) new nodes or decommission existing nodes.
Don’t issue DDL-related queries like TRUNCATE.

Depending on your infrastructure, workloads, and consistency levels, you might want to avoid other operations that could require excessive resources or coordination across mixed-version nodes. For example, BATCH operations can be resource intensive.
Don’t alter any schemas.

Schema versions must be consistent across the cluster before starting the upgrade.

Propagation of schema changes between mixed-version nodes can cause query failures, data consistency issues, and other errors. Take action to prevent schema changes from occurring during the upgrade process.

During the upgrade, it is normal for nodes on different versions to show schema disagreements. These disagreements should resolve after the entire cluster is upgraded.
Don’t rebuild indexes during the upgrade.
Disable services that can cause data inconsistencies and interfere with the upgrade process:
- Disable backup services, scheduled backups, and automated snapshots.
If you disabled the Performance Service before the upgrade, don’t reenable it during the upgrade.
DSE Analytics nodes: Don’t run analytics jobs until all nodes are upgraded.

When upgrading to a new major or minor version of DSE, all nodes in a DSE datacenter that run Spark must be on the same version of Spark, and the Spark jobs must be compiled for that version. To determine the Spark version, see the DSE release notes.

Before reinitiating Spark jobs, all datacenters comprising a Spark cluster must be on the same upgraded DSE version. If any Spark jobs run against Graph keyspaces, don’t reinitiate Spark jobs until you have upgraded all Analytics and non-Analytics nodes in the entire cluster. When Graph keyspaces are involved, Spark jobs fail due to mixed Spark and DSE versions across all nodes.

Prepare to upgrade with the DataStax Installer

Follow these steps to prepare each node for the upgrade:

Upgrade to the latest patch release for your current version.

Always upgrade to latest patch release for your current version before you upgrade to a new major or minor version. Patch releases help prepare your clusters for the upgrade by minimizing the differences between the current version and the target version.

Get your current DSE version with dse -v, and then compare it with the latest version available for your DSE release series. You can find the latest version in the DSE release notes.

If you need to apply a patch release, upgrade all nodes to the latest patch release before applying any further upgrades. Treat patch releases as you would any major or minor upgrade by carefully reviewing the release notes and following the recommended upgrade procedure.
Familiarize yourself with the changes and features in the new release, reviewing all releases between your current version and the latest 5.1 patch release as well as Apache Cassandra® changes:
Check Java runtime compatibility and install a mutually supported Java runtime version if needed.

For upgrades to DSE 5.1, install the latest build of OpenJDK 8 (recommended) or Oracle Java SE 8 (JRE or JDK) (supported). OpenJDK is recommended because DataStax does more extensive testing on OpenJDK.
1. Get your current Java version:
  
  java -version
2. Make sure the output shows a supported Java runtime. If not, install a supported Java runtime.
  
  For example, if OpenJDK 8 is installed:
  
  openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
3. If your environment has multiple Java versions installed, set the $JAVA_HOME environment variable to Java 8.
4. If you changed $JAVA_HOME, restart the node.

Use the DataStax Installer to upgrade DSE on Linux

Contact IBM Support to download the Installer for your target major and minor version, such as DataStaxEnterprise-5.1.10-linux-x64-installer.run.
Change to the directory where you downloaded the Installer, and then make it executable:
```
chmod +x DataStaxEnterprise-5.1.10-linux-x64-installer.run
```

Run the Installer with sudo:

sudo ./DataStaxEnterprise-5.1.10-linux-x64-installer.run

Follow the instructions in the setup wizard.

For more information about the settings, see the DSE 5.1 installation instructions.
For DSE Tiered Storage tables, you must remove all txn_compaction log files from second-level tiers and lower.

Data loss will occur if you don’t remove the txn_compaction log files before starting DSE.

You can find the tiers and directories in the dse.yaml configuration file. For example, given the following tiered_storage_options configuration, you would remove txn_compaction log files from the /mnt2 and /mnt3 directories:
```
tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3
```
The following example removes the files from the second and third tier directories using find commands:
```
find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete
```
After completing the install wizard process and removing txn_compaction files (if needed), start DSE:
```
sudo service dse start
```
If you upgraded from DSE 4.7 or 4.8, start the DataStax Agent on the node:
```
sudo service datastax-agent start
```
In DSE 5.1, use DSE OpsCenter to manage agents.
Verify that DSE is running:
```
nodetool status
```
Repeat for all nodes in the cluster.

Post-upgrade steps

Restart any services or automated operations that you disabled before starting the upgrade.

For more information, see Avoid certain operations during the upgrade.
Edit any scripts that use DSE file paths to the new file paths in DSE 5.1.

For default file locations in DSE 5.1, see Default file locations for package and Installer-Services installations and Default file locations for tarball and Installer-No Services installations.