Upgrade DSE 4.7 or 4.8 to 5.0

Improper upgrades can result in downtime, data loss, and other errors.

To minimize risk for an upgrade, review the entire upgrade guide before you begin, including warnings, restrictions, release notes, and all pre-upgrade, upgrade, and post-upgrade steps.

You must prepare all nodes in the cluster before upgrading any nodes. Failure to prepare all nodes can result in mismatched configurations, repair states, and schemas that can disrupt the upgrade process.

Follow these instructions to manually upgrade an earlier version of DataStax Enterprise (DSE) to DSE 5.0.

Upgrade path to avoid data loss

TTL expiration timestamps are susceptible to the year 2038 problem. If the TTL value is long and an expiration date that is greater than the maximum threshold of 2038-01-19T03:14:06+00:00, the data is immediately expired and purged on the next compaction. To protect against silent data loss, upgrade to DSE 5.0.15 or later and recover and reinsert rows with overflowed timestamps.

DSE 4.7, DSE 4.8, and DataStax Community 3.0.x can upgrade directly to DSE 5.0.

If you are on an earlier version of DSE, upgrade to DSE 4.7 or 4.8 before proceeding to 5.0.

Upgrades from the DataStax Distribution of Apache Cassandra® 3.x aren’t supported. For assistance, contact IBM Support.

Avoid certain operations during the upgrade

The upgrade process for DSE can be completed with minimal downtime (ideally zero) by upgrading nodes in a rolling fashion: upgrade and restart one node at a time while the other nodes continue to operate.

The moment that you install the new version on one node in a cluster, the cluster enters a partially upgraded state. In this state, the cluster continues to operate as though all nodes are running the earlier, pre-upgraded version. The cluster remains in this state until you have upgraded all nodes in the cluster.

You must avoid or block the following operations during the upgrade to maintain cluster stability and prevent errors:

Don’t enable new features.

This restriction includes net new features and existing features that you weren’t using previously. For example, don’t enable Change Data Capture (CDC) for the first time during an upgrade. Upgrade the entire cluster, and then enable the feature.
Don’t change credentials, permissions, or any other security settings unless explicitly instructed to do so.

To preserve your existing configuration, you will transfer your existing security settings from your current configuration files to the new configuration files during the upgrade process. The new files might use different settings due to enhancements, deprecations, and removals between versions. This mapping is required for a successful upgrade, and it isn’t the same as an extraneous change to the security configuration.

Examples of security changes to avoid include modifications to authenticators, authorizers, CQL roles, and CQL permissions. Postpone all such changes until after the upgrade.

If you aren’t already using Kerberos, don’t set up Kerberos authentication immediately before upgrading. First upgrade the cluster, and then set up Kerberos. If you are already using Kerberos, you can continue to use it during the upgrade.
Don’t run repairs manually or automatically:
- Disable all automated and scheduled repairs, including Reaper, crontab, and scripts that run nodetool repair.
- Don’t manually run commands like nodetool repair.
- Don’t repair SSTables while the cluster is in a partially upgraded state. Only repair SSTables before and after upgrading the entire cluster.
  
  If enabled, stop the DSE OpsCenter Repair Service.
You must complete the cluster-wide upgrade before the expiration of gc_grace_seconds (approximately 10 days by default) to ensure any post-upgrade repairs complete successfully.

Attempting to run repairs on partially upgraded clusters can cause performance issues, data inconsistencies, and repair failures. If you absolutely must run repairs on a partially upgraded cluster, determine which SSTable formats and streaming protocols are supported on each node, and then thoroughly test your intended repair operation on an isolated non-production cluster. Small incremental repairs are less risky but they aren’t risk free.
Don’t add (bootstrap) new nodes or decommission existing nodes.
Don’t issue DDL-related queries like TRUNCATE.

Depending on your infrastructure, workloads, and consistency levels, you might want to avoid other operations that could require excessive resources or coordination across mixed-version nodes. For example, BATCH operations can be resource intensive.
Don’t alter any schemas.

Schema versions must be consistent across the cluster before starting the upgrade.

Propagation of schema changes between mixed-version nodes can cause query failures, data consistency issues, and other errors. Take action to prevent schema changes from occurring during the upgrade process.

During the upgrade, it is normal for nodes on different versions to show schema disagreements. These disagreements should resolve after the entire cluster is upgraded.
Don’t rebuild indexes during the upgrade.
Disable services that can cause data inconsistencies and interfere with the upgrade process:
- Disable backup services, scheduled backups, and automated snapshots.
If you disabled the Performance Service before the upgrade, don’t reenable it during the upgrade.
DSE Analytics nodes: Don’t run analytics jobs until all nodes are upgraded.

When upgrading to a new major or minor version of DSE, all nodes in a DSE datacenter that run Spark must be on the same version of Spark, and the Spark jobs must be compiled for that version. To determine the Spark version, see the DSE release notes.

Before reinitiating Spark jobs, all datacenters comprising a Spark cluster must be on the same upgraded DSE version. If any Spark jobs run against Graph keyspaces, don’t reinitiate Spark jobs until you have upgraded all Analytics and non-Analytics nodes in the entire cluster. When Graph keyspaces are involved, Spark jobs fail due to mixed Spark and DSE versions across all nodes.

Driver compatibility and live application traffic

Incompatible drivers and unspecified protocol versions can cause connection failures and other issues during the upgrade.

To maintain live application traffic throughout the upgrade process, your Cassandra driver must be compatible with your current installation and your target DSE version. Additionally, you must configure protocol version negotiation to ensure backwards compatibility throughout the upgrade.

Find a mutually compatible driver version

Verify that your current Cassandra driver version is compatible with your target DSE version. For available drivers and compatibility information, see Cassandra driver compatibility.

If your current driver version isn’t compatible with your target DSE version, upgrade your driver to a mutually compatible version.

When upgrading your driver, check your driver’s release notes for breaking changes that might require code changes. Review all release notes from your current driver version through the new version. Recompile your client application code, if needed.

No mutually compatible version exists

If there is no mutually compatible driver version, you have two options:

Intermediate upgrade (recommended): Determine if your upgrade path requires (or supports) an intermediate upgrade before proceeding to your target DSE version. Find driver versions that are compatible with each segment of the upgrade path (current to intermediate and intermediate to target), and then upgrade your driver before each segment.
Delayed upgrade: Upgrade to the latest driver version that is compatible with your current installation, and then plan to upgrade your driver again when your cluster is on your target DSE version. This can require code or configuration changes to accommodate the temporary driver version, such as changing authentication methods or avoiding unsupported query patterns. Consider testing this approach in an isolated environment to identify potential compatibility issues before applying it to your production environment.

Configure protocol negotiation

When a cluster is in a partially upgraded state, upgraded nodes might support different Cassandra native protocol versions than non-upgraded nodes. By default, a driver negotiates the protocol version with the first host (node) that it connects to. The driver might select a protocol version that works across all nodes, but this isn’t guaranteed if the driver connects to a random node each time. It is better to prevent protocol incompatibility in one of two ways:

Set protocol version: Set a mutually compatible protocol version explicitly in your application at start up. After you upgrade all nodes in the cluster, you can switch your applications to the latest supported protocol version.
Set initial contact points: Specify a list of initial contact points that contains only hosts with the oldest database platform version or mutually compatible protocol version. You must upgrade these nodes last.

For more information about protocol version negotiation, see your driver’s documentation.

DSE-only drivers are unsupported

All DSE-only drivers have reached end-of-life (EOL); they are unsupported and unmaintained. If your applications use DSE-only drivers, you must upgrade to a recent version of a supported driver.

For legacy applications that require DSE-only drivers, compatibility with your target DSE version might be limited or nonexistent. For example, newer DSE, Cassandra, and CQL features might not be supported by the driver, or queries that use deprecated data types, functions, or syntax might fail. Consider modernizing or sunsetting legacy applications if possible, particularly because EOL drivers don’t receive any updates, including security updates.

Although the DSE-only drivers are compatible with legacy applications that require DSE 5.0, if you intend to upgrade past DSE 5.0, you will need to use a newer driver eventually.

Strict upgrade order for cluster stability

Don’t upgrade any nodes until you have completed the pre-upgrade steps on all nodes.

When upgrading nodes, always upgrade seed nodes before non-seed nodes.

To avoid cluster-wide downtime, you must upgrade and restart one node at a time in a specific order based on workload types and deployment architecture:

For multi-cluster deployments, upgrade one cluster at a time.
For multi-datacenter clusters, upgrade one datacenter at a time.
If using racks, upgrade one rack at a time within a datacenter.
Within a datacenter (or rack), upgrade one node at a time, starting with seed nodes.
If using advanced workloads (Analytics, Graph, Search), upgrade datacenters or nodes according to the following workload type hierarchy.

If each datacenter has only one workload type, upgrade your datacenters in the given order. Within each datacenter, proceed rack-by-rack (if applicable) and node-by-node, starting with the seed nodes.

If a datacenter has multiple workload types, upgrade nodes within that datacenter in the given order, starting with the seed nodes. If using racks, apply the hierarchy to each rack.

The workload type hierarchy is:
1. Upgrade DSE Analytics datacenters:
  
  Upgrade the DSE Hadoop Job Tracker node.
  
  Upgrade Hadoop nodes.
  
  Upgrade Spark nodes.
2. Upgrade transactional (Cassandra) and DSE Graph datacenters.
3. Upgrade DSE Search datacenters.
Repeat until you have upgraded all nodes in all racks, datacenters, and clusters.

Use an isolated deployment to test the upgrade.

For example, in multi-datacenter clusters with an active-passive configuration, you can complete the entire upgrade process (pre-upgrade, upgrade, and post-upgrade) on the passive datacenter first, run performance tests on that datacenter, and then upgrade the active datacenter if the tests are successful. If the tests fail, you can destroy the passive datacenter and rebuild it without impacting the active datacenter.

Similarly, if you have a test deployment with no connectivity to your production nodes, datacenters, or clusters, then you can safely perform the entire upgrade process in that isolated deployment before rolling out the upgrade to your production deployment.

Prepare to upgrade

Before upgrading any node, you must prepare all nodes for the upgrade. Complete the pre-upgrade steps on every node in the recommended order. Following the recommended order ensures that you don’t miss any nodes, although some steps aren’t applicable to every node.

Some pre-upgrade steps provide information about actions that you must take later in the upgrade process. This information is given in advance because it might impact your upgrade plan. Be sure to revisit these steps at the appropriate time.

Upgrade to the latest patch release for your current version.

Always upgrade to latest patch release for your current version before you upgrade to a new major or minor version. Patch releases help prepare your clusters for the upgrade by minimizing the differences between the current version and the target version.

Get your current DSE version with dse -v, and then compare it with the latest version available for your DSE release series. You can find the latest version in the DSE release notes.

If you need to apply a patch release, upgrade all nodes to the latest patch release before applying any further upgrades. Treat patch releases as you would any major or minor upgrade by carefully reviewing the release notes and following the recommended upgrade procedure. For specific instructions, see apply a DSE 4.8 patch release.
Familiarize yourself with the changes and features in the new release, reviewing all releases between your current version and the latest 5.0 patch release as well as Apache Cassandra changes.

The Cassandra version included with DSE changed in DSE 5.0 to Cassandra 3.0.
Check Java runtime compatibility and install a mutually supported Java runtime version if needed.

For upgrades from DSE 4.x to DSE 5.0, install the latest build of Oracle Java SE JDK 8 (1.8.0_40 minimum). This is the minimum mutually compatible Java runtime for DSE 4.x and 5.0. After you upgrade your clusters to DSE 5.0, you can install your preferred Java 8 runtime and set $JAVA_HOME accordingly.
1. Get your current Java version:
  
  java -version
2. Make sure the output shows Oracle Java SE JDK 8. If not, install it.
  
  For example:
  
  java version "1.8.0_40" Java(TM) SE Runtime Environment (build 1.8.0_40-b25) Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
3. If your environment has multiple Java versions installed, set the $JAVA_HOME environment variable to Java 8.
4. If you changed $JAVA_HOME, restart the node.
Make sure each node has adequate free disk space for the upgrade.

The required overhead depends on the compaction strategy. For more information, see Disk space.
1. Get the size of the DSE data directory:
  sudo du -sh /var/lib/cassandra/data/
2. Check available space on disk:
  sudo df -hT /
  The following output includes one disk named /dev/sda1. Your output depends on your disk configuration.
  Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 59G 16G 41G 28% /
3. If necessary, make adjustments to your disks or data directory configuration to allow for more disk space.
Replace SELECT * queries using prepared statements with SELECT statements using the relevant columns.

If your applications must use SELECT * prepared statements, DataStax strongly recommends that you continue your upgrade to DSE 6.x and upgrade to the latest version of a compatible driver before reinstating this logic. However, SELECT * statements are considered an anti-pattern and generally discouraged.

Upgrade all SSTables on all nodes.

To avoid significant performance degradation and issues with incompatible SSTables formats, always upgrade SSTables before and after any database upgrade, including minor and patch releases.

This process is recommended for all upgrades, and it is critical for upgrades that involve a major underlying Cassandra version change.

On each node, run nodetool upgradesstables with the -a flag to ensure that all SSTables use the latest SSTable format and version for your current database version:

nodetool upgradesstables -a

DataStax recommends running the upgradesstables command on one node at a time or, when using racks, one rack at a time. Running the command on too many nodes at once can degrade performance.

You can use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

If all SSTables are already on the latest version and format, the command returns immediately and takes no action.

Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:
```
nodetool repair -pr
```
Repair nodes on a regular basis to minimize the overhead and time required for repair operations.

If you use DSE OpsCenter, you can repair nodes with the Repair Service but you must disable the Repair Service before upgrading any nodes.

Back up all DSE data and files, and then store the backups separately from your existing DSE installation.

Backup your existing installation before every upgrade, regardless of the extent of the upgrade.

To avoid data loss, you must store your backup configuration files in a separate directory from your DSE installation. The new version overwrites configuration files that are present in the installation directory, replacing all existing values with new default values.

Examples of data and files to back up include:

SSTable snapshots
Commit logs
Logs
Configuration files
Encrypted table recovery data:
- A snapshot of the entire dse_system keyspace with the encrypted_keys table
- All system keys

Some backup files are used during the upgrade process to align the new version with the previous version, and a complete backup is required if you need to rollback to the previous version or restore lost data.

If your upgrade requires significant pre-upgrade changes, consider creating a backup before making those changes and again immediately before upgrading the first node. Create the final backup as close to the beginning of your upgrade as possible to ensure your backup contains the most recent configuration files and data.

For instructions, see Back up and restore a DSE package installation or Back up and restore a DSE tarball installation.

Additional preparation for DSE Search nodes

Complete these additional pre-upgrade steps on DSE Search nodes while DSE 4.7 or 4.8 is still running.

DataStax recommends running nodetool repair and creating a fresh backup after making these changes, even if you already did so during the pre-upgrade steps.

Lucene field cache deprecated

The Lucene field cache (solr_field_cache_enabled) file is deprecated. This field was located in the dse.yaml file.

For fields that are sorted, faceted, or grouped by: Set docValues="true" on the field in the schema.xml file, and then reload the core and reindex.

During mixed versions upgrades, you can re-enable the field cache (solr_field_cache_enabled: true) to allow running queries but not reindexing. The default value is false. To override false, set useFieldCache=true in the Solr request.

Unique key fields must be indexed

All unique key elements must be indexed in the Solr schema. In schema.xml, make sure all unique key fields have indexed=true.

If you change the schema, do a full reindex:

dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=false

HTTP-based Solr shard transport option deprecated

Starting in DSE 5.0, shard_transport_options is deprecated in favor of internode_messaging_options, and all internode messaging requests use this internal messaging service. These options are configured in dse.yaml.

While a cluster is in a partially upgraded state, DSE runs two different servers for backward compatibility: one server uses shard_transport_options, and the other server uses internode_messaging_options. After all nodes are upgraded to 5.0, only internode_messaging_options are used.

If you plan to continue your upgrade to DSE 5.1, be aware that the deprecated HTTP options are removed in 5.1.

DSE Search partition key names

If you have COMPACT STORAGE tables that are backed by DSE Search indexes, the partition key names must match the uniqueKey in schema.xml.

For example, given the following table schema:

CREATE TABLE keyspace_name.table_name (key text PRIMARY KEY, foo text, solr_query text)
WITH COMPACT STORAGE

If the Solr schema.xml is <uniqueKey>id</uniqueKey>, then rename the key in the table to match the Solr schema:

ALTER TABLE ks.table RENAME key TO id;

Tune the schema before you upgrade

Starting in DSE 5.0.10, all field definitions in the schema are validated and must be compatible with DSE Search. This requirement applies to all fields, including fields with docValues applied, fields used as a copy-field source, and fields that aren’t indexed.

By default, automatic resource generation includes all columns. To improve performance, take action to prevent fields from being loaded from the database. For example, make sure your schema includes only required fields by removing or commenting out unused fields.

If you change the schema, reload the Solr core.

Upgrade steps

Upgrade one node at a time in the recommended order. Following the recommended order minimizes downtime and ensures that you don’t miss any nodes.

DSE Analytics nodes: Kill all Spark worker processes.
Flush the commit log of the current installation:
```
nodetool -h hostname drain
```
This step saves time when nodes start up after the upgrade and prevents DSE Search nodes from having to reindex data.

This step is mandatory for upgrades that include a major Cassandra version change because the previous version’s commit logs are incompatible with the new version’s SSTable format.
Stop the node.
Install DSE 5.0, but don’t start DSE.

Don’t start DSE yet. Before restarting the node, you must reconfigure DSE as explained in the next steps.

Use the same installation type as your current version. Mismatched installation types can cause problems with the upgrade.

For RHEL-based systems with DSE demos installed, you must specify the package installation in a single line, and specify the version for dse-full and dse-demos. For example:
```
sudo yum install dse-full-5.0.15-1 dse-demos-5.0.15-1
```

Configure the new DSE version by comparing the new configuration files to your backup configuration files.

Patch releases are unlikely to introduce significant changes, but you must ensure that your existing settings are transferred to the new version. The extent of the changes depends on your workloads, previous configurations, and the number of patch releases between your current and new versions. For example:

Remove deprecated and removed settings.
Configure new settings.
Migrate previous custom configuration values to your new configuration files.
Modify previous configurations to account for changes described in the DSE release notes.

You must use the new configuration files included with your new installation. Copy values from your old configuration files into the new files. Don’t overwrite entire files with the old files.

You can use the yaml_diff tool to compare your backup YAML files with the new YAML files:

cd /usr/share/dse/tools/yamls

./yaml_diff path/to/yaml-file-old path/to/yaml-file-new

The output is a diff between the specified YAML files. For example:

...
 CHANGES
=========
authenticator:
- AllowAllAuthenticator
+ com.datastax.bdp.cassandra.auth.DseAuthenticator

authorizer:
- AllowAllAuthorizer
+ com.datastax.bdp.cassandra.auth.DseAuthorizer

roles_validity_in_ms:
- 2000
+ 120000
...

If the cluster will run Hadoop in a Kerberos secure environment, change the task-controller file ownership to root and access permissions to 4750.

For example, using a DSE Installer-Services or package installation:
```
sudo chown root /usr/share/dse/resources/hadoop/native/Linux-amd64-64/bin/task-controller
sudo chmod 4750 /usr/share/dse/resources/hadoop/native/Linux-amd64-64/bin/task-controller
```
DSE Analytics nodes: Manually set the Hive metastore configuration if you need a cluster to continue processing DSE Analytics workloads during the upgrade.

Prior to DSE 5.0, Spark used the Hive metastore table HiveMetaStore.MetaStore. Starting in DSE 5.0, the Hive and Spark metastore tables are separate, and Spark uses the HiveMetaStore.sparkmetastore table.

If the metastore table is missing when you start a DSE 5.0 node, then the node waits for the entire cluster to be upgraded before starting Spark because it has to create the metastore table first. While the cluster is in a partially upgraded state, the node cannot create the metastore table because, by design, this table cannot exist on non-upgraded nodes.

To allow Spark nodes to create the metastore table and elect a Master in a mixed-version cluster, you can manually update the Hive metastore configuration. This allows Analytics to restart and to continue serving requests while you upgrade the other nodes in that cluster.

If you don’t make this change, then the Spark Master is elected automatically after the entire datacenter is upgraded to DSE 5.0. Analytics nodes won’t start until the entire cluster is upgraded.

To manually update the Hive metastore configuration, do the following:
1. Set the warehouse path in hive-site.xml:
  
  sudo perl -i -pe 's|cfs:///user/spark/warehouse|cfs:///user/hive/warehouse|g' /etc/dse/spark/hive-site.xml
  
  For package installations, replace /etc/dse/spark/hive-site.xml with /usr/local/lib/dse/resources/spark/conf/hive-site.xml.
2. Set the name of the Spark metastore table in hive-site.xml:
  
  sudo perl -i -pe 's|sparkmetastore|MetaStore|g' /etc/dse/spark/hive-site.xml
  
  For package installations, replace /etc/dse/spark/hive-site.xml with /usr/local/lib/dse/resources/spark/conf/hive-site.xml.
3. If needed, edit the contact points in your client applications for compatibility with the Hive metastore configuration change.
  
  This choice depends on the data that you want applications to access during the upgrade.
  
  If the Spark contact point is set to a DSE 5.0 node, it can use only DSE 5.0 replicas to access data.
  
  If the contact point is set to a DSE 4.8 node, it is able to access data on all replicas in the cluster.
4. After all nodes are upgraded, you can revert the manual configuration changes and allow nodes to elect a Master and use contact points in the usual way.
Ensure that keyspace replication factors are correct for your environment:
1. Set the keyspace replication factor for analytics keyspaces.
2. Set the keyspace replication factor for system_auth and dse_security keyspaces.
Start the node:
- Installer-Services and Package installations: Start DSE as a service
- Installer-No Services and Tarball installations: Start DSE as a standalone process
Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition.

This is only relevant when using NetworkTopologyStrategy.
1. Get the node’s datacenter name:
  nodetool status | grep "Datacenter"
2. Get the replication configuration for a keyspace:
  cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"
3. If the datacenter name in replication does not match the node’s datacenter name, alter or recreate the keyspace accordingly:
  ALTER KEYSPACE keyspace-name WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter-name': '3'};
Review the logs for warnings, errors, and exceptions:
```
grep -w 'WARN\|ERROR\|exception' /var/log/cassandra/*.log
```
If the default log location returns no results, check dse-env.sh for custom log locations.

Warnings, errors, and exceptions are frequently found in the logs when starting an upgraded node. Some messages provide information to help you complete specific upgrade steps, and some messages can be ignored while the cluster is in a partially upgraded state. If you find warnings, errors, or exceptions that are unexpected or persist after upgrading the entire cluster, contact IBM Support.

If you made schema changes shortly before upgrading to DSE 5.0, you can ignore warnings related to commitlog replays. For example:
```
WARN [main] 2016-06-23 12:01:59,693 CommitLogReplayer.java:154 - Skipped 31 mutations from unknown (probably removed) CF with id b0f22357-4458-3cdb-9631-c43e59ce3676
WARN [main] 2016-06-23 12:01:59,693 CommitLogReplayer.java:154 - Skipped 1 mutations from unknown (probably removed) CF with id 3aa75225-4f82-350b-8d5c-430fa221fa0a
WARN [main] 2016-06-23 12:01:59,696 CommitLogReplayer.java:154 - Skipped 1 mutations from unknown (probably removed) CF with id 45f5b360-24bc-3f83-a363-1034ea4fa697
WARN [main] 2016-06-23 12:01:59,696 CommitLogReplayer.java:154 - Skipped 1 mutations from unknown (probably removed) CF with id 0359bc71-7123-3ee1-9a4a-b9dfb11fc125
WARN [main] 2016-06-23 12:01:59,697 CommitLogReplayer.java:154 - Skipped 1 mutations from unknown (probably removed) CF with id 296e9c04-9bec-3085-827d-c17d3df2122a
```
For DSE Analytics nodes, Task Tracker exceptions are expected from nodes that aren’t upgraded to DSE 5.0 yet. The jobs should succeed after the entire cluster is upgraded.

Because DSE 5.0 uses Cassandra 3.0, you can ignore warnings in the output.log file about the following options:
- sstable_compression
- chunk_length_kb
- memory_allocator
- memtable_allocation_type
- offheap_objects
- netty_server_port
The netty_server_port option is used only during the upgrade to 5.0. After all nodes are upgraded to 5.0, coordinator nodes no longer contact other nodes on this port when handling requests. Instead, requests use internode_messaging_options that are used by several DSE components. For 5.0 and later, all internode messaging requests use this service.
Repeat the upgrade process on each node in the cluster following the recommended order.

Post-upgrade steps

The post-upgrade steps include multi-node and cross-datacenter operations that can cause errors while a cluster is in a partially upgraded state. For example, repairs can propagate incompatible data between mixed-version nodes, and SSTable snapshots can be corrupted or unreadable due to mixed SSTable formats and versions.

Only start the post-upgrade steps after you upgrade all dependent nodes. Node dependencies are determined by your deployment architecture:

Architecture	Examples	Post-upgrade path
Isolated or non-production deployment	Upgrading a test deployment before rolling out the upgrade to production. Upgrading a passive datacenter in an active-passive multi-datacenter deployment.	Start the post-upgrade steps after upgrading all nodes in the test deployment or passive datacenter. For example, in a test deployment with two datacenters and cross-datacenter replication, you must upgrade all nodes in both datacenters before starting the post-upgrade steps. In contrast, in a test deployment with one datacenter and no cross-datacenter connections, you must upgrade all nodes in that datacenter before starting the post-upgrade steps.
Production deployment	Rolling out an upgrade to an entire production cluster. Upgrading a multi-datacenter deployment that has cross-datacenter replication.	Start the post-upgrade steps after upgrading all nodes included in the deployment. For example, in a multi-datacenter deployment with cross-datacenter replication, you must upgrade all nodes in all datacenters before starting the post-upgrade steps.

Architecture

Examples

Post-upgrade path

Isolated or non-production deployment

Upgrading a test deployment before rolling out the upgrade to production.
Upgrading a passive datacenter in an active-passive multi-datacenter deployment.

Start the post-upgrade steps after upgrading all nodes in the test deployment or passive datacenter.

For example, in a test deployment with two datacenters and cross-datacenter replication, you must upgrade all nodes in both datacenters before starting the post-upgrade steps.

In contrast, in a test deployment with one datacenter and no cross-datacenter connections, you must upgrade all nodes in that datacenter before starting the post-upgrade steps.

Production deployment

Rolling out an upgrade to an entire production cluster.
Upgrading a multi-datacenter deployment that has cross-datacenter replication.

Start the post-upgrade steps after upgrading all nodes included in the deployment.

For example, in a multi-datacenter deployment with cross-datacenter replication, you must upgrade all nodes in all datacenters before starting the post-upgrade steps.

Complete the post-upgrade steps progressively in the recommended order. Modify the recommended order as needed for your deployment architecture and workloads.

Upgrade all SSTables again.

Your database upgrade isn’t complete until you upgrade all SSTables.