Upgrade DataStax Enterprise (DSE) 5.1 to 6.9

Change all use of HTTP API writes to instead use CQL commands for updates and inserts.

Edit the search index config and make these changes as needed. See Search index config for valid options to change query behavior for search indexes.

Ensure the presence of the catalina.properties and context.xml files in the Tomcat conf directory.

If earlier DSE versions use a custom configuration for the Solr UI web.xml, change the following configuration:

<filter-class>com.datastax.bdp.search.solr.auth.DseAuthenticationFilter</filter-class>

<filter-class>com.datastax.bdp.cassandra.auth.http.DseAuthenticationFilter</filter-class>

Be aware that the StallMetrics MBean is removed in DSE 6.9.

If you programmatically set the shuffle parameter, you must change the code for applications that use conf.set("spark.shuffle.service.port", port). Instead, use dse spark-submit which automatically sets the correct service port based on the authentication state.

If DSEFS is enabled, copy the CFS hivemetastore directory (if it exists) to DSEFS:

DSE_HOME/bin/dse hadoop fs -cp cfs://node_ip_address/user/spark/warehouse/ dsefs://node_ip_address/user/spark/warehouse/

If present in your DSE 5.1 installation, remove the Cassandra File System (CFS) by removing the cfs and cfs_archive keyspaces before upgrading. For more information, see Copying data from CFS to DSEFS.

cqlsh> DROP KEYSPACE cfs

cqlsh> DROP KEYSPACE cfs_archive

Make sure any use of the SPARK_LOCAL_DIRS and SPARK_EXECUTOR_DIRS environment variables match their use as described in 5.1@dse:spark:configure-node.adoc#configureEnvVar.

For applications to use a compatible Spark Jobserver API from the DataStax repository, you must migrate any jobs that extend from SparkHiveJob or SparkSqlJob to SparkSessionJob. See the DemoSparkSessionJob example in the demos directory, which is located at /usr/share/dse/demos for package installations and <installation_location>/demos for tarball installations.

Upgrade to the latest patch release on your current version. The latest patch release includes fixes that can simplify the upgrade process.

bin/dse -v
current_dse_version

Familiarize yourself with the changes and features in the new release as listed in the DSE 6.9 release notes.

Ensure that each node has enough free disk space for the upgrade. The required space depends on the compaction strategy.

Replace ITriggers and custom interfaces.

Because Thrift-compatible tables (COMPACT STORAGE) are no longer supported, you must migrate all non-system tables that have COMPACT STORAGE to CQL table format before upgrading.

cqlsh -e 'DESCRIBE FULL SCHEMA;' > schema_file

cat schema_file | while read -d $';\n' line ; do
  if echo "$line"|grep 'COMPACT STORAGE' 2>&1 > /dev/null ; then
    TBL="`echo $line|sed -e 's|^CREATE TABLE \([^ ]*\) .*$|\1|'`"
    if echo "$TBL"|egrep -v '^system' 2>&1 > /dev/null; then
      echo "ALTER TABLE $TBL DROP COMPACT STORAGE;" >> schema-drop-list
    fi
  fi
done

cqlsh -f schema-drop-list

Upgrade the SSTables on each node to ensure that all SSTables are on the current version:

nodetool upgradesstables

Ensure that keyspace replication factors are correct for your environment:

cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"
CREATE KEYSPACE keyspace-name WITH replication = {'class': 'NetworkTopologyStrategy, 'replication_factor': '3'} AND durable_writes = true;

Verify the Java runtime version and upgrade to a supported version if needed:

java -version

java version "11.0.x" YYYY-MM-DD LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.x+xx-LTS-219)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.x+xx-LTS-219, mixed mode)

Verify the Python version is between Python 3.8 and 3.11:

python --version

Python 2.7.18

Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:

nodetool repair -pr

Install the libaio package for optimal performance.

Back up any customized configuration files since they can be overwritten with default values when installing the new version.

In multiple datacenter clusters, upgrade every node in one datacenter before upgrading another datacenter.

Upgrade the seed nodes within a datacenter first.

Upgrade DSE Analytics datacenters.

Upgrade transactional (DSE Graph) datacenters.

Upgrade DSE Search nodes or datacenters.

Flush the commit log of the current installation:

nodetool drain

Stop the node:

sudo service dse stop

installation_dir/bin/dse cassandra-stop

Install the new product version on a supported platform using the same installation type as your current installation. Mismatched installation types can cause problems with the upgrade.

After upgrading but before restarting a node, compare changes in the new configuration files with your backup configuration files. Remove deprecated settings and update any new settings if required.

cd /usr/share/dse/tools/yamls
./yaml_diff path/to/yaml-file-old path/to/yaml-file-new

...
 CHANGES
=========
authenticator:
- AllowAllAuthenticator
+ com.datastax.bdp.cassandra.auth.DseAuthenticator

authorizer:
- AllowAllAuthorizer
+ com.datastax.bdp.cassandra.auth.DseAuthorizer

roles_validity_in_ms:
- 2000
+ 120000
...

If you are performing an interrim upgrade to versions earlier than 5.1.16, 6.0.8, or 6.7.4, and any tables use DSE Tiered Storage, you must remove all txn_compaction log files from second-level tiers and lower.

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

Remove any previously installed JTS (Java Topology Suite) JAR files from the CLASSPATHS in your DSE installation. JTS is distributed with DSE versions 6.7 and later.

After making all the configuration changes, start the node.

sudo service dse start

installation_dir/bin/dse cassandra

Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition:

Review the logs for warnings, errors, and exceptions:

grep -w 'WARNING\|ERROR\|exception' /var/log/cassandra/*.log

Repeat the upgrade process on each node in the cluster following the recommended upgrade order.

After the entire cluster upgrade is complete, use upgradesstables to upgrade the SSTables on one node at a time. When using racks, upgrade one rack at a time. This sequence is required to avoid degraded performance from upgrading too many nodes at once.

nodetool upgradesstables

If you use the OpsCenter Repair Service, turn it on.

If you encounter serialization-header errors, stop the node and repair them using the sstablescrub -e option:

sstablescrub -e fix-only keyspace table

Learn about DSE Metrics Collector.

The system no longer uses the appender SolrValidationErrorAppender or the logger SolrValidationErrorLogger, so you can safely remove them from logback.xml.

In contrast to earlier versions, DataStax recommends accepting the new default value of 1024 for back_pressure_threshold_per_core in dse.yaml. For more information, see Tune DSE Search for maximum indexing throughput.

Fully reindex all encrypted search indexes on each node in your cluster.

dsetool reload_core keyspace_name.table_name distributed=false reindex=true deleteAll=true

Check the replication factor for the dse_analytics keyspace, which is a new keyspace that stores all DSE Analytics internal system data. DataStax recommends setting the replication strategy to NetworkTopologyStrategy (NTS) with a minimum replication factor of 3 in each of your DSE Analytics datacenters. If a datacenter has more nodes, consider setting a larger replication factor.

cqlsh --execute "DESCRIBE KEYSPACE dse_analytics;" | grep "replication"
CREATE KEYSPACE keyspace-name WITH replication = {'class': 'replication-strategy, 'datacenter-name': 'replication-factor'};

If you are using Spark SQL tables, migrate them to the new Hive metastore format:

dse client-tool spark metastore migrate --from 5.1.0 --to 6.9.0

Because the Spark Jobserver uses DSE custom version 0.8.0.56, you must ensure that applications use the compatible Spark Jobserver API from the DataStax repository.