Upgrading DataStax Enterprise 5.1 to 6.7 or 6.8

In multiple datacenter clusters, upgrade every node in one datacenter before upgrading another datacenter.

Upgrade the seed nodes within a datacenter first.

DSE Analytics datacenters

Transactional/DSE Graph datacenters

DSE Search nodes or datacenters

Ensure all use of HTTP API writes are changed to use CQL commands for updates and inserts.

Edit the search index config and make these changes, as needed. See Search index config for valid options to change query behavior for search indexes.

Ensure that the catalina.properties and context.xml files are present in the Tomcat conf dir.

If earlier DSE versions use a custom configuration for the Solr UI web.xml, change:

<filter-class>com.datastax.bdp.search.solr.auth.DseAuthenticationFilter</filter-class>

<filter-class>com.datastax.bdp.cassandra.auth.http.DseAuthenticationFilter</filter-class>

The StallMetrics MBean is removed.

If you programmatically set the shuffle parameter, you must change the code for applications that use conf.set("spark.shuffle.service.port", port). Instead, use dse spark-submit which automatically sets the correct service port based on the authentication state. See Configuring Spark for more information.

If DSEFS is enabled, copy CFS hivemetastore directory to dse:

DSE_HOME/bin/dse hadoop fs -cp cfs://node_ip_address/user/spark/warehouse/ dsefs://node_ip_address/user/spark/warehouse/

Cassandra File System (CFS) is removed. Remove the cfs and cfs_archive keyspaces before upgrading. See the From CFS to DSEFS blog post and the Copying data from CFS to DSEFS documentation for more information.

DROP KEYSPACE cfs

DROP KEYSPACE cfs_archive

Make sure any use of the SPARK_LOCAL_DIRS and SPARK_EXECUTOR_DIRS environment variables match their use as described in Setting environment variables.

For applications to use the compatible Spark Jobserver API in DataStax repository, migrate jobs that extend from SparkHiveJob and SparkSqlJob to SparkSessionJob. See example in the DemoSparkSessionJob in the demos directory.

Upgrade to the latest patch release on your current version. Fixes included in the latest patch release can simplify the upgrade process.

bin/dse -v
current_dse_version

Familiarize yourself with the changes and features in the new release:

Before upgrading, be sure that each node has adequate free disk space.

sudo du -sh /var/lib/cassandra/data/
3.9G	/var/lib/cassandra/data/

sudo df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   59G   16G   41G  28% /

Replace ITriggers and custom interfaces.

Support for Thrift-compatible tables (COMPACT STORAGE) is dropped. Before upgrading, migrate all non-system tables that have COMPACT STORAGE to CQL table format:

cqlsh -e 'DESCRIBE FULL SCHEMA;' > schema_file

cat schema_file | while read -d $';\n' line ; do
  if echo "$line"|grep 'COMPACT STORAGE' 2>&1 > /dev/null ; then
    TBL="`echo $line|sed -e 's|^CREATE TABLE \([^ ]*\) .*$|\1|'`"
    if echo "$TBL"|egrep -v '^system' 2>&1 > /dev/null; then
      echo "ALTER TABLE $TBL DROP COMPACT STORAGE;" >> schema-drop-list
    fi
  fi
done

cqlsh -f schema-drop-list

Upgrade the SSTables on each node to ensure that all SSTables are on the current version:

nodetool upgradesstables

Ensure that keyspace replication factors are correct for your environment:

cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"
CREATE KEYSPACE keyspace-name WITH replication = {'class': 'NetworkTopologyStrategy, 'replication_factor': '3'}  AND durable_writes = true;

Verify the Java runtime version and upgrade to the recommended version.

java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:

nodetool repair -pr

Install the libaio package for optimal performance.

sudo yum install libaio

sudo apt-get install libaio1

Back up any customized configuration files since they may be overwritten with default values during installation of the new version.

Flush the commit log of the current installation:

nodetool drain

Stop the node:

Use the appropriate method to install the new product version on a supported platform:

To configure the new version:

When upgrading DSE to versions earlier than 5.1.16, 6.0.8, or 6.7.4 inclusive, if any tables are using DSE Tiered Storage, remove all txn_compaction log files from second-level tiers and lower. For example, given the following dse.yaml configuration, remove txn_compaction log files from /mnt2 and /mnt3 directories:

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

Remove any previously installed JTS JAR files from the CLASSPATHS in your DSE installation. JTS (Java Topology Suite) is distributed with DSE 6.7.

Start the node.

Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition:

Review the logs for warnings, errors, and exceptions:

grep -w 'WARNING\|ERROR\|exception' /var/log/cassandra/*.log

Repeat the upgrade process on each node in the cluster following the recommended order.

After the entire cluster upgrade is complete: upgrade the SSTables on one node at a time or, when using racks, one rack at a time.

nodetool upgradesstables

If you use the OpsCenter Repair Service, turn on the Repair Service (6.8).

If you encounter serialization-header errors, stop the node and repair them using the sstablescrub -e option:

sstablescrub -e fix-only keyspace table

DSE 6.8 introduces, and enables by default, the DSE Metrics Collector, a diagnostics information aggregator used to help facilitate DSE problem resolution. For more information on the DSE Metrics Collector, or to disable metrics collection, see DataStax Enterprise Metrics Collector (6.8).

The appender SolrValidationErrorAppender and the logger SolrValidationErrorLogger are no longer used and may safely be removed from logback.xml.

In contrast to earlier versions, DataStax recommends accepting the new default value of 1024 for back_pressure_threshold_per_core (6.8) in dse.yaml. See Configuring and tuning indexing performance (6.8).

Do a full reindex of all encrypted search indexes on each node in your cluster:

dsetool reload_core keyspace_name.table_name distributed=false reindex=true deleteAll=true

Check the replication factor for the dse_analytics keyspace, a new keyspace stores all DSE Analytics internal system data. DataStax recommends setting the replication strategy to NetworkTopologyStrategy (NTS) with a replication factor of at least 3 in each of DSE Analytics datacenters. If a datacenter has more nodes, a larger replication factor should be considered.

cqlsh --execute "DESCRIBE KEYSPACE dse_analytics;" | grep "replication"
CREATE KEYSPACE keyspace-name WITH replication = {'class': 'replication-strategy, 'datacenter-name': 'replication-factor'};

If you are using Spark SQL tables, migrate them to the new Hive metastore format:

dse client-tool spark metastore migrate --from 5.0.0 --to 6.7.0

Spark Jobserver uses DSE custom version 8.0.4.45. Ensure that applications use the compatible Spark Jobserver API from the DataStax repository.

Install yum-versionlock (one-time operation):

sudo yum install yum-versionlock

Lock the current DSE version:

sudo yum versionlock dse-*