Upgrading DataStax Enterprise 5.0 to 6.0

In multiple datacenter clusters, upgrade every node in one datacenter before upgrading another datacenter.

Upgrade the seed nodes within a datacenter first.

DSE Analytics datacenters

Transactional/DSE Graph datacenters

DSE Search datacenters.

Change HTTP API queries to CQL queries:

If any Solr core was created on DSE 4.6 or earlier and never reindexed after being upgraded to DSE 4.7 or later, you must reindex on DSE 5.0 before upgrading to DSE 6.x:

dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=false

If you are using Apache Solr SolrJ, the minimum required Solr version is 6.0.0. To find the current Solr version:

installation_directory/bin/solr status

For SpatialRecursivePrefixTreeFieldType (RPT) in search schemas, you must adjust your queries for these changes:

The Circle syntax is no longer a part of Well-Known-Text (WKT); therefore, Spatial Search queries such as:

Intersects(Circle(10 10 d=2))

Intersects(BUFFER(POINT(10 10), 2)

Edit the solrconfig.xml file and make these changes, as needed:

Update the schema and configuration for the Solr field types that are removed from Solr 5.5 and later.

If using BCDStrField:

Tune the schema before you upgrade. After the upgrade, all field definitions in the schema are validated and must be DSE Search compatible, even if the fields are not indexed, have docValues applied, or used for copy-field source. The default behavior of automatic resource generation includes all columns. To improve performance, take action to prevent the fields from being loaded from the database. Include only the required fields in the schema by removing or commenting out unused fields in the schema.

Remove these unsupported Solr requestHandlers:

Change the directoryFactory from:

<directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>

<directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

Remove <unlockOnStartup>.

Reload the core:

dsetool reload_core keyspace_name.table_name reindex=false

DSE versions earlier than 5.1 use an older version of Spark and applications written using that version (1.6) may not be compatible with Spark 2.2. You must recompile all DSE 5.0 Scala Spark applications against Scala 2.11 and use only Scala 2.11 third-party libraries.

Spark applications should use dse:// URLs instead of spark://spark_master_IP:Spark_RPC_port_number URLs, as described in Specifying Spark URLs.

Cassandra File System (CFS) is removed. Remove the cfs and cfs_archive keyspaces before upgrading. See the From CFS to DSEFS blog post and the Copying data from CFS to DSEFS documentation for more information.

DROP KEYSPACE cfs

DROP KEYSPACE cfs_archive

If DSEFS is enabled, copy CFS hivemetastore directory to dse:

DSE_HOME/bin/dse hadoop fs -cp cfs://node_ip_address/user/spark/warehouse/ dsefs://node_ip_address/user/spark/warehouse/

Modify calls to setMasterv and setAppName.

val conf = new SparkConf(true)
.setMaster("spark://192.168.123.10:7077")
.setAppName("cassandra-demo")
.set("cassandra.connection.host" , "192.168.123.10") // initial contact
.set("cassandra.username", "cassandra")
.set("cassandra.password", "cassandra")
val sc = new SparkContext(conf)

val conf = new SparkConf(true)
**.appName**("cassandra-demo")
**.master**("dse://192.168.123.10:7077")
.set("cassandra.connection.host" , "192.168.123.10") // initial contact
.set("cassandra.username", "cassandra")
.set("cassandra.password", "cassandra")
val sc = new SparkContext(conf)

Upgrade to the latest patch release on your current version. Fixes included in the latest patch release can simplify the upgrade process.

bin/dse -v
current_dse_version

Familiarize yourself with the changes and features in the new release:

Ensure that each node has adequate free disk space before upgrading.

sudo du -sh /var/lib/cassandra/data/
3.9G	/var/lib/cassandra/data/

sudo df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   59G   16G   41G  28% /

Replace ITriggers and custom interfaces.

Support for Thrift-compatible tables (COMPACT STORAGE) is dropped. Before upgrading, migrate all non-system tables that have COMPACT STORAGE to CQL table format:

cqlsh -e 'DESCRIBE FULL SCHEMA;' > schema_file

cat schema_file | while read -d $';\n' line ; do
  if echo "$line"|grep 'COMPACT STORAGE' 2>&1 > /dev/null ; then
    TBL="`echo $line|sed -e 's|^CREATE TABLE \([^ ]*\) .*$|\1|'`"
    if echo "$TBL"|egrep -v '^system' 2>&1 > /dev/null; then
      echo "ALTER TABLE $TBL DROP COMPACT STORAGE;" >> schema-drop-list
    fi
  fi
done

cqlsh -f schema-drop-list

If audit logging is configured to use CassandraAuditWriter, run these CQL commands as superuser on DSE 5.0 nodes:

ALTER TABLE dse_audit.audit_log ADD authenticated text;

ALTER TABLE dse_audit.audit_log ADD consistency text

nodetool describecluster
Cluster Information:
	Name: Test Cluster
	Snitch: com.datastax.bdp.snitch.DynamicEndpointSnitch
	DynamicEndPointSnitch: enabled
	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
	Schema versions:
		0fffd971-b7a4-33ae-859d-8ca792cd2852: [10.116.138.23]

Upgrade the SSTables on each node to ensure that all SSTables are on the current version:

nodetool upgradesstables

Ensure that keyspace replication factors are correct for your environment:

cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"
CREATE KEYSPACE keyspace-name WITH replication = {'class': 'NetworkTopologyStrategy, 'replication_factor': '3'}  AND durable_writes = true;

Verify the Java runtime version and upgrade to the recommended version.

java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:

nodetool repair -pr

Install the libaio package for optimal performance.

sudo yum install libaio

sudo apt-get install libaio1

Back up any customized configuration files since they may be overwritten with default values during installation of the new version.

Flush the commit log of the current installation:

nodetool drain

DSE Analytics nodes only: Kill all Spark worker processes:

for pid in $(jps | grep Worker | awk '{print $1}'); do kill -9 $pid; done

Stop the node:

Use the appropriate method to install the new product version on a supported platform:

To configure the new product version:

When upgrading DSE to versions earlier than 5.1.16, 6.0.8, or 6.7.4 inclusive, if any tables are using DSE Tiered Storage, remove all txn_compaction log files from second-level tiers and lower. For example, given the following dse.yaml configuration, remove txn_compaction log files from /mnt2 and /mnt3 directories:

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

DSE Analytics nodes only: If your DSE 5.0 clusters had any datacenters running in Analytics Hadoop mode and if the DseSimpleSnitch was used, you must use one of these options for starting nodes in your cluster. Select the option that works best for your environment:

Start the node.

Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition:

Review the logs for warnings, errors, and exceptions:

grep -w 'WARNING\|ERROR\|exception' /var/log/cassandra/*.log

Repeat the upgrade process on each node in the cluster following the recommended order.

After the entire cluster upgrade is complete: upgrade the SSTables on one node at a time or, when using racks, one rack at a time.

nodetool upgradesstables

If you use the OpsCenter Repair Service, turn on the Repair Service (6.8).

If you encounter serialization-header errors, stop the node and repair them using the sstablescrub -e option:

sstablescrub -e fix-only keyspace table

Drop the following legacy tables, if they exist: system_auth.users, system_auth.credentials, and system_auth.permissions:

DROP TABLE IF EXISTS system_auth.users;

DROP TABLE IF EXISTS system_auth.credentials;

DROP TABLE IF EXISTS system_auth.permissions;

Review your security configuration. To use security, enable and configure DSE Unified Authentication.

TimeWindowCompactionStrategy (TWCS) (TWCS (6.7) | TWCS (6.8)) is set only on new dse_perf and dse_audit_log tables. Manually change dse_perf and dse_audit_log tables that were created in earlier releases to use TWCS. For example:

ALTER TABLE dse_perf.read_latency_histograms WITH COMPACTION={'class':'TimeWindowCompactionStrategy'};

ALTER TABLE dse_audit_log.audit_log WITH COMPACTION={'class':'TimeWindowCompactionStrategy'};

The appender SolrValidationErrorAppender and the logger SolrValidationErrorLogger are no longer used and may safely be removed from logback.xml.

In contrast to earlier versions, DataStax recommends accepting the new default value of 1024 for back_pressure_threshold_per_core in dse.yaml. See Configuring and tuning indexing performance.

If SpatialRecursivePrefixTreeFieldType (RPT) is used in the search schema, replace the units field type with a suitable (degrees, kilometers, or miles) distanceUnits, and then verify that spatial queries behave as expected.

If you are using HTTP API writes with JSON documents (deprecated), a known issue may cause the auto-generated solrconfig.xml to have invalid requestHandler for JSON core creations. If necessary, change the auto generated solrconfig.xml:

<requestHandler name="/update/json" class="solr.UpdateUpdateRequestHandler" startup="lazy"/>

<requestHandler name="/update/json" class="solr.UpdateRequestHandler" startup="lazy"/>

Do a full reindex of all encrypted search indexes on each node in your cluster:

dsetool reload_core keyspace_name.table_name distributed=false reindex=true deleteAll=true

Spark Jobserver uses DSE custom version 8.0.4.45. Ensure that applications use the compatible Spark Jobserver API from the DataStax repository.

If you are using Spark SQL tables, migrate them to the new Hive metastore format:

dse client-tool spark metastore migrate --from 5.0.0 --to 6.0.0

Install yum-versionlock (one-time operation):

sudo yum install yum-versionlock

Lock the current DSE version:

sudo yum versionlock dse-*