Upgrades for DataStax Enterprise 5.1.x patch releases

Familiarize yourself with the changes and features in the new release:

Stop the OpsCenter Repair Service if enabled:

Before upgrading, be sure that each node has adequate free disk space.

sudo du -sh /var/lib/cassandra/data/
3.9G	/var/lib/cassandra/data/

sudo df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   59G   16G   41G  28% /

Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:

nodetool repair -pr

DSE Search nodes: Ensure that the catalina.properties and context.xml files are present in the Tomcat conf dir. DSE will not start after upgrade if these files are missing. The default location of the Tomcat conf directory depends on the type of installation:

Back up any customized configuration files since they may be overwritten with default values during installation of the new version.

Upgrade the SSTables on each node to ensure that all SSTables are on the current version:

nodetool upgradesstables

Upgrades from DSE 5.1.0 and 5.1.1 to DSE 5.1.2 and later DSE 5.1.x releases

-Dcassandra.force_3_0_protocol_version=true

installation_location/bin/dse cassandra -Dcassandra.force_3_0_protocol_version=true

In multiple datacenter clusters, upgrade every node in one datacenter before upgrading another datacenter.

Upgrade the seed nodes within a datacenter first.

DSE Analytics datacenters

Transactional/DSE Graph datacenters

DSE Search nodes or datacenters

DSE Analytics nodes only: Kill all Spark worker processes:

for pid in $(jps | grep Worker | awk '{print $1}'); do kill -9 $pid; done

Flush the commit log of the current installation:

nodetool drain

Stop the node:

Use the appropriate method to install the new product version on a supported platform:

Compare changes in the new configuration files with the backup configuration files after the upgrade but before restarting, remove deprecated settings, and update any new settings if required.

cd /usr/share/dse/tools/yamls

./yaml_diff path/to/yaml-file-old path/to/yaml-file-new
...
 CHANGES
=========
authenticator:
- AllowAllAuthenticator
+ com.datastax.bdp.cassandra.auth.DseAuthenticator

authorizer:
- AllowAllAuthorizer
+ com.datastax.bdp.cassandra.auth.DseAuthorizer

roles_validity_in_ms:
- 2000
+ 120000
...

When upgrading DSE to versions *earlier than 5.1.16 , if any tables are using DSE Tiered Storage, remove all txn_compaction log files from second-level tiers and lower. For example, given the following dse.yaml configuration, remove txn_compaction log files from /mnt2 and /mnt3 directories:

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

Start the node.

Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition:

Review the logs for warnings, errors, and exceptions:

grep -w 'WARNING\|ERROR\|exception' /var/log/cassandra/*.log

Repeat the upgrade process on each node in the cluster following the recommended order.

If you use the OpsCenter Repair Service restart it:

After the entire cluster upgrade is complete: upgrade the SSTables on one node at a time or, when using racks, one rack at a time.

nodetool upgradesstables

Index time boost support is removed in DSE 5.1.1 and later. Use query time boosting instead. Delete any _docBoost columns in backing CQL tables:

DELETE _docBoost FROM table-name IF EXISTS;

Do a full reindex of all encrypted search indexes on each node in your cluster:

dsetool reload_core keyspace_name.table_name distributed=false reindex=true deleteAll=true