Upgrade DSE 5.0 to 5.1

Improper upgrades can result in downtime, data loss, and other errors.

To minimize risk for an upgrade, review the entire upgrade guide before you begin, including warnings, restrictions, release notes, and all pre-upgrade, upgrade, and post-upgrade steps.

You must prepare all nodes in the cluster before upgrading any nodes. Failure to prepare all nodes can result in mismatched configurations, repair states, and schemas that can disrupt the upgrade process.

Follow these instructions to manually upgrade from DataStax Enterprise (DSE) 5.0 to 5.1.

As an alternative to an in-place upgrade on a live cluster, you can use the Zero Downtime Migration (ZDM) tools to orchestrate live traffic while you prepare a new cluster with your desired DSE version, configuration, and data. After testing the new cluster, you can redirect your application traffic and safely decommission your current cluster. If you need to rollback, you can seamlessly discard the new cluster and continue using your existing cluster.

Upgrade path to avoid data loss

TTL expiration timestamps are susceptible to the year 2038 problem. If the TTL value is long and an expiration date that is greater than the maximum threshold of 2038-01-19T03:14:06+00:00, the data is immediately expired and purged on the next compaction.

To protect against silent data loss, upgrade to DSE 5.1.7 or later and recover and reinsert rows with overflowed timestamps.

Avoid certain operations during the upgrade

The upgrade process for DSE can be completed with minimal downtime (ideally zero) by upgrading nodes in a rolling fashion: upgrade and restart one node at a time while the other nodes continue to operate.

The moment that you install the new version on one node in a cluster, the cluster enters a partially upgraded state. In this state, the cluster continues to operate as though all nodes are running the earlier, pre-upgraded version. The cluster remains in this state until you have upgraded all nodes in the cluster.

You must avoid or block the following operations during the upgrade to maintain cluster stability and prevent errors:

  • Don’t enable new features.

    This restriction includes net new features and existing features that you weren’t using previously. For example, don’t enable Change Data Capture (CDC) for the first time during an upgrade. Upgrade the entire cluster, and then enable the feature.

  • Don’t change credentials, permissions, or any other security settings unless explicitly instructed to do so.

    To preserve your existing configuration, you will transfer your existing security settings from your current configuration files to the new configuration files during the upgrade process. The new files might use different settings due to enhancements, deprecations, and removals between versions. This mapping is required for a successful upgrade, and it isn’t the same as an extraneous change to the security configuration.

    Examples of security changes to avoid include modifications to authenticators, authorizers, CQL roles, and CQL permissions. Postpone all such changes until after the upgrade.

    If you aren’t already using Kerberos, don’t set up Kerberos authentication immediately before upgrading. First upgrade the cluster, and then set up Kerberos. If you are already using Kerberos, you can continue to use it during the upgrade.

  • Don’t run repairs manually or automatically:

    • Disable all automated and scheduled repairs, including Reaper, crontab, and scripts that run nodetool repair.

    • Don’t manually run commands like nodetool repair.

    • Don’t repair SSTables while the cluster is in a partially upgraded state. Only repair SSTables before and after upgrading the entire cluster.

    You must complete the cluster-wide upgrade before the expiration of gc_grace_seconds (approximately 10 days by default) to ensure any post-upgrade repairs complete successfully.

    Attempting to run repairs on partially upgraded clusters can cause performance issues, data inconsistencies, and repair failures. If you absolutely must run repairs on a partially upgraded cluster, determine which SSTable formats and streaming protocols are supported on each node, and then thoroughly test your intended repair operation on an isolated non-production cluster. Small incremental repairs are less risky but they aren’t risk free.

  • Don’t add (bootstrap) new nodes or decommission existing nodes.

  • Don’t issue DDL-related queries like TRUNCATE.

    Depending on your infrastructure, workloads, and consistency levels, you might want to avoid other operations that could require excessive resources or coordination across mixed-version nodes. For example, BATCH operations can be resource intensive.

  • Don’t alter any schemas.

    Schema versions must be consistent across the cluster before starting the upgrade.

    Propagation of schema changes between mixed-version nodes can cause query failures, data consistency issues, and other errors. Take action to prevent schema changes from occurring during the upgrade process.

    During the upgrade, it is normal for nodes on different versions to show schema disagreements. These disagreements should resolve after the entire cluster is upgraded.

  • Don’t rebuild indexes during the upgrade.

  • Disable services that can cause data inconsistencies and interfere with the upgrade process:

    • Disable backup services, scheduled backups, and automated snapshots.

  • If you disabled the Performance Service before the upgrade, don’t reenable it during the upgrade.

  • DSE Analytics nodes: Don’t run analytics jobs until all nodes are upgraded.

    When upgrading to a new major or minor version of DSE, all nodes in a DSE datacenter that run Spark must be on the same version of Spark, and the Spark jobs must be compiled for that version. To determine the Spark version, see the DSE release notes.

    Before reinitiating Spark jobs, all datacenters comprising a Spark cluster must be on the same upgraded DSE version. If any Spark jobs run against Graph keyspaces, don’t reinitiate Spark jobs until you have upgraded all Analytics and non-Analytics nodes in the entire cluster. When Graph keyspaces are involved, Spark jobs fail due to mixed Spark and DSE versions across all nodes.

  • DSE Search Nodes: DSE 5.1 uses a different Apache Lucene™ codec than DSE 5.0 for new search cores. Segments written with this new codec cannot be read by earlier versions of DSE.

    If you need to downgrade after creating new search cores in the new version, you must clear the entire data directory for the rolled-back search index.

Driver compatibility and live application traffic

Incompatible drivers and unspecified protocol versions can cause connection failures and other issues during the upgrade.

To maintain live application traffic throughout the upgrade process, your Cassandra driver must be compatible with your current installation and your target DSE version. Additionally, you must configure protocol version negotiation to ensure backwards compatibility throughout the upgrade.

Find a mutually compatible driver version

Verify that your current Cassandra driver version is compatible with your target DSE version. For available drivers and compatibility information, see Cassandra driver compatibility.

If your current driver version isn’t compatible with your target DSE version, upgrade your driver to a mutually compatible version.

When upgrading your driver, check your driver’s release notes for breaking changes that might require code changes. Review all release notes from your current driver version through the new version. Recompile your client application code, if needed.

No mutually compatible version exists

If there is no mutually compatible driver version, you have two options:

  • Intermediate upgrade (recommended): Determine if your upgrade path requires (or supports) an intermediate upgrade before proceeding to your target DSE version. Find driver versions that are compatible with each segment of the upgrade path (current to intermediate and intermediate to target), and then upgrade your driver before each segment.

  • Delayed upgrade: Upgrade to the latest driver version that is compatible with your current installation, and then plan to upgrade your driver again when your cluster is on your target DSE version. This can require code or configuration changes to accommodate the temporary driver version, such as changing authentication methods or avoiding unsupported query patterns. Consider testing this approach in an isolated environment to identify potential compatibility issues before applying it to your production environment.

Configure protocol negotiation

When a cluster is in a partially upgraded state, upgraded nodes might support different Cassandra native protocol versions than non-upgraded nodes. By default, a driver negotiates the protocol version with the first host (node) that it connects to. The driver might select a protocol version that works across all nodes, but this isn’t guaranteed if the driver connects to a random node each time. It is better to prevent protocol incompatibility in one of two ways:

  • Set protocol version: Set a mutually compatible protocol version explicitly in your application at start up. After you upgrade all nodes in the cluster, you can switch your applications to the latest supported protocol version.

  • Set initial contact points: Specify a list of initial contact points that contains only hosts with the oldest database platform version or mutually compatible protocol version. You must upgrade these nodes last.

For more information about protocol version negotiation, see your driver’s documentation.

DSE-only drivers are unsupported

All DSE-only drivers have reached end-of-life (EOL); they are unsupported and unmaintained. If your applications use DSE-only drivers, you must upgrade to a recent version of a supported driver.

For legacy applications that require DSE-only drivers, compatibility with your target DSE version might be limited or nonexistent. For example, newer DSE, Cassandra, and CQL features might not be supported by the driver, or queries that use deprecated data types, functions, or syntax might fail. Consider modernizing or sunsetting legacy applications if possible, particularly because EOL drivers don’t receive any updates, including security updates.

Strict upgrade order for cluster stability

Don’t upgrade any nodes until you have completed the pre-upgrade steps on all nodes.

When upgrading nodes, always upgrade seed nodes before non-seed nodes.

To avoid cluster-wide downtime, you must upgrade and restart one node at a time in a specific order based on workload types and deployment architecture:

  1. For multi-cluster deployments, upgrade one cluster at a time.

  2. For multi-datacenter clusters, upgrade one datacenter at a time.

  3. If using racks, upgrade one rack at a time within a datacenter.

  4. Within a datacenter (or rack), upgrade one node at a time, starting with seed nodes.

  5. If using advanced workloads (Analytics, Graph, Search), upgrade datacenters or nodes according to the following workload type hierarchy.

    If each datacenter has only one workload type, upgrade your datacenters in the given order. Within each datacenter, proceed rack-by-rack (if applicable) and node-by-node, starting with the seed nodes.

    If a datacenter has multiple workload types, upgrade nodes within that datacenter in the given order, starting with the seed nodes. If using racks, apply the hierarchy to each rack.

    The workload type hierarchy is:

    1. Upgrade DSE Analytics datacenters:

      1. Upgrade the DSE Hadoop Job Tracker node.

      2. Upgrade Hadoop nodes.

      3. Upgrade Spark nodes.

    2. Upgrade transactional (Cassandra) and DSE Graph datacenters.

    3. Upgrade DSE Search datacenters.

  6. Repeat until you have upgraded all nodes in all racks, datacenters, and clusters.

Use an isolated deployment to test the upgrade.

For example, in multi-datacenter clusters with an active-passive configuration, you can complete the entire upgrade process (pre-upgrade, upgrade, and post-upgrade) on the passive datacenter first, run performance tests on that datacenter, and then upgrade the active datacenter if the tests are successful. If the tests fail, you can destroy the passive datacenter and rebuild it without impacting the active datacenter.

Similarly, if you have a test deployment with no connectivity to your production nodes, datacenters, or clusters, then you can safely perform the entire upgrade process in that isolated deployment before rolling out the upgrade to your production deployment.

Prepare to upgrade

Before upgrading any node, you must prepare all nodes for the upgrade. Complete the pre-upgrade steps on every node in the recommended order. Following the recommended order ensures that you don’t miss any nodes, although some steps aren’t applicable to every node.

Some pre-upgrade steps provide information about actions that you must take later in the upgrade process. This information is given in advance because it might impact your upgrade plan. Be sure to revisit these steps at the appropriate time.

  1. Upgrade to the latest patch release for your current version.

    Always upgrade to latest patch release for your current version before you upgrade to a new major or minor version. Patch releases help prepare your clusters for the upgrade by minimizing the differences between the current version and the target version.

    Get your current DSE version with dse -v, and then compare it with the latest version available for your DSE release series. You can find the latest version in the DSE release notes.

    If you need to apply a patch release, upgrade all nodes to the latest patch release before applying any further upgrades. Treat patch releases as you would any major or minor upgrade by carefully reviewing the release notes and following the recommended upgrade procedure. For specific instructions, see Apply a DSE 4.8 or 5.0 patch release.

  2. Familiarize yourself with the changes and features in the new release, reviewing all releases between your current version and the latest 5.1 patch release as well as Apache Cassandra® changes:

  3. Plan to upgrade DSE OpsCenter.

    DSE OpsCenter 6.5 supports both DSE 5.0 and 5.1; however, this version is EOSL. The minimum recommended DSE OpsCenter version for DSE 5.1 is DSE OpsCenter 6.8.4, but this version doesn’t support DSE 5.0. For assistance with upgrading DSE OpsCenter, contact IBM Support.

  4. Compare your current operating system version to the supported platforms for DSE 5.1:

    • If your current OS version is compatible with your new DSE version, no action is needed.

    • If your current OS is compatible but outdated, plan to upgrade your OS before or after upgrading DSE to minimize complexity and risk.

    • If you need to make significant system environment changes to support your new DSE version, an in-place upgrade might not be feasible or reasonable. Consider using platform migration tools, like the Zero Downtime Migration (ZDM) tools, to migrate to a new cluster with the required infrastructure rather than attempt an in-place upgrade. For assistance, contact IBM Support.

  5. Check Java runtime compatibility and install a mutually supported Java runtime version if needed.

    For upgrades from DSE 5.0 to DSE 5.1, install the latest build of OpenJDK 8 (recommended) or Oracle Java SE 8 (JRE or JDK) (supported). OpenJDK is recommended because DataStax does more extensive testing on OpenJDK.

    1. Get your current Java version:

      java -version

    2. Make sure the output shows a supported Java runtime. If not, install a supported Java runtime.

      For example, if OpenJDK 8 is installed:

      openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

    3. If your environment has multiple Java versions installed, set the $JAVA_HOME environment variable to Java 8.

    4. If you changed $JAVA_HOME, restart the node.

  6. Install the libaio package:

    RHEL
    sudo yum install libaio
    Debian
    sudo apt-get install libaio1
    Ubuntu 24.04 or later
    sudo apt-get install libaio1t64

    On Ubuntu 24.04 and later, the libaio library was renamed to libaio1t64. DSE requires a symlink from the old library name to the new one to detect it properly. Without this link, DSE issues a warning about the missing library and performance is significantly degraded.

    sudo ln -s /lib/x86_64-linux-gnu/libaio.so.1t64 /lib/x86_64-linux-gnu/libaio.so.1

  7. Make sure each node has adequate free disk space for the upgrade.

    The required overhead depends on the compaction strategy. For more information, see Disk space.

    1. Get the size of the DSE data directory:

      sudo du -sh /var/lib/cassandra/data/

    2. Check available space on disk:

      sudo df -hT /

      The following output includes one disk named /dev/sda1. Your output depends on your disk configuration.

      Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   59G   16G   41G  28% /

    3. If necessary, make adjustments to your disks or data directory configuration to allow for more disk space.

  8. Ensure that keyspace replication factors are correct for your environment.

    Check user-defined, analytics, system_auth, and dse_security keyspaces.

    1. Get the current replication factor:

      cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"

    2. If necessary, alter or recreate keyspaces with the correct replication factor:

      ALTER KEYSPACE keyspace-name WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;

      If you use NetworkTopologyStrategy, the replication factor is set for each datacenter in the form of 'DATACENTER_NAME': 'FACTOR'. Use commas to separate datacenter-factor pairs.

      For more information, see Setting the replication factor for analytics keyspaces and Configure the security keyspaces replication factors.

  9. Upgrade all SSTables on all nodes.

    To avoid significant performance degradation and issues with incompatible SSTables formats, always upgrade SSTables before and after any database upgrade, including minor and patch releases.

    This process is recommended for all upgrades, and it is critical for upgrades that involve a major underlying Cassandra version change.

    On each node, run nodetool upgradesstables with the -a flag to ensure that all SSTables use the latest SSTable format and version for your current database version:

    nodetool upgradesstables -a

    DataStax recommends running the upgradesstables command on one node at a time or, when using racks, one rack at a time. Running the command on too many nodes at once can degrade performance.

    You can use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

    If all SSTables are already on the latest version and format, the command returns immediately and takes no action.

  10. Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:

    nodetool repair -pr

    Repair nodes on a regular basis to minimize the overhead and time required for repair operations.

    If you use DSE OpsCenter, you can repair nodes with the Repair Service but you must disable the Repair Service before upgrading any nodes.

  11. Back up all DSE data and files, and then store the backups separately from your existing DSE installation.

    Backup your existing installation before every upgrade, regardless of the extent of the upgrade.

    To avoid data loss, you must store your backup configuration files in a separate directory from your DSE installation. The new version overwrites configuration files that are present in the installation directory, replacing all existing values with new default values.

    Examples of data and files to back up include:

    • SSTable snapshots

    • Commit logs

    • Logs

    • Configuration files

    • Encrypted table recovery data:

      • A snapshot of the entire dse_system keyspace with the encrypted_keys table

      • All system keys

    Some backup files are used during the upgrade process to align the new version with the previous version, and a complete backup is required if you need to rollback to the previous version or restore lost data.

    If your upgrade requires significant pre-upgrade changes, consider creating a backup before making those changes and again immediately before upgrading the first node. Create the final backup as close to the beginning of your upgrade as possible to ensure your backup contains the most recent configuration files and data.

    For instructions, see Back up and restore a DSE package installation or Back up and restore a DSE tarball installation.

  12. Upgrades from 5.0.0 through 5.0.7: Restart the node with -Dcassandra.force_3_0_protocol_version=true to enforce a backwards-compatible protocol version:

    installation_location/bin/dse cassandra -Dcassandra.force_3_0_protocol_version=true

    After the upgrade is complete, you can remove this flag.

    This step isn’t required if you are already running 5.0.8 or later.

    While mixed versions exist during the upgrade, don’t add or remove columns from existing tables.

Additional preparation for DSE Search and SearchAnalytics nodes

Complete these additional pre-upgrade steps on DSE Search and SearchAnalytics nodes while DSE 5.0 is still running.

DataStax recommends running nodetool repair and creating a fresh backup after making these changes, even if you already did so during the pre-upgrade steps.

There are extensive changes to DSE Search and DSE SearchAnalytics between version 5.0 and 5.1. Plan sufficient time to implement and test the required changes before the upgrade. For assistance, contact IBM Support.

Schema changes might require a full reindex, and configuration changes require reloading the core.

  1. Change HTTP API queries to Cassandra Query Language (CQL) queries:

    • Delete-by-id is removed. Use CQL DELETE by primary key instead.

    • Delete-by-query no longer supports wildcards. Use CQL TRUNCATE instead.

  2. Any Solr core created on DSE 4.6 or earlier that wasn’t reindexed after upgrading to DSE 4.7 or later must be reindexed before upgrading to DSE 5.1:

    dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=false

    You must reindex all nodes before beginning the upgrade.

  3. If you are using SolrJ, you must use, at minimum, Solr version 6.0.0.

    To get your current Solr version:

    installation_directory/bin/solr status

    For more information, see Upgrading Solr.

  4. For SpatialRecursivePrefixTreeFieldType (RPT) in search schemas, you must adjust your queries for the following changes:

    • IsDisjointTo is no longer supported in queries on SpatialRecursivePrefixTreeFieldType. Replace IsDisjointTo with a NOT Intersects query. For example:

      foo:0,0 TO 1000,1000 AND -"Intersects(POLYGON((338 211, 338 305, 404 305, 404 211, 338 211)))")

    • The ENVELOPE syntax is required for Well-Known-Text (WKT) style queries against SpatialRecursivePrefixTreeFieldType fields. For example, you must specify ENVELOPE(10, 15, 15, 10) where queries on earlier releases could specify 10 10 15 15. For more information on using distanceUnits in spatial queries, see Apache Solr Spatial Search.

    • The Circle syntax is removed from WKT. Therefore, Spatial Search queries like Intersects(Circle(10 10 d=2)) must be rewritten as Intersects(BUFFER(POINT(10 10), 2)).

  5. In solrconfig.xml, remove unsupported Solr requestHandler classes:

    • XmlUpdateRequestHandler

    • BinaryUpdateRequestHandler

    • CSVRequestHandler

    • JsonUpdateRequestHandler

    • DataImportHandler

    For example, the following configurations are invalid because the class attribute references unsupported Solr request handler classes:

    <requestHandler name="/dataimport" class="solr.DataImportHandler"/>
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler"/>

  6. In solrconfig.xml, use the simplified class path for directoryFactory.

    For example, given the following configuration:

    <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>

    Change the class attribute to the simplified path:

    <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

  7. In solrconfig.xml, remove <unlockOnStartup> because it is no longer supported.

  8. In solrconfig.xml, remove the class attribute from updateLog.

    For example, given the following configuration:

    <updateLog class="solr.FSUpdateLog" force="false">

    Remove the class attribute:

    <updateLog force="false">

  9. Replace unsupported Solr types with supported types.

    Some types require special handling, including multiple rounds of schema and application changes before, during, and after the upgrade.

    Adapt these steps for your infrastructure and workloads, repeating or skipping steps based on the Solr types and query patterns that you use. For example, if you apply post-upgrade schema changes, repeat the schema tuning and reindexing steps as you would for typical Solr core maintenance.

    Review the guidance for all types in the following steps before taking action.

    1. Understand the sorting limitations that exist during this upgrade.

      While a cluster is in a partially upgraded state, nodes on the the new DSE version consider unsupported Solr types invalid, even if those types are found on a non-upgraded node. Until all nodes are upgraded and use valid types, sorting on invalid types is not possible.

      This limitation is due to incompatibility in the way the old and new types handle marshalling and unmarshalling of sort values during distributed queries.

      Specifically, the following type transitions are problematic:

      Removed Solr field type Supported Solr field type Upgrade strategy to address sorting limitations

      ByteField

      TrieIntField

      Avoid sorting on ByteField until the upgrade is complete for all nodes in the datacenter being queried.

      DateField

      TrieDateField

      Avoid sorting on DateField until the upgrade is complete for all nodes in the datacenter being queried.

      BCDIntField

      TrieIntField

      Temporarily replace BCDIntField with SortableIntField.

      Although the SortableIntField type is removed, it is sort-compatible with TrieIntField. Therefore, queries can successfully sort on either TrieIntField or SortableIntField during the upgrade regardless of a node’s DSE version.

      Change the schema in a distributed fashion, but don’t reindex. After applying this change to all nodes, tune the schema (remove or comment out unused fields) and then reindex. More information about schema tuning is available at the end of this section.

      BCDLongField

      TrieLongField

      Temporarily replace BCDLongField with SortableLongField.

      Although the SortableLongField type is removed, it is sort-compatible with TrieLongField. Therefore, queries can successfully sort on either SortableLongField or TrieLongField during the upgrade regardless of a node’s DSE version.

      Change the schema in a distributed fashion, but don’t reindex. After applying this change to all nodes, tune the schema (remove or comment out unused fields) and then reindex. More information about schema tuning is available at the end of this section.

      If you have multiple Search datacenters, you can isolate queries to a single datacenter while you change the schema and reindex another datacenter. Then, isolate queries to the newly reindexed datacenter while you change the schema and reindex the first datacenter.

      With this strategy, the isolated datacenter can continue to process sorts based on the types supported by that datacenter’s DSE version. After all nodes are upgraded and the schema changes are fully applied to all datacenters, you can remove the isolation and allow queries to be processed across all datacenters.

    2. Update the schema and configuration for Solr field types that are removed in Solr 5.5 and later.

      Removed Solr field type Supported Solr field type Comments

      ByteField

      TrieIntField

      DateField

      TrieDateField

      DoubleField

      TrieDoubleField

      FloatField

      TrieFloatField

      IntField

      TrieIntField

      LongField

      TrieLongField

      ShortField

      TrieIntField

      SortableDoubleField

      TrieDoubleField

      SortableFloatField

      TrieFloatField

      SortableIntField

      TrieIntField

      SortableLongField

      TrieLongField

      BCDIntField

      TrieIntField

      If you are temporarily using SortableIntField to address the sort limitations described in the previous step, replace SortableIntField with TrieIntField after upgrading all nodes in a datacenter.

      BCDLongField

      TrieLongField

      If you are temporarily using SortableLongField to address the sort limitations described in the previous step, replace SortableLongField with TrieLongField after upgrading all nodes in a datacenter.

      BCDStrField

      TrieIntField

      In earlier versions, DSE mapped Cassandra text columns to BCDStrField. DSE cannot map text to TrieIntField.

      If your schema has BCDStrField, you must do one or more of the following:

      • Recommended: If you no longer use BCDStrField, remove the unused BCDStrField fields from your Solr schema. If this is your only schema change, then reindexing isn’t required. If you have some active BCDStrField field, consider the following options for those fields.

      • Recommended: To index the field as a TrieIntField, change the underlying database column to the int type, and change BCDStrField to TrieIntField in the Solr schema. A full reindex is required. If a full reindex isn’t feasible, consider the following alternatives.

      • To keep the database column as text and only run simple matching queries on the indexed field, replace BCDStrField with StrField in the Solr schema. With this option, you cannot use the field for numeric range queries or sorting because StrField uses a lexicographic order rather than a numeric one. Typically, reindexing isn’t required.

      • Not recommended: This option is intended for sub-optimal data models, such as a text column with values that cannot convert to int. Use this option only if you must keep the database column as text and run numeric range queries and sorts on the BCDStrField field. In the Solr schema, change the BCDStrField field to StrField with indexed=false. Then, add a new copy field with the type TrieIntField that copies values from the original BCDStrField field (now StrField). Reindexing is required to populate the new copy field.

    3. After you make the schema changes for the removed types, reload the core with full reindexing in a rolling fashion (one node at a time):

      dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=false

      If you have multiple Search datacenters, upgrade one datacenter at a time, and reload the core with distributed=true and deleteAll=true:

      dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=true

    4. Make sure solrconfig.xml uses type mapping version 1 or 2:

      • If the Solr core is backed by a CQL table and type mapping is unspecified, use <dseTypeMappingVersion>2</dseTypeMappingVersion>. In DSE 5.1 and later, auto-generated schemas use data type mapper version 2.

      • If the core isn’t backed by a CQL table and type mapping is specified with <dseTypeMappingVersion>0</dseTypeMappingVersion>, change it to <dseTypeMappingVersion>1</dseTypeMappingVersion>.

      • If the core isn’t backed by a CQL table and type mapping is unspecified, either verify that it automatically uses version 1, or specify <dseTypeMappingVersion>1</dseTypeMappingVersion>.

      Then, reload the core:

      dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath

      Or do a full reindex, node-by-node, as shown in the previous step.

  10. Tune the schema before you upgrade.

    After the upgrade, all field definitions in the schema are validated and must be compatible with DSE Search. This requirement applies to all fields, including fields with docValues applied, fields used as a copy-field source, and fields that aren’t indexed.

    By default, automatic resource generation includes all columns. To improve performance, take action to prevent fields from being loaded from the database. For example, make sure your schema includes only required fields by removing or commenting out unused fields.

    Be sure to reload the core (with reindexing, if needed) after making schema changes.

Additional preparation for DSE Graph nodes

Complete these additional pre-upgrade steps on DSE Graph nodes with Search indexes while DSE 5.0 is still running.

DataStax recommends running nodetool repair and creating a fresh backup after making these changes, even if you already did so during the pre-upgrade steps.

These configuration changes require reloading the core. Plan sufficient time to implement and test these changes before starting the upgrade.

  1. In solrconfig.xml, remove unsupported Solr requestHandler classes:

    • XmlUpdateRequestHandler

    • BinaryUpdateRequestHandler

    • CSVRequestHandler

    • JsonUpdateRequestHandler

    • DataImportHandler

    For example, the following configurations are invalid because the class attribute references unsupported Solr request handler classes.

    <requestHandler name="/dataimport" class="solr.DataImportHandler"/>
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler"/>

  2. In solrconfig.xml, use the simplified class path for directoryFactory.

    For example, given the following configuration:

    <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>

    Change the class attribute to the simplified path:

    <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

  3. In solrconfig.xml, remove <unlockOnStartup> because it is no longer supported.

  4. Reload the core:

    dsetool reload_core keyspace_name.table_name reindex=false

Additional preparation for DSE Analytics nodes

Complete these additional pre-upgrade steps on DSE Analytics nodes while DSE 5.0 is still running.

DataStax recommends running nodetool repair and creating a fresh backup after making these changes, even if you already did so during the pre-upgrade steps.

  1. Recompile DSE 5.0 Scala Spark applications.

    DSE versions earlier than 5.1 use an older version of Spark (1.6). Applications written using Spark 1.6 might be incompatible with Spark 2.2. Therefore, you must recompile all DSE 5.0 Scala Spark applications against Scala 2.11 using only Scala 2.11 third-party libraries.

    Changing the dse-spark-dependencies in your build files is not sufficient to change the compilation target. For examples of how to set up your build files, see the DataStax Spark example projects.

  2. Your Spark applications must use dse:// URLs instead of spark://spark_master_IP:Spark_RPC_port_number URLs, as described in Specifying Spark URLs.

  3. Modify calls to setMaster and setAppName.

    For example, the following code works in DSE 5.0 but not DSE 5.1:

    val conf = new SparkConf(true)
.setMaster("spark://192.168.123.10:7077")
.setAppName("cassandra-demo")
.set("cassandra.connection.host" , "192.168.123.10") // initial contact
.set("cassandra.username", "cassandra")
.set("cassandra.password", "cassandra")
val sc = new SparkContext(conf)

    To connect in DSE 5.1, modify the call to setMaster:

    val conf = new SparkConf(true)
**.appName**("cassandra-demo")
**.master**("dse://192.168.123.10:7077")
.set("cassandra.connection.host" , "192.168.123.10") // initial contact
.set("cassandra.username", "cassandra")
.set("cassandra.password", "cassandra")
val sc = new SparkContext(conf)

Upgrade steps

Upgrade one node at a time in the recommended order. Following the recommended order minimizes downtime and ensures that you don’t miss any nodes.

  1. DSE Analytics nodes: Kill all Spark worker processes:

    for pid in $(jps | grep Worker | awk '{print $1}'); do kill -9 $pid; done

  2. Flush the commit log of the current installation:

    nodetool drain

    This step is mandatory for upgrades that include a major Cassandra version change because the previous version’s commit logs are incompatible with the new version’s SSTable format.

  3. Stop the node.

  4. Install the new DSE version, but don’t start DSE.

    Don’t start DSE yet. Before restarting the node, you must reconfigure DSE as explained in the next steps.

    If you use DSE OpsCenter, you can use Lifecycle Manager (LCM) to clone a configuration profile and run an upgrade job on a datacenter or node. This approach can help streamline the install and reconfiguration process.

    Use the same installation type as your current version. Mismatched installation types can cause problems with the upgrade.

  5. Migrate any custom connectors from your previous installation’s server.xml to the new installation’s server.xml.

    The new server.xml is for Apache Tomcat® 8.

  6. Configure the new DSE version by comparing the new configuration files to your backup configuration files.

    Specific modifications are described in the next steps. Additional changes depend on your workloads and previous configurations. For example:

    • Remove deprecated and removed settings.

    • Configure new settings.

    • Migrate previous custom configuration values to your new configuration files.

    • Modify previous configurations to account for changes described in the DSE release notes.

    You must use the new configuration files included with your new installation. Copy values from your old configuration files into the new files. Don’t overwrite entire files with the old files.

    For tarball installations of DSE 5.1 and later, some files previously located at INSTALL_DIRECTORY/conf moved to INSTALL_DIRECTORY/resources/cassandra/conf. For package installations, these files moved from /etc/cassandra to /etc/dse/cassandra. For default configuration file locations in DSE 5.1, see Default file locations for package and Installer-Services installations and Default file locations for tarball and Installer-No Services installations.

    You can use the yaml_diff tool to compare your backup YAML files with the new YAML files:

    cd /usr/share/dse/tools/yamls
    ./yaml_diff path/to/yaml-file-old path/to/yaml-file-new

    The output is a diff between the specified YAML files. For example:

    ...
 CHANGES
=========
authenticator:
- AllowAllAuthenticator
+ com.datastax.bdp.cassandra.auth.DseAuthenticator

authorizer:
- AllowAllAuthorizer
+ com.datastax.bdp.cassandra.auth.DseAuthorizer

roles_validity_in_ms:
- 2000
+ 120000
...

  7. In dse.yaml, under shard_transport_options, set netty_client_request_timeout to 60000, and then remove all other options under shard_transport_options:

    shard_transport_options:
  netty_client_request_timeout: 60000
# Remove deprecated shard_transport_options sub-options:
#  type: html
#  netty_server_port: 8984
#  netty_server_acceptor_threads:
#  netty_server_worker_threads:
#  netty_client_worker_threads:
#  netty_client_max_connections:
#  http_shard_client_conn_timeout: 0
#  http_shard_client_socket_timeout: 0

  8. To configure DSEFS, uncomment the dsefs_options section in dse.yaml.

    Starting in DSE 5.1.0, DSEFS is enabled by default but the dsefs_enabled section is commented out in dse.yaml.

  9. Upgrades to 5.1.0 through 5.1.15: If you are intentionally installing a version earlier than 5.1.16 and any tables use DSE Tiered Storage, then you must remove all txn_compaction log files from second-level tiers and lower.

    At versions 5.1.0 to 5.1.15, data loss can occur if you don’t remove the txn_compaction log files before starting DSE. Additionally, versions 5.1.0 to 5.1.7 are susceptible to the Year 2038 problem. To avoid these issues, DataStax strongly recommends that you upgrade to 5.1.16 or later.

    You can find the tiers and directories in the dse.yaml configuration file. For example, given the following tiered_storage_options configuration, you would remove txn_compaction log files from the /mnt2 and /mnt3 directories:

    tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

    The following example removes the files from the second and third tier directories using find commands:

    find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

  10. DSE Analytics nodes: If you have any datacenters running in Analytics Hadoop mode with DseSimpleSnitch, you must use the following options to restart upgraded nodes in those datacenters:

    • Nodes in Analytics Hadoop mode: Start these nodes in Spark mode.

      DSE Hadoop is removed in DSE 5.1. The byoh-env.sh file (located at /etc/dse/byoh-env.sh for package installations and INSTALL_DIRECTORY/bin/byoh-env.sh for tarball installations) isn’t present or used in DSE 5.1 and later.

    • Other nodes: Add the start-up parameter -Dcassandra.ignore_dc=true to the node, and then start the node in cassandra mode.

      This flag is required only once after upgrading. Subsequent restarts don’t use this flag. You can leave the flag in the configuration file or remove it after the first restart of each upgraded node.

  11. Start the node:

  12. Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition.

    This is only relevant when using NetworkTopologyStrategy.

    1. Get the node’s datacenter name:

      nodetool status | grep "Datacenter"

    2. Get the replication configuration for a keyspace:

      cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"

    3. If the datacenter name in replication does not match the node’s datacenter name, alter or recreate the keyspace accordingly:

      ALTER KEYSPACE keyspace-name WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter-name': '3'};

  13. Review the logs for warnings, errors, and exceptions:

    grep -w 'WARN\|ERROR\|exception' /var/log/cassandra/*.log

    If the default log location returns no results, check dse-env.sh for custom log locations.

    Warnings, errors, and exceptions are frequently found in the logs when starting an upgraded node. Some messages provide information to help you complete specific upgrade steps, and some messages can be ignored while the cluster is in a partially upgraded state. If you find warnings, errors, or exceptions that are unexpected or persist after upgrading the entire cluster, contact IBM Support.

    For DSE Advanced Replication nodes, WriteTimeoutExceptions are expected while the cluster is in a partially upgraded state. These exceptions are caused by write consistency limitations related to the upgrade process, and they should resolve after all nodes are upgraded.

    For DSE Graph nodes, warnings are logged for gremlin_server properties from earlier versions of DSE that are no longer required. For example:

    WARN [main] 2017-08-31 12:25:30,523 GREMLIN DseWebSocketChannelizer.java:149 - Configuration for the org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0 serializer in dse.yaml overrides the DSE default - typically it is best to allow DSE to configure these.

    These warnings are generated if any such properties exist in dse.yaml. You can ignore these warnings, or you can modify dse.yaml to use only the required Gremlin server properties.

  14. Repeat the upgrade process on each node in the cluster following the recommended order.

Post-upgrade steps

The post-upgrade steps include multi-node and cross-datacenter operations that can cause errors while a cluster is in a partially upgraded state. For example, repairs can propagate incompatible data between mixed-version nodes, and SSTable snapshots can be corrupted or unreadable due to mixed SSTable formats and versions.

Only start the post-upgrade steps after you upgrade all dependent nodes. Node dependencies are determined by your deployment architecture:

Architecture Examples Post-upgrade path

Isolated or non-production deployment

  • Upgrading a test deployment before rolling out the upgrade to production.

  • Upgrading a passive datacenter in an active-passive multi-datacenter deployment.

Start the post-upgrade steps after upgrading all nodes in the test deployment or passive datacenter.

For example, in a test deployment with two datacenters and cross-datacenter replication, you must upgrade all nodes in both datacenters before starting the post-upgrade steps.

In contrast, in a test deployment with one datacenter and no cross-datacenter connections, you must upgrade all nodes in that datacenter before starting the post-upgrade steps.

Production deployment

  • Rolling out an upgrade to an entire production cluster.

  • Upgrading a multi-datacenter deployment that has cross-datacenter replication.

Start the post-upgrade steps after upgrading all nodes included in the deployment.

For example, in a multi-datacenter deployment with cross-datacenter replication, you must upgrade all nodes in all datacenters before starting the post-upgrade steps.

Complete the post-upgrade steps progressively in the recommended order. Modify the recommended order as needed for your deployment architecture and workloads.

  1. Upgrade all SSTables again.

    Your database upgrade isn’t complete until you upgrade all SSTables.

    To avoid significant performance degradation and issues with incompatible SSTables formats, always upgrade SSTables before and after any database upgrade, including minor and patch releases.

    This process is recommended for all upgrades, and it is critical for upgrades that involve a major underlying Cassandra version change.

    On each node, run nodetool upgradesstables with the -a flag to ensure that all SSTables use the latest SSTable format and version for your current database version:

    nodetool upgradesstables -a

    DataStax recommends running the upgradesstables command on one node at a time or, when using racks, one rack at a time. Running the command on too many nodes at once can degrade performance.

    You can use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

    If all SSTables are already on the latest version and format, the command returns immediately and takes no action.

  2. If you encounter serialization-header errors, stop the node, and then repair the errors using the sstablescrub -e option:

    sstablescrub -e fix-only keyspace table

    For more information, see DSE 5.0 SSTables with UDTs corrupted after upgrading to DSE 5.1, 6.0, or 6.7.

  3. Review your security configuration.

    To use security, enable and configure DSE Unified Authentication.

    In cassandra.yaml, the default authenticator is DseAuthenticator and the default authorizer is DseAuthorizer. Security is disabled in dse.yaml by default.

  4. Change the compaction strategy for dse_perf and dse_audit_log tables that were created in earlier versions of DSE to use TimeWindowCompactionStrategy (TWCS):

    ALTER TABLE dse_perf.read_latency_histograms WITH COMPACTION={'class':'TimeWindowCompactionStrategy'};
    ALTER TABLE dse_audit_log.audit_log WITH COMPACTION={'class':'TimeWindowCompactionStrategy'};

    Starting in DSE 5.1, TWCS is used for new dse_perf and dse_audit_log tables. DataStax recommends this change to keep compaction consistent across all dse_perf and dse_audit_log tables.

  5. Drop the legacy tables system_auth.users, system_auth.credentials, and system_auth.permissions, if they exist:

    DROP TABLE IF EXISTS system_auth.users;
    DROP TABLE IF EXISTS system_auth.credentials;
    DROP TABLE IF EXISTS system_auth.permissions;

  6. Edit any scripts that use DSE file paths to the new file paths in DSE 5.1.

    For default file locations in DSE 5.1, see Default file locations for package and Installer-Services installations and Default file locations for tarball and Installer-No Services installations.

  7. Restart any services or automated operations that you disabled before starting the upgrade.

    For more information, see Avoid certain operations during the upgrade.

Post-upgrade steps for DSE Search nodes

The following steps require multiple Search schema changes followed by long-running reindexing operations. To minimize the impact of reindexing, complete all relevant schema changes before reindexing. Plan sufficient time after the upgrade to run this operation on all nodes.

  1. Update any commands or scripts that interact with logback.xml.

    For tarball installations of DSE 5.1 and later, logback.xml moved from INSTALL_DIRECTORY/resources/logback.xml to INSTALL_DIRECTORY/resources/cassandra/conf/logback.xml.

    For package installations, this file remains at /etc/dse/cassandra/logback.xml.

  2. Don’t attempt unbounded facet searches with facet.limit=-1.

    Starting in DSE 5.1.17, unbounded facet searches are no longer allowed using facet.limit=-1. The maximum facet limit value is 20,000 as set by solr.max.facet.limit.size. DataStax doesn’t recommend overriding the facet limit size using -Dsolr.max.facet.limit.size in the JVM options.

  3. Discontinue use of index time boost in CQL tables that back DSE Search indexes.

    Support for index time boost is removed in DSE 5.1.1. You can use query time boosting instead.

    You must drop _docBoost columns from all relevant CQL tables except Thrift tables, which cannot drop this column. Thrift tables with a _docBoost column are allowed but the _docBoost column is ignored.

    DELETE _docBoost FROM table-name IF EXISTS;

  4. If SpatialRecursivePrefixTreeFieldType (RPT) is used in the search schema, replace the units field type with a suitable distanceUnits (degrees, kilometers, or miles), and then verify that spatial queries behave as expected.

  5. For optimal indexing of multipolygon shapes, you must set useJtsMulti="false".

    For example:

    <fieldType autoIndex="true" useJtsMulti="false"
                class="solr.SpatialRecursivePrefixTreeFieldType" distErrPct="0.0125"
                distanceUnits="kilometers" geo="true" name="WktField" spatialContextFactory="org.locationtech.
                spatial4j.context.jts.JtsSpatialContextFactory"/>

  6. If you are using HTTP API writes with JSON documents (deprecated), change the auto-generated solrconfig.xml if necessary.

    A known issue can cause the auto-generated solrconfig.xml to have an invalid requestHandler for JSON core creations.

    For example, if the auto-generated solrconfig.xml has:

    <requestHandler name="/update/json" class="solr.UpdateUpdateRequestHandler" startup="lazy"/>

    Change the requestHandler element to:

    <requestHandler name="/update/json" class="solr.UpdateRequestHandler" startup="lazy"/>

  7. Migrate encrypted tables and indexes.

    Encrypted tables and indexes must be migrated after upgrading DSE.

Post-upgrade steps for DSE Analytics nodes

  1. If you are using Spark SQL tables, migrate them to the new Hive metastore format:

    dse spark-sql-metastore-migrate

  2. Configure authorization for the Spark web UI if security is enabled, as explained in Monitoring Apache Spark™ with the web interface.

  3. To use the new schema for DSEFS, you must create a new DSEFS keyspace, and then copy any existing DSEFS data into the new keyspace.

    Multi-datacenter clusters that use DSEFS must apply the new DSEFS schema because DSEFS isn’t designed to support multi-datacenter clusters.

    If you need to preserve any existing DSEFS data, create a backup before clearing the directories. If you created a backup of your installation and data before the upgrade, you can restore the data from that backup.

    No action is needed for single-datacenter clusters or multi-datacenter clusters without DSEFS. In these clusters, DSEFS will continue to work with the DSE 5.0 schema.

    1. Stop the node.

    2. Clear the DSEFS data directories on each node.

      DSEFS data directories are set in the dsefs_options section in dse.yaml. For example:

      dsefs_options:
     ...
     data_directories:
         - dir: /var/lib/dsefs/data

      The following command removes all data from the /var/lib/dsefs/data directory on the node:

      rm -r /var/lib/dsefs/data/*

    3. In the dsefs_options section of dse.yaml, change the keyspace_name parameter to a different name:

      ##########################
# DSE File System options
dsefs_options:
    ...
    keyspace-name: new_keyspace_name

      This creates a new DSEFS keyspace with the new schema when the node starts.

    4. Start the node:

    5. If you backed up existing DSEFS data before the upgrade, copy that data to your DSEFS data directory from your local backup store:

      dse hadoop fs -cp /local_backup_location/* /dsefs_data_directory/

    6. Optional: Drop the original dsefs keyspace:

      DROP KEYSPACE dsefs

    7. Repeat for each node.

Post-upgrade steps for DSE Advanced Replication

DSE 5.1 uses DSE Advanced Replication v2, which is substantially different from Advanced Replication v1 in earlier DSE versions. You must migrate to Advanced Replication v2 after upgrading to DSE 5.1.

To enable this migration, DSE 5.1 supports both Advanced Replication v1 and v2. However, new features are developed against v2 only, and DSE 6.x supports Advanced Replication v2 only.

To migrate from Advanced Replication v1 to v2:

  1. Create a v2 destination for the hub that you configured in v1:

    dse advrep destination create \
  --name dest_name \
  --addresses ip_address \
  --transmission-enabled false

  2. Create a v2 channel for each v1 channel:

    dse advrep channel create \
  --source-keyspace keyspace_name \
  --source-table table_name \
  --destination dest_name \
  --source-id source_id \
  --source-id-column source_column_id \
  --priority 1

  3. Resume collection and transmission:

    dse advrep channel resume \
  --source-keyspace keyspace_name \
  --source-table table_name \
  --collection-enabled true \
  --transmission-enabled true

  4. Make sure the v2 channels are running and replicating, and then disable collection on the v1 channels:

    dse advrep --v1 edge channel pause \
  --keyspace keyspace_name \
  --table table_name

  5. Wait for the v1 replication log to drain, proceeding only when the count reaches zero.

    To check for a zero count:

    dse advrep --v1 edge rl-count

  6. Delete v1 channels and disable the v1 hub.

    You can use a script to facilitate this process. For example:

    for f in `dse advrep --v1 edge list-conf|cut -f1 -d' '|sed 1,2d | sed s/_/-/g`; do dse advrep --v1 edge remove-conf --$f; done;

Lock DSE package versions

After upgrading a DSE package installation, you can prevent unintended upgrades by locking the package version:

Yum

  1. Install yum-versionlock if it isn’t already installed:

    sudo yum install yum-versionlock

  2. Lock the current DSE version:

    sudo yum versionlock dse-*

  3. When you need to allow an upgrade, clear the version lock:

    sudo yum versionlock clear
APT

  1. Use apt-mark to hold the dse package at the current version:

    sudo apt-mark hold dse-*

  2. When you need to allow an upgrade, remove the hold:

    sudo apt-mark unhold dse-*

See also

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM