Migrate from DataStax Enterprise (DSE) to Hyper-Converged Database (HCD) 1.2

You can migrate from DSE to HCD 1.2 using the Zero Downtime Migration (ZDM) tools or an in-place upgrade. This guide explains how to perform an in-place upgrade.

DataStax strongly recommends using the ZDM tools for this migration. The ZDM tools provide support for blue-green deployments and seamless rollback capabilities without the need for progressive cluster reconfiguration. DataStax recommends that you perform an in-place migration only if you cannot use the ZDM tools.

Upgrade path to avoid data loss

Improper upgrades can result in downtime, data loss, and other errors. Specifically, some databases with UDT or tuple data require special handling to avoid data loss.

Review the entire upgrade guide before you begin, including warnings, restrictions, release notes, and all pre-upgrade, upgrade, and post-upgrade steps.

The required migration path depends on your current DSE version. The greater the gap between the current version and the target version, the more complex the upgrade. Upgrades from earlier versions can require multiple intermediate upgrades.

Current DSE version UDT or tuple data In-place upgrade path Cross-cluster migration path (recommended)

DSE 6.9

Not relevant

  1. Upgrade to the latest DSE 6.9 patch release.

  2. Upgrade from DSE 6.9 to HCD 1.2, as explained in this guide.

Use ZDM tools.

DSE 6.8

No UDT or tuple data

  1. Upgrade to the latest DSE 6.8 patch release.

  2. Upgrade from DSE 6.8 to HCD 1.2, as explained in this guide.

Use ZDM tools.

DSE 6.8.0 to 6.8.34

Database has UDT or tuple data

You must upgrade to DSE 6.8.35 first to avoid loss of UDT and tuple data. Follow this upgrade path:

  1. Upgrade to exactly DSE 6.8.35 using the instructions in the DSE 6.8 patch release guide.

  2. Upgrade all SSTables with nodetool upgradesstables -a and run sstablescrub --fix.

  3. Upgrade to the latest DSE 6.8 patch release.

  4. Upgrade from DSE 6.8 to HCD 1.2, as explained in this guide.

Use ZDM tools.

DSE 6.8.35

Database has UDT or tuple data

DSE 6.8.35 is the boundary of the tuple data loss issue. Prepare your SSTables before upgrading further to avoid data loss.

  1. If you haven’t done so already, upgrade all SSTables with nodetool upgradesstables -a and run sstablescrub --fix.

  2. Upgrade to the latest DSE 6.8 patch release.

  3. Upgrade from DSE 6.8 to HCD 1.2, as explained in this guide.

Use ZDM tools.

DSE 6.8.36 and later

Database has UDT or tuple data

If you are running DSE 6.8.36 or later, you have already passed the tuple data loss boundary.

  1. Upgrade to the latest DSE 6.8 patch release.

  2. Upgrade from DSE 6.8 to HCD 1.2, as explained in this guide.

Use ZDM tools.

DSE 5.1

No UDT or tuple data

  1. Upgrade to the latest DSE 5.1 patch release.

  2. Upgrade from DSE 5.1 to HCD 1.2, as explained in this guide.

Use ZDM tools.

DSE 5.1

Database has UDT or tuple data

You must upgrade to DSE 6.8.35 first to avoid loss of UDT and tuple data. Follow this upgrade path:

  1. Upgrade from DSE 5.1 to DSE 6.8.35.

  2. Upgrade all SSTables with nodetool upgradesstables -a and run sstablescrub --fix.

  3. Upgrade to the latest DSE 6.8 patch release.

  4. Upgrade from DSE 6.8 to HCD 1.2, as explained in this guide.

Use ZDM tools.

Any unsupported DSE version (6.7, 6.0, 5.0, or earlier)

Not relevant

  1. Upgrade to DSE 5.1, 6.8, or 6.9.

  2. Follow the HCD migration path for your DSE version.

Some versions are compatible with ZDM tools.

Use storage port 7000 for online upgrades

Online upgrades require the default storage port 7000. Check the storage_port setting in your cluster’s cassandra.yaml or JVM options. If your cluster uses a non-default storage_port value, you must use the ZDM tools instead of an in-place upgrade.

Avoid certain operations during the upgrade

The upgrade process for HCD can be completed with minimal downtime (ideally zero) by upgrading nodes in a rolling fashion: upgrade and restart one node at a time while the other nodes continue to operate.

The moment that you install the new version on one node in a cluster, the cluster enters a partially upgraded state. In this state, the cluster continues to operate as though all nodes are running the earlier, pre-upgraded version. The cluster remains in this state until you have upgraded all nodes in the cluster.

You must avoid or block the following operations during the upgrade to maintain cluster stability and prevent errors:

  • Don’t enable new features.

    This restriction includes net new features and existing features that you weren’t using previously. For example, if don’t enable Change Data Capture (CDC) for the first time during an upgrade. Upgrade the entire cluster, and then enable the feature.

  • Don’t change credentials, permissions, or any other security settings unless explicitly instructed to do so.

    To preserve your existing configuration, you will transfer your existing security settings from your current configuration files to the new configuration files during the upgrade process. The new files might use different settings due to enhancements, deprecations, and removals between versions. This mapping is required for a successful upgrade, and it isn’t the same as an extraneous change to the security configuration.

    Examples of security changes to avoid include modifications to authenticators, authorizers, CQL roles, and CQL permissions. Postpone all such changes until after the upgrade.

  • Don’t run repairs manually or automatically:

    • Disable all automated and scheduled repairs, including Reaper, crontab, and scripts that run nodetool repair.

    • Don’t manually run commands like nodetool repair.

    • Don’t repair SSTables while the cluster is in a partially upgraded state. Only repair SSTables before and after upgrading the entire cluster.

    You must complete the cluster-wide upgrade before the expiration of gc_grace_seconds (approximately 10 days by default) to ensure any post-upgrade repairs complete successfully.

    Attempting to run repairs on partially upgraded clusters can cause performance issues, data inconsistencies, and repair failures. If you absolutely must run repairs on a partially upgraded cluster, determine which SSTable formats and streaming protocols are supported on each node, and then thoroughly test your intended repair operation on an isolated non-production cluster. Small incremental repairs are less risky but they aren’t risk free.

  • Don’t add (bootstrap) new nodes or decommission existing nodes.

  • Don’t issue DDL-related queries like TRUNCATE.

    Depending on your infrastructure, workloads, and consistency levels, you might want to avoid other operations that could require excessive resources or coordination across mixed-version nodes. For example, BATCH operations can be resource intensive.

  • Don’t alter any schemas.

    Schema versions must be consistent across the cluster before starting the upgrade.

    Propagation of schema changes between mixed-version nodes can cause query failures, data consistency issues, and other errors. Take action to prevent schema changes from occurring during the upgrade process.

    During the upgrade, it is normal for nodes on different versions to show schema disagreements. These disagreements should resolve after the entire cluster is upgraded.

  • Don’t rebuild indexes during the upgrade.

  • Disable services that can cause data inconsistencies and interfere with the upgrade process:

    • Disable backup services, scheduled backups, and automated snapshots.

    • Disable nodesync.

  • Disable DSE Management Services that aren’t available in HCD. Don’t attempt to reenable these services after the migration. Mission Control is the recommended way to access these types of services with HCD.

Driver compatibility and live application traffic

Incompatible drivers and unspecified protocol versions can cause connection failures and other issues during the upgrade.

To maintain live application traffic throughout the upgrade process, your Cassandra driver must be compatible with your current installation and your target HCD version. Additionally, you must configure protocol version negotiation to ensure backwards compatibility throughout the upgrade.

Find a mutually compatible driver version

Verify that your current Cassandra driver version is compatible with your target HCD version. For available drivers and compatibility information, see Cassandra driver compatibility.

If your current driver version isn’t compatible with your target HCD version, upgrade your driver to a mutually compatible version.

When upgrading your driver, check your driver’s release notes for breaking changes that might require code changes. Review all release notes from your current driver version through the new version. Recompile your client application code, if needed.

No mutually compatible version exists

If there is no mutually compatible driver version, you have two options:

  • Intermediate upgrade (recommended): Determine if your upgrade path requires (or supports) an intermediate upgrade before proceeding to your target HCD version. Find driver versions that are compatible with each segment of the upgrade path (current to intermediate and intermediate to target), and then upgrade your driver before each segment.

  • Delayed upgrade: Upgrade to the latest driver version that is compatible with your current installation, and then plan to upgrade your driver again when your cluster is on your target HCD version. This can require code or configuration changes to accommodate the temporary driver version, such as changing authentication methods or avoiding unsupported query patterns. Consider testing this approach in an isolated environment to identify potential compatibility issues before applying it to your production environment.

Configure protocol negotiation

When a cluster is in a partially upgraded state, upgraded nodes might support different Cassandra native protocol versions than non-upgraded nodes. By default, a driver negotiates the protocol version with the first host (node) that it connects to. The driver might select a protocol version that works across all nodes, but this isn’t guaranteed if the driver connects to a random node each time. It is better to prevent protocol incompatibility in one of two ways:

  • Set protocol version: Set a mutually compatible protocol version explicitly in your application at start up. After you upgrade all nodes in the cluster, you can switch your applications to the latest supported protocol version.

  • Set initial contact points: Specify a list of initial contact points that contains only hosts with the oldest database platform version or mutually compatible protocol version. You must upgrade these nodes last.

For more information about protocol version negotiation, see your driver’s documentation.

DSE-only drivers are unsupported

All DSE-only drivers have reached end-of-life (EOL); they are unsupported and unmaintained. If your applications use DSE-only drivers, you must upgrade to a recent version of a supported driver.

For legacy applications that require DSE-only drivers, compatibility with your target HCD version might be limited or nonexistent. For example, newer HCD, Cassandra, and CQL features might not be supported by the driver, or queries that use deprecated data types, functions, or syntax might fail. Consider modernizing or sunsetting legacy applications if possible, particularly because EOL drivers don’t receive any updates, including security updates.

Strict upgrade order for cluster stability

An in-place upgrade requires that you fully install and configure HCD 1.2 on each node in your cluster. This process completely replaces the database software while preserving your existing data.

For cluster stability, you must upgrade and restart one node at a time in a specific order based on your deployment architecture:

  1. For multi-cluster deployments, upgrade one cluster at a time.

  2. For multi-datacenter clusters, upgrade one datacenter at a time.

  3. If using racks, upgrade one rack at a time within a datacenter.

  4. Within a datacenter (or rack), upgrade one node at a time, starting with seed nodes.

    Optionally, advanced users can consider upgrading nodes in parallel under the following conditions:

    • If using racks, you can upgrade nodes in parallel only when using NetworkTopologyStrategy.

    • For multi-datacenter clusters, you can upgrade datacenters in parallel only when using LOCAL_* consistency levels.

  5. Repeat until you have upgraded all nodes in all racks, datacenters, and clusters.

Test before upgrading in production

DataStax strongly recommends testing the entire upgrade process (pre-upgrade, upgrade, and post-upgrade steps) on an isolated node, datacenter, or cluster before upgrading in production.

Testing helps you determine the impact of the changes on your existing installation and the required configuration changes you need to make before, during, and after the upgrade.

A full explanation of migration testing is outside the scope of this guide. As an example, in a multi-datacenter cluster with an active-passive configuration, you can upgrade the passive datacenter first, run performance tests on that datacenter, and then upgrade the active datacenter if the tests are successful. If the tests fail, you can destroy the passive datacenter and rebuild it without impacting the active datacenter.

Major differences between DSE and HCD

There are significant differences between DSE and HCD that you must address before, during, and after the migration process. The following table summarizes notable differences that require your action:

Difference Severity Action

No support for DSE advanced workloads

Critical

See DSE advanced workloads aren’t supported

No support for COMPACT STORAGE tables

Critical

See COMPACT STORAGE tables aren’t supported

Data directories commented out by default

Critical

Addressed during the Upgrade steps

Different default token count (vnodes) per node

Critical

Addressed during the Upgrade steps

Different authenticator and authorizer classes

Moderate

See Authentication and authorization changes

No support for DSE Management Services

Moderate

See Avoid certain operations during the upgrade

Different management and service commands

Low

Addressed during the Upgrade steps

Different default file paths and directory names

Low

Addressed during the Upgrade steps

A list of all possible configuration differences is outside the scope of this guide. For more information about HCD features, review the HCD 1.2 release notes, and make note of any changes that might be incompatible with your existing installation. If you have specific concerns, contact IBM Support.

DSE advanced workloads aren’t supported

Data loss and application failures will occur if you migrate to HCD without addressing advanced workloads.

HCD doesn’t support any DSE advanced workloads (DSE Search, DSE Analytics, and DSE Graph).

If you use advanced workloads, you must make all necessary schema and application changes to migrate off of these features before migrating to HCD:

  1. Identify all applications using these workloads.

  2. Plan alternative solutions, such as an external Spark cluster.

  3. Migrate data to supported formats within your DSE cluster.

    All HCD nodes are transactional (Cassandra) nodes.

  4. Update application code to use new APIs.

For assistance, contact IBM Support.

Authentication and authorization changes

DSE uses DSE-specific authenticator and authorizer classes, whereas HCD uses the DataStax unified security classes.

HCD automatically handles this transition, but you must verify that authentication works after migration to ensure that existing users can connect and permissions are preserved.

Notably, HCD doesn’t support Kerberos authentication. If you use only Kerberos authentication (not recommended), you must set a backup scheme before migrating:

  1. In your DSE cassandra.yaml file, find the authenticator settings.

  2. If default_scheme is set to kerberos, make sure additional_scheme has at least one other authentication scheme defined.

DataStax recommends always setting internal as an additional scheme to allow access to the database superuser role. This prevents complete lockout in the event your primary authentication scheme is misconfigured or fails.

COMPACT STORAGE tables aren’t supported

For migrations from DSE 5.1, you must migrate all non-system tables that use COMPACT STORAGE to the standard CQL table format.

The COMPACT STORAGE format isn’t supported in HCD 1.2, DSE 6.8, or DSE 6.9. HCD won’t start if COMPACT STORAGE tables are present.

  1. Get the schema so you can check for tables using COMPACT STORAGE.

    The following command outputs the full schema to a file named schema_file:

    cqlsh -e 'DESCRIBE FULL SCHEMA;' > schema_file

  2. If any non-system tables use COMPACT STORAGE, use ALTER TABLE to drop COMPACT STORAGE from those tables:

    ALTER TABLE keyspace.table DROP COMPACT STORAGE;

If you have a lot of tables with COMPACT STORAGE, you can use a script to automate this process. The following example reads the schema_file generated in the previous steps, uses grep to find lines containing COMPACT STORAGE, and then writes the required ALTER TABLE commands to a file named schema-drop-list:

cat schema_file | while read -d $';\n' line ; do
  if echo "$line"|grep 'COMPACT STORAGE' 2>&1 > /dev/null ; then
    TBL="`echo $line|sed -e 's|^CREATE TABLE \([^ ]*\) .*$|\1|'`"
    if echo "$TBL"|egrep -v '^system' 2>&1 > /dev/null; then
      echo "ALTER TABLE $TBL DROP COMPACT STORAGE;" >> schema-drop-list
    fi
  fi
done

Then, you can run the ALTER TABLE commands individually by copying them from schema-drop-list, or you can pass the entire schema-drop-list file to cqlsh to run all of the ALTER TABLE commands:

cqlsh -f schema-drop-list

Prepare to upgrade

Don’t upgrade any nodes until you have completed the pre-upgrade steps on all nodes.

Complete the pre-upgrade steps on every node in the recommended order. Following the recommended order ensures that you don’t miss any nodes, although some steps aren’t applicable to every node.

Some pre-upgrade steps provide information about actions that you must take later in the upgrade process. This information is given in advance because it might impact your upgrade plan. Be sure to revisit these steps at the appropriate time.

  1. Make sure you have root or sudo access to all cluster nodes.

  2. Make sure there is network connectivity between all nodes.

  3. Upgrade to the latest patch release of DSE 5.1, 6.8 or 6.9, if you aren’t already running it.

    Patch releases help prepare your clusters for the upgrade by minimizing the differences between the current version and the target version.

    1. Get your current DSE version with dse -v.

    2. Compare the output with the latest version in the DSE release notes.

    3. If you aren’t on the latest patch release, upgrade all nodes to the latest patch release before applying any further upgrades.

  4. Review and address all major differences between DSE and HCD, if you haven’t done so already.

    Changes are required before, during, and after the migration. The migration will fail if these differences aren’t addressed at the appropriate time.

  5. Make sure each node has adequate free disk space for the upgrade.

    DataStax recommends a minimum of 50 percent free space. The required space depends on the compaction strategy: HCD requires free disk space matching your x largest STCS tables, where x is the value of concurrent_compactors (typically between two and eight). For more information, see Disk space.

    1. Get the size of the DSE data directory:

      sudo du -sh /var/lib/cassandra/data/

    2. Check available space on disk:

      sudo df -hT /

      The following output includes one disk named /dev/sda1. Your output depends on your disk configuration.

      Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   59G   16G   41G  28% /

    3. If necessary, make adjustments to your disks or data directory configuration to allow for more disk space.

  6. Install Java 11 and set $JAVA_HOME for compatibility with HCD 1.2:

    Migration path Java compatibility Required action

    DSE 6.9 to HCD 1.2

    Both use Java 11.

    Make sure $JAVA_HOME is set to Java 11.

    DSE 6.8 to HCD 1.2

    DSE 6.8 supports Java 8 and Java 11.

    Make sure Java 11 is installed, and set $JAVA_HOME to Java 11.

    DSE 5.1 to HCD 1.2

    DSE 5.1 supports only Java 8.

    Install both Java 8 and Java 11. For $JAVA_HOME, see the following steps.

    1. Get your current Java version:

      java -version

    2. Make sure the output shows Java 11. If not, install Java 11.

      For example:

      openjdk version "11.0.25" 2024-10-15 LTS
OpenJDK Runtime Environment (build 11.0.25+9-LTS-256)
OpenJDK 64-Bit Server VM (build 11.0.25+9-LTS-256, mixed mode)

    3. Set the $JAVA_HOME environment variable to your Java 11 runtime:

      export JAVA_HOME=INSTALL_DIRECTORY/openjdk11
export PATH=$JAVA_HOME/bin:$PATH

    4. If you changed $JAVA_HOME, restart the node.

    If DSE 5.1 nodes experience issues when $JAVA_HOME is set to Java 11, set $JAVA_HOME to Java 8 and restart the nodes. When you install HCD, set $JAVA_HOME to Java 11 on each upgraded node.

    To avoid Java runtime management when migrating from DSE 5.1, you can upgrade from DSE 5.1 to 6.8 and then upgrade from DSE 6.8 to HCD. Because HCD 6.8 supports both Java 8 and Java 11, you can use Java 8 for the first upgrade, and then use Java 11 for the second upgrade.

Upgrade SSTables and run repairs

  1. Upgrade all SSTables on all nodes.

    To avoid significant performance degradation and issues with incompatible SSTables formats, always upgrade SSTables before and after any database upgrade, including minor and patch releases.

    This process is recommended for all upgrades, and it is critical for upgrades that involve a major underlying Cassandra version change.

    On each node, run nodetool upgradesstables with the -a flag to ensure that all SSTables use the latest SSTable format and version for your current database version:

    nodetool upgradesstables -a

    DataStax recommends running the upgradesstables command on one node at a time or, when using racks, one rack at a time. Running the command on too many nodes at once can degrade performance.

    You can use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

    If all SSTables are already on the latest version and format, the command returns immediately and takes no action.

  2. Run nodetool repair to ensure that data on each replica is consistent with data on other nodes.

    Run repairs as close to the beginning of your upgrade as possible.

    nodetool repair -pr

    Repairing nodes on a regular basis minimizes the overhead and time required for repair operations.

Back up your existing files and data

A complete backup is required to reconfigure the new HCD installation and, if needed, restore lost data or rollback to DSE.

Back up all DSE data and configuration files before starting the upgrade process:

  • Create SSTable snapshots for all keyspaces and tables.

    If you use automatic backup services, make sure the backup process runs after you upgrade SSTables.

    To manually create SSTable snapshots, run nodetool snapshot.

  • Backup logs, including commitlogs.

  • Backup configuration files.

    Store backup configuration files separately from your existing DSE installation to ensure that they aren’t unintentionally overwritten when you install HCD.

If your upgrade requires significant pre-upgrade changes, consider creating a backup before making those changes and again immediately before upgrading the first node. Create the final backup as close to the beginning of your upgrade as possible to ensure your backup contains the most recent configuration files and data.

For manual backup instructions, see Backing up a tarball installation or Backing up a package installation. If your existing cluster is managed by Mission Control, you can use Mission Control to automatically create backups.

Upgrade steps

Upgrade one node at a time in the recommended order. Following the recommended order minimizes downtime and ensures that you don’t miss any nodes.

  1. Flush the commit log of the current installation:

    nodetool drain

  2. Stop the DSE service:

    DSE package installations
    sudo service dse stop
    DSE tarball installations
    INSTALL_DIRECTORY/bin/dse cassandra-stop

  3. Verify that the node is using a supported platform.

  4. Install HCD in the same way that you installed DSE (package or tarball).

    Don’t start HCD yet.

    Before you start HCD, you must configure HCD for compatibility with your DSE installation, as explained in the next steps. Otherwise, HCD cannot find your existing data or join the cluster correctly.

    Use the same installation type that you used for DSE (package or tarball).

    HCD is installed alongside your existing installation.

  5. In the HCD 1.2 cassandra.yaml, update num_tokens to match your DSE configuration.

    HCD defaults to 16 tokens per node, which might not match your DSE configuration. If token counts don’t match, HCD fails to start with Cannot change the number of tokens from X to Y.

    The following commands get the current token count from the DSE configuration, and then update the HCD configuration to match. To use these commands, modify them for your environment and file paths.

    # Check DSE token count
existing_num_tokens=$(grep "num_tokens" /path/to/dse/resources/cassandra/conf/cassandra.yaml)
# Update new cassandra.yaml to match
sed -i "s/num_tokens: 16/num_tokens: $\{existing_num_tokens}/" INSTALL_DIRECTORY/resources/cassandra/conf/cassandra.yaml

  6. In the HCD 1.2 cassandra.yaml, uncomment and set the *_directory parameters to the same directories you used for DSE.

    Data directories are commented out by default in HCD. To ensure that HCD can find and use your existing DSE data directories and the data stored within them, you must uncomment and explicitly set them to match your previous configuration.

    The following example uses default values. Set these parameters to the actual paths as set in your DSE cassandra.yaml file.

    data_file_directories:
	 - /var/lib/cassandra/data

metadata_directory: /var/lib/cassandra/metadata

commitlog_directory: /var/lib/cassandra/commitlog

hints_directory: /var/lib/cassandra/hints

cdc_raw_directory: /var/lib/cassandra/cdc_raw

  7. In the HCD JVM options, set -Dcassandra.ignore_dc=true to bypass datacenter name validation during startup:

    echo 'JVM_OPTS="$JVM_OPTS -Dcassandra.ignore_dc=true"' >> INSTALL_DIRECTORY/resources/cassandra/conf/cassandra-env.sh

    If datacenter names don’t match between your DSE and HCD configurations, HCD fails to start with Cannot start node if snitch’s datacenter differs from previous datacenter.

    DSE and HCD use the same default datacenter names depending on the snitch configuration: datacenter1 for SimpleSnitch (or DseSimpleSnitch in DSE) and dc1 for GossipingPropertyFileSnitch (or as set in cassandra-rackdc.properties). Even if you use the default names, the ignore_dc flag still serves as a failsafe against this startup error.

  8. Compare your backup DSE configuration files with the new HCD configuration files, and then restore or reconfigure any deprecated, removed, changed, or moved settings.

    Specific changes are described in the next steps and Major differences between DSE and HCD. The extent of your changes depends on your configuration settings.

    Don’t overwrite entire configuration files with your backup configuration files. You must compare each backup configuration file with the new configuration file, and then manually copy or reconfigure your previous settings.

    The yaml_diff tool isn’t available in HCD installations. You must use another comparison tool or manually compare the backup YAML files with the upgraded YAML files. For example:

    # Generate a diff to compare cassandra.yaml files manually
diff /etc/dse/cassandra/cassandra.yaml.backup INSTALL_DIRECTORY/resources/cassandra/conf/cassandra.yaml

    HCD has different default file paths and directory names than DSE. For example, /etc/dse/cassandra/ is replaced by resources/cassandra/conf/.

  9. Start the node and verify that it’s running correctly:

    INSTALL_DIRECTORY/bin/hcd cassandra

  10. Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition.

    Compared to DSE, HCD uses different management and service commands, and HCD uses direct binary commands rather than system services. Where you previously ran commands with dse, you now use hcd. For example, hcd nodetool instead of dse nodetool, and hcd start instead of systemctl start dse.

    1. Get the node’s datacenter name:

      INSTALL_DIRECTORY/bin/hcd nodetool status | grep "Datacenter"

    2. Verify that the node’s datacenter name matches the datacenter name for a keyspace:

      INSTALL_DIRECTORY/bin/hcd cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"

  11. Review the logs for warnings, errors, and exceptions:

    grep -w 'WARNING\|ERROR\|exception' INSTALL_DIRECTORY/logs/*.log

    Warnings, errors, and exceptions are frequently found in the logs when starting an upgraded node. Some messages provide information to help you complete specific upgrade steps, and some messages can be ignored while the cluster is in a partially upgraded state. If you find warnings, errors, or exceptions that are unexpected or persist after upgrading the entire cluster, contact IBM Support.

    In DSE, the default log location is /var/log/cassandra/. In HCD, the default log location is either /var/log/cassandra/ or INSTALL_DIRECTORY/logs/. Non-default log locations are configured in hcd-env.sh.

  12. Repeat the upgrade process on each node in the cluster following the recommended order.

Post-upgrade steps

The post-upgrade steps include multi-node and cross-datacenter operations that can cause errors while a cluster is in a partially upgraded state. For example, repairs can propagate incompatible data between mixed-version nodes, and SSTable snapshots can be corrupted or unreadable due to mixed SSTable formats and versions.

Only start the post-upgrade steps after you upgrade all dependent nodes. Node dependencies are determined by your deployment architecture:

Architecture Examples Post-upgrade path

Isolated or non-production deployment

  • Upgrading a test deployment before rolling out the upgrade to production.

  • Upgrading a passive datacenter in an active-passive multi-datacenter deployment.

Start the post-upgrade steps after upgrading all nodes in the test deployment or passive datacenter.

For example, in a test deployment with two datacenters and cross-datacenter replication, you must upgrade all nodes in both datacenters before starting the post-upgrade steps.

In contrast, in a test deployment with one datacenter and no cross-datacenter connections, you must upgrade all nodes in that datacenter before starting the post-upgrade steps.

Production deployment

  • Rolling out an upgrade to an entire production cluster.

  • Upgrading a multi-datacenter deployment that has cross-datacenter replication.

Start the post-upgrade steps after upgrading all nodes included in the deployment.

For example, in a multi-datacenter deployment with cross-datacenter replication, you must upgrade all nodes in all datacenters before starting the post-upgrade steps.

Complete the post-upgrade steps progressively in the recommended order. Modify the recommended order as needed for your deployment architecture and workloads.

  1. Upgrade all SSTables again.

    Your database upgrade isn’t complete until you upgrade all SSTables.

    To avoid significant performance degradation and issues with incompatible SSTables formats, always upgrade SSTables before and after any database upgrade, including minor and patch releases.

    This process is recommended for all upgrades, and it is critical for upgrades that involve a major underlying Cassandra version change.

    On each node, run nodetool upgradesstables with the -a flag to ensure that all SSTables use the latest SSTable format and version for your current database version:

    nodetool upgradesstables -a

    DataStax recommends running the upgradesstables command on one node at a time or, when using racks, one rack at a time. Running the command on too many nodes at once can degrade performance.

    You can use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

    If all SSTables are already on the latest version and format, the command returns immediately and takes no action.

  2. Run the following commands to verify that HCD is running correctly:

    # Check node status
# Make sure all nodes show UN (Up/Normal) status
./bin/hcd nodetool status

# Verify all original keyspaces are accessible
./bin/hcd cqlsh --execute "DESCRIBE KEYSPACES;"

# Confirm data integrity
# Make sure data load matches pre-migration values
./bin/hcd cqlsh --execute "SELECT count(*) FROM system.local;"

# Check that DSE-specific keyspaces are preserved
./bin/hcd cqlsh --execute "SELECT keyspace_name FROM system_schema.keyspaces WHERE keyspace_name LIKE 'dse_%';"

  3. Based on the output from the previous commands and other information provided by your HCD installation, make sure the following are true:

    • HCD node status shows UN (Up/Normal)

    • All original keyspaces are accessible

    • Data load matches pre-migration values

    • Host ID and tokens are preserved

    • You have upgraded SSTables

    • Authentication and authorization work correctly

    • No critical errors in logs

    If all conditions are true, then the migration was successful. If any conditions are false, see Troubleshoot migration.

Troubleshoot migration

You might encounter these issues during migration:

Cannot change the number of tokens

  • Cause: Token count mismatch between DSE and HCD configurations.

  • Solution: Update num_tokens in the HCD cassandra.yaml to match the original DSE configuration.

Cannot start node if snitch’s datacenter differs

  • Cause: Datacenter name mismatch between DSE and HCD.

  • Solution: Add -Dcassandra.ignore_dc=true to JVM options in cassandra-env.sh.

Connection refused

  • Cause: HCD failed to start due to configuration issues.

  • Solution: Check logs for specific error messages and address configuration compatibility issues.

To help debug issues, check the HCD logs for detailed error information:

# Check system logs
tail -50 /var/log/cassandra/system.log | grep -E "(ERROR|WARN|Exception|Failed|Cannot start|Fatal)"

# Check debug logs for more details
tail -50 /var/log/cassandra/debug.log | grep -E "(ERROR|WARN|Exception|Failed)"

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM