Migrate Hyper-Converged Database (HCD) 1.2 or later to HCD 2.0

You can migrate from HCD 1.2 or later to HCD 2.0 using the Zero Downtime Migration (ZDM) tools or an in-place upgrade. This guide explains how to perform an in-place upgrade.

DataStax strongly recommends using the ZDM tools for this migration. The ZDM tools provide support for blue-green deployments and seamless rollback capabilities without the need for progressive cluster reconfiguration.

Only perform an in-place upgrade if you cannot use the ZDM tools.

Migration warnings

This section contains important information you must understand before beginning the migration process. To improve your chances of a successful, error-free migration, review these limitations and considerations, taking action or planning ahead as needed.

Use storage port 7000 for online upgrades

Online upgrades require the default storage port 7000. A cluster that uses non-default storage_port values must use the ZDM tools instead of an in-place upgrade.

Verify your storage port configuration before you begin the upgrade process.

Avoid certain operations while nodes are partially upgraded

Don’t change credentials, permissions, or any other security settings.

Complete the cluster-wide upgrade before the expiration of gc_grace_seconds(approximately 13 days) to ensure any repairs complete successfully.

When you upgrade one or more nodes in a cluster, but before all nodes in the cluster run the new version, the cluster enters a partially upgraded state. In this state, the cluster continues to operate as though it runs on the earlier, pre-upgraded version.

Certain restrictions and limitations apply to a cluster in a partially upgraded state. During upgrade, and while a cluster is in a partially upgraded state:

  • Don’t enable new features.

  • Don’t run repairs.

    Before beginning the upgrade process, you should disable all automated/scheduled repairs. This includes disabling tools like Reaper, crontab, and any scripts that call nodetool repair.

  • Don’t repair SSTables during the upgrade process. However, DataStax recommends that you run regular repairs before starting the upgrade.

  • Don’t add/bootstrap new nodes to the cluster or decommission any existing nodes.

  • Don’t issue TRUNCATE or other DDL-related queries.

  • Do not alter schemas for any workloads.

    Propagation of schema changes between mixed-version nodes can have unexpected results. Take action to prevent schema changes from occurring during the upgrade process.

    It’s normal for nodes on different versions to show schema disagreements during the upgrade.

Check your driver version to avoid connection failures

Before you upgrade your clusters, your Cassandra drivers must be compatible with your current installation and your target HCD version. This means the driver language and version must be compatible with both installations. Incompatible driver versions can cause connection failures and other issues.

For available drivers, see Cassandra driver compatibility. Check the driver’s documentation to find a suitable version for your migration path, and then upgrade your driver before migrating to HCD:

  • Depending on the new driver version, you might need to recompile your client application code. Check your driver’s release notes for breaking changes that could require code changes. Review all release notes from your current driver version through the new version.

  • If your migration path requires an interim cluster upgrade, look for a driver version that is compatible with your current, interim, and target versions. If no such version exists, you must upgrade your driver before each cluster upgrade, choosing a compatible version for each phase.

  • If there is no minimum compatible version, you have two options:

    • Reevaluate your migration plan to determine if you need an interim cluster upgrade before migrating to your target HCD version. This approach allows you to upgrade the driver before each cluster upgrade, which maintains compatibility throughout the migration.

    • Upgrade to the latest driver version that is compatible with your current installation, and then plan to upgrade your driver again soon after the cluster migration. This can require code or configuration changes to accommodate the temporary driver version, such as using different cluster authentication methods or disabling certain queries.

  • All DSE-only drivers have reached end-of-life (EOL), and they are unsupported. If your applications use DSE-only drivers, you must upgrade to a recent version of a supported driver before migrating your clusters. For legacy applications that require DSE-only drivers, migration might be impossible until you mitigate this dependency. For example, you might choose to rewrite the application to use a supported driver. Alternatively, you might decide to sunset the application, migrating only the databases to HCD for historical purposes.

During upgrades, clusters with mixed driver versions might experience driver-specific impacts. If your cluster has mixed versions, the driver negotiates the protocol version with the first host to which it connects. Some drivers automatically select a protocol version that works across nodes. To avoid driver version incompatibility during upgrades, use these workarounds:

  • Protocol version: Set the protocol version explicitly in your application at start up. Switch to the new protocol version only after you complete the upgrade on all nodes in the cluster.

  • Initial contact points: Ensure that the list of initial contact points contains only hosts with the oldest database platform version or protocol version. For example, the initial contact points contain only protocol version 2.

For more information about protocol version negotiation, see your driver’s documentation.

Testing and upgrade restrictions

Carefully test upgrades in staging environments and on the first production node. You might find edge-case bugs during the upgrade process. Prepare to contact IBM Support for assistance.

Consider the following before you begin the upgrade process:

  • Ensure that schema versions are consistent across the cluster before starting the upgrade

  • Turn off snapshots during the whole upgrade process

Prepare to upgrade

This section contains all the steps you must complete to prepare your infrastructure and clusters before beginning the actual upgrade process. You must complete these preparations for a successful migration.

Ensure that you have the following prerequisites before you begin the upgrade process:

  • The latest version of HCD 1.2.x installed and running. See the HCD release notes for a version list.

  • Adequate disk space. DataStax recommends a minimum of 50% free space. HCD requires free disk space matching your x largest STCS tables, where x is concurrent_compactors, typically 2-8.

  • A backup of your current HCD installation.

  • Root or sudo access to all cluster nodes.

  • Network connectivity between all nodes.

Disable services that can cause data inconsistencies

To reduce the chance of a migration failure, data loss, or startup failure, disable the following services during the entire upgrade process:

  • Backups: Disable all backup services and scheduled backups.

  • Snapshots: Turn off all snapshot operations.

These services might interfere with the upgrade process and cause data inconsistencies.

Pre-upgrade preparation steps

To prepare each node for the upgrade, do the following:

  1. Verify that the HCD version is the latest:

    bin/hcd -v

    See the HCD release notes for a version list.

  2. Before upgrading, verify that each node has adequate free disk space.

    Determine the current HCD data disk space usage:

    sudo du -sh /var/lib/cassandra/data/
3.9G	/var/lib/cassandra/data/

    Determine available disk space:

    sudo df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   59G   16G   41G  28% /

    The required space depends on the compaction strategy.

  3. Run nodetool repair to ensure that data on each replica is consistent with data on other nodes:

    nodetool repair -pr

    Make sure to have run repairs recently, before starting the upgrade This step ensures data consistency across the cluster before the upgrade process begins.

Upgrade and backup SSTables to avoid data loss

DataStax recommends running the upgradesstables command on one node at a time or, when using racks, one rack at a time.

Upgrade the SSTables on each node to ensure that all SSTables run on the current version:

nodetool upgradesstables

If you fail to upgrade SSTables when required, you will experience a significant performance impact and increased disk usage.

Use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

If the SSTables already run on the current version, the command returns immediately and takes no action.

Verify Java runtime and back up configuration files

  1. Verify the Java runtime version and upgrade to the recommended version:

    java -version
openjdk version "11.0.25" 2024-10-15 LTS
OpenJDK Runtime Environment (build 11.0.25+9-LTS-256)
OpenJDK 64-Bit Server VM (build 11.0.25+9-LTS-256, mixed mode)

    HCD 1.2 supports Java 11, and HCD 2.0 supports Java 11 and 17. When upgrading to HCD 2.0, you must use Java 11. You can install and use Java 17 after completing the upgrade. DataStax recommends upgrading to the latest version of OpenJDK 17.

  2. Back up any customized configuration files since the new version installation might overwrite them with default values.

    If you backed up your installation using the instructions in Back up your existing installation, the backup includes your original configuration files.

Back up your existing installation

Back up your data prior to any version upgrade.

A backup enables you to revert and restore all the data that you used in the previous version if necessary.

Instead of manual processes, automate the management of enterprise-wide backup and restore cluster operations using Mission Control.

Understand data directory and configuration handling

During the upgrade process, HCD 2.0 will automatically use the existing data directories from your HCD 1.2 installation.

The default data directory locations are:

  • Data files: $CASSANDRA_HOME/data/data (or as configured in cassandra.yaml)

  • Commit logs: $CASSANDRA_HOME/data/commitlog

  • Saved caches: $CASSANDRA_HOME/data/saved_caches

  • Do not move or modify data files during the upgrade process

  • Configuration files (cassandra.yaml, cassandra-env.sh) from HCD 1.2 are compatible with HCD 2.0

  • Review any custom configuration settings after upgrade to ensure they are still appropriate

Upgrade to HCD 2.0

After completing all preparatory steps, you are ready to upgrade your clusters to HCD 2.0.

An in-place upgrade requires that you fully install and configure HCD 2.0 on each node in your cluster. This process completely replaces the database software while preserving your existing data.

Upgrade order

Upgrade each node, one at a time, in the following order:

  1. Move from node to node within one rack.

    Advanced users can upgrade nodes in parallel only when using NetworkTopologyStrategy.

  2. Move from rack to rack within one datacenter.

  3. Move from datacenter to datacenter within one cluster.

    Advanced users can upgrade datacenters in parallel only when using LOCAL_* consistency levels.

  4. Move to the next cluster, repeating the upgrade process by node, rack, and datacenter.

  5. Repeat until you have upgraded all clusters.

Upgrade one node to HCD

Complete then entire upgrade process for one node before moving to the next node in the recommended upgrade order.

To upgrade each node, do the following:

  1. Flush the commit log of the HCD 1.2 installation:

    nodetool drain

  2. Stop the HCD 1.2 service:

    INSTALL_DIRECTORY/bin/hcd cassandra-stop

  3. Verify the node is using a supported platform.

  4. Install HCD on the node.

    Install HCD using the same installation type as your current system (package or tarball).

    HCD 2.0 is installed alongside your existing installation. HCD 2.0 uses your existing HCD 1.2 data directories.

  5. Configure HCD 2.0 for compatibility with your HCD 1.2 installation.

    For a successful migration, you must complete these configuration changes before starting HCD 2.0.

    1. Update the token count to match your HCD 1.2 configuration:

      # Check HCD 1.2 token count
existing_num_tokens=$(grep "num_tokens" PATH_TO_HCD_1.2_INSTALL_DIRECTORY/resources/cassandra/conf/cassandra.yaml)
# Update HCD 2.0 cassandra.yaml to match
sed -i "s/num_tokens: 16/num_tokens: $\{existing_num_tokens}/" PATH_TO_HCD_2.0_INSTALL_DIRECTORY/resources/cassandra/conf/cassandra.yaml

    2. In your HCD 2.0 cassandra.yaml, set the following *_directory parameters to use existing directories.

      This ensures that the node continues to use your database’s existing data.

      The following example uses default values. Set these parameters to the actual paths in the node’s file system.

      		 
      data_file_directories:
	 - /var/lib/cassandra/data

metadata_directory: /var/lib/cassandra/metadata

commitlog_directory: /var/lib/cassandra/commitlog

hints_directory: /var/lib/cassandra/hints

cdc_raw_directory: /var/lib/cassandra/cdc_raw

    3. To avoid startup failures due to mismatched datacenter names, add the datacenter compatibility flag to JVM options:

      echo 'JVM_OPTS="$JVM_OPTS -Dcassandra.ignore_dc=true"' >> INSTALL_DIRECTORY/resources/cassandra/conf/cassandra-env.sh

    4. Compare and update other configuration settings as needed:

      # Generate a diff to compare cassandra.yaml files manually
diff /etc/hcd/cassandra/cassandra.yaml.backup INSTALL_DIRECTORY/resources/cassandra/conf/cassandra.yaml

  6. Start the node and verify it’s running correctly:

    INSTALL_DIRECTORY/bin/hcd cassandra

  7. Verify that the upgraded datacenter names match the datacenter names in the keyspace schema definition:

    1. Get the node’s datacenter name:

      INSTALL_DIRECTORY/bin/hcd nodetool status | grep "Datacenter"

    2. Verify that the node’s datacenter name matches the datacenter name for a keyspace:

      INSTALL_DIRECTORY/bin/hcd cqlsh --execute "DESCRIBE KEYSPACE keyspace-name;" | grep "replication"

    3. Review the logs for warnings, errors, and exceptions:

      grep -w 'WARNING\|ERROR\|exception' INSTALL_DIRECTORY/logs/*.log

      Warnings, errors, and exceptions frequently appear in the logs when starting an upgraded node. Some of these log entries provide informational help for executing specific upgrade-related steps. If you find unexpected warnings, errors, or exceptions, contact IBM Support.

      The default location is /etc/hcd/hcd-env.sh for package installations and installation_location/bin/hcd-env.sh for tarball installations. You can change the location in hcd-env.sh.

  8. After successfully upgrading and verifying one node, repeat the entire process on the next node in the recommended upgrade order. Proceed through all nodes, one by one, until you have upgraded all nodes, racks, datacenters, and clusters.

Post-upgrade SSTable upgrade and verification

After you upgrade all nodes and clusters, finalize and verify the upgrade:

  1. Run upgradesstables on one node at a time, or one rack at a time when using racks.

    If you don’t upgrade SSTables, you will experience significant performance degradation, increased disk usage, and possible data loss.

    		 
    installation_location/bin/hcd nodetool upgradesstables

    You can use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes the impact on the cluster. Set to 0 to use all available compaction threads.

  2. Run the following commands to verify that HCD is running correctly:

# Check node status
# Make sure all nodes show UN (Up/Normal) status
./bin/hcd nodetool status

# Verify all original keyspaces are accessible
./bin/hcd cqlsh --execute "DESCRIBE KEYSPACES;"

# Confirm data integrity
# Make sure data load matches pre-migration values
./bin/hcd cqlsh --execute "SELECT count(*) FROM system.local;"

# Check that HCD-specific keyspaces are preserved
./bin/hcd cqlsh --execute "SELECT keyspace_name FROM system_schema.keyspaces WHERE keyspace_name LIKE 'hcd_%';"

  1. Based on the output from the previous commands and other information provided by your HCD installation, make sure the following are true:

    • HCD node status shows UN (Up/Normal)

    • All original keyspaces are accessible

    • Data load matches pre-migration values

    • Host ID and tokens are preserved

    • You have upgraded SSTables

    • Authentication and authorization work correctly

    • No critical errors in logs

    If all conditions are true, then the migration was successful. If any conditions are false, see Troubleshoot migration.

Troubleshoot migration

You might encounter these issues during migration:

Cannot change the number of tokens

  • Cause: Token count mismatch between the HCD 1.2 and HCD 2.0 configurations.

  • Solution: Update num_tokens in the HCD 2.0 cassandra.yaml to match the original HCD 1.2 configuration.

Cannot start node if snitch’s data center differs

  • Cause: Datacenter name mismatch between HCD 1.2 and HCD 2.0.

  • Solution: Add -Dcassandra.ignore_dc=true to JVM options in cassandra-env.sh.

Connection refused

  • Cause: HCD failed to start due to configuration issues.

  • Solution: Check logs for specific error messages and address configuration compatibility issues.

To help debug issues, check the HCD logs for detailed error information:

# Check system logs
tail -50 /var/log/cassandra/system.log | grep -E "(ERROR|WARN|Exception|Failed|Cannot start|Fatal)"

# Check debug logs for more details
tail -50 /var/log/cassandra/debug.log | grep -E "(ERROR|WARN|Exception|Failed)"

Was this helpful?

Give Feedback

How can we improve the documentation?

© Copyright IBM Corporation 2026 | Privacy policy | Terms of use |  Manage Privacy Choices

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: Contact IBM