Upgrading from DataStax Enterprise 5.1 to 6.0

The upgrade process for DataStax Enterprise provides minimal downtime (ideally zero). During this process, upgrade and restart one node at a time while other nodes continue to operate online. With a few exceptions, the cluster continues to work as though it were on the earlier version of DataStax Enterprise until all of the nodes in the cluster are upgraded.

Several internal and beta extension points were modified to necessitate core storage engine refactoring. All custom implementations, including the following interfaces, must be replaced with supported implementations when upgrading to DSE 6.0. Because a rewrite of the following interfaces is required for DSE 6.0: (For help contact the DataStax Services team.)

• The org.apache.cassandra.triggers.ITrigger interface was modified from augment to augmentNonBlocking for non-blocking internal architecture. Updated trigger implementations must be provided on upgraded nodes. If unsure, drop all existing triggers before upgrading. To check for existing triggers:

  SELECT * FROM system_schema.triggers

• The org.apache.cassandra.index.Index interface was modified to comply with the core storage engine changes. Updated implementations are required. If unsure, drop all existing custom secondary indexes before upgrading, except DSE Search indexes, which do not need to be replaced. To check for existing indexes:

  SELECT * FROM system_schema.indexes

• The org.apache.cassandra.cql3.QueryHandler, org.apache.cassandra.db.commitlog.CommitLogReadHandler, and other extension points have been changed. See QueryHandlers.

The required space depends on the compaction strategy. See Disk space.

• DataStax Enterprise 6.0 release notes.
• General upgrading advice for any version. Be sure to read NEWS.txt all the way back to your current version.
• DataStax Enterprise changes in CHANGES.txt.
• DataStax driver changes.

dse -v

The configuration files are overwritten with default values during installation of the new version.

Always upgrade to latest patch release on your current version before you upgrade to a higher version. Fixes included in the latest patch release might help or smooth the upgrade process.

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken.

java -version

• Recommended. OpenJDK 8 (1.8.0_151 minimum)
  Note: Recommendation changed due to the end of public updates for Oracle JRE/JDK 8. See Oracle Java SE Support Roadmap.
• Supported. Oracle Java SE 8 (JRE or JDK) (1.8.0_151 minimum)

1. Support for Thrift-compatible tables (COMPACT STORAGE) is dropped. Before upgrading, follow the steps to migrate all tables that have COMPACT STORAGE to CQL table format while DSE 5.x.x is running.
   Note: Do not migrate system.* tables, COMPACT STORAGE is removed by DSE internally. Modifying the system keyspace is not supported; modification attempts generate an error.
   For DSE Analytics, drop compact storage from all the tables in the "HiveMetaStore" and PortfolioDemo keyspaces.
   After COMPACT STORAGE is dropped, columns to support migration to CQL-compatible table format are added as described in migrating from compact storage.

   Attention: DSE 6.0 will not start if COMPACT STORAGE tables are present. Creating a COMPACT STORAGE table in a mixed-version cluster is not supported. Driver connections to the latest DSE 5.0.x and DSE 5.1.x run in a "NO_COMPACT" mode that causes compact tables to appear as if the compact flags were already dropped, but only for the current session.
2. If you programmatically set the shuffle parameter, you must change the code for applications that use conf.set("spark.shuffle.service.port", port). Instead, use dse spark-submit which automatically sets the correct service port based on the authentication state. See Configuring Spark for more information.
3. If DSEFS is enabled, copy CFS hivemetastore directory to dse:

   DSE_HOME/bin/dse hadoop fs -cp cfs://127.0.0.1/user/spark/warehouse/ dsefs://127.0.0.1/user/spark/warehouse/

   After upgrade is complete migrate Spark SQL tables (if used) to the new Hive metastore format:

   dse client-tool spark metastore migrate --from 5.0.0 --to 6.0.0

4. Cassandra File System (CFS) is removed. Remove the cfs and cfs_archive keyspaces before upgrading. See the From CFS to DSEFS blog post and the Copying data from CFS to DSEFS documentation for more information.
5. Make sure any use of the SPARK_LOCAL_DIRS and SPARK_EXECUTOR_DIRS environment variables match their use as described in Setting environment variables.
6. For applications to use the compatible Spark Jobserver API in DataStax repository, migrate jobs that extend from SparkHiveJob and SparkSqlJob to SparkSessionJob. See example in the DemoSparkSessionJob in the demos directory.
   Note: Spark Jobserver is the DSE custom version 8.0.4.45.
   The default location of the demos directory depends on the type of installation:

   • Package installations: /usr/share/dse/demos
   • Tarball installations: installation_location/demos

• Ensure all use of HTTP writes are changed to use CQL commands for updates and inserts.
• Edit the search index config and make these changes, as needed. See Search index config for valid options to change query behavior for search indexes.
  • Remove the unsupported dataDir option. You can still set the location of search indexes.
  • Remove mergePolicy, maxMergeDocs, and mergeFactor. For example:

    <mergeFactor>25</mergeFactor>
    <maxMergeDocs>...
    <mergePolicy>...

    Use mergePolicyFactory instead, and add mergeScheduler:

    <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
        <int name="maxThreadCount">16</int>
        <int name="maxMergeCount">32</int>
    </mergeScheduler>
    ...
    <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
      <int name="maxMergeAtOnce">10</int>
      <int name="segmentsPerTier">10</int>
    </mergePolicyFactory>

  • Remove any instance of ExtractingRequestHandler.
  • Remove DSENRTCachingDirectoryFactory. Change:

    <directoryFactory name="DirectoryFactory" class="com.datastax.bdp.search.solr.DSENRTCachingDirectoryFactory"/>

    to:

    <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

• Ensure that the catalina.properties and context.xml files are present in the Tomcat conf dir. DSE will not start after upgrade if these files are missing.
  The default location of the Tomcat conf directory depends on the type of installation:

  • Package installations: /etc/dse/tomcat/conf
  • Tarball installations: installation_location/resources/tomcat/conf
• If earlier DSE versions use a custom configuration for the Solr UI web.xml, change:

  <filter-class>com.datastax.bdp.search.solr.auth.DseAuthenticationFilter</filter-class>

  to

  <filter-class>com.datastax.bdp.cassandra.auth.http.DseAuthenticationFilter</filter-class>

• StallMetrics MBean is removed. Change operators that use the MBean.

• Ensure that edge label names and property key names use only the supported characters. Edge label names and property key names allow only [a-zA-Z0-9], underscore, hyphen, and period. In earlier versions, edge label names and property key names allowed nearly unrestricted Unicode.
  • schema.describe() displays the entire schema, even if it contains illegal names.
  • In-place upgrades allow existing schemas with invalid edge label names and property key names.
  • Schema elements with illegal names cannot be updated or added.

nodetool -h hostname drain

• To stop DataStax Enterprise running as a service:

  sudo service dse stop

• To stop DataStax Enterprise running as a stand-alone process:

  bin/dse cassandra-stop

• Package installer using YUM
• Package installer using APT
• Binary tarball installer

1. Compare your backup configuration files to the new configuration files:
   • Look for any deprecated, removed, or changed settings.
   • DSE Search nodes
     • While the node is down, edit dse.yaml and remove these options:
       • cql_solr_query_executor_threads
       • enable_back_pressure_adaptive_nrt_commit
       • max_solr_concurrency_per_core
       • solr_indexing_error_log_options
       DSE 6.0 will not start with these options present.
   • Be sure you are familiar with the Apache Cassandra and DataStax Enterprise changes and features in the new release.
   • Ensure that keyspace replication factors are correct for your environment:
     • Check the keyspace replication factor for analytics keyspaces.
     • Check the keyspace replication factor for system_auth and dse_security keyspaces.
2. Edit the cassandra.yaml file and comment out or remove these credentials cache settings, if they exist:

   credentials_validity_in_ms
   credentials_update_interval_in_ms

   Caches are optimized without these settings.
3. Merge the applicable modifications into the new version.

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

The following example removes the files using the find command:

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

• Package installations: See Starting DataStax Enterprise as a service.
• Tarball installations: See Starting DataStax Enterprise as a stand-alone process.

nodetool status

Warnings, errors, and exceptions are frequently found in the logs when starting an upgraded node. Some of these log entries are informational to help you execute specific upgrade-related steps. If you find unexpected warnings, errors, or exceptions, contact DataStax Support.

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken. See SSTable compatibility and upgrade version.

Use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes impact on the cluster. Set to 0 to use all available compaction threads. DataStax recommends running the upgradesstables command on one node at a time or when using racks, one rack at a time.

• DSE 5.0.x, downgrade to 5.0.15 or later
• DSE 5.1.x, downgrade to 5.1.16 or later