Upgrading from DataStax Enterprise 5.0 to 6.0

• Delete-by-id is removed, use CQL DELETE by primary key instead.
• Delete-by-query no longer supports wildcards, use CQL TRUNCATE instead.

shard_transport_options:
    netty_client_request_timeout: 60000

• IsDisjointTo is no longer supported in queries on SpatialRecursivePrefixTreeFieldType. Replace IsDisjointTo with a NOT Intersects query. For example:

  foo:0,0 TO 1000,1000 AND -"Intersects(POLYGON((338 211, 338 305, 404 305, 404 211, 338 211)))")

• The ENVELOPE syntax is now required for WKT-style queries against SpatialRecursivePrefixTreeFieldType fields. You must specify ENVELOPE(10, 15, 15, 10), where queries on earlier releases could specify 10 10 15 15.

• Change the stored attribute value of all copyField directives from true to false in the schema.xml file and then use dsetool reload_core to reload the modified schema.
• You must ensure that application designs and implementations recognize this change.

• Remove these requestHandlers: XmlUpdateRequestHandler, BinaryUpdateRequestHandler, CSVRequestHandler, JsonUpdateRequestHandler, DataImportHandler. Solr deprecated and then removed these requestHandlers.

  For example:

  <requestHandler name="/dataimport" class="solr.DataImportHandler"/>

  or

  <requestHandler name="/update" class="solr.XmlUpdateRequestHandler"/>

• Change the directoryFactory from:

  <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>

  to

  <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

• <unlockOnStartup> is now unsupported as a result of LUCENE-6508 and SOLR-7942.
• Change the updateLog from:

  <updateLog class="solr.FSUpdateLog" force="false">

  to

  <updateLog force="false">

1. Avoid sorting on removed Solr field types until the upgrade to DSE 5.1 or later is complete for all nodes in the datacenter being queried.
   Tip: When using two search datacenters, isolate queries to a single datacenter and then change the schema and reindex the other datacenter. Then isolate queries to the newly reindexed datacenter while you change the schema and upgrade the first datacenter.
2. If you are using BCDIntField or BCDLongField, update the schema to replace BCDIntField and BCDLongField with types that are sort-compatible with the supported Solr types TrieIntField and TrieLongField:

   Removed Solr field types	Interim sort-compatible supported Solr field types
   BCDIntField	SortableIntField
   BCDLongField	SortableLongField

   Change the schema in a distributed fashion, and do not reindex. After the schema is updated on all nodes, then go on to 9.

• Update the schema to replace unsupported Solr field types with supported Solr field types:

  Removed Solr field types	Supported Solr field types
  ByteField	TrieIntField
  DateField	TrieDateField
  DoubleField	TrieDoubleField
  FloatField	TrieFloatField
  IntField	TrieIntField
  LongField	TrieLongField
  ShortField	TrieIntField
  SortableDoubleField	TrieDoubleField
  SortableFloatField	TrieFloatField
  SortableIntField	TrieIntField
  SortableLongField	TrieLongField
  BCDIntField	TrieIntField
  BCDLongField	TrieLongField
  BCDStrField (see 10 if used)	TrieIntField

• If you are using type mapping version 0, or you do not specify a type mapper, verify or update the solrconfig.xml to use dseTypeMappingVersion 1:

  <dseTypeMappingVersion>1</dseTypeMappingVersion>

  If the Solr core is backed by a CQL table and the type mapping is unspecified, use type mapping version 2.
• Reload the core:

  dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath

  If you were using the unsupported data types, do a full reindex node-by-node:

  dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=false

Changing the dse-spark-dependencies in your build files is not sufficient to change the compilation target. See the example projects for how to set up your build files.

You no longer need to specify the Spark master IP address or hostname when using dse:// URLs. Connecting to any Spark node will redirect the request to the master node.

For example, the following code worked in DSE 5.0 but will not work in DSE 5.1 or later.

val conf = new SparkConf(true) 
.setMaster("spark://192.168.123.10:7077") 
.setAppName("cassandra-demo") 
.set("cassandra.connection.host" , "192.168.123.10") // initial contact 
.set("cassandra.username", "cassandra") 
.set("cassandra.password", "cassandra") 
val sc = new SparkContext(conf)

To connect to DSE 5.1 and later, you no longer need to call setMaster. This code will work in DSE 5.1 and later:

val conf = new SparkConf(true)
.setAppName("cassandra-demo") 
.set("cassandra.connection.host" , "192.168.123.10") // initial contact 
.set("cassandra.username", "cassandra") 
.set("cassandra.password", "cassandra") 
val sc = new SparkContext(conf)

If you need to specify the master using setMaster, use the dse:// URL format.

The required space depends on the compaction strategy. See Disk space.

• DataStax Enterprise 6.0 release notes.
• General upgrading advice for any version. Be sure to read NEWS.txt all the way back to your current version.
• DataStax Enterprise changes in CHANGES.txt.
• DataStax driver changes.

Several internal and beta extension points were modified to necessitate core storage engine refactoring. All custom implementations, including the following interfaces, must be replaced with supported implementations when upgrading to DSE 6.0. Because a rewrite of the following interfaces is required for DSE 6.0: (For help contact the DataStax Services team.)

• The org.apache.cassandra.triggers.ITrigger interface was modified from augment to augmentNonBlocking for non-blocking internal architecture. Updated trigger implementations must be provided on upgraded nodes. If unsure, drop all existing triggers before upgrading. To check for existing triggers:

  SELECT * FROM system_schema.triggers

• The org.apache.cassandra.index.Index interface was modified to comply with the core storage engine changes. Updated implementations are required. If unsure, drop all existing custom secondary indexes before upgrading, except DSE Search indexes, which do not need to be replaced. To check for existing indexes:

  SELECT * FROM system_schema.indexes

• The org.apache.cassandra.cql3.QueryHandler, org.apache.cassandra.db.commitlog.CommitLogReadHandler, and other extension points have been changed. See QueryHandlers.

ALTER TABLE dse_audit.audit_log ADD authenticated text;
ALTER TABLE dse_audit.audit_log ADD consistency text;

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken.

java -version

• Recommended. OpenJDK 8 (1.8.0_151 minimum)
  Note: Recommendation changed due to the end of public updates for Oracle JRE/JDK 8. See Oracle Java SE Support Roadmap.
• Supported. Oracle Java SE 8 (JRE or JDK) (1.8.0_151 minimum)

• Support for Thrift-compatible tables (COMPACT STORAGE) is dropped. Before upgrading, follow the steps to migrate all tables that have COMPACT STORAGE to CQL table format while DSE 5.x.x is running.
  Note: Do not migrate system.* tables, COMPACT STORAGE is removed by DSE internally. Modifying the system keyspace is not supported; modification attempts generate an error.
  For DSE Analytics, drop compact storage from all the tables in the "HiveMetaStore" and PortfolioDemo keyspaces.
  After COMPACT STORAGE is dropped, columns to support migration to CQL-compatible table format are added as described in migrating from compact storage.

  Attention: DSE 6.0 will not start if COMPACT STORAGE tables are present. Creating a COMPACT STORAGE table in a mixed-version cluster is not supported. Driver connections to the latest DSE 5.0.x and DSE 5.1.x run in a "NO_COMPACT" mode that causes compact tables to appear as if the compact flags were already dropped, but only for the current session.
• If you programmatically set the shuffle parameter, you must change the code for applications that use conf.set("spark.shuffle.service.port", port). Instead, use dse spark-submit which automatically sets the correct service port based on the authentication state. See Configuring Spark for more information.
• If DSEFS is enabled, copy CFS hivemetastore directory to dse:

  DSE_HOME/bin/dse hadoop fs -cp cfs://127.0.0.1/user/spark/warehouse/ dsefs://127.0.0.1/user/spark/warehouse/

  After upgrade is complete migrate Spark SQL tables (if used) to the new Hive metastore format:

  dse client-tool spark metastore migrate --from 5.0.0 --to 6.0.0

• Cassandra File System (CFS) is removed. Remove the cfs and cfs_archive keyspaces before upgrading. See the From CFS to DSEFS blog post and the Copying data from CFS to DSEFS documentation for more information.

• DSE Search in DataStax Enterprise 6.0 uses Apache Solr™ 6.0. Complete all of the steps in Advanced preparation for upgrading DSE Search and SearchAnalytics nodes.
• Ensure all use of HTTP writes are changed to use CQL commands for updates and inserts.
• Edit the search index config and make these changes, as needed. See Search index config for valid options to change query behavior for search indexes.
  • Remove the unsupported dataDir option. You can still set the location of search indexes.
  • Remove mergePolicy, maxMergeDocs, and mergeFactor. For example:

    <mergeFactor>25</mergeFactor>
    <maxMergeDocs>...
    <mergePolicy>...

    Use mergePolicyFactory instead, and add mergeScheduler:

    <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
        <int name="maxThreadCount">16</int>
        <int name="maxMergeCount">32</int>
    </mergeScheduler>
    ...
    <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
      <int name="maxMergeAtOnce">10</int>
      <int name="segmentsPerTier">10</int>
    </mergePolicyFactory>

  • Remove any instance of ExtractingRequestHandler.
  • Remove DSENRTCachingDirectoryFactory. Change:

    <directoryFactory name="DirectoryFactory" class="com.datastax.bdp.search.solr.DSENRTCachingDirectoryFactory"/>

    to:

    <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

• Ensure that the catalina.properties and context.xml files are present in the Tomcat conf dir. DSE will not start after upgrade if these files are missing.
  The default location of the Tomcat conf directory depends on the type of installation:

  • Package installations: /etc/dse/tomcat/conf
  • Tarball installations: installation_location/resources/tomcat/conf
• If earlier DSE versions use a custom configuration for the Solr UI web.xml, change:

  <filter-class>com.datastax.bdp.search.solr.auth.DseAuthenticationFilter</filter-class>

  to

  <filter-class>com.datastax.bdp.cassandra.auth.http.DseAuthenticationFilter</filter-class>

• StallMetrics MBean is removed. Change operators that use the MBean.

• If your graph nodes have search indexes that you added with gremlin, complete the steps in Advanced preparation for upgrading DSE Graph nodes with search indexes.
• Ensure that edge label names and property key names use only the supported characters. Edge label names and property key names allow only [a-zA-Z0-9], underscore, hyphen, and period. In earlier versions, edge label names and property key names allowed nearly unrestricted Unicode.
  • schema.describe() displays the entire schema, even if it contains illegal names.
  • In-place upgrades allow existing schemas with invalid edge label names and property key names.
  • Schema elements with illegal names cannot be updated or added.

The configuration files are overwritten with default values during installation of the new version.

The messaging protocol version in DSE 5.1.2 has been changed to VERSION_3014. Schema migrations rely on exact messaging protocol versions. To accommodate schema changes that might occur during the upgrade, force a backward compatible messaging protocol.

-Dcassandra.force_3_0_protocol_version=true

installation_location/bin/dse cassandra -Dcassandra.force_3_0_protocol_version=true

After the upgrade is complete on all nodes, restart nodes without this flag.

nodetool -h hostname drain

• To stop DataStax Enterprise running as a service:

  sudo service dse stop

• To stop DataStax Enterprise running as a stand-alone process:

  bin/dse cassandra-stop

• Package installer using YUM
• Package installer using APT
• Binary tarball installer

1. Compare your backup configuration files to the new configuration files:
   • Look for any deprecated, removed, or changed settings.
     • DSE Search nodes
       • While the node is down, edit dse.yaml and remove these options:
         • cql_solr_query_executor_threads
         • enable_back_pressure_adaptive_nrt_commit
         • max_solr_concurrency_per_core
         • solr_indexing_error_log_options
         DSE 6.0 will not start with these options present.
     • DSE Analytics nodes
       Note: Although DSEFS is enabled by default in DSE 5.1.0 and later, the dsefs.enabled setting is commented out in dse.yaml. To enable DSEFS, uncomment the dsefs_options.enabled setting. (DSP-13310)
   • The upgrade installs a new server.xml for Tomcat 8. If your existing server.xml has custom connectors, migrate those connectors to the new server.xml before starting the upgraded nodes.
   • Be sure you are familiar with the Apache Cassandra and DataStax Enterprise changes and features in the new release.
   • Ensure that keyspace replication factors are correct for your environment:
     • Check the keyspace replication factor for analytics keyspaces.
     • Check the keyspace replication factor for system_auth and dse_security keyspaces.

• For nodes in the datacenters running in Analytics Hadoop mode, start those nodes in Spark mode.
• Add the special start-up parameter -Dcassandra.ignore_dc=true for each node, then start in cassandra mode. This flag is required only once after upgrading. Subsequent restarts do not use this flag. You can leave the flag in the configuration file or remove it after the first restart of each node.

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

The following example removes the files using the find command:

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

• Package installations: See Starting DataStax Enterprise as a service.
• Tarball installations: See Starting DataStax Enterprise as a stand-alone process.

nodetool status

Warnings, errors, and exceptions are frequently found in the logs when starting an upgraded node. Some of these log entries are informational to help you execute specific upgrade-related steps. If you find unexpected warnings, errors, or exceptions, contact DataStax Support.

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken. See SSTable compatibility and upgrade version.

Use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes impact on the cluster. Set to 0 to use all available compaction threads. DataStax recommends running the upgradesstables command on one node at a time or when using racks, one rack at a time.

• DSE 5.0.x, downgrade to 5.0.15 or later
• DSE 5.1.x, downgrade to 5.1.16 or later