Upgrading from DataStax Enterprise 5.0 to 6.7

• Delete-by-id is removed, use CQL DELETE by primary key instead.
• Delete-by-query no longer supports wildcards, use CQL TRUNCATE instead.

shard_transport_options:
    netty_client_request_timeout: 60000

• IsDisjointTo is no longer supported in queries on SpatialRecursivePrefixTreeFieldType. Replace IsDisjointTo with a NOT Intersects query. For example:

  foo:0,0 TO 1000,1000 AND -"Intersects(POLYGON((338 211, 338 305, 404 305, 404 211, 338 211)))")

• The ENVELOPE syntax is now required for WKT-style queries against SpatialRecursivePrefixTreeFieldType fields. You must specify ENVELOPE(10, 15, 15, 10), where queries on earlier releases could specify 10 10 15 15.

• Remove these unsupported Solr requestHandlers:
  • XmlUpdateRequestHandler
  • BinaryUpdateRequestHandler
  • CSVRequestHandler
  • JsonUpdateRequestHandler
  • DataImportHandler

  For example:

  <requestHandler name="/dataimport" class="solr.DataImportHandler"/>

  or

  <requestHandler name="/update" class="solr.XmlUpdateRequestHandler"/>

• Change the directoryFactory from:

  <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>

  to

  <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

• <unlockOnStartup> is unsupported as a result of LUCENE-6508 and SOLR-7942.
• Change the updateLog from:

  <updateLog class="solr.FSUpdateLog" force="false">

  to

  <updateLog force="false">

1. Avoid sorting on removed Solr field types until the upgrade is complete for all nodes in the datacenter being queried.
   Tip: When using two search datacenters, isolate queries to a single datacenter and then change the schema and reindex the other datacenter. Then isolate queries to the newly reindexed datacenter while you change the schema and upgrade the first datacenter.
2. If you are using BCDIntField or BCDLongField, update the schema to replace BCDIntField and BCDLongField with types that are sort-compatible with the supported Solr types TrieIntField and TrieLongField:

   Removed Solr field types	Interim sort-compatible supported Solr field types
   BCDIntField	SortableIntField
   BCDLongField	SortableLongField

   Change the schema in a distributed fashion, and do not reindex. After the schema is updated on all nodes, then go on to 9.

• Update the schema to replace unsupported Solr field types with supported Solr field types:

  Removed Solr field types	Supported Solr field types
  ByteField	TrieIntField
  DateField	TrieDateField
  DoubleField	TrieDoubleField
  FloatField	TrieFloatField
  IntField	TrieIntField
  LongField	TrieLongField
  ShortField	TrieIntField
  SortableDoubleField	TrieDoubleField
  SortableFloatField	TrieFloatField
  SortableIntField	TrieIntField
  SortableLongField	TrieLongField
  BCDIntField	TrieIntField
  BCDLongField	TrieLongField
  BCDStrField (see 10 if used)	TrieIntField

• If you are using type mapping version 0, or you do not specify a type mapper, verify or update the solrconfig.xml to use dseTypeMappingVersion 1:

  <dseTypeMappingVersion>1</dseTypeMappingVersion>

  If the Solr core is backed by a CQL table and the type mapping is unspecified, use type mapping version 2.
• Reload the core:

  dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath

  If you were using the unsupported data types, do a full reindex node-by-node:

  dsetool reload_core keyspace_name.table_name schema=filepath solrconfig=filepath reindex=true deleteAll=true distributed=false

Changing the dse-spark-dependencies in your build files is not sufficient to change the compilation target. See the example projects for how to set up your build files.

You no longer need to specify the Spark master IP address or hostname when using dse:// URLs. Connecting to any Spark node will redirect the request to the master node.

For example, the following code worked in DSE 5.0 but will not work in DSE 5.1 or later.

val conf = new SparkConf(true) 
.setMaster("spark://192.168.123.10:7077") 
.setAppName("cassandra-demo") 
.set("cassandra.connection.host" , "192.168.123.10") // initial contact 
.set("cassandra.username", "cassandra") 
.set("cassandra.password", "cassandra") 
val sc = new SparkContext(conf)

To connect to DSE 6.7, you no longer need to call setMaster:

val conf = new SparkConf(true)
.setAppName("cassandra-demo") 
.set("cassandra.connection.host" , "192.168.123.10") // initial contact 
.set("cassandra.username", "cassandra") 
.set("cassandra.password", "cassandra") 
val sc = new SparkContext(conf)

To specify the master using setMaster, use the dse:// URL format.

The required space depends on the compaction strategy. See Disk space.

• DataStax Enterprise 6.7 release notes.
• General upgrading advice for any version. Be sure to read NEWS.txt all the way back to your current version.
• DataStax Enterprise changes in CHANGES.txt.
• DataStax driver changes.

Several internal and beta extension points were modified to necessitate core storage engine refactoring. All custom implementations, including the following interfaces, must be replaced with supported implementations when upgrading to DSE 6.7. Because a rewrite of the following interfaces is required for DSE 6.7: (For help contact the DataStax Services team.)

• The org.apache.cassandra.triggers.ITrigger interface was modified from augment to augmentNonBlocking for non-blocking internal architecture. Updated trigger implementations must be provided on upgraded nodes. If unsure, drop all existing triggers before upgrading. To check for existing triggers:

  SELECT * FROM system_schema.triggers

• The org.apache.cassandra.index.Index interface was modified to comply with the core storage engine changes. Updated implementations are required. If unsure, drop all existing custom secondary indexes before upgrading, except DSE Search indexes, which do not need to be replaced. To check for existing indexes:

  SELECT * FROM system_schema.indexes

• The org.apache.cassandra.cql3.QueryHandler, org.apache.cassandra.db.commitlog.CommitLogReadHandler, and other extension points have been changed. See QueryHandlers.

ALTER TABLE dse_audit.audit_log ADD authenticated text;
ALTER TABLE dse_audit.audit_log ADD consistency text;

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken.

java -version

• Recommended. OpenJDK 8 (1.8.0_151 minimum)
  Note: Recommendation changed due to the end of public updates for Oracle JRE/JDK 8. See Oracle Java SE Support Roadmap.
• Supported. Oracle Java SE 8 (JRE or JDK) (1.8.0_151 minimum)

• If you programmatically set the shuffle parameter, you must change the code for applications that use conf.set("spark.shuffle.service.port", port). Instead, use dse spark-submit which automatically sets the correct service port based on the authentication state. See Configuring Spark for more information.

• If DSEFS is enabled, copy CFS hivemetastore directory to dse:

  DSE_HOME/bin/dse hadoop fs -cp cfs://127.0.0.1/user/spark/warehouse/ dsefs://127.0.0.1/user/spark/warehouse/

  After upgrade is complete migrate Spark SQL tables (if used) to the new Hive metastore format:

  dse client-tool spark metastore migrate --from 5.0.0 --to 6.0.0

• Cassandra File System (CFS) is removed. Remove the cfs and cfs_archive keyspaces before upgrading. See the From CFS to DSEFS blog post and the Copying data from CFS to DSEFS documentation for more information.

• DSE Search in DSE 6.7 uses Apache Solr™ 6.0. Complete all of the steps in Advanced preparation for upgrading DSE Search and SearchAnalytics nodes.
• Ensure all use of HTTP writes are changed to use CQL commands for updates and inserts.
• Edit the search index config and make these changes, as needed. See Search index config for valid options to change query behavior for search indexes.
  • Remove the unsupported dataDir option. You can still set the location of search indexes.
  • Remove mergePolicy, maxMergeDocs, and mergeFactor. For example:

    <mergeFactor>25</mergeFactor>
    <maxMergeDocs>...
    <mergePolicy>...

    Use mergePolicyFactory instead, and add mergeScheduler:

    <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
        <int name="maxThreadCount">16</int>
        <int name="maxMergeCount">32</int>
    </mergeScheduler>
    ...
    <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
      <int name="maxMergeAtOnce">10</int>
      <int name="segmentsPerTier">10</int>
    </mergePolicyFactory>

  • Remove any instance of ExtractingRequestHandler.
  • Remove DSENRTCachingDirectoryFactory. Change:

    <directoryFactory name="DirectoryFactory" class="com.datastax.bdp.search.solr.DSENRTCachingDirectoryFactory"/>

    to:

    <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>

• Ensure that the catalina.properties and context.xml files are present in the Tomcat conf dir. DSE will not start after upgrade if these files are missing.
  The default location of the Tomcat conf directory depends on the type of installation:

  • Package installations: /etc/dse/tomcat/conf
  • Tarball installations: installation_location/resources/tomcat/conf
• If earlier DSE versions use a custom configuration for the Solr UI web.xml, change:

  <filter-class>com.datastax.bdp.search.solr.auth.DseAuthenticationFilter</filter-class>

  to

  <filter-class>com.datastax.bdp.cassandra.auth.http.DseAuthenticationFilter</filter-class>

• StallMetrics MBean is removed. Change operators that use the MBean.

• If your graph nodes have search indexes that you added with gremlin, complete the steps in Advanced preparation for upgrading DSE Graph nodes with search indexes.
• Ensure that edge label names and property key names use only the supported characters. Edge label names and property key names allow only [a-zA-Z0-9], underscore, hyphen, and period. In earlier versions, edge label names and property key names allowed nearly unrestricted Unicode.
  • schema.describe() displays the entire schema, even if it contains illegal names.
  • In-place upgrades allow existing schemas with invalid edge label names and property key names.
  • Schema elements with illegal names cannot be updated or added.

The configuration files are overwritten with default values during installation of the new version.

nodetool -h hostname drain

• Package installer using YUM
• Package installer using APT
• Binary tarball installer

1. Compare your backup configuration files to the new configuration files:
   • Review changes in cassandra.yaml and dse.yaml.

     After the upgrade and before restarting with 6.7.0, remove deprecated settings and use the new settings.

     cassandra.yaml changes

     Memtable settings

     Deprecated cassandra.yaml settings

     memtable_heap_space_in_mb memtable_offheap_space_in_mb

     Replace with this setting

     memtable_space_in_mb Governs heap and offheap space allocation to set a threshold for automatic memtable flush. The calculated default is 1/4 of the heap size.

     Changed setting

     memtable_allocation_type: offheap_objects The default method the database uses to allocate and manage memtable memory is  offheap_objects.

     User-defined functions (UDF) settings

     Deprecated cassandra.yaml settings

     user_defined_function_warn_timeout user_defined_function_fail_timeout

     Replace with these settings

     user_defined_function_warn_micros: 500 user_defined_function_fail_micros: 10000 user_defined_function_warn_heap_mb: 200 user_defined_function_fail_heap_mb: 500 user_function_timeout_policy: die Settings are in microseconds since Java UDFs run faster. The new timeouts are not equivalent to the deprecated settings.

     Internode encryption settings

     Deprecated cassandra.yaml setting

     server_encryption_options: store_type: JKS

     Replace with these settings

     server_encryption_options: keystore_type: JKS truststore_type: JKS Valid type options are JKS, JCEKS, PKCS12, or PKCS11.

     Client-to-node encryption settings

     Deprecated cassandra.yaml setting

     client_encryption_options: store_type: JKS

     Replace with these settings

     client_encryption_options: keystore_type: JKS truststore_type: JKS Valid type options are JKS, JCEKS, PKCS12, or PKCS11.

     dse.yaml changes

   • Look for any deprecated, removed, or changed settings.

     Shard transport

     Deprecated dse.yaml settings

     shard_transport_options: type: netty netty_server_port: 8984 netty_server_acceptor_threads: netty_server_worker_threads: netty_client_worker_threads: netty_client_max_connections: netty_client_request_timeout: The http transport type is removed. shard_transport_options: type: http http_shard_client_conn_timeout: 0 http_shard_client_socket_timeout: 0

     New dse.yaml settings

     shard_transport_options: netty_client_request_timeout: 60000 The shard_transport_options supports only netty_client_request_timeout. Remove any other options under shard_transport_options.

     DSE Search nodes

     Deprecated dse.yaml settings

     Remove these options: cql_solr_query_executor_threads enable_back_pressure_adaptive_nrt_commit max_solr_concurrency_per_core solr_indexing_error_log_options DSE 6.7 will not start with these options present.

     DSE Analytics nodes: DSEFS settings

     Changed dse.yaml settings

     Although DSEFS is enabled by default, the dsefs.enabled setting is commented out. To enable DSEFS, uncomment all dsefs_options settings.

   • Ensure that keyspace replication factors are correct for your environment:
     • Check the keyspace replication factor for analytics keyspaces.
     • Check the keyspace replication factor for system_auth and dse_security keyspaces.
   • The upgrade installs a new server.xml for Tomcat 8. If your existing server.xml has custom connectors, migrate those connectors to the new server.xml before starting the upgraded nodes.
   • Be sure you are familiar with the Apache Cassandra and DataStax Enterprise changes and features in the new release.

• For nodes in the datacenters running in Analytics Hadoop mode, start those nodes in Spark mode.
• Add the special start-up parameter -Dcassandra.ignore_dc=true for each node, then start in cassandra mode. This flag is required only once after upgrading. Subsequent restarts do not use this flag. You can leave the flag in the configuration file or remove it after the first restart of each node.

tiered_storage_options:
    strategy1:
        tiers:
            - paths:
                - /mnt1
            - paths:
                - /mnt2
            - paths:
                - /mnt3

The following example removes the files using the find command:

find /mnt2 -name "*_txn_compaction_*.log" -type f -delete &&
find /mnt3 -name "*_txn_compaction_*.log" -type f -delete

• Package installations
• Tarball installations

nodetool status

Warnings, errors, and exceptions are frequently found in the logs when starting an upgraded node. Some of these log entries are informational to help you execute specific upgrade-related steps. If you find unexpected warnings, errors, or exceptions, contact DataStax Support.

Upgrading and restarting each node is called a rolling restart.

nodetool upgradesstables

If the SSTables are already on the current version, the command returns immediately and no action is taken. See SSTable compatibility and upgrade version.

Use the --jobs option to set the number of SSTables that upgrade simultaneously. The default setting is 2, which minimizes impact on the cluster. Set to 0 to use all available compaction threads. DataStax recommends running the upgradesstables command on one node at a time or when using racks, one rack at a time.

• DSE 5.0.x, downgrade to 5.0.15 or later
• DSE 5.1.x, downgrade to 5.1.16 or later

-Dcassandra.commitlog.ignorereplayerrors=true