Upgrading from DataStax Enterprise 2.2 to 3.2

Follow these instructions to upgrade from DataStax Enterprise 2.2.x to 3.2.x.

Review this information for upgrades from DataStax Enterprise 2.2.x to 3.2.x.

Tip: DataStax is offering a complimentary half-day Upgrade Assessment. This assessment is a DataStax Services engagement designed to assess the upgrade compatibility of your existing DSE deployment to later DSE versions, including 5.1, 6.0, and 6.7. Contact the DataStax Services team to schedule your assessment.

Security recommendations

Upgrade the entire cluster before setting up security and then do another rolling restart.

Hadoop

The ownership of the Hadoop mapred staging directory in the CassandraFS has changed. After upgrading, you need to set the owner of /tmp/hadoop-dseuser/mapred/staging to the DataStax Enterprise user. For example, if you run DataStax Enterprise 3.1 as root, use the following command on Linux:
dse hadoop fs -chown root /tmp/hadoop-root/mapred/staging

Solr

Do not issue Solr queries after upgrading from DataStax Enterprise 2.1.x or earlier until all nodes are upgraded and schema disagreements are resolved.

Solr configuration files from previous versions of DataStax Enterprise will be invalidated by the new version of Solr included in this release. Follow these steps to update your Solr configuration file on the first Solr node you upgrade, before upgrading any other nodes:

  1. Open the system.log file and look for the message about the Solr error.

    The error message briefly describes the changes you need to make.

  2. Correct these errors in your solrconfig.xml files, then post the corrected files.

    Existing cores cannot be loaded until the solrconfig.xml errors are resolved.

  3. Issue the following command to recover indexes on each upgraded Solr node. On the first node upgraded, this process should happen after the Solr configuration file has been uploaded. Note that in the command below you will need to substitute the name of your Solr core.
    curl -v "http://localhost:8983/solr/admin/cores?action=CREATE&solr core.solr&recovery=true"

The following is an example of how to perform these steps using our Solr-based demos. If you wish to do this on a test cluster, first run the solr, wiki and logging demos on a test cluster running the earlier version of DataStax Enterprise.

Go to the directory containing your Solr application. For example, go to the demos directory:

  • Binary installation

    cd install_location/demos

  • Package installation

    cd /usr/share/dse-demos

Run the following commands to HTTP-POST your modified custom solrconfig.xml to DSE Search. For example, from the demos or dse-demos directory, run the following commands:

  • From the solr_stress directory:

    curl -v --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'
http://localhost:8983/solr/resource/demo.solr/solrconfig.xml

  • From the wikipedia directory:

    curl -v --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'
http://localhost:8983/solr/resource/wiki.solr/solrconfig.xml

  • From the log_search directory:

    curl -v --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'
http://localhost:8983/solr/resource/Logging.log_entries/solrconfig.xml

    After running each curl command, a SUCCESS message appears.

This step is only required once, when the first node is upgraded.

After each node is upgraded, run the CREATE command with the recovery option set to true, and the distributed option set to false:

curl -v "http://localhost:8983/solr/admin/cores?action=CREATE&name=demo.solr&recovery=true"
$ curl -v  "http://localhost:8983/solr/admin/cores?action=CREATE&name=wiki.solr&recovery=true"
$ curl -v  "http://localhost:8983/solr/admin/cores?action=CREATE&name=Logging.log_entries&recovery=true"

Partitioner

Edit the cassandra.yaml file to change the partitioner setting to match the previous partitioner. The default RandomPartitioner (org.apache.cassandra.dht.RandomPartitioner) was the default partitioner prior to Apache Cassandra™ 1.2.

CQL 3

Do not issue any CQL 3 queries until all nodes are upgraded and schema disagreements are resolved.

Security recommendations

The client_encryption_options for enabling client-to-node SSL have been removed from dse.yaml starting in 3.1.2. To enable client-to-node SSL, set the option in the cassandra.yaml file.

Before upgrading from 3.0.x to 3.2.x, if you use these DataStax Enterprise security features, adjust the replication strategy and options in the cassandra.yaml file to configure a replication factor for the dse_auth keyspace greater than 1:
  • Kerberos
  • Object permission management (internal authorization)
  • Internal authentication
Adjust the replication factor for dse_auth on each node in the cluster. After updating the cassandra.yaml file and restarting the node, run nodetool repair to repair the first range returned by the partitioner for the keyspace:
nodetool repair dse_auth -pr

This should only take a few seconds to complete.

The new version of Apache Cassandra™ updates the security options. First simply merge the following settings into the new configuration files:
  • authenticator
  • authorizer
  • auth_replication_strategy
  • auth_replication_options
  • any other diffs
Use the old settings while you are upgrading the cluster so that backward compatibility is maintained. For example, the new file contains the old, Cassandra 1.1 authenticator and authorizer options at this point:
  • authenticator: com.datastax.bdp.cassandra.auth.PasswordAuthenticator
  • authorizer: org.apache.cassandra.auth.CassandraAuthorizer
If you are upgrading a secure cluster, there could be a significant delay to each node's first startup as the security migration takes place (up to 1 minute). The delay is due to ensuring that the ring is fully connected before the migration starts. During the upgrade of a secure cluster, you might see a security related error message (documented below). You will see the following message in the log when the node has completed the migration:
INFO [NonPeriodicTasks:1 ] 2013-06-22 15:01:08,173
Auth.java  (line 208 ) Migration of legacy auth data is complete.
You should now switch to org.apache.cassandra.auth implementations in cassandra.yaml.

After all nodes have been upgraded, change these options to the new Cassandra 1.2 values and perform a rolling restart as explained below.

Note: If using Kerberos authentication, there are no credentials data to migrate, but user records must still be updated. Merge the related diffs from the old to the new file.
  1. Edit the cassandra.yaml to switch to the official Apache versions of PasswordAuthenticator and CassandraAuthorizer:
    authenticator: org.apache.cassandra.auth.PasswordAuthenticator
authorizer: org.apache.cassandra.auth.CassandraAuthorizer
  2. Remove or comment out these options from the cassandra.yaml file:
    • auth_replication_strategy
    • auth_replication_options
    • replication_factor
    Note:

    If you have not disabled both auth_replication_strategy and replication_factor, you will see an error. For information about correcting this error, see Issues in the DataStax Enterprise 3.2.5 release notes.

  3. Optionally, adjust the replication factor of the system_auth keyspace. The amount of data in this keyspace is typically very small, so leaving it replicated across the cluster is relatively cheap.

Virtual nodes (vnodes)

DataStax recommends using vnodes only on datacenters running Cassandra workloads. To disable vnodes on datacenters that run Hadoop or Solr workloads, set num_tokens to 1 in cassandra.yaml.

Solr

If you make changes to the configuration of a Solr node after upgrading, you must set the type mapping correctly as explained in Configuring the Solr type mapping version.