OpsCenter Release Notes

OpsCenter release notes provide information about new and improved features, known and resolved issues, and bug fixes.

Release impacts

Understand upgrade impacts, compatibility with DSE versions, and known issues.

Before reading release notes, review the following information to understand upgrade impacts, compatibility with DataStax Enterprise (DSE) versions, and known issues for your version of OpsCenter.

Upgrade Information

Important: Review the 6.7 changes pertinent to the release as noted in Upgrading DSE OpsCenter. Configuration and other notable changes are provided in detail.

Compatibility

To see which versions of DataStax Enterprise (DSE) are supported with OpsCenter 6.x, see the DSE OpsCenter compatibility with DSE.

Known Issues

Important: Review the list of known issues before running a new OpsCenter version on a production DSE cluster.

OpsCenter 6.7.0 release notes

Release notes for the OpsCenter and Lifecycle Manager version 6.7.0 release.

5 December 2018

Highlights

  • Upgraded to the new DataStax Enterprise Java driver with full support for DSE 6.7.
  • Added Azure Blob Storage and Amazon S3-compatible storage support for backup and restore operations.
  • When running a restore, OpsCenter now uses a new default restore mechanism that bypasses SSTableloader to significantly increase performance.
  • Meld, the remote execution component of Lifecycle Manager that deploys configuration changes to DSE nodes, no longer depends on a Python runtime.
  • LCM managed Java installations now use OpenJDK instead of Oracle Java.

Review the list of new features in OpsCenter to see the latest additions and compare across versions.

Changes in 6.7.0

The following changes are included in this release.

Core
  • Enhanced OpsCenter to support LDAP searches for users without specifying an Organizational Unit (OU). Also added the ability to follow LDAP referrals. (OPSC-13384)
  • Added a dismissable banner notification concerning Oracle Java policy changes. (OPSC-14672)
  • Upgraded to the new DataStax Enterprise Java driver with full support for DSE 6.7. (OPSC-15060)
Backup Service
  • Added Azure Blob Storage support for backup and restore operations. (OPSC-4777)
  • Added Amazon S3-compatible storage support for backup and restore operations. (OPSC-4994)
  • Removed requirement that the backup_storage_dir must be on the same partition as the DataStax Agent tmp_dir. (OPSC-13108)
  • Changed the restore of materialized views to happen after data is restored. (OPSC-14574)
  • Moved AWS S3 destination validation from OpsCenter to the DataStax Agent. (OPSC-14611)
  • Fixed an issue with where the AWS Credentials Provider Chain was not respected in relations to IAM Roles. (OPSC-14939)
  • Fixed an issue in OpsCenter where editing an S3 destination after a restart shows Enable S3 server-side encryption and Enable S3 transfer acceleration enabled when they are not. (OPSC-14982)
  • Corrected an issue that caused the restore status to initially show 100% then reset to 0%. (OPSC-14995)
  • Added a loading indicator to the Configure and Restore screen when populating the list of keyspaces and graphs. (OPSC-14999)
  • Fixed a memory leak in the Backup Service job execution cache. (OPSC-15015)
  • When running a restore, users now have the ability to bypass SSTableloader. (OPSC-15038)
  • Fixed an issue with point-in-time restores when an On Server destination is the only destination defined. (OPSC-15052)
  • Fixed Solr restore handling to be case sensitive. (OPSC-15117)
  • Fixed a bug that generated an error indicating that a Solr core could not be created because the associated table did not exist. This error occurred when tables backed by Solr cores were dropped before running the restore, but the keyspace was not dropped. (OPSC-15187)
  • Updated restore logic so materialized views will be restored if their underlying table is recreated. (OPSC-15188)
  • Added a note and a warning that when restoring a DSE Graph backup without selecting Use sstableloader, DSE must be restarted to ensure all data is available. (OPSC-15283)
Provisioning
  • Improved LCM error messages for Oracle JRE download errors. (OPSC-13846)
  • Meld, the remote execution component of Lifecycle Manager that deploys configuration changes to DSE nodes, no longer depends on a Python runtime. (OPSC-14076)
  • Changed Opscenterd to OpsCenter in various error-messages and strings. If API clients or log parsers key behavior off of that string, they must be updated. (OPSC-14191)
  • Updated third-party dependencies for LCM. (OPSC-14495)
  • Improved error messages when LCM fails to upload Meld to the target nodes. (OPSC-14517)
  • LCM managed Java installations now use OpenJDK instead of Oracle Java. Existing installs will be migrated to OpenJDK during the next install/upgrade job if no action is taken. See the Upgrade Guide for details. (OPSC-14674)
  • Updated the Java setup page to only display JCE options when manage-java is selected. (OPSC-14897)
  • Improved LCM error message when the Oracle JRE download times out on a target node. (OPSC-14844)
  • The DSE health check timeout (dse_healthcheck_startup_timeout) can now be set to configure how long LCM will wait for DSE to start up. (OPSC-15014)
  • Field descriptions now display by default in LCM configuration profiles. (OPSC-15036)
  • LCM health checks for DSE startup now retry if the service script says the service is not running. It was observed that the status might be inaccurate early on during service start. (OPSC-15043)
  • LCM now defaults to setting auto_bootstrap to true for all nodes. This change improves install-job robustness, but requires that initial node startup be performed one at a time. Concurrency policies other than Automatic (default) are not recommended. (OPSC-15189)
  • The default job concurrency policy in LCM now behaves the same as the Single install policy. This default improves robustness of install jobs at the cost of executing jobs more slowly. (OPSC-15192)
Repair Service
  • Fixed a bug for Distributed Subrange Repair (DSR) to honor the max_parallel_repairs property, which was remaining at a value of 1 regardless of the specified value. (OPSC-14947)

Known and resolved issues for OpsCenter 6.x

Known issues, workarounds, and resolved issues for the OpsCenter and Lifecycle Manager 6.0 and later versions.

The following are known issues that exist in OpsCenter 6.x versions. Each item has a link to more details including workarounds when available. These issues will be addressed in future releases where possible. If you have any questions, please contact DataStax Support for assistance.

  • Users may observe a large number of log messages about requests to /pit-cleanup if there are a large number of existing commit logs in the staging directory. (OPSC-8349)
  • Insufficient permissions on the staging directory can cause the agent to exhaust inotify watches on the system over time. (OPSC-10732)
  • Users will see an ungraceful error+stack trace in opscenterd.log if accessing a cluster through the UI/API that no longer exists. The error message contains "ERROR: Unhandled error in Deferred: There are no clusters with name or ID...". This error message is harmless. (OPSC-8819)
  • Enabling SNMP alerts may cause opscenterd to hang on startup in some slower environments. (OPSC-9314; see More Details)
  • Failure to follow the required prerequisite instructions to install Oracle Java SE Runtime Environment 8 (JRE or JDK) before installing OpsCenter 6.0 on Ubuntu 16.04 results in installation of OpenJDK 9, which is not currently supported. (OPSC-10778)
  • For DSE versions earlier than 5.0.7, the agent can only estimate partition sizes and counts per node or keyspace for repairs by using JMX stats. For DSE versions 5.0.7 and later, the agent queries the system size_estimates table for a much more precise estimate of partition sizes and counts per range. (OPSC-11417, OPSC-11590)
  • For DSE versions 5.0 and later, object permissions currently are not persisted with an OpsCenter backup and thus are not re-applied when that backup is restored. As a result, users must manually manage object permissions externally from OpsCenter. For more details (no workaround available at this time), see the KB support article. (OPSC-11015)
  • The solr-index-size (displayed as Search: Core Size) metric in the OpsCenter Monitoring UI is unavailable for DSE versions 5.1.0 through 5.1.3. (OPSC-12267)
  • (Applicable to OpsCenter version 6.1 or later only) For DSE versions 5.1 and later, slow query data is only available since the last time the DataStax agent was restarted. (OPSC-11702)
  • (Applicable to OpsCenter version 6.1 or later only) If there are approximately 75 or more keyspaces, the DataStax agent /tokenranges API call runs out of memory with the default heap size. As a temporary workaround, adjust the agent heap size. (OPSC-11975)
  • (Applicable to OpsCenter version 6.1 or later only) When using OpsCenter to restore a backup that contains multiple SASI indexes, some or all of these indexes might not restore correctly. The indexes appear in the table schema but might not function correctly. Indexes should be validated at restore time and rebuilt if errors are detected. For more information, see CREATE CUSTOM INDEX (SASI). (OPSC-11746)
    Note: SASI indexes are experimental for DSE. DataStax does not support SASI indexes for production.
  • (Applicable to OpsCenter versions 6.0.x, 6.1.x, and 6.5.0) OpsCenter does not automatically remove compact storage from its keyspaces when upgrading to OpsCenter 6.5.0. For important details, see Compact storage no longer supported. (OPSC-14442)
  • Lifecycle Manager (LCM)
    • Lifecycle Manager is not currently compatible with DSE Transparent data encryption. See Encrypted DSE configuration values for more details. (OPSC-7529)
    • OPSC-8851, in 6.0.2, improved resiliency of Lifecycle Manager in situations where there is high latency between the OpsCenter daemon and nodes in the cluster. This release improves upon the problem; however, there are still known issues in high latency scenarios that will be addressed in a future release. (OPSC-9853)
    • DSE Graph properties (DSE 5.0.1+ only): DSE Graph configuration in dse.yaml, which is configurable through LCM Config Profiles. All Graph properties in dse.yaml can be managed through the LCM UI with the exception of gremlin_server.serializers and gremlin_server.scriptEngines. If you are using LCM and need to customize these properties, be sure to leverage the LCM API to make the changes. Future changes to the Config Profile using the LCM UI will retain properties set through the API.
    • When configuring credentials in a Repository, special characters such as #, $, and so forth are supported, but non-ascii unicode characters are not. (OPSC-8921)
    • (Applicable to OpsCenter 6.5 only) When running an LCM job and attempting to abort or terminate the job while it is in progress, termination fails to stop the job unless the abort request is issued prior to the first node completing. Issuing a terminate or abort command after the first node has finished running has no effect; the job will continue to run to completion as if the terminate command had not been issued. (OPSC-14410)
Known Issues Fixed in OpsCenter 6.0.8
Known Issues Fixed in OpsCenter 6.0.6
  • When modifying the Config Profile of an existing cluster in Lifecycle Manager, the Cluster Connection Settings in OpsCenter are now automatically updated after running a configure job. (OPSC-8544)
  • Fixed an issue where OpsCenter failed to retrieve the diagnostic tarballs from the agents if SSL was enabled between OpsCenter and the agents. (OPSC-10701)
Known Issues Fixed in OpsCenter 6.0.5
  • Fixed LCM repository authentication bug when special characters exist in credentials (as with most DataStax Academy usernames). (OPSC-10817)
Known Issues Fixed in OpsCenter 6.0.3
  • Any encrypted config values generated since Opscenter 6.0 (and prior to fix in 6.0.3) will need to be re-encrypted. (OPSC-10244).
  • Backups/Restores will not work with keyspace names longer than 32 characters on DSE 4.7 or 4.8. (OPSC-9563)
  • The Repair Service fails to auto-restart after a node is decommissioned. Manually starting the Repair Service resolves this issue. (OPSC-9244)
  • Some items in the Lifecycle Manager UI may not automatically update if they are modified outside of the current UI session; for example, via the API directly or in another UI session. If multiple users might be concurrently modifying the same cluster, please be sure to refresh the UI before making any changes. (OPSC-9306)
Known Issues Fixed in OpsCenter 6.0.2
  • g1-gc-opts in cassandra-env.sh are not immediately editable when using G1 garbage collection by default. To workaround this issue, change the garbage collector to something other than G1 and back again. (OPSC-9556)
  • Max heap size in cassandra-env.sh is not editable using the LCM UI. To workaround this issue, users can set the -Xmx and -Xms JVM properties directly via additional-jvm-opts further down on the cassandra-env.sh section in the Config Properties page. (OPSC-9546)
  • The use_tls setting in email alerts does not currently work as expected. Users can still configure email alerts to work with TLS-enabled servers by setting use_ssl=1 and use_tls=0. Please contact DataStax Support if you have any issues. (OPSC-9451)
  • Automatic definition file updates are not dynamically reloaded for new versions of DSE. If you see an error for "Unsupported or invalid version of DSE" in the UI, try restarting opscenterd. (OPSC-9468)
  • Some users may see intermittent job failures with an IncompleteRead error (OPSC-8851; see More Details)
  • The LCM UI has some rendering issues in older versions of Safari (<=8). The workaround is to use a newer version of Safari or another supported browser. (OPSC-9123)
Known Issues Fixed in OpsCenter 6.0.1
  • opscenterd fails to properly resolve relative symlinks to Java. (OPSC-9344; see More Details)
  • When installing an agent on a node for the first time, address.yaml is owned by the root user. The only OpsCenter functionality this affects directly is Automatic Failover, which will not work until ownership or permissions are updated. If the agent has previously been installed on the node, ownership is not affected. (OPSC-9336; see More Details)
  • S3 and Local FS backups fail for keyspaces leveraging the new Materialized Views feature in DSE 5.0. On Server backups are not affected. (OPSC-9328; see More Details)
  • Users must ensure tables that leverage the new User Defined Aggregates and User Defined Functions features in DSE 5.0 exist prior to running a restore. OpsCenter cannot automatically re-create these tables, but can successfully restore the data to existing tables. (OPSC-9261; see More Details)