OpsCenter Release Notes

7 May 2020

Changes in 6.7.8

The following changes are included in this release.

• Permissions are now more restrictive on the SSL directory for package installs. (OPSC-16136)
• Added the ability to enable hostname verification on LDAP SSL/TLS connections. (OPSC-16354)
• Changed rpc address logging from info to debug. (OPSC-16456)
• New LDAP authentication options which also enable support for multiple authentication sources. (OPSC-16464)
• Removed IP constraint on [agents] reported_interface to allow use of hostnames for failover. (OPSC-16470)
• Fixed an issue where OpsCenter metrics storage would limit queries incorrectly to a subset of nodes. (OPSC-12413)
• Improved failover logging. (OPSC-13463)
• Changed OpsCenter disconnect cluster so that it does not automatically remove the associated LCM cluster. (OPSC-15026)
• Updated dependency of JGroup to prevent vulnerabilities in OpsCenter. (OPSC-16105)
• Updated c3p0 library to prevent vulnerabilities in OpsCenter. (OPSC-16109)
• Fixed vulnerabilities in the OpsCenter user interface. (OPSC-16221)
• Fixed an issue that caused a 404 error when retrieving a DataStax Agent configuration value of false. (OPSC-16384)
• DataStax Agent now cleans up diagnostic tarball files on disk after transfer. (OPSC-16507)
• Fixed an issue where OpsCenter user interface session caches were invalidated but not cleaned up properly. (OPSC-16572)
• Fixed issues in how OpsCenter generates redirect responses during login. (OPSC-16011)
• Prevented OpsCenter from recommending upgrading to OpsCenter 6.8.x if there is a DSE 5.1.x cluster. (OPSC-16577)
• Improved the documentation for [ui] storagemap_ttl in opscenterd.conf to help users correctly configure the parameter. (OPSC-14320)
• Corrected an issue which caused the nodes in the ring view to appear gray if the client-to-node encryption section was missing from cassandra.yaml. (OPSC-16255)

• Added a [labs] swift_cli_skip_diff_after_upload Boolean configuration option to increase performance after files are uploaded. (OPSC-16312)
• Added datacenter selection to restore. (OPSC-16369)
• Performance improvements to restores that do not use sstableloader. (OPSC-16429)
• Changed the restore process to store size information with the backup to a destination so it no longer has to be queried before starting a restore. (OPSC-16430)
• File transfers from destinations during a restore can now be configured to run in parallel using [agents] restore_parallel_factor. This property defaults to 1 to remain compatible in behavior with older releases. (OPSC-16541)
• Fixed several issues that could prevent destinations from getting cleaned up from the cluster configuration file after a restore. (OPSC-13253)
• Improved error message when a node does not have access to the data required for a restore. (OPSC-16042)
• Corrected an issue that prevented a datacenter snapshot from being taken when nodes outside of that datacenter had problems with their DataStax Agents. (OPSC-16338)
• Fixed issues preventing a sstableloader bypass restore of a single datacenter. This works with only new backups of the datacenter. (OPSC-16341)
• When running a backup for select datacenters, the disk space check no longer queries nodes not included in the backup. (OPSC-16407)
• Fixed an Azure backup issue where not all data was being returned when a container had more than 5,000 items. (OPSC-16409)
• Fixed an issue causing unneeded checks of remote file sizes during restore from a destination. (OPSC-16431)
• Removed the need for host-ids to match when doing a restore without sstableloader. (OPSC-16528)
• Added support for Swift S3 backup and restore. (OPSC-16240)
• OpsCenter now cleans up directories created during a diagnostic. (OPSC-16300)
• Fixed an issue that caused a restore to fail when working with tens of thousands of SSTables for a single node and keyspace. (OPSC-16463)
• Fixed an issue which could cause a StackOverflowException during backups, restores, and metrics gathering. (OPSC-15977)
• Fixed check-2i-cardinality warning caused by OpsCenter backup_reports index. (OPSC-15895)

• Fixed Lifecyle Manager (LCM) job failure conditions that previously resulted in jobs stuck in RUNNING status that could not be forcefully terminated. (OPSC-16116)
• Fixed OpsCenter and LCM workflows that resulted in seemingly identical clusters. (OPSC-16520)

• Fixed and issue with the All Graphs option when adding a graph metric to a dashboard graph. (OPSC-16034)
• Fixed an issue that could cause OpsCenter to failover during a diagnostics tarball collection. (OPSC-16228)
• Added chrony output to the diagnostic tarball. (OPSC-16560)
• Fixed an issue where the diagnostic tarball download link would be incorrect when behind a proxy with a subpath. (OPSC-15566)
• Improved the performance of the metric fetcher when querying values from the storage cluster. (OPSC-16559)

• Repair Service temporary files are cleaned up more quickly. (OPSC-15982)

4 December 2019

Changes in 6.7.7

The following changes are included in this release.

• Upgraded OpsCenter and DataStax Agent dependencies to address security vulnerabilities. (OPSC-16090, OPSC-16148)

• Added OpsCenter labs feature that allows Swift CLI as a backup location. (OPSC-16061)
• Added [labs] swift_cli_sync_status_delay_seconds configuration option to set a delay between a Swift CLI backup and when its contents can be verified. (OPSC-16266)

• Changed default Best Practice rule schedules so they will execute over an hour rather than all at once. (OPSC-16023)

• Improved configuration of percentile alerts with a configurable duration and separate histogram aggregation window for the calculation. (OPSC-16115)
• Fixed errors showing up in the agent.log when iostat does not support -s option. (OPSC-16181)
• Fixed an issue that could prevent alerts rules from populating in the user interface after enabling or disabling an alert rule. (OPSC-16233)

• Fixed Lifecycle Manager (LCM) jobs hanging under certain conditions. Such conditions will now result in job failure with details in the opecenterd logs. (OPSC-16176)
• Fixed a bug that caused Meld to ignore the PID file of an existing Meld process. (OPSC-16100)

• Fixed issues with the DataStax Agent diagnostic download after failover. (OPSC-16166)
• Corrected an issue that caused the DataStax Agent rapidly spawn new threads when trying to restart Repair Service while OpsCenter is down. This issue caused the DataStax Agent to reach the maximum operating system thread limit. (OPSC-16213)

5 November 2019

Changes in 6.7.6

The following change is included in this release.

• Validates the SSTable format change introduced in DSE 6.7.6, which prevented some restores from completing. Changed the SSTable version check to pass for all compatible SSTable versions without requiring the exact SSTable version used by the DSE version. (OPSC-16237)

23 September 2019

Changes in 6.7.5

The following changes are included in this release.

• If a keystore or truststore file fails to load, OpsCenter logs the keystore or truststore file that failed to load. (OPSC-13632)
• Sensitive information, including passwords and S3 tokens, are omitted from diagnostic tarball collection. (OPSC-14760)
• The OpsCenter user interface properly displays responses returned from its API. (OPSC-15815)

• When trying to view NodeSync metrics for a table that is ignored by the metrics system, a warning displays. When enabling NodeSync for an OpsCenter rollup table, a warning displays. (OPSC-14614)
• Provides new Insights diagnostic data tarball for download only as requested by DataStax Support. (OPSC-15945)

• Corrected an issue that prevented the search index from rebuilding after a point-in-time restore. (OPSC-15809)
• Fixed a bug in the commitlog cleanup throttle that prevented future cleanups from running. (OPSC-15869)
• Improved debug logging to provide table and keyspace information for troubleshooting restore issue. (OPSC-16111)

• Fixed a bug in LCM that falsely reported old and new cluster passwords as invalid when the cause of the authentication failure was a failure to achieve consistency level QUORUM. (OPSC-15830)
• Improved the usability of configuring an HTTP proxy when adding a repository in LCM. (OPSC-15526)
• After three unsuccessful attempts to update definitions, OpsCenter prints a log message:

  Experienced 3 consecutive failures downloading definitions, disabling updates until the next restart. Instead of continuing to check for definitions and logging a stacktrace error, OpsCenter does not attempt to update definitions again until OpsCenter restarts. (OPSC-10468)

• LCM jobs fail if the datacenter num_tokens value does match the number of initial_tokens supplied for the node. Added the ability to edit initial_token for nodes in LCM. (OPSC-14431)
• When adding a repository with the Manually configure DataStax repo option on systems that use the yum package manager, LCM will no longer falsely report this error message: the dse-full package cannot be found. (OPSC-15621)
• Corrected default performance_max_threads value in LCM from dse.yaml for DSE 6.0 and DSE 6.7. (OPSC-15699)
• Added memtable_space_in_mb parameter to LCM. Fixed memtable_allocation_type default in LCM definitions to match DSE 6.7. (OPSC-15769)
• Fixed a bug that showed duplicate clusters in OpsCenter when re-running an install job on an existing cluster. (OPSC-15888)
• Fixed a bug that prevented changes to the LCM datacenter model after the first complete install job. (OPSC-15892)

• Improved logging statements in Repair Service to clarify which type of repair job is being logged. (OPSC-15913)
• Removed URI length restriction for [repair_service] ignore_keyspaces and [repair_service] ignore_tables to ensure specified keyspaces and tables are excluded from subrange repairs. To use this new endpoint, upgrade both opscenterd and all the DataStax Agents. (OPSC-13245)

21 June 2019

Changes in 6.7.4

The following change is included in this release.

• Agents will now detect if there are symlinks in their DataStax Enterprise (DSE) data directories. If symlinks are detected, OpsCenter displays a warning and On Server snapshot cleanup will not run. (OPSC-15974)
• Corrected an issue where an On Server bypass restore would place symlinks in working data directories instead of hard links. (OPSC-15957)

8 May 2019

Highlights

Core

• Even when configured clusters are unavailable, users can interact with the Cannot connect to cluster screen, including being able to select text or click hyperlinks.

Backup Service

• If an attempt to create a hard link fails, improved restore error handling allows the restore to fall back to using copy for all future files.
• The Backup Service now uses the LZ4 algorithm for compression, which makes compressions four times faster, but results in 10% larger files. Older backups that use gzip can still be restored with OpsCenter.

Lifecycle Manager (LCM)

• Can configure time between polling if DSE is up and running with dse_healthcheck_attempt_wait_time configuration option for LCM.
• LCM now displays a useful error and link to documentation when trying to edit or use a Config Profile with an unsupported DSE version.

Changes in 6.7.3

The following changes are included in this release.

• Improved memory handling of tooltips. (OPSC-6524)
• Fixed an issue where the wrong log4j.properties configuration file was included in the installed_location/agent/conf directory of opscenterd tarball distributions. (OPSC-14729)
• Fixed an issue where users could not interact with the Cannot connect to cluster screen, including being able to select text or click hyperlinks, when configured clusters are unavailable. (OPSC-15767)

• Improved metric queries to use Best Practice rules by using prepared statements with parameters for queries from rollup tables. (OPSC-13149)
• Fixed an issue where alert types that permit immediate notification could not be edited if immediate notification was selected. (OPSC-13704)
• When recreating a keyspace with nested UDTs, fixed dependency order of UDTs to control the order OpsCenter restores the UDTs. (OPSC-15127)
• Improved restore error handling when an attempt to create a hard link fails. After one such failure, the restore will now fall back to using copy for all future files. (OPSC-15608)
• Backup service now uses the LZ4 algorithm for compression, which makes compressions four times faster, but results in 10% larger files. Older backups that use gzip can still be restored with OpsCenter. (OPSC-15633)
• Fixed an issue that prevented some detailed information about backups from displaying in the OpsCenter interface. (OPSC-15200)
• Added logging to clarify errors the can occur when creating a snapshot with DSE 6.0.3 or DSE 6.0.4. (OPSC-15309)
• Fixed an issue where the default locations for commit log backups could not be entered in the OpsCenter interface. (OPSC-15683)
• Fixed a bug where only a subset of keyspaces were shown in the Restore from Backup dialog keyspace selector. (OPSC-15717)
• Upgraded DSE Java Driver to 1.8.1. (OPSC-15719)
• Fixed bug in parsing tiered_storage_options. (OPSC-15724)
• Fixed an issue that would cause a SSTableloader bypass restore to fail if restoring an entire keyspace with materialized views from an On Server location without specifying any tables. (OPSC-15739)
• Fixed an issue that caused the minimum percentile alerts to return a high value instead of 0 when given only zeros for data. (OPSC-15762)

• Include all queries and tables in the error message relating to the Use prepared statements Best Practice rule. (OPSC-15647)

• Corrected an issue where an alert could trigger emails even after it was deleted. (OPSC-13861)

• Definitions added to allow configuration of CASSANDRA_HEAPDUMP_DIR in cassandra-env.sh. (OPSC-12377)
• LCM will now install appropriate packages to read from the apt-transport-https repository on Debian Stretch. (OPSC-15018)
• Added more logging to the post-install script for the Debian package installation. (OPSC-15106)
• LCM now displays a useful error and link to documentation when trying to edit or use a Config Profile with an unsupported DSE version. (OPSC-15381)
• LCM jobs now always stop the DSE service, even if it thinks the service is already stopped. This can happen if the user is using a different service management system to manually start/stop DSE outside of LCM. (OPSC-15556)
• Updated LCM to properly call systemctl daemon-reload. (OPSC-15563)
• LCM install jobs with manual repository setups on systems that use the yum package manager will no longer falsely report that the dse-full package cannot be found. (OPSC-15621)
• Fixed LCM API uniqueness check for entities on PUT requests when one or more unique key fields are missing from the request. The LCM API now gives an error when the user submits a change for a read-only field. Previously, the LCM API ignored these changes with no error. (OPSC-15656)
• LCM cluster model and related resources are now immutable while there are associated jobs in the queue. (OPSC-15714)
• LCM import cluster and DataStax Agent installation jobs will no longer run concurrently with any other job type. This change prevents the troublesome situation where one of these job types is run on a cluster that is already managed while another job type is active on that cluster. (OPSC-15716)
• Added dse_healthcheck_attempt_wait_time configuration option for LCM to configure the time between polling if DSE is running. (OPSC-15731)
• Improved tarball extraction in meld to avoid potential too many open files issue. (OPSC-15831)
• Removed auto download of Oracle JRE due to Oracle licensing changes. (OPSC-15871)

• Fixed to prevent leaking file descriptors by cleaning up open resources after each repair job completes. (OPSC-15466)
• Fixed Repair Service alert regression when opscenterd is restarted during repairs. (OPSC-15746)

4 March 2019

Highlights

Core

• Rollover log files for opscenterd and other configured rollover log files are now included in the diagnostic tarball. (OPSC-12141)
• Made a change to always remove the server response header from opscenterd web server responses for security purposes to combat vulnerabilities in a known version of Twisted web server. (OPSC-14866)
• OpsCenter now adds the HttpOnly flag to its login session cookie to help prevent XSS attacks. (OPSC-14868)
• Packages now include extra build information in the following files to aid in troubleshooting and support: ds_branch.txt, ds_version.txt, and ds_timestamp.txt. These files now include branch, commit, version, and timestamp information. (OPSC-15201)
• Added an authentication plugin framework to allow custom authentication strategies. (OPSC-14507)

Backup Service

• Backups to Amazon S3 locations now support Amazon S3 Transfer Acceleration. (OPSC-10271)
• Added KMS Managed Encryption (SSE-KMS) as an option for Amazon S3 backups. (OPSC-15170)
• Improved the speed of backups to local file system destinations. (OPSC-15530)
• Changed restore operations that bypass SSTableloader to use hard links (if possible) instead of copying files to reduce the required disk space when restoring from a destination. (OPSC-14894)
• Increased the speed when backing up to a local file system and fixed a bug with throttling speed to the local file system. (OPSC-14876)
• Fixed a bug that could cause problems when restoring materialized views. (OPSC-14727)
• Fixed an issue where AWS role-based credentials were being held in memory after expiration. (OPSC-15138)
• Fixed an issue where a restore would fail if the backup was taken shortly after dropping a column from a table. (OPSC-13029)

Lifecycle Manager (LCM)

• Added configuration files in LCM to enable exporting collected metrics from DSE Metrics Collector to supported monitoring tools. (OPSC-15094)
• LCM now performs client-side health checks against each node in the job by executing a local query before the job is considered successful on that node. (OPSC-14848)
• Fixed an issue where LCM jobs would fail to terminate when aborted. (OPSC-14410)

Changes in 6.7.1

The following changes are included in this release.

• Logging levels for OpsCenter and all DataStax Agents in a cluster can now be set with a curl command. (OPSC-7105)
• The DataStax Agent now supports Transport Layer Security (TLS) with remote JMX. (OPSC-8375)
• Disconnecting a cluster no longer fails if the cluster configuration file was already removed. (OPSC-11318)
• The OpsCenter UI now properly escapes all JSON responses returned from its API. (OPSC-11508)
• Corrected an issue that prevented the failover OpsCenter instance from connecting to the DataStax Agents during failover. (OPSC-11742)
• Fixed an issue when sending alerts to multiple recipients via email. (OPSC-12131)
• Rollover log files for opscenterd and other configured rollover log files are now included in the diagnostic tarball. (OPSC-12141)
• Added multi-role support for LDAP authentication. Added additional roles field to /users and /users/\{username\} GET responses for getting all roles that a user belongs to. (OPSC-12740)
• Added documentation for the /logout API. (OPSC-13147)
• Made changes to include the output of DESCRIBE FULL SCHEMA in the diagnostic tarball downloaded from OpsCenter. (OPSC-13290)
• Fixed an issue with the POST URL event plugin that was logging error messages during successful POST operations. (OPSC-13643)
• Improved favicon display in several web browsers. (OPSC-13788)
• Removed non-determinism from RollupReporter restart. (OPSC-13798)
• DNS names no longer try to resolve during configuration validation. (OPSC-14181)
• Fixed an issue where OpsCenter indicated that a change to the OpsCenter keyspace replication strategy failed, when selecting the link from the notification about the OpsCenter keyspace using SimpleStrategy for replication in a multi-datacenter environment. (OPSC-14406)
• OpsCenter now drops compact storage option from all tables inside the configured OpsCenter keyspace. (OPSC-14442)
• Enhanced OpsCenter to properly log exceptions from LDAP if group names contain Unicode characters. (OPSC-14452)
• Fixed an issue where tmp_dir settings in the [agents] section of the opscenterd.conf configuration file was not being sent to DataStax Agents. (OPSC-14722)
• Made a change to always remove the server response header from opscenterd web server responses for security purposes to combat vulnerabilities in a known version of Twisted web server. (OPSC-14866)
• OpsCenter now adds the HttpOnly flag to its login session cookie to help prevent XSS attacks. (OPSC-14868)
• Fixed an issue where a benign warning message would be logged when opscenterd started. (OPSC-14912)
• Reduced memory usage in opscenterd when requests are made to the DataStax Agent. (OPSC-15037)
• Packages now include extra build information in the following files to aid in troubleshooting and support: ds_branch.txt, ds_version.txt, and ds_timestamp.txt. These files now include branch, commit, version, and timestamp information. (OPSC-15201)
• Updated DataStax Agent key generation to utilize RSA instead of DSA and updated documentation. (OPSC-15123)
• Fixed an issue where STOMP would not come up on some platforms using the LANG=C.UTF-8 variable. (OPSC-15251)
• Upgraded Dojo to version 1.14. This Dojo release addresses security vulnerabilties. See the NIST website for more information. OpsCenter does not use the vulnerable features and previous OpsCenter versions are not affected. (OPSC-15327)
• Fixed an issue where STOMP is attempting to reconnect, causing OpsCenter to hang. (OPSC-15357)
• Fixed an issue where OpsCenter generates too many asynchronous CQL queries, which results in a NoHostAvailableException. (OPSC-15461)
• Fixed an issue where OpsCenter returned an error page instead of redirecting to the login page when authentication was enabled. (OPSC-15630)

• Added proxy server support for S3 backup and restore. (OPSC-6978)
• Added support to use system default credentials for Amazon S3 backups. (OPSC-5161)
• Backups to Amazon S3 locations now support Amazon S3 Transfer Acceleration. (OPSC-10271)
• Enabled support for restores on DSE clusters using configuration encryption and client-to-node encryption. (OPSC-12312)
• Improved exception handling relating to periodic failures before and after running the backup script. (OPSC-12405)
• Fixed an issue that caused backup history pagination to fail when many events shared the same time. (OPSC-12836)
• Added support for backing up encryption keys other than system_key, plus support for backing up multiple keys. (OPSC-12914)
• Fixed an issue where a restore would fail if the backup was taken shortly after dropping a column from a table. (OPSC-13029)
• Removed requirement that the backup_storage_dir must be on the same partition as the DataStax Agent tmp_dir. (OPSC-13108)
• Fixed an issue where snapshots containing one or more SASI index could not be restored due to an invalid SSTable name exception during the validation checks. (OPSC-13314)
• Changed commit log archiving when starting the DataStax Agent to handle all commit logs as a batch instead of processing each file individually. (OPSC-13782)
• Commit logs can now be archived while backing up a snapshot. The execution of the commit log retention policy has been refactored and will now be more consistent. (OPSC-14126)
• Fixed a small rendering issue in the Restore from Backup: Other Location form. (OPSC-14226)
• Fixed a bug requiring the user to double click on the plus (+) button when selecting a keyspace from the Create Backup dialog. (OPSC-14228)
• Upgraded Java AWS SDK to version 1.11.328. (OPSC-14454)
• Fixed the destination pre-check to fail the entire backup if the pre-check fails. (OPSC-14508)
• Fixed an issue when using multi-level prefix paths as locations for the Backup Service. (OPSC-14687)
• Restored marker in the backups location dialogue indicating that AWS key and secret are required for user supplied credentials. (OPSC-14702)
• Fixed an error that displayed when clicking Cancel after OpsCenter prompts whether you want to delete a scheduled job. (OPSC-14715)
• Fixed a bug that could cause problems when restoring materialized views. (OPSC-14727)
• Increased the speed when backing up to a local file system and fixed a bug with throttling speed to the local file system. (OPSC-14876)
• Fixed an issue where AWS regions specified as remote_backup_region values in the cluster configuration file were not used as bucket defaults. (OPSC-14775)
• Changed restore operations that bypass SSTableloader to use hard links (if possible) instead of copying files to reduce the required disk space when restoring from a destination. (OPSC-14894)
• Fixed an issue that caused schema files to be sent repeatedly to a destination during a backup. (OPSC-15009)
• Reduced memory required when Backup Service is taking a snapshot. (OPSC-15046)
• Fixed an issue with Point In Time restores when an On Server destination is the only destination defined. (OPSC-15052)
• Added a fix to sort keyspaces in the selection list. (OPSC-15114)
• Added node IP in Destination validation error messages. (OPSC-15166)
• Added KMS Managed Encryption (SSE-KMS) as an option for Amazon S3 backups. (OPSC-15170)
• Fixed a bug that generated an error indicating that a Solr core could not be created because the associated table did not exist. This error occurred when tables backed by Solr cores were dropped before running the restore, but the keyspace was not dropped. (OPSC-15187)
• Prevented errors about missing schema.cql for system tables when taking a backup (OPSC-15198)
• Fixed missing destination UI bug by keeping UI cache of destinations in sync with the server when a backup is run. (OPSC-15206)
• Fixed a bug where the text value of the button label was passed in the parameter to remove the selected backup destination when selecting Delete Backup Data. (OPSC-15215)
• Fixed an issue causing restores to fail when restoring a keyspace containing user defined types (UTD). (OPSC-15308)
• OpsCenter no longer truncates tables in the OpsCenter keyspace even if selected as part of the restore. (OPSC-15310)
• Changed permissions so that the diagnostic tarball only requires read permissions on files. (OPSC-15382)
• Fixed a bug with region selection when using the Amazon AWS CLI to use the region specified for the destination. (OPSC-15435)
• Fixed an issue with the Backup Service and SSTable attached secondary indexes. (OPSC-15489)
• Improved the speed of backups to local file system destinations. (OPSC-15530)
• Improved handling of exceptions when trying to back up to an Amazon S3 bucket that does not exist. (OPSC-15544)
• Fixed a bug where the Location form does not close when adding a new location for a Point In Time restore. (OPSC-15573)
• Removed dependency of az-cli when using Microsoft Azure for backups. (OPSC-15602)
• Redacted storage keys from log files when running commands from AzCopy to improve security. (OPSC-15672)

• Fixed an error in the Secondary indexes cardinality Best Practice rule where a list of nodes displayed instead of information about too many secondary indexes in keyspaces and tables. (OPSC-15209)

• Support added for new Read Coordination and Hint metrics added in DSE 5.0.12 (Read Requests - Local Node Non Replica, Read Requests - Preferred Other Replicas, Hints on Disk, Hint Replay Success Rate, Hint Replay Error Rate, Hint Replay Timeout Rate, Hint Replay Received Rate). (OPSC-12230)
• Improved the color scheme for node status in OpsCenter. (OPSC-12618)
• Fixed an issue where the graph zoom button does not work when the legend is expanded. (OPSC-13413)
• Fixed an issue causing invalid data for the Total Compactions Completed metric. The fix will be published as a live definition file update for all existing OpsCenter 6.7.x installations. Metric data collected prior to deploying the fix will remain invalid. (OPSC-13657)
• Fixed an issue where nodes were sometimes incorrectly indicated as DOWN in OpsCenter when they were actually UP. (OPSC-14299)
• Corrected an unhandled exception when retrieving metrics from clusters with datacenters that contain hyphens in their names. (OPSC-14747)
• Added current_value_timestamp to data returned by the /alerts/fired route. (OPSC-14762)
• Implemented a change so that data for average time and average request for Solr cores comes from QueryMetrics MBean rather than older Solr MBeans. (OPSC-14845)

• Fixed an issue where decommissioning a node would sometimes result in an error indicating that OpsCenter Tried to cancel an already-cancelled event. (OPSC-14016)

• Enhanced link styling on the NodeSync status page to make clickable entities more distinct and obvious. (OPSC-15514)

• Fixed an issue where some configuration parameters in the [agent_config] section of cluster_name.conf could not be parsed by the DataStax Agents. (OPSC-12258)
• Removed blocking cql queries to improve application performance. (OPSC-15574)

• Fixed an issue where column names overlapped in the key_provider edit dialog in LCM. (OPSC-10990)
• Added information about upgrade jobs to the cluster workspace tooltip. (OPSC-13107)
• Disallow LCM cluster import when opscenterd is not fully communicating with a cluster. (OPSC-13367)
• Enlarged the ssh-key field in LCM to improve readability when entering SSH keys. (OPSC-13509)
• Fixed a bug in LCM cluster import when there are datacenter or node-specific configuration options. (OPSC-13546)
• Fixed a bug where LCM UI form dialogs would reset values to the original state while being edited. (OPSC-14025)
• Fixed an issue where the native_transport_port_ssl setting was ignored when changing the default CQL password. (OPSC-14030)
• Added tooltips for several custom URLs to provide examples of what LCM needs them to point to. (OPSC-14060)
• LCM UI form dialogs now display a loading animation while loading form values on slow connections. Form inputs will be grayed out while in the loading state. (OPSC-14123)
• Modified the locations that LCM uses to traverse the cluster model when determining the SSH management port. (OPSC-14258)
• Fixed an issue where LCM jobs would fail to terminate when aborted. (OPSC-14410)
• Improve LCM error messages when invalid characters are submitted for usernames or entity names. (OPSC-14411)
• Improved error message when LCM attempts to update the default admin password in the cassandra CQL account, but no new password has been specified on the LCM cluster model. (OPSC-14586)
• Performing a minor upgrade on yum-based systems when dse-demos is installed no longer upgrades DSE to the latest available version. (OPSC-14608)
• LCM now performs client-side health checks against each node in the job by executing a local query before the job is considered successful on that node. (OPSC-14848)
• Added configuration files in LCM to enable exporting collected metrics from DSE Metrics Collector to supported monitoring tools. (OPSC-15094)
• Fixed a bug preventing node_install_idle_timeout from being respected in LCM. (OPSC-15376)
• Fixed a bug in LCM platform detection code for Amazon Linux 2. (OPSC-15637)

• Implemented a fix to prevent the first Repair Service progress alert from being triggered before the period (in seconds) configured by error_logging_window elapses first. (OPSC-13166)
• When a distributed subrange repair is paused, the current-task file is not removed from the DataStax Agent file system. (OPSC-14612)
• Parsed the DataStax Agent tokenranges JSON size value as Java long rather than int to avoid crashing due to int overflow during subrange repairs. (OPSC-15182)
• Prevent Repair Service from crashing due to a Java long overflow when a repair task is persisted in SQLite. (OPSC-15225)
• Change to use a Java long primitive for calculation to prevent the Repair Service from crashing due to a Java integer overflow. (OPSC-15255)

5 December 2018

Highlights

• Upgraded to the new DataStax Enterprise Java driver with full support for DSE 6.7.
• Added Azure Blob Storage and Amazon S3-compatible storage support for backup and restore operations.
• When running a restore, OpsCenter now uses a new default restore mechanism that bypasses SSTableloader to significantly increase performance.
• Meld, the remote execution component of Lifecycle Manager that deploys configuration changes to DSE nodes, no longer depends on a Python runtime.
• LCM managed Java installations now use OpenJDK instead of Oracle Java.

Review the list of new features in OpsCenter to see the latest additions and compare across versions.

Changes in 6.7.0

The following changes are included in this release.

• Enhanced OpsCenter to support LDAP searches for users without specifying an Organizational Unit (OU). Also added the ability to follow LDAP referrals. (OPSC-13384)
• Added a dismissable banner notification about the end of public availability of Oracle Java 8. (OPSC-14672)
• Upgraded to the new DataStax Enterprise Java driver with full support for DSE 6.7. (OPSC-15060)

• Added Azure Blob Storage support for backup and restore operations. (OPSC-4777)
• Added Amazon S3-compatible storage support for backup and restore operations. (OPSC-4994)
• Removed requirement that the backup_storage_dir must be on the same partition as the DataStax Agent tmp_dir. (OPSC-13108)
• Changed the restore of materialized views to happen after data is restored. (OPSC-14574)
• Moved AWS S3 destination validation from OpsCenter to the DataStax Agent. (OPSC-14611)
• Fixed an issue with where the AWS Credentials Provider Chain was not respected in relations to IAM Roles. (OPSC-14939)
• Fixed an issue in OpsCenter where editing an S3 destination after a restart shows Enable S3 server-side encryption and Enable S3 transfer acceleration enabled when they are not. (OPSC-14982)
• Corrected an issue that caused the restore status to initially show 100% then reset to 0%. (OPSC-14995)
• Added a loading indicator to the Configure and Restore screen when populating the list of keyspaces and graphs. (OPSC-14999)
• Fixed a memory leak in the Backup Service job execution cache. (OPSC-15015)
• When running a restore, users now have the ability to bypass SSTableloader. (OPSC-15038)
• Fixed an issue with point-in-time restores when an On Server destination is the only destination defined. (OPSC-15052)
• Fixed Solr restore handling to be case sensitive. (OPSC-15117)
• Fixed a bug that generated an error indicating that a Solr core could not be created because the associated table did not exist. This error occurred when tables backed by Solr cores were dropped before running the restore, but the keyspace was not dropped. (OPSC-15187)
• Updated restore logic so materialized views will be restored if their underlying table is recreated. (OPSC-15188)
• Added a note and a warning that when restoring a DSE Graph backup without selecting Use sstableloader, DSE must be restarted to ensure all data is available. (OPSC-15283)

• Improved LCM error messages for Oracle JRE download errors. (OPSC-13846)
• Meld, the remote execution component of Lifecycle Manager that deploys configuration changes to DSE nodes, no longer depends on a Python runtime. (OPSC-14076)
• Changed Opscenterd to OpsCenter in various error-messages and strings. If API clients or log parsers key behavior off of that string, they must be updated. (OPSC-14191)
• Updated third-party dependencies for LCM. (OPSC-14495)
• Improved error messages when LCM fails to upload Meld to the target nodes. (OPSC-14517)
• LCM managed Java installations now use OpenJDK instead of Oracle Java. Existing installs will be migrated to OpenJDK during the next install/upgrade job if no action is taken. See the Upgrade Guide for details. (OPSC-14674)
• Updated the Java setup page to only display JCE options when manage-java is selected. (OPSC-14897)
• Improved LCM error message when the Oracle JRE download times out on a target node. (OPSC-14844)
• The DSE health check timeout (dse_healthcheck_startup_timeout) can now be set to configure how long LCM will wait for DSE to start up. (OPSC-15014)
• Field descriptions now display by default in LCM configuration profiles. (OPSC-15036)
• LCM health checks for DSE startup now retry if the service script says the service is not running. It was observed that the status might be inaccurate early on during service start. (OPSC-15043)
• LCM now defaults to setting auto_bootstrap to true for all nodes. This change improves install-job robustness, but requires that initial node startup be performed one at a time. Concurrency policies other than Automatic (default) are not recommended. (OPSC-15189)
• The default job concurrency policy in LCM now behaves the same as the Single install policy. This default improves robustness of install jobs at the cost of executing jobs more slowly. (OPSC-15192)
• The meld.pyz file name changed to meld. (OPSC-15817)

• Fixed a bug for Distributed Subrange Repair (DSR) to honor the max_parallel_repairs property, which was remaining at a value of 1 regardless of the specified value. (OPSC-14947)

The following are known issues that exist in OpsCenter 6.1 and later versions. Each item has a link to more details including workarounds when available. These issues will be addressed in future releases where possible. If you have any questions, contact DataStax Support for assistance.

• Data loss can occur if you delete the contents of the snapshot directory after using OpsCenter to perform a restore operation from an on-server backup to a cluster with an unchanged topology. (OPSC-15957)
  Warning: To prevent data loss on OpsCenter 6.7.1 or 6.7.3, do not delete snapshots after a restore. For more information, see the DataStax Support Alerts or contact DataStax Support.

• When running an LCM job and attempting to abort or terminate the job while it is in progress, termination fails to stop the job unless the abort request is issued prior to the first node completing. Issuing a terminate or abort command after the first node has finished running has no effect. The job will continue to run to completion as if the terminate command had not been issued. (OPSC-14410)

• OpsCenter does not automatically remove compact storage from its keyspaces when upgrading to OpsCenter 6.5.0. For important details, see Compact storage no longer supported. (OPSC-14442)
• When restoring materialized views, OpsCenter does not correctly wait for the cluster schema to settle, which can cause errors when data is restored to the table on which the view is based. (OPSC-13029)

• For DSE versions 5.1 and later, slow query data is only available since the last time the DataStax agent was restarted. (OPSC-11702)
• If there are approximately 75 or more keyspaces, the DataStax Agent /tokenranges API call runs out of memory with the default heap size. As a temporary workaround, adjust the agent heap size. (OPSC-11975)
• When using OpsCenter to restore a backup that contains multiple SASI indexes, some or all of these indexes might not restore correctly. The indexes appear in the table schema but might not function correctly. Indexes should be validated at restore time and rebuilt if errors are detected. For more information, see CREATE CUSTOM INDEX (SASI). (OPSC-11746)
  Note: SASI indexes are experimental for DSE. DataStax does not support SASI indexes for production.

• A large number of log messages might display regarding requests to /pit-cleanup if there are a large number of existing commit logs in the staging directory. (OPSC-8349)
• Insufficient permissions on the staging directory can cause the agent to exhaust inotify watches on the system over time. (OPSC-10732)
• Users will see an ungraceful error+stack trace in opscenterd.log if accessing a cluster through the UI/API that no longer exists. The error message contains ERROR: Unhandled error in Deferred: There are no clusters with name or ID.... This error message is harmless. (OPSC-8819)
• Enabling SNMP alerts may cause opscenterd to hang on startup in some slower environments. (OPSC-9314; see More Details)
• For DSE versions earlier than 5.0.7, the DataStax Agent can only estimate partition sizes and counts per node or keyspace for repairs by using JMX stats. For DSE versions 5.0.7 and later, the Datastax Agent queries the system size_estimates table for a more precise estimate of partition sizes and counts per range. (OPSC-11417, OPSC-11590)
• For DSE versions 5.0 and later, object permissions currently are not persisted with an OpsCenter backup and thus are not re-applied when that backup is restored. As a result, users must manually manage object permissions externally from OpsCenter. For more details (no workaround available at this time), see the KB support article. (OPSC-11015)
• The solr-index-size (displayed as Search: Core Size) metric in the OpsCenter Monitoring UI is unavailable for DSE versions 5.1.0 through 5.1.3. (OPSC-12267)

• Lifecycle Manager (LCM)

  • Lifecycle Manager is not currently compatible with DSE Transparent data encryption. See Encrypted DSE configuration values for more details. (OPSC-7529)
  • DSE Graph properties: DSE Graph configuration in dse.yaml, which is configurable through LCM Config Profiles. All Graph properties in dse.yaml can be managed through the LCM UI with the exception of gremlin_server.serializers and gremlin_server.scriptEngines. If you are using LCM and need to customize these properties, be sure to leverage the LCM API to make the changes. Future changes to the Config Profile using the LCM UI will retain properties set through the API.
  • When configuring credentials in a Repository, special characters such as #, $, and so forth are supported, but non-ascii unicode characters are not. (OPSC-8921)