New features in DSE OpsCenter 6.1

Changes in features, configuration files, metrics, and APIs in OpsCenter 6.1.

New features

opscenterd.conf

The location of the opscenterd.conf file depends on the type of installation:
  • Package installations: /etc/opscenter/opscenterd.conf
  • Tarball installations: install_location/conf/opscenterd.conf

address.yaml

The location of the address.yaml file depends on the type of installation:
  • Package installations: /var/lib/datastax-agent/conf/address.yaml
  • Tarball installations: install_location/conf/address.yaml

cluster_name.conf

The location of the cluster_name.conf file depends on the type of installation:
  • Package installations: /etc/opscenter/clusters/cluster_name.conf
  • Tarball installations: install_location/conf/clusters/cluster_name.conf

The following new and improved features are highlighted for the current DataStax Enterprise (DSE) OpsCenter version 6.1 release.

DataStax Enterprise 5.1 Support for provisioning and monitoring DataStax Enterprise 5.1 clusters:
Repair Service Overhauled the OpsCenter Repair Service, including refactoring for reliability and performance:
Backup Service Added features for flexible backup and restore workflows:
  • Ability to enable backup failure alerts as an event to the Event log, which can be configured to alert as an email or posted URL to a chat room.
  • Back up and restore a datacenter.
  • Added a rule to the Best Practice Service when Commit Log Archiving is inconsistent.
  • Added an unset throttle warning when restoring a backup.
  • Cloning a backup from a Local FS location.
  • Point-in-time restores from a Local FS location.
DataStax agents improvements Improved flexibility for managing DataStax agents:
  • Added backward-compatibility with agent versions that allow similar but different agent versions to co-exist with the installed OpsCenter version without loss of feature service. This feature provides flexibility with regard to maintaining a DSE cluster within OpsCenter.
  • Exposed the DataStax Agent API in a Swagger UI console. See Enable and access the DataStax Agent API.
  • Enhanced UI design for conveying Agents Status:
    • Vastly improved visibility of agent version and agent compatibility, including better built-in guidance on when it is imperative that all agents get upgraded.
    • Tooltips explain each column purpose in the Agent Status.
    • A red icon appears on the Agents tab for increased visibility when agents are unhealthy.
    • Replaced text in each status column and row with intuitive icons with tooltips for faster scanning of agent status.
New metrics Added metrics for DSE Graph and messaging latency:
Best Practice Service Added a rule for Commit Log Archiving Setting Enabled Consistency Best Practice Rule. The rule ensures new nodes added to a cluster with Commit Log Archiving (CLA) enabled also have CLA enabled, which helps prevent data loss particularly during a Point-in-Time restore using the Backup Service. The rule checks to make sure all nodes within a cluster have Commit Log Archiving enabled.
Access Logs from the OpsCenter UI Access critical logs from within the Node Details dialog in OpsCenter monitoring. These logs are a labs feature. See Viewing logs from node details.
Lifecycle Manager (LCM) LCM now runs jobs concurrently across different clusters for faster job completion.

Updates from OpsCenter 6.0

The following changes are updates from the OpsCenter 6.0 major releases.

Support for using EC2 IAM roles for OpsCenter S3 backups

Added support to use system default credentials for Amazon S3 backups as described in Working with AWS Credentials. See Adding an Amazon S3 backup location for instructions on adding an Amazon S3 backup location.

Support for multiple user roles using LDAP authentication

Users can have multiple roles when using LDAP authentication. If the list of a user's groups map to more than one role in OpsCenter, the user will be granted each of the listed roles, and their resulting OpsCenter permissions will be the merging of permissions for all of their roles. See Adding a role for an LDAP user.

Important: In OpsCenter 6.1.10 and later and OpsCenter 6.5.3 and later, you must update custom scripts and applications that use the OpsCenter API if you want to use multiple user roles with LDAP authentication. If a custom script or application that uses the OpsCenter API did not account for multiple user roles, and a user has multiple roles, the script or application will fail because the role attribute cannot be found. The single role attribute will be provided for users that have only one role. If your application or script has users with only one role, then updates are not required for continued use.

Hadoop deprecated

DataStax has deprecated support for Hadoop as of DSE 5.1 and OpsCenter 6.1. DataStax Support can discuss alternative options for your organization.

Attempts to import a DSE cluster that has Hadoop enabled will fail. Lifecycle Manager can only import clusters that do not have Hadoop datacenter workloads. LCM cannot run jobs on clusters, datacenters, or nodes that have Hadoop enabled. Please contact DataStax Support for assistance with disabling Hadoop before retrying an import or attempting to run the job again.

The hive-site.xml configuration file has been relocated to the Spark section of the Configuration Profiles in LCM. The Hadoop section has been removed from LCM Config Profiles.

The Vnodes enabled on Hadoop nodes Best Practice Rule has been removed from OpsCenter monitoring.

Deprecated Hadoop-related configuration

The following port configuration option has been removed from opscenterd.conf: [hadoop] base_job_tracker_proxy_port.

The following configuration options have been removed from cluster_name.conf:
  • [hadoop] job_tracker_http_port
  • [hadoop] job_tracker_port
  • [hadoop] job_tracker_proxy_port
  • [kerberos] job_tracker_client_principal
The following agent configuration options have been removed from address.yaml:
  • hadoop_conf_location
  • hadoop_log_location

Repair Service configuration changes

The Repair Service features have been significantly simplified and improved from both a usability and reliability standpoint. For complete details, see the Repair Service documentation in OpsCenter.

The following configuration options have been removed:
  • incremental_range_repair
  • ks_update_period
  • max_err_threshold
  • alert_on_repair_failure
  • repair_estimation_factor
The following configuration options have been added:
  • incremental_sleep
  • incremental_threshold
  • prioritization_page_size
  • offline_splits
  • ignore_keyspaces
  • ignore_table
  • time_to_completion_target_percentage
  • incremental_repair_datacenters
  • tokenranges_http_timeout
  • max_down_node_retry

Backup Service configuration changes

OpsCenter uses the Amazon SDK for S3 functionality. The [cloud] accepted_certs configuration option is no longer necessary and has been removed from opscenterd.conf.

The backup file queue has been modified into a backup job queue. The backup_file_queue_size configuration option has been removed from address.yaml.

The following configuration options were added to opscenterd.conf under [backup_service] for the Amazon S3 proxy:

  • s3_proxy_host
  • s3_proxy_port
Improvements to the S3 throttle feature available in the Add Location dialog in the Backup Service have deprecated the following options:
  • max-seconds-to-sleep
  • seconds-to-read-kill-channel
  • multipart-chunk-size
  • read-buffer-size
  • write-buffer-size

The options have been removed from address.yaml.

Best Practice Service Rule changes

The following changes were made to the Best Practice Service:

  • The Commit Log Archiving Setting Enabled Consistency rule has been added. The rule ensures new nodes added to a cluster with Commit Log Archiving (CLA) enabled also have CLA enabled to prevent any potential data loss, particularly during a Point-in-Time restore using the Backup Service. If the Commit Log Archiving setting is not consistent across all nodes in a cluster, the rule fails. For more information, see Configuring commit log backups.
  • The Vnodes enabled on Hadoop nodes Best Practice Rule has been removed.

Deprecated agent install script and configuration option

The following files related to legacy agent install features are no longer needed in OpsCenter 6.x and have been removed:
  • ./agent/pkg/bin/install_agent.sh
  • ./opscenterd/agent/bin/install_agent.sh

The agent install process is now internal to Opscenter, and no external script is necessary.

The path_to_installscript configuration option has been removed from the [agents] section of opscenterd.conf. The option if present in your configuration is ignored for OpsCenter 6.1 and later. Updating your configuration to remove the path_to_installscript is not necessary.

DataStax Agent backward compatibility

DataStax Agent backward-compatibility starts with OpsCenter 6.1 going forward. The DataStax Agent has backward compatibility with the installed OpsCenter version, which provides flexibility with the timing of upgrading agents. Before DataStax Agent backward compatibility, the OpsCenter version and the DataStax Agent version were required to match exactly. The DataStax Agent Status View lists any potential features that will not operate for a specific agent version and provides guidance on when it is imperative to upgrade agents.

Note: The OpsCenter 6.1 server is not compatible with DataStax Agent versions earlier than 6.1.0. All DataStax Agents must be upgraded to version 6.1.0.

Check the OpsCenter release notes and known issues for any changes that break DataStax Agent backward-compatibility for features.

Definitions files directory relocation

OpsCenter definitions are no longer considered configuration files, and have been moved from /etc/opscenter/definitions to /var/lib/opscenter/definitions. Existing definitions files will be left in place in /etc/opscenter/definitions, but will be ignored by OpsCenter. The existing definitions files in /etc/opscenter/definitions may optionally be deleted manually.