Recommendations for DSE OpsCenter production deployments

DSE OpsCenter is purpose-built to visually manage, provision, upgrade, backup and restore, and monitor DSE clusters in production. When running at significant scale, it is important to follow OpsCenter recommendations and suggested settings for your deployment. This information outlines important considerations for deploying OpsCenter at scale.

The following recommendations are generally recommended for all OpsCenter production deployments.

Use a separate OpsCenter storage cluster

Use a separate storage cluster for OpsCenter data for optimal performance.

By default, OpsCenter stores its data in the OpsCenter keyspace on the DataStax Enterprise (DSE) cluster being monitored. While storing OpsCenter data on the monitored cluster is convenient, in production environments, it is highly recommended to store OpsCenter data on a separate DSE cluster.

Because OpsCenter metrics can consume excessive disk space, configuring a separate storage cluster for metrics and controlling data collection prevents issues with performance or collecting metrics when using the same cluster.

The OpsCenter storage cluster must be sized appropriately to handle the necessary workload. Each node in the monitored DSE cluster writes its metrics data directly to the OpsCenter keyspace on the separate storage cluster. The metrics data is subsequently read by the OpsCenter server to show related graphs in the OpsCenter interface.

For a cluster with one table, each monitored node writes ~340 times per minute to the separate storage cluster to store its granular monitoring data. The number of writes increases by 40 with each additional table.

Therefore, in a cluster with 10 nodes and one table, there would be 3,400 writes per minute to the storage cluster (340 writes x 10 nodes). Adding 10 tables increases the number of writes by 400 per node (40 writes x 10 tables). Adding those additional writes to the original number of writes (400 writes + 340 writes) equates to 740 writes per minute, or 7,400 writes for a cluster with 10 nodes and 10 tables.

These calculations assume that the replication factor remains the same, and that the data is distributed evenly among your partition keys. Increasing the replication factor or distributing data unevenly across partition keys impacts linear scaling.

It is imperative to consider this performance need when designing your deployment, in addition to the overhead for reads and other (less intensive) operations.

At a minimum, DataStax recommends that your separate storage cluster be between three and six nodes for production.

Set the OpsCenter keyspace replication strategy

Use the NetworkToplogy replication strategy for the OpsCenter keyspace.

OpsCenter creates the OpsCenter keyspace with the SimpleStrategy replication strategy (not recommended for production deployments). The OpsCenter keyspace replication factor and strategy should be properly configured based on the configuration of your storage cluster to ensure OpsCenter data is performant and always available.

If configured incorrectly, the following symptoms can surface:

OpsCenter dashboards load slowly.
Metrics render slowly.
OpsCenter freezes intermittently.

DataStax recommends the NetworkTopologyStrategy replication strategy and a replication factor of two or greater.

The OpsCenter Replication Advisor includes rules that check for these scenarios so that they can be addressed. If these rules are triggered, take the appropriate action to fix the issue before problems arise.

Configure Java heap settings

Tune the Java heap settings for the OpsCenter server and DataStax Agent.

If using the OpsCenter Backup Service or Repair Service, DataStax recommends increasing the default heap settings for the OpsCenter server (to 4 GB) and DataStax Agent (to 2 GB) when running in production. In addition, DataStax recommends adding an additional 1 GB of heap to the OpsCenter server for each additional cluster being monitored.

Use LCM for online and offline DSE deployments

DataStax strongly recommends using Lifecycle Manager (LCM) to visually install, configure, and manage DSE deployments in a secure and automated manner. LCM simplifies complex tasks such as certificate generation and configuration management.

For deployments that have limited or no access to the external internet, LCM can use a package proxy to install DSE.

Configure DSE Metrics Collector

Configure DSE Metrics Collector for DSE 6.8 clusters to enhance OpsCenter Monitoring.

When monitoring clusters that are running DataStax Enterprise (DSE) 6.8 or greater, the DSE Metrics Collector provides the ability to send DSE and OS metrics directly to the centralized monitoring tool of your choice to complement OpsCenter Monitoring. Follow the instructions to use DSE Metrics Collector with preconfigured dashboards to show critical metrics that are most important to your organization.

Cluster synchronization

Use the DSE OpsCenter NodeSync Service or Repair Service to maintain cluster synchronization.

The built-in NodeSync feature eliminates the need to run the Repair Service. NodeSync reduces operator complexity and is the recommended solution for cluster synchronization.

NodeSync settings

Ensure that NodeSync is enabled for all relevant keyspaces and tables through the OpsCenter interface. Monitor NodeSync and tune the NodeSync rate if needed to ensure that synchronization happens within the NodeSync deadline. For more information, see Use the NodeSync Service.

Repair Service settings

Use the Repair Service to run repair operations across nodes and their replicas.

When enabled, the Repair Service runs repair operations, which synchronize the most current data across nodes and their replicas, including repairing any corrupted data encountered at the file system level. Once enabled, the Repair Service resumes running where it left off. If this behavior is not desired (for example, when altering tokenranges_partitions), stop and start the Repair Service in the OpsCenter interface to restart synchronization.

Ensure that you are running the latest OpsCenter patch version (6.8.x) when running the Repair Service. Nodes managed by OpsCenter 6.8.x must be running DSE 6.0.0 or later.

Parallel repair operations

The max_parallel_repairs property controls how many repair operations are issued to the cluster in parallel. This number is computed automatically by dividing the number of nodes by the largest replication factor of any keyspace in the cluster included in the Repair Service. The automatic calculation ensures nodes do not receive multiple repair requests.

This value might need to be tuned manually when keyspaces with a high replication factors exist. However, modifying this value might result in multiple repair requests being sent to a node.

However, ensure that the maximum for parallel subrange repairs is not being automatically calculated to a very low value. This automatic calculation is typically caused by having a keyspace in a cluster with a high replication factor.

Slower repairs

Slowing the progression of the Repair Service might benefit clusters that have a limited amount of data, but can reduce the impact of the Repair Service.

Consider the following advanced Repair Service configurations:

Increasing the min_repair_time property (by doubling it or more) allows more time to pass between Repair Service requests.
Increasing the time_to_completion_target_percentage throttle property (from current value up to 100) further slows Repair Service tasks.

Faster repairs

Increasing the speed of the Repair Service progression is necessary when a cluster is large or dense, which requires the Repair Service to run more quickly to synchronize the cluster. Increasing the Repair Service speed will increase the load on a cluster due to the increased synchronization activities.

Consider the following changes to increase the speed of the Repair Service:

Enabling distributed subrange repairs can increase throughput when repairing large clusters (100+ nodes).
Decreasing the min_repair_time property (can be as low as 0) shortens the amount of time between Repair Service tasks.
Decreasing the time_to_completion_target_percentage throttle property (can be as low as 0) increases the speed of Repair Service tasks.

Backup Service settings

Configure the DSE OpsCenter Backup Service to ensure that data is backed up without error.

When backing up data to a destination using the Backup Service, ensure that OpsCenter has permission to read and write to and from the backup location, as well as to and from the DataStax Enterprise (DSE) data directory. Additionally, ensure that the backup destination is used exclusively for OpsCenter backups and contains no additional files or directories.

Local filesystem: Ensure that the user running the DataStax Agent has read and write permission for the backup directory.
AWS S3 and Generic S3: Proper privileges for Amazon Web Services (AWS) S3 and Generic S3 backups must be configured for the IAM role accessing the AWS S3 bucket used for backups. For production workloads, DataStax strongly recommends enabling the S3 AWS CLI feature for better performance.
Microsoft Azure Blob storage: Be aware of the dependencies for each DSE node when you add an Azure backup location to back up data to Microsoft Azure.