Storing collection data on a separate cluster

Configure a separate storage cluster for metrics. Store collection data on a separate DataStax Enterprise (DSE) cluster as an alternative to the default of OpsCenter storing data in an OpsCenter keyspace on the same DSE cluster being monitored.

OpsCenter metrics can consume excessive disk space. Configuring a separate storage cluster for metrics and controlling data collection can prevent issues with performance or collecting metrics.

In production environments, DataStax strongly recommends storing data in a separate DSE cluster.

There are two different cluster connections in the DataStax Agent. When monitored and storage clusters are separate, the DataStax Agent checks the health of the monitored cluster on one of those connections, and writes metrics to the storage cluster on the other connection. If a separate storage cluster is not configured, both connection pools point to the same DSE cluster instance.

OpsCenter supports connecting to a DSE storage cluster when TLS/SSL is enabled. See Configuring SSL/TLS for DSE using LCM.

OpsCenter supports distinct SSL or authentication connection settings between a monitored cluster and its corresponding storage cluster. Previously, the authentication mechanism and credentials had to be identical between the storage cluster and the monitored clusters.

Prerequisites

The seed nodes must be accessible without Kerberos security.
A unique keyspace must be used for each DSE cluster monitored by OpsCenter. If you are storing data for multiple clusters, DataStax recommends adding the cluster name as a suffix to the default keyspace name of OpsCenter. For example, set the keyspace name to OpsCenter_Cluster1Storage for the storage cluster to differentiate it from the OpsCenter keyspace for the monitored cluster.

You can set most of the DataStax Agent configuration options in the [agent_config] section of the cluster_name.conf file. The options in the [agent_config] section must match the corresponding configuration option name in address.yaml. Setting DataStax Agent options through the cluster configuration file sets the corresponding property in address.yaml on every node. Some properties or some cases might require setting these properties directly in address.yaml on applicable DataStax Agents. Setting DataStax Agent configuration options that require a list entry (value1,value2,value3 and so forth) is not supported in cluster_name.conf.

For a complete list of cluster_name.conf options, see Cassandra connection properties. For a complete list of address.yaml options, see DataStax Agent configuration.

Procedure

Locate the cluster_name.conf file. The location of this file depends on the type of installation:
- Package installations: /etc/opscenter/clusters/cluster_name.conf
- Tarball installations: installation_location/conf/clusters/cluster_name.conf
Open the storage cluster configuration file cluster_name.conf for editing. The cluster_name.conf represents the named configuration file for a particular cluster. Replace cluster_name with your actual cluster name.
Similar authentication: If your monitored and storage clusters use similar authentication, add a [storage_cassandra] section with the applicable storage configuration options for your environment.
```
[storage_cassandra]
username = opscuser
password = password
seed_hosts = host1, host2
cql_port = 9042
keyspace = OpsCenter_Cluster1Storage
```
See Similar authentication between the monitored and storage cluster for a list of available configuration options.
Different authentication: If your storage and monitored clusters require different authentication or SSL settings, open cluster_name.conf for editing and add settings in the [agents], [storage_cassandra], and [cassandra] sections as applicable for your environment. The settings in the [cassandra] section are for the monitored cluster:
```
[agents]
ssl_keystore = /etc/dse/keystore
ssl_keystore_password = example
storage_ssl_keystore = /etc/dse/keystore_storage
storage_ssl_keystore_password = example2

[storage_cassandra]
username = opscuser
password = password
seed_hosts = host1, host2
cql_port = 9042
keyspace = OpsCenter_Cluster1Storage

[cassandra] #monitored
username = baz
password = test
```
See Different authentication or SSL settings for a list of available configuration options.
If your environment requires setting options in address.yaml for reasons such as integration with third-party CMS applications, set the storage configuration options (without the monitored_ prefix) and corresponding monitored_ configuration options as appropriate for your environment. If you set these DataStax Agent options in address.yaml, you must do so for every node in the cluster, whereas setting the options in cluster_name.conf automatically propagates the settings to all agents.

See Integrating with other applications for a list of available configuration options.

Locate the address.yaml file. The location of this file depends on the type of installation:
- Package installations: /var/lib/datastax-agent/conf/address.yaml
- Tarball installations: install_location/conf/address.yaml
Restart OpsCenter for the changes to take effect.

After the storage cluster is online, decide whether to keep the metrics from the OpsCenter keyspace on the original cluster.

Choice

Steps

If you need to keep the OpsCenter keyspace metrics and the OpsCenter storage cluster is monitored by OpsCenter

1. Back up the OpsCenter keyspace.

2. Clone it to the OpsCenter storage cluster.

If you need to keep the metrics and the OpsCenter storage cluster is not monitored by OpsCenter

1. Back up the OpsCenter keyspace.

2. Take a snapshot of the OpsCenter keyspace.

3. Load the snapshot into the OpsCenter storage cluster with sstableloader.

If you don’t need the metrics and use Lifecycle Manager (LCM)

Remove the duplicate storage cluster that appears in LCM.

[NOTE] When you configure a separate storage cluster for metrics, the OpsCenter keyspace is recreated on the storage cluster. Because the OpsCenter UUID changes, you will see duplicate clusters in LCM. To remove the duplicate cluster in Lifecycle Manager (LCM), resync the OpsCenter cluster UUID.

If you don’t need metrics and don’t use LCM

Delete the OpsCenter keyspace on the original cluster.

Similar authentication between the monitored and storage cluster

The following [storage_cassandra] configuration options are available in cluster_name.conf for monitored and storage clusters:

[storage_cassandra] seed_hosts: Configure when using a separate OpsCenter storage cluster. For OpsCenter storage clusters, the agents only communicate with the listed seed hosts, consequently ensuring that a sufficient number of storage cluster nodes are placed in the comma-delimited list and that the list is reviewed as nodes are added and removed from the storage cluster.
[storage_cassandra] cql_port: Configure when using a different cluster for OpsCenter storage. The CQL port configured for your cluster, the default port is 9042.
[storage_cassandra] local_dc_pref: To reliably determine cluster information, OpsCenter can require a minimum of 2 nodes to connect to. If you specified a single value for seed_hosts, OpsCenter selects a second node in the cluster to fulfill this requirement. You can specify local_dc_pref as a datacenter name to constrain OpsCenter to use that datacenter to pick the second node.
[storage_cassandra] used_hosts_per_remote_dc: Configure when using a different cluster for OpsCenter storage. If using local_dc_pref, this option specified how many remote dc connections may be used as a fallback, the default value is 1.
[storage_cassandra] connect_timeout: Configure when using a different cluster for OpsCenter storage. Sets the timeout, in seconds, of a native connection from OpsCenter to Cassandra. The default value is 6.0.
[storage_cassandra] bind_interface: Configure when using a different cluster for OpsCenter storage. The interface used for native connections.
[storage_cassandra] connection_pool_size: Configure when using a different cluster for OpsCenter storage. The number of connections to build for the connection pool. The default value is 5.
[storage_cassandra] username: Configure when using a different cluster for OpsCenter storage. The username used to connect to Cassandra if authentication is enabled.
[storage_cassandra] password: Configure when using a different cluster for OpsCenter storage. The password used to connect to Cassandra if authentication is enabled.
[storage_cassandra] send_rpc: Configure when using a different cluster for OpsCenter storage. Specifies whether to send the Cassandra RPC IP to DataStax Agents. The default value is True.
[storage_cassandra] keyspace: The name of the keyspace used for OpsCenter data. The keyspace name must be unique for each managed cluster.
[storage_cassandra] ssl_keystore: The SSL keystore location for OpsCenter to use to connect to Cassandra directly.
[storage_cassandra] ssl_keystore_password: The SSL keystore password for OpsCenter to use to connect to Cassandra directly.
[storage_cassandra] ssl_truststore: The SSL truststore location for OpsCenter to use to connect to Cassandra directly.
[storage_cassandra] ssl_truststore_password: The SSL truststore password for OpsCenter to use to connect to Cassandra directly.

Different authentication or SSL settings

The following configuration options are available in cluster_name.conf for monitored and storage clusters:

[agents] ssl_keystore: The SSL keystore location for DataStax Agents to use to connect to CQL on the monitored cluster.
[agents] ssl_keystore_password: The SSL keystore password for DataStax Agents to use to connect to CQL on the monitored cluster.
[agents] storage_ssl_keystore: The SSL keystore location for DataStax Agents to use to connect to CQL on the storage cluster.
[agents] storage_ssl_keystore_password: The SSL keystore password for DataStax Agents to use to connect to CQL on the storage cluster.
[cassandra] seed_hosts: A Cassandra seed node is used to determine the ring topology and obtain gossip information about the nodes in a cluster. This list should be the same comma-delimited list of seed nodes as the one configured for the DataStax Enterprise cluster by the seeds property in the cassandra.yaml configuration file. The default value is localhost.
[cassandra] rolling_restart_retry_delay: The number of seconds to wait between retry attempts when connecting to Cassandra after restarting a node. Default: 5.
[cassandra] rolling_restart_retry_attempts: The maximum number of connection retry attempts after restarting a Cassandra node. Default: 25.
[cassandra] rolling_restart_error_threshold: A rolling restart will be cancelled if the number of errors during the restart reaches this number. This helps prevent having too many nodes down in a cluster if something catastrophic happens during a rolling restart. Default: 1.
[cassandra] restart_delay: During a rolling restart, the time in seconds OpsCenter waits after sending the command to stop Cassandra before sending the command to start it again. The default is 30 seconds.

Integrating with other applications

The following configuration options are available in address.yaml for storage and monitored clusters:

cassandra_port: Port used to connect to the storage cassandra node. The native transport port. Example: cassandra_port: 9042
cassandra_user: The Username used to connect to storage cassandra when authentication is enabled. Example: cassandra_user: cassandra
cassandra_pass: The password used to connect to storage cassandra when authentication is enabled. Example: cassandra_pass: cassandra [This field may be encrypted for additional security.]
ssl_keystore: The SSL keystore location for the storage cluster that agents use to connect to CQL. Example: ssl_keystore: /etc/dse/conf/.keystore
ssl_keystore_password: The SSL keystore password for the storage cluster that agents use to connect to CQL. Example: ssl_keystore_password: keystore-pass [This field may be encrypted for additional security.]
ssl_truststore: The SSL truststore location for the storage cluster that agents use to connect to CQL. Example: ssl_truststore: /etc/dse/conf/.truststore
ssl_truststore_password: The SSL truststore password for the storage cluster that agents use to connect to CQL. Example: ssl_truststore_password: truststore-pass [This field may be encrypted for additional security.]
monitored_cassandra_port: Port used to connect to the monitored cassandra node. The native transport port. Example: monitored_cassandra_port: 9042
monitored_cassandra_user: The Username used to connect to monitored cassandra when authentication is enabled. Example: monitored_cassandra_user: cassandra
monitored_cassandra_pass: The password used to connect to monitored cassandra when authentication is enabled. Example: monitored_cassandra_pass: cassandra-pass [This field may be encrypted for additional security.]
monitored_ssl_keystore: The SSL keystore location for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_keystore: /etc/dse/conf/.keystore
monitored_ssl_keystore_password: The SSL keystore password for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_keystore_password: keystore-pass [This field may be encrypted for additional security.]
monitored_ssl_truststore: The SSL truststore location for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_truststore: /etc/dse/conf/.truststore
monitored_ssl_truststore_password: The SSL truststore password for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_truststore_password: truststore-pass [This field may be encrypted for additional security.]

Removing duplicate storage cluster with LCM

When you configure a separate storage cluster for metrics, the OpsCenter keyspace is recreated on the storage cluster. Because the OpsCenter UUID changes, you will see duplicate clusters in LCM. To remove the duplicate cluster in Lifecycle Manager (LCM), resync the OpsCenter cluster UUID.

In OpsCenter 6.8.1 and later, LCM finds the OpsCenter cluster by its UUID first and then the intersecting seed list. This improvement prevents LCM from listing a duplicate cluster.

Prerequisites

Configure a separate OpsCenter storage cluster for metrics.
Create a unique keyspace to use for each DSE cluster monitored by OpsCenter. If you are storing data for multiple clusters, DataStax recommends adding the cluster name as a suffix to the default keyspace name of OpsCenter. For example, set the keyspace name to OpsCenter_Cluster1Storage for the storage cluster to differentiate it from the OpsCenter keyspace for the monitored cluster.

Procedure

Get the unique_cluster_id for the OpsCenter storage cluster.

curl http://127.0.0.1:8888/cluster_name/cluster/unique_cluster_id

// 20190604153731
// http://127.0.0.1:8888/cluster_name/cluster/unique_cluster_id

"unique_cluster_id"

Get a list of LCM clusters to get the unique opsc-cluster-id and href_URL for the LCM cluster.

curl http://127.0.0.1:8888/api/v2/lcm/clusters/

// 20190604153842
// http://127.0.0.1:8888/api/v2/lcm/clusters

{
  "next": null,
  "previous": null,
  "last": 1,
  "count": 1,
  "per-page": 50,
  "current": 1,
  "results": [
    {
      "opsc-cluster-id": "opsc_cluster_id",
      "name": "*_cluster_name",
      "id": "id",
      "href": "href_URL",
	  ...
    }
  ]
}

The opsc_cluster_id is not expected to match the unique_cluster_id from the curl result.

If opsc-cluster-id and unique_cluster_id do not match, replace the opsc-cluster-id value with the unique_cluster_id using the LCM cluster href_URL.
```
curl -X PUT href_URL -H "Content-Type: application/json" -d '{"opsc-cluster-id": "unique_cluster_id"}'
```
If the OpsCenter and LCM cluster UUIDs match and you have duplicate clusters, contact DataStax Support.
Refresh the LCM web-interface.

The duplicate cluster should no longer be visible.

Removing duplicate from db_monitoring cluster

Get the unique_cluster_id from db_monitoring cluster.

curl http://127.0.0.1:8888/db_monitoring/cluster/unique_cluster_id

Based on the results, the unique_cluster_id is 6885d453-a816-4c9e-b06f-7050d28c50b5.

// 20190604153731
// http://127.0.0.1:8888/db_monitoring/cluster/unique_cluster_id

"6885d453-a816-4c9e-b06f-7050d28c50b5"

Get the unique opsc-cluster-id and href_URL for the db_monitoring LCM cluster.

curl http://127.0.0.1:8888/api/v2/lcm/clusters/

Based on the results, the opsc_cluster_id is f1da1420-7846-4c84-bf7d-399d48e6a51e and the href_URL is [http://127.0.0.1:8888/api/v2/lcm/clusters/cd658793-e763-4e5a-9960-b19fe935f0db](http://127.0.0.1:8888/api/v2/lcm/clusters/cd658793-e763-4e5a-9960-b19fe935f0db).

// 20190604153842
// http://127.0.0.1:8888/api/v2/lcm/clusters

{
  "next": null,
  "previous": null,
  "last": 1,
  "count": 1,
  "per-page": 50,
  "current": 1,
  "results": [
    {
      "opsc-cluster-id": "f1da1420-7846-4c84-bf7d-399d48e6a51e",
      "name": "db_monitoring",
      "id": "cd658793-e763-4e5a-9960-b19fe935f0db",
      "href": "http://127.0.0.1:8888/api/v2/lcm/clusters/cd658793-e763-4e5a-9960-b19fe935f0db",
	  ...
    }
  ]
}

Replace the opsc-cluster-id value with the unique_cluster_id.

curl -X PUT http://127.0.0.1:8888/api/v2/lcm/clusters/cd658793-e763-4e5a-9960-b19fe935f0db -H "Content-Type: application/json" -d '{"opsc-cluster-id": "6885d453-a816-4c9e-b06f-7050d28c50b5"}'

The results show that the opsc-cluster-id matches the unique_cluster_id.

{
  "opsc-cluster-id": "6885d453-a816-4c9e-b06f-7050d28c50b5",
  "name": "db_monitoring",
  "id": "cd658793-e763-4e5a-9960-b19fe935f0db",
  "href": "http://127.0.0.1:8888/api/v2/lcm/clusters/cd658793-e763-4e5a-9960-b19fe935f0db",
  ...
}