Storing collection data on a separate cluster

Configure a separate storage cluster for metrics. Store collection data on a separate DataStax Enterprise cluster as an alternative to OpsCenter storing data in an OpsCenter keyspace on the same DataStax Enterprise cluster being monitored.

Configure a separate storage cluster for metrics. Store collection data on a separate DataStax Enterprise cluster as an alternative to the default of OpsCenter storing data in an OpsCenter keyspace on the same DataStax Enterprise cluster being monitored.

OpsCenter metrics can consume excessive disk space. Configuring a separate storage cluster for metrics and controlling data collection can prevent issues with performance or collecting metrics.

Important: In production environments, DataStax strongly recommends storing data in a separate DataStax Enterprise cluster.

There are two different cluster connections in the agent. When monitored and storage clusters are separate, the agent monitors the health of the monitored cluster on one of those connections, and writes metrics to the storage cluster on the other connection. If there is not a separate storage cluster, both connection pools point to the same DataStax Enterprise cluster instance. OpsCenter supports connecting to a DataStax Enterprise storage cluster when TLS/SSL is enabled. See Configuring SSL/TLS for DSE using LCM.

Note: In versions earlier than OpsCenter 5.2.2, the authentication mechanism and credentials must be identical between the storage cluster and the monitored clusters. As of OpsCenter 5.2.2 and later, OpsCenter supports distinct SSL or authentication connection settings between a monitored cluster and its corresponding storage cluster.

address.yaml

The location of the address.yaml file depends on the type of installation:

Package installations: /var/lib/datastax-agent/conf/address.yaml
Tarball installations: install_location/conf/address.yaml

cluster_name.conf

The location of the cluster_name.conf file depends on the type of installation:

Package installations: /etc/opscenter/clusters/cluster_name.conf
Tarball installations: install_location/conf/clusters/cluster_name.conf

Prerequisites

The seed nodes must be accessible without Kerberos security.
A unique keyspace must be used for each DataStax Enterprise cluster monitored by OpsCenter. If you are storing data for multiple clusters, DataStax recommends adding the cluster name as a suffix to the default keyspace name of OpsCenter. For example, set the keyspace name to OpsCenter_Cluster1 for the storage cluster to differentiate it from the OpsCenter keyspace for the monitored cluster.

Procedure

Open the storage cluster configuration file cluster_name.conf for editing. The cluster_name.conf represents the named configuration file for a particular cluster. Replace cluster_name with your actual cluster name.
Add a [storage_cassandra] section with the applicable storage configuration options for your environment.
The following example configuration assumes similar authentication between the monitored and storage cluster:
```
[storage_cassandra]
username = cassandra
password = password
seed_hosts = host1, host2
api_port = 9160
cql_port = 9042
keyspace = OpsCenter_Cluster1
```
Tip: You can set most of the agent configuration options in the [agent_config] section of the cluster_name.conf file. The options in the [agent_config] section must match the corresponding configuration option name in address.yaml. Setting agent options through the cluster configuration file sets the corresponding property in address.yaml on every node. Some properties or some cases might require setting these properties directly in address.yaml on applicable agents. Setting agent configuration options that require a list entry (value1,value2,value3 and so forth) is not supported in cluster_name.conf.

Available cluster configuration options for monitored and storage clusters:

[storage_cassandra] seed_hosts

Configure when using a separate OpsCenter storage cluster. For OpsCenter storage clusters, the agents only communicate with the listed seed hosts, consequently ensuring that a sufficient number of storage cluster nodes are placed in the comma-delimited list and that the list is reviewed as nodes are added and removed from the storage cluster.

[storage_cassandra] api_port

Configure when using a different cluster for OpsCenter storage. The Thrift remote procedure call port configured for your cluster. Same as the rpc_port property in the cassandra.yaml configuration file. Default is 9160.

[storage_cassandra] cql_port

Configure when using a different cluster for OpsCenter storage. The CQL port configured for your cluster, the default port is 9042.

[storage_cassandra] local_dc_pref

To reliably determine cluster information, OpsCenter can require a minimum of 2 nodes to connect to. If you specified a single value for seed_hosts, OpsCenter selects a second node in the cluster to fulfill this requirement. You can specify local_dc_pref as a datacenter name to constrain OpsCenter to use that datacenter to pick the second node.

[storage_cassandra] used_hosts_per_remote_dc

Configure when using a different cluster for OpsCenter storage. If using local_dc_pref, this option specified how many remote dc connections may be used as a fallback, the default value is 1.

[storage_cassandra] connect_timeout

Configure when using a different cluster for OpsCenter storage. Sets the timeout, in seconds, of a thrift connection from OpsCenter to Cassandra. The default value is 6.0.

[storage_cassandra] bind_interface

Configure when using a different cluster for OpsCenter storage. The interface used for thrift connections.

[storage_cassandra] connection_pool_size

Configure when using a different cluster for OpsCenter storage. The number of connections to thrift to build for the connection pool. The default value is 5.

[storage_cassandra] username

Configure when using a different cluster for OpsCenter storage. The username used to connect to Cassandra if authentication is enabled.

[storage_cassandra] password

Configure when using a different cluster for OpsCenter storage. The password used to connect to Cassandra if authentication is enabled.

[storage_cassandra] send_rpc

Configure when using a different cluster for OpsCenter storage. Specifies whether to send the Cassandra RPC IP to DataStax Agents. The default value is True.

[storage_cassandra] keyspace

The name of the keyspace used for OpsCenter data. The keyspace name must be unique for each managed cluster.

[storage_cassandra] ssl_keystore

The SSL keystore location for OpsCenter to use to connect to Cassandra directly.

[storage_cassandra] ssl_keystore_password

The SSL keystore password for OpsCenter to use to connect to Cassandra directly.

[storage_cassandra] ssl_truststore

The SSL truststore location for OpsCenter to use to connect to Cassandra directly

[storage_cassandra] ssl_truststore_password

The SSL truststore password for OpsCenter to use to connect to Cassandra directly
If your storage and monitored clusters require different authentication or ssl settings, open cluster_name.conf for editing and add the settings in an [agents], [storage_cassandra], and [cassandra] (monitored) sections as applicable for your environment:
```
[agents]
ssl_keystore = /etc/dse/keystore
ssl_keystore_password = example
storage_ssl_keystore = /etc/dse/keystore_storage
storage_ssl_keystore_password = example2

[storage_cassandra]
username = cassandra
password = password
seed_hosts = host1, host2
api_port = 9160
cql_port = 9042
keyspace = OpsCenter_Cluster1

[cassandra] #monitored 
username = baz
password = test
```
Note: If only the cassandra_user settings were configured for a storage cluster prior to 5.2.2, with no differentiation in authentication, opscenterd sets both the cassandra_user for storage and the monitored_cassandra_user settings on your behalf when upgrading to 5.2.2.

Available cluster configuration options for monitored and storage clusters:

[agents] ssl_keystore

The SSL keystore location for DataStax Agents to use to connect to CQL on the monitored cluster.

[agents] ssl_keystore_password

The SSL keystore password for DataStax Agents to use to connect to CQL on the monitored cluster.

[agents] storage_ssl_keystore

The SSL keystore location for DataStax Agents to use to connect to CQL on the storage cluster.

[agents] storage_ssl_keystore_password

The SSL keystore password for DataStax Agents to use to connect to CQL on the storage cluster.

[cassandra] seed_hosts

A Cassandra seed node is used to determine the ring topology and obtain gossip information about the nodes in the cluster. This should be the same comma-delimited list of seed nodes as the one configured for your DataStax Enterprise cluster by the seeds property in the cassandra.yaml configuration file. The default value is localhost.

[cassandra] rolling_restart_retry_delay

The number of seconds to wait between retry attempts when connecting to Cassandra after restarting a node. Default: 5.

[cassandra] rolling_restart_retry_attempts

The maximum number of connection retry attempts after restarting a Cassandra node. Default: 25.

[cassandra] rolling_restart_error_threshold

A rolling restart will be cancelled if the number of errors during the restart reaches this number. This helps prevent having too many nodes down in your cluster if something catastrophic happens during a rolling restart. Default: 1

[cassandra] restart_delay

During a rolling restart, the time in seconds OpsCenter waits after sending the command to stop Cassandra before sending the command to start it again. The default is 30 seconds.

Note: For a complete list of cluster_name.conf options, see Cassandra connection properties. For a complete list of address.yaml options, see DataStax Agent configuration.
If your environment requires setting options in address.yaml for reasons such as integration with third-party CMS applications, set the storage configuration options (sans the monitored_ prefix) and corresponding monitored_ configuration options as appropriate for your environment. If you set these agent options in address.yaml, you must do so for every node, whereas setting the options in cluster_name.conf automatically propagates the settings to all agents.

Available configuration options for storage and monitored clusters in address.yaml:

cassandra_port

Port used to connect to the storage cassandra node. The native transport port. Example: cassandra_port: 9042

thrift_port

Port used to connect to storage thrift server. The default setting is 9160. This information will be sent by opscenterd for convenience, but can be configured locally as needed. Example: thrift_port: 9160

cassandra_user

The Username used to connect to storage cassandra when authentication is enabled. Example: cassandra_user: cassandra

cassandra_pass

The password used to connect to storage cassandra when authentication is enabled. Example: cassandra_pass: cassandra [This field may be encrypted for additional security.]

ssl_keystore

The SSL keystore location for the storage cluster that agents use to connect to CQL. Example: ssl_keystore: /etc/dse/conf/.keystore

ssl_keystore_password

The SSL keystore password for the storage cluster that agents use to connect to CQL. Example: ssl_keystore_password: keystore-pass [This field may be encrypted for additional security.]

ssl_truststore

The SSL truststore location for the storage cluster that agents use to connect to CQL. Example: ssl_truststore: /etc/dse/conf/.truststore

ssl_truststore_password

The SSL truststore password for the storage cluster that agents use to connect to CQL. Example: ssl_truststore_password: truststore-pass [This field may be encrypted for additional security.]

monitored_cassandra_port

Port used to connect to the monitored cassandra node. The native transport port. Example: monitored_cassandra_port: 9042

monitored_thrift_port

Port used to connect to monitored thrift server. The default setting is 9160. This information will be sent by opscenterd for convenience, but can be configured locally as needed. Example: monitored_thrift_port: 9160

monitored_cassandra_user

The Username used to connect to monitored cassandra when authentication is enabled. Example: monitored_cassandra_user: cassandra

monitored_cassandra_pass

The password used to connect to monitored cassandra when authentication is enabled. Example: monitored_cassandra_pass: cassandra-pass [This field may be encrypted for additional security.]

monitored_ssl_keystore

The SSL keystore location for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_keystore: /etc/dse/conf/.keystore

monitored_ssl_keystore_password

The SSL keystore password for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_keystore_password: keystore-pass [This field may be encrypted for additional security.]

monitored_ssl_truststore

The SSL truststore location for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_truststore: /etc/dse/conf/.truststore

monitored_ssl_truststore_password

The SSL truststore password for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_truststore_password: truststore-pass [This field may be encrypted for additional security.]
Restart OpsCenter for the changes to take effect.

After the storage cluster is online, decide whether to keep the metrics from the OpsCenter keyspace on the original cluster.

Choice	Steps
If you need to keep the OpsCenter keyspace metrics and the OpsCenter storage cluster is monitored by OpsCenter	Back up the OpsCenter keyspace. Clone it to the OpsCenter storage cluster.
If you need to keep the metrics and the OpsCenter storage cluster is not monitored by OpsCenter	Back up the OpsCenter keyspace. Take a snapshot of the OpsCenter keyspace. Load the snapshot into the OpsCenter storage cluster with sstableloader.
If you don't need the metrics and use Lifecycle Manager (LCM)	Remove the duplicate storage cluster that appears in LCM. Note: When you configure a separate storage cluster for metrics, the OpsCenter keyspace is recreated on the storage cluster. Because the OpsCenter UUID changes, you will see duplicate clusters in LCM. To remove the duplicate cluster in Lifecycle Manager (LCM), resync the OpsCenter cluster UUID.
If you don't need metrics and don't use LCM	Delete the OpsCenter keyspace on the original cluster.