Storing collection data on a separate cluster

Configure a separate storage cluster for monitored clusters. Store collection data on a separate DataStax Enterprise cluster as an alternative to OpsCenter storing data in an OpsCenter keyspace on the same DataStax Enterprise cluster being monitored.

cluster_name.conf

The location of the cluster_name.conf file depends on the type of installation:

Package installations: /etc/opscenter/clusters/cluster_name.conf
Tarball installations: install_location/conf/clusters/cluster_name.conf

address.yaml

The location of the address.yaml file depends on the type of installation:

Package installations: /var/lib/datastax-agent/conf/address.yaml
Tarball installations: install_location/conf/address.yaml

If you do not want OpsCenter to store data in an OpsCenter keyspace on the DataStax Enterprise cluster being monitored, you can store the data on a separate DataStax Enterprise cluster. OpsCenter supports connecting to a DataStax Enterprise storage cluster when SSL is enabled.

Important: In production environments, DataStax strongly recommends storing data in a separate DataStax Enterprise cluster.

There are two different cluster connections in the agent. When monitored and storage clusters are separate, the agent monitors the health of the monitored cluster on one of those connections, and writes metrics to the storage cluster on the other connection. If there is not a separate storage cluster, both connection pools point to the same DataStax Enterprise cluster instance.

Note: In versions earlier than OpsCenter 5.2.2, the authentication mechanism and credentials must be identical between the storage cluster and the monitored clusters. As of OpsCenter 5.2.2 and later, OpsCenter supports distinct SSL or authentication connection settings between a monitored cluster and its corresponding storage cluster.

Version availability for configuration options:

The SSL configuration options are only applicable to OpsCenter versions 5.1.1 and later.
The monitored_* and storage_ssl_ keystore* settings are only applicable to OpsCenter versions 5.2.2 and later.
The rolling_restart_retry_* settings for both [cassandra] (monitored) and [storage_cassandra] sections are only applicable to OpsCenter versions 6.0.8 and later.

Prerequisites

The seed nodes must be accessible without Kerberos security.
A unique keyspace must be used for each DataStax Enterprise cluster monitored by OpsCenter. If you are storing data for multiple clusters, DataStax recommends adding the cluster name as a suffix to the default keyspace name of OpsCenter. For example, set the keyspace name to OpsCenter_Cluster1.
If you are using SSL to access the storage cluster and have a CER-encoded certificate, use the following command to convert it:
```
$ openssl x509 -inform der -in certificate.cer -out certificate.pem
```

Procedure

Open the storage cluster configuration file cluster_name.conf for editing. The cluster_name.conf represents the named configuration file for a particular cluster. Replace cluster_name with your actual cluster name.
Add a [storage_cassandra] section with the applicable storage configuration options for your environment.
The following example configuration assumes similar authentication between the monitored and storage cluster:
```
[storage_cassandra]
username = opsusr
password = opscenter
seed_hosts = host1, host2
api_port = 9160
cql_port = 9042
keyspace = OpsCenter_Cluster1
```
Tip: You can set most of the agent configuration options in the [agent_config] section of the cluster_name.conf file. The options in the [agent_config] section must match the corresponding configuration option name in address.yaml. Setting agent options through the cluster configuration file sets the corresponding property in address.yaml on every node. Some properties or some cases might require setting these properties directly in address.yaml on applicable agents. Setting agent configuration options that require a list entry (value1,value2,value3 and so forth) is not supported in cluster_name.conf.

Available cluster configuration options for monitored and storage clusters:

[agents] ssl_keystore

The SSL keystore location for agents to use to connect to CQL on the monitored cluster.

[agents] ssl_keystore_password

The SSL keystore password for agents to use to connect to CQL on the monitored cluster.

[agents] storage_ssl_keystore

The SSL keystore location for agents to use to connect to CQL on the storage cluster.

[agents] storage_ssl_keystore_password

The SSL keystore password for agents to use to connect to CQL on the storage cluster.

[cassandra] seed_hosts

A Cassandra seed node is used to determine the ring topology and obtain gossip information about the nodes in the cluster. This should be the same comma-delimited list of seed nodes as the one configured for your DataStax Enterprise cluster by the seeds property in the cassandra.yaml configuration file. The default value is localhost.

[storage_cassandra] seed_hosts

Configure when using a different cluster for OpsCenter storage. A Cassandra seed node is used to determine the ring topology and obtain gossip information about the nodes in the cluster. This should be the same comma-delimited list of seed nodes as the one configured for your DataStax Enterprise cluster by the seeds property in the cassandra.yaml configuration file.

[storage_cassandra] api_port

Configure when using a different cluster for OpsCenter storage. The Thrift remote procedure call port configured for your cluster. Same as the rpc_port property in the cassandra.yaml configuration file. Default is 9160.

[storage_cassandra] cql_port

Configure when using a different cluster for OpsCenter storage. The CQL port configured for your cluster, the default port is 9042.

[storage_cassandra] local_dc_pref

To reliably determine cluster information, OpsCenter can require a minimum of 2 nodes to connect to. If you specified a single value for seed_hosts, OpsCenter selects a second node in the cluster to fulfill this requirement. You can specify local_dc_pref as a datacenter name to constrain OpsCenter to use that datacenter to pick the second node.

[storage_cassandra] used_hosts_per_remote_dc

Configure when using a different cluster for OpsCenter storage. If using local_dc_pref, this option specified how many remote dc connections may be used as a fallback, the default value is 1.

[storage_cassandra] connect_timeout

Configure when using a different cluster for OpsCenter storage. Sets the timeout, in seconds, of a thrift connection from OpsCenter to Cassandra. The default value is 6.0.

[storage_cassandra] bind_interface

Configure when using a different cluster for OpsCenter storage. The interface used for thrift connections.

[storage_cassandra] connection_pool_size

Configure when using a different cluster for OpsCenter storage. The number of connections to thrift to build for the connection pool. The default value is 5.

[storage_cassandra] username

Configure when using a different cluster for OpsCenter storage. The username used to connect to Cassandra if authentication is enabled.

[storage_cassandra] password

Configure when using a different cluster for OpsCenter storage. The password used to connect to Cassandra if authentication is enabled.

[storage_cassandra] send_rpc

Configure when using a different cluster for OpsCenter storage. Specifies whether to send the Cassandra RPC IP to agents. The default value is True.

[storage_cassandra] keyspace

The name of the keyspace used for OpsCenter data. The keyspace name must be unique for each managed cluster.

[storage_cassandra] ssl_keystore

The SSL keystore location for OpsCenter to use to connect to Cassandra directly.

[storage_cassandra] ssl_keystore_password

The SSL keystore password for OpsCenter to use to connect to Cassandra directly.

[storage_cassandra] ssl_truststore

The SSL truststore location for OpsCenter to use to connect to Cassandra directly

[storage_cassandra] ssl_truststore_password

The SSL truststore password for OpsCenter to use to connect to Cassandra directly

[cassandra] rolling_restart_retry_delay

The number of seconds to wait between retry attempts when connecting to Cassandra after restarting a node. Default: 5.

[cassandra] rolling_restart_retry_attempts

The maximum number of connection retry attempts after restarting a Cassandra node. Default: 25.

[cassandra] rolling_restart_error_threshold

A rolling restart will be cancelled if the number of errors during the restart reaches this number. This helps prevent having too many nodes down in your cluster if something catastrophic happens during a rolling restart. Default: 1

[cassandra] restart_delay

During a rolling restart, the time in seconds OpsCenter waits after sending the command to stop Cassandra before sending the command to start it again. The default is 30 seconds.
If your storage and monitored clusters require different authentication or ssl settings, open cluster_name.conf for editing and add the settings in an [agents], [storage_cassandra], and [cassandra] (monitored) sections as applicable for your environment:
```
[agents]
ssl_keystore = /etc/dse/keystore
ssl_keystore_password = example
storage_ssl_keystore = /etc/dse/keystore_storage
storage_ssl_keystore_password = example2

[storage_cassandra]
username = foo
password = bar

[cassandra] #monitored 
username = baz
password = test
```
Note: If only the cassandra_user settings were configured for a storage cluster prior to 5.2.2, with no differentiation in authentication, opscenterd sets both the cassandra_user for storage and the monitored_cassandra_user settings on your behalf when upgrading to 5.2.2.
If your environment requires setting options in address.yaml for reasons such as integration with third-party CMS applications, set the storage configuration options (sans the monitored_ prefix) and corresponding monitored_ configuration options as appropriate for your environment. If you set these agent options in address.yaml, you must do so for every node, whereas setting the options in cluster_name.conf automatically propagates the settings to all agents.

Available configuration options for storage and monitored clusters in address.yaml:

cassandra_port

Port used to connect to the storage cassandra node. The native transport port. Example: cassandra_port: 9042

thrift_port

Port used to connect to storage thrift server. The default setting is 9160. This information will be sent by opscenterd for convenience, but can be configured locally as needed. Example: thrift_port: 9160

cassandra_user

The Username used to connect to storage cassandra when authentication is enabled. Example: cassandra_user: cassandra

cassandra_pass

The password used to connect to storage cassandra when authentication is enabled. Example: cassandra_pass: cassandra [This field may be encrypted for additional security.]

ssl_keystore

The SSL keystore location for the storage cluster that agents use to connect to CQL. Example: ssl_keystore: /etc/dse/conf/.keystore

ssl_keystore_password

The SSL keystore password for the storage cluster that agents use to connect to CQL. Example: ssl_keystore_password: keystore-pass [This field may be encrypted for additional security.]

monitored_cassandra_port

Port used to connect to the monitored cassandra node. The native transport port. Example: monitored_cassandra_port: 9042

monitored_thrift_port

Port used to connect to monitored thrift server. The default setting is 9160. This information will be sent by opscenterd for convenience, but can be configured locally as needed. Example: monitored_thrift_port: 9160

monitored_cassandra_user

The Username used to connect to monitored cassandra when authentication is enabled. Example: monitored_cassandra_user: cassandra

monitored_cassandra_pass

The password used to connect to monitored cassandra when authentication is enabled. Example: monitored_cassandra_pass: cassandra-pass [This field may be encrypted for additional security.]

monitored_ssl_keystore

The SSL keystore location for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_keystore: /etc/dse/conf/.keystore

monitored_ssl_keystore_password

The SSL keystore password for the monitored cluster that agents use to connect to CQL. Example: monitored_ssl_keystore_password: keystore-pass [This field may be encrypted for additional security.]
Restart OpsCenter for the changes to take effect.