Configuring for very large clusters
About this task
OpsCenter can manage very large clusters.
|
Lifecycle Manager can provision and manage nodes per cluster within its UI. See Supported capabilities for more details. |
When working with very large clusters, the performance of OpsCenter decreases with the default settings. To improve performance, adjust the cluster settings to increase the time period between polls of a cluster’s nodes and token lists.
After adding a very large cluster to OpsCenter, change the following default settings:
Procedure
-
Locate the
cluster_name.conffile. The location of this file depends on the type of installation:-
Package installations: /etc/opscenter/clusters/cluster_name.conf
-
Tarball installations: install_location/conf/clusters/cluster_name.conf
-
-
Open
cluster_name.conffor editing.-
Increase the node list poll period to 30 minutes by setting the
nodelist_poll_periodoption to 1800 under[collection]:-
[collection] nodelist_poll_period
The interval in seconds OpsCenter waits to poll the nodes in a cluster. The default value is 30.
[collection] nodelist_poll_period = 1800
-
-
If an agent is overloaded, increase the default
http_timeoutif necessary:-
[agents] http_timeout
The timeout, in seconds, for an HTTP call to the agent. The default value is 10.
[agents] http_timeout = 20
-
-
-
Set the following properties in the
[agent_config]section ofcluster_name.confon theopscenterdmachine and the properties propagate automatically to all DataStax Agents.Conversely, setting these options in
address.yamlmust be done for every node.Increasing the
[agent_config]values reduces the speed at which the UI displays updates of node status, such as the Ring view.-
[agent_config] realtime_interval
The length of time, in seconds, between polling attempts to capture rapidly changing
realtimeinformation. Default value: 5[agent_config] realtime_interval = 40
-
[agent_config] shorttime_interval
The length of time, in seconds, between polling attempts to capture information that changes frequently. Default value: 10
[agent_config] shorttime_interval = 80
-
[agent_config] longtime_interval
The length of time, in seconds, between polling attempts to capture information that changes infrequently. Default value: 300
[agent_config] longtime_interval = 2400
-
[agent_config] status_reporting_interval
The length of time, in seconds, between sending agent health information. Default value: 20
[agent_config] status_reporting_interval = 160
-
[agent_config] disk_usage_update_period
The length of time, in seconds, to wait between attempts to poll the disk for usage. Agents send storage capacity data to
opscenterdbased on the disk_usage_update_period value inaddress.yamlor incluster_name.conf. Default value: 30[agent_config] disk_usage_update_period = 240
-
-
Locate the
opscenterd.conffile. The location of this file depends on the type of installation:-
Package installations: /etc/opscenter/opscenterd.conf
-
Tarball installations: install_location/conf/opscenterd.conf
-
-
Open
opscenterd.conffor editing and adjust the following settings:-
[agents] not_seen_threshold
The maximum time in seconds since the last agent status about a specific connection, such as
stomp, was sent before that agent connection is considered down. This threshold also affects how long OpsCenter waits before marking node health as unknown. Default value: 180 seconds. -
[agents] http_poll_period
The frequency in seconds between attempts to poll agent http health. Default value: 60 seconds.
-
[ui] default_api_timeout
The default timeout value in seconds for an API call from the OpsCenter UI to the OpsCenter API. Default value: 30 seconds. Some API calls require a timeout longer than 30 seconds. In those cases, the API call timeouts are scaled relative to the
default_api_timeout(for example, 6 *default_api_timeout). Changing thedefault_api_timeoutaffects those timeouts accordingly. For example:[agents] not_seen_threshold = 620 http_poll_period = 500 [ui] default_api_timeout = 60
-
-
Locate the
address.yamlfile. The location of this file depends on the type of installation:-
Package installations: /var/lib/datastax-agent/conf/address.yaml
-
Tarball installations: install_location/conf/address.yaml
-
-
Open
address.yamlfor editing, and adjust the following setting to override the default of 30 seconds:-
stomp-setup-timeout
The maximum time in seconds before the agents try to reconnect via stomp. Increasing this timeout to 120 seconds better allows agents to reconnect after restarting OpsCenter in a large cluster.
stomp-setup-timeout: 120
-
-
Use the environmental variable
OPSC_JVM_OPTSto override the default parameters for the OpsCenter JVM.The following command doubles the heap size to
4096m(4GB).export OPSC_JVM_OPTS=-Xmx4096mSee Configuring the OpsCenter JVM for additional information.
-
If you continually receive
OutOfMemoryerrors, consider Configuring the DataStax Agent JVM.
