Configure automatic OpsCenter failover from the primary OpsCenter instance to the
designated backup OpsCenter instance.
Follow these steps to enable automatic failover from the primary OpsCenter instance
to the designated backup OpsCenter instance. Enabling failover requires minimal
initial set up on the backup OpsCenter.
Important: When configuring automatic failover, both the primary and
secondary OpsCenter instances must be running the same OpsCenter version. The
OpsCenter daemon (opscenterd) and the DataStax Agents must be running the same
version before configuring high availability. DataStax cannot guarantee results if
the primary and secondary OpsCenter instances are running different versions.
To enable automatic OpsCenter failover:
Prerequisites
Warning: Ensure that
is not being managed by third-party
Configuration Management. During failover, OpsCenter automatically changes
stomp_interface
in address.yaml to point
to the backup OpsCenter instance. If a separate Configuration Management system is
managing address.yaml, that change might be undone when the
Configuration Management system pushes its next update.
Procedure
- Optional:
Set up a hostname/IP that can switch between primary and backup OpsCenter
instances to avoid changing the browser URL for OpsCenter if a failover occurs.
If you do not set up a hostname or IP for seamless URL switching post-failover,
inform your OpsCenter users of any alternate URL to access OpsCenter.
-
Mirror the configuration directories stored on the primary OpsCenter instance
to the backup OpsCenter instance using the method you prefer, such as NFS mount
or rysnc. See Backing up critical configuration data for instructions on
backing up the active OpsCenter instance to the standby instance.
-
Create and run an automated script to keep the mirrored directories in sync.
The following example cron scripts run rsync to synchronize the configuration
directories every 5 minutes.
Package installation:
*/5 * * * * /usr/bin/rsync -az /etc/opscenter <user>@<backup_host>:/etc/opscenter
*/5 * * * * /usr/bin/rsync -az /var/lib/opscenter/ssl <user>@<backup_host>:/var/lib/opscenter/ssl
Tarball installation:
*/5 * * * * /usr/bin/rsync -az install_location/conf <user>@<backup_host>:install_location/conf
*/5 * * * * /usr/bin/rsync -az install_location/ssl <user>@<backup_host>:install_location/ssl
Warning: When a failover occurs, you must manually stop the sync
scripts on the former primary OpsCenter instance and start the sync scripts
on the new primary OpsCenter instance. Failure to do so will result in
configuration changes on the new primary OpsCenter instance being
overwritten by stale files from the former primary OpsCenter
instance.
- Optional:
If you want to override the default values, edit the [failover] section of the OpsCenter
configuration file opscenterd.conf.
OpsCenter daemon failover default configuration parameters
Option |
Description |
Default |
heartbeat_period |
Frequency (in seconds) with which the primary OpsCenter
instance sends a heartbeat to the backup OpsCenter
instance. |
10 |
heartbeat_reply_period |
Frequency (in seconds) with which the OpsCenter backup
instance sends a heartbeat to the primary OpsCenter
instance. |
300 |
heartbeat_fail_window |
Amount of time (in seconds) that must elapse before the
lack of a heartbeat triggers a failover. |
60 |
failover_configuration_directory |
Directory location where failover-specific configuration
is stored. The failover_id file is also
located in the failover directory.
|
- /var/lib/opscenter/failover/
(package installs)
- install_location/failover/ (tarball
installs)
|
override_primary_redirect_url |
If set, overrides the primary OpsCenter URL automatically
generated during failover. Example:
https://opscenter:80 . This option
provides more flexibility for setting a port in the redirect
URL. |
Empty. Allows OpsCenter to generate the redirect URL for
the newly primary OpsCenter instance switched from backup
OpsCenter instance status. |
-
On the backup OpsCenter instance, in the failover
directory, create a
primary_opscenter_location
configuration
file that indicates the IP address of the primary OpsCenter daemon to
monitor:
/var/lib/opscenter/failover/primary_opscenter_location
(package installs)
install_location/failover/primary_opscenter_location
(tarball installs)
The
primary_opscenter_location
file should only contain
the IP address of the primary OpsCenter instance and nothing
more:
$ cat primary_opscenter_location
55.100.200.300
Ensure the user running OpsCenter has at least read permission for the
primary_opscenter_location
file. Before the backup
OpsCenter instance can take over as the primary OpsCenter instance, the backup
OpsCenter instance deletes the primary_opscenter_location
file
in the event of a failover. After a failover, recreate the
primary_opscenter_location
file on the newly designated
backup OpsCenter instance.