Configuring commit log backups

Configure commit log backups to facilitate point-in-time restores.

Commit log backups facilitate restoring backup data to a particular point-in-time.

If commit log archiving is enabled on a cluster, OpsCenter does not automatically enable commit log archiving on the new nodes when adding new nodes to a cluster using Lifecycle Manager. The Commit Log Archiving Setting Enabled Consistency Rule in the Best Practice Service checks to make sure all nodes within a cluster have Commit Log Archiving enabled.

Important: Enabling commit log archiving modifies the Apache Cassandra™ commitlog_archiving.properties configuration file. The archive_commitlog.sh file is created when commit log archiving is enabled. If you add nodes to a cluster and commit log archiving is enabled, you must manually copy commitlog_archiving.properties and archive_commitlog.sh to the new nodes prior to starting DataStax Enterprise, or re-enable commit log archiving.

Prior to being routed to the backup storage directory configured with the backup_storage_dir option in address.yaml (default: /var/lib/datastax-agent/backups/), the commit logs temporarily reside in the backup staging directory for commit logs. After the commit logs have been sent to any other configured locations, the commit logs are moved from the backup staging to the backup storage directory.

address.yaml

The location of the address.yaml file depends on the type of installation:
  • Package installations: /var/lib/datastax-agent/conf/address.yaml
  • Tarball installations: install_location/conf/address.yaml

Prerequisites

  • Java 8 is required when backing up commit logs.

Procedure

  1. Click cluster name > Services.
  2. Click the Details link for the Backup Service.
  3. Click the Settings tab.

    Configure optional Backup Service features in the Settings tab

  4. Click the Configure link for the Commit log backup feature.

    The Configure commit log backup dialog appears.

    Configure commit log backup dialog for Point-in-Time restores

  5. Set the slider to On.
  6. If you want to back up the commit logs to a local filesystem location or to a cloud storage provider such as Amazon S3, click Add Location. For more information about Location and Retention Policy, see adding a backup location.
    Tip: Adding another backup location besides On Server is a recommended best practice.
  7. Required: Enter the path in the Backup Staging Directory for staging the commit logs prior to storage on each node.

    The Backup Staging Directory is configured with the backup_staging_dir option in address.yaml (default: /var/lib/datastax-agent/commitlogs/).

    The location set under Backup Staging Directory must be writable both by the user running DataStax Enterprise and the DataStax agent user. The Backup Staging Directory is not to be confused with the Commit log locations, which store backups of the commit logs. The DataStax agent user and the DataStax Enterprise user are by default the same user. If that is not the case for your environment, see Setting permissions to run the DataStax Agent as the DSE user.

    Important: If the location runs out of disk space, the backup fails. Monitor the free disk space using available alerts. Configure the free disk space threshold to prevent backups from starting if there is not enough free space available. Commit log files record every mutation of the data in a cluster, and can grow quite large over time (see Logging writes and memtable storage). The disk performance of the commit log backup location is extremely important because the disk write speed can likely be a bottleneck for write-heavy use cases; and read performance a bottleneck for restore operations.
  8. Enter the path to the Backup Storage Directory where commit logs are stored and retained on each node.

    Retention is based on the Retention Policy set for the specified On Server location, which corresponds to the value set by the backup_storage_dir in the agent address.yaml.

  9. Click Save.
  10. After enabling (or disabling) commit log backups, OpsCenter prompts you to confirm a rolling restart of the cluster.

    The Status column displays On for the Status of the Commit Log Backup feature.

  11. Click View Archive Times at any time to view the date and time for the archived commit logs.

    The Commit log backups dialog displays the Last Archived date and time for each node. When restoring a backup to a specific point in time, all nodes in the cluster must have a commit log backup for the desired time.