Restoring a backup to a specific point-in-time

A point-in-time restore uses commit log archives to restore data from a backup to a specific date and time using OpsCenter.

For a point-in-time restore, OpsCenter intelligently chooses which snapshots and commit logs to restore from based on the date and time you are restoring the cluster to. If an acceptable combination of snapshots and commit logs cannot be found, the restore fails. A detailed error message is visible in the Activity section of the OpsCenter UI.

dse-env.sh

The default location of the dse-env.sh file depends on the type of installation:

Package installations
Installer-Services installations

 /etc/dse/dse-env.sh

Tarball installations
Installer-No Services installations

 installation_location/bin/dse-env.sh

Prerequisites

  • For point-in-time restores to work, you must have enabled commit log backups and performed at least one snapshot backup before the time to which you are restoring.
  • The Restore feature of the Backup Service leverages the sstableloader utility, which currently requires enabling the thrift server on all nodes before restoring. Before restoring, ensure the thrift server is enabled on all nodes.
    Note: The thrift server is only required for DSE versions earlier than 5.0 (DSE 4.8.x versions).
  • When performing a point-in-time restore, the cluster topology must not have changed since the backup. Attempting to perform a point-in-time restore on a cluster whose topology has changed results in a failure. DataStax strongly recommends performing a snapshot backup both before and after changing the cluster topology. After changing the topology, you can then restore the cluster based on that backup. If reverting to the previous topology, you can use the backup with the original topology to restore the cluster.
  • Known limitations:
    • Point-in-time restore cannot restore commit logs for keyspaces or tables that would have to be recreated in Cassandra 2.1 and later, and DataStax Enterprise 4.7 and later.
    • Point-in-time restore fails if any tables were recreated during the time period of the actual point-in-time restore.
Important: The Backup Service requires control over the data and structure of its destination locations. The AWS S3 bucket and the Local file system destinations must be dedicated for use only by OpsCenter. Any additional directories or files in those destinations can prevent the Backup Service from properly conducting a Backup or Restore operation.
Note: Consider the following caveats when creating and restoring backups:
  • Restoring a snapshot that contains only the system keyspace is not allowed. There must be both system and non-system keyspaces, or only non-system keyspaces in the snapshot you want to restore.
  • Restoring a snapshot that does not contain a table definition is not allowed.
  • Restoring from a backup while Kerberos is enabled is not currently supported by OpsCenter.
  • Restoring a snapshot to a location with insufficient disk space fails. The Restore Report indicates which nodes do not have sufficient space and how much space is necessary for a successful restore. For more information and tips for preventative measures, see Monitoring sufficient disk space for restoring backups.

Procedure

  1. Click cluster name > Services.
  2. Click the Details link for the Backup Service.
  3. Click Restore Backup.
    The Restore from Backup, Step 1 of 2: Select Backup dialog appears.
  4. Click the Point In Time tab.

  5. Complete your selections:
    1. Point in Time: Set the date and time to which you want to restore your data.
    2. Commit logs location: Select the location of the commit logs; either On Server, Local FS, or another location on Amazon S3. The location of commit logs is configured when enabling commit log backups.
    3. Backup location: Select the location of the snapshot; either On Server, Local FS, or Amazon S3. Click the +Add Location link to add another location.
    4. Click Next.
      The Restore from Backup, Step 2 of 2: Configure and Restore dialog appears.


  6. Complete your selections:
    1. Keyspaces: Select the tables included in the backup you want to restore. Click the keyspace name to include all the tables in the keyspace. Click All Keyspaces to restore all the keyspaces.

      To select only specific tables, expand the keyspace name and select the tables.

      Note: Consider the following caveats when creating and restoring backups:
      • Restoring a snapshot that contains only the system keyspace is not allowed. There must be both system and non-system keyspaces, or only non-system keyspaces in the snapshot you want to restore.
      • Restoring a snapshot that does not contain a table definition is not allowed.
      • Restoring from a backup while Kerberos is enabled is not currently supported by OpsCenter.
      • Restoring a snapshot to a location with insufficient disk space fails. The Restore Report indicates which nodes do not have sufficient space and how much space is necessary for a successful restore. For more information and tips for preventative measures, see Monitoring sufficient disk space for restoring backups.
    2. Under Location: Select the target cluster for the restored data.
      • The Location list is only available when there are multiple clusters and both clusters are managed by the same instance of OpsCenter.
      • If you select a different cluster than the one that was backed up, the data is cloned to the selected cluster. See cloning cluster data.
      Note: Restoring encrypted tables to a different cluster does not work unless the encryption keys are identical, which is typically not the case.
    3. Optional: To remove the existing keyspace data before the data is restored, select Truncate/delete existing data before restore. This completely removes any updated data in the cluster for the keyspaces you are restoring.
    4. Optional: To prevent overloading the network, set a maximum transfer rate for the restore. Select Throttle stream throughput at ____ MB and set the maximum MB per second.
    5. Optional: Change the staging directory if necessary by setting the backup_staging_directory configuration option in address.yaml.
    6. Optional: Click the Edit Restart Settings link to adjust settings for the rolling restart.
  7. Click Restore Backup.
    The Confirm Restore dialog appears.
    Confirm restore operation dialog
    Warning: If a value was not set for throttling stream output, a warning message indicates the consequences of unthrottled restores. Take one of the following actions:
    • Click Cancel and set the throttle value in the Restore from Backup dialog.
    • Set the stream_throughput_outbound_megabits_per_sec and inter_dc_stream_throughput_outbound_megabits_per_sec values in cassandra.yaml.
    • Proceed anyway at the risk of creating network bottlenecks.
    Tip: If you are using LCM to manage DSE cluster configuration, update Cluster Communication settings in cassandra.yaml in the config profile for the cluster and run a configuration job. Stream throughput (not inter-dc) is already set to 200 in LCM defaults.

    Update throttle options in LCM

  8. Click Start Restore to confirm when prompted.

Results

OpsCenter retrieves the backup data and sends the data to the nodes in the cluster. A snapshot restore is completed first, following the same process as a normal snapshot restore. After the snapshot restore successfully completes, OpsCenter instructs all agents in parallel to download the necessary commit logs, followed by a rolling commit log replay across the cluster. Each node is configured for replay and restarted after the previous node finishes successfully.

If an error occurs during a point-in-time restore for a subset of tables, you might need to manually the revert changes made to some cluster nodes. To clean up a node, edit dse-env.sh and remove the last line that specifies JVM_OPTS. For example:

export JVM_OPTS="$JVM_OPTS -Dcassandra.replayList=Keyspace1.Standard1"