Clone Astra DB Serverless databases

The clone feature for Astra DB Serverless is available only to qualified participants in the private preview release. Development is ongoing, and the features and functionality are subject to change. This private preview is governed by your Agreement and the DataStax Preview Terms.

If you are interested in this feature, contact your account representative or IBM Support.

You can clone the schema, data, and custom guardrails (if any) from one Astra DB Serverless database to another, empty Astra DB Serverless database.

Cloning is a one-time event that populates an empty database by copying a point-in-time snapshot (backup) from another database. There is no direct connection between the databases because the clone process reads from a snapshot, not the live database.

You can clone databases through the DevOps API or the Astra Portal.

Limitations

Cloning isn’t a continuous replication or synchronization process. There is no permanent relationship between the databases. Any changes made to the original database during or after cloning aren’t reflected in the clone. Likewise, any changes made to the clone aren’t reflected in the original database.
You can clone the same database multiple times, but there is a limit of one active clone operation per database. For example, if you clone from database A to database B, then you cannot start another clone operation from database A or B until the active clone operation ends.
Cloning isn’t supported for Astra Managed Clusters databases.

Prepare databases

Before starting the cloning process, do the following:

Identify your source and target databases:
- Source database: The database that contains data you want to clone.
- Target database: The empty database that you want to clone data into.
Make sure both databases are in the same Astra organization.
Verify that the database deployment types are compatible:
- If the source database is a Serverless (non-vector) database, then the target database can be a Serverless (non-vector) or Serverless (vector) database.
- If the source database is a Serverless (vector) database, then the target database must be a Serverless (vector) database.
  
  If your use case requires that you clone data from a Serverless (vector) database to a Serverless (non-vector) database, you can use the DevOps API to attempt the clone operation, but schema errors and incompatibility issues might occur during or after cloning. To avoid these errors, DataStax recommends that you use another tool, such as DataStax Bulk Loader (DSBulk), to copy only the table data.
- If the source database has custom guardrails, DataStax recommends that you clone to the same deployment type.
Ensure that your source database is eligible for cloning:
- The source database must contain some data to clone.
- The source database must have at least one snapshot (backup) available to clone from.
  
  Active databases capture snapshots regularly, and they usually have multiple snapshots available.
  
  Hibernated databases don’t record new backups after entering hibernation. A database that is hibernated for more than 19 consecutive days has no backups due to the backup retention schedule.
  
  For more information and to check available snapshots, see Choose snapshot.
Make sure the target database is a single-region database (deployed to one region only).

Before cloning, the target database must not be a multi-region database. You can deploy secondary regions after cloning.

This requirement applies to the target database only. The source database can be a single-region or multi-region database.
If the target database is in a different cloud provider or region than the source database, be aware that clone operations involving cross-region or cross-cloud data transfer can incur additional billed charges due to the necessary data transfer operations.

The target database doesn’t need to be deployed to the same region or cloud provider as the source database. However, if possible, DataStax recommends cloning between databases in the same region and cloud provider to minimize costs and latency.
Make sure the target database is empty, except for one empty keyspace.

The easiest way to ensure that the target database is empty is to create a new database. If you want to clone to an existing database, you must delete all additional keyspaces, and then delete all collections and tables from the last keyspace.

Make sure the target database belongs to a PCU group.

This is a strict requirement, and cloning will fail if the target database isn’t associated with a PCU group.

Don’t change the target database’s PCU group association or PCU group configuration during the cloning process.

You must ensure that the PCU group has sufficient capacity to support the cloning process based on the source database’s configuration and data, including custom guardrails, keyspaces, and tables. Cloning is a resource intensive process that rapidly spikes database activity by writing a large amount of data and applying configuration changes to an empty database.

The PCU group’s minimum capacity is critical to the success of the clone operation because it is always ready and available to use. If the group’s minimum capacity is too low, then the cloning process can stall or fail while Astra scales up additional capacity. Make sure the minimum capacity is set high enough to anticipate the spike in database activity caused by the cloning process. DataStax also recommends setting the maximum capacity higher than the minimum capacity to allow additional capacity to be provisioned if needed.

If multiple databases are in the same PCU group, be aware that the cloning process can significantly reduce the capacity available to other databases in the group, causing latency spikes and timeouts. To isolate the cloning process from other databases, consider creating a flexible capacity PCU group that you can use exclusively for cloning. After cloning, you can move the database to another PCU group, and then delete or park the flexible capacity group. Alternatively, you can temporarily increase the minimum and maximum capacity to accommodate the cloning process, and then decrease the capacity after cloning is complete.

If the source database uses vectorize integrations to generate embeddings, add the target database to the embedding provider credentials' scoped databases before starting the cloning process.

This ensures that vectorize integrations will function in the target database after cloning.

Clone with the Astra Portal

When cloning databases in the Astra Portal, the source snapshot and the target database must have the same cloud provider and region.

For cross-provider and cross-cloud cloning, you must clone with the DevOps API.

In the Astra Portal, click the name of the source database that you want to clone.

You can also start clone operations from the Clones page in your organization’s settings.
Click Snapshots.
Find the snapshot that you want to clone, click More, and then select Clone.

For more information about snapshots, see Choose snapshot.
On the Clone snapshot dialog, select the target database for the clone operation.

The Astra Portal performs some checks on the target database before cloning, but it cannot verify all requirements. To avoid failed clone operations, make sure the target database meets all requirements described in Prepare databases.

The Source database, Region, and Snapshot fields are locked when cloning from a database’s Snapshots tab. To change these fields, click Cancel, and then choose a different snapshot.
Click Clone snapshot to start the cloning process.
To monitor the progress of the cloning process, see Monitor cloning.
After cloning is complete, finalize the target database configuration.

Clone with the DevOps API

Multiple API requests are required to clone databases with the DevOps API. Environment variables help manage the identifiers and credentials required for these requests.

Prepare environment

Generate an application token that has a role with the Create DB permission, such as Database Administrator, Organization Administrator, or a custom role with sufficient permissions.
Get the database IDs for your source and target databases.
Set the following environment variables using your preferred method, such as a .env or .curlrc file:
```
export APPLICATION_TOKEN="APPLICATION_TOKEN"
export SOURCE_CLONE_DATABASE_ID="SOURCE_DATABASE_ID"
export TARGET_CLONE_DATABASE_ID="TARGET_DATABASE_ID"
export SNAPSHOT_REGION="SOURCE_SNAPSHOT_REGION"
```
The SNAPSHOT_REGION variable is only necessary when cloning from a multi-region database. Specify the name of the region that you want to clone from, such as us-east-1 or eu-west-1.
Optional: Install jq to format the JSON responses from the DevOps API.

The example commands in this guide use jq. If you use a different formatter or no formatter, you must modify the last line of each command accordingly.

For more information about setting environment variables and forming DevOps API requests, see Get started with the Astra DevOps API.

Choose snapshot

The cloning process replicates the schema and data from a point-in-time snapshot (backup) of the source database.

Cloning from a snapshot avoids performance impacts and data inconsistencies that could occur from real-time reads and writes required to clone from a live database.

As explained in Prepare databases, the source database must have at least one snapshot to clone from.

To get the list of available snapshots and select a snapshot to clone, do the following:

Use GET /v2/databases/SOURCE_DATABASE_ID/snapshots to get a list of available snapshots for your source database.
Primary region snapshots

By default, the endpoint returns snapshots for the primary region. For a single-region database, this is the only region. For a multi-region database, this is the first region where the database was deployed.

curl -sS -L -X GET \ "https://api.astra.datastax.com/v2/databases/$SOURCE_CLONE_DATABASE_ID/snapshots" \ -H "Authorization: Bearer $APPLICATION_TOKEN" \ | jq --monochrome-output .

Secondary region snapshots (multi-region databases)

For multi-region databases, each region has its own snapshots. To get snapshots from a secondary region of a multi-region database, use the sourceRegion query parameter to specify the region to get snapshots from, such as us-east-1 or eu-west-1.

curl -sS -L -X GET \ "https://api.astra.datastax.com/v2/databases/$SOURCE_CLONE_DATABASE_ID/snapshots?sourceRegion=$SNAPSHOT_REGION" \ -H "Authorization: Bearer $APPLICATION_TOKEN" \ | jq --monochrome-output .

If you didn’t set a SNAPSHOT_REGION environment variable, replace $SNAPSHOT_REGION with the region name.
From the response, get the ID of the snapshot that you want to clone.

If you recently made significant schema or data changes to the source database, pay attention to the snapshot timestamp in the response. Make sure you choose a snapshot that reflects the desired state of the target database.

For example, if you deleted a table from the source database that you don’t want to clone to the target database, make sure you choose a snapshot that was recorded after the deletion. In contrast, if you clone from a pre-deletion snapshot, then the deleted table will be cloned to the target database.
Recommended: Store the snapshot ID in an environment variable:
```
export CLONE_SNAPSHOT_ID="SNAPSHOT_ID"
```

Start cloning

You must provide the snapshot ID as a query parameter with your DevOps API requests. The clone operation will fail if the snapshot ID is missing, invalid, passed in the body, or belongs to a different database than specified in SOURCE_CLONE_DATABASE_ID.

To start the cloning process, use POST /v2/databases/TARGET_DATABASE_ID/cloneFrom/SOURCE_DATABASE_ID with the snapshotID query parameter:

Clone from primary region snapshot

curl -sS -L -X POST \
  "https://api.astra.datastax.com/v2/databases/$TARGET_CLONE_DATABASE_ID/cloneFrom/$SOURCE_CLONE_DATABASE_ID?snapshotID=$CLONE_SNAPSHOT_ID" \
  -H "Authorization: Bearer $APPLICATION_TOKEN" \
  | jq --monochrome-output .

Clone from secondary region snapshot

If your snapshot is from a secondary region of a multi-region database, use the sourceRegion query parameter to specify the region that the snapshot is from, such as us-east-1 or eu-west-1.

curl -sS -L -X POST \
  "https://api.astra.datastax.com/v2/databases/$TARGET_CLONE_DATABASE_ID/cloneFrom/$SOURCE_CLONE_DATABASE_ID?snapshotID=$CLONE_SNAPSHOT_ID&sourceRegion=$SNAPSHOT_REGION" \
  -H "Authorization: Bearer $APPLICATION_TOKEN" \
  | jq --monochrome-output .

If you didn’t set a SNAPSHOT_REGION environment variable, replace $SNAPSHOT_REGION with the region name.

From the response, get the clone operation ID, and then store it in an environment variable.

This ID is shown only once. You need it to monitor the cloning process.
```
export CLONE_OPERATION_ID="CLONE_OPERATION_ID"
```
To monitor the progress of the cloning process, see Monitor cloning.
After cloning is complete, finalize the target database configuration.

Monitor cloning

Upon initiating the cloning process, the target database enters Maintenance status, and it remains unavailable until cloning is complete. Clone time varies based on factors like the size of the source database. Once cloning is complete, the target database returns to Active status.

The source database is unaffected by the cloning process because the cloned data is replicated from a snapshot, not the live database.

You can check the status of the cloning process using the DevOps API or the Astra Portal:

If you used the DevOps API to start cloning, use GET /v2/databases/TARGET_DATABASE_ID/cloneStatus/CLONE_OPERATION_ID to check cloning progress:

curl -s -L -X GET \
  "https://api.astra.datastax.com/v2/databases/$TARGET_CLONE_DATABASE_ID/cloneStatus/$CLONE_OPERATION_ID" \
  -H "Authorization: Bearer $APPLICATION_TOKEN" \
  | jq --monochrome-output .

If you used the Astra Portal to start cloning, inspect clone history in organization settings:
1. In the Astra Portal header, click Settings.
2. In the Settings navigation menu, make sure the enterprise/organization filter is set to the organization that you want to manage.
  
  If the organization belongs to an enterprise, you must filter on the enterprise, and then click the organization name in the Organizations list.
3. In the Settings navigation menu, click Clones.
4. Review the information for each clone operation:
  
  Target database: The target database name or Terminated database if the original target database was terminated.
  
  Source database: The source database name.
  
  Snapshot ID: The timestamp of the snapshot used for cloning.
  
  Region: The snapshot’s region.
  
  User: The user who started the clone operation.
  
  Created at: The timestamp for the start of the clone operation.
  
  Status: The current status of the clone operation, such as Cloned (complete) or Failed. If a clone operation fails, see Troubleshoot cloning.
There is a limit on the number of clone operations stored in the Astra Portal. Older entries are discarded when the limit is reached.

Finalize target database configuration

After the cloning process replicates the schema and data to the target database’s primary region, you can finalize the target database’s configuration as needed. For example:

Deploy secondary regions if you want the target database to be a multi-region database.
Configure security settings, such as private endpoints.
Connect to the target database using an application token or Secure Connect Bundle (SCB) for the target database.

Troubleshoot cloning

There are several common reasons why cloning operations can fail or cause issues in the target database:

Target database isn’t empty: The target database must have only one keyspace and the keyspace must be empty (no tables).

The first step of the clone process is to create the schema (keyspaces and tables) in the target database. If there are any extra keyspaces or tables present in the target database, the operation will fail.

Additionally, if a clone operation fails after schema cloning begins, then the target database can contain all or part of the source schema. If you don’t delete these keyspaces and tables from the target database, then subsequent clone attempts will fail because the target database must be empty before cloning. Before you retry a failed clone operation, make sure the target database has only one empty keyspace (no tables).
PCU group changed during cloning: Don’t change the target database’s PCU group association or PCU group configuration during the cloning process.
Source database deleted during cloning: Deleting a database also deletes all of its snapshots. If you delete the source database while a clone operation is ongoing, you also delete the snapshot that is being used for that clone operation. Then, cloning fails due to the snapshot being unavailable.

If a snapshot no longer exists, you cannot start or restart a cloning operation with that snapshot ID. You must choose a different snapshot.
Snapshot exited retention window during cloning: Snapshots are stored on a rolling backup retention schedule. If the snapshot that you’re cloning from exits the retention window during cloning, then cloning can fail due to the snapshot being unavailable. To avoid this issue, make sure to start cloning from a snapshot that was recorded recently enough to remain in the retention window for the expected duration of the cloning process.

Daily backups are most stable because they are retained for several days. In contrast, hourly backups are stored for 24 hours in a rolling fashion, aside from a single daily backup. For example, if you clone from an hourly backup that was recorded 23 hours ago, and your clone operation takes longer than 1 hour, then the snapshot could exit the retention window while cloning is ongoing.

If a snapshot no longer exists, you cannot start or restart a cloning operation with that snapshot ID. You must choose a different snapshot.
Invalid snapshot ID: Make sure the snapshot ID is correct, exists, and belongs to the database specified in SOURCE_CLONE_DATABASE_ID. Additionally, make sure the snapshot ID is passed as a query parameter, not in the request body.

If a snapshot no longer exists, you cannot start or restart a cloning operation with that snapshot ID. You must choose a different snapshot.
Vectorize integration stops working after cloning: If the source database uses vectorize integrations to generate embeddings, make sure you add the target database to the scoped databases. If the target database isn’t in the scoped databases, then the integration breaks after cloning.
Schema errors after cloning: This is most likely to occur when using the DevOps API to clone a Serverless (vector) source database to a Serverless (non-vector) target database.

If your use case requires that you clone data from a Serverless (vector) database to a Serverless (non-vector) database, and you encounter schema errors during or after cloning, consider using another tool, such as DataStax Bulk Loader (DSBulk), to copy only the table data.