Clean up Astra DB Sideloader migrations

Astra DB Sideloader has an automatic cleanup process. You can also manually start or reschedule a cleanup.

The cleanup process deletes all SSTable snapshots from the migration directory, revokes any unexpired upload credentials, and then closes the migration.

Each migration ID has its own cleanup schedule, and the cleanup process deletes only the files and credentials associated with the specific migration ID that is being cleaned up. Cleaning up one migration doesn’t affect other migrations associated with the same database.

Idle timeout and automatic cleanup

A migration becomes idle if it is not in Initializing or ImportInProgress status. If a migration remains continuously idle for one week, it hits the idle timeout and triggers the automatic migration cleanup process.

A migration’s idle timer starts when you initialize the migration, and it automatically restarts when you import data.

The idle time doesn’t restart when you upload snapshots or take any other action besides importing data.

You can override the idle timer by manually starting or scheduling a cleanup. However, you cannot permanently prevent the cleanup process.

DataStax recommends that you reschedule the cleanup if your migration could be idle for several days. This includes time spent completely idle, as well as time required to upload snapshots or import data to the target database.

For mutli-terabyte and cross-region migrations, it can take several days to upload snapshots or import data. Make sure you reschedule the cleanup to avoid automatic cleanup.

Manually start a cleanup

Use the DevOps API to immediately start the cleanup process for a migration:
```
curl -X POST \
    -H "Authorization: Bearer ${token}" \
    https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/cleanup \
    | jq .
```
The cleanup process never runs on migrations in ImportInProgress status. If the request fails due to ImportInProgress, you must either wait for the import process to end, abort the migration, or reschedule the cleanup process for a later time.
Wait a few minutes, and then check the migration status:
```
curl -X GET \
    -H "Authorization: Bearer ${token}" \
    https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID} \
    | jq .
```
A successful response contains a MigrationStatus object. It can take a few minutes for the DevOps API to reflect status changes during a migration. Immediately calling this endpoint after starting a new phase of the migration might not return the actual current status.

While the cleanup is running, the migration status is CleaningUpFiles. When complete, the migration status is Closed.

Reschedule a cleanup

If you reschedule a cleanup, the cleanup timer doesn’t reset when you import data.

Keep a record of your rescheduled cleanups so you can reschedule them again, if necessary.

For example, you might need to reschedule a cleanup if your migration needs more time. Alternatively, if your migration is complete, you might reschedule the cleanup to minimize storage costs for the migration bucket.

You can use the DevOps API to schedule a migration cleanup for a specific date and time:

curl -X POST \
    -H "Authorization: Bearer ${token}" \
    https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/cleanup \
    ?option.cleanupTime=CLEANUP_TIME \
    | jq .

Replace the following:

Set your dbID and migrationID environment variables according to the migration that you want to reschedule.

This endpoint overrides the idle timeout for the specified migration ID only.
Replace CLEANUP_TIME with the date and time that you want the cleanup process to run. You must use ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ), such as option.cleanupTime=2025-03-31T14:30Z.

Setting a cleanup time in the past immediately starts the cleanup process.

At your scheduled time, the cleanup process runs on the specified migration ID. The cleanup process never runs on migrations in ImportInProgress status. If the migration is in ImportInProgress at the scheduled cleanup time, the cleanup process will start when the migration’s status changes.

Clean up Astra DB Sideloader migrations

Idle timeout and automatic cleanup

Manually start a cleanup

Reschedule a cleanup

Was this helpful?

Give Feedback