Decommission a datacenter

This procedure shows how to safely decommission a datacenter to prevent data loss and authentication outages.

Decommission an HCD datacenter

Before you begin, adjust the replication of all user-defined keyspaces to exclude the datacenter you are removing, and update the replication settings for the system_auth keyspace as part of the decommissioning workflow. Run a full repair on the system_auth keyspace in the remaining datacenter.

HCD requires you to explicitly include every datacenter with active nodes in the replication settings for system_auth. Simply dropping a datacenter from the replication map before removing the nodes results in a ConfigurationException as long as any live nodes remain in that datacenter. This validation prevents authentication outages in datacenters that are actively serving clients.

Remove datacenter dc2 and keep dc1

In this example, you decommission dc2 and keep dc1.

Run a full repair on the system_auth keyspace in the datacenter that you are keeping (dc1):
```
nodetool repair -full system_auth
```
This ensures that authentication data is consistent in dc1 before you remove any nodes from dc2.
Set the replication factor of the datacenter that you are removing (dc2) to 1 instead of 0.

You cannot set the replication factor to 0 while nodes are still present in that datacenter. Reduce the replication factor to 1 first to keep the datacenter in the replication map.
```
ALTER KEYSPACE system_auth
WITH REPLICATION = {
    'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy',
    'dc1': '3',
    'dc2': '1'
};
```
Decommission all nodes in dc2 except for the last node.

Run nodetool decommission on each node in dc2, one node at a time. Wait for each decommissioning process to complete before you proceed to the next node.
```
nodetool decommission
```
For the final remaining node in dc2, run the decommission command with the --force option.

Because the final node cannot stream data to other nodes in the same datacenter, you must force it to decommission. Run this command from the last node in dc2:
```
nodetool decommission --force
```
Using the --force option skips data streaming. Verify that the full repair completed successfully before you force-decommission the last node.

Remove dc2 from the replication map:

ALTER KEYSPACE system_auth
WITH REPLICATION = {
    'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy',
    'dc1': '3'
};

This command succeeds because no active nodes remain in dc2.

From any remaining node in dc1, confirm that no endpoints from dc2 remain in the gossip or ring output:
```
nodetool gossipinfo
```

nodetool ring

Decommission a datacenter

Decommission an HCD datacenter

Remove datacenter dc2 and keep dc1

Was this helpful?

Give Feedback