Recover from a single disk failure using JBOD

In a Just a Bunch Of Disks (JBOD) array, Hyper-Converged Database (HCD) might not fail completely from the loss of one disk, but some reads and writes might fail if:

A read/write operation’s consistency level is ALL.
Data being read or written is stored on the defective disk.
Data to be compacted is on the defective disk.

Use the following steps to recover from a single disk failure in a JBOD array. These steps proceed from the simplest recovery action to more complex recovery actions if earlier attempts fail. In some cases you can replace the disk, and then restart HCD and run nodetool repair. If HCD cannot start or repair fails, this is an indication that the disk crash corrupted system tables, and you must remove the incomplete data from the other disks in the array. The procedure for doing this depends on whether the cluster uses vnodes or a single-token architecture.

Verify the disk failure and replace the disk

To verify that the node has a defective disk and to identify the defective disk, check the logs on the affected node.

Disk failures are logged in FILE NOT FOUND entries that identify the failed mount point or disk.
If the node is still running, stop HCD and shut down the node.
Replace the defective disk.
Attempt to start HCD on the node.

If HCD doesn’t start, try Restart without bootstrapping.

If HCD starts, run nodetool repair on the node. If the repair succeeds, the node is restored to production and recovery is complete. If the repair fails, try Advanced recovery.

Restart without bootstrapping

If a node cannot restart after replacing a failed disk, try restarting HCD without bootstrapping the node. To do this, use the -Dcassandra.allow_unsafe_replace=true start up option to bypass bootstrapping.

Package installations

Add the option to the jvm-server.options file, and then start HCD:

JVM_OPTS="$JVM_OPTS -Dcassandra.allow_unsafe_replace=true"

Tarball installations

sudo INSTALL_DIRECTORY/bin/hcd cassandra -Dcassandra.allow_unsafe_replace=true

If HCD doesn’t start with -Dcassandra.allow_unsafe_replace=true, replace the node.

If HCD starts, run nodetool repair on the node.

If the repair succeeds, the node is restored to production and recovery is complete. For package installations, you can remove -Dcassandra.allow_unsafe_replace=true from jvm-server.options now that the node is repaired and rejoined the cluster.

If the repair fails, try Advanced recovery. If you already attempted advanced recovery and repair still fails, replace the node.

Advanced recovery

If the previous recovery attempts failed, you must clear the system directories on the unaffected drives on the node with the failed disk.

For clusters using vnodes, manually set the initial token range so that the node can rejoin the cluster with the correct token range assignment.

After the node is recovered and repaired, you can remove the manually set initial token to reenable automatic token assignment. This isn’t necessary for single-token architectures because the initial token is already set manually in cassandra.yaml.
1. On a healthy node in the same cluster, run nodetool ring to get information about the cluster, and then retrieve the list of tokens for the node that had the failed disk.
  
  For example, use grep with the node’s IP address:
  
  nodetool ring | grep NODE_IP_ADDRESS | awk ' {print $NF ","}' | xargs
2. Reformat the list of tokens into a comma-separated list.
  
  For example, copy the output of nodetool ring into a spreadsheet file (space-delimited), delete all columns except the list of tokens, and then export or use a function like CONCAT to combine all token values into a single comma-separated list.
3. On the node that had the failed disk, edit the cassandra.yaml file, and then set the value of the initial_token property to the comma-separated list of tokens that you retrieved from nodetool ring.
4. If needed, change any other configuration properties on the recovered node to match the existing nodes.
  
  Use the diff command to compare configuration files and reconcile any differences.
On the node that had the failed disk, clear the system directory on the remaining functional drives.

For example, given a node with the following three-disk JBOD array:
```
-/mnt1/cassandra/data
-/mnt2/cassandra/data
-/mnt3/cassandra/data
```
If mnt1 failed, you would run the following commands to clear the system directories on the other two functioning drives:
```
rm -fr /mnt2/cassandra/data/system &&
rm -fr /mnt3/cassandra/data/system
```
Restart HCD without bootstrapping, and then repair the node.
For clusters using vnodes, after the node starts and repairs successfully, you can remove the manually set initial_token value from the cassandra.yaml file to reenable automatic token assignment.

Recover from a single disk failure using JBOD

Verify the disk failure and replace the disk

Restart without bootstrapping

Advanced recovery

Was this helpful?

Give Feedback