Recovering from a single disk failure using JBOD

Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).

The DataStax Distribution of Apache Cassandra™ (DDAC) might not fail from the loss of one disk in a JBOD array, but some reads and writes may fail when:

The operation's consistency level is ALL.
The data being requested or written is stored on the defective disk.
The data to be compacted is on the defective disk.

It's possible that you can simply replace the disk, restart Cassandra, and run nodetool repair. However, if the disk crash corrupted system table, you must remove the incomplete data from the other disks in the array. The procedure for doing this depends on whether the cluster uses vnodes or single-token architecture.

cassandra-env.sh

The cassandra-env.sh file is located in the installation_location/conf directory.

cassandra.yaml

The cassandra.yaml file is located in the installation_location/conf directory.

Procedure

Verify that the node has a defective disk and identify the disk, by checking the logs on the affected node.

Disk failures are logged in FILE NOT FOUND entries, which identifies the mount point or disk that has failed.
If the node is still running, stop Cassandra and shut down the node.
Replace the defective disk and restart the node.
If the node cannot restart:
1. Try restarting Cassandra without bootstrapping the node:
  Start Cassandra with this option:
```
sudo bin/cassandra Dcassandra.allow_unsafe_replace=true
```
If Cassandra restarts, run nodetool repair on the node. If not, replace the node.
If the repair succeeds, the node is restored to production. Otherwise, go to 7 or 8.
For a cluster using vnodes:
1. On the affected node, clear the system directory on each functioning drive.
  Example for a node with a three disk JBOD array:
```
$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
```
  If mnt1 has failed:
```
$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
```
2. Restart Cassandra without bootstrapping as described in 4:
```
-Dcassandra.allow_unsafe_replace=true
```
3. Run nodetool repair on the node.
  
  If the repair succeeds, the node is restored to production. If not, replace the dead node.
For a cluster single-token nodes:
1. On one of the cluster's working nodes, run nodetool ring to retrieve the list of the repaired node's tokens:
```
$ nodetool ring | grep ip_address_of_node | awk ' {print $NF ","}' | xargs
```
2. Copy the output of the nodetool ring into a spreadsheet (space-delimited).
3. Edit the output, keeping the list of tokens and deleting the other columns.
4. On the node with the new disk, open the cassandra.yaml file and add the tokens (as a comma-separated list) to the initial_token property.
5. Change any other non-default settings in the new nodes to match the existing nodes. Use a diff command to find and merge any differences between the nodes.
  
  If the repair succeeds, the node is restored to production. If not, replace the node.
6. On the affected node, clear the system directory on each functioning drive.
  Example for a node with a three disk JBOD array:
```
$ -/mnt1/cassandra/data
$ -/mnt2/cassandra/data
$ -/mnt3/cassandra/data
```
  If mnt1 has failed:
```
$ rm -fr /mnt2/cassandra/data/system
$ rm -fr /mnt3/cassandra/data/system
```
7. Restart Cassandra without bootstrapping as described in 4:
```
-Dcassandra.allow_unsafe_replace=true
```
8. Run nodetool repair on the node.
  
  If the repair succeeds, the node is restored to production. If not, replace the node.