Recovering from a single disk failure using JBOD

Steps for recovering from a single disk failure in a disk array using JBOD (just a bunch of disks).

1. Verify that the node has a defective disk and identify the disk, by checking the logs on the affected node.

   Disk failures are logged in FILE NOT FOUND entries, which identifies the mount point or disk that has failed.

2. If the node is still running, stop DSE and shut down the node.
3. Replace the defective disk and restart the node.
4. If the node cannot restart:
   1. Try restarting DSE without bootstrapping the node:
      Package and Installer-Services installations:

      1. Add the following option to cassandra-env.sh file:

         JVM_OPTS="$JVM_OPTS -Dcassandra.allow_unsafe_replace=true

      2. Starting DataStax Enterprise as a service.
      3. After the node bootstraps, remove the -Dcassandra.allow_unsafe_replace=true parameter from cassandra-env.sh.
      4. Starting DataStax Enterprise as a service.

      Tarball and Installer-No Services installations:

      • Start DataStax Enterprise with this option:

        sudo bin/dse cassandra Dcassandra.allow_unsafe_replace=true

        Tarball and Installer No-Services path:

        installation_location

5. If DSE restarts, run nodetool repair on the node. If not, replace the node.
6. If the repair succeeds, the node is restored to production. Otherwise, go to 7 or 8.
7. For a cluster using vnodes:
   1. On the affected node, clear the system directory on each functioning drive.
      Example for a node with a three disk JBOD array:

      $ -/mnt1/cassandra/data
      $ -/mnt2/cassandra/data
      $ -/mnt3/cassandra/data

      If mnt1 has failed:

      $ rm -fr /mnt2/cassandra/data/system
      $ rm -fr /mnt3/cassandra/data/system

   2. Restart DSE without bootstrapping as described in 4:

      -Dcassandra.allow_unsafe_replace=true

   3. Run nodetool repair on the node.

      If the repair succeeds, the node is restored to production. If not, replace the dead node.

8. For a cluster single-token nodes:
   1. On one of the cluster's working nodes, run nodetool ring to retrieve the list of the repaired node's tokens:

      $ nodetool ring | grep ip_address_of_node | awk ' {print $NF ","}' | xargs

   2. Copy the output of the nodetool ring into a spreadsheet (space-delimited).
   3. Edit the output, keeping the list of tokens and deleting the other columns.
   4. On the node with the new disk, open the cassandra.yaml file and add the tokens (as a comma-separated list) to the initial_token property.
   5. Change any other non-default settings in the new nodes to match the existing nodes. Use the diff command to find and merge any differences between the nodes.

      If the repair succeeds, the node is restored to production. If not, replace the node.

   6. On the affected node, clear the system directory on each functioning drive.
      Example for a node with a three disk JBOD array:

      $ -/mnt1/cassandra/data
      $ -/mnt2/cassandra/data
      $ -/mnt3/cassandra/data

      If mnt1 has failed:

      $ rm -fr /mnt2/cassandra/data/system
      $ rm -fr /mnt3/cassandra/data/system

   7. Restart DSE without bootstrapping as described in 4:

      -Dcassandra.allow_unsafe_replace=true

   8. Run nodetool repair on the node.

      If the repair succeeds, the node is restored to production. If not, replace the node.