Migrating to incremental repairs

Repairing SSTables using anti-entropy repairis a necessary part of Cassandra maintenance. A full repair of all SSTables on a node takes a lot of time and is resource-intensive. You can manage repairs with less service disruption using incremental repair. Incremental repair consumes less time and resources because it skips SSTables that are already marked as repaired.

Incremental repair works equally well with any compaction scheme — Size-Tiered Compaction (STCS), Date-Tiered Compaction(DTCS), Time-Window Compaction(TWCS), or Leveled Compaction (LCS).

In Cassandra 3.0 and later, switching from full repair to incremental repair is easier than before. However, the first system-wide incremental repair can take a long time, as Cassandra recompacts all SSTables according to the chosen compaction shceme. You can make this process less disruptive by migrating the cluster to incremental repair one node at a time.

Overview of the procedure

To migrate one Cassandra node to incremental repair:

Disable autocompaction on the node.
Run a full, sequential repair.
Stop the node.
Set the repairedAt metadata value to each SSTable that existed before you disabled compaction.
Restart Cassandra on the node.
Re-enable autocompaction on the node.

Prerequisites

Listing SSTables

Before you run a full repair on the node, list its SSTables. The existing SSTables may not be changed by the repair process, and the incremental repair process you run later will not process these SSTables unless you set the repairedAt value for each SSTable (see Step 4 below).

You can find the node's SSTables in one of the following locations:

Package installations: /var/lib/cassandra/data
Tarball installations: install_location/data/data

This directory contains a subdirectory for each keyspace. Each of these subdirectories contains a set of files for each SSTable. The name of the file that contains the SSTable data has the following format:

<version_code>-<generation>-<format>-Data.db

Note: You can mark multiple SSTables as a batch by running sstablerepairedset with a text file of filenames — see Step 4.

Migrating the node to incremental repair

Note: In RHEL and Debian installations, you must install the tools packages before you can follow these steps.

Disable autocompaction on the node
From the install_directory:
```
$ bin/nodetool disableautocompaction
```
Running this command without parameters disables autocompaction for all keyspaces. For details, see disableautocompaction.
Run the default full, sequential repair
From the install_directory:
```
$ bin/nodetool repair
```
Running this command without parameters starts a full sequential repair of all SSTables on the node. This may take a substantial amount of time. For details, see repair.
Stop the node.
Set the repairedAt metadata value to each SSTable that existed before you disabled compaction.
Use sstablerepairedset. To mark a single SSTable SSTable-example-Data.db:
```
sudo bin/sstablerepairedset --really-set --is-repaired SSTable-example-Data.db
```
To do this as a batch process using a text file of SSTable names:
```
sudo bin/sstablerepairedset --really-set --is-repaired -f SSTable-names.txt
```
Note: The value of the repairedAt metadata is the timestamp of the last repair. The sstablerepairedset command applies the current date/time. To check the value of the repairedAt metadata for an SSTable, use:
```
$ bin/sstablemetadata example-keyspace-SSTable-example-Data.db | grep "Repaired at"
```
Restart the node.

What's next

After you have migrated all nodes, you will be able to run incremental repairs using nodetool repair with the -inc option. For details, see https://www.datastax.com/dev/blog/more-efficient-repairs.