NodeSync: Continuous background repair

NodeSync is an easy to use continuous background repair that has low overhead and provides consistent performance and virtually eliminates manual efforts to run repair operations in a DataStax cluster.

NodeSync is an easy to use continuous background repair that has low overhead and provides consistent performance.

NodeSync service

By default, each node runs the NodeSync service. The service is idle unless it has something to validate. NodeSync is enabled/disabled on each table individually. The service continuously validates local data ranges for NodeSync-enabled tables and repairs any inconsistency found. The local data ranges are split into small segments, which act as validation save points. Segments are prioritized in order to try to meet the per-table deadline target.

Segments

A segment is a small local token range of a table. It is created by NodeSync by recursively splitting local ranges in half a certain number of times (depth). The depth calculated using the total table size, assuming equal distribution of data, create segments that cover no more than 200 MB. Token ranges can be no smaller than a single partition, so very large partitions can result in segments larger than the configured size.

Validation process and status

After a segment is selected for validation, NodeSync reads the entirety of the data it covers from all replica (using paging), checks for inconsistencies, and repairs if needed. When a node validates a segment, it “locks” it in a system table to avoid work duplication by other nodes. It is not a race-free lock; there is a possibility of duplicated work which saves the complexity and cost of true distributed locking. Segment validation is saved on completion in the system_distributed.nodesync_status table, which is used internally for resuming on failure, prioritization, segment locking, and by tools; not meant to be read directly.