Setting the NodeSync deadline

Each table with NodeSync enabled has a deadline_target_sec property. This is the target for the maximum time between 2 validations of the same data. As long as the deadline is met, all parts of the ring (for the table) are validated at least that often.

The deadline (deadline_target_sec) relates to grace period (gc_grace_seconds). The deadline should always be less than or equal to the grace period. As long as the deadline is met, no data is resurrected due to tombstone purging.

The deadline defaults to which ever is longer, the grace period or four days. Typically an acceptable default, unless the table has a grace period of zero. For testing, the deadline value can be less than the grace period. Verify for a few weeks if a lower gc_grace value is realistic without taking risk before changing it.

NodeSync prioritize segments in order to try to meet the deadline. The next segment to validate at any given time is the one the closest to missing its deadline. For example, if table 1 has half the deadline of table 2, table 1 validates approximately twice as often as table 2.

Use OpsCenter to get a graphical representation of the NodeSync validation status. See Viewing NodeSync Status.

The syntax to change the per-table nodesync property:

ALTER TABLE <table_name>
WITH nodesync = { 'enabled': 'true',
                  'deadline_target_sec': <value> };

If your set deadline is too aggressive or the NodeSync rate is too low, you may encounter a warning log such as:

WARN  [NodeSyncMaintenanceTasks:1] 2023-10-11 21:21:25,117  NodeSyncMaintenanceTasks.java:250 -
The configured NodeSync rate on this node (256kB/s) is too low to possibly validate all NodeSync
-enabled tables within their respective deadline ('deadline_target_sec' property). This can be
fixed by increasing the rate and/or increasing table deadlines. With the current deadlines and
current table size, the theoretical minimum rate would be 264kB/s, but we would recommend a
_minimum_ of 422kB/s and ideally 774kB/s to account for node failures, temporary slow nodes
and future data growth. Please check 'nodetool nodesyncservice ratesimulator' for more details
on how those values are computed.

Setting the NodeSync deadline

Was this helpful?

Give Feedback