Use the NodeSync Service
NodeSync automatically and continuously synchronizes replicas of designated keyspaces and tables of a cluster as a background process. The DSE OpsCenter NodeSync Service is initially enabled by default, but keyspaces and tables must be opted in to the NodeSync Service.
NodeSync Service, Repair Service, and limitations on repair operations
Unlike incremental repairs in the Repair Service, the NodeSync Service does not build a Merkle tree for comparisons and stream differences for compaction. Rather, the NodeSync incrementally scans a token range, compares data a page at a time, and sends data over the standard write path.
|
The |
The legacy Repair Service can work in conjunction with the NodeSync Service.
Tables without NodeSync can use the legacy Repair Service freely.
If preferable for an environment, tables can be repaired (synchronized) using either the legacy Repair Service or the NodeSync Service. The Repair Service does not start any subrange or incremental repairs on tables that have NodeSync enabled and running until the NodeSync operation completes. If a legacy repair (also known as classic incremental or subrange repairs) is in progress for a table that has NodeSync enabled, NodeSync waits for the classic repair to finish before proceeding. If a classic repair is launched against a keyspace with NodeSync enabled on some but not all tables, a warning is logged and classic repair jobs are started for the non-NodeSync tables.
|
For operational simplicity and better performance, DataStax recommends running NodeSync Service exclusively rather than the legacy Repair Service. |
Prepare your DSE cluster for NodeSync
-
NodeSync is enabled in DSE by default. If you disabled NodeSync in DSE, you must enable NodeSync to use the NodeSync Service.
Incremental NodeSync is available in DataStax Enterprise (DSE) 6.8 and later.
-
If any keyspaces are empty, create at least one table in each keyspace that you want to opt in to NodeSync.
You cannot enable NodeSync on empty keyspaces in OpsCenter.
-
-
If OpsCenter authentication is enabled, set user permissions for the NodeSync Service for the applicable roles and clusters.
Enable the NodeSync Service for keyspaces and tables
NodeSync and incremental NodeSync are enabled by default on a cluster, but you must opt in keyspaces and tables for synchronization:
-
Click the name of the cluster that you want to manage, and then click Services.
-
Select Details for the NodeSync Service.
-
Select the Settings tab.
NodeSync Settings shows only keyspaces and tables eligible for NodeSync. Some tables are not shown, and they are exempt from NodeSync because the keyspace has a replication factor of 1, or it is a reserved system keyspace. Select View excluded tables to view the excluded keyspaces and tables.
-
Select the keyspaces and tables to configure NodeSync for your environment, and then select Configure.
By default, you can configure up to 50 tables at once. If you encounter schema agreement errors when enabling or disabling multiple tables, increase the value of
max_request_tables. -
Enable NodeSync or incremental NodeSync for the selected keyspaces and tables.
Enabling NodeSync sends an
ALTER TABLEstatement that is effective immediately; You don’t need to restart DSE.
After you enable NodeSync, monitor NodeSync operations for keyspace and tables:
Monitor NodeSync
There are several ways to monitor NodeSync operations.
NodeSync metrics
NodeSync metrics are available to indicate progress on repaired data and objects (rows and range tombstones), validated data and objects, and processed pages details. Set up alerts and dashboard graphs based on NodeSync metrics pertinent to your production environment.
NodeSync rule for the Best Practice Service
The NodeSync Not Running rule fails if the NodeSync Service is not running on a node, unless the rule has been turned off in the Best Practice Service.

NodeSync status and details
View the progress and status of NodeSync operations for keyspace and tables in OpsCenter Monitoring. The NodeSync Status page provides summary and detailed health status information.
-
Click the name of the cluster you want to inspect, and then click Services.
-
Select Details for the NodeSync Service.
The Status tab shows the current progress of NodeSync operations, including the time of the latest data refresh.
Each cluster, keyspace, and table has a color-coded legend that conveys its NodeSync status (listed from best to worst) when hovering over a particular progress bar:
-
Green: Percentage of the data was synced within the first 50% of the time period for the NodeSync deadline.
-
Yellow: Percentage of the data was synced within the first 90% of the time period for the NodeSync deadline.
-
Orange: Percentage of the data was synced within the last 10% of the time period for the NodeSync deadline.
-
Red: Percentage of the data was not synced before the NodeSync deadline. DataStax recommends that you increase your NodeSync throughput.
The NodeSync deadline is a table parameter that specifies the maximum number of seconds to validate all segments of the table. If the deadline is not explicitly set for a table, the deadline defaults to whichever is greater:
gc_grace_secondsor the minimum deadline target of 4 days. You can set the deadline value with thedeadline_target_secparameter in anALTER TABLEstatement. For example:ALTER TABLE table_name WITH nodesync={ 'enabled': 'true', 'deadline_target_sec' : <some value>}Select a progress bar section to view a tooltip with more details about progress status. The Services panel at Cluster > Nodes also shows the NodeSync progress bar.
-
-
Select the Keyspaces or Tables link to view the Detailed Health Status.
-
Select a table within Detailed Health Status to view NodeSync Details for that table.
Configure the NodeSync data refresh interval
The NodeSync refresh_period is the frequency, in seconds, at which NodeSync status data is refreshed in the NodeSync metrics table in OpsCenter Monitoring.
The default is 300 seconds (5 minutes).
The interval is measured from the end of the last refresh operation.
For example, If a status refresh takes 2 minutes to complete, the refresh period timer starts again and refreshes NodeSync Status after another 5 minutes.
To modify the refresh_period, do the following:
-
Edit the relevant configuration file for the configuration scope and your installation type:
File Scope Package installation location Tarball installation location opscenterd.confGlobal, all clusters
/etc/opscenter/opscenterd.confINSTALL_DIRECTORY/conf/opscenterd.confCLUSTER_NAME.confCluster-specific
/etc/opscenter/clusters/INSTALL_DIRECTORY/conf/clusters/address.yamlNode-specific
/var/lib/datastax-agent/conf/address.yamlINSTALL_DIRECTORY/conf/address.yaml -
In the
[nodesync]section, set therefresh_periodoption.The following example sets the interval to 15 minutes (900 seconds):
[nodesync] refresh_period=900 -
If you edited
address.yaml, restart each DataStax Agent.
Tune throughput for NodeSync
NodeSync is not self-tuning.
If NodeSync operations don’t finish before the expiration of a table’s GC grace period, you must adjust NodeSync throughput with the rate_in_kb parameter.
You can set this value in a Lifecycle Manager (LCM) Config Profile, and then deploy the configuration update by running a configuration job, as explained in the following steps.
Alternatively, you can manually configure NodeSync with nodetool nodesyncservice.
Configure the NodeSync rate_in_kb for DSE using LCM:
-
Select Config Profiles from the LCM navigation menu.
-
Select the edit icon for the configuration profile to edit, or click Add config profile if you have not already created a profile.
-
In the Config Profile pane under the Cassandra section, click cassandra.yaml.
-
Scroll down to the NodeSync pane.
-
Edit a value for rate_in_kb.
The default is 1024 KB.
To determine the optimal rate, use
nodetool nodesyncservice ratesimulator. For more information, see Setting the NodeSync rate. -
Select Save to save the configuration profile.
-
Go to the Clusters workspace in LCM, and then select the configuration profile to apply at the cluster, datacenter, or node level.
-
Run a configuration job to push the updated configuration to all applicable nodes.
-
View NodeSync Status in OpsCenter Monitoring to ensure that the new rate allows NodeSync jobs to complete within the required time frame.