Query timestamps
Timestamps determine the order of precedence for operations on the same column value from different queries. In Apache Cassandra™ and DataStax Enterprise (DSE), each mutation—update, insert, delete—is assigned a microsecond-precision timestamp to order operations relative to each other. The order of precedence for operations on the same column value is:
-
Data with the latest timestamp.
-
If the operations have the same timestamp, deletes have priority over inserts and updates.
-
Otherwise, the lexically larger value of data has priority. For example, 2 is chosen over 1.
Timestamps can be assigned by the driver client or the server-side node coordinating the request. All recent versions of the DataStax drivers use client-generated timestamps by default for Cassandra versions 2.1 and later and DSE versions 4.7 and later. Older versions of Cassandra and DSE do not support client timestamps, as they were introduced in the CQL native protocol version 3.
Client-side timestamp generation is the default to keep order of operations predictable from the perspective of a single client. Through monotonically increasing client-side timestamps, the driver ensures that all operations are written in the sequential order that they were executed within the scope of that instance.
Without client timestamps, the client is at the whim of timestamps assigned by coordinating nodes. Coordinating nodes assign timestamps based on their internal system clock. It is difficult to keep the different nodes system clock synchronized in a distributed system. Each node is subject to clock drifts ranging from tens of milliseconds to seconds, even when the nodes use NTP or other clock synchronization software.
For example, consider the following scenario where server timestamps are used.
-
A client executes the following query:
DELETE FROM tbl_a WHERE key = 0The query is sent to Node A, which creates a delete mutation with timestamp 10.
-
The client then executes:
UPDATE tbl_a SET x = ‘hello’ where key = 0The query is sent to Node B, which creates an update mutation with timestamp 9.
-
The client executes:
SELECT x from tbl_a where key = 0and receives a result set with 0 rows.
It should be surprising that no rows were returned from the SELECT query in step 3.
Even though the DELETE operation in step 1 was executed before the UPDATE operation in step 2, it takes precedence because the largest timestamp (10) was assigned to it.
This scenario is avoided completely by using client timestamps.
When to use server timestamps
One possible downside to using client timestamps is that the number of client application servers often outnumber DataStax Enterprise nodes in production environments. It is not unusual for different applications using the same DSE cluster to be managed by different teams. In these cases, it may be operationally challenging to keep the clocks synchronized between many different client application servers.
Out-of-sync client application server clocks is an issue only when there are clients making updates to the same partition values as other clients within a window that would be smaller than the expected clock drift between client nodes. Even in this case, it may not be important that updates made in this window be properly ordered in the sequence in which they were executed. It is possible that these updates were made by different parties who are not aware of one another. If it is important, then consider using lightweight transactions.
Lightweight transactions and client timestamps
When executing lightweight transactions (LWTs), any client timestamp assigned to those operations is discarded. This is because DSE maintains a separate timestamp generator that ensures the timestamp assigned is monotonically increased across all LWTs.
One common mistake users make is mixing the use of LWTs and other mutation operations on a single table. This is not recommended, especially since the timestamp mechanism used for normal operations is different than the one used by LWTs, even when using server timestamps.
Keeping clocks in sync across servers
Regardless of the timestamp strategy, DataStax strongly recommends using a service like NTP to keep the system clocks synchronized across all machines in the data ecosystem.
DataStax also recommends organizations measure and understand the degree of clock drift among all the servers in their production environment to understand the time windows that may exist between nodes.
Use utilities and commands, such as clockdiff, ntpdate -q, and ntp -q, to measure clock differences between servers.
