Write request processing
When Astra DB Serverless receives a write request, a coordinator node distributes the write request to the relevant replica nodes for processing, and then manages the response to the client. Replicas that receive a write process it independently through the write path. The coordinator node’s activities and the client response are related but separate from each replica’s internal write path.
The write path is a multi-stage process designed for data durability and consistency. It preserves the integrity of the database when writing data to disk, and it reduces the chance of lost or incomplete writes in the event of a failure on the write path. The stages of the write path are as follows:
-
Append the write request to the commit log.
-
Write the data to memtable storage.
-
Flush memtables.
-
Store the data on disk in immutable SSTables.
|
The timestamp for all writes is Universal Time Coordinated (UTC). |
Commit log and memtable storage of writes
When a write occurs, the database immediately does the following:
-
Appends the write to the commit log
-
Writes the data to a memtable.
Memtables are table-specific, in-memory data structures that temporarily store writes in sorted order until they are flushed to disk as immutable SSTables. Flushes can occur manually or automatically based on configurable thresholds. Functionally, the memtable is a write-back cache of data partitions that the database looks up by key.
The commit log ensures that writes are durably recorded before being acknowledged to the client. These durable writes survive permanently even if the node experiences an outage along the write path.
|
Commit logs aren’t a guarantee that data will be written to disk, and they cannot prevent all possible dropped writes. For example, if a network outage occurs before a write request reaches the cluster, the database cannot append it to the commit log because it never received the request. |
Memtable flushes
To flush data from the memtable, the database writes data to disk in the memtable-sorted order. A partition index is also created on the disk that maps the tokens to a location on disk.
When the memtable or commit log exceed a predefined threshold, the memtable is put in a queue that is flushed to disk.
Commit log segments and crash recovery
When recovering from a crash, the database can use the commit log to rebuild memtables at startup.
During normal operations, the commit log is divided into segments. Writes are recorded in order, and new segments are created when the current segment reaches the predefined commit log segment size.
The database purges commit log segments only after all data in a segment is flushed to disk from the memtable. If the commit log directory reaches a predefined maximum size, then the oldest segments are purged and the corresponding memtables are flushed to disk.
Data stored on disk in SSTables
The commit log is shared among all tables, but each table has its own memtables and SSTables.
SSTables are immutable. After the memtable is flushed, the database doesn’t write to that SSTable again. Instead, new writes are stored in new memtables that are then flushed to new SSTables. Consequently, a partition is typically stored across multiple SSTable files.
To perform read operations, the database uses SSTable data files and other generated SSTable structures. Read request latency increases relative to the number of SSTables that the database must access to read an entire row. Ideally, one query shouldn’t require more than 3 SSTables. To minimize the number of SSTables, the database periodically runs a compaction process that consolidates SSTables.
Updates and upserts
Astra DB Serverless treats each row in a write request as an upsert:
-
If the incoming row has the same primary key as that of an existing row, the database processes it as an update to the existing row.
-
If the incoming row’s primary key doesn’t match any existing row, then the database processes it as an insert of a new row.
During a write, Astra adds each new row to the database without checking whether a duplicate record exists on any other replica. This means that multiple versions of the same row can exist simultaneously on different replicas. Because Astra adheres to eventual consistency, it doesn’t immediately reconcile multiple versions of the same row across replicas.
During memtable flushes and SSTable compaction, if the database encounters multiple versions of the same row, it only writes the most recent version to the new SSTable. After compaction, the node drops the original SSTables, deleting the outdated rows. However, each node performs compaction independently, even if it is in the same replica set as another recently compacted node. Therefore, when compaction drops outdated versions of a row from one node, the outdated rows can still exist on another node. Compaction eventually reconciles rows as it runs on all the nodes, but this might not happen fast enough for rows that are frequently updated.
To compensate for eventual consistency among the replicas, the database performs another round of comparisons on the read path. When a client requests data with a particular primary key, the database retrieves the row from one or more replicas, and then selects the version with the most recent timestamp to return to the client. This is known as last write wins, and it applies to each column individually. This is because writes don’t always include all columns, so replicas can store partially inconsistent versions of a row. During a compaction or read, the database assembles a complete version of each row from all partial updates, using the most recent version of each column.
