Selectively update maps and UDTs based on Kafka fields

Selectively update maps and UDTs based on Kafka fields.

In the Kafka topic mapping, you can selectively update maps and User Defined Types (UDTs) based on whether Kafka fields have present values. That is, any value other than null.

The goal is to minimize tombstones in the database. If Maps or UDTs are in the database table that the Kafka Connector is configured to write to, in order to minimize tombstones, you can use the CQL command UPDATE to only update the fields present in the Kafka record if the nullToUnset parameter is set to true.

To do so, you can leverage the query parameter with the udtColNotFrozen keyword. The UDT field definition must be not frozen. For example, consider the following DDL statement:

CREATE TYPE IF NOT EXISTS myudt (udtmem1 int, udtmem2 text);

In order for CREATE TYPE in this example to work, the myudt field must have been defined with:

udtColNotFrozen myudt

Once you will have the UDT defined, you can provide a CQL query parameter. Assume you have the following mapping:

bigintcol=key, udtcol1=value.udtmem1, udtcol2=value.udtmem2

You would use an UPDATE command with the udtColNotFrozen keyword. For example:

UPDATE ks.table set udtColNotFrozen.udtmem1=:udtcol1, udtColNotFrozen.udtmem2=:udtcol2 where bigintCol=:bigintcol

When the new record arrives, DataStax Apache Kafka™ Connector sets only the not null fields; it will not override fields of UDT that have a null value (or are not present) in the Kafka record, assuming nullToUnset is set to true. The behavior of the Update versus Override is defined via the nullToUnset parameter. See its description in Kafka topic-to-table settings. For related information about Kafka maps and UDTs, see Mapping messages to table that has a User Defined Type.