Selectively update maps and UDTs based on Pulsar fields
In the Apache Pulsar™ topic mapping, you can selectively update maps and User Defined Types (UDTs) based on whether Pulsar fields have present values. That is, any value other than
The goal is to minimize tombstones in the database. If Maps or UDTs are in the database table that the DataStax Apache Pulsar™ Connector is configured to write to, in order to minimize tombstones, you can use the CQL command
UPDATE to only update the fields present in the Pulsar record if the
nullToUnset parameter is set to
To do so, you can leverage the query parameter with the
The UDT field definition must be not frozen.
For example, consider the following DDL statement:
CREATE TYPE IF NOT EXISTS myudt (udtmem1 int, udtmem2 text);
In order for
CREATE TYPE in this example to work, the
myudt field must have been defined with:
Once you will have the UDT defined, you can provide a CQL query parameter. Assume you have the following mapping:
bigintcol=key, udtcol1=value.udtmem1, udtcol2=value.udtmem2
You would use an
UPDATE command with the
UPDATE ks.table set udtColNotFrozen.udtmem1=:udtcol1, udtColNotFrozen.udtmem2=:udtcol2 where bigintCol=:bigintcol
When the new record arrives, DataStax Apache Pulsar™ Connector sets only the not null fields;
it will not override fields of UDT that have a null value (or are not present) in the Pulsar record, assuming
nullToUnset is set to
The behavior of the Update versus Override is defined via the
See its description in nullToUnset.
For related information about Pulsar maps and UDTs, see Mapping messages to table that has a User Defined Type.