• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Pulsar Connector

    • Getting Started
      • About the DataStax Apache Pulsar™ Connector
        • System requirements
      • Pulsar Connector release notes
      • Installing DataStax Apache Pulsar™ Connector
      • Pulsar Connector single instance quick start
    • Guides and examples
      • Configuration
        • Configuring parallelism
        • Specify writetime timestamp column
        • Setting row-level TTL values from Pulsar fields
        • Pass Pulsar Connector settings directly to the DataStax Java driver
        • Mapping pulsar topics to database tables
          • Determining topic data structure
          • Mapping basic messages to table columns
          • Mapping a message that contain JSON fields
            • Mapping a message that contains both basic and JSON fields
            • Mapping JSON messages
          • Mapping Avro messages
          • Extract Pulsar record header values
          • Mapping messages to table that has a User Defined Type
          • Mapping a topic to multiple tables
          • Multiple topics to multiple tables
          • Provide CQL queries in mappings
          • The now() function in mappings
      • Operations
        • About operating and maintaining the DataStax Connector
        • Scaling the DataStax Apache Pulsar™ Connector
        • Changing the topic or table schema
        • Restarting the DataStax Apache Pulsar™ Connector
        • Displaying the DataStax Apache Pulsar™ Connector configuration
        • Updating the DataStax Apache Pulsar™ Connector configuration
        • Deleting the DataStax Apache Pulsar™ Connector
        • Getting the DataStax Connector status
      • Security
        • Using internal or LDAP authentication
      • DataStax Apache Pulsar™ Connector metrics
      • Troubleshooting
        • Record fails to write
        • Writing fails because of mutation size
        • Data parsing fails
        • Loading balancing datacenter is not specified
    • Reference
      • DataStax Apache Pulsar™ Connector details
      • DataStax connection
      • Pulsar topic-to-table settings
      • Converting date and times for a topic
      • Using the DataStax Apache Pulsar™ Connector with DataStax Enterprise authentication
        • Internal or LDAP authentication
      • SSL encrypted connection
      • Configure error handling
  • DataStax Pulsar Connector
  • Selectively update maps and UDTs based on Pulsar fields

Selectively update maps and UDTs based on Pulsar fields

In the Apache Pulsar™ topic mapping, you can selectively update maps and User Defined Types (UDTs) based on whether Pulsar fields have present values. That is, any value other than null.

The goal is to minimize tombstones in the database. If Maps or UDTs are in the database table that the DataStax Apache Pulsar™ Connector is configured to write to, in order to minimize tombstones, you can use the CQL command UPDATE to only update the fields present in the Pulsar record if the nullToUnset parameter is set to true.

To do so, you can leverage the query parameter with the udtColNotFrozen keyword. The UDT field definition must be not frozen. For example, consider the following DDL statement:

CREATE TYPE IF NOT EXISTS myudt (udtmem1 int, udtmem2 text);

In order for CREATE TYPE in this example to work, the myudt field must have been defined with:

udtColNotFrozen myudt

Once you will have the UDT defined, you can provide a CQL query parameter. Assume you have the following mapping:

bigintcol=key, udtcol1=value.udtmem1, udtcol2=value.udtmem2

You would use an UPDATE command with the udtColNotFrozen keyword. For example:

UPDATE ks.table set udtColNotFrozen.udtmem1=:udtcol1, udtColNotFrozen.udtmem2=:udtcol2 where bigintCol=:bigintcol

When the new record arrives, DataStax Apache Pulsar™ Connector sets only the not null fields; it will not override fields of UDT that have a null value (or are not present) in the Pulsar record, assuming nullToUnset is set to true. The behavior of the Update versus Override is defined via the nullToUnset parameter. See its description in nullToUnset. For related information about Pulsar maps and UDTs, see Mapping messages to table that has a User Defined Type.

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage