Setting row-level TTL values from Pulsar fields

In the Apache Pulsar™ topic mapping, you can optionally specify which column should be used as the ttl (time-to-live) of the record being inserted into the database table. The mapping setting provides a special property, __ttl. The format is:

topic:
  my_topic:
    my_ks:
      my_table:
        mapping: 'col1=key.f1, col2=value.f1, __ttl=value.f2'

When you specify a ttl value to be used while inserting the record, by mapping __ttl or .ttl static configuration, you can further specify a timeUnit. The default is SECONDS. If you want to specify a ttl value in a unit such as HOURS, set timeUnit to HOURS. DataStax Apache Pulsar Connector automatically transforms the value to SECONDS, which is the DSE ttl unit. The transformation also applies for static .ttl field configuration:

topic:
  my_topic:
    my_ks:
      my_table:
        ttlTimeUnit: SECONDS

If the ttl is provided via mapping, such as mapping: `__ttl=value.f2, and by the ttl static configuration (ttl: 86400) the ttl value from the mapping takes precedence and is used. If you set ttl in both resources, DataStax Apache Pulsar™ Connector writes a warning message in the log file.

When you configure ttl, all rows for that topic table will have this same TTL value. DataStax Apache Pulsar™ Connector appends AND TTL <configured-ttl-value> to the INSERT statement for those rows.

When doing dynamic configuration, if the ttl field value is negative, an error is thrown during the insert of the record into a database table. However, because the Pulsar connector is a streaming system, DataStax Apache Pulsar Connector resorts to inserting the record without the ttl value.

Was this helpful?

Edit this page