Mapping pulsar topics to database tables
DataStax Apache Pulsar™ Connector has a simple yet powerful syntax for mapping fields from a Pulsar record to columns in a supported database table. An instance of the DataStax Apache Pulsar™ Connector can capture multiple topics and supports mapping a single topic to one or more tables.
DataStax Apache Pulsar™ Connector topic mapping
DataStax Apache Pulsar™ Connector supports topic-to-table mappings with supported databases:
-
Determining topic data structure Display messages to determine the data structure of the topic messages.
-
Mapping basic messages to table columns Create a topic-table map for Pulsar messages that only contain a key and value in each record.
-
Mapping a message that contain JSON fields For JSON fields, map individual fields in the structure to columns.
-
Mapping Avro messages Supports mapping individual fields from a Avro format field.
-
Extract Pulsar record header values Extract values from Pulsar record header and write to the database table.
-
Mapping messages to table that has a User Defined Type Write complex types directly into User-defined Types (UDT).
-
Mapping a topic to multiple tables Ingest a single topic into multiple tables using a single connector instance.
-
Multiple topics to multiple tables Ingest multiple topics and write to different tables using a single connector instance.
-
Selectively update maps and UDTs based on Pulsar fields Selectively update maps and UDTs based on Pulsar fields.
-
Provide CQL queries in mappings Provide CQL queries when new record arrives on the Pulsar topic.
-
The now() function in mappings You can use the now() function in mappings.
