Map Avro messages
The DataStax Apache Pulsar™ connector supports mapping individual fields from Avro formatted key or values.
Mapping requirements
When mapping fields to columns, the following requirements apply:
-
The data in the Pulsar field must be compatible with the mapped column’s data type. For example, a string field cannot map to a number column.
-
A Pulsar field mapped to a table’s primary key (PK) column must always contain data because primary key columns don’t allow nulls.
-
All primary key columns must have a mapping because primary key columns don’t allow nulls.
Avro mapping example
In this example, the key is a basic string and the value is Avro format.
The Pulsar topic name is users_topic and have the following records:
| Key | Value |
|---|---|
user0 |
|
user1 |
|
user2 |
|
user3 |
|
user4 |
|
The value field uses the following Avro Schema:
{
"namespace": "example.avro",
"type": "record",
"name": "user",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": "int"},
{"name": "favorite_color", "type": "string"}
]
}The following table describes the desired outcome from the mapping when the messages are written to the CQL table:
| userid | name | favoritenumber | favoritecolor |
|---|---|---|---|
user0 |
chris |
14 |
blue |
user1 |
jack |
56 |
pink |
user2 |
shereen |
7 |
black |
user3 |
kimberly |
11 |
orange |
user4 |
taryn |
37 |
green |
To achieve this outcome, the connector’s mapping must be configured as follows.
For this example, the topic name is users_topic, the keyspace name is users_keyspace, and the table name is users_table.
topic:
users_topic:
users_keyspace:
users_table:
mapping: 'userid=key, name=value.name, favoritenumber=value.favorite_number, favoritecolor=value.favorite_color'