Mapping Avro messages
Supports mapping individual fields from a Avro format field.
The DataStax Apache Kafka™ Connector supports mapping individual fields from Avro formatted key or values.
In this example, the key is a basic string and the value is Avro format. The Kafka topic name
is
users_topic
and have the following records:Note: The
kafka-avro-consumer
outputs the Avro field as JSON to the console. When
Avro records are written to Kafka they are often done so using GenericRecord objects, which are fundamentally
different than JSON.key | value |
---|---|
user0 |
|
user1 |
|
user2 |
|
user3 |
|
user4 |
|
The value field uses the following Avro
Schema:
{
"namespace": "example.avro",
"type": "record",
"name": "user",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": “int”},
{"name": "favorite_color", "type": “string”}
]
}
The DataStax keyspace name is
users_keyspace
and table is
users_table
. userid | name | favoritenumber | favoritecolor |
---|---|---|---|
user0 | chris | 14 | blue |
user1 | jack | 56 | pink |
user2 | shereen | 7 | black |
user3 | kimberly | 11 | orange |
user4 | taryn | 37 | green |
To create the table use the following
command:
CREATE TABLE users_keyspace.users_table (userid text primary key, name text,
favoritenumber int, favoritecolor text)
Configure the connector and use the following map specification:
“topic.users_topic.users_keyspace.users_table.mapping”: “userid=key,
name=value.name, favoritenumber=value.favorite_number,
favoritecolor=value.favorite_color”
See
the DataStax Kafka Examples for more.