Multiple topics to multiple tables

The DataStax Apache Pulsar™ Connector allows for mapping multiple topics to multiple tables in a single connector instance.

Most Apache Pulsar™ systems store all messages in the same format and DataStax Apache Pulsar™ Connect workers only support a single converter class for key and value.

In the example stocks_topic, the key is a basic string and the value is regular JSON.

key value

key	value
APPLE	`{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}`
EXXON MOBIL	`{"symbol":"M", "value":80, "exchange":"NYSE", "industry":"ENERGY", "ts":"2018-11-26T19:26:27.483"}`
GENERAL MOTORS	`{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}`
AT&T	`{"symbol":"AT&T", "value":33, "exchange":"NYSE", "industry":"TELECOM", "ts":"2018-11-26T19:26:27.483"}`
FORD MOTOR	`{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}`

APPLE

{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}

EXXON MOBIL

{"symbol":"M",
"value":80,
"exchange":"NYSE",
"industry":"ENERGY",
"ts":"2018-11-26T19:26:27.483"}

GENERAL MOTORS

{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}

AT&T

{"symbol":"AT&T",
"value":33,
"exchange":"NYSE",
"industry":"TELECOM",
"ts":"2018-11-26T19:26:27.483"}

FORD MOTOR

{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}

And baseball_topic has the same structure, basic key and JSON value.

key value

key	value
redsox	`{"number":50, "firstname":"mookie", "lastname":"betts", "street":"4 yawkey way", "city":"boston"}`
redsox	`{"number":28, "firstname":"jd", "lastname":"martinez", "street":"4 yawkey way", "city":"boston"}`
redsox	`{"number":16, "firstname":"andrew", "lastname":"benintendi", "street":"4 yawkey way", "city":"boston"}`
redsox	`{"number":41, "firstname":"chris", "lastname":"sale", "street":"4 yawkey way", "city":"boston"}`
redsox	`{"number":24, "firstname":"david", "lastname":"price", "street":"4 yawkey way", "city":"boston"}`

redsox

{"number":50,
"firstname":"mookie",
"lastname":"betts",
"street":"4 yawkey way",
"city":"boston"}

redsox

{"number":28, "firstname":"jd", "lastname":"martinez", "street":"4 yawkey way", "city":"boston"}

redsox

{"number":16,
"firstname":"andrew",
"lastname":"benintendi",
"street":"4 yawkey way",
"city":"boston"}

redsox

{"number":41, "firstname":"chris", "lastname":"sale", "street":"4 yawkey way", "city":"boston"}

redsox

{"number":24,
"firstname":"david",
"lastname":"price",
"street":"4 yawkey way",
"city":"boston"}

Map the topics into different keyspaces and tables, stocks_keyspace.stocks_table and baseball_keyspace.baseball_table.

The DataStax Schema definitions:

CREATE TABLE stocks_keyspace.stocks_table_by_symbol (
   symbol text,
   ts timestamp,
   exchange text,
   industry text,
   name text,
   value double,
   PRIMARY KEY (symbol, ts));

CREATE TABLE baseball_keyspace.baseball_table (
   team text primary key,
   number int,
   firstname text,
   lastname text,
   street text,
   city text);

In the connector configuration, add the following settings:

topic:
   stocks_topic:
      stocks_keyspace:
         stocks_table_by_symbol:
            mapping: 'symbol=value.symbol, ts=value.dateTime, exchange=value.exchange, industry=value.industry, name=key.name, value=value.value'

topic:
   baseball_topic:
      baseball_keyspace:
         baseball_table:
            mapping: 'team=key, number=value.number, firstname=value.firstname, lastname=value.lastname, street=value.street, city=value.city'

Multiple topics to multiple tables

Was this helpful?

Give Feedback