Multiple topics to multiple tables

The DataStax Apache Pulsar™ Connector allows for mapping multiple topics to multiple tables in a single connector instance.

Most Apache Pulsar™ systems store all messages in the same format and DataStax Apache Pulsar™ Connect workers only support a single converter class for key and value.

In the example stocks_topic, the key is a basic string and the value is regular JSON.

key value

APPLE

{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}

EXXON MOBIL

{"symbol":"M",
"value":80,
"exchange":"NYSE",
"industry":"ENERGY",
"ts":"2018-11-26T19:26:27.483"}

GENERAL MOTORS

{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}

AT&T

{"symbol":"AT&T",
"value":33,
"exchange":"NYSE",
"industry":"TELECOM",
"ts":"2018-11-26T19:26:27.483"}

FORD MOTOR

{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}

And baseball_topic has the same structure, basic key and JSON value.

key value

redsox

{"number":50,
"firstname":"mookie",
"lastname":"betts",
"street":"4 yawkey way",
"city":"boston"}

redsox

{"number":28, "firstname":"jd", "lastname":"martinez", "street":"4 yawkey way", "city":"boston"}

redsox

{"number":16,
"firstname":"andrew",
"lastname":"benintendi",
"street":"4 yawkey way",
"city":"boston"}

redsox

{"number":41, "firstname":"chris", "lastname":"sale", "street":"4 yawkey way", "city":"boston"}

redsox

{"number":24,
"firstname":"david",
"lastname":"price",
"street":"4 yawkey way",
"city":"boston"}

Map the topics into different keyspaces and tables, stocks_keyspace.stocks_table and baseball_keyspace.baseball_table.

The DataStax Schema definitions:

  • CREATE TABLE stocks_keyspace.stocks_table_by_symbol (
       symbol text,
       ts timestamp,
       exchange text,
       industry text,
       name text,
       value double,
       PRIMARY KEY (symbol, ts));
  • CREATE TABLE baseball_keyspace.baseball_table (
       team text primary key,
       number int,
       firstname text,
       lastname text,
       street text,
       city text);

In the connector configuration, add the following settings:

  • topic:
       stocks_topic:
          stocks_keyspace:
             stocks_table_by_symbol:
                mapping: 'symbol=value.symbol, ts=value.dateTime, exchange=value.exchange, industry=value.industry, name=key.name, value=value.value'
  • topic:
       baseball_topic:
          baseball_keyspace:
             baseball_table:
                mapping: 'team=key, number=value.number, firstname=value.firstname, lastname=value.lastname, street=value.street, city=value.city'