Mapping a topic to multiple tables

The DataStax Apache Pulsar™ Connector allows for mapping a single topic to multiple tables for a single connector instance.

In the example stocks_topic, the key is a basic string and the value is regular JSON.

key value

key	value
APPLE	`{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}`
EXXON MOBIL	`{"symbol":"M", "value":80, "exchange":"NYSE", "industry":"ENERGY", "ts":"2018-11-26T19:26:27.483"}`
GENERAL MOTORS	`{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}`
AT&T	`{"symbol":"AT&T", "value":33, "exchange":"NYSE", "industry":"TELECOM", "ts":"2018-11-26T19:26:27.483"}`
FORD MOTOR	`{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}`

APPLE

{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}

EXXON MOBIL

{"symbol":"M",
"value":80,
"exchange":"NYSE",
"industry":"ENERGY",
"ts":"2018-11-26T19:26:27.483"}

GENERAL MOTORS

{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}

AT&T

{"symbol":"AT&T",
"value":33,
"exchange":"NYSE",
"industry":"TELECOM",
"ts":"2018-11-26T19:26:27.483"}

FORD MOTOR

{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}

In the DataStax keyspace stocks_keyspace, create three different tables that optimized with different schemas.

stocks_table_by_symbol

CREATE TABLE stocks_keyspace.stocks_table_by_symbol (
   symbol text,
   ts timestamp,
   exchange text,
   industry text,
   name text,
   value double,
   PRIMARY KEY (symbol, ts));

stocks_table_by_exhange

CREATE TABLE stocks_keyspace.stocks_table_by_exchange (
   symbol text,
   ts timestamp,
   exchange text,
   industry text,
   name text,
   value double,
   PRIMARY KEY (exchange, ts));

stocks_table_by_industry

CREATE TABLE stocks_keyspace.stocks_table_by_industry (
   symbol text,
   ts timestamp,
   exchange text,
   industry text,
   name text,
   value double,
   PRIMARY KEY (industry, ts));

Configure the connector and add all the map specifications:

stocks_topic to stocks_table_by_symbol

topic:
   stocks_topic:
      stocks_keyspace:
         stocks_table_by_symbol:
            mapping: 'symbol=value.symbol, ts=value.ts, exchange=value.exchange, industry=value.industry, name=key.name, value=value.value'

stocks_topic to stocks_table_by_exchange

“topic:
   stocks_topic:
      stocks_keyspace:
         stocks_table_by_exchange:
            mapping: 'symbol=value.symbol, ts=value.ts, exchange=value.exchange, industry=value.industry, name=key.name, value=value.value'

stocks_topic to stocks_table_by_industry

topic:
   stocks_topic:
      stocks_keyspace:
         stocks_table_by_industry:
            mapping: 'symbol=value.symbol, ts=value.ts, exchange=value.exchange, industry=value.industry, name=key.name, value=value.value'

Mapping a topic to multiple tables

Was this helpful?

Give Feedback