• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Pulsar Connector

    • Getting Started
      • About the DataStax Apache Pulsar™ Connector
        • System requirements
      • Pulsar Connector release notes
      • Installing DataStax Apache Pulsar™ Connector
      • Pulsar Connector single instance quick start
    • Guides and examples
      • Configuration
        • Configuring parallelism
        • Specify writetime timestamp column
        • Setting row-level TTL values from Pulsar fields
        • Pass Pulsar Connector settings directly to the DataStax Java driver
        • Mapping pulsar topics to database tables
          • Determining topic data structure
          • Mapping basic messages to table columns
          • Mapping a message that contain JSON fields
            • Mapping a message that contains both basic and JSON fields
            • Mapping JSON messages
          • Mapping Avro messages
          • Extract Pulsar record header values
          • Mapping messages to table that has a User Defined Type
          • Mapping a topic to multiple tables
          • Multiple topics to multiple tables
          • Provide CQL queries in mappings
          • The now() function in mappings
      • Operations
        • About operating and maintaining the DataStax Connector
        • Scaling the DataStax Apache Pulsar™ Connector
        • Changing the topic or table schema
        • Restarting the DataStax Apache Pulsar™ Connector
        • Displaying the DataStax Apache Pulsar™ Connector configuration
        • Updating the DataStax Apache Pulsar™ Connector configuration
        • Deleting the DataStax Apache Pulsar™ Connector
        • Getting the DataStax Connector status
      • Security
        • Using internal or LDAP authentication
      • DataStax Apache Pulsar™ Connector metrics
      • Troubleshooting
        • Record fails to write
        • Writing fails because of mutation size
        • Data parsing fails
        • Loading balancing datacenter is not specified
    • Reference
      • DataStax Apache Pulsar™ Connector details
      • DataStax connection
      • Pulsar topic-to-table settings
      • Converting date and times for a topic
      • Using the DataStax Apache Pulsar™ Connector with DataStax Enterprise authentication
        • Internal or LDAP authentication
      • SSL encrypted connection
      • Configure error handling
  • DataStax Pulsar Connector
  • Reference
  • Pulsar topic-to-table settings

Pulsar topic-to-table settings

Capture Apache Pulsar™ topics in the supported database by specifying a target keyspace, table, and then map the topic fields to table columns in the name of the parameter.

Parameters

Use the following syntax for standalone properties file:

topics: topic_list
topic:
  topic_name:
    keyspace_name:
      table_name:
        mapping: mapping_specification
        consistencyLevel: WRITE_CONSISTENCY_LEVEL
        ttlTimeUnit: seconds
        ttl: -1
        nullToUnset: true
        deletesEnabled: true
topics

A comma separated list of all topics to which the DataStax Connector subscribes.

where:

  • topic_name - Pulsar topic name.

  • keyspace_name - database keyspace where the table is located.

  • table_name - database table where data is written.

mapping

Required, field-to-column mapping. See Mapping pulsar topics to database tables.

consistencyLevel

Query consistency level. DSE settings are:

  • ALL

  • EACH QUORUM

  • QUORUM

  • LOCAL_QUORUM

  • ONE

  • TWO

  • THREE

  • LOCAL\_ONE (default)

  • ANY

    See How are consistent read and write operations handled?

    Default: LOCAL_ONE

nullToUnset

Whether to treat nulls in Pulsar as UNSET in DSE. DataStax recommends using the default to avoid creating unnecessary tombstones.

Default: true

ttl

Time-to-live. Set to the number of seconds before the data is automatically deleted from the DSE table. When you configure topic.<topic-name>.<keyspace-name>.<table-name>.ttl, all rows for that topic table will have this same TTL value. DataStax Pulsar Connector appends AND TTL <configured-ttl-value> to the INSERT statement for those rows.

Default: -1 (disabled)

deletesEnabled

When enabled, treat records that after mapping would result in only non-null values for primary key columns as deletes, rather than inserting/updating nulls for all regular columns.

Only triggers if the setting is enabled and the mapping specification contains all DSE table columns.

Default: true

DataStax connection Converting date and times for a topic

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage