• Glossary
  • Support
  • Downloads
  • DataStax Home
Get Live Help
Expand All
Collapse All

DataStax Pulsar Connector

    • Getting Started
      • About the DataStax Apache Pulsar™ Connector
        • System requirements
      • Pulsar Connector release notes
      • Installing DataStax Apache Pulsar™ Connector
      • Pulsar Connector single instance quick start
    • Guides and examples
      • Configuration
        • Configuring parallelism
        • Specify writetime timestamp column
        • Setting row-level TTL values from Pulsar fields
        • Pass Pulsar Connector settings directly to the DataStax Java driver
        • Mapping pulsar topics to database tables
          • Determining topic data structure
          • Mapping basic messages to table columns
          • Mapping a message that contain JSON fields
            • Mapping a message that contains both basic and JSON fields
            • Mapping JSON messages
          • Mapping Avro messages
          • Extract Pulsar record header values
          • Mapping messages to table that has a User Defined Type
          • Mapping a topic to multiple tables
          • Multiple topics to multiple tables
          • Provide CQL queries in mappings
          • The now() function in mappings
      • Operations
        • About operating and maintaining the DataStax Connector
        • Scaling the DataStax Apache Pulsar™ Connector
        • Changing the topic or table schema
        • Restarting the DataStax Apache Pulsar™ Connector
        • Displaying the DataStax Apache Pulsar™ Connector configuration
        • Updating the DataStax Apache Pulsar™ Connector configuration
        • Deleting the DataStax Apache Pulsar™ Connector
        • Getting the DataStax Connector status
      • Security
        • Using internal or LDAP authentication
      • DataStax Apache Pulsar™ Connector metrics
      • Troubleshooting
        • Record fails to write
        • Writing fails because of mutation size
        • Data parsing fails
        • Loading balancing datacenter is not specified
    • Reference
      • DataStax Apache Pulsar™ Connector details
      • DataStax connection
      • Pulsar topic-to-table settings
      • Converting date and times for a topic
      • Using the DataStax Apache Pulsar™ Connector with DataStax Enterprise authentication
        • Internal or LDAP authentication
      • SSL encrypted connection
      • Configure error handling
  • DataStax Pulsar Connector
  • Guides and examples
  • DataStax Apache Pulsar™ Connector metrics

DataStax Apache Pulsar™ Connector metrics

You can view DataStax Apache Pulsar™ Connector metrics in raw Prometheus format at the Apache Pulsar metrics endpoint:

http://<pulsar_instance>:8080/metrics

You can also configure a Prometheus instance to monitor your Pulsar instance as described here, Getting Started.

Prometheus query roots

The entries in this reference are formatted as Prometheus query strings. The root of each query takes the format:

com_datastax_oss_pulsar_sink + sink_name + metric_class

where:

sink_name

is the name of a configured and running DataStax Pulsar Connector sink.

metric_class

is the general class of the metric, for example, Count or SnapShotSize

An example query root would be:

com_datastax_oss_pulsar_sink_dse_sink_kv_Count

Prometheus query attributes

In addition to the Prometheus query root, you can append a name attribute to zero in on specific topics of interest.

An example query would be:

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{name="failedRecordCount"}

and would return the number of failed records:

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="failedRecordCount", table="pulsar_kv", topic="example_topic"}
25

The entries in this reference are separated into the Prometheus root query, and available metrics for the name attribute are listed within.

Specifying only the root query returns all of the available named metrics.

com_datastax_oss_pulsar_sink_<sink_name>_Count

Integer count for various DataStax Pulsar Connect metrics.

cql-client-timeouts

Number of CQL client timeouts.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{driver="driver", instance="localhost:8080", job="pulsar", name="cql-client-timeouts", session="s0"}

cql-requests

Number of CQL requests.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

failedRecordCount

Running total of the number of failed records encountered by the connector instance.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="failedRecordCount", table="pulsar_kv", topic="example_topic"}

recordCount

Running total of records processed.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="recordCount", table="pulsar_kv", topic="example_topic"}

failedRecordsWithUnknownTopic

Number of failed record transactions with an unknown topic.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Count{instance="localhost:8080", job="pulsar", name="failedRecordsWithUnknownTopic"}

com_datastax_oss_pulsar_sink_<sink_name>_SnapshotSize

Snapshot size.

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_SnapshotSize{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_SnapshotSize{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_FiveMinuteRate

Moving average of records processed in the last five minutes.

cql-requests

Moving average of CQL requests processed in the last five minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FiveMinuteRate{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

failedRecordCount

Moving average of failed records processed in the last five minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FiveMinuteRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="failedRecordCount", table="pulsar_kv", topic="example_topic"}

recordCount

Moving average of records processed in the last five minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FiveMinuteRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="recordCount", table="pulsar_kv", topic="example_topic"}

failedRecordsWithUnknownTopic

Moving average of failed records with unknown topics processed in the last five minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FiveMinuteRate{instance="localhost:8080", job="pulsar", name="failedRecordsWithUnknownTopic"}

com_datastax_oss_pulsar_sink_<sink_name>_95thPercentile

95th percentile latency histogram.

cql-requests

Number of CQL requests in the 95th percentile latency histogram.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_95thPercentile{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_95thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_95thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_dse_sink_kv_50thPercentile

50th percentile latency histogram.

cql-requests

Number of CQL requests in the 50th percentile latency histogram.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_50thPercentile{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_50thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_50thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_999thPercentile

999th percentile latency histogram.

cql-requests

Number of CQL requests in the 999th percentile latency histogram.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_999thPercentile{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_999thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_999thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_98thPercentile

98th percentile latency histogram.

cql-requests

Number of CQL requests in the 98th percentile latency histogram.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_98thPercentile{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_98thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_98thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_99thPercentile

99th percentile latency histogram.

cql-requests

Number of CQL requests in the 99th percentile latency histogram.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_99thPercentile{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_99thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_99thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_Mean

Mean number of requests processed.

cql-requests

Mean number of CQL requests processed.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Mean{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Mean{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Mean{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_Max

Maximum number of requests.

cql-requests

Maximum number of CQL requests.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Max{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Max{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Max{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_FifteenMinuteRate

Moving average in the last 15 minutes.

cql-requests

Number of CQL requests in the last 15 minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FifteenMinuteRate{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

failedRecordCount

Moving average of failed records in the last 15 minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FifteenMinuteRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="failedRecordCount", table="pulsar_kv", topic="example_topic"}

recordCount

Moving average of records in the last 15 minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FifteenMinuteRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="recordCount", table="pulsar_kv", topic="example_topic"}

failedRecordsWithUnknownTopic

Moving average of failed records with an unknown topic in the last 15 minutes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_FifteenMinuteRate{instance="localhost:8080", job="pulsar", name="failedRecordsWithUnknownTopic"}

com_datastax_oss_pulsar_sink_<sink_name>_Min

Minimum number of records processed.

CQL Requests

Minimum number of CQL requests processed.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Min{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Min{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_Min{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_MeanRate

Average number of records processed since the connector was created.

cql-requests

Average number of CQL requests processed since the connector was created.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_MeanRate{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

failedRecordCount

Average number of failed records processed since the connector was created.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_MeanRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="failedRecordCount", table="pulsar_kv", topic="example_topic"}

recordCount

Average number of records processed since the connector was created.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_MeanRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="recordCount", table="pulsar_kv", topic="example_topic"}

failedRecordsWithUnknownTopic

Average number of failed records with unknown topics processed since the connector was created.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_MeanRate{instance="localhost:8080", job="pulsar", name="failedRecordsWithUnknownTopic"}

com_datastax_oss_pulsar_sink_<sink_name>_75thPercentile

75th percentile latency histogram.

cql-requests

Number of CQL requests in the 75th percentile latency histogram.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_75thPercentile{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_75thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_75thPercentile{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_OneMinuteRate

Moving average in the last minute.

cql-requests

Number of CQL requests in the last minute.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_OneMinuteRate{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

failedRecordCount

Moving average of failed records in the last minute.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_OneMinuteRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="failedRecordCount", table="pulsar_kv", topic="example_topic"}

recordCount

Moving average of records in the last minute.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_OneMinuteRate{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="recordCount", table="pulsar_kv", topic="example_topic"}

com_datastax_oss_pulsar_sink_<sink_name>_StdDev

Standard deviation.

cql-requests

Standard deviation for CQL requests.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_StdDev{driver="driver", instance="localhost:8080", job="pulsar", name="cql-requests", session="s0"}

batchSize

Returns the number of statements in the CQL batch used to write records to the database.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_StdDev{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSize", table="pulsar_kv", topic="example_topic"}

batchSizeInBytes

Calculated size of every batch statement in bytes.

Example

com_datastax_oss_pulsar_sink_dse_sink_kv_StdDev{instance="localhost:8080", job="pulsar", keyspace="pulsar_qs", name="batchSizeInBytes", table="pulsar_kv", topic="example_topic"}
Using internal or LDAP authentication Record fails to write

General Inquiries: +1 (650) 389-6000 info@datastax.com

© DataStax | Privacy policy | Terms of use

DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc. and its subsidiaries in the United States and/or other countries.

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Kubernetes is the registered trademark of the Linux Foundation.

landing_page landingpage