Connectors

Astra Streaming offers fully-managed Apache Pulsarâ„¢ connectors.

Create, monitor, and manage both source and sink connectors through our simple UI, the pulsar-admin CLI, or RESTful API. Connect popular data sources to Apache Pulsar topics or sink data from Pulsar topics to popular systems.

Below is a list of Apache Pulsar source and sink connectors supported by Astra Streaming.

Astra Streaming does not support custom sink or source connectors.

Sink Connectors

AstraDB sink

The AstraDB sink connector reads messages from Apache Pulsar topics and writes them to AstraDB systems.

Cloud Storage sink

The Cloud Storage sink connector reads messages from Apache Pulsar topics and writes them to Cloud Storage systems.

ElasticSearch sink

The Elasticsearch sink connector reads messages from Apache Pulsar topics and writes them to Elasticsearch systems.

Google BigQuery sink

The Google BigQuery sink connector reads messages from Apache Pulsar topics and writes them to BigQuery systems.

JDBC-Clickhouse sink

The JDBC-ClickHouse sink connector reads messages from Apache Pulsar topics and writes them to JDBC-ClickHouse systems.

JDBC-MariaDB sink

The JDBC-MariaDB sink connector reads messages from Apache Pulsar topics and writes them to JDBC-MariaDB systems.

JDBC-PostgreSQL sink

The JDBC-PostgreSQL sink connector reads messages from Apache Pulsar topics and writes them to JDBC-PostgreSQL systems.

JDBC-SQLite

The JDBC-SQLite sink connector reads messages from Apache Pulsar topics and writes them to JDBC-SQLite systems.

Kafka

The Kafka sink connector reads messages from Apache Pulsar topics and writes them to Kafka systems.

Kinesis

The Kinesis sink connector reads messages from Apache Pulsar topics and writes them to Kinesis systems.

Snowflake

The Snowflake sink connector reads messages from Apache Pulsar topics and writes them to Snowflake systems.

Source Connectors

Data Generator source

The Data generator source connector produces messages for testing and persists the messages to Pulsar topics.

Debezium MongoDB source

The Debezium MongoDB source connector reads data from Debezium MongoDB systems and produces data to Pulsar topics.

Debezium MySQL source

The Debezium MySQL source connector reads data from Debezium MySQL systems and produces data to Pulsar topics.

Debezium Oracle source

The Debezium Oracle source connector reads data from Debezium Oracle systems and produces data to Pulsar topics.

Debezium Postgres source

The Debezium PostgreSQL source connector reads data from Debezium PostgreSQL systems and produces data to Pulsar topics.

Debezium SQL Server source

The Debezium SQL Server source connector reads data from Debezium SQL Server systems and produces data to Pulsar topics.

Kafka source

The Kafka source connector reads data from Kafka systems and produces data to Pulsar topics.

AWS Kinesis source

The AWS Kinesis source connector reads data from Kinesis systems and produces data to Pulsar topics.

Experimental Connectors

DataStax is always experimenting with connectors. Below are the connectors currently in development that have not yet been promoted to official support in Astra Streaming.

If you would like to be given access to these connectors, please send a request to astrastreaming@datastax.com (include your Astra account UUID).

Sink Connectors (experimental)

Kinetica
Aerospike
Azure DocumentDB
Azure Data Explorer (Kusto)
Batch Data Generator
CoAP
Couchbase
DataDog
Diffusion
Flume
Apache Geode
Hazelcast
Apache HBase
HDFS 2
HDFS 3
Humio
InfluxDB
JMS
Apache Kudu
MarkLogic
MongoDB
MQTT
Neo4J
New Relic
OrientDB
Apache Phoenix
PLC4X
RabbitMQ
Redis
SAP HANA
SingleStore
Apache Solr
Splunk
XTDB
Zeebe

Source Connectors (experimental)

Cassandra Source
Kinetica
Azure DocumentDB
Batch Data Generator
Big Query
canal
CoAP
Couchbase
datadog
diffusion
DynamoDB
file
flume
Apache Geode
Hazelcast
Humio
JMS
Apache Kudu
MarkLogic
MongoDB
MQTT
Neo4J
New Relic
NSQ
OrientDB
Apache Phoenix
PLC4X
RabbitMQ
Redis
SAP HANA
SingleStore
Splunk
Twitter
XTDB
Zeebe

Listing Sink Connectors

To list available sink connectors in your Astra Streaming tenant, use any of the following.

  • Pulsar Admin

  • cURL

Assuming you have downloaded client.conf to the Pulsar folder:

./bin/pulsar-admin sinks available-sinks

You’ll need to create an Astra Streaming API token to be used with the Rest API. This is different from your Astra tokens.

Navigate to the "Settings" area in the Astra Streaming UI and choose "Create Token".

Retrieve the web service URL from the "Connect" tab in the Astra Streaming UI.

export WEB_SERVICE_URL=<replace-me>
export ASTRA_STREAMING_TOKEN=<replace-me>
curl "$WEB_SERVICE_URL/admin/v3/sinks/builtinsinks" -H "Authorization: $ASTRA_STREAMING_TOKEN"

Listing Source Connectors

To list available source connectors in your Astra Streaming tenant, use any of the following.

  • Pulsar Admin

  • cURL

Assuming you have downloaded client.conf to the Pulsar folder:

./bin/pulsar-admin sources available-sources

You’ll need to create an Astra Streaming API token to be used with the Rest API. This is different from your Astra tokens.

Navigate to the "Settings" area in the Astra Streaming UI and choose "Create Token".

Retrieve the web service URL from the "Connect" tab in the Astra Streaming UI.

export WEB_SERVICE_URL=<replace-me>
export ASTRA_STREAMING_TOKEN=<replace-me>
curl "$WEB_SERVICE_URL/admin/v3/sources/builtinsources" -H "Authorization: $ASTRA_STREAMING_TOKEN"

What’s next?

For more on Pulsar IO connectors, see the Pulsar documentation.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com