Pulsar I/O connectors in IBM Elite Support for Apache Pulsar
|
IBM Elite Support for Apache Pulsar is the new name for Luna Streaming after IBM acquired DataStax. In this documentation, IBM Elite Support for Apache Pulsar and Luna Streaming refer to the same product. For technical specifications, inclusions, and exclusions of the IBM Elite Support for Apache Pulsar offering, see the IBM Elite Support for Apache Pulsar page in the IBM documentation. |
IBM Elite Support for Apache Pulsar offers fully-managed versions of Apache Pulsar™ sink and source connectors.
|
There are three versions of the IBM Elite Support for Apache Pulsar distribution.
The |
Sink connectors
Sink connectors export messages from Pulsar topics to external services.
- Supported sink connectors
-
The following sink connectors are included in IBM Elite Support for Apache Pulsar:
- Experimental sink connectors
-
The following sink connectors are experimental. They are available in the
luna-streaming-allversion of IBM Elite Support for Apache Pulsar, and you can use them in your IBM Elite Support for Apache Pulsar cluster. They are considered experimental because they aren’t fully supported or tested with Astra Streaming.-
Aerospike
-
Apache Geode™
-
Apache HBase®
-
Apache Kudu™
-
Apache Phoenix™
-
Apache Solr™
-
Batch Data Generator
-
CoAP
-
Couchbase
-
DataDog
-
Diffusion
-
Flume
-
Hazelcast
-
HDFS 2
-
HDFS 3
-
Humio
-
InfluxDB
-
JMS
-
Kinetica
-
MarkLogic
-
Microsoft Azure Data Explorer (Kusto)
-
Microsoft Azure DocumentDB
-
MongoDB
-
MQTT
-
Neo4J
-
New Relic
-
OrientDB
-
PLC4X
-
RabbitMQ
-
Redis
-
SAP HANA
-
SingleStore
-
Splunk
-
XTDB
-
Zeebe
-
Source connectors
Source connectors ingest messages from external services into Pulsar topics.
- Supported source connectors
-
The following source connectors are included in IBM Elite Support for Apache Pulsar:
- Experimental source connectors
-
The following source connectors are experimental. They are available in the
luna-streaming-allversion of IBM Elite Support for Apache Pulsar, and you can use them in your IBM Elite Support for Apache Pulsar cluster. They are considered experimental because they aren’t fully supported or tested with Astra Streaming.-
Apache Cassandra
-
Apache Geode™
-
Apache Kudu™
-
Apache Phoenix™
-
Batch Data Generator
-
Big Query
-
Canal
-
CoAP
-
Couchbase
-
DataDog
-
Diffusion
-
DynamoDB
-
File
-
Flume
-
Hazelcast
-
Humio
-
JMS
-
Kinetica
-
MarkLogic
-
Microsoft Azure DocumentDB
-
MongoDB
-
MQTT
-
Neo4J
-
Netty
-
New Relic
-
NSQ
-
OrientDB
-
PLC4X
-
RabbitMQ
-
Redis
-
SAP HANA
-
SingleStore
-
Splunk
-
Twitter
-
XTDB
-
Zeebe
-
Create and manage connectors
You can use the Admin Console and the pulsar-admin CLI to create, monitor, and manage sink and source connectors.
Although you use the same base commands to create and update all Pulsar connectors, each connector has different configuration options. For example commands and configuration details, see the documentation for your preferred sink connectors and source connectors.
pulsar-admin CLI sink operations
- Get available sink connectors
-
Get a list of sink connectors that are available in your Pulsar tenant:
./bin/pulsar-admin sinks available-sinks
- Get sink connector configuration data
-
Get the configuration for an existing sink connector:
# Get information about a connector ./bin/pulsar-admin sinks get \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" - Start a sink connector
-
# Start all instances of a connector ./bin/pulsar-admin sinks start \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only start an individual instance - Stop a sink connector
-
# Stop all instances of a connector ./bin/pulsar-admin sinks stop \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only stop an individual instance - Restart a sink connector
-
# Restart all instances of a connector ./bin/pulsar-admin sinks restart \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only restart an individual instance - Get sink connector status
-
# Check connector status ./bin/pulsar-admin sinks status \ --instance-id "$SINK_INSTANCE_ID" \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" - Delete a sink connector
-
# Delete all instances of a connector ./bin/pulsar-admin sinks delete \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT"
pulsar-admin CLI source operations
- Get available source connectors
-
Get a list of source connectors that are available in your Pulsar tenant:
./bin/pulsar-admin sources available-sources
- Get source connector configuration data
-
Get the configuration for an existing source connector:
# Get information about connector ./bin/pulsar-admin sources get \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" - Start a source connector
-
# Start all instances of a connector ./bin/pulsar-admin sources start \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only start an individual instance - Stop a source connector
-
# Stop all instances of a connector ./bin/pulsar-admin sources stop \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only stop an individual instance - Restart a source connector
-
# Restart all instances of a connector ./bin/pulsar-admin sources restart \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only restart an individual instance - Get source connector status
-
# Check connector status ./bin/pulsar-admin sources status \ --instance-id "$SOURCE_INSTANCE_ID" \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" - Delete a source connector
-
# Delete all instances of a connector ./bin/pulsar-admin sources delete \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT"
Connector metrics
IBM Elite Support for Apache Pulsar exposes Prometheus-formatted metrics for every connector. For more information, see Pulsar Heartbeat.
