Pulsar I/O connectors in Luna Streaming
Luna Streaming offers fully-managed versions of Apache Pulsar™ sink and source connectors.
|
There are three versions of the DataStax Luna Streaming distribution.
The |
Sink connectors
Sink connectors export messages from Pulsar topics to external services.
- Supported sink connectors
-
The following sink connectors are included in Luna Streaming:
Source connectors
Source connectors ingest messages from external services into Pulsar topics.
- Supported source connectors
-
The following source connectors are included in Luna Streaming:
Create and manage connectors
You can use the Admin Console and the pulsar-admin CLI to create, monitor, and manage sink and source connectors.
Although you use the same base commands to create and update all Luna Streaming Pulsar connectors, each connector has different configuration options. For example commands and configuration details, see the documentation for your preferred sink connectors and source connectors.
pulsar-admin CLI sink operations
- Get available sink connectors
-
Get a list of sink connectors that are available in your Luna Streaming Pulsar tenant:
./bin/pulsar-admin sinks available-sinks
- Get sink connector configuration data
-
Get the configuration for an existing sink connector:
# Get information about a connector ./bin/pulsar-admin sinks get \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" - Start a sink connector
-
# Start all instances of a connector ./bin/pulsar-admin sinks start \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only start an individual instance - Stop a sink connector
-
# Stop all instances of a connector ./bin/pulsar-admin sinks stop \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only stop an individual instance - Restart a sink connector
-
# Restart all instances of a connector ./bin/pulsar-admin sinks restart \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only restart an individual instance - Get sink connector status
-
# Check connector status ./bin/pulsar-admin sinks status \ --instance-id "$SINK_INSTANCE_ID" \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT" - Delete a sink connector
-
# Delete all instances of a connector ./bin/pulsar-admin sinks delete \ --namespace "$NAMESPACE" \ --name "$SINK_NAME" \ --tenant "$TENANT"
pulsar-admin CLI source operations
- Get available source connectors
-
Get a list of source connectors that are available in your Luna Streaming Pulsar tenant:
./bin/pulsar-admin sources available-sources
- Get source connector configuration data
-
Get the configuration for an existing source connector:
# Get information about connector ./bin/pulsar-admin sources get \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" - Start a source connector
-
# Start all instances of a connector ./bin/pulsar-admin sources start \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only start an individual instance - Stop a source connector
-
# Stop all instances of a connector ./bin/pulsar-admin sources stop \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only stop an individual instance - Restart a source connector
-
# Restart all instances of a connector ./bin/pulsar-admin sources restart \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" # optionally add --instance-id to only restart an individual instance - Get source connector status
-
# Check connector status ./bin/pulsar-admin sources status \ --instance-id "$SOURCE_INSTANCE_ID" \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT" - Delete a source connector
-
# Delete all instances of a connector ./bin/pulsar-admin sources delete \ --namespace "$NAMESPACE" \ --name "$SOURCE_NAME" \ --tenant "$TENANT"
Connector metrics
Luna Streaming exposes Prometheus-formatted metrics for every connector. For more information, see Pulsar Heartbeat.
