Kinesis

archive

true

The connector type, like 'builtin://debezium-mysql'

batchBuilder

false

BatchBuilder provides two types of batch construction methods, DEFAULT and KEY_BASED. The default value is: DEFAULT

batchSourceConfig

false

Batch source config key/value (as a JSON string)

className

true

The connector type’s class reference, like 'org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource'

configs

false

{}

JSON key/value config of source type specific settings. Example: {"property1":"1234","property2":{"subProperty":"asdf"}}

customRuntimeOptions

false

A string that encodes options to customize the runtime, see Apache Pulsar docs for configured runtime for details

name

true

Give your source a good name for later reference. The name must start with a lowercase alphabetic character. It can only contain lowercase alphanumeric characters, and hyphens (kebab-case).

namespace

true

The namespace you’d like the source created under

parallelism

true

1

The number of a Pulsar Source instances to run

processingGuarantees

true

ATLEAST_ONCE

The delivery semantics applied to the Pulsar Sink. Values are 'ATLEAST_ONCE', 'ATMOST_ONCE', 'EFFECTIVELY_ONCE'

producerConfig

false

The custom producer configuration (as a JSON string)

resources

false

The compute resources that need to be allocated per source instance (applicable only to the process)(as a JSON string). Example: {"cpu": 0.25,"disk":1000000000,"ram":500000000}

runtimeFlags

false

A string that encodes options to customize the runtime, see Apache Pulsar docs for configured runtime for details

schemaType

false

The schema type (either a builtin schema like 'avro', 'json', etc.. or custom Schema class name to be used to encode messages emitted from the Pulsar Source

secrets

false

This is a map of secretName(that is how the secret is going to be accessed in the function via context) to an object that encapsulates how the secret is fetched by the underlying secrets provider. The type of an value here can be found by the SecretProviderConfigurator.getSecretObjectType() method

serdeClassName

false

The SerDe classname for the Pulsar Source

tenant

true

The tenant you’d like the source created under

topicName

true

The name of an existing topic in Astra Streaming, where messages will be published to. Should be in the format of [non-]persistent://<tenant>/<namespace>/<topic-name>

initialPositionInStream

InitialPositionInStream

false

LATEST

The position where the connector starts from. Below are the available options:
* AT_TIMESTAMP: start from the record at or after the specified timestamp.
* LATEST: start after the most recent data record.
* TRIM_HORIZON: start from the oldest available data record.

startAtTime

Date

false

" " (empty string)

If set to AT_TIMESTAMP, it specifies the point in time to start consumption.

applicationName

String

false

Pulsar IO connector

The name of the Amazon Kinesis application.
By default, the application name is included in the user agent string used to make AWS requests. This can assist with troubleshooting, for example, distinguish requests made by separate connector instances.

checkpointInterval

long

false

60000

The frequency of the Kinesis stream checkpoint in milliseconds.

backoffTime

long

false

3000

The amount of time to delay between requests when the connector encounters a throttling exception from AWS Kinesis in milliseconds.

numRetries

int

false

3

The number of re-attempts when the connector encounters an exception while trying to set a checkpoint.

receiveQueueSize

int

false

1000

The maximum number of AWS records that can be buffered inside the connector.
Once the receiveQueueSize is reached, the connector does not consume any messages from Kinesis until some messages in the queue are successfully consumed.

dynamoEndpoint

String

false

" " (empty string)

The Dynamo end-point URL, which can be found at here.

cloudwatchEndpoint

String

false

" " (empty string)

The Cloudwatch end-point URL, which can be found at here.

useEnhancedFanOut

boolean

false

true

If set to true, it uses Kinesis enhanced fan-out. +If set to false, it uses polling.

awsEndpoint

String

false

" " (empty string)

The Kinesis end-point URL, which can be found at here.

awsRegion

String

false

" " (empty string)

The AWS region. Example us-west-1, us-west-2

awsKinesisStreamName

String

true

" " (empty string)

The Kinesis stream name.

awsCredentialPluginParam

String

false

" " (empty string)

The JSON parameter to initialize awsCredentialsProviderPlugin.

awsCredentialPluginName

String

false

" " (empty string)

The fully-qualified class name of implementation of {@inject: github:AwsCredentialProviderPlugin:/pulsar-io/aws/src/main/java/org/apache/pulsar/io/aws/AwsCredentialProviderPlugin.java}
awsCredentialProviderPlugin has the following built-in plugs:
org.apache.pulsar.io.kinesis.AwsDefaultProviderChainPlugin: this plugin uses the default AWS provider chain. For more information, see using the default credential provider chain.
org.apache.pulsar.io.kinesis.STSAssumeRoleProviderPlugin: this plugin takes a configuration via the awsCredentialPluginParam that describes a role to assume when running the KCL.
JSON configuration example
{"roleArn": "arn…​", "roleSessionName": "name"}
awsCredentialPluginName is a factory class which creates an AWSCredentialsProvider that is used by Kinesis sink.
If awsCredentialPluginName set to empty, the Kinesis sink creates a default AWSCredentialsProvider which accepts json-map of credentials in awsCredentialPluginParam.