Elasticsearch sink connector

Name

Type

Required

Default

Description

apiKey

String

false

" " (empty string)

The apiKey used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured.

bulkActions

Integer

false

1000

The maximum number of actions per elasticsearch bulk request. Use -1 to disable it.

bulkConcurrentRequests

Integer

false

0

The maximum number of in flight elasticsearch bulk requests. The default 0 allows the execution of a single request. A value of 1 means 1 concurrent request is allowed to be executed while accumulating new bulk requests.

bulkEnabled

Boolean

false

false

Enable the elasticsearch bulk processor to flush write requests based on the number or size of requests, or after a given period.

bulkFlushIntervalInMs

Long

false

1000

The maximum period of time to wait for flushing pending writes when bulk writes are enabled. -1 or zero means the scheduled flushing is disabled.

bulkSizeInMb

Integer

false

5

The maximum size in megabytes of elasticsearch bulk requests. Use -1 to disable it.

canonicalKeyFields

Boolean

false

false

Whether to sort the key fields for JSON and Avro or not. If it is set to true and the record key schema is JSON or AVRO, the serialized object does not consider the order of properties.

compatibilityMode

enum (AUTO,ELASTICSEARCH,ELASTICSEARCH_7,OPENSEARCH)

false

AUTO

Specify compatibility mode with the ElasticSearch cluster. AUTO value will try to auto detect the correct compatibility mode to use. Use ELASTICSEARCH_7 if the target cluster is running ElasticSearch 7 or prior. Use ELASTICSEARCH if the target cluster is running ElasticSearch 8 or higher. Use OPENSEARCH if the target cluster is running OpenSearch.

compressionEnabled

Boolean

false

false

Enable elasticsearch request compression.

connectionIdleTimeoutInMs

Integer

false

5

Idle connection timeout to prevent a read timeout.

connectionRequestTimeoutInMs

Integer

false

1000

The time in milliseconds for getting a connection from the elasticsearch connection pool.

connectTimeoutInMs

Integer

false

5000

The elasticsearch client connection timeout in milliseconds.

createIndexIfNeeded

Boolean

false

false

Manage index if missing.

elasticSearchUrl

String

true

" " (empty string)

The URL of elastic search cluster to which the connector connects.

idHashingAlgorithm

enum(NONE,SHA256,SHA512)

false

NONE

Hashing algorithm to use for the document id. This is useful in order to be compliant with the ElasticSearch _id hard limit of 512 bytes.

indexName

String

false

" " (empty string)

The index name to which the connector writes messages. The default value is the topic name. It accepts date formats in the name to support event time based index with the pattern %{+<date-format>}. For example, suppose the event time of the record is 1645182000000L, the indexName is logs-%{+yyyy-MM-dd}, then the formatted index name would be logs-2022-02-18.

indexNumberOfReplicas

int

false

1

The number of replicas of the index.

indexNumberOfShards

int

false

1

The number of shards of the index.

keyIgnore

Boolean

false

true

Whether to ignore the record key to build the Elasticsearch document _id. If primaryFields is defined, the connector extract the primary fields from the payload to build the document _id If no primaryFields are provided, elasticsearch auto generates a random document _id.

malformedDocAction

enum (IGNORE,WARN,FAIL)

false

FAIL

How to handle elasticsearch rejected documents due to some malformation. Possible options are IGNORE, DELETE or FAIL. Default is FAIL the Elasticsearch document.

maxRetries

Integer

false

1

The maximum number of retries for elasticsearch requests. Use -1 to disable it.

maxRetryTimeInSec

Integer

false

86400

The maximum retry time interval in seconds for retrying an elasticsearch request.

nullValueAction

enum (IGNORE,DELETE,FAIL)

false

IGNORE

How to handle records with null values, possible options are IGNORE, DELETE or FAIL. Default is IGNORE the message.

password

String

false

" " (empty string)

The password used by the connector to connect to the elastic search cluster. <br /><br />If username is set, then password should also be provided.

primaryFields

String

false

"id"

The comma separated ordered list of field names used to build the Elasticsearch document _id from the record value. If this list is a singleton, the field is converted as a string. If this list has 2 or more fields, the generated _id is a string representation of a JSON array of the field values.

retryBackoffInMs

Integer

false

100

The base time to wait when retrying an Elasticsearch request (in milliseconds).

schemaEnable

Boolean

false

false

Turn on the Schema Aware mode.

socketTimeoutInMs

Integer

false

60000

The socket timeout in milliseconds waiting to read the elasticsearch response.

ssl

ElasticSearchSslConfig

false

string

Configuration for TLS encrypted communication. See ElasticSearchSslConfig structure.

stripNonPrintableCharacters

Boolean

false

true

Whether to remove all non-printable characters from the document or not. If it is set to true, all non-printable characters are removed from the document.

stripNulls

Boolean

false

true

If stripNulls is false, elasticsearch _source includes 'null' for empty fields (for example {"foo": null}), otherwise null fields are stripped.

token

String

false

" " (empty string)

The token used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured.

typeName

String

false

"_doc"

The type name to which the connector writes messages to. The value should be set explicitly to a valid type name other than "_doc" for Elasticsearch version before 6.2, and left to default otherwise.

username

String

false

" " (empty string)

The username used by the connector to connect to the elastic search cluster. If username is set, then password should also be provided.