Configuring the Starlight for Kafka extension

This page describes configuration options for Starlight for Kafka:

Listeners

This section lists configurations for topic listeners.

Name Description

kafkaListeners

Comma-separated list of URIs that we will listen on and the listener names. For example: PLAINTEXT://localhost:9092,SSL://localhost:9093.

Each URI’s scheme represents a listener name if kafkaProtocolMap is configured. Otherwise, the scheme must be a valid protocol in PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL.

If the hostname is not set, it will be bound to the default interface.

kafkaAdvertisedListeners

Listeners to publish to ZooKeeper for clients to use. The format is the same as kafkaListeners.

kafkaProtocolMap

Comma-separated map of listener name and protocol. For example: PRIVATE:PLAINTEXT,PRIVATE_SSL:SSL,PUBLIC:PLAINTEXT,PUBLIC_SSL:SSL.

listeners

Deprecated. kafkaListeners is used.

Among all configurations, only kafkaListeners or listeners (deprecated) is required.

Multiple listeners

To support multiple listeners, specify different listener names in kafkaListeners and kafkaAdvertisedListeners, then map the listener name to the proper protocol in kafkaProtocolMap.

For example, to listen on ports 9092 and 19092 with the PLAINTEXT protocol, the associated names are kafka_internal and kafka_external.

Add the following configurations:

kafkaListeners=kafka_internal://0.0.0.0:9092,kafka_external://0.0.0.0:19092
kafkaProtocolMap=kafka_internal:PLAINTEXT,kafka_external:PLAINTEXT
kafkaAdvertisedListeners=kafka_internal://localhost:9092,kafka_external://localhost:19092

In the above example,

  • kafkaListener is split into multiple tokens by a comma (,), the token is in a format of <listener-name>://<host>:<port>.

  • kafkaProtocolMap is split into multiple tokens by a comma (,), the token is in a format of <listener-name>:<protocol>.

  • kafkaAdvertisedListeners is split into multiple tokens by a comma(,), the token is in a format of <listener-name>:<scheme>://<host>:<port>.

Logger

Starlight for Kafka shares the same configuration files with the Pulsar broker, e.g. conf/broker.conf or conf/standalone.conf. The log configuration is modified in conf/log4j2.yaml file as below:

Logger:
  - name: io.streamnative.pulsar.handlers.kop
    level: warn
    additivity: false
    AppenderRef:
      - ref: Console

Namespace

Pulsar is a multi-tenant system that requires users to specify the tenant and namespace. While most Kafka users just specify the short topic name, Starlight for Kafka provides the following configurations to specify the default namespace.

Name Description Default

kafkaTenant

The default tenant of Kafka topics

public

kafkaNamespace

The default namespace of Kafka topics

default

kafkaMetadataTenant

The tenant used for storing Kafka metadata topics

public

kafkaEnableMultiTenantMetadata

Use the SASL username as kafkaMetadataTenant

true

kafkaMetadataNamespace

The namespace used for storing Kafka metadata topics

kafka

kopSchemaRegistryNamespace

The namespace used for storing Kafka metadata topics

kafka_schemaregistry

kafkaManageSystemNamespaces

Control whether the system should automatically manage the namespaces.

true

kopAllowedNamespaces

The allowed namespace to list topics with a comma separator. e.g. public/default,public/kafka.

If it’s not set or empty, the allowed namespaces will be <kafkaTenant>/<kafkaNamespace>.

Enable kafkaEnableMultiTenantMetadata to use separate tenants when handling system metadata. This enhancement results in fully isolated tenants in the Pulsar cluster, which is not available in pure Kafka where the system metadata is shared among all users.

Performance

This section lists configurations that may affect performance.

Name Description Default

maxReadEntriesNum

The maximum number of entries that are read from the cursor once per time. Increasing this value can make FETCH request read more bytes each time.

Currently, Starlight for Kafka does not check the maximum byte limit. Therefore, if the value is too great, the response size may be over the network limit.

5

Network

This section lists configurations for network communication.

Name Description Default

maxQueuedRequests

Limit the queue size for request, like queued.max.requests in Kafka server.

500

requestTimeoutMs

Limit the timeout in milliseconds for request, like request.timeout.ms in Kafka client. If a request was not processed in the timeout, Starlight for Kafka would return an error response to client.

30000

connectionMaxIdleMs

The idle connection timeout in milliseconds. If the idle connection timeout (such as connections.max.idle.ms used in the Kafka server) is reached, the server handler will close this idle connection.

If it is set to -1, it indicates that the idle connection timeout is disabled.

600000

failedAuthenticationDelayMs

Connection close delay on failed authentication: this is the time (in milliseconds) by which connection close will be delayed on authentication failure, like connection.failed.authentication.delay.ms in Kafka server.

300

brokerLookupTimeoutMs

The timeout for broker lookups (in milliseconds).

30000

These limits are based on each connection.

Prometheus

This section lists configurations for Prometheus metrics.

Name Description Default

kopPrometheusStatsLatencyRolloverSeconds

Starlight for Kafka metrics exposed to prometheus rollover latency in seconds.

60

Group Coordinator

This section lists configurations about the group coordinator and the __consumer_offsets topic that is used to store committed offsets.

Name Description Default

groupMinSessionTimeoutMs

The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection while require more frequent consumer heart beating, which can overwhelm broker resources.

6000

groupMaxSessionTimeoutMs

The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages between heartbeats while require longer time to detect failures.

300000

groupInitialRebalanceDelayMs

The time the group coordinator waits for more consumers to join a new group before performing the first rebalance. A longer delay potentially reduces rebalances, but increases the time until processing begins.

3000

offsetsTopicCompressionCodec

Compression codec for the offsets topic.

NONE

offsetMetadataMaxSize

The maximum size in bytes for a metadata entry associated with an offset commit.

4096

offsetsRetentionMinutes

Offsets older than this retention period are discarded.

4320

offsetsMessageTTL

The offsets message TTL in seconds.

259200

offsetsRetentionCheckIntervalMs

The frequency at which to check for stale offsets.

600000

offsetsTopicNumPartitions

The number of partitions for the offsets topic.

50

Transactions

This section lists configurations for transactions.

Name Description Default

enableTransactionCoordinator

Whether to enable transaction coordinator.

false

brokerId

The broker ID that is used to create the producer ID.

1

txnLogTopicNumPartitions

the number of partitions for the transaction log topic.

50

txnAbortTimedOutTransactionCleanupIntervalMs

The interval in milliseconds at which to rollback transactions that have timed out.

10000

enableTransactionalIdExpiration

Whether to enable transactional ID expiration.

true

transactionalIdExpirationMs

The time (in ms) that the transaction coordinator waits without receiving any transaction status updates for the current transaction before expiring its transactional ID.

604800

transactionsRemoveExpiredTransactionalIdCleanupIntervalMs

The interval (in ms) at which to remove expired transactions.

3600

SSL encryption

This section lists configurations for SSL encryption.

Name Description Default

kopSslProtocol

Kafka SSL configuration map with: SSL_PROTOCOL_CONFIG = ssl.protocol

TLS

kopSslProvider

Kafka SSL configuration map with: SSL_PROVIDER_CONFIG = ssl.provider

kopSslCipherSuites

Kafka SSL configuration map with: SSL_CIPHER_SUITES_CONFIG = ssl.cipher.suites

kopSslEnabledProtocols

Kafka SSL configuration map with: SSL_ENABLED_PROTOCOLS_CONFIG = ssl.enabled.protocols

TLSv1.2, TLSv1.1, TLSv1

kopSslKeystoreType

Kafka SSL configuration map with: SSL_KEYSTORE_TYPE_CONFIG = ssl.keystore.type

JKS

kopSslKeystoreLocation

Kafka SSL configuration map with: SSL_KEYSTORE_LOCATION_CONFIG = ssl.keystore.location

kopSslKeystorePassword

Kafka SSL configuration map with: SSL_TRUSTSTORE_PASSWORD_CONFIG = ssl.truststore.password

N/A

kopSslTruststoreType

Kafka SSL configuration map with: SSL_KEYSTORE_TYPE_CONFIG = ssl.keystore.type

JKS

kopSslTruststoreLocation

Kafka SSL configuration map with: SSL_TRUSTSTORE_LOCATION_CONFIG = ssl.truststore.location

kopSslTruststorePassword

Kafka SSL configuration map with: SSL_TRUSTSTORE_PASSWORD_CONFIG = ssl.truststore.password

kopSslKeymanagerAlgorithm

Kafka SSL configuration map with: SSL_KEYMANAGER_ALGORITHM_CONFIG = ssl.keymanager.algorithm

SunX509

kopSslTrustmanagerAlgorithm

Kafka SSL configuration map with: SSL_TRUSTMANAGER_ALGORITHM_CONFIG = ssl.trustmanager.algorithm

SunX509

kopSslSecureRandomImplementation

Kafka SSL configuration map with: SSL_SECURE_RANDOM_IMPLEMENTATION_CONFIG = ssl.secure.random.implementation

What’s next?

Configure and manage Starlight for Kafka based on your requirements. Check the following guides for more details.

Was this helpful?

Edit this page

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com