Starlight for Kafka Schema Registry

Starlight for Kafka implements a schema registry that is compatible with the Confluent Schema Registry® and the Apicurio Schema Registry, the most commonly used schema registries in the Kafka ecosystem.

The Kafka Schema Registry stores schemas in the $<tenant>/kafka/schema-registry topic. It is automatically created as a non-partitioned topic with infinite retention.

To avoid losing data, it is very important to disable retention on the __kafka namespace, or set it only for that topic if you have “topic level policies” enabled in your Pulsar system.

The Starlight for Kafka adapter creates automatically the $<tenant>/kafkaschemaregistry namespace and sets infinite retention for you.

You can use kopSchemaRegistryNamespace configuration parameter to set the name of the namespace to be used.

Starlight for Kafka Schema Registry Implementation

  1. To enable the Schema Registry, add this configuration in conf/broker.conf:

    kopSchemaRegistryEnable=true
    kopSchemaRegistryProxyPort=8001
  2. If you’re using the Kafka proxy, add this configuration in conf/proxy.conf:

    kopSchemaRegistryEnable=true
    kopSchemaRegistryProxyPort=8081
  3. Verify that the Schema Registry is working properly by connecting to http://pulsar-broker:8001/config or http://pulsar-proxy:8081/config (if you’re using the proxy). You should see something like this:

    {
      "compatibilityLevel" : "NONE"
    }

The Proxy works only if the broker is working properly, because it simply forwards the requests to a broker. If the broker fails or it is not accessible, the Proxy will answer with an error message.

Enable TLS on schema registry

  1. To enable TLS on the schema registry, you must use the Kafka proxy.
    To set the SchemaRegistry endpoint on the broker to use TLS, add this configuration in conf/broker.conf:

    kopSchemaRegistryEnableTls=true
    kopSchemaRegistryProxyEnableTls=true
  2. To set the SchemaRegistry endpoint on the proxy to use TLS, add this configuration in conf/proxy.conf:

    kopSchemaRegistryEnableTls=true
    kopSchemaRegistryProxyEnableTls=true
  3. TLS configuration is then picked up automatically by the Pulsar proxy configuration.

Authentication for the schema registry

If authentication is enabled, you can only use the schema registry with HTTP Basic Authentication:

Username: the name of the “tenant” to connect to
Password: token:xxxxxxxx (the JWT token used to connect to Pulsar/Kafka)

If you are using the Confluent Java client or the Kafka Tools, set these properties:

basic.auth.credentials.source=USER_INFO
basic.auth.user.info=TENANTNAME:token:xxxxxxxx
schema.registry.url=http://pulsar-proxy:8081

What’s next?

You can configure and manage Starlight for Kafka based on your requirements. Check the following guides for more details.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000, info@datastax.com