Getting started with Astra Streaming
Astra Streaming is a cloud native data streaming and event stream processing service tightly integrated into the Astra Portal and powered by Apache Pulsar™. Using Astra Streaming, customers can quickly create Pulsar instances, manage their clusters, scale across cloud regions, and manage Pulsar resources such as topics, connectors, functions and subscriptions.
Follow this guide to create a new account in DataStax Astra, along with a Streaming Tenant running Apache Pulsar.
Prerequisites
You will need the following to complete this guide:
-
A working email address
Create your Astra account
-
Navigate to the Astra portal sign in page.
-
Click the "Sign Up" link (toward the bottom).
-
Provide your information along with a valid email.
-
You should receive an email from DataStax verifying your address - click the button within.
-
If you aren’t automatically signed in, visit the Astra sign in page and use your new account creds.
Your first streaming tenant
Think of a tenant in Astra Streaming as your safe space to work. It is a portion of DataStax’s managed Apache Pulsar that is only yours. Learn more about the concept of tenancy in the Pulsar docs.
The steps in the below tabs will guide you through creating a Streaming Tenant. You’ll use this tenant to create namespaces, topics, functions, and pretty much everything else. The only difference between the tabs is how your tenant is created - they all have the same result.
-
Astra Portal
-
Astra CLI
-
Once signed in, click the "Create a Stream" button on the portal home page.
-
Name your Streaming Tenant something memorable like "my-stream-<rand>". Replace
<rand>with random letters or numbers - all tenant names in Astra Streaming must be unique. Choose your preferred cloud provider and region. For your first example tenant, the region doesn’t really matter (it’s all free).
-
Click the "Create Tenant" button.
-
You will be directed to the quickstart page for your new tenant.
Astra CLI is a set of commands for creating and managing all Astra resources. For more, see the documentation.
-
Set the required variables, replacing <rand> with a few random numbers or letters. Optionally you can choose a different cloud provider and region. Use
astra streaming list-regionsto see values.TENANT="my-stream-<rand>" CLOUD_PROVIDER_STREAMING="gcp" CLOUD_REGION_STREAMING="uscentral-1" -
Run the following script to create a new streaming tenant.
# Create a tenant within the "astra-streaming-examples" astra org astra streaming create \ --config "astra-streaming-examples" \ --cloud "$CLOUD_PROVIDER_STREAMING" \ --region "$CLOUD_REGION_STREAMING" \ "$TENANT" -
Within the newly created tenant, you’re ready to create a namespace.
A namespace to hold topics
A namespace exists within a tenant. A namespace is a logical grouping of message topics. Tenants usually have many namespaces. What is contained within namespace is limited only by your imagination. It could be an environment (dev, stage, prod) or by application (catalog, cart, user) or whatever logical grouping makes sense to you. Learn more about namespaces in the Pulsar docs.
|
Astra Streaming automatically created a namespace named "default" when you created your tenant. If you would like to use the default namespace instead of creating a new namespace, that’s perfectly fine. |
-
Astra Portal
-
Pulsar Admin
-
Curl
-
Navigate to the "Namespace And Topics" tab.
-
Click the "Create Namespace" button, and give your namespace a super original name like "my-namespace".
-
Click "Create" to create the namespace.
Not sure about this? Learn more about configuring your local environment for Astra Streaming.
-
Set the required variables.
NAMESPACE="my-namespace" # TENANT="my-stream-<rand>" # set previously -
Run the following script to create a new namespace.
# Create a new namespace in an existing pulsar tenant ./bin/pulsar-admin namespaces create "$TENANT/$NAMESPACE"
Not sure about this? Learn more about interacting with Astra Streaming through curl here.
-
Set the required variables.
PULSAR_TOKEN="<REPLACE-ME-WITH-TOKEN>" WEB_SERVICE_URL="<REPLACE-ME-WITH-URL>" NAMESPACE="my-namespace" # TENANT="my-stream-<rand>" # set previously -
Run the following script to create a new namespace.
# Create a new namespace in an existing pulsar tenant curl -sS --fail --location \ --request PUT \ --header "Authorization: Bearer $PULSAR_TOKEN" \ "$WEB_SERVICE_URL/admin/v3/namespaces/$TENANT/$NAMESPACE"
A topic to organize messages
Topics are the core construct of a messaging system. Topics provide a way to group messages matching certain criteria. The name of the topic usually loosely defines the criteria, and more advanced characteristics like schemas might be applied at the topic level as well. Topics are where others can "listen" for new messages. Consumers subscribe to topics to "listen" for messages, and functions and connectors can also "listen" for messages and automate workflows. In Pulsar, topic addresses look like a URL (ie: persistent://tenant/namespace/topic). Learn more about topics in the Pulsar docs.
-
Astra Portal
-
Pulsar Admin
-
Curl
-
In the Namespace And Topics tab, locate the namespace created above and click its Add Topic button.
-
Provide a name for the topic like "my-topic". It must start with a lowercase letter, be alphanumeric, and can contain hyphens.
-
Leave the choice of persistence and partitioning alone for now - those can be a part of future learning.
-
Click the Add Topic button to create the topic within the namespace.
Not sure about this? Learn more about connecting to Astra Streaming with the pulsar-admin CLI here.
-
Set the required variables.
TOPIC="my-topic" # NAMESPACE="my-namespace" # set previously # TENANT="my-stream-<rand>" # set previously -
Run the following script to create a new topic.
# Create a new persistent topic in an existing pulsar tenant/namespace ./bin/pulsar-admin topics create "persistent://$TENANT/$NAMESPACE/$TOPIC" # Or create a non-persistent topic #./bin/pulsar-admin topics create "non-persistent://$TENANT/$NAMESPACE/$TOPIC"
Not sure about this? Learn more about interacting with Astra Streaming through curl here.
-
Set the required variables.
TOPIC="my-topic" # PULSAR_TOKEN="<REPLACE-ME-WITH-TOKEN>" # set previously # WEB_SERVICE_URL="<REPLACE-ME-WITH-URL>" # set previously # NAMESPACE="my-namespace" # set previously # TENANT="my-stream-<rand>" # set previously -
Run the following script to create a new topic.
# Create a new topic in an existing pulsar tenant curl -sS --fail --location \ --request PUT \ --header "Authorization: Bearer $PULSAR_TOKEN" \ "$WEB_SERVICE_URL/admin/v3/topics/$TENANT/$NAMESPACE/$TOPIC"
Putting your topic to work
Now that you have a topic created, it’s time to produce and consume messages in your new message topic. There are several different ways to accomplish this. Choose the right one for your needs (they all end up doing the same thing).
Astra Portal |
Pulsar Client |
Runtime Clients |
