Astra Streaming limits
Astra Streaming includes guardrails and limits to ensure best practices, foster availability, and promote optimal configurations for your databases.
Unless otherwise noted, the following limits apply to shared clusters. In general, Streaming Capacity Units have more relaxed limits than shared clusters, and some limits can be changed by contacting DataStax Support.
Astra Streaming guardrails
Guardrails are provisioned in the default settings for Astra Streaming. You can’t directly change these guardrails. For Streaming Capacity Units, some guardrails can be changed by contacting DataStax Support.
| Guardrail | Limit | Comments |
|---|---|---|
Number of tenants per organization |
|
This limit can’t be changed for shared clusters. |
Number of namespaces per tenant |
All clusters: 20 |
|
Number of topics per namespace |
|
|
Max message size |
|
|
Max throughput per topic for consumers |
|
This limit applies to non-partitioned topics and individual partitions of partitioned topics. |
Max throughput per topic for producers |
|
This limit applies to non-partitioned topics and individual partitions of partitioned topics, regardless of the number of producers to a topic. If there are multiple producers to a topic, they can produce a combined maximum of 5000 messages per second. |
Max throughput per topic for subscriptions |
|
This limit applies to non-partitioned topics and individual partitions of partitioned topics, regardless of the number of subscribers to a topic. If there are multiple subscriptions to a topic, they can dispatch a combined maximum of 5000 messages per second. |
Auto topic creation |
All clusters: Disabled |
|
Max number of partitions per topic |
All clusters: 128 |
|
Number of producers per topic |
|
|
Number of consumers per topic |
|
|
Number of subscriptions per topic |
|
|
Max unacknowledged messages per consumer |
All clusters: 50000 |
|
Max unacknowledged messages per subscription |
All clusters: 200000 |
|
Subscription expiry |
All clusters: Not set |
Astra Streaming namespace policies
These namespace policies are initially provisioned in the default settings for Astra Streaming, and you can change them:
| Policy | Default Setting | Available in the Astra Portal |
|---|---|---|
Max retention time |
Disabled/no data retention by default |
Yes |
Max retention size |
Disabled/no data retention by default |
Yes |
Max TTL |
Infinite, minimum setting is 2 days |
Yes |
Backlog size |
5GB, minimum setting is -1, which disables backlog size quota enforcement |
Yes |
Backlog Quota Retention Policy |
Producer Exception |
Yes |
Unload topic |
Enabled |
Yes |
Message deduplicate |
Disabled |
No |
Subscription auto creation |
Enabled |
No |
Subscription expiration per namespace |
Default policy: |
No |
Inactive topic policy |
Default policy: |
No |
Character limit
The total characters of tenant name, namespace name, and function name cannot exceed 54 characters. This is a Kubernetes restriction based on a pod label’s maximum size of 63 characters. You can read more about Kubernetes pod naming restrictions here.
Namespace policy configuration file
The following is an example of a default namespace policy with limits set.
Default namespace policy with limits set
{
"auth_policies" : {
"namespace_auth" : {
"client;{client key}" : [ "consume", "produce" ],
"websocket" : [ "consume", "produce" ]
},
"destination_auth" : { },
"subscription_auth_roles" : { }
},
"replication_clusters" : [ "pulsar-gcp-europewest1" ],
"bundles" : {
"boundaries" : [ "0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 4
},
"backlog_quota_map" : {
"destination_storage" : {
"limit" : 1000000000,
"policy" : "producer_exception"
}
},
"clusterDispatchRate" : { },
"topicDispatchRate" : {
"pulsar-gcp-europewest1" : {
"dispatchThrottlingRateInMsg" : 5000,
"dispatchThrottlingRateInByte" : 25000000,
"relativeToPublishRate" : false,
"ratePeriodInSecond" : 1
}
},
"subscriptionDispatchRate" : {
"pulsar-gcp-europewest1" : {
"dispatchThrottlingRateInMsg" : 5000,
"dispatchThrottlingRateInByte" : 25000000,
"relativeToPublishRate" : false,
"ratePeriodInSecond" : 1
}
},
"replicatorDispatchRate" : { },
"clusterSubscribeRate" : {
"pulsar-gcp-europewest1" : {
"subscribeThrottlingRatePerConsumer" : 5000,
"ratePeriodInSecond" : 30
}
},
"persistence" : {
"bookkeeperEnsemble" : 2,
"bookkeeperWriteQuorum" : 2,
"bookkeeperAckQuorum" : 2,
"managedLedgerMaxMarkDeleteRate" : 0.0
},
"deduplicationEnabled" : false,
"autoTopicCreationOverride" : {
"allowAutoTopicCreation" : false,
"topicType" : "",
"defaultNumPartitions" : 0
},
"publishMaxMessageRate" : {
"pulsar-gcp-europewest1" : {
"publishThrottlingRateInMsg" : 1000,
"publishThrottlingRateInByte" : 5000000
}
},
"latency_stats_sample_rate" : { },
"message_ttl_in_seconds" : 0,
"subscription_expiration_time_minutes" : 0,
"retention_policies" : {
"retentionTimeInMinutes" : 2880,
"retentionSizeInMB" : -1
},
"deleted" : false,
"encryption_required" : false,
"inactive_topic_policies" : {
"inactiveTopicDeleteMode" : "delete_when_no_subscriptions",
"maxInactiveDurationSeconds" : 86400,
"deleteWhileInactive" : true
},
"subscription_auth_mode" : "None",
"max_producers_per_topic" : 50,
"max_consumers_per_topic" : 50,
"max_consumers_per_subscription" : 50,
"max_unacked_messages_per_consumer" : 50000,
"max_unacked_messages_per_subscription" : 200000,
"compaction_threshold" : 0,
"offload_threshold" : -1,
"offload_deletion_lag_ms" : 0,
"schema_auto_update_compatibility_strategy" : "Full",
"schema_compatibility_strategy" : "UNDEFINED",
"is_allow_auto_update_schema" : true,
"schema_validation_enforced" : false
}Astra Streaming topic and namespace actions
These topic and namespace actions are provisioned in the default settings by Astra Streaming. You can perform these actions in the Astra Portal or programmatically with a sufficiently authorized role or authentication token.
| Allowed Action | Default Setting | Available in the Astra Portal |
|---|---|---|
Terminate topic |
Enabled |
No |
Unload namespace |
Enabled |
No |
Clear backlog at topic level |
Enabled |
No |
Clear backlog at namespace level |
Enabled |
No |
Set compaction threshold at namespace level |
Disabled |
No |
Trigger compaction at topic level |
Enabled |
No |
Topic compaction |
Enabled |
No |
All subscription expiration |
Disabled |
No |
Function and connector resources
A function instance is a unit used for scaling Apache Pulsar™ functions or Pulsar IO connectors running on Astra Streaming. Each function instance is assigned a specific amount of CPU and memory which can be allocated to a Pulsar function. Each Pulsar function running on Astra Streaming has a minimum of one function instance allocated to it.
Functions and connector resources for function instances are set in the control plane based as resource limits. You can’t customize these resources.
The default settings are as follows:
| Cluster type | CPU per instance | RAM per instance | Maximum function instances per namespace |
|---|---|---|---|
Shared clusters |
0.25 core |
500MB |
Varies by plan |
Streaming Capacity Units (dedicated clusters) |
0.50 core |
1GB |
5000 |
Immutable configurations
The following configurations can’t be changed:
-
Data persistency (
En,Qw,Qa) -
Managedledgerpolicy/deletion -
Namespace bundle configurations:
-
Bundle split
-
Bundle level clear backlog
-
Bundle level unload
-
Bundle level subscribe and unsubscribe
-
-
Replication
-
Delayed delivery
-
Offload policy
-
Offload deletion
Unsupported pulsar-admin commands
Astra Streaming doesn’t support the following pulsar-admin commands because they aren’t applicable in a cloud environment or they could cause privacy or data integrity issues:
-
bookies -
brokers -
broker-stats -
clusters -
ns-isolation-policies -
resource-quotas -
tenants
