Troubleshooting

To troubleshoot Luna Streaming, look at these potential issues:

Insufficient Quota
Containers in Pending State
Using the Bastion Pod

Check quotas

In some cases, pods can become unhealthy. The root cause might be an insufficient quota. You can check quotas in the cloud provider’s user interface.
For example, in the Google Cloud (GCP) console, check for any unhealthy pods in your GKE project. Then in the IAM & Admin section of the GCP console, navigate to Quotas. Look for any reported issues with backend services:

From the GCP Console Quotas display:

Select the row for the service name that is reporting a quota issue.
Select All Quotas from the Details column.
Check the box for the affected quota, and select Edit Quota.

The dialog indicates: "Enter a new quota limit. Your request will be sent to your service provider for approval."
Examine the displayed current value and set a new value.
Enter a brief request description, and select Next.
Verify your contact information, and select Submit Request.

In the following example, the Backend services quota is set to '5'. We’re changing it to '50'. For the K8ssandra deployments, such as Stargate, cass-operator, Reaper, and Medusa, 10 might be a sufficient quota.

Containers in Pending state

If your pods are stuck in a Pending state after installation or your cloud provider is warning you about Unschedulable Pods, there are a few ways to work through this:

If some of your pods start, but others like pulsar-adminconsole and pulsar-grafana are left in an Unschedulable state, you might need to add CPUs to your existing nodes or an additional node pool. Luna Streaming requires more resources than Apache Pulsar.
To examine a specific pod, use kubectl describe. For example, if your pulsar-bookkeeper-0 pod is not scheduling, use kubectl describe pods/pulsar-bookkeeper-0 to view detailed output on the pod’s state, dependencies, and events.
To view log output for a specific pod, use kubectl logs. Continuing with the pulsar-bookkeeper-0 example, to get logs for this pod use kubectl logs pulsar-bookkeeper-0. If this pod spans multiple containers, add the flag --all-containers=true.

Bastion pod

The bastion pod deploys with Luna Streaming and allows a user to execute administrative commands in a bash shell of the cluster. This can be a helpful troubleshooting tool for testing the health of your cluster.

Access bastion pod

To use the bastion pod after installation, access the shell of the bastion pod. Let’s use the namespace default:

kubectl -n default exec $(kubectl get pods --namespace default -l "app=pulsar,component=bastion" -o jsonpath="{.items[0].metadata.name}") -it -- bash

Your bash shell will then show that you are in your bastion pod:

pulsar@pulsar-bastion-78d4cd74ff-ctvvh:/pulsar$

Test bastion pod

Ensure your bastion pod sees your cluster with pulsar admin namespaces list public. It should return a list of your public namespaces:

"public/default"
"public/functions"

If you get bash: admin: command not found, move into the bin directory with cd bin. You might need to append your command with ./, for example ./pulsar-admin get pods. If the shell finds no resources, you might not have any public namespaces. Create a public namespace with pulsar-admin namespaces create public/default.

Publish a message

To test your Pulsar cluster with the bastion pod, produce a message with pulsar-client through the bastion pod shell:

pulsar-client produce my-test-topic --messages "hello-pulsar"

You should receive a confirmation the message was produced:

00:16:37.970 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced

This means your Pulsar cluster is functional. If the message isn’t produced, double-check your message syntax.

What’s next?

To view your cluster, brokers, and connectors in a user interface, see the Admin Console Tutorial.