Debugging and cleaning up the real-time data pipeline

This guide is part of a series that creates a real-time data pipeline with Astra Streaming and Decodable. For context and prerequisites, start here.

Debugging the pipeline

If your pipeline didn’t work as expected, there are a few things you can do to debug the issue.

First, find where the pipeline broke by following the data flows described in Follow the flow of the product data clicks.

With Decodable, you can click through the connections, streams, and pipelines and visualize where the connection failed. For example, if you accidentally named a stream click-stream instead of clicks-stream, you can follow the click event to the outbound clicks-stream and then to the pipeline. There, you’ll see that the pipeline isn’t receiving the message because its inbound stream is misnamed click-stream. Then, you can stop the pipeline, fix the inbound stream name, and then restart the pipeline to get data flowing again.

If that doesn’t resolve the issue, you can test input data at the point of failure. In this case, you want to determine if the input data is malformed, or if the object itself is failing. There are two tools you can use to debug this at various stages of the pipleine:

In Decodable, you can use the Preview feature to see a sample of data processing in each pipeline.
In Astra Streaming, each tenant has a Try Me feature where you can simulate producing and consuming messages in specific topics.

Clean up

To clean up the resources created for this tutorial, you need to remove the Decodable and Astra objects you created.

Removing Decodable objects

For each "Connection" click the 3 dots and choose "Stop".
Once "Stopped", click the 3 dots again and choose "Delete". This will remove the object from your account.
Repeat this for each "Connection" and "Pipeline". You can also remove the "Streams" if you prefer.

Removing Astra objects

In the Astra Portal header, click Applications, and then select Streaming.
Find the webstore-clicks tenant, click More, and then select Delete.

This removes the tenant and all associated sinks, topics, messages, and namespaces.
In the Astra Portal header, click Applications, and then select Astra.
Find the webstore-clicks database, click More, and then select Terminate.

This removes the database and all associated keyspaces and tables. This can take a few minutes.

Next step

Now that you’ve created a sample pipeline, try modifying the tutorial to create a pipeline for your own use cases, like fraud detection or removing personally identifiable information (PII).