Debugging and cleaning up the real-time data pipeline

This guide is part of a series that creates a real-time data pipeline with Astra and Decodable. For context and prerequisites, start here.

Debugging the pipeline

Maybe things didn’t go so smoothly for you. Or maybe you’re just having a bad day :(. Whatever the case, we can fix this!

Follow the data flows described in Step 3 to find where in the pipeline things are broken. Decodable is a great visualization tool for this, because you can click through the connections, streams, and pipelines and actually see where connections are failing. For example, if you accidentally named a stream "click-stream" instead of "clicks-stream", you can follow the click event to the outbound "clicks-stream" and then to the pipeline, and see that the pipeline is not receiving the message because its inbound stream is misnamed "click-stream". Stop the pipeline, choose the correct stream, and restart the pipeline to get data flowing again.

Test input data at the point of failure. The first question to ask is if the input data is malformed, or if the object itself is failing. Decodable’s UI gives you the ability to “Preview” the processing in each pipeline. This is a very powerful debugging tool. Additionally, your Astra Tenant has the “Try Me” area where you can produce and consume messages in specific topics, which is great for testing purposes. These tools can help you recreate each step of the pipeline and debug issues.

Clean up

Removing Decodable objects

  1. For each "Connection" click the 3 dots and choose "Stop".

    Stop decodable connection

  2. Once "Stopped", click the 3 dots again and choose "Delete". This will remove the object from your account.

    Delete decodable connection

  3. Repeat this for each "Connection" and "Pipeline". You can also remove the "Streams" if you prefer.

Removing Astra objects

  1. From the Astra portal home navigate to the "Streaming" area to see a list of your tenants.

  2. Click the 3 dots associated with "webstore-clicks" tenant and choose the "Delete". This will remove all sinks, topics, messages, and namespaces.

    Delete Astra tenant

  3. Navigate to the "Databases" area and click the 3 dots associated with "webstore-clicks". Choose "Terminate" to remove all traces of the database (it might take a few seconds).

    Terminate Astra database

You have now removed all objects and are ready to start fresh! Your next processing pipeline can solve a business challenge like fraud detection, or removing personally identifiable information (PII), or maybe how to get to the moon! Have fun!

Next step

Return to the guide home.

Was this helpful?

Give Feedback

How can we improve the documentation?

© 2024 DataStax | Privacy policy | Terms of use

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Pulsar, Pulsar, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries. Kubernetes is the registered trademark of the Linux Foundation.

General Inquiries: +1 (650) 389-6000,