Publish and Consume Events using Python SDK

Pipeline Access Token

To publish and consume events using the SDK, you only need to provide the PIPELINE_ACCESS_TOKEN.

Key Features of Pipeline Access Tokens

• Scope-Limited: These tokens are restricted to a specific pipeline, ensuring better security and access control.
• Default & Custom Tokens: Each pipeline comes with a default pipeline access token, and you can create additional ones as needed.
• Manageability: Tokens can be created and deleted easily, making integration seamless with external systems such as third-party data tools or application code.

The PIPELINE_ACCESS_TOKEN is distinct from the Personal Access Token, which provides full management capabilities for your GlassFlow account. You can get the Pipeline Access Token from the Pipeline detail page on the Web Application or get it from the Pipeline object using the SDK

Publish Events

Send events in json format using the Python SDK to your pipeline. PipelineDataSource module provides a publish method that takes in a json data and publishes it to the pipeline.
📌 Full API reference PipelineDataSource.publish

Consume Events

Retrieve transformed events from your pipeline using the PipelineDataSink module. GlassFlow automatically tracks the last consumed event and ensures you receive only unprocessed events. If there are no new events, the consume method returns an empty response with status_code 204.
📌 Full API reference PipelineDataSink.consume

Consume Failed Events

If an event transformation fails due to issues like corrupted data, schema changes, or transformation bugs, it is placed in a failed queue. You can consume and reprocess these failed events to ensure data integrity.
📌 Full API reference PipelineDataSink.consume_failed

Summary

• Use PipelineDataSource.publish() to send JSON events.
• Retrieve events with PipelineDataSink.consume().
• Access failed events with PipelineDataSink.consume_failed().
• Secure access using PIPELINE_ACCESS_TOKEN.

By leveraging the GlassFlow Python SDK, you can easily integrate event-driven data processing into your applications while maintaining reliability and security.