Deploying Pipelines using GlassFlow Python SDK
GlassFlow provides a powerful Python SDK that enables users to build, manage, and interact with data pipelines entirely in code.
GlassFlow provides a powerful Python SDK that enables users to build, manage, and interact with data pipelines entirely in code. With the GlassFlow SDK, you can:
- Define and deploy pipelines programmatically
- Manage pipeline updates
- Create and Retrieve access tokens
- Send and receive data from pipelines seamlessly
Installation
To install the GlassFlow Python SDK, use pip:
For more details, visit the GlassFlow SDK documentation on PyPI.
Note:
GlassFlow Python SDK authenticates itself to your GlassFlow account using a Personal Access Token. You can find your token at https://app.glassflow.dev/profile
Getting Started
1. Define Your Pipeline Transformation Code
Create a transform.py file with your transformantion code.
A basic echo example can be like:
If your pipeline transformation code needs external python dependencies, you can add them on a requirements.txt file.
Example:
Save the transform.py and requirements.txt locally to use them in the next step.
2. Setup the pipeline using GlassFlow SDK
Create a python script called create_pipeline.py with the following code. This creates an example space on GlassFlow and
creates a echo-pipeline.
Run the above script in a python environment which has glassflow python pacakage installed. This creates a pipeline on GlassFlow which is immediately deployed and ready to receive events. You can go to GlassFlow WebApp to see your pipeline, view logs and other pipeline parameters. You can also make changes to the pipeline directly on the Web UI.
Note:
The above example only shows a basic echo pipeline. You can build more advanced pipelines with custom requirements.txt, environment variables and managed connectors. See glassflow-examples github repo to see examples of more advanced pipelines.
Managing Pipelines
The SDK provides functionalities to:
- Start, stop, and monitor pipelines
- Modify and update pipeline configurations
- Retrieve execution logs and metrics
For detailed examples, check our GitHub repository. For detailed documentation on Python SDK, visit the GlassFlow SDK documentation on PyPI.
Sending and Receiving Data
GlassFlow SDK allows seamless data exchange with pipelines:
For more in-depth use cases, refer to the Jupyter notebook examples on GitHub.