- Streaming pipelines
- Overview
Streaming pipelines
Overview
Introduction
Each streaming pipeline has three components:
- Source: The data stream source to consume message from.
- Transformer: Write python code to transform the message before writing to destination.
- Sink (destination): The data stream destination to write message to.
Check out this tutorial to set up an example streaming pipeline.
Supported sources
Supported sinks (destinations)
Test pipeline execution
After finishing configuring the streaming pipeline, you can click the button Execution pipeline
to test streaming pipeline execution.
Contributing guide
Follow this doc to add a new source or destination (sink) to Mage streaming pipeline.