- Streaming pipelines
Each streaming pipeline has three components:
- Source: The data stream source to consume message from.
- Transformer: Write python code to transform the message before writing to destination.
- Sink (destination): The data stream destination to write message to.
Check out this tutorial to set up an example streaming pipeline.
Supported sinks (destinations)
Test pipeline execution
After finishing configuring the streaming pipeline, you can click the button
Execution pipeline to test streaming pipeline execution.
Follow this doc to add a new source or destination (sink) to Mage streaming pipeline.