Overview
Introduction
Each streaming pipeline has three components:
- Source: The data stream source to consume message from.
- Transformer: Write python code to transform the message before writing to destination.
- Sink (destination): The data stream destination to write message to.
Check out this tutorial to set up an example streaming pipeline.
0.8.80 or greater.You can build a streaming pipeline with one source, multiple transformers, and multiple sinks. Here is an example streaming
pipeline:
Supported sources
Supported sinks (destinations)
Test pipeline execution
After finishing configuring the streaming pipeline, you can click the button Execution pipeline to test streaming pipeline execution.
Run pipeline in production
Create the trigger in triggers page to run streaming pipelines in production.
Executor count
If you want to run multiple executors at the same time to scale the streaming pipeline execution, you can set the executor_count variable
in the pipeline’s metadata.yaml file. Here is an example:
blocks:
- ...
- ...
executor_count: 10
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline
Executor type
When running Mage on Kubernetes cluster, you can also configure streaming pipeline to be run on separate k8s pods by setting executor_type
field in the pipeline’s metadata.yaml to k8s.
Example config:
blocks:
- ...
- ...
executor_type: k8s
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline
Contributing guide
Follow this doc to add a new source or destination (sink) to Mage streaming pipeline.