1. Streaming pipelines
  2. Overview

Introduction

Each streaming pipeline has three components:

  • Source: The data stream source to consume message from.
  • Transformer: Write python code to transform the message before writing to destination.
  • Sink (destination): The data stream destination to write message to.

Check out this tutorial to set up an example streaming pipeline.

Supported sources

Supported sinks (destinations)

Test pipeline execution

After finishing configuring the streaming pipeline, you can click the button Execution pipeline to test streaming pipeline execution.

Contributing guide

Follow this doc to add a new source or destination (sink) to Mage streaming pipeline.