Kafka

Watch 2 min demo

Play with live tool

Get instant help

Mage

You can install Mage using Docker, pip, or conda.

Setup

Setup and install

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Contributing

ETL pipeline

Load, transform, and export restaurant data

Complete example

Use a completed pipeline

DBT orchestration

Set up DBT models & orchestrate DBT runs

ML training pipeline

Train model on Titanic dataset

Every user experience and technical design decision adheres to these principles.

Core design principles

These are the fundamental concepts that Mage uses to operate.

Core abstractions

Orchestration

Data pipeline management

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

Overview

Write your code for fetching data from a remote source or loading it from disk.

Data loader

Use these blocks to clean, transform, and enhance data from other blocks.

Transformer

Once you’re done transforming your data, write code in these types of blocks to store that data or to train models and store those models elsewhere.

Data exporter

Use these blocks to experiment and write throw away code.

Scratchpad

A sensor is a block that continuously evaluates a condition until it’s met or until a period of time has elapsed.

Sensor

Use the output of a block to dynamically create more blocks.

Dynamic blocks

An extension block adds extra functionality to blocks in your pipeline.

Extension

Extension block

Data integration is synchronizing data from an external system to an internal system or vice versa.

A source is a system that you want to load data from and synchronize it into another system. A source can be a 3rd party API, SaaS, database, data warehouse, or a data lake.

Sources technical documentation

Create a new source

Adapt existing source

PostgreSQL

Change Data Capture (CDC) with PostgreSQL

Destinations technical documentation

Create a new destination

Adapt existing destination

DBT Integration

If you have an existing DBT project, you can add it to your Mage project.

Add existing project

Add an existing DBT project to Mage

Run a model

Run a single model

Run multiple models

Run selected models

Testing

Running DBT tests

Get data from blocks

DBT sources and upstream dependencies

Connection profiles

DBT connection profiles

To serve DBT docs in production, you will need to enable a container to host the DBT docs webserver in the cloud service you are using.

Docs

Serve DBT docs in production

How to set up a folder containing your Mage project code and other files.

Project setup

Batch pipeline

Batch pipeline tutorial

Here are the high level steps to build a data integration pipeline:

Data integration pipeline

How to build a data integration pipeline

Build pipelines that ingest data from event streaming sources like Kafka.

Streaming pipeline

Callbacks

SQL blocks

R blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

Custom

Custom blocks

Use 3rd party Python packages in your project and pipelines.

requirements.txt

Install Python packages

You can import any Python code from any file that is in your Mage project directory.

Custom files

Import your own Python code

Build pipelines that ingest data from event streaming sources like RabbitMQ.

Streaming pipeline (RabbitMQ)

Runtime variables

Pipeline variables and keyword arguments

Every data loader and transformer block has data validation capabilities built-in.

Data validation

Test the data produced by pipeline blocks using Great Expectations.

Great Expectations

Secrets

Use GCP credentials locally when developing.

Setting up Git on Cloud

Mage can automatically keep track of previous versions of a file. Any update to the contents of a file will create a new version that can be used to restore a file to a previous state. This also includes changes to a pipeline.

File versions

File versioning in Mage

Mage is designed and built to be cloud native.

Multi-development environment in the cloud

AWS ECS

AWS Elastic Container Service

Google Cloud Run

Kubernetes

Build your data pipelines in your own text editor without the Mage notebook UI.

Developing w/o UI

Using your own text editor

Troubleshooting

Schedules

Schedule pipelines to run periodically

You can trigger a pipeline by making an API request.

Webhooks

Triggering pipeline via API request

Backfilling pipelines

Guide

Backfill guide

Create users, manage users, and require sign in to authenticate and use Mage.

User authentication

There are 4 development and deployment workflows.

Terraform

Helm

Production Configuration

Secrets management

Resources

Compute resources

Version control your Mage project with Git

Git integration

Logging

You can monitor many metrics for each of your pipelines and blocks.

Monitoring

Datadog

Writing metrics to Datadog

Get status updates in your Slack channel.

Slack

Alerting status updates in Slack

Get status updates in your Teams channel.

Teams

Alerting status updates in Teams

Get status updates sent to your email inbox.

Email

Alerting status updates in Email

This is a guide for using Spark (PySpark) with Mage in different cloud providers or Kubernetes cluster.

Spark

Spark and PySpark

This is a guide for using Databricks Spark cluster with Mage.

Databricks

BigQuery

MySQL

Redshift

Snowflake

Use with Mage

Run Mage pipelines in Airflow

In this tutorial, we’ll create a DAG in Airflow for scheduling and running a data pipeline; all from the Mage UI.

Kafka

Basic config

SSL authentication

SASL authentication

Deserialize message with protobuf schema

Pass raw message to transformer

​Basic config

​SSL authentication

​SASL authentication

​Deserialize message with protobuf schema

​Pass raw message to transformer

Basic config

SSL authentication

SASL authentication

Deserialize message with protobuf schema

Pass raw message to transformer