Slack

Watch 2 min demo

Play with live tool

Get instant help

Mage

You can install Mage using Docker, pip, or conda.

Setup

Setup and install

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Contributing

ETL pipeline

Load, transform, and export restaurant data

Complete example

Use a completed pipeline

DBT orchestration

Set up DBT models & orchestrate DBT runs

ML training pipeline

Train model on Titanic dataset

Every user experience and technical design decision adheres to these principles.

Core design principles

These are the fundamental concepts that Mage uses to operate.

Core abstractions

Orchestration

Data pipeline management

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

Overview

Write your code for fetching data from a remote source or loading it from disk.

Data loader

Use these blocks to clean, transform, and enhance data from other blocks.

Transformer

Once you’re done transforming your data, write code in these types of blocks to store that data or to train models and store those models elsewhere.

Data exporter

Use these blocks to experiment and write throw away code.

Scratchpad

A sensor is a block that continuously evaluates a condition until it’s met or until a period of time has elapsed.

Sensor

Use the output of a block to dynamically create more blocks.

Dynamic blocks

Data integration is synchronizing data from an external system to an internal system or vice versa.

A source is a system that you want to load data from and synchronize it into another system. A source can be a 3rd party API, SaaS, database, data warehouse, or a data lake.

Sources technical documentation

Create a new source

Adapt existing source

PostgreSQL

Change Data Capture (CDC) with PostgreSQL

Destinations technical documentation

Create a new destination

Adapt existing destination

DBT Integration

If you have an existing DBT project, you can add it to your Mage project.

Add existing project

Add an existing DBT project to Mage

Run a model

Run a single model

Run multiple models

Run selected models

Testing

Running DBT tests

Get data from blocks

DBT sources and upstream dependencies

Connection profiles

DBT connection profiles

To serve DBT docs in production, you will need to enable a container to host the DBT docs webserver in the cloud service you are using.

Docs

Serve DBT docs in production

Batch pipeline

Batch pipeline tutorial

Here are the high level steps to build a data integration pipeline:

Data integration pipeline

How to build a data integration pipeline

Build pipelines that ingest data from event streaming sources like Kafka.

Streaming pipeline

SQL blocks

R blocks

Runtime variables

Pipeline variables and keyword arguments

Every data loader and transformer block has data validation capabilities built-in.

Data validation

Use GCP credentials locally when developing.

Version control

Setting up Git on Cloud

Mage is designed and built to be cloud native.

Multi-development environment in the cloud

AWS ECS

AWS Elastic Container Service

Google Cloud Run

Kubernetes

Troubleshooting

Schedules

Schedule pipelines to run periodically

An event-type trigger will instruct the pipeline to run whenever a specific event occurs.

Event triggers

Trigger pipeline from event

You can trigger a pipeline by making an API request.

Webhooks

Triggering pipeline via API request

Backfilling pipelines

Guide

Backfill guide

There are 4 development and deployment workflows.

Terraform

Secrets

Secrets management

Resources

Compute resources

Databases

Logging

You can monitor many metrics for each of your pipelines and blocks.

Monitoring

Get status updates in your Slack channel.

Alerting status updates in Slack

Get status updates in your Teams channel.

Teams

Alerting status updates in Teams

Get status updates sent to your email inbox.

Email

Alerting status updates in Email

This is a guide for using Spark (PySpark) with Mage in different cloud providers (see a specific section for the cloud provider you use).

Spark

Spark and PySpark

BigQuery

MySQL

Redshift

Snowflake

Use with Mage

Run Mage pipelines in Airflow

In this tutorial, we’ll create a DAG in Airflow for scheduling and running a data pipeline; all from the Mage UI.