- Introduction
- Mage
๐ง A modern replacement for Airflow.
Watch demo
Watch our 2 min video
Live demo
Open our interactive demo
Documentation
Learn more about Mage
Community chat
Join our Slack community
โGive your data team magical
powers
-
Integrate and synchronize data from 3rd party sources.
-
Build real-time and batch pipelines to transform data using Python, SQL, and R.
-
Run, monitor, and orchestrate thousands of pipelines without losing sleep
โ1. Build
Have you met anyone who said they loved developing in Airflow? Thatโs why we designed an easy developer experience that youโll enjoy.
- Easy developer experience: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
- Language of choice: Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.
- Engineering best practices built-in: Each step in your pipeline is a standalone file containing modular code thatโs reusable and testable with data validations. No more DAGs with spaghetti code.

โ2. Preview
Stop wasting time waiting around for your DAGs to finish testing. Get instant feedback from your code each time you run it.
- Interactive code: Immediately see results from your codeโs output with an interactive notebook UI.
- Data is a first-class citizen: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
- Collaborate on cloud: Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.

โ3. Launch
Donโt have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.
- Fast deploy: Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.
- Scaling made simple: Transform very large datasets directly in your data warehouse or through a native integration with Spark.
- Observability: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.

โ๐ง Intro
Mage is an open-source data pipeline tool for transforming and integrating data.
โ๐โโ๏ธ Quick start
You can install and run Mage using Docker (recommended), pip
, or conda
.
โInstall using Docker
- Create a new project and launch tool (change
demo_project
to any other name if you want):
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
/app/run_app.sh mage start demo_project
- Open http://localhost:6789 in your browser and build a pipeline.
โUsing pip
or conda
- Install Mage
pip install mage-ai
or
conda install -c conda-forge mage-ai
For additional packages (e.g. spark
, postgres
, etc), please see
Installing extra packages.
If you run into errors, please see
Install errors.
- Create new project and launch tool (change
demo_project
to any other name if you want):
mage start demo_project
- Open http://localhost:6789 in your browser and build a pipeline.
โ๐ฎ Demo
Build and run a data pipeline with our live demo app.
The live demo is public to everyone, please donโt save anything sensitive (e.g. passwords, secrets, etc).
โDemo video (2 min)
โ๐ฉโ๐ซ Tutorials
Load data from API, transform it, and export it to PostgreSQL
Set up DBT models and orchestrate DBT runs
Train model on Titanic dataset
Integrate Mage into an existing Airflow project
โ๐ฎ Features
๐ถ | Orchestration | Schedule and manage data pipelines with observability. |
---|---|---|
๐ | Notebook | Interactive Python, SQL, & R editor for coding data pipelines. |
๐๏ธ | Data integrations | Synchronize data from 3rd party sources to your internal destinations. |
๐ฐ | Streaming pipelines | Ingest and transform real-time data. |
โ | DBT | Build, run, and manage your DBT models with Mage. |
A sample data pipeline defined across 3 files โ
- Load data โ
@data_loader
def load_csv_from_file():
return pd.read_csv('default_repo/titanic.csv')
- Transform data โ
@transformer
def select_columns_from_df(df, *args):
return df[['Age', 'Fare', 'Survived']]
- Export data โ
@data_exporter
def export_titanic_data_to_disk(df) -> None:
df.to_csv('default_repo/titanic_transformed.csv')
What the data pipeline looks like in the UI โ

New? We recommend reading about blocks and learning from a hands-on tutorial.
Ask us questions on Slack
โ๐๏ธ Core design principles
Every user experience and technical design decision adheres to these principles.
๐ป | Easy developer experience | Open-source engine that comes with a custom notebook UI for building data pipelines. |
---|---|---|
๐ข | Engineering best practices built-in | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
๐ณ | Data is a first-class citizen | Designed from the ground up specifically for running data-intensive workflows. |
๐ช | Scaling is made simple | Analyze and process large data quickly for rapid iteration. |
โ๐ธ Core abstractions
These are the fundamental concepts that Mage uses to operate.
Project | Like a repository on GitHub; this is where you write all your code. |
---|---|
Pipeline | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
Block | A file with code that can be executed independently or within a pipeline. |
Data product | Every block produces data after itโs been executed. These are called data products in Mage. |
Trigger | A set of instructions that determine when or how a pipeline should run. |
Run | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
โ๐โโ๏ธ Contributing and developing
Add features and instantly improve the experience for everyone.
Check out the contributing guide to setup your development environment and start building.
โ๐จโ๐ฉโ๐งโ๐ฆ Community
Individually, weโre a mage.
๐ง Mage
Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, weโre Magers!
๐งโโ๏ธ๐ง Magers (
/หmฤjษr/
)A group of mages who help each other realize their full potential! Letโs hang out and chat together โ
Hang out on Slack
For real-time news, fun memes, data engineering topics, and more, join us on โ
GitHub
โ๐ค Frequently Asked Questions (FAQs)
Check out our FAQ page to find answers to some of our most asked questions.
โ๐ชช License
See the LICENSE file for licensing information.
- Give your data team magical powers
- 1. Build
- 2. Preview
- 3. Launch
- ๐ง Intro
- ๐โโ๏ธ Quick start
- Install using Docker
- Using pip or conda
- ๐ฎ Demo
- Demo video (2 min)
- ๐ฉโ๐ซ Tutorials
- ๐ฎ Features
- ๐๏ธ Core design principles
- ๐ธ Core abstractions
- ๐โโ๏ธ Contributing and developing
- ๐จโ๐ฉโ๐งโ๐ฆ Community
- ๐ค Frequently Asked Questions (FAQs)
- ๐ชช License