Mage

​

Give your data team magical powers

• Integrate and synchronize data from 3rd party sources.

• Build real-time and batch pipelines to transform data using Python, SQL, and R.

• Run, monitor, and orchestrate thousands of pipelines without losing sleep

​

1. Build

Have you met anyone who said they loved developing in Airflow? That’s why we designed an easy developer experience that you’ll enjoy.

• Easy developer experience: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
• Language of choice: Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.
• Engineering best practices built-in: Each step in your pipeline is a standalone file containing modular code that’s reusable and testable with data validations. No more DAGs with spaghetti code.

​

2. Preview

Stop wasting time waiting around for your DAGs to finish testing. Get instant feedback from your code each time you run it.

• Interactive code: Immediately see results from your code’s output with an interactive notebook UI.
• Data is a first-class citizen: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
• Collaborate on cloud: Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.

​

3. Launch

Don’t have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.

• Fast deploy: Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.
• Scaling made simple: Transform very large datasets directly in your data warehouse or through a native integration with Spark.
• Observability: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.

​

🧙 Intro

Mage is an open-source data pipeline tool for transforming and integrating data.

​

🏃‍♀️ Quick start

You can install and run Mage using Docker (recommended), pip, or conda.

​

Install using Docker

1. Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage start demo_project

2. Open http://localhost:6789 in your browser and build a pipeline.

​

Using pip or conda

1. Install Mage

pip install mage-ai

or

conda install -c conda-forge mage-ai

For additional packages (e.g. spark, postgres, etc), please see Installing extra packages. If you run into errors, please see Install errors.

2. Create new project and launch tool (change demo_project to any other name if you want):

mage start demo_project

3. Open http://localhost:6789 in your browser and build a pipeline.

​

🎮 Demo

Build and run a data pipeline with our live demo app.

​

Demo video (2 min)

​

👩‍🏫 Tutorials

​

🔮 Features

A sample data pipeline defined across 3 files ➝

1. Load data ➝

@data_loader
def load_csv_from_file():
    return pd.read_csv('default_repo/titanic.csv')

2. Transform data ➝

@transformer
def select_columns_from_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]

3. Export data ➝

@data_exporter
def export_titanic_data_to_disk(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI ➝

New? We recommend reading about blocks and learning from a hands-on tutorial.

​

🏔️ Core design principles

Every user experience and technical design decision adheres to these principles.

​

🛸 Core abstractions

These are the fundamental concepts that Mage uses to operate.

​

🙋‍♀️ Contributing and developing

Add features and instantly improve the experience for everyone.

Check out the contributing guide to setup your development environment and start building.

​

👨‍👩‍👧‍👦 Community

Individually, we’re a mage.

🧙 Mage

Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, we’re Magers!

🧙‍♂️🧙 Magers (/ˈmājər/)

A group of mages who help each other realize their full potential! Let’s hang out and chat together ➝

For real-time news, fun memes, data engineering topics, and more, join us on ➝

​

🤔 Frequently Asked Questions (FAQs)

Check out our FAQ page to find answers to some of our most asked questions.

​

🪪 License

See the LICENSE file for licensing information.