🧙 A modern replacement for Airflow.
Integrate and synchronize data from 3rd party sources
Build real-time and batch pipelines to transform data using Python, SQL, and R
Run, monitor, and orchestrate thousands of pipelines without losing sleep
Have you met anyone who said they loved developing in Airflow? That’s why we designed an easy developer experience that you’ll enjoy.
Stop wasting time waiting around for your DAGs to finish testing. Get instant feedback from your code each time you run it.
Don’t have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.
Mage is an open-source data pipeline tool for transforming and integrating data.
You can install and run Mage using Docker (recommended),
- 1.Create a new project and launch tool (change
demo_projectto any other name if you want):docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \mage start demo_projectWant to use Spark or other integrations? Read more about integrations.
- 2.Create new project and launch tool (change
demo_projectto any other name if you want):mage start demo_project
WARNINGThe live demo is public to everyone, please don’t save anything sensitive (e.g. passwords, secrets, etc).
Click the image to play video
A sample data pipeline defined across 3 files ➝
- 1.Load data ➝@data_loaderdef load_csv_from_file():return pd.read_csv('default_repo/titanic.csv')
- 2.Transform data ➝@transformerdef select_columns_from_df(df, *args):return df[['Age', 'Fare', 'Survived']]
- 3.Export data ➝@data_exporterdef export_titanic_data_to_disk(df) -> None:df.to_csv('default_repo/titanic_transformed.csv')
What the data pipeline looks like in the UI ➝
data pipeline overview
Every user experience and technical design decision adheres to these principles.
These are the fundamental concepts that Mage uses to operate.
Add features and instantly improve the experience for everyone.
Individually, we’re a mage.
🧙 MageMagic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, we’re Magers!
🧙♂️🧙 Magers (
/ˈmājər/)A group of mages who help each other realize their full potential! Let’s hang out and chat together ➝
For real-time news, fun memes, data engineering topics, and more, join us on ➝