๐งโโ๏ธ Welcome to Mage
Mage is an open-source, hybrid framework for transforming and integrating data. โจ
๐ฆ Make data magical
We put the fun in fun-ctional programming. Mage is a hybrid framework that combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.
- Extract and synchronize data from 3rd party sources.
- Transform data with real-time and batch pipelines using Python, SQL, and R.
- Load data into your data warehouse or data lake using our pre-built connectors.
- Run, monitor, and orchestrate thousands of pipelines without losing sleep.
๐จ Build
Have you met anyone who loves developing in Airflow? Thatโs why we designed an easy developer experience that youโll enjoy.
- Simple developer experience: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
- Language of choice: Write code in Python, SQL, or R in the same pipeline for ultimate flexibility.
- Engineering best practices built-in: Every step in your pipeline is a standalone file with modular code thatโs reusable and testable. No more DAGs with spaghetti code ๐.
- Bring your favorite tools: Write dbt models, use your favorite IDE, track changes with Git, and much much more.
๐ Preview
Donโt waste time waiting for your DAGs to finish testing. Get instant feedback from your code every time you run it.
- Interactive code: Immediately see results from your codeโs output with an interactive notebook UI.
- Data is a first-class citizen: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
- Collaborate on cloud: Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.
๐ Launch
Donโt have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.
- Fast deploy: Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.
- Scaling made simple: Transform very large datasets directly in your data warehouse or through a native integration with Spark.
- Observability: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.
๐ฎ Features
| ๐ถ | Orchestration | Schedule and manage data pipelines with observability. |
| ๐ | Notebook editor | Interactive Python, SQL, & R editor for coding data pipelines. |
| ๐๏ธ | Data integration | Synchronize data from 3rd party sources to your internal destinations. |
| ๐ฐ | Streaming | Ingest and transform real-time data. |
| ๐งฑ | dbt | Build, run, and manage your dbt models with Mage. |
๐๏ธ Core design principles
Every user experience and technical design decision adheres to these principles.
| ๐ป | Easy developer experience | Open-source engine that comes with a custom notebook UI for building data pipelines. |
| ๐ข | Engineering best practices | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
| ๐ณ | Data as a first-class citizen | Designed from the ground up specifically for running data-intensive workflows. |
| ๐ช | Scaling made simple | Analyze and process large data quickly for rapid iteration. |
๐ธ Core abstractions
These are the fundamental concepts that Mage uses to operate.
| ๐ข | Project | Like a repository on GitHub; this is where you write all your code. |
| ๐ช | Pipeline | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
| ๐งฑ | Block | A file with code that can be executed independently or within a pipeline. |
| ๐ค | Data product | Every block produces data after itโs been executed. These are called data products in Mage. |
| โฐ | Trigger | A set of instructions that determine when or how a pipeline should run. |
| ๐โโ๏ธ | Run | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
๐จโ๐ฉโ๐งโ๐ฆ Community
Individually, weโre a mage.
๐ง Mage
Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology).
Together, weโre Magers!
๐งโโ๏ธ๐ง Magers (
/หmฤjษr/)A group of mages who help each other realize their full potential! Join us on Slack.
โจ This documentation & project are brought to you by the following magical individuals (learn more about contributing here):
๐ชช License
Was this page helpful?