Python package versionAnaconda versionGitHub StarsDocker pullspip installs

๐Ÿฆ„ Make data magical

We put the fun in fun-ctional programming. Mage is a hybrid framework that combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.

  • Extract and synchronize data from 3rd party sources.
  • Transform data with real-time and batch pipelines using Python, SQL, and R.
  • Load data into your data warehouse or data lake using our pre-built connectors.
  • Run, monitor, and orchestrate thousands of pipelines without losing sleep.
1

๐Ÿ”จ Build

Have you met anyone who loves developing in Airflow? Thatโ€™s why we designed an easy developer experience that youโ€™ll enjoy.

  • Simple developer experience: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
  • Language of choice: Write code in Python, SQL, or R in the same pipeline for ultimate flexibility.
  • Engineering best practices built-in: Every step in your pipeline is a standalone file with modular code thatโ€™s reusable and testable. No more DAGs with spaghetti code ๐Ÿ.
  • Bring your favorite tools: Write dbt models, use your favorite IDE, track changes with Git, and much much more.

2

๐Ÿ” Preview

Donโ€™t waste time waiting for your DAGs to finish testing. Get instant feedback from your code every time you run it.

  • Interactive code: Immediately see results from your codeโ€™s output with an interactive notebook UI.
  • Data is a first-class citizen: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
  • Collaborate on cloud: Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.

3

๐Ÿš€ Launch

Donโ€™t have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.

  • Fast deploy: Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.
  • Scaling made simple: Transform very large datasets directly in your data warehouse or through a native integration with Spark.
  • Observability: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.

๐Ÿ”ฎ Features

๐ŸŽถOrchestrationSchedule and manage data pipelines with observability.
๐Ÿ““Notebook editorInteractive Python, SQL, & R editor for coding data pipelines.
๐Ÿ—๏ธData integrationSynchronize data from 3rd party sources to your internal destinations.
๐ŸšฐStreamingIngest and transform real-time data.
๐ŸงฑdbtBuild, run, and manage your dbt models with Mage.

๐Ÿ”๏ธ Core design principles

Every user experience and technical design decision adheres to these principles.

๐Ÿ’ปEasy developer experienceOpen-source engine that comes with a custom notebook UI for building data pipelines.
๐ŸšขEngineering best practicesBuild and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts.
๐Ÿ’ณData as a first-class citizenDesigned from the ground up specifically for running data-intensive workflows.
๐ŸชScaling made simpleAnalyze and process large data quickly for rapid iteration.

๐Ÿ›ธ Core abstractions

These are the fundamental concepts that Mage uses to operate.

๐ŸขProjectLike a repository on GitHub; this is where you write all your code.
๐ŸชˆPipelineContains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.
๐ŸงฑBlockA file with code that can be executed independently or within a pipeline.
๐Ÿค“Data productEvery block produces data after itโ€™s been executed. These are called data products in Mage.
โฐTriggerA set of instructions that determine when or how a pipeline should run.
๐Ÿƒโ€โ™‚๏ธRunStores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc.

๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ Community

Individually, weโ€™re a mage.

๐Ÿง™ Mage

Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology).

Together, weโ€™re Magers!

๐Ÿง™โ€โ™‚๏ธ๐Ÿง™ Magers (/หˆmฤjษ™r/)

A group of mages who help each other realize their full potential! Join us on Slack.

โœจ This documentation & project are brought to you by the following magical individuals (learn more about contributing here):

๐Ÿชช License