> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mage.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# It’s magic.

> Give your data team magical powers.

export const ProButton = ({href, label = 'Get started with Mage Pro for free', source = 'documentation'}) => <div style={{
  height: 32,
  position: 'relative'
}}>
    <a target="_blank" className="group px-4 py-1.5 relative inline-flex items-center text-sm font-medium rounded-full" href={href ?? `https://cloud.mage.ai/sign-up?source=${source}`}>
      <span className="absolute inset-0 bg-primary-dark dark:bg-primary-light/10 border-primary-light/30 rounded-full dark:border group-hover:opacity-[0.9] dark:group-hover:border-primary-light/60">
      </span>

      <div className="mr-0.5 space-x-2.5 flex items-center">
        <span class="z-10 text-white dark:text-primary-light">
          {label}
        </span>

        <svg width="3" height="24" viewBox="0 -9 3 24" class="h-5 rotate-0 overflow-visible text-white/90 dark:text-primary-light">
          <path d="M0 0L3 3L0 6" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"></path>
        </svg>
      </div>
    </a>
  </div>;

export const urls = {
  chat: 'https://www.mage.ai/chat',
  oss: 'https://www.mage.ai/oss',
  pro: 'https://cloud.mage.ai/sign-up'
};

<div
  style={{
display: "flex"
}}
>
  <div style={{ marginRight: 8 }}>
    <a href={urls.oss} target="_blank">
      <img alt="Mage AI GitHub repo stars" src="https://img.shields.io/github/stars/mage-ai/mage-ai?style=for-the-badge&logo=github&labelColor=0500ff&logoColor=FFFFFF&label=stars&color=0500ff" className="hidden dark:block" noZoom />

      <img alt="Mage AI GitHub repo stars" src="https://img.shields.io/github/stars/mage-ai/mage-ai?style=for-the-badge&logo=github&labelColor=000000&logoColor=FFFFFF&label=stars&color=0500ff" className="block dark:hidden" noZoom />
    </a>
  </div>

  <div style={{ marginRight: 8 }}>
    <a href="https://hub.docker.com/r/mageai/mageai" target="_blank">
      <img alt="Mage AI Docker downloads" src="https://img.shields.io/docker/pulls/mageai/mageai?style=for-the-badge&logo=docker&labelColor=956fff&logoColor=FFFFFF&label=pulls&color=956fff" className="hidden dark:block" noZoom />

      <img alt="Mage AI Docker downloads" src="https://img.shields.io/docker/pulls/mageai/mageai?style=for-the-badge&logo=docker&labelColor=000000&logoColor=FFFFFF&label=pulls&color=956fff" className="block dark:hidden" noZoom />
    </a>
  </div>

  <div style={{ marginRight: 8 }}>
    <a href="https://github.com/mage-ai/mage-ai/blob/master/LICENSE" target="_blank">
      <img alt="Mage AI license" src="https://img.shields.io/github/license/mage-ai/mage-ai?style=for-the-badge&logo=codeigniter&labelColor=FFCC19&logoColor=000000&label=license&color=FFCC19" className="hidden dark:block" noZoom />

      <img alt="Mage AI license" src="https://img.shields.io/github/license/mage-ai/mage-ai?style=for-the-badge&logo=codeigniter&labelColor=000000&logoColor=FFFFFF&label=license&color=FFCC19" className="block dark:hidden" noZoom />
    </a>
  </div>

  <div style={{ marginRight: 0 }}>
    <a href={urls.chat} target="_blank">
      <img alt="Join the Mage AI community" src="https://img.shields.io/badge/Join%20the%20community-black.svg?style=for-the-badge&logo=lightning&labelColor=DD55FF&logoColor=FFFFFF&label=&color=DD55FF&logoWidth=20" className="hidden dark:block" noZoom />

      <img alt="Join the Mage AI community" src="https://img.shields.io/badge/Join%20the%20community-black.svg?style=for-the-badge&logo=lightning&labelColor=000000&logoColor=FFFFFF&label=&color=DD55FF&logoWidth=20" className="block dark:hidden" noZoom />
    </a>
  </div>
</div>

<a href="https://cloud.mage.ai/sign-up?source=introduction">
  <picture>
    <source media="(prefers-color-scheme: dark)" />

    <img alt="Mage AI hero" src="https://mage-ai.github.io/assets/oss/hero.png" />
  </picture>
</a>

Mage is a hybrid framework for transforming and integrating data. It combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.

* Extract and synchronize data from 3rd party sources.
* Transform data with real-time and batch pipelines using Python, SQL, and R.
* Load data into your data warehouse or data lake using our pre-built connectors.
* Run, monitor, and orchestrate thousands of pipelines without losing sleep.

Plus hundreds of enterprise-class features, infrastructure innovations, and magical surprises.

***

**Available in two spellbinding versions**

<CardGroup cols={2}>
  <Card href={`${urls.pro}?source=introduction`} horizontal>
    <div
      style={{
    display: "flex"
  }}
    >
      <div style={{ marginRight: 16 }}>
        <img src="https://mage-ai.github.io/assets/oss/card-pro.png" noZoom />
      </div>

      <div style={{ marginTop: 8 }}>
        <b>For teams.</b> Fully managed platform
        for integrating and transforming data.
      </div>
    </div>
  </Card>

  <Card href={`${urls.oss}?source=introduction`} horizontal>
    <div
      style={{
    display: "flex"
  }}
    >
      <div style={{ marginRight: 16 }}>
        <img src="https://mage-ai.github.io/assets/oss/card-oss.png" noZoom />
      </div>

      <div style={{ marginTop: 8 }}>
        <b>Self-hosted.</b> System to build, run, and
        manage data pipelines.
      </div>
    </div>
  </Card>
</CardGroup>

<br />

<ProButton source="introduction" />

***

## Getting started

Get quickly started on building and deploying magical data pipelines
by watching a short product video and trying out the interactive demo playground.

<CardGroup>
  <Card title="Product tour" icon="circle-play" href="https://youtu.be/GswOdShLGmg" horizontal>
    Get a 5 minute overview
  </Card>

  <Card title="Live demo" icon="bullseye-pointer" href="https://demo.mage.ai" horizontal>
    Open our interactive demo
  </Card>
</CardGroup>

## 🦄 Make data magical

We put the *fun* in fun-ctional programming. Mage is a hybrid framework that combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.

* **Extract** and synchronize data from 3rd party sources.
* **Transform** data with real-time and batch pipelines using Python, SQL, and R.
* **Load** data into your data warehouse or data lake using our pre-built connectors.
* Run, monitor, and **orchestrate** thousands of pipelines without losing sleep.

<Steps titleSize="h2">
  <Step title="🔨 Build">
    Have you met anyone who loves developing in Airflow? That's why we designed an easy developer experience that you'll enjoy.

    * **Simple developer experience**: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
    * **Language of choice**: Write code in Python, SQL, or R in the same pipeline for *ultimate* flexibility.
    * **Engineering best practices built-in**: Every step in your pipeline is a standalone file with modular code that's reusable and testable. No more DAGs with spaghetti code 🍝.
    * **Bring your favorite tools:** Write [dbt](/guides/dbt/overview) models, use your favorite [IDE](/guides/developer-ux/code-editor), [track changes with Git](/guides/data-sync/version-control), and much much more.

    <p align="center">
      <img src="https://mage-ai.github.io/assets/overview/mage-build.gif" />
    </p>
  </Step>

  <Step title="🔍 Preview">
    Don't waste time waiting for your DAGs to finish testing. Get instant feedback from your code *every* time you run it.

    * **Interactive code**: Immediately see results from your code's output with an interactive notebook UI.
    * **Data is a first-class citizen**: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
    * **Collaborate on cloud**: Develop collaboratively on [cloud resources](/production/deploying-to-cloud/architecture), version control with [Git](/development/git/setting-up-git), and [test](/observability/testing/data-validation) pipelines without waiting for an available shared staging environment.

    <p align="center">
      <img src="https://mage-ai.github.io/assets/overview/mage-preview.gif" />
    </p>
  </Step>

  <Step title="🚀 Launch">
    Don't have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.

    * **Fast deploy**: Deploy Mage to [AWS](/production/deploying-to-cloud/aws/setup), [GCP](/production/deploying-to-cloud/gcp/setup), or [Azure](/production/deploying-to-cloud/azure/setup) with only 2 commands using maintained Terraform templates.
    * **Scaling made simple**: Transform very large datasets directly in your data warehouse or through a native integration with [Spark](/integrations/spark-pyspark).
    * **Observability**: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.

    <p align="center">
      <img src="https://mage-ai.github.io/assets/overview/observability.gif" />
    </p>
  </Step>
</Steps>

## 🔮 [Features](/about/features)

|     |                                                                         |                                                                        |
| --- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| 🎶  | [Orchestration](/design/data-pipeline-management)                       | Schedule and manage data pipelines with observability.                 |
| 📓  | [Notebook editor](/about/features#notebook-for-building-data-pipelines) | Interactive Python, SQL, & R editor for coding data pipelines.         |
| 🏗️ | [Data integration](/data-integrations)                                  | Synchronize data from 3rd party sources to your internal destinations. |
| 🚰  | [Streaming](/guides/streaming/introduction)                             | Ingest and transform real-time data.                                   |
| 🧱  | [dbt](/guides/dbt)                                                      | Build, run, and manage your dbt models with Mage.                      |

## 🏔️ [Core design principles](/design/core-design-principles)

Every user experience and technical design decision adheres to these principles.

|    |                                                                                                  |                                                                                                                              |
| -- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| 💻 | [Easy developer experience](/design/core-design-principles#easy-developer-experience)            | Open-source engine that comes with a custom notebook UI for building data pipelines.                                         |
| 🚢 | [Engineering best practices](/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
| 💳 | [Data as a first-class citizen](/design/core-design-principles#data-is-a-first-class-citizen)    | Designed from the ground up specifically for running data-intensive workflows.                                               |
| 🪐 | [Scaling made simple](/design/core-design-principles#scaling-is-made-simple)                     | Analyze and process large data quickly for rapid iteration.                                                                  |

## 🛸 [Core abstractions](/design/core-abstractions)

These are the fundamental concepts that Mage uses to operate.

|       |                                                        |                                                                                                                                                             |
| ----- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 🏢    | [Project](/design/core-abstractions#project)           | Like a repository on GitHub; this is where you write all your code.                                                                                         |
| 🪈    | [Pipeline](/design/core-abstractions#pipeline)         | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.        |
| 🧱    | [Block](/design/core-abstractions#block)               | A file with code that can be executed independently or within a pipeline.                                                                                   |
| 🤓    | [Data product](/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage.                                                                 |
| ⏰     | [Trigger](/design/core-abstractions#trigger)           | A set of instructions that determine when or how a pipeline should run.                                                                                     |
| 🏃‍♂️ | [Run](/design/core-abstractions#run)                   | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |

## 👨‍👩‍👧‍👦 Community

Individually, we're a mage.

> 🧙 Mage
>
> Magic is indistinguishable from advanced technology. A mage is someone who
> uses magic (aka advanced technology).

Together, we're Magers!

> 🧙‍♂️🧙 Magers (`/ˈmājər/`)
>
> A group of mages who help each other realize their full potential! Join us on [Slack](https://www.mage.ai/chat).

✨ This documentation & project are brought to you by the following *magical* individuals (learn more about contributing [here](/contributing/overview)):

<p align="center">
  <a href="https://github.com/mage-ai/mage-ai/graphs/contributors">
    <img src="https://contrib.rocks/image?repo=mage-ai/mage-ai&columns=10" />
  </a>
</p>

***

<p align="center">
  <img src="https://mage-ai.github.io/assets/mascots/mascots-shorter.jpeg" noZoom />
</p>