# Changelog
Source: https://docs.mage.ai/about/changelog

Latest new features and changelog.

> **[Changelog](https://github.com/mage-ai/mage-ai/releases)**
>
> Learn more about the latest features and improvements


# Code of Conduct
Source: https://docs.mage.ai/about/code-of-conduct

We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a *magical* ✨ experience for everyone.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our
community include:

* 🪄 Deliver magical experiences: demonstrating empathy and kindness toward others.
  Being respectful of differing opinions, viewpoints, and experiences
* 🔋 Give people power-ups: in data engineering *and* open-source development,
  doing your best to help others learn and grow
* 👬 Victorious as a team: working together to build the best product possible,
  recognizing that teamwork can amplify outcomes beyond that of an individual
* 🙅‍♂️ No Ego: giving and gracefully accepting constructive feedback

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or advances of
  any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
  without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
  professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement— please reach
out to [the Mage team](mailto:hello@mage.ai) with any concerns.
All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the
reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate.

### 2. Warning

**Community Impact**: A violation through a single incident or series of
actions.

**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within the
community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code\_of\_conduct.html][v2.1].

Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].

[homepage]: https://www.contributor-covenant.org

[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html

[Mozilla CoC]: https://github.com/mozilla/diversity

[FAQ]: https://www.contributor-covenant.org/faq

[translations]: https://www.contributor-covenant.org/translations


# Features
Source: https://docs.mage.ai/about/features



![](https://user-images.githubusercontent.com/78053898/198751496-22d51f03-f09f-454c-8f98-409b5dbec8f9.svg)

> **[Data pipeline management](/design/data-pipeline-management)**

> **[Notebook for building data pipelines](/about/features#1-data-centric-editor)**

> **[Changelog](https://github.com/mage-ai/mage-ai/releases)**

> **[Roadmap](https://airtable.com/shrJS0cDOmQywb8vp)**

## Data pipeline management

👉 See more
[details here](/design/data-pipeline-management).

![](https://mage-ai.github.io/assets/orchestration-overview.gif)

## Notebook for building data pipelines

### 1. Data centric editor

An interactive coding experience designed for preparing data to train ML models.

Visualize the impact of your code every time you load, clean, and transform
data.

![Data centric editor](https://mage-ai.github.io/assets/data-centric-editor.png)

### 2. Production ready code

No more writing throw away code or trying to turn notebooks into scripts.

Each block (aka cell) in this editor is a modular file that can be tested,
reused, and chained together to create an executable data pipeline locally or in
any environment.

Read more about [blocks](/design/blocks) and how they work.

![Production ready code](https://mage-ai.github.io/assets/data-pipeline.png)

Run your data pipeline end-to-end using the command line function:
`$ mage run [project] [pipeline]`

You can run your pipeline in production environments with the orchestration
tools

* [Airflow](/guides/integrate-mage-airflow)

* [Prefect](/integrations/prefect)

### 3. Extensible

Easily add new functionality directly in the source code or through plug-ins
(coming soon).

Adding new API endpoints ([Tornado](https://www.tornadoweb.org/en/stable/)),
transformations (Python, PySpark, SQL), and charts (using
[React](https://reactjs.org/)) is easy to do (tutorial coming soon).

![Extensible charts](https://mage-ai.github.io/assets/extensible-charts.gif)


# Frequently Asked Questions
Source: https://docs.mage.ai/about/frequently-asked-questions

Here are some frequently asked questions about Mage and our best answers.

![](https://mage-ai.github.io/assets/notion-faq.png)

> **What is Mage?**
>
> Mage is an open-source data pipeline tool for transforming and integrating data.
>
> 🧙 A mage is someone who uses magic. Advanced technology is indistinguishable from magic.
>
> We’re on a mission to make AI technology more accessible by building data tools for engineers and scientists.
>
> Find out more about our story: [https://www.mage.ai/blog/mage-heros-journey-fantasy-epic-on-how-a-startup-rose-from-the-ashes](https://www.mage.ai/blog/mage-heros-journey-fantasy-epic-on-how-a-startup-rose-from-the-ashes)

> **Who is the ideal user for this tool?**
>
> Our tool was built with data engineers and data scientists in mind, but is not limited to those roles. Other data professionals could find value in the tool.

> **How difficult is Mage to setup?**
>
> You can quickly and easily get started by installing Mage using Docker (recommended), **`pip`**, or **`conda`**. Click [here](https://docs.mage.ai/introduction/overview#quick-start) for details.

> **How much does Mage cost?**
>
> Mage is free as long as you are self-hosted (AWS, GCP, Azure, or Digital Ocean).

> **How is Mage’s data pipeline engine software different from Airflow, Dagster, etc?**
>
> Our 4 [core design principles](/design/core-design-principles) that differentiate ourselves are:
>
> 1. Easy developer experience
>
> 2. Engineering best practices built-in
>
> 3. Data is a first-class citizen
>
> 4. Scaling is made simple
>
> Features that set us apart (some of the others might eventually have these features):
>
> 1. Mix and match SQL and Python in data pipeline tasks.
> 2. UI/IDE for building and managing data pipelines.
> 3. Data centric: we designed and built a pipeline engine ONLY for moving and transforming data. This makes it possible for us to make datasets a 1st class citizen; enabling native features such as partitioning, versioning, backfilling, data validation, testing, and data quality monitoring.
> 4. Extensible: we designed and built the tool with developers in mind, making sure it’s really easy to add new functionality to the source code or through plug-ins.
> 5. Scalable: the tool can handle very, very large datasets while transforming the data or charting it.
> 6. Production ready: when you build your data pipeline, it runs exactly the same in development as it does in production. Deploying the tool and managing the infrastructure in production is very easy and simple, unlike Airflow.
> 7. Modular: every block/cell you write is a standalone file that is interoperable; meaning it can be used in other pipelines or in other code bases.

> **What’s the difference between Mage and Fivetran?**
>
> Check out our blog [Mage vs. Fivetran](https://www.mage.ai/blog/mage-vs-fivetran).

> **What’s the difference between Mage and Airbyte?**
>
> Check out our blog [Mage vs Airbyte](https://www.mage.ai/blog/mage-vs-airbyte).

> **What’s the difference between Mage and Prefect?**
>
> Mage provides an interactive notebook with built-in engineering best practices for building pipelines, which makes prototyping and building production-ready pipelines much easier.
>
> Mage supports writing pipelines in multiple languages which include Python, SQL, and R.
>
> Mage supports multiple types of pipelines natively such as:
>
> * Standard batch pipelines
> * Data integration pipelines
> * Streaming pipelines
> * Spark pipelines
> * DBT pipelines

> **What languages does Mage support?**
>
> We currently support SQL, Python, R, and PySpark. Coming soon: Spark SQL.

> **Does Mage integrate with Spark?**
>
> Yes! [Here](https://docs.mage.ai/integrations/spark-pyspark) is a step-by-step tutorial to use Mage with Spark on EMR.

> **What’s the difference between Mage and Sagemaker?**
>
> Sagemaker is used to train machine learning models and serve them via api.
>
> Mage is an engine for running data pipelines that can move and transform data. That data can then be stored anywhere (e.g. S3) and used to train models in Sagemaker.

> **What’s the difference between Mage and Databricks?**
>
> Databricks provides infrastructure to run Spark. They also provide notebooks that can run your code in Spark as well.
>
> Mage can execute your code in a Spark cluster, managed by AWS, GCP, or even Databricks.

> **How do I send pipeline notifications to Slack?**
>
> [Here is a doc](/integrations/observability/alerting-slack) to help you set up alerting for pipeline status updates in [Slack](https://www.mage.ai/chat).

> **How can I contribute or request features?**
>
> We love and welcome community contributions! [Here is a doc](/contributing/overview) to get you started.
>
> To request features, add a “Feature request” using the `New issue` button in GitHub from this [link](https://github.com/mage-ai/mage-ai/issues), or join our [feature-request](https://www.mage.ai/chat) Slack channel.

*Can’t find what you’re looking for? [Ask a question here](https://github.com/mage-ai/mage-ai/discussions/categories/q-a) or [join our slack](https://mage.ai/chat) for additional support!*


# Releases
Source: https://docs.mage.ai/about/releases

Latest new features and changelog.

> **[Update Mage](/development/updating-mage)**
>
> Install the newest version of Mage

Mage Pro may show a short feature highlight when you first open the app after an
update. These highlights are available in Mage Pro only and spotlight selected
features; the full release notes remain available below. Select **Next** to move
through the highlights, then **Done**, or choose a feature action to continue into
the app. The background stays locked until you finish or choose an action; Escape
and clicking outside do not close the announcement. Finishing or choosing a feature
action marks all highlights offered in that announcement as seen, including any
remaining screens. This is remembered for your account and deployment in the current
browser; clearing browser storage or using another browser can show them again.
Older highlights can still appear if they remain included in the installed release
and you have not completed them in that browser. Video previews play automatically
without sound. Some previews may repeat.

***

> **[All releases](https://github.com/mage-ai/mage-ai/releases)**
>
> Install the newest version of Mage

**Kaos**

0.9.74

[Read full release notes](https://github.com/mage-ai/mage-ai/releases/tag/0.9.74)

### ☁️ Google cloud storage source

You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.

![](https://mage-ai.github.io/assets/pipelines/integration-pipelines/gcs_integration_test.png)

by @TalaatHasanin in [https://github.com/mage-ai/mage-ai/pull/5334](https://github.com/mage-ai/mage-ai/pull/5334)

### 🌬️ 🍽️ Airtable Integration

Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.

![](https://mage-ai.github.io/assets/pipelines/integration-pipelines/airtable_integration_test.png)

by @TalaatHasanin in [https://github.com/mage-ai/mage-ai/pull/5404](https://github.com/mage-ai/mage-ai/pull/5404)

### ✂️ Add trim reformat action to transformer block

A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.

by @cristopheridlc in [https://github.com/mage-ai/mage-ai/pull/5321](https://github.com/mage-ai/mage-ai/pull/5321)

### 🐻‍❄️ Enable polars dataframe in GCS data exporter

This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.

![](https://media0.giphy.com/media/v1.Y2lkPTc5MGI3NjExbDM2amtld2toajMxMmR2cmt2eHc4MmdlZ2V6Y3IxcXdncmI1ZGN5YSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/aK4wh0UE3oddS/giphy.webp?raw=true)

by @TalaatHasanin in [https://github.com/mage-ai/mage-ai/pull/5348](https://github.com/mage-ai/mage-ai/pull/5348)

### Add support to scheduler name on k8s executor

This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.

by @messerzen in [https://github.com/mage-ai/mage-ai/pull/5412](https://github.com/mage-ai/mage-ai/pull/5412)

**Deadpool & Wolverine**

0.9.73

[Read full release notes](https://github.com/mage-ai/mage-ai/releases/tag/0.9.73)

### 🧠 Memory management upgrade

This feature enhances system performance and stability by optimizing allocation, deallocation, and overall utilization of RAM resources, resulting in improved efficiency and reduced memory-related errors.

Check out the [doc](https://www.notion.so/mageai/Memory-management-upgrades-e6c5e6e360ce410094091e00c46e3bb6?pvs=4) for details.

### 🔄 Dynamic blocks 2.0

Introducing an enhanced Dynamic blocks feature for creating dynamic content blocks that can adapt based on user input or data changes, offering improved flexibility and interactivity for users. This update aims to streamline workflows and enhance the overall user experience by making content more responsive and customizable.

Learn more in this [doc](https://www.notion.so/mageai/Dynamic-Blocks-2-0-d2fc11a4f48148d68b817e9bfb38a732).

### 🔐 Azure DB connection via Key Vault

Securely retrieve a database connection URL stored in Azure Key Vault using environment variables for authentication.

```
AZURE_KEY_VAULT_URL
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID
```

by [@wangxiaoyou1993](https://github.com/wangxiaoyou1993) in [#5302](https://github.com/mage-ai/mage-ai/pull/5302)

### 👁️ Workspace monitoring

Add an "Overview" page and "Pipeline runs" page to the Workspace Management UI. This provides some overall monitoring for all of the pipeline runs in the different workspaces without having to individually open up each workspace instance.

![](https://github.com/user-attachments/assets/0b8a1d83-3292-4e10-928b-5d625f817d40)

by @johnson-mage in [https://github.com/mage-ai/mage-ai/pull/5311](https://github.com/mage-ai/mage-ai/pull/5311)

**House Targaryen**

0.9.72

[Read full release notes](https://github.com/mage-ai/mage-ai/releases/tag/0.9.72)

[House Targaryen Release](https://www.youtube.com/embed/abiOckshzu8)

**X-Men**

0.9.71

[Read full release notes](https://github.com/mage-ai/mage-ai/releases/tag/0.9.71)

[X-Men Release](https://www.youtube.com/embed/m8iw0e8346k)

**Fallout**

0.9.70

[Read full release notes](https://github.com/mage-ai/mage-ai/releases/tag/0.9.70)

[Fallout Release](https://www.youtube.com/embed/rAp7psCTxC8)

**Invincible**

0.9.68

[Read full release notes](https://github.com/mage-ai/mage-ai/releases/tag/0.9.68)

[Invincible Release](https://www.youtube.com/embed/ck1UGjUeGGA)

* [0.9.66 | Shogun Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.66)
* [0.9.65 | Demon Slayer Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.65)
* [0.9.64 | Maestro Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.64)
* [0.9.63 | Halo Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.63)
* [0.9.62 | The Beekeeper Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.62)
* [0.9.60 | Yusuke Urameshi Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.60)
* [0.9.59 | Year of the Dragon Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.59)
* [0.9.50 | Wonka Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.50)
* [0.9.48 | The Boy and the Heron Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.48)
* [0.9.46 | Wish Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.46)
* [0.9.45 | Yuji Itadori Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.45)
* [0.9.43 | Attack on Titan Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.43)
* [0.9.41 | Halloween Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.41)
* [0.9.38 | Goosebumps Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.38)
* [0.9.35 | Loki Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.35)
* [0.9.34 | C-3PO Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.34)
* [0.9.30 | Cowboy Bebop Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.30)
* [0.9.28 | The Creator Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.28)
* [0.9.26 | Expend4bles Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.26)
* [0.9.23 | One Piece Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.23)
* [0.9.21 | Ahsoka Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.21)
* [0.9.19 | The Equalizer Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.19)
* [0.9.16 | Gran Turismo Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.16)
* [0.9.14 | Blue Beetle Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.14)
* [0.9.11 | Mutant Mayhem Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.11)
* [0.9.10 | Haunted Mansion Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.10)
* [0.9.80 | Dead Reckoning Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.8)
* [0.9.40 | Solar Flare Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.4)
* [0.9.00 | The Dial of Destiny Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.0)
* [0.8.93 | Rise of the Beasts Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.93)
* [0.8.86 | Fast Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.86)
* [0.8.83 | Fury of the Gods Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.83)
* [0.8.78 | Rise of the Machines Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.78)
* [0.8.75 | Guardians Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.75)
* [0.8.69 | Hairy Otter Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.69)
* [0.8.58 | Once & Always Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.58)
* [0.8.52 | The Super Mario Bros. Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.52)
* [0.8.44 | Dungeons and Dragons Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.44)
* [0.8.37 | Foundation Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.37)
* [0.8.29 | Wick Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.29)
* [0.8.27 | Shazam Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.27)
* [0.8.24 | Merlin Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.24)
* [0.8.15 | Creed Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.15)
* [0.8.11 | The Mandalorian Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.11)
* [0.8.3 | Everything Everywhere All at Once Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.3)
* [0.7.98 | Quantumania Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.98)
* [0.7.90 | That '90s Show Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.90)
* [0.7.84 | Rabbit Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.84)
* [0.7.74 | Lunar Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.74)


# Roadmap
Source: https://docs.mage.ai/about/roadmap



> **Roadmap**
>
> Mage's 2024 Roadmap coming soon!

[https://airtable.com/embed/applEuqJsK4zF5yJK/shrJctaGt4tvU27uF?backgroundColor=red\&viewControls=on](https://airtable.com/embed/applEuqJsK4zF5yJK/shrJctaGt4tvU27uF?backgroundColor=red\&viewControls=on)


# Help improve the tool
Source: https://docs.mage.ai/about/statistics

Please contribute usage statistics to help improve the developer experience for you and everyone in the community 🤝.

> [!TIP]
>
> All usage statistics are completely **anonymous**.
>
> It’s **impossible** for Mage to know which statistics belongs to whom.

![Combine our powers](https://media2.giphy.com/media/0Av9l0VIc01y1isrDw/giphy.gif)

## 🙏 Why is this important?

By opting into sending usage statistics to [Mage](https://www.mage.ai/),
it’ll help the team and community of contributors (Magers) learn what’s going wrong with the tool
and what improvements can be made.

In addition to helping reduce potential errors,
you’ll help inform which features are useful and which need work.

## 🤔 What usage statistics am I sending?

### Project UUID

Each project will have a universally unique identifier.
This will help Mage count how many projects have been created.

> [!NOTE]
>
> It’s **impossible** to associate a UUID with a project without knowing the pair together.

Your project UUID is stored in the project’s `metadata.yaml` file,
located at the root of your project: `[project_name]/metadata.yaml`.

Here is an example of what it could look like:

```yaml theme={"system"}
variables_dir: ~/.mage_data

# ...

project_uuid: 4279d28ab1f64644b1f2f4f779be7b7e
```

### Number of pipelines

Sending usage statistics will include the number of pipelines in a single project.
This will help improve the coding experience when building pipelines.

### Number of pipeline runs

Sending usage statistics will include the number of times any pipeline has ran in a single project.
This will help add better pipeline management features.

### Number of users

Sending usage statistics will include the number of users in a single project.
This will help improve the collaboration capabilities of the tool.

> [!NOTE]
>
> This usage statistic is only included if
> [user authentication](/production/authentication/overview) is enabled.

### Errors

When an application error occurs in Mage, the error type, error message, and offending line of code
will be included in the usage statistics.
This will help fix bugs and improve the developer experience.

### Platform

The operating system, release, version, etc of the machine that Mage is running on.
This information will help reproduce errors.

## 🤷‍♀️ How does this work?

Usage statistics are anonymously sent to Mage’s online server.

Here’s a sample of the JSON payload containing usage statistics that could be sent:

```json theme={"system"}
{
  "usage_statistic": {
    "project_uuid": "4279d28ab1f64644b1f2f4f779be7b7e",
    "pipelines": 40,
    "pipeline_runs": 357,
    "users": 13,
    "platform": "Linux-5.15.49-linuxkit-aarch64-with-glibc2.31",
    "version": "0.8.70",
    "error": {
      "message": "...",
      "traceback": "..."
    }
  }
}
```

### Enable

To enable sending usage statistics, add a key in the project’s `metadata.yaml` file called
`help_improve_mage` with the value `true`.

Here is an example:

```yaml theme={"system"}
project_uuid: 4279d28ab1f64644b1f2f4f779be7b7e
help_improve_mage: true
```

### Disable

To disable, change the value of `help_improve_mage` to `false`.

Here is an example:

```yaml theme={"system"}
project_uuid: 4279d28ab1f64644b1f2f4f779be7b7e
help_improve_mage: false
```


# Authentication
Source: https://docs.mage.ai/agent/authentication

Log in to your Mage Pro cluster and manage credentials.

## Interactive login

```bash theme={"system"}
mage-agent login
```

The CLI prompts for:

1. Your Mage Pro cluster URL (e.g. `https://app.mage.ai`)
2. Email
3. Password

On success, mage-agent writes your credentials to `~/.mage-agent/config.json` and fetches remote skill files from the cluster.

## Non-interactive login

Pass all values as flags to skip prompts — useful in CI or scripted environments:

```bash theme={"system"}
mage-agent login \
  --cluster-url https://your-mage-url \
  --email YOUR_EMAIL \
  --password YOUR_PASSWORD
```

## Credential file

Credentials are stored at `~/.mage-agent/config.json` with `600` permissions (owner read/write only):

```json theme={"system"}
{
  "cluster_url": "https://your-mage-url",
  "oauth_token": "YOUR_OAUTH_TOKEN"
}
```

> [!WARNING]
>
> Do not commit `~/.mage-agent/config.json` to version control or share it. The OAuth token provides full access to your Mage Pro cluster.

You can also create this file manually if you already have an OAuth token — mage-agent will pick it up automatically.

## Re-authenticating

If you see `AUTH_EXPIRED` errors, your session token has expired. Run `mage-agent login` again to get a fresh token.

## How authentication is used

All MCP tool calls and sync operations send the OAuth token as a bearer credential to your cluster. You can override the default API key by adding an `api_key` field to the config file:

```json theme={"system"}
{
  "cluster_url": "https://your-mage-url",
  "oauth_token": "YOUR_OAUTH_TOKEN",
  "api_key": "YOUR_API_KEY"
}
```


# CLI Reference
Source: https://docs.mage.ai/agent/cli-reference

All mage-agent commands and their options.

## Global flags

These flags apply to every command:

| Flag | Default | Description |
| - | - | - |
| `--log-level` | `INFO` | Log verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `--format-output` | `json` | Output format: `json` or `human` |

You can also set `MAGE_AGENT_LOG_LEVEL` as an environment variable:

```bash theme={"system"}
MAGE_AGENT_LOG_LEVEL=DEBUG mage-agent sync-status
```

***

## `mage-agent login`

Authenticate with your Mage Pro cluster and save credentials.

```bash theme={"system"}
mage-agent login [OPTIONS]
```

| Option | Description |
| - | - |
| `--cluster-url URL` | Mage Pro cluster URL |
| `--email EMAIL` | Account email |
| `--password PASSWORD` | Account password |

If any option is omitted, the CLI prompts interactively.

On success, writes `~/.mage-agent/config.json` and fetches remote skill files from the cluster.

***

## `mage-agent mcp`

Start the MCP stdio server. This is the command your IDE or AI agent launches.

```bash theme={"system"}
mage-agent mcp
```

No options. The server reads JSON-RPC requests from stdin and writes responses to stdout. Logs go to stderr.

***

## `mage-agent sync-project-to-local`

Pull all remote project files to a local directory.

```bash theme={"system"}
mage-agent sync-project-to-local [--local-project-dir PATH]
```

| Option | Default | Description |
| - | - | - |
| `--local-project-dir` | `.mage-agent/local_sync` | Destination directory |

***

## `mage-agent sync-status`

Show what has changed between local and remote.

```bash theme={"system"}
mage-agent sync-status [--local-project-dir PATH]
```

Output groups files into: `remote_to_local`, `local_to_remote`, `conflicts`, `in_sync`.

Use `--format-output human` for a readable table.

***

## `mage-agent sync-local-to-remote`

Push locally changed files to the cluster.

```bash theme={"system"}
mage-agent sync-local-to-remote [--local-project-dir PATH]
```

Only files with `local_to_remote` status are uploaded.

***

## `mage-agent sync`

Bidirectional sync with interactive conflict resolution.

```bash theme={"system"}
mage-agent sync [--local-project-dir PATH]
```

For each conflicting file, prompts: `[r]emote / [l]ocal / [s]kip`.


# IDE Integration
Source: https://docs.mage.ai/agent/ide-integration

Connect Cursor, Claude Code, Claude Desktop, OpenAI Codex CLI, or another MCP-compatible AI client to your Mage Pro cluster via MCP.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mage-agent-ide-integration)

mage-agent exposes a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server over stdio. Any MCP-compatible AI client can connect by launching `mage-agent mcp` as a subprocess.

## Prerequisites

For manual MCP configuration, complete [Installation](/agent/installation)
and [Authentication](/agent/authentication) before setting up IDE integration.

For Claude Desktop, use the [bundled Desktop extension](/agent/mage-agent-plugin#claude-desktop)
and its setup flow. It includes the executable and does not require the Claude
Code CLI or a Python installation.

***

## Cursor

**1. Open MCP settings**

In Cursor, go to **Settings → MCP** (or open the Command Palette and search for "MCP Servers").

**2. Add mage-agent**

Add the following server config:

```json theme={"system"}
{
  "mcpServers": {
    "mage-agent": {
      "command": "mage-agent",
      "args": ["mcp"]
    }
  }
}
```

**3. Reload Cursor**

Save the config and reload the window. Confirm that `mage-agent` tools appear in the MCP tool list.

***

## Claude Code

**1. Add the MCP server via CLI**

```bash theme={"system"}
claude mcp add mage-agent -- mage-agent mcp
```

Or add it manually to `.claude/settings.json` in your project (or `~/.claude/settings.json` globally):

```json theme={"system"}
{
  "mcpServers": {
    "mage-agent": {
      "command": "mage-agent",
      "args": ["mcp"]
    }
  }
}
```

**2. Restart Claude Code**

Restart the Claude Code session. Run a quick sanity check:

> "Use the `pipeline_list` tool to list my Mage pipelines"

***

## OpenAI Codex CLI

**1. Edit your Codex config**

Add `mage-agent` to the `mcpServers` section of your Codex config file:

```json theme={"system"}
{
  "mcpServers": {
    "mage-agent": {
      "command": "mage-agent",
      "args": ["mcp"]
    }
  }
}
```

**2. Restart Codex CLI**

Restart Codex CLI and verify tools and resources are visible.

## PATH issues

If `mage-agent` is not found when the IDE launches the MCP server, use the absolute path to the executable:

```bash theme={"system"}
which mage-agent
# e.g. /Users/you/.venv/bin/mage-agent
```

```json theme={"system"}
{
  "mcpServers": {
    "mage-agent": {
      "command": "/Users/you/.venv/bin/mage-agent",
      "args": ["mcp"]
    }
  }
}
```

***

## Skill resources

When connected, your AI assistant can read bundled Markdown guidance through
MCP resources. The catalog includes pipeline, execution, data preview, and chart
guidance. Examples:

| Resource URI | Content |
| - | - |
| `skill://pipeline` | Pipeline CRUD patterns and conventions |
| `skill://blocks` | Block inspection and editing guidance |
| `skill://executions` | Run triggers and log retrieval |
| `skill://sync` | Sync operations and conflict handling |
| `skill://data_previews` | Explore Data discovery and preview guidance |
| `skill://visualization_charts` | Saved chart configuration and analysis |
| Installed clients expose AI-ready data tools and guidance without a local | |
| `AI_READY_DATA_ENABLED` flag. Operations still require support, configuration, | |
| and permissions on the connected cluster; unsupported or disabled calls can | |
| return API errors. Clients decide how to display or load MCP resources. | |


# Installation
Source: https://docs.mage.ai/agent/installation

System requirements and install options for mage-agent.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mage-agent-installation)

## Requirements

* A Mage Pro cluster (URL + account credentials)
* Your chosen AI client
* Python **3.10** or newer for PyPI or source installation; bundled installers
  and the Claude Desktop extension include the Mage Agent executable

## Download from Mage Pro

In Mage Pro, open **Settings** > **Plugins** > **Mage Agent** to download the latest Mage Agent installer for Claude Code or OpenAI Codex CLI.

The page lists the latest GitHub release assets for macOS and Windows, including file sizes and checksums. Download links use GitHub's stable latest-release asset URLs, so Mage Pro does not need source changes when release tags or asset URLs change. Use the download that matches your operating system and AI coding client, then follow the prompts from the installer.

## Claude Desktop extension

Claude Desktop uses a separate `.mcpb` extension for macOS or Windows. It
includes the standalone Mage Agent executable and does not require Python or
Claude Code. See the [plugin installation guide](/agent/mage-agent-plugin#claude-desktop)
for package selection and setup. The Claude Code `.pkg` and `.zip` installers
are for Claude Code.

## Install from PyPI

```bash theme={"system"}
pip install mage-agent
```

We recommend installing inside a virtual environment:

```bash theme={"system"}
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install mage-agent
```

Verify:

```bash theme={"system"}
mage-agent --version
```

## Install from source

```bash theme={"system"}
git clone https://github.com/mage-ai/mage-agent.git
cd mage-agent
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

## IDE PATH requirement

When you configure your IDE to launch `mage-agent mcp`, the IDE may use a different `PATH` than your terminal. If the MCP server fails to start, use the absolute path to the executable:

```bash theme={"system"}
which mage-agent   # e.g. /Users/you/.venv/bin/mage-agent
```

Then update your IDE config:

```json theme={"system"}
{
  "mcpServers": {
    "mage-agent": {
      "command": "/Users/you/.venv/bin/mage-agent",
      "args": ["mcp"]
    }
  }
}
```


# Introduction
Source: https://docs.mage.ai/agent/introduction

Build and manage Mage data pipelines directly from your AI coding assistant.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mage-agent)

**Mage Agent** connects your local development environment to your Mage Pro cluster through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io). Your AI assistant can inspect, create, modify, and run Mage pipelines using natural language — and a built-in sync engine keeps local files in step with the cluster.

> **[Quickstart](/agent/quickstart)**
>
> Get connected in under five minutes.

> **[IDE Integration](/agent/ide-integration)**
>
> Connect Cursor, Claude Code, OpenAI Codex CLI, or another MCP-compatible client.

> **[Sync](/agent/sync)**
>
> Mirror your Mage project to local files.

> **[MCP Tools](/agent/mcp-tools)**
>
> Full reference for every tool your AI can call.

## How it works

```
Your AI assistant  ──MCP──►  mage-agent  ──REST──►  Mage Pro cluster
      │                           │
      │                     sync engine
      └── local files ◄──────────┘
```

1. **MCP server** — mage-agent runs as a background stdio process. Your IDE or AI agent sends tool calls over JSON-RPC; mage-agent translates them into Mage Pro REST API requests.
2. **Skill resources** — the server exposes Markdown guidance files that give your AI assistant context on how to interact with pipelines, blocks, runs, and sync operations.
3. **Sync engine** — a separate CLI layer lets you pull the full remote project tree to a local directory, push local edits back, and resolve conflicts file by file.

## Supported clients

> **Cursor**

> **Claude Code**

> **OpenAI Codex CLI**

Any MCP-compatible client that can launch a subprocess will work with the same configuration.


# Mage Agent Plugin
Source: https://docs.mage.ai/agent/mage-agent-plugin

Install the Mage Agent Plugin and connect it to a Mage cluster.

# Mage Agent Plugin Installation Guide

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mage-agent-plugin)

## What It Is

The Mage Agent Plugin connects an AI agent client to Mage so the agent can work
with the Mage clusters you authorize. After installation and sign-in, the agent
can help inspect and operate Mage workflows, including pipelines, blocks,
schedules, runs, logs, outputs, files, and other Mage project resources.

## Where To Download It

Download the installer from the public Mage Agent Plugin releases repository:

```text theme={"system"}
https://github.com/mage-ai/agent-plugin/releases
```

The `mage-ai/agent-plugin` repository is the public release-asset repository.
The plugin source and release automation are maintained in the Mage plugin
source under:

```text theme={"system"}
mage-pro/plugins/mage-agent-plugin
```

## Choose The Correct Installer

Use the package that matches your agent client and operating system. Claude
Code and Claude Desktop use different packages.

macOS:

```text theme={"system"}
Mage-Agent-MacOS-Codex.pkg
Mage-Agent-MacOS-Claude.pkg
```

Windows:

```text theme={"system"}
Mage-Agent-Windows-Codex.zip
Mage-Agent-Windows-Claude.zip
```

Use the Codex package if you are installing the plugin for Codex. Use the
Claude package if you are installing it for Claude Code. For Claude Desktop,
use the separate `.mcpb` extension described below.

### Claude Desktop

The Claude Desktop extension bundles the standalone Mage Agent executable;
it does not require Python or the Claude Code CLI. Choose the file for your
computer:

| Computer | Extension file |
| - | - |
| macOS with Apple silicon | `Mage-Agent-Claude-Desktop-darwin-arm64.mcpb` |
| macOS with an Intel processor | `Mage-Agent-Claude-Desktop-darwin-x64.mcpb` |
| Windows x64 | `Mage-Agent-Claude-Desktop-windows-x64.mcpb` |

It exposes the same shared API-backed tools and bundled skill guidance through
MCP. Claude Desktop decides how to display and load that guidance.

Use a Desktop extension supplied with the release or test build you are
installing. The `.pkg` and `.zip` files named for Claude Code install into
Claude Code and do not register a Claude Desktop extension.

1. In Claude Desktop, open **Settings** > **Extensions** > **Advanced
   settings** > **Install Extension**, select the `.mcpb` file, and follow the
   installation prompts.
2. Enable the Mage extension, then start a conversation.
3. Ask: "Connect Mage to my cluster" and provide the cluster name or URL.
4. Complete the Mage sign-in in your browser when prompted, then return to
   Claude Desktop.
5. Ask: "What Mage tools are available?" or "List my Mage pipelines."

The extension starts the bundled executable directly. You do not need to
enter a token into extension settings. The first tool that needs
authentication can also start browser sign-in automatically.
If Mage Cloud tools are missing after your first sign-in, restart the
extension to refresh its tool list.

Claude Desktop reuses the Mage connection stored in
`~/.mage/agent/config.json`. The selected cluster is shared with other Mage
Agent clients on the same computer. To change it, ask "Connect Mage to
`https://your-mage-cluster-url`"; this updates the shared connection. If a
tool fails, check the active cluster, your access, and the server's feature
configuration before retrying.

## Prerequisites

Before installing, make sure you have:

* Access to the Mage cluster you want the agent to use.
* The cluster name or cluster URL.
* The correct agent client installed, such as Codex, Claude Code, or Claude
  Desktop with desktop extension support.
* For Claude Code, the Claude Code CLI must be installed and support plugins.

For Claude Code, verify:

```bash theme={"system"}
claude plugin --help
```

## Install On macOS

1. Open the GitHub Releases page:

```text theme={"system"}
https://github.com/mage-ai/agent-plugin/releases
```

2. Download the correct `.pkg` file:

```text theme={"system"}
Mage-Agent-MacOS-Codex.pkg
Mage-Agent-MacOS-Claude.pkg
```

3. Open the downloaded package.
4. Follow the installer prompts.
5. When the installer finishes, the Mage plugin is installed for the selected
   agent client.

The macOS installer starts the Mage OAuth sign-in helper after installing the
plugin.

## Install On Windows

1. Open the GitHub Releases page:

```text theme={"system"}
https://github.com/mage-ai/agent-plugin/releases
```

2. Download the correct `.zip` file:

```text theme={"system"}
Mage-Agent-Windows-Codex.zip
Mage-Agent-Windows-Claude.zip
```

3. Extract the zip file.
4. Run the installer command inside the extracted folder.

For Codex:

```powershell theme={"system"}
.\Install-Codex.cmd
```

For Claude Code:

```powershell theme={"system"}
.\Install-Claude-Code.cmd
```

## Sign In To Mage

After installing, sign in to Mage and connect the plugin to a cluster.

If the installer opens the sign-in flow automatically, complete the browser
login and select or provide the Mage cluster.

If you need to sign in manually from the plugin source checkout, run:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-login <cluster-name-or-url>
```

Example:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-login mageai-1-tutorials
```

You can also use a full cluster URL:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-login https://your-mage-cluster-url
```

## Switch Clusters

To connect the plugin to a different Mage cluster, run:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-switch-cluster
```

To skip the cluster selection screen, pass the cluster name or URL directly:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-switch-cluster <cluster-name-or-url>
```

Example:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-switch-cluster mageai-1-tutorials
```

## Verify The Active Cluster

Check which Mage cluster is currently connected:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-current-cluster
```

List saved local cluster state:

```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-list-clusters
```

## What Gets Saved Locally

After sign-in, the plugin stores the active Mage connection locally at:

```text theme={"system"}
~/.mage/agent/config.json
```

That file stores the selected `cluster_url` and the token needed for the plugin
to call Mage on your behalf.

OAuth helper logs are written to:

```text theme={"system"}
~/.mage/agent/codex-oauth.log
```

## Confirm It Works

After installation and sign-in:

1. Open your agent client.
2. Ask the agent to use Mage.
3. Try a read-only request first, such as listing available Mage pipelines or
   checking the current cluster.
4. If the agent asks you to authenticate, complete the browser sign-in and retry
   the request.

## Quick Checklist

* Download installer from `https://github.com/mage-ai/agent-plugin/releases`.
* Pick Codex, Claude Code, or Claude Desktop.
* Pick the matching macOS `.pkg`, Windows `.zip`, or Desktop `.mcpb` file.
* Install the package.
* Sign in to Mage.
* Provide or select the Mage cluster name or URL.
* Verify the active cluster.
* Ask the agent to run a simple Mage read-only request.


# MCP Tools
Source: https://docs.mage.ai/agent/mcp-tools

Full reference for every tool your AI assistant can call via mage-agent.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mage-agent-mcp-tools)

Once mage-agent is connected to your IDE, your AI assistant has access to the following tools. All tools require valid authentication.

> [!NOTE]
>
> In MCP clients that support rendered tool output, mage-agent returns compact
> resource widgets for supported resource actions. Widgets are currently
> available for pipelines, blocks, pipeline runs, block runs, block outputs,
> pipeline schedules, and pipeline triggers. They do not change the tool
> arguments or authentication flow; clients that do not support rendered output
> still receive the normal structured tool response.

## Resource Widget Contract

The widget path is an additive MCP Apps enhancement for read-only resource
tools. A widget-enabled tool descriptor includes both `_meta.ui.resourceUri` and
`_meta["openai/outputTemplate"]` pointing at the registered HTML resource. When
the tool succeeds, mage-agent returns:

* `structuredContent.kind = "mage_resource_widget"`
* `structuredContent.schema_version` for renderer compatibility
* `structuredContent.resource` and `structuredContent.action`
* `structuredContent.state` as `ok`, `empty`, `loading`, or `error`
* sanitized `records`, `summary`, `columns`, `filters`, and `pagination`
* optional sanitized `mutation` metadata for create, update, or delete results
* optional sanitized `graph` data for pipeline-oriented detail widgets, using
  normalized `nodes` and `edges` instead of raw pipeline YAML
* `content[0].text` as a JSON fallback for clients that do not render widgets
* `_meta["openai/widgetData"]` with widget-only rendering hints

List and get widgets render the normal table or detail view. Mutating actions
may render the returned record with a mutation summary banner and footer, such as
`Operation: Create`; delete results without a returned record render a compact
success state. The adapter must redact sensitive fields before they enter
`structuredContent`, fallback `content`, widget `_meta`, or activity logs.

Pipeline-oriented widgets may include a compact graph preview when Mage returns
block dependency metadata. The graph shape is:

```json theme={"system"}
{
  "nodes": [
    {
      "id": "load_customers",
      "label": "Load customers",
      "type": "data_loader",
      "status": "completed",
      "selected": false
    }
  ],
  "edges": [
    {
      "source": "load_customers",
      "target": "segment_customers"
    }
  ],
  "totalNodes": 12,
  "truncated": false
}
```

Renderers should treat graph data as optional. If it is missing, malformed, or
too large, the widget must fall back to the existing list/detail table without
blocking the tool response.

## Adding Widget Support

To add another Mage resource widget:

1. Add the resource/action pair to `widget_contracts.py`.
2. Add presentation columns, summary fields, empty/error copy, and guidance in
   `widget_registry.py`.
3. Add adapter tests that cover populated, empty, malformed, large-result, and
   secret-bearing payloads.
4. Add renderer coverage in `widgets/resource_widget` and the app-shell preview
   route under `/apps/test/resource-widgets`.
5. Run the MCP protocol tests and an EKS mocked app-shell visual check before
   calling the change QA-ready.

For normal Mage trigger inventory, active trigger counts, API/webhook trigger
lists, run-once schedules, and next run dates, use `pipeline_schedules`. Use
`pipeline_triggers` only when the user explicitly asks for the advanced
`PipelineTriggerResource` config surface scoped to a pipeline.

***

## AI Chat context

> **current\_page\_context**
>
> Read the visible non-chat text that Mage Pro AI Chat captured from the user's current browser page for the active chat turn.
>
> This tool is only registered when Mage Pro AI Chat starts with captured page context. Standard IDE MCP sessions do not expose it unless `MAGE_AGENT_CURRENT_PAGE_CONTEXT_PATH` points to a valid context sidecar file.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `include_text` | boolean | no | Return the captured page text. Set to `false` to return metadata only. |
>
> > [!NOTE]
> >
> > The context is scoped to the authenticated user and chat session, excludes the chat UI, and should only be used for page-relative requests such as "explain this page" or "what am I looking at?"

***

## Pipelines

> **pipeline\_list**
>
> List all pipelines in the project.
>
> **Arguments:** none
>
> **Example prompt:** *"List all my Mage pipelines"*

> **pipeline\_get**
>
> Get details for a single pipeline.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `uuid` | string | yes | Pipeline UUID |

> **pipeline\_create**
>
> Create a new pipeline.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `name` | string | yes | Pipeline name |
> | `type` | string | no | `python` (default), `streaming`, or `integration` |
>
> **Example prompt:** *"Create a new streaming pipeline called `realtime_events`"*

> **pipeline\_update**
>
> Update fields on an existing pipeline.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `uuid` | string | yes | Pipeline UUID |
> | `name` | string | no | New name |
> | `description` | string | no | Updated description |

> **pipeline\_delete**
>
> Delete a pipeline by UUID.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `uuid` | string | yes | Pipeline UUID |
>
> > [!WARNING]
> >
> > Deletion is permanent and cannot be undone.

***

## Blocks

> **block\_list**
>
> List all blocks in a pipeline.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `pipeline_uuid` | string | yes | Pipeline UUID |

> **block\_get**
>
> Get block details including source code.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `pipeline_uuid` | string | yes | Pipeline UUID |
> | `uuid` | string | yes | Block UUID |

> **block\_create**
>
> Create a new block in a pipeline.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `pipeline_uuid` | string | yes | Pipeline UUID |
> | `name` | string | yes | Block name |
> | `type` | string | yes | Block type (see below) |
> | `language` | string | no | `python` (default), `sql`, `r`, `yaml`, `markdown` |
> | `content` | string | no | Initial source code |
> | `upstream_blocks` | array | no | UUIDs of upstream blocks |
> | `config` | object | no | Additional block configuration |
>
> **Supported block types:**
> `ai`, `data_loader`, `transformer`, `data_exporter`, `sensor`, `callback`, `chart`, `conditional`, `custom`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`
>
> AI blocks are available in Mage Pro only. For Sidekick-powered agent blocks, use `type="ai"`, `language="yaml"`, and YAML content with top-level `type: agent`.
>
> > [!NOTE]
> >
> > If no `config` is provided, mage-agent automatically matches a block template for you. For `integration` pipelines, it also auto-detects available sources and destinations.
>
> **Example prompt:** *"Add a SQL transformer block to the `customer_etl` pipeline that filters rows where revenue > 1000"*

> **block\_update**
>
> Update a block's code or configuration.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `pipeline_uuid` | string | yes | Pipeline UUID |
> | `uuid` | string | yes | Block UUID |
> | `content` | string | no | New source code |
> | `name` | string | no | New name |
> | `upstream_blocks` | array | no | Updated upstream block UUIDs |

> **block\_templates\_list**
>
> List available block templates.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `force_refresh` | boolean | no | Bypass the daily cache and re-fetch from cluster |
>
> Templates are cached for 24 hours at `~/.mage-agent/cache/block_templates.json`.

***

## Runs & Logs

> **run\_pipeline**
>
> Trigger a pipeline run.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `pipeline_uuid` | string | yes | Pipeline UUID |
> | `payload` | object | no | Optional run payload (variables, etc.) |
>
> **Example prompt:** *"Run the `daily_refresh` pipeline"*

> **run\_logs**
>
> Fetch logs for a specific pipeline run.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `pipeline_run_id` | string | yes | Pipeline run ID (returned by `run_pipeline`) |
>
> **Example prompt:** *"Show me the logs for the last run"*

***

## Sync

> **sync\_project\_to\_local**
>
> Pull all remote project files to a local directory.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `local_project_dir` | string | no | Local directory path (default: `.mage-agent/local_sync`) |

> **sync\_status**
>
> Show what has changed between local and remote.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `local_project_dir` | string | no | Local directory path |

> **sync\_local\_to\_remote**
>
> Push locally changed files to the cluster.
>
> | Parameter | Type | Required | Description |
> | - | - | - | - |
> | `local_project_dir` | string | no | Local directory path |

***

## Mage Cloud Backend

Mage Cloud Backend tools are exposed only for installed-agent sessions that have
Cloud handoff credentials. These tools are generated from the Cloud Backend
resource inventory, so call `mage_tool_contract` before invoking a Cloud tool
when payload or query shape matters.

> **Cloud Backend contracts**
>
> Cloud Backend create and update requests use the same resource-root envelope
> that the backend API expects. Do not send flat mutation payloads.
>
> | Tool | Mutation payload root |
> | - | - |
> | `teams` | `team` |
> | `cloud_users` | `user` |
> | `team_memberships` | `team_membership` |
> | `invitations` | `invitation` |
>
> Role fields use Cloud Backend integer values: owner `0`, admin `1`,
> editor `2`, viewer `3`.
>
> For example, creating a team invitation uses:
>
> ```json theme={"system"}
> {
>   "action": "create",
>   "identifiers": {
>     "team_id": "42"
>   },
>   "payload": {
>     "invitation": {
>       "invitee_email": "user@example.com",
>       "role": 3
>     }
>   }
> }
> ```
>
> `cluster_stats` requires `query.cluster_uuid`; `cluster_uuid` is not a path
> identifier for that tool.
>
> `cloud_users` supports `list`, `get`, and `update`. It does not expose
> `create`, because Cloud Backend treats `users.create` as the public signup
> path rather than a private team-admin user creation API.

***

## Error responses

All tool errors return a structured envelope:

```json theme={"system"}
{
  "status": "error",
  "error": {
    "code": "AUTH_EXPIRED",
    "message": "Session token expired. Run `mage-agent login` to re-authenticate."
  }
}
```

| Error code | Cause |
| - | - |
| `CONFIG_INVALID` | Missing or malformed `~/.mage-agent/config.json` |
| `AUTH_REQUIRED` | No credentials found |
| `AUTH_EXPIRED` | 401/403 from cluster — run `mage-agent login` |
| `API_ERROR` | Cluster API returned an error |
| `TOOL_NOT_FOUND` | Requested tool name is not registered |
| `INTERNAL_ERROR` | Unexpected internal failure |


# Quickstart
Source: https://docs.mage.ai/agent/quickstart

Install mage-agent, log in, and make your first AI-assisted pipeline call in under five minutes.

**1. Install mage-agent**

```bash theme={"system"}
pip install mage-agent
```

Verify the install:

```bash theme={"system"}
mage-agent --version
```

**2. Log in to your Mage Pro cluster**

```bash theme={"system"}
mage-agent login
```

The CLI prompts for your cluster URL, email, and password. On success, credentials are saved to `~/.mage-agent/config.json`.

> [!TIP]
>
> To skip the prompts, pass flags directly:
>
> ```bash theme={"system"}
> mage-agent login \
>   --cluster-url https://your-mage-url \
>   --email YOUR_EMAIL \
>   --password YOUR_PASSWORD
> ```

**3. Connect your AI assistant**

Add the following to your IDE's MCP server config and restart the client:

```json theme={"system"}
{
  "mcpServers": {
    "mage-agent": {
      "command": "mage-agent",
      "args": ["mcp"]
    }
  }
}
```

See [IDE Integration](/agent/ide-integration) for client-specific setup instructions.

**4. Try it**

Ask your AI assistant:

> "List all my Mage pipelines"

> "Create a new Python transformer block in the `customer_etl` pipeline"

> "Run the `daily_refresh` pipeline and show me the logs"

The assistant calls the appropriate MCP tool automatically — no manual API calls required.

## Next steps

> **[Sync local files](/agent/sync)**
>
> Mirror your Mage project to your machine for local editing.

> **[MCP tools reference](/agent/mcp-tools)**
>
> See every tool available to your AI assistant.


# Syncing Local Files
Source: https://docs.mage.ai/agent/sync

Mirror your Mage Pro project to a local directory and push edits back to the cluster.

The sync engine lets you work on pipeline files in your local IDE and keep them in sync with your Mage Pro cluster — pulling remote changes down and pushing local edits back up.

## Supported file types

`.py`, `.sql`, `.r`, `.yaml`, `.yml`, `.md`, `.txt`

***

## Pull remote files to local

Download the full remote project tree to a local directory:

```bash theme={"system"}
mage-agent sync-project-to-local --local-project-dir ./my-mage-project
```

If `--local-project-dir` is omitted, the CLI prompts for a path (default: `.mage-agent/local_sync`).

***

## Check sync status

See what has changed between your local copy and the cluster:

```bash theme={"system"}
mage-agent sync-status --local-project-dir ./my-mage-project
```

Files are grouped into four categories:

| Status | Meaning |
| - | - |
| `remote_to_local` | Remote changed since last sync — pull needed |
| `local_to_remote` | Local changed since last sync — push needed |
| `conflicts` | Both sides changed since last sync |
| `in_sync` | No changes on either side |

For a human-readable table instead of JSON:

```bash theme={"system"}
mage-agent sync-status --local-project-dir ./my-mage-project --format-output human
```

***

## Push local changes to the cluster

```bash theme={"system"}
mage-agent sync-local-to-remote --local-project-dir ./my-mage-project
```

Only files flagged as `local_to_remote` are uploaded.

***

## Bidirectional sync with conflict resolution

Run a full two-way sync in one command:

```bash theme={"system"}
mage-agent sync --local-project-dir ./my-mage-project
```

For each conflicting file, mage-agent prompts:

```
pipelines/customer_etl/transformers/clean_names.py — CONFLICT
  Remote modified: 2024-11-01 14:32
  Local modified:  2024-11-01 15:10
  Keep [r]emote / [l]ocal / [s]kip?
```

Choose `r`, `l`, or `s` for each file. Conflicts are never silently overwritten.

***

## Sync via AI assistant

You can also trigger sync operations through your AI assistant using MCP tools:

| Tool | What it does |
| - | - |
| `sync_project_to_local` | Pull remote files to local directory |
| `sync_status` | Show remote/local diff |
| `sync_local_to_remote` | Push local changes to cluster |

Example prompt:

> "Pull the latest remote changes to my local project at `./my-mage-project`"

***

## How sync state is tracked

mage-agent stores a state file at `<local-project-dir>/.mage-agent-sync-state.json` after each sync. This file records the last-known timestamps for all tracked files, enabling accurate detection of what changed on each side. Do not delete or edit it manually.

> [!NOTE]
>
> Files inside hidden directories (`.git`, `.venv`, etc.) and `__pycache__` folders are automatically excluded from sync.


# Troubleshooting
Source: https://docs.mage.ai/agent/troubleshooting

Solutions for common mage-agent issues.

## Authentication errors

> **"AUTH\_EXPIRED" or "AUTH\_REQUIRED" errors**
>
> Your session token has expired or is missing.
>
> ```bash theme={"system"}
> mage-agent login
> ```
>
> This fetches a new token and overwrites the existing config.

> **"CONFIG\_INVALID" error**
>
> The config file at `~/.mage-agent/config.json` is missing or malformed.
>
> Re-run login to recreate it:
>
> ```bash theme={"system"}
> mage-agent login
> ```
>
> Or create the file manually:
>
> ```json theme={"system"}
> {
>   "cluster_url": "https://your-mage-url",
>   "oauth_token": "YOUR_OAUTH_TOKEN"
> }
> ```

***

## MCP / IDE issues

> **MCP tools not showing in the IDE**
>
> 1. Confirm `mage-agent` is accessible from the IDE's runtime PATH — or use an absolute path (see [Installation](/agent/installation#ide-path-requirement)).
> 2. Save the MCP config and **restart** the IDE or agent client.
> 3. Test the server directly in a terminal:
>
> ```bash theme={"system"}
> mage-agent mcp
> ```
>
> If it starts without errors, the issue is the IDE config. If it errors, check authentication first.

> **"command not found: mage-agent" in IDE**
>
> The IDE cannot find the executable. Run:
>
> ```bash theme={"system"}
> which mage-agent
> ```
>
> Copy the full path and use it in your IDE config:
>
> ```json theme={"system"}
> {
>   "mcpServers": {
>     "mage-agent": {
>       "command": "/full/path/to/mage-agent",
>       "args": ["mcp"]
>     }
>   }
> }
> ```

> **Tools are listed but calls fail**
>
> Run login again — the MCP server may have started before credentials were saved, or the token expired during the session:
>
> ```bash theme={"system"}
> mage-agent login
> ```
>
> Then restart the IDE to pick up fresh credentials.

***

## Sync issues

> **Unexpected conflicts after syncing**
>
> Conflicts are detected by comparing timestamps on each side. Small clock differences between your machine and the cluster can cause false positives.
>
> Check what the conflict looks like before choosing a side:
>
> ```bash theme={"system"}
> mage-agent sync-status --local-project-dir ./my-project --format-output human
> ```
>
> Then run `mage-agent sync` and choose `r` (remote) or `l` (local) for each affected file.

> **Files not appearing in sync-status**
>
> Only these file extensions are tracked: `.py`, `.sql`, `.r`, `.yaml`, `.yml`, `.md`, `.txt`.
>
> Also check that the files are not inside a hidden directory (`.git`, `.venv`) or a `__pycache__` folder — those are always excluded.

> **Sync state file is missing or corrupted**
>
> The state file lives at `<local-project-dir>/.mage-agent-sync-state.json`. If it is missing, mage-agent treats all files as new and will re-pull the full project on the next `sync-project-to-local`. Delete it intentionally only if you want a clean baseline.

***

## Enable debug logging

Add `--log-level DEBUG` to any command to see full request/response traces:

```bash theme={"system"}
mage-agent --log-level DEBUG sync-status --local-project-dir ./my-project
mage-agent --log-level DEBUG mcp
```

Or set the environment variable permanently in your shell profile:

```bash theme={"system"}
export MAGE_AGENT_LOG_LEVEL=DEBUG
```

***

## Still stuck?

Open an issue at [github.com/mage-ai/mage-agent](https://github.com/mage-ai/mage-agent/issues) and include the debug log output.


# Agents
Source: https://docs.mage.ai/ai/agents

Create YAML-backed AI agents with a backing pipeline and API endpoint.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents)

Agents are AI blocks stored as YAML files under the project `agents/` folder. Each agent YAML file is the source of truth for the agent name, description, default instructions, backing pipeline, saved actions, status, and published endpoint metadata.

Creating an agent from the Agents page or Agents API creates one backing pipeline named after the agent and one YAML block file in `agents/`. When a UUID is omitted, Mage derives it from the cleaned agent name. If that UUID already exists, the request is rejected so you can choose a different name or UUID.

```yaml theme={"system"}
type: agent
uuid: customer_support_agent_a1b2
name: Customer support agent
description: Routes support questions.
instructions: |
  Classify the request and recommend the next support workflow.
pipeline_uuid: customer_support_agent
actions:
  - pipeline_uuid: support_workflows
    block_uuid: triage_ticket
status: draft
endpoint: {}
```

## Fields

Use `instructions` for the saved default prompt. Runtime requests use `message`, which can be sent when running the agent. Existing `type: agent` AI blocks can still use `message` as the per-run user request.

* `name`: Display name for the agent.
* `description`: Short summary shown in the Agents list and detail view.
* `instructions`: Default instructions stored with the agent.
* `pipeline_uuid`: Backing pipeline used to execute and publish the agent.
* `actions`: List of action references. Mage stores this list but runtime handling is defined separately.
* `status`: `draft`, `published`, or `paused`.
* `endpoint`: API trigger metadata written when the agent is published.

Only YAML files with `type: agent` and `pipeline_uuid` appear in the Agents list. This keeps standalone agent blocks hidden unless they are associated with a backing pipeline.

## Publishing

Publishing an agent marks it live by upserting an API trigger on the backing pipeline. Mage stores the trigger metadata in the agent YAML under `endpoint`, including the schedule ID and streaming run path.

Publishing is idempotent. Re-publishing the same agent reuses the existing API trigger when possible instead of creating duplicate triggers.

## Running

Publish the agent to enable its streaming endpoint. On the agent's **Endpoint** page, edit the JSON request body, then select **Run agent**. Set `pipeline_run.variables.message` to the request for this run and add any other runtime variables your agent needs:

```json theme={"system"}
{
  "pipeline_run": {
    "variables": {
      "message": "Summarize yesterday's support tickets."
    }
  }
}
```

The button POSTs that payload to the displayed streaming endpoint and displays the server-sent events (SSE) from the same response. The endpoint still creates a backing pipeline run, so its run ID and execution history remain available. See [Run agent](/extensibility/api-reference/agents/run-agent) for authentication and API examples.

## Chatting with an agent

**Chat with agent** opens a Sidekick conversation with the agent's saved instructions and allowed actions. The conversation carries out the agent's work directly; it does not start the agent's own backing pipeline. When an action is needed, Sidekick can execute the configured action pipeline and report its run separately. Asking the agent to run does not make every assigned action mandatory: the agent chooses actions based on its instructions and the available data.


# AI Clients
Source: https://docs.mage.ai/ai/ai-client

Use and enable different AI clients.

## AI Client

You now have the option to select from various AI clients to harness
[the capabilities of Mage AI](https://docs.mage.ai/guides/ai/overview), as detailed in the
Mage AI capabilities documentation. Currently, we offer support for OpenAI and Hugging Face,
with the promise of additional AI clients being added in the future.

## Use Hugging Face Client

### Setup

#### Hugging Face Inference Endpoint

In order to utilize the Hugging Face AI Client, it is necessary to establish a Hugging Face inference endpoint.
You can set it up following [this guide](https://huggingface.co/inference-endpoints).

This process is quite straightforward. It entails

* selecting the specific model you wish to use,
* determining the hosting environment (AWS or Azure),
* specifying the geographical region,
* choosing the type of GPU.

For your convenience and based on our testing, we recommend using the
"mistralai/Mistral-7B-Instruct-v0.1" model.

Once the Inference endpoint is operational, it will provide you with an API URL and a corresponding
token for establishing a secure connection.

#### Mage Project Setup

Within your Mage project's metadata YAML configuration, please include the subsequent "ai\_config" section:

```
ai_config:
  mode: 'hugging_face'
  open_ai_config:
    openai_api_key: key
  hugging_face_config:
    huggingface_api: api_url
    huggingface_inference_api_token: api_token
```

The "mode" parameter determines your selection of the AI client to be employed.
It can be specified as either "open\_ai" or "hugging\_face," with the default value being set to "open\_ai."

"hugging\_face\_config" as a mandatory configuration if you choose to use the hugging face client.
This configuration includes the two essential elements obtained from the Hugging Face inference
endpoint, namely, the API and Token.

You are ready to go once the "ai\_config" is setup. At this point,
you can fully leverage Mage AI's capabilities, such as generating blocks
with text description, automatically write comments for your functions, etc.

## How to add a new AI Client

You may find it necessary to employ an AI client other than those offered by OpenAI and Hugging Face.
Additionally, you might wish to make direct calls to your Language Model (LLM).
This can be accomplished by enabling a new AI client for your specific needs.

This is an [example PR](https://github.com/mage-ai/mage-ai/pull/3850).

### Create new AI config

Create a dedicated configuration to save the params required to connect to LLM in the [config.py](https://github.com/mage-ai/mage-ai/blob/92c372b24e08148863d799d9afcdd44483c11c89/mage_ai/orchestration/ai/config.py#L19).
For instance, when using the Hugging Face client, the LLM is hosted within the
inference endpoint, mandating both the API and Token for invoking the service
for inference. In the OpenAI client, the OpenAI key is required to facilitate model inference.

### Create dedicated AI Client

Inherit the [AIClient interface](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/ai/ai_client.py)
and implements the two required functions: “inference\_with\_prompt” and “find\_block\_params”.

* Inference\_with\_prompt function does the LLM model inference.
  It takes the prompt template, required variables being used in the prompt and
  return the inference result.
* Find\_block\_params function does a multi classification based on code description
  to generate required types including block\_type, pipeline\_type, language,
  action type and data source.

You can read your configuration in the Setup function and initialize the client
to talk to your service.

### Enable in llm\_pipeline\_wizard

The last action to take is modifying the Setup function within "[mage\_ai/ai/llm\_pipeline\_wizard.py](https://github.com/mage-ai/mage-ai/blob/92c372b24e08148863d799d9afcdd44483c11c89/mage_ai/ai/llm_pipeline_wizard.py#L195)"
to introduce a new mode of your client and initialize your AI client.


# AI Blocks
Source: https://docs.mage.ai/ai/blocks

Use AI blocks to turn prompts, upstream data, and pipeline context into reusable pipeline outputs.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=ai-blocks)

[AI Blocks](https://www.youtube.com/embed/28pRdPk2DAQ)

AI blocks bring generative AI into the same pipeline canvas as your loaders, transformers, exporters, and monitors. Use them to summarize data, classify records, draft messages, create structured JSON, generate code, or hand work to Mage Pro AI Sidekick when a task needs project context.

Because AI blocks run inside a Mage pipeline, their answers can become real pipeline outputs. Downstream blocks can consume the final response just like any other block result.

## Create an AI Block

Add an AI block to a pipeline, then describe the job in YAML.

```yaml theme={"system"}
prompt: |
  Summarize the customer feedback from upstream blocks.
  Return the top themes and recommended next actions.
```

Run the block from the editor, or run the full pipeline from a trigger. Mage sends the prompt to the configured AI runtime and stores the final response as the block output.

Use AI blocks when you want to:

* summarize reports, logs, feedback, or operational data
* extract structured fields from text
* classify rows or events into business categories
* generate code or SQL for a controlled workflow
* turn upstream block outputs into plain-language recommendations
* automate a decision step before the next pipeline block runs

## Use an Agent Block

Create an Agent block when you want a Mage Pro pipeline step to run through Mage Pro AI Sidekick. Agent blocks are useful when the block should understand pipeline context, inspect upstream block outputs, and produce a final answer with the same project awareness users expect from Sidekick.

```yaml theme={"system"}
prompt: |
  Use the upstream customer rows to summarize churn risk.
message: Summarize the highest-risk customers and recommend next steps.
settings:
  speed: fast
output:
  level: default
runtime:
  report_name: weekly_churn_review
```

Agent blocks use the block type `agent` and are saved as YAML under the `agents/` folder. New Agent block YAML does not need a top-level `type: agent` field; existing AI blocks that already use YAML `type: agent` still run for compatibility. Agent blocks run as part of the pipeline and return the final Sidekick answer as the block output. While Sidekick works, Mage prints progress and streamed response events to the block logs so users can see that the agent is running.

Use Agent blocks when the task benefits from pipeline context:

* "Read the upstream customer output and explain the highest churn risk."
* "Review this pipeline run's logs and summarize what failed."
* "Compare the two upstream datasets and write a short executive summary."
* "Use the runtime variables and upstream rows to draft a customer follow-up."

## Fields

### `prompt` and `instructions`

The main instruction for a standard AI block is `prompt`. YAML-backed Agents use `instructions` for the saved default prompt stored in the `agents/` folder. Keep either field clear and specific. Multi-line prompts are easier to read and maintain.

```yaml theme={"system"}
prompt: |
  Write a three-bullet summary of the upstream sales report.
  Include one recommended action for the revenue team.
```

```yaml theme={"system"}
type: agent
instructions: |
  You are a customer retention analyst.
```

### `message`

An optional user-style request for `type: agent` blocks. Use `instructions` for saved standing instructions on YAML-backed Agents, or `prompt` on standalone AI blocks, and use `message` for the specific request to answer on this run.

```yaml theme={"system"}
prompt: |
  You are a customer retention analyst.
message: Which accounts need attention this week?
```

### `settings`

Controls how the AI runtime should behave. For agent blocks, `speed: fast` is a good default when you want a quick operational answer.

```yaml theme={"system"}
settings:
  speed: fast
```

### `runtime`

Optional context that travels with the block run. Use it for business labels, report names, or request metadata that helps the AI understand the job.

```yaml theme={"system"}
runtime:
  customer_segment: enterprise
  report_name: renewal_risk
```

### `output`

Use `output` when you need the response to follow a predictable shape or control agent block logs.

For Agent blocks, `output.level` controls how much Sidekick runtime output Mage prints while the block runs:

* `default`: print streamed answer, narration/status, tool call, and error events.
* `all`: print every runtime event, including completion and other low-level events.
* `none`: do not print Sidekick runtime output; the final answer still becomes the block output.

```yaml theme={"system"}
prompt: Review the upstream data and summarize the result.
message: What changed since the last run?
output:
  level: all
```

Use `format` when a non-agent AI block should return structured data.

```yaml theme={"system"}
prompt: Extract customer details from the text.
output:
  format:
    name:
      type: string
      description: Customer's full name
    email:
      type: string
      format: email
    priority:
      type: string
      enum:
        - low
        - medium
        - high
```

### `validation`

Use validation when the answer has to meet a business rule before it moves downstream.

```yaml theme={"system"}
prompt: Draft a customer escalation summary.
validation: Make sure the response includes the customer name, issue, impact, and next step.
```

### `tools`

Use tools when a standard AI block should choose and trigger another pipeline as part of the workflow.

```yaml theme={"system"}
prompt: Analyze the request and choose the next processing pipeline.
tools:
  prompt: Choose the best pipeline for the request.
  required: true
  pipelines:
    - uuid: customer_follow_up
      description: Creates customer follow-up tasks.
      variables:
        customer_id:
          type: string
```

## Upstream Data

AI and Agent blocks can sit downstream of regular Mage blocks. For Agent blocks, Sidekick receives references to upstream block outputs and can inspect those outputs through Mage instead of copying raw datasets into the prompt.

```yaml theme={"system"}
prompt: |
  Read the upstream customer scores and event history.
  Explain the biggest churn risk in plain English.
message: What should the account team do next?
settings:
  speed: fast
```

The final answer becomes the AI block output. Downstream blocks can store it, send it to another system, or combine it with additional data.

## Examples

### Minimal Summary

```yaml theme={"system"}
prompt: Summarize this document in five bullet points.
```

### Structured Extraction

```yaml theme={"system"}
prompt: Extract the renewal request from the customer note.
output:
  format:
    customer_name:
      type: string
    renewal_date:
      type: string
      format: date
    requested_discount:
      type: number
    risk_level:
      type: string
      enum:
        - low
        - medium
        - high
```

### Agent Block With Upstream Outputs

```yaml theme={"system"}
prompt: |
  You are reviewing customer health data from upstream blocks.
  Use the available customer scores and recent events to identify retention risk.
message: Summarize the top risks and recommended outreach.
settings:
  speed: fast
runtime:
  audience: account_team
```

### Agent Block With No Message

```yaml theme={"system"}
prompt: |
  Use the upstream rows to produce a short operational summary.
settings:
  speed: fast
```

Use this pattern when the prompt itself is the complete request.

## Best Practices

* Give the block one clear job.
* Put reusable instructions in `instructions` for YAML-backed Agents or `prompt` for standalone AI blocks.
* Put the run-specific ask in `message` for Agent blocks.
* Use upstream blocks for data and the AI block for interpretation.
* Use structured output when downstream blocks need predictable fields.
* Keep runtime context business-focused, such as segment, audience, or report name.
* Use Agent blocks when project context or upstream block output matters.


# Customized AI resources for training and inference
Source: https://docs.mage.ai/ai/custom-resources



## Custom resources

Customized GPU accelerated resources for running AI/ML/LLM pipelines.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=ai-resources)

## Inference endpoints

Deploy high-performance, low-latency API endpoints for executing blocks and returning output data,
such as inference endpoints.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=ai-endpoints)


# Machine learning pipeline tutorial
Source: https://docs.mage.ai/ai/ml/train-model

Build a machine learning pipeline to train a model on the Titanic dataset.

![Titanic](https://media.graphcms.com/TIobzTjRniW8wUAna6Ht)

In this tutorial, we’ll create a pipeline that does the following:

1. Load data from an online endpoint
2. Select columns and fill in missing values
3. Train a model to predict which passengers will survive

If you prefer to skip the tutorial and view the finished code, follow
[this guide](/guides/train/complete-project).

If you haven’t setup a project before, check out the [setup guide](/getting-started/setup)
before starting.

## 1. Setup

### 1a. Add Python packages to project

In the left sidebar (aka file browser), click on the `requirements.txt` file
under the `demo_project/` folder.

![requirements](https://mage-ai.github.io/assets/requirements.png)

Then add the following dependencies to that file:

```text theme={"system"}
matplotlib
requests
scikit-learn
```

Then, save the file by pressing `⌘ + S`.

### 2a. Install dependencies

The simplest way is to run pip install from the tool.

Add a scratchpad block by pressing the `+ Scratchpad` button. Then run the
following command:

```bash theme={"system"}
pip install -r demo_project/requirements.txt
```

Alternatively, here are other ways of installing dependencies (depending on if
you are using Docker or not):

#### Docker

Get the name of the container that is running the tool:

```bash theme={"system"}
docker ps
```

Sample output:

```bash Sample output theme={"system"}
CONTAINER ID   IMAGE       COMMAND                  CREATED         STATUS         PORTS     NAMES
214e1155f5c3   mage/data   "python mage_ai/comm…"   5 seconds ago   Up 2 seconds             mage-ai_server_run_6f8d367ac405
```

The container name in the above sample output is
`mage-ai_server_run_6f8d367ac405`.

Then run this command to install Python packages in the
`demo_project/requirements.txt` file:

```bash theme={"system"}
docker exec [container_name] pip3 install -r demo_project/requirements.txt
```

### pip

If you aren’t using Docker, just run the following command in your terminal:

```bash theme={"system"}
pip3 install -r demo_project/requirements.txt
```

## 2. Create new pipeline

In the top left corner, click `File > New pipeline`. Then, click the name of the
pipeline next to the green dot to rename it to `titanic survivors`.

![create new pipeline](https://mage-ai.github.io/assets/create-new-pipeline.gif)

## 3. Play around with scratchpad

There are 4 buttons, click on the `+ Scratchpad` button to add a block.

Paste the following sample code in the block:

```python theme={"system"}
import matplotlib.pyplot as plt
import numpy as np


t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t, s)

plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('About as simple as it gets, folks')
plt.grid(True)
plt.show()
```

Then click the `Play button` on the right side of the block to run the code.
Alternatively, you can use the following keyboard shortcuts to execute code in
the block:

* ⌘ + Enter
* Control + Enter
* Shift + Enter (run code and add a new block)

![](https://mage-ai.github.io/assets/scratchpad.gif)

Now that we’re done with the scratchpad, we can leave it there or delete it. To
delete a block, click the trash can icon on the right side or use the keyboard
shortcut by typing the letter D and then D again.

## 4. Load data

1. Click the `+ Data loader` button, select `Python`, then click the template
   called `API`.
2. Rename the block to `load dataset`.
3. In the function named `load_data_from_api`, set the `url` variable to:
   `https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv`.
4. Run the block by clicking the play icon button or using the keyboard
   shortcuts `⌘ + Enter`, `Control + Enter`, or `Shift + Enter`.

After you run the block (⌘ + Enter), you can immediately see a sample of the
data in the block’s output.

![load data](https://mage-ai.github.io/assets/load-data.gif)

Here is what the code should look like:

```python theme={"system"}
import io
import pandas as pd
import requests
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_api(**kwargs) -> DataFrame:
    """
    Template for loading data from API
    """
    url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'

    response = requests.get(url)
    return pd.read_csv(io.StringIO(response.text), sep=',')


@test
def test_output(df) -> None:
    """
    Template code for testing the output of the block.
    """
    assert df is not None, 'The output is undefined'
```

## 5. Transform data

We’re going to select numerical columns from the original dataset, then fill in
missing values for those columns (aka impute).

1. Click the `+ Transformer` button, select `Python`, then click
   `Generic (no template)`.
2. Rename the block to `extract and impute numbers`.
3. Paste the following code in the block:

```python theme={"system"}
from pandas import DataFrame
import math


if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


def select_number_columns(df: DataFrame) -> DataFrame:
    return df[['Age', 'Fare', 'Parch', 'Pclass', 'SibSp', 'Survived']]


def fill_missing_values_with_median(df: DataFrame) -> DataFrame:
    for col in df.columns:
        values = sorted(df[col].dropna().tolist())
        median_age = values[math.floor(len(values) / 2)]
        df[[col]] = df[[col]].fillna(median_age)
    return df


@transformer
def transform_df(df: DataFrame, *args) -> DataFrame:
    return fill_missing_values_with_median(select_number_columns(df))
```

After you run the block (⌘ + Enter), you can immediately see a sample of the
data in the block’s output.

![Transform data](https://mage-ai.github.io/assets/transform-data.gif)

## 6. Train model

In this part, we’re going to accomplish the following:

1. Split the dataset into a training set and a test set.
2. Train logistic regression model.
3. Calculate the model’s accuracy score.
4. Save the training set, test set, and model artifact to disk.

Here are the steps to take:

1. Add a new data exporter block by clicking `+ Data exporter` button, select
   `Python`, then click `Generic (no template)`.
2. Rename the block to `train model`.
3. Paste the following code in the block:

```python theme={"system"}
from pandas import DataFrame
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
import os
import pickle

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


LABEL_COLUMN = 'Survived'


def build_training_and_test_set(df: DataFrame) -> None:
    X = df.drop(columns=[LABEL_COLUMN])
    y = df[LABEL_COLUMN]

    return train_test_split(X, y)


def train_model(X, y) -> None:
    model = LogisticRegression()
    model.fit(X, y)

    return model


def score_model(model, X, y) -> None:
    y_pred = model.predict(X)

    return accuracy_score(y, y_pred)


@data_exporter
def export_data(df: DataFrame) -> None:
    X_train, X_test, y_train, y_test = build_training_and_test_set(df)
    model = train_model(X_train, y_train)

    score = score_model(model, X_test, y_test)
    print(f'Accuracy: {score}')

    cwd = os.getcwd()
    filename = f'{cwd}/finalized_model.lib'
    print(f'Saving model to {filename}')
    pickle.dump(model, open(filename, 'wb'))

    print(f'Saving training and test set')
    X_train.to_csv(f'{cwd}/X_train')
    X_test.to_csv(f'{cwd}/X_test')
    y_train.to_csv(f'{cwd}/y_train')
    y_test.to_csv(f'{cwd}/y_test')
```

Run the block (⌘ + Enter).

![Train model](https://mage-ai.github.io/assets/train-model.gif)

## 7. Run pipeline

We can now run the entire pipeline end-to-end. In your terminal, execute the
following command:

```bash Docker theme={"system"}
./scripts/run.sh demo_project titanic_survivors
```

```bash pip theme={"system"}
bash mage run demo_project titanic_survivors
```

You can also run the pipeline from the UI. Click on the **Execute pipeline&#x20;**&#xA;from right bottom panel.

![](https://mage-ai.github.io/assets/execute-pipeline-button.png)

Your output should look something like this:

```text theme={"system"}
Executing data_loader block: load_dataset...DONE
Executing transformer block: extract_and_impute_numbers...DONE
Executing data_exporter block: train_model...Accuracy: 0.757847533632287
Saving model to /home/src/finalized_model.lib
Saving training and test set
DONE
```

***

## Congratulations!

You’ve successfully built an ML pipeline that consists of modular code blocks
and is reproducible in any environment.

If you have more questions or ideas, please live chat with us in
[Slack](https://www.mage.ai/chat).


# Retrieval Augmented Generation (RAG) pipeline builder
Source: https://docs.mage.ai/ai/rag-pipeline

Build and deploy RAG pipelines with ease.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=rag-pipeline)

![Mage Pro RAG pipeline builder](https://mage-ai.github.io/assets/pro/features/rag-pipeline.jpg)


# Using AI (artificial intelligence)
Source: https://docs.mage.ai/ai/setup

Use AI to perform various actions in Mage.

![Using artificial intelligence](https://steamuserimages-a.akamaihd.net/ugc/813376249149384255/EC3222795DC2961599F7FC0B69E0241C8EC7293E/?imw=5000\&imh=5000\&ima=fit\&impolicy=Letterbox\&imcolor=%23000000\&letterbox=false)

Mage uses AI to help you build data pipelines faster. With our OpenAI integration and Mage Pro AI Chat, you can automate the tedious parts of pipeline development and focus on the fun stuff.

> [!NOTE]
>
> Some AI actions that use direct OpenAI integration require an OpenAI API key. Mage Pro AI Chat uses Mage Pro API proxy authentication for model requests.

We currently support the following AI features:

* Generate pipelines
* Generate blocks
* Chat with AI in Mage Pro
* Document code
  * Block documentation
  * Block comments
  * Pipeline documentation
  * Simultaneously document *all* blocks in a pipeline

![](https://mage-ai.github.io/assets/overview/ai-demo.gif)

To see examples of how to use AI in Mage, check out our [AI guides](/guides/ai/overview).

## Setup

For AI actions that use direct OpenAI integration, add an OpenAI API key to your project.

> [!NOTE]
>
> Read [OpenAI’s documentation](https://help.openai.com/en/articles/4936850-where-do-i-find-my-secret-api-key)
> to get your API key.

Once you have your OpenAI API key, open the account menu and select **Settings** → **Workspace** → **Preferences**. Under **AI services**, enter the key in **OpenAI Key**.

## AI Chat in Mage Pro

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=ai-sidekick)

Mage Pro includes AI Chat for workspace assistance. Open AI Chat from Mage Pro, or visit `/ai/chat` in an authenticated workspace as a user with the Editor role or higher.

### Build dbt workflows with AI Chat

Ask AI Chat to inspect an existing dbt project, create or update models and tests, build Mage dbt blocks, or run and validate a workflow. For example:

```text theme={"system"}
Create a dbt project for daily customer revenue, add tests for its final model,
and create a Mage pipeline that runs the production reporting scope. Then run it
and summarize the result.
```

AI Chat reuses matching projects, pipelines, and blocks when they already exist, so a follow-up request can update or validate the workflow without creating duplicates.

### Models, selectors, and commands

* When you ask to run one named model or one snapshot file, AI Chat creates a Mage SQL dbt block that points directly to that file.
* When you ask for a selector, multiple models, tests, or bulk snapshots, it creates a Mage YAML dbt block with the requested scope. It includes upstream models only when they are needed for the requested block to run on its own.
* `dbt docs generate` is not currently supported by Mage's native dbt blocks. AI Chat explains this limitation instead of creating a block that cannot run.

### When AI Chat runs dbt

Creating or editing a dbt workflow does not run it automatically, even when the request says to create a block that runs a model or to add tests. To run it in the same request, explicitly say **run**, **execute**, **test**, or **validate through execution**; no separate mid-request approval is needed.

When you authorize execution, AI Chat runs the requested block or pipeline and reports the result only after it finishes. Its response includes the outcome, relevant run identifiers, and a redacted dbt command when available. If you explicitly ask to run, test, or validate again, it starts a fresh execution; if a repair flow is incomplete or fails safely, it retries the original run.

### Credentials and production-quality work

AI Chat does not display credentials, resolved environment-variable values, or raw dbt profile content. It asks for an adapter, profile name, or target only when that non-secret choice cannot be determined safely, and it tells you when it cannot verify that a project can run with its configured profile.

Every dbt request uses existing project conventions, checks its changes, and explains assumptions. Requests for production readiness—or higher-risk work such as incremental models, snapshots, backfills, sensitive data, destructive schema changes, and large or costly workloads—also receive deeper design, data-quality, security, reliability, and operational checks. AI Chat may ask for a decision only when it is necessary for a safe implementation. A production-quality result requires an explicitly authorized fresh execution and all applicable checks to pass; asking for production-ready work alone does not start execution.

For each chat turn, Mage captures the visible non-chat text from the current browser page, along with the page title and URL, so AI Chat can answer page-relative prompts such as "explain this page" or "what am I looking at?" The captured context is scoped to the authenticated user and chat session, excludes the chat UI, and is capped at 20,000 characters.

The AI Chat sidebar lists active and archived chats, lets you search chat history by title, and loads older chats with **Show more** when more results are available. Mage Pro remembers the selected chat and whether you were viewing active or archived chats for the current workspace, then restores that state when you refresh or reopen AI Chat.

When AI Chat replies with Markdown image links that resolve through Mage Pro's authenticated preview flow, AI Chat renders those image previews inline. Markdown links to other workspace files still open in the code editor.

### Inspect delegated subagent work

When a chat turn delegates work, select **Show work** in the AI Chat header. Its **Subagents** summary shows how many subagents are working and how many are done.

Selecting **Subagents** opens the right panel, where subagents are grouped into **Active** and **Done**. Each row shows the subagent name and its latest activity. Select a row to inspect that subagent's turn output, tool activity, and generated outputs; use the back button to return to the grouped list.

AI Chat supports session-scoped file attachments from the composer File action or paste, folder attachments from the composer Folder action, and file or folder attachments by drag and drop. Uploaded attachments remain available on follow-up messages in the same chat session, even after the composer clears for the next message. Folder uploads appear as a single folder attachment in the chat; for browser-selected folders, Mage Pro packages every browser-exposed file into one zip archive before upload, preserves the relative paths inside that archive, and does not filter files from the selected folder. AI Chat receives an uploaded-file manifest and bounded text previews from readable non-archive attachments so it can answer questions about uploaded files without guessing from workspace files.

AI Chat supports push-to-record voice dictation when the Mage Pro API OpenAI proxy is configured and the signed-in user has the same Editor-or-higher permission required to send AI Chat messages. Click the microphone in the composer, then click the stop control to insert the transcription into the draft for editing, or click the send arrow while recording to stop capture, transcribe the audio, and send or queue the dictated message. Audio is sent from the browser to the Mage Pro workspace, then to the Mage Pro API proxy; the workspace and browser never receive the raw OpenAI API key.

Use the execution settings menu beside the chat composer to choose a **Speed** and **Effort** level for the current chat. Choose **Standard** for the default response tier, or **Fast** to use the priority service tier for that chat. Choose **Light**, **Medium**, **High**, or **Extra High** effort based on how much model effort the task needs. The menu stays open after a selection so you can confirm both checked choices, and its button displays the active effort level. When **Fast** is selected, the button appends the speed after the effort level, such as **Medium Fast**.

For guided Sidekick workflows in Mage Pro, AI Chat can show inline forms directly in the chat transcript. Use these forms to choose sources, streams, destinations, and connector settings without leaving the conversation. Required fields must be completed before continuing, large option lists include filtering, and **Cancel** stops the guided setup. Submitting or cancelling the form resolves the original Sidekick turn instead of adding a separate chat message. After submission, AI Chat replaces the form with a submitted summary that shows safe submitted values, truncates long values, and masks sensitive values such as passwords or credentials.

When AI Chat returns a project-local PNG, JPEG, GIF, or WebP path, AI Chat renders the image inline. The path can be on its own line, embedded in prose, such as `Here it is: /home/src/demo/cyborg_mage.png`, or used as a Markdown link target, such as `[output/anime_styled_mage.png](output/anime_styled_mage.png)`. The preview is served through a short-lived authenticated URL, so the image file does not need a public URL.

When Sidekick media tools generate image, audio, or video files, AI Chat renders them at the end of the assistant answer as chat artifacts instead of only inside tool-usage details. Images render inline, audio renders with browser playback controls, and video renders with browser playback controls. Media previews use five-minute authenticated tokens. Supported preview files are GIF, JPEG, PNG, and WebP images up to 25 MB; AAC, MP3, Opus, WAV, and WebM audio up to 50 MB; and MOV, MP4, and WebM video up to 250 MB. Files must be under the current project or the current Sidekick media directory.

Mage Pro administrators configure AI Chat connectivity and model access when the
workspace is deployed. Per-chat execution settings take precedence over the
deployment defaults. Codex CLI commands launched independently from the Mage
terminal continue to use that terminal's own Codex configuration.

When Codex model capacity is temporarily unavailable, AI Chat shows a **Waiting for model capacity** status and automatically retries the run within the configured retry window. If the retry budget is exhausted, AI Chat shows the rate-limit error.

Voice dictation is available when it is enabled by the Mage Pro administrator.
The workspace validates supported audio formats and applies configured upload,
duration, and usage limits before sending audio for transcription.

## Generate pipeline using AI

When creating a new pipeline, select the option labeled **Using AI**.

Then, type the description of what the pipeline should do.

For example:

*Load data from an API,
then clean the column names,
and finally export the dataframe to PostgreSQL.*

![Generate pipeline using AI](https://mage-ai.github.io/assets/guides/ai/generate-pipeline.png)

## Generate block using AI

> [!NOTE]
>
> You must turn on the feature named `add_new_block_v2`
> in **Settings** → **Workspace** → **Preferences**, available from the account menu.

When adding a new block, type in the description of what you want the block to do.
In the autocomplete dropdown list, select the 1st option with a label starting with

**Generate block using AI: ...**

![Generate block using AI](https://mage-ai.github.io/assets/guides/ai/generate-block.png)

## Document code using AI

![Add documentation using AI](https://mage-ai.github.io/assets/guides/ai/document.png)

### Add documentation for a block

On the edit pipeline page in the top right corner of a block, click the AI actions icon.
Select the option labeled **Document block**.

### Add documentation for a pipeline and all its blocks

On the edit pipeline page in the top right corner of a block, click the AI actions icon.
Select the option labeled **Document pipeline and all blocks**.

### Add comments in a block

On the edit pipeline page in the top right corner of a block, click the AI actions icon.
Select the option labeled **Add comments in code**.


# AI Sidekick
Source: https://docs.mage.ai/ai/sidekick/index

Use Mage Pro AI Sidekick to build, debug, operate, explore, and automate data work from inside Mage.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=ai-sidekick)

AI Sidekick is the Mage Pro assistant built for data teams. It understands the
Mage page you are viewing, the project you are working in, your pipelines,
blocks, runs, logs, data outputs, connected sources, docs, and chat history, then
turns natural-language requests into useful answers and real Mage actions.

Use Sidekick when you want to move from an idea to a working pipeline, explain a
production failure, explore data without opening a separate tool, generate an
Analytics chart, update project files, or ask a teammate-style question about
what is happening in Mage.

![AI Sidekick creating Titanic survival charts from inside Mage Pro.](https://mintcdn.com/mage/xlfFBQh2VT61fQmA/media/ai-sidekick/ai-sidekick-hero.png?fit=max\&auto=format\&n=xlfFBQh2VT61fQmA\&q=85\&s=2341d4aa5a13fc36e4df9228526a0553)

In this example, Sidekick is working from the open analytics workflow. It turns
plain-language chart requests into visual analysis, keeps the generated charts
in context, and gives you a path to add useful results to a dashboard without
leaving Mage.

## How to Access AI Sidekick

AI Sidekick is available in Mage Pro wherever contextual help or automation is
most useful.

1. Open a Mage Pro workspace.
2. Open a pipeline, block, run, Analytics page, data preview, or code page.
3. Click the AI Sidekick entry point in the page header, right panel, output
   toolbar, chart workflow, or contextual action menu.
4. Ask in plain English, choose a suggested prompt, or attach the current page,
   file, block output, chart, or data preview as context.

Sidekick starts with the context of the page you are on. On a pipeline page, it
can see the pipeline shape and active block. On an Analytics page, it can keep
the target dashboard or chart in focus. In code-oriented surfaces, it can work
with the active file and selected source text. When the prompt is ambiguous, it
can ask you to choose the right source, table, pipeline, or action before making
changes.

## How to Use It

Start with the outcome you want, not the implementation details.

```text theme={"system"}
Create a pipeline that loads Salesforce opportunities, joins them with Stripe
invoices, calculates expansion revenue, and runs every weekday morning.
```

```text theme={"system"}
This run failed overnight. Explain the root cause, fix the broken block, and
retry the failed work from the same run.
```

```text theme={"system"}
Show active customers by region from Snowflake, filter to enterprise accounts,
then make a bar chart for the Analytics dashboard.
```

```text theme={"system"}
Open the current block output, find outliers in order_value, and explain what
changed from the previous run.
```

Sidekick streams its progress as it works. For changes that affect Mage objects
or project files, it can read back the result, refresh the open page, and keep
the conversation tied to the exact run, block, preview, chart, or file it used.

## What AI Sidekick Can Do

### Build Pipelines from a Prompt

Describe the data workflow you want and Sidekick can create the pipeline,
choose the right block types, generate block code, wire dependencies, apply
templates, and set up the supporting configuration. It can work with batch
pipelines, streaming pipelines, integration pipelines, dbt workflows, dynamic
blocks, callbacks, conditionals, sensors, charts, global data products, and
custom project code.

Examples:

* "Build a daily customer health pipeline from HubSpot, Stripe, and Snowflake."
* "Add a dbt model step after this loader and run only the selected model."
* "Create a streaming pipeline for Kafka events and write clean records to S3."
* "Split this block into dynamic children by country, then reduce the results."

### Generate, Edit, and Repair Blocks

Sidekick can create blocks, update existing logic, refactor code, explain what a
block does, change dependencies, and keep block metadata aligned with the
pipeline. It understands common Mage block languages and patterns, including
Python, SQL, R, YAML, Markdown, dbt, and data integration templates.

Use it for everyday engineering work:

* turn a plain-language transform into reusable code
* add retries, timeouts, variables, secrets, or execution settings
* convert exploratory code into production-ready blocks
* update upstream or downstream dependencies
* fix syntax errors, runtime exceptions, and broken imports
* preserve the current editor context while applying file-backed updates

### Debug Runs and Self-Heal Failures

When a pipeline run fails, Sidekick can inspect the run, block statuses, logs,
outputs, and recent changes, then explain what happened in terms a data team can
act on. If you ask it to fix the problem, it can update the broken code and
retry the failed or incomplete work on the original run instead of creating a
separate run with different context.

Examples:

* "Why did this run fail?"
* "Fix the failed block and retry the downstream blocks."
* "Summarize all ERROR logs for the last three runs."
* "Find the schema change that caused this exporter to break."

### Explore Data in Chat

Sidekick can discover authorized data sources and pipeline outputs, inspect
schemas, ask for missing warehouse, database, schema, dataset, table, or catalog
selection, and open a windowed data preview directly in chat. For SQL-capable
sources, it can create read-only query previews for grouped results, joins,
derived columns, filters, and comparisons.

The preview stays interactive. You can sort, filter, select columns, ask
follow-up questions, return to a parent preview, or branch into another
refinement without copying a full dataset into the conversation.

Examples:

* "Show revenue by month from the warehouse."
* "Only include enterprise accounts and sort by ARR descending."
* "Compare this block output against yesterday's run."
* "Count rows by status, then open the result as a table."

### Create Analytics and Data Products

Sidekick can turn pipeline outputs, data previews, SQL results, and existing
charts into Analytics assets. It can generate Plotly code, create chart metadata,
save bounded chart-ready datasets, attach charts to the right dashboard, and
refresh the open Analytics page after the chart is saved.

Ask for:

* operational dashboards from pipeline outputs
* revenue, usage, churn, or conversion charts from warehouse data
* chart edits such as new filters, titles, colors, or grouping
* table-backed data products for analysts and stakeholders
* chart explanations for people who did not build the pipeline

### Understand the Page You Are On

Sidekick can answer page-relative questions such as "what am I looking at?",
"why is this empty?", "what changed here?", or "what should I do next?" because
Mage passes it scoped page context for the active chat turn. It can also use
recent interaction context, selected text, current file context, active preview
context, and chart workflow attachments when those are available.

![AI Sidekick response explaining a pipeline and recommending a monitoring chart.](https://mintcdn.com/mage/xlfFBQh2VT61fQmA/media/ai-sidekick/pipeline-sidekick-chat.jpg?fit=max\&auto=format\&n=xlfFBQh2VT61fQmA\&q=85\&s=e42b55ebf3982617dd9f5995d0f60ab2)

This makes Sidekick useful for onboarding and review:

* explain a pipeline graph to a new teammate
* summarize the selected code in Apps Coder
* describe why a chart or preview is filtered
* compare what the UI shows with the last successful run
* keep an Analytics edit focused on the selected chart

### Search Docs, Project Knowledge, and Chat History

Sidekick can search Mage documentation, read relevant pages, inspect project
instructions when they matter, and use previous chat turns or tool outputs to
continue work without making you repeat context. When retrieval is enabled for a
project, it can reference the project files, configuration, docs, and code that
you permit it to read.

Use this for:

* "How should I configure this trigger?"
* "Find the docs for dynamic blocks and apply the pattern here."
* "Continue from the chart we created earlier."
* "Read the last tool output and explain why it failed."

### Operate Production Workflows

Sidekick can help with the operational surfaces around pipelines: schedules,
triggers, backfills, runs, jobs, block runs, outputs, logs, variables, secrets,
notifications, permissions, users, roles, deployments, compute, Spark, global
hooks, version control, and workspace settings. It can list, inspect, create, or
update supported Mage resources through the same authenticated Mage APIs that
power the product.

Examples:

* "Create a weekday trigger for this pipeline with these runtime variables."
* "Backfill this pipeline for the last seven days."
* "Show active triggers and the next run time for each."
* "Check whether this user has permission to view the workspace."
* "Review Spark executor status for the slow run."

### Work with Files and Versioned Project Code

Sidekick can read and update project files, upload or download supported files,
record changes so Mage file versions stay accurate, inspect file history, and
coordinate code changes with pipeline or block metadata updates. It can also use
sync workflows when local and remote project files need to be reconciled.

Use it when you want to:

* update a utility module used by multiple blocks
* change a YAML config and apply the related Mage metadata update
* inspect file versions before accepting a change
* create a new project file from a prompt
* pull or push project changes through an agent workflow

### Coordinate Longer AI Workflows

For larger requests, Sidekick can break work into smaller AI tasks, wait for
results, cancel stale work, and summarize progress back into the main chat. This
is useful for independent review, broad discovery, test-gap analysis, migration
work, or multi-step implementation where several pieces can be investigated at
the same time.

Examples:

* "Audit these five pipelines and summarize the risky patterns."
* "Have one agent inspect docs and another inspect code before proposing a fix."
* "Generate a migration plan and check the source project inventory."
* "Run an independent review of the changes before I ship them."

### Generate and Understand Media

In Sidekick sessions that enable media tools, Sidekick can transcribe audio,
translate audio to English, create spoken audio from text, generate images, edit
images, and create or download short videos. Generated files are saved to the
Sidekick media area for the active chat and are returned through authenticated
preview links.

Use this for:

* transcribing a stakeholder recording into requirements
* generating a narrated explanation of a pipeline result
* drafting images for reports, demos, or internal docs
* editing chart screenshots or data-product visuals
* producing short demo clips from a written prompt

## Built for Data Teams

AI Sidekick is not a generic chatbot placed next to your pipeline editor. It is
connected to Mage's data engineering model: projects, pipelines, blocks,
dependencies, schedules, logs, previews, dashboards, files, users, roles,
deployments, compute resources, and production history. That connection lets it
answer with context and take action where the work already happens.

## Safety and Control

Sidekick uses your Mage Pro authentication and respects the same project,
workspace, and resource boundaries as the product. It uses redacted tool
responses for sensitive fields, works with preview IDs and bounded row windows
instead of dumping full datasets into chat, validates read-only SQL previews
before execution, and asks clarifying questions when a request could target
multiple resources.

For changes that mutate pipelines, blocks, files, schedules, secrets, users,
permissions, or other resources, review the proposed result in Mage and ask
Sidekick to explain what changed, show the relevant diff or resource state, or
run the affected workflow again.


# Blog
Source: https://docs.mage.ai/community/blog



> **[Blog](https://www.mage.ai/blog)**
>
> Want to be entertained?


# Contact
Source: https://docs.mage.ai/community/contact



> **[Contact](mailto:hello@mage.ai)**
>
> Send us an email


# GitHub
Source: https://docs.mage.ai/community/github



> **[GitHub](https://github.com/mage-ai/mage-ai)**
>
> Open-source project


# LinkedIn
Source: https://docs.mage.ai/community/linkedin



> **[LinkedIn](https://www.linkedin.com/company/magetech)**
>
> Let’s connect professionally


# Slack
Source: https://docs.mage.ai/community/slack



> **[Slack](https://www.mage.ai/chat)**
>
> Instantly chat with the team and other community members


# Twitter
Source: https://docs.mage.ai/community/twitter



> **[Twitter](https://twitter.com/mage_ai)**
>
> Ready for funny memes?


# Adding an IO class
Source: https://docs.mage.ai/contributing/backend/io/adding-a-class

IO classes power Mage sources and destinations. Read on to learn how you can contribute an IO class to Mage.

## What's an IO class?

IO classes are at the heart of Mage— they're the code that enables Mage to read and write data. When you configure your [`io_config`](https://docs.mage.ai/development/io_config) to use a source or destination, you're telling Mage to use a specific IO class.

IO classes are defined [here](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/io) in the Mage repository. You can see that there are a few different types of IO classes— some are for reading data, some are for writing data, and some are for both.

## Why should I contribute?

If you have a favorite IO class *that's not currently supported*, you can reach out *or* contribute it to Mage! Contributing is a great way to build your skills, become a part of the Mage community, and help others.

## How do I contribute?

### Configure your development environment

The first step to contributing an IO class is to configure your development environment. You can find instructions for doing so [here](https://docs.mage.ai/contributing/development-environment).

### Create a new IO class

Once configured, you'll want to create a new file in the `mage_ai/io` directory. The file should be named after the IO class you're contributing. For example, if you're contributing an IO class called `MyIOClass`, you should create a file called `my_io_class.py`.

Some IO classes are relatively straightforward, like [exporting to local files](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/file.py) or [writing to Google Sheets](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/google_sheets.py). Others, like full database integrations, can be pretty complex. [Google BigQuery](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/bigquery.py) is a good example of a complex IO class.

Most classes are platform specific, but for a database, you might need the following methods:

* `alter_table`: Alter the table schema
* `load`: Load data from the database
* `export`: Export data to the database
* `execute`: Execute a query on the database
  * `execute_queries`: Execute multiple queries on the database

Additionally, every class will need to have the following methods:

* `__init__`: Initialize the class
* `with_config`: Initialize the database client from the configuration loader. You'll notice that this method is used in *all* of our templates.
* `test_connection`: Validate credentials or reachability with a lightweight, side-effect-free check. Return normally (`None`) when the probe succeeds and let connection or probe failures raise so callers can surface the underlying error. This method should not load preview data, export records, or mutate remote resources. If the safest probe depends on a target resource (for example a bucket, index, or sheet), accept that target as an optional argument and document any broader no-target fallback permissions.

### Test your IO class

In order to test your class, you should perform all of the methods you defined in the class itself. Once you pass our linting checks (outlined in the development environment link above) *and* your tests pass, you're ready to submit a pull request! Learn more about pull requests on the contribution.

## Create a pull request

<Snippet />


# Contributing to the backend server
Source: https://docs.mage.ai/contributing/backend/overview

Mage backend code is written in Python 🐍 and our server uses the Tornado 🌪️ framework. Here are some guides on adding features to the Mage backend.

## Guides

### API

* [Overview](/api-reference/overview)

### Data integrations

* [Add a new source to the data integration pipeline](/contributing/data-integrations/add-new-source)
* [Add a new destination to the data integration pipeline](/contributing/data-integrations/add-new-destination)

### Streaming pipelines

* [Add a new source or destination to the streaming pipeline](/contributing/backend/streaming/sources-and-destinations)

### IO classes

* [Add a new IO class](/contributing/backend/io/adding-a-class)

## Style guide

### Linter

Install `flake8` in your IDE to lint the Python code.

To run the linter locally, execute this script:

```bash theme={"system"}
./scripts/server/lint.sh
```

## Testing

### Unit tests

Add unit tests for the feature in
[mage\_ai/tests](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/tests) directory.

To run the tests locally, execute this script:

```bash theme={"system"}
./scripts/server/test.sh
```

It is also possible to run unit tests directly in a live docker instance, as given in the following steps.

1. Find out the backend server container name, with the command `docker container ls` in a terminal:

```bash theme={"system"}
 % docker container ls

CONTAINER ID   IMAGE       COMMAND                  CREATED      STATUS          PORTS                    NAMES
8dbcfbe41755   mage/data   "./scripts/install_a…"   5 days ago   Up 38 minutes   0.0.0.0:3000->3000/tcp   mage-ai-app-1
b9d811e1e3e8   mage/data   "python mage_ai/serv…"   5 days ago   Up 38 minutes   0.0.0.0:6789->6789/tcp   mage-ai-server-1
```

2. Start an interactive `bash` session with the backend server container:

```bash theme={"system"}
% docker exec -it mage-ai-server-1 /bin/bash
```

3. Run unit tests with the following command:

```bash theme={"system"}
root@b9d811e1e3e8:/home/src# python3 -m pytest mage_ai -x
```

## Debugging

<Snippet />


# Contributing
Source: https://docs.mage.ai/contributing/backend/streaming/sources-and-destinations



This doc talks about how to add a new source or destination to the streaming pipeline.

## Prerequisites

Follow this [doc](/contributing/overview) to set up the development environment for Mage.

## Add a new source

Example PR for adding a source: [https://github.com/mage-ai/mage-ai/pull/1953](https://github.com/mage-ai/mage-ai/pull/1953)

* Add the new source type to [mage\_ai/streaming/constants.py](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/constants.py).
* Add the source file to the folder [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/streaming/sources](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/streaming/sources)
  * Define the source class which inherits from the `BaseSource`.
  * Define the source config class.
  * Implement the methods:
    * `init_client`: Initialize the client used to connect to the source.
    * `batch_read`: Batch read messages from the source and use `handler` method to process the messages.
* Add the new source to the [SourceFactory](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/sources/source_factory.py).
* Add a template file with example config to the folder: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/data\_preparation/templates/data\_loaders/streaming](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/data_preparation/templates/data_loaders/streaming)
* Add new source type to the frontend code.
  * [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L31](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L31)
  * [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L3](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L3)
  * [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L22](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L22)
* Add unit tests: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/tests/streaming/sources](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/tests/streaming/sources)
* Add doc: [https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/sources](https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/sources)

***

## Add a new destination (sink)

Example PR for adding a sink: [https://github.com/mage-ai/mage-ai/pull/2121](https://github.com/mage-ai/mage-ai/pull/2121)

* Add the new sink type to [mage\_ai/streaming/constants.py](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/constants.py).
* Add the sink file to the folder [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/streaming/sinks](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/streaming/sinks)
  * Define the sink class which inherits from the `BaseSink`.
  * Define the sink config class.
  * Implement the methods:
    * `init_client`: Initialize the client used to connect to the destination.
    * `batch_write`: Batch write messages to destination.
* Add the new sink to the [SinkFactory](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/sinks/sink_factory.py).
* Add a template file with example config to the folder: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/data\_preparation/templates/data\_exporters/streaming](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/data_preparation/templates/data_exporters/streaming)
* Add new source type to the frontend code.
  * [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L37](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L37)
  * [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L3](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L3)
  * [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L22](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L22)
* Add unit tests: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/tests/streaming/sinks](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/tests/streaming/sinks)
* Add doc: [https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/destinations](https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/destinations)


# Testing
Source: https://docs.mage.ai/contributing/backend/testing/overview

Making Mage so invincible that nothing can break it, not even itself.

![Testing](https://i.pinimg.com/originals/05/70/56/057056dae3c4c8a5c5426f6c458ba001.gif)

## Errors

### SQL Alchemy migration error

Delete the `test.db` file located in the directory where you are running the
`python3 -m pytest ... -x` command.

Example:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: index ix_pipeline_run_execution_date already exists

...

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) index ix_pipeline_run_execution_date already exists
[SQL: CREATE INDEX ix_pipeline_run_execution_date ON pipeline_run (execution_date)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

----------------------------------------------------------------------
Ran 0 tests in 0.038s

FAILED (errors=1)
```


# How to add a chart
Source: https://docs.mage.ai/contributing/charts/how-to-add



## 1. Add a new chart type

Add a new type in [`mage_ai/data_preparation/models/widget/constants.py`](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/models/widget/constants.py):

```python theme={"system"}
class ChartType(str, Enum):
    # ...
    PIE_CHART = 'pie_chart'
```

Add a new type in [`mage_ai/frontend/interfaces/ChartBlockType.ts`](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/ChartBlockType.ts):

```javascript theme={"system"}
export enum ChartTypeEnum {
  // ...
  PIE_CHART = 'pie_chart',
}
export const CHART_TYPES = [
  // ...
  ChartTypeEnum.PIE_CHART,
];
```

## 2. Define the configuration options for the chart

In the file `mage_ai/frontend/components/ChartBlock/constants.ts`,
add a list of options for the user to configure their chart with:

```javascript theme={"system"}
export const CONFIGURATIONS_BY_CHART_TYPE = {
  [ChartTypeEnum.PIE_CHART]: [
    {
      label: () => 'variable name of values',
      monospace: true,
      uuid: 'super_cool_uuid',
    },
  ],
};
```

Also, add the new variable name to the following constant in the
file `mage_ai/frontend/interfaces/ChartBlockType.ts`,:

```javascript theme={"system"}
export const VARIABLE_NAMES = [
  'super_cool_uuid',
];
```

If 1 or more of the configuration options is for a variable name that can be defined in the
chart block’s code and referenced at presentation time, then you must add the following
in the file `mage_ai/data_preparation/models/widget/constants.py`:

```python theme={"system"}
VARIABLE_NAMES_BY_CHART_TYPE = {
    ChartType.PIE_CHART: [
        'super_cool_uuid',
    ],
}
```

### Add default settings to the configuration

When a user selects a chart type, you can set default values for these options.
In the file `mage_ai/frontend/interfaces/ChartBlockType.ts`, add the following:

```javascript theme={"system"}
export const DEFAULT_SETTINGS_BY_CHART_TYPE = {
  [ChartTypeEnum.PIE_CHART]: {
    configuration: () => ({
      [VARIABLE_NAME_X]: 'x',
    }),
    content: ({
      upstream_blocks: upstreamBlocks = [],
    }: BlockType) => {
      const uuid = upstreamBlocks[0];
      return `x = ${uuid}[${uuid}.columns[0]]`;
    },
  },
};
```

### Add helpful information about the variable type that the chart expects

Each chart requires 1 or more input values.
The chart knows how to access those input values by referencing a variable name that the user
inputs into the configuration options.

Each variable can be of a certain type; array/list of strings, integers, etc.

To help the user know how to set the variables properly,
add the following information into the file `mage_ai/frontend/components/ChartBlock/constants.ts`:

```javascript theme={"system"}
export const VARIABLE_INFO_BY_CHART_TYPE = {
  [ChartTypeEnum.PIE_CHART]: {
    [VARIABLE_NAME_X]: (): string => 'must be a list of booleans, dates, integers, floats, or strings.',
  },
};
```

## 3. Define post processing logic for chart data

In the file `mage_ai/data_preparation/models/widget/__init__.py`,
add additional logic for parsing the input values from the
variables defined in the chart block’s code.

The output of this method is provided to the front-end React components for rendering.

```python theme={"system"}
class Widget(Block):
    def post_process_variables(self, variables):
        if ChartType.PIE_CHART == self.chart_type:
            variables = {}
        return variables
```

## 4. Create React component for rendering chart

In the file `mage_ai/frontend/components/ChartBlock/ChartController.tsx`,
import your React component for your new chart and render it.

For example:

```javascript theme={"system"}
function ChartController({
  block,
  data,
  width,
}: ChartControllerProps) {
  const {
    configuration,
  } = block;
  const chartType = configuration?.chart_type;
  if (ChartTypeEnum.PIE_CHART === chartType) {
    return <PieChart />;
  }
}
```


# Adapt an existing source
Source: https://docs.mage.ai/contributing/data-integrations/adapt-existing-source

Mage builds data integrations from [Singer taps](https://www.singer.io/#taps), so if there's a Singer source you want to use that isn't supported yet, you can adapt it yourself!

## Getting started

Mage sources live in the `/mage_integrations/mage_integrations` folder. To add a new source, you'll need to create a new folder in the `/sources` subdirectory.

The folder name should be the name of the tap. For existing sources, that starts with pulling in the GitHub repo for the tap. For example, for the [GitHub tap](https://github.com/MeltanoLabs/tap-github), we would:

```bash theme={"system"}
cd mage_integrations/mage_integrations
git clone https://github.com/MeltanoLabs/tap-github.git sources/github
```

The result:

```
.
├── LICENSE
├── README.md
├── config-sample.json
├── config.json
├── meltano.yml
├── poetry.lock
├── pyproject.toml
├── pytest.ini
├── tap-github.sh
├── tap_github
│   ├── __init__.py
│   ├── authenticator.py
│   ├── client.py
│   ├── organization_streams.py
│   ├── repository_streams.py
│   ├── schema_objects.py
│   ├── scraping.py
│   ├── streams.py
│   ├── tap.py
│   ├── tests
│   ├── user_streams.py
│   └── utils
└── tox.ini
```

## Configuring the source

We now have a folder with our tap and a bunch of miscellaneous files. Our first step is to clear out everything we *don't* need. We can delete all files/folders other than `tap_your_source` and `tests`, but hold onto anything that looks like a `config.json` file or a sample config.

Next, we need to create a configuration file and an init file:

```bash theme={"system"}
mkdir sources/github/templates \
&& touch sources/github/templates/config.json \
&& touch sources/github/__init__.py
```

We can now populate our `config.json` file with the sample from the tap. In the GitHub example, this is `config.sample.json`. It looks like this:

```json theme={"system"}
{
    "access_token": "abcdefghijklmnopqrstuvwxyz1234567890ABCD",
    "repository": "singer-io/target-stitch",
    "start_date": "2021-01-01T00:00:00Z",
    "request_timeout": 300,
    "base_url": "https://api.github.com"
}
```

These are the parameters we'll need to configure our tap. After cleanup, our tree looks like:

```
.
├── __init__.py
├── tap_github
│   ├── __init__.py
│   ├── authenticator.py
│   ├── client.py
│   ├── organization_streams.py
│   ├── repository_streams.py
│   ├── schema_objects.py
│   ├── scraping.py
│   ├── streams.py
│   ├── tap.py
│   ├── tests
│   ├── user_streams.py
│   └── utils
└── templates
    └── config.json
```

Now, we need to override some methods in the class to make it work with Mage. This is where things get fun! Copy and paste the following template into `your_tap/__init__.py`:

```python theme={"system"}
from typing import List

from mage_integrations.sources.base import Source, main
from mage_integrations.sources.catalog import Catalog

class YourSource(Source):
    def discover(self, streams: List[str] = None) -> Catalog:
        pass

    def sync(self, catalog: Catalog) -> None:
        pass
    
    def test_connection(self) -> None:
        client = YourSource(YOUR_CONFIG)
        client.close()

if __name__ == "__main__":
    main(YourSource, schemas_folder="tap_your_tap/schemas")
```

Replace YourSource with the name of your source, e.g. `Github` for Github, and `tap_your_tap` with the name of your tap, e.g. `tap_github`.

### Overwrite tap methods

Our goal is to overwrite the `discover` and `sync` methods to make them work with Mage. Discover is likely in `tap_your_tap/discover.py` and sync is likely in `tap_your_tap/sync.py`. It requires a bit of nuance, but our job is to look through the existing Singer tap, understand how the `sync` and `discover` methods were implemented, then execute them in Mage `__init__.py` under the corresponding method. This is necessarily different for every tap, since each project is entirely independent.

Don't forget to use absolute imports for these files. [HubSpot](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/hubspot) is another good example of a completed tap.

### Add test connection method

<Snippet />

### Overwrite `write_schema` method

Finally, we need to overwrite the `write_schema` method in the `tap_your_tap/sync.py` file. Add the following import:

`from mage_integrations.sources.messages import write_schema`

Now, switch out the default function (likely `singer.write_schema`) for the mage function using the default arguments. Here's an example:

```python theme={"system"}
  write_schema(
        stream_name=catalog['tap_stream_id'],
        schema=schema,
        key_properties=catalog.get('key_properties', ['id']),
        bookmark_properties=catalog.get('bookmark_properties'),
        disable_column_type_check=catalog.get('disable_column_type_check'),
        partition_keys=catalog.get('partition_keys'),
        replication_method=catalog.get('replication_method'),
        stream_alias=catalog.get('stream_alias'),
        unique_conflict_method=catalog.get('unique_conflict_method'),
        unique_constraints=catalog.get('unique_constraints'),
    )
```

### Add logging

To add verbose logging to Mage, add a logger argument each tap file, `tap_github/sync.py`, `tap_github/discover.py`:

```python theme={"system"}
def discover(client, logger=None):
    """
    Run the discovery mode, prepare the catalog file and return catalog.
    """
    mage_logger = logger or LOGGER
```

Then replace usage of `LOGGER` with `mage_logger`.

```python theme={"system"}

for stream_name, schema_dict in schemas.items():
        try:
            schema = Schema.from_dict(schema_dict)
            mdata = field_metadata[stream_name]
        except Exception as err:
            # These were LOGGER previously
            mage_logger.error(err)
            mage_logger.error('stream_name: %s', stream_name)
            mage_logger.error('type schema_dict: %s', type(schema_dict))
            raise err
```

This will enable Mage to better log Singer taps! 🥳

## Add source to the Mage UI

Add the new source to the `SOURCES` list constant in [this](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_integrations/sources/constants.py) file

This will make your new source visible in the Mage UI.

## Test your source

Sources should be tested both in the console and via the UI. Follow the directions [here](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/README.md#testing-sync-locally) to configure a lightweight test of your sources taps via a bash terminal and the Mage UI in the dev docker instance. This entails:

* Setting up a sample source and connection locally.
* Running the tap directly, via Python.
* Writing to an intermediate file to inspect the output.
* Performing an end-to-end sync.

## Populate `README.md`

Finally, populate a README file in the source folder, e.g. `/sources/github/README.md`. This should include:

* The name/logo of the tap.
* A configuration section.
* The method for accessing/creating an API key to use with the folder.
* See [HubSpot](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/hubspot) or [GitHub](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/github) for examples.


# Create a new destination
Source: https://docs.mage.ai/contributing/data-integrations/add-new-destination



Here is an example PR for adding a new destination: [https://github.com/mage-ai/mage-ai/pull/2277](https://github.com/mage-ai/mage-ai/pull/2277)

In this guide, we'll build a sample destination called `SampleDest`. When you're building your own destination, you can swap out the `SampleDest` name for the real name of your new destination.

1. Create a directory for the new destination
2. Define custom destination class
3. Add main function
4. Add your destination to the UI (optional)
5. Test your destination

## 1. Create a directory for the new destination

In the `mage_integrations/destinations/` directory, add a new directory named after your destination.
Use snake case and lowercase for your directory name: `mage_integrations/destinations/sample_dest/`.

In this new directory, create the following subdirectories and files:

* `mage_integrations/destinations/sample_dest/templates/config.json`
* `mage_integrations/destinations/sample_dest/__init__.py`
* `mage_integrations/destinations/sample_dest/README.md`

The directory structure should look like this:

```
mage_integrations/
|   destinations/
|   |   sample_dest/
|   |   |   templates/
|   |   |   |   config.json
|   |   |   __init__.py
|   |   |   README.md
```

### Templates folder

This folder contains a sample configuration JSON file that’s
displayed to the user when they are setting up a synchronization using this destination.

The `config.json` file contains keys and values that are used to configure the
behavior of the destination as well as credentials to authenticate requests to the destination.

#### Naming convention

You must use the exact filename `config.json`.

#### Examples

```json theme={"system"}
{
  "api_key": "",
  "secret_key": ""
}
```

### `__init__.py`

This is where majority of the destination logic will exist.

#### Examples

[`mage_integrations/destinations/amazon_s3/__init__.py`](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/amazon_s3/__init__.py)

### `README.md`

Document how to configure and use your destination in the `README.md` file.

## 2. Define custom destination class

In the `mage_integrations/destinations/sample_dest/__init__.py`,
create a new class named after your destination and subclass the
[base destination class](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/base.py).

If you're adding a destination for a SQL data warehouse or database, you can subclass the
[base sql destination class](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/sql/base.py)

```python theme={"system"}
from mage_integrations.destinations.base import Destination


class SampleDest(Destination):
    pass
```

### Override the `export_batch_data` method

The base `Destination` class has an instance method called `export_batch_data`. Here is the interface:

```python theme={"system"}
def export_batch_data(self, record_data: List[Dict], stream: str) -> None:
    ...
```

Override this method to contain the logic for exporting data that is specific to your destination.

## 3. Add main function

In the file `mage_integrations/destinations/sample_dest/__init__.py`, add the main function to the bottom of the file
like below:

```python theme={"system"}
if __name__ == '__main__':
    destination = SampleDest(
        argument_parser=argparse.ArgumentParser(),
        batch_processing=True,
    )
    destination.process(sys.stdin.buffer)
```

## 4. Add your destination to the UI (optional)

Add the new destination to the `DESTINATIONS` list constant in this file: [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/data\_integrations/destinations/constants.py](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_integrations/destinations/constants.py)

## 5. Test your destination

Follow this [doc](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations) to test your new destination.

To test the destination in the UI, you can install your updated `mage_integrations` module by running the following commands in Mage terminal:

```bash theme={"system"}
pip uninstall -y mage_integrations
pip install "git+https://github.com/your_repo.git@your_branch#egg=mage-integrations&subdirectory=mage_integrations"
```


# Create a new source
Source: https://docs.mage.ai/contributing/data-integrations/add-new-source

This guide details adding a new source to Mage. If your source already exists as a [Singer tap](https://www.singer.io/#taps), check out [our guide](/contributing/data-integrations/adapt-existing-source) for adapting an existing source instead.

Mage Sources are built on Singer Taps. You can read more about the Singer spec [here](https://github.com/singer-io/getting-started). In this guide, we'll copy over our sample source to get started and build from there.

If you haven't already, start by cloning the Mage repo and navigating to the folder:

```bash theme={"system"}
git clone https://github.com/mage-ai/mage-ai.git
cd mage-ai
```

For our source, we'll need:

* A schema that represents the source data
* A template configuration file
* An `__init__.py` file that contains the logic for fetching data from the source.
  * A `test_connection` method that tests the connection to our source.
  * A `discover` method that returns a [Catalog](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/catalog.py) of streams.
  * A `load_data` method that yields data from the source as a dictionary.

We'll start from a template to make things easy.

## Copy the template source

Navigate to `/mage-ai/mage_integrations/mage_integrations/sources` and create a copy of the `titanic` directory:

```bash theme={"system"}
cd ./mage_integrations/mage_integrations/sources
cp -R titanic MY_SOURCE
```

Our titanic source simply reads data from a CSV file, but we'll modify it to do much more. You'll notice the following files and folders in the `MY_SOURCE` directory:

```
mage_integrations/sources/MY_SOURCE
├── README.md
├── __init__.py
├── schemas
│   └── passengers.json
└── templates
    └── config.json
```

So we'll need to update or overwrite each of these.

## Update schemas

This folder contains all the known schemas from your source. For sources that have dynamic schemas (e.g. database tables from MySQL), this folder may be empty since the schema is dependent on the individual's source data.

The JSON format of these schema files follows the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#schemas).

When naming schemas, use the plural name of the object you're referencing. This plural name will be displayed to the individual who is setting up a synchronization using this source. Take the [sample "passengers" schema](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/titanic/schemas/passengers.json) for example:

```json theme={"system"}
{
  "properties": {
    "Survived": {
      "type": ["null", "integer"]
    },
    "Name": {
      "type": ["null", "string"]
    }
  },
  "type": ["null", "object"]
}
```

## Add templates

This folder contains a sample configuration JSON file that's displayed to the user when they are setting up a synchronization using this source.

The `config.json` file contains keys and values that are used to configure the behavior of the source as well as credentials to authenticate requests to the source. You must use the exact filename `config.json`, regardless of your source's name.

The following simple example is present in the [Titanic source](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/titanic/templates/config.json).

```json theme={"system"}
{
  "api_key": "",
  "secret_key": ""
}
```

## Update `__init__.py`

A majority of source logic lives in the `__init__.py` file. Most of the work in adding your own source will involve creating/overwriting methods from `__init__.py`. [Our sample](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/titanic/__init__.py) contains the following:

```python theme={"system"}
from mage_integrations.sources.base import Source, main
from typing import Dict, Generator, List
import requests


URL = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'

class Titanic(Source):

    def load_data(
        self,
        **kwargs,
    ) -> Generator[List[Dict], None, None]:
        text = requests.get(URL).text

        rows = []
        lines = text.rstrip().split('\n')
        columns = lines[0].split(',')

        for line in lines[1:]:
            values = line.split(',')
            rows.append({col: values[idx] for idx, col in enumerate(columns)})

        yield rows

    def test_connection(self):
        request = requests.get(URL)

        if request.status_code != 200:
            raise Exception('Could not fetch titanic data')


if __name__ == '__main__':
    main(Titanic)
```

### Rename source class

First, rename the source class to match your new source

```python theme={"system"}
from mage_integrations.sources.base import Source

class MY_SOURCE(Source):
    pass
```

### Edit methods

#### The `load_data()` method

Override this method to contain the logic for fetching data specific to your source. The `Titanic` source's `load_data` method reads data from a CSV file and yields a list of dictionaries:

```python theme={"system"}
def load_data(
    self,
    **kwargs,
) -> Generator[List[Dict], None, None]:
    url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'
    text = requests.get(url).text
    rows = []
    lines = text.rstrip().split('\n')
    columns = lines[0].split(',')
    for line in lines[1:]:
        values = line.split(',')
        rows.append({col: values[idx] for idx, col in enumerate(columns)})
    yield rows
```

Your `load_data` method should *also* yield a list of dictionaries. There is a keyword argument named `query` in the `load_data` method that is a dictionary. When Mage runs a source, the following keys and values are automatically available on each run:

| Key | Description | Sample value |
| - | - | - |
| `_execution_date` | The date and time (in ISO format) of when the pipeline started running. | `2022-10-21T17:24:49.443559` |
| `_execution_partition` | An automatically formatted partition of the pipeline run using the execution date. | `20221021T172557` (e.g. format `%Y%m%dT%H%M%S`) |
| `_start_date` | You can define this variable as a [runtime variable](/getting-started/runtime-variable) in your pipeline or it'll be automatically filled in using the date and time your pipeline runs minus 1 hour, day, week, etc (based on your schedule's interval). | `2022-10-01T00:00:00.000000` |
| `_end_date` | You can define this variable as a [runtime variable](/getting-started/runtime-variable) in your pipeline or it'll be automatically filled in using the date and time your pipeline runs. | `2022-10-02T00:00:00.000000` |

#### The `discover()` method

The discover method should return a [Catalog](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/catalog.py) of streams that define the data in your source. For example discover methods, see our [Google Sheets](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/google_sheets/__init__.py) source or our [DynamoDB source](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/dynamodb/__init__.py).

#### The `test_connection()` method

<Snippet />

### Update `main()` function

Change the source name in the `main()` function to match your new source:

```python theme={"system"}
if __name__ == '__main__':
    main(MY_SOURCE)
```

## Add your source to the UI

The list of sources is available [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_integrations/sources/constants.py), add yours to make it accessible via the UI.

## Document your source

Document how to configure and use your source in the `README.md` file at the source's root directory. Documentation helps Magers and future contributor understand exactly how your source works! Be sure to be thorough and descriptive. [Here's an example](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/salesforce/README.md) of documentation done well.

## Test your source

You'll first need to configure your [development environment](https://docs.mage.ai/contributing/development-environment). Once complete, follow this [doc](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations) to test your new source.


# Configure your development environment
Source: https://docs.mage.ai/contributing/development-environment

Want to get started contributing code? You're in the right place! Read on to learn how to set up your development environment.

## Getting started

Configuring a development environment can be *hard*, but at Mage we're trying to make things as simple and straightforward as possible.

To get started, you'll need Docker installed on your machine. You can read more about Docker [here](https://www.docker.com/).

## Environment setup

Next, you'll need to configure your development environment. This should be as simple as running a few commands. You can watch the video below or skip to the numbered instructions.

[YouTube video player](https://www.youtube.com/embed/mxKh2062sTc)

**1. Configure your development environment**

Configure your coding environment for local development: `pyenv`, pre-commit hooks, etc. Follow the instructions [here](https://github.com/mage-ai/mage-ai/blob/master/README_dev.md).

**2. Create a new project**

```bash theme={"system"}
./scripts/init.sh [project_name]
```

**3. Build the dev docker image and run all services**

```bash theme={"system"}
./scripts/dev.sh [project_name]
```

If you only work on backend, use the command below for better performance

```bash theme={"system"}
./scripts/start.sh [project_name]
```

**4. Open Mage**

When developing frontend, use port 3000. If you only need to develop backend, you can use port 6789. Find these ports at [http://localhost:3000](http://localhost:3000) & [http://localhost:6789](http://localhost:6789).

## Push code changes

Once your environment is configured, you can make edits and see changes in real time! Now you're ready to contribute to the Mage repo. This is very similar to contributing to other open source projects on GitHub.

If you *don't* have any experience contributing code and this is confusing, feel free to reach out in Slack and we'll help you get started.

Otherwise, follow the steps below to push your code changes to the Mage repo.

<Snippet />

## Troubleshoot

### Permission denied

If you get an error relating to permission denied when trying to execute a script,
it might have something to do with SELinux.

Two or more containers are trying to mount the same volume. With SELinux it’s not possible without
the `z` option.

For more information about using the `z` option,
read [Docker’s documentation](https://docs.docker.com/storage/bind-mounts/).

### No such file or directory for "install\_and\_run.sh" file

If you are using a Windows machine and run into the following error while executing the
`docker-compose.yml` file: `exec ./scripts/install_and_run.sh: no such file or directory`,
it may be due to a difference in line-endings used on Windows versus Macs.

You can check the `mage_ai/frontend/scripts/install_and_run.sh` file in VSCode to see if
it is using CRLFs (carriage return line feeds), and if it is, change it to LFs (line feeds)
and save the file. Then try running the `docker-compose` command again.

![](https://mage-ai.github.io/assets/development/windows/update-crlf-to-lf-vscode.gif)

### ModuleNotFoundError: No module named 'x'

If there were added new libraries you should manually handle new dependencies. It can be done in 2 ways:

1. `docker-compose build` from project root will fully rebuild an image with new dependencies - it can take lots of time
2. `pip install x` from inside the container will only install the required dependency - it should be much faster


# Documentation
Source: https://docs.mage.ai/contributing/documentation/overview

Like every other part of Mage, our documentation is open-source. Read on to learn how to make edits or even write entirely new docs for Mage!

![](https://github.com/mage-ai/mage-ai/assets/59450879/769a5faf-37ea-453c-8bd6-83270366fe00)

## Overview

Mage uses [Mintlify](https://www.mintlify.com/) as our documentation framework. Mintlify provides hosting and deployment for our docs, which makes editing quite simple. In this guide, we'll discuss different "levels" of contributing to our docs, starting from simplest to most advanced.

All of our docs are written in plain text as Markdown files. If you're not familiar with Markdown, check out [this guide](https://www.markdownguide.org/basic-syntax/) to get started. You can take a look at the structure and contents of our docs by navigating this site and checking out the [docs directory](https://github.com/mage-ai/mage-ai) on GitHub.

## Level 1: Suggesting Edits 🦸🏻‍♂️

The very simplest way to contribute to the Mage docs is directly in the browser, using GitHub's file editing feature. This is a great way to suggest small edits to existing docs, but it's not the best way to add entirely new docs or complex formatting, images, etc.

To make edits, simply click on the "Suggest Edit" button at the bottom of every page. This will open a new tab with the file you're editing. Make your changes, then click "Commit changes." This will open a new pull request with your changes. Be sure to add appropriate details to your pull request & tag `@wangxiaoyou1993`. Once the pull request is merged, your changes will be live on the site!

![](https://mage-ai.github.io/assets/contributing-docs.png)

## Level 2: Add complex components 💪

### Images

To add images to the Mage docs, you'll need to contact a team member or host the images in GitHub. Our assets currently live in a private repo to keep the docs repo lightweight.

Reach out in [Slack](https://www.mage.ai/chat) if you're editing documentation that requires additional assets and we'll get them hosted for you.

When adding images, we use centered divs or the Mintlify frame component. You can see the code in this page for examples of both.

### Mintlify elements

To learn more about Mintlify-specific elements, visit the Mintlify docs [here](https://mintlify.com/docs/content/page). You can add things like accordion groups, interactive lists, html elements, and more. Anything you can do in Mintlify, you can do in our docs.

## Level 3: Add entirely new docs 🪄

### Adding a new page

Each page is an MDX file that should begin with the following metadata syntax with `---` at the start and end. Here are basics to get you started:

1. [Page basics](https://mintlify.com/docs/content/page)
2. [Headers and text](https://mintlify.com/docs/content/text)
3. [Images and embeds](https://mintlify.com/docs/content/image-embeds)

Once you add a new page in the appropriate folder in our `docs` directory, you'll need to add it to our site navigation. That is accomplished through the `mint.json` file in the root of our `docs` directory.

You'll need to add a new entry to the `pages` array in that file. The `path` should be the relative path to your new page, and the `title` should be the title of your page. Paths in `mint.json` should *not* include file extensions.

The best way to understand the structure is to read about it in the [Mintlify docs](https://mintlify.com/docs/settings/global) and to take a look at our `mint.json` file [here](https://github.com/mage-ai/mage-ai/blob/master/docs/mint.json).

### Developing locally in Mintlify

If you *really* want to level up your docs contributions, you can create a local development environment. This will allow you to see changes *live* as you make them on your machine.

Follow the instructions from Mintlify [here](https://mintlify.com/docs/quickstart#2-updating-your-docs) to get started. Once configured, you'll be able to run a local server that will allow you to see your changes in real-time.

Simply `cd mage-ai/docs` + `mintlify dev` and you'll be able to navigate to a live version at `localhost:3000`.

### Adding icons

Some elements in Mintlify have icons. Those are sourced from [fontawesome](https://fontawesome.com/). Any icon you find there can be added as an icon to the Mage docs, where appropriate. We only add icons to top-level items in our sidebar.

## Level 4: Adding entirely new sections 🧙🏻‍♂️

If you'd like to contribute a larger, more structured section of our documentation, please reach out to a member of our team via Slack. They'll be able to guide you through the process of making sweeping changes.


# Contributing to the front-end client
Source: https://docs.mage.ai/contributing/frontend/overview

Guides on adding features to the front-end client.

| | |
| - | - |
| Language | TypeScript |
| Framework | React, Next.js |
| Code directory | `mage_ai/frontend/` |

## Style guide

Mage follows the [Airbnb JavaScript style guide](https://airbnb.io/javascript/react/).

### Linter

To run the linter locally, execute this script:

```bash theme={"system"}
./scripts/test.sh
```

## Setup

```bash theme={"system"}
yarn global add eslint-config-next
yarn global add eslint_d
```


# Frontend testing
Source: https://docs.mage.ai/contributing/frontend/testing

Making Mage invincible so that nothing can break it, not even itself.

## Prerequisites

* It is not recommended to run tests on your working project, since failed tests can leave behind
  empty pipelines, triggers, etc. Set up your
  [development environment](https://docs.mage.ai/contributing/development-environment).
* Make sure [user authentication](https://docs.mage.ai/production/authentication/overview) is enabled.
* Make sure your development server is running.

## Configuration

* Tests default to "fail" if a client error occurs. Add the following to the top of the test file to disable this check.

```ts theme={"system"}
test.use({ failOnClientError: false });
```

* Certain setting features are enabled by default. Refer to `base.ts` for more information, and use the following snippet when needed.

```ts theme={"system"}
test.use({ settingFeaturesToDisable: { [FeatureUUIDEnum.LOCAL_TIMEZONE]: true } });
```

## Running the tests

You have the option to just run one specific test or all tests at once. There are
[several ways](https://playwright.dev/docs/running-tests) to run specific tests.

```bash theme={"system"}
# Run everything, headless by default.
yarn test
# Run everything with a browser.
yarn test:h

# Run test files that have the word "pipelines" in their names.
yarn test pipelines
```

You can run the frontend tests using the CI/CD Playwright configuration with the following command:

```bash theme={"system"}
yarn test:ci
```

## Creating a test

Refer to the [Playwright docs](https://playwright.dev/docs/writing-tests) for examples on writing frontend tests.

## Debugging

The easiest way to debug is to run tests in UI mode so you can see what went wrong.

```bash theme={"system"}
yarn test --ui

# Or
yarn test:ci --ui
```

If you are running into issues with the app and would like to report it, you can tell us about it either on [GitHub](https://github.com/mage-ai/mage-ai/issues/new/choose) or [Slack](https://www.mage.ai/chat).


# Contributing
Source: https://docs.mage.ai/contributing/overview

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

![Contributing](https://media.giphy.com/media/5cZbRBLhW4tc4/giphy.gif)

## 🥾 Choose your adventure

There *so* many ways to contribute to Mage! We welcome all contributions, no matter how small.

Start your contribution journey below, whether you're an *apprentice* or a *master of spells*. Once you feel confident in your path, [read on](/contributing/development-environment) to learn how to get started.

> **🧪 Alchemist**
>
> Thanks for leading the charge! The easiest way to contribute is to report bugs or suggest new features.
>
> ### Report bugs
>
> Head on over to our [Slack](https://mage.ai/chat) and let us know what's up. We recommend the `#bugs-troubleshoot-questions` channel.
>
> ### Squash bugs
>
> We're always trying to make our project better, but we need your help! Take a look at our [GitHub issues](https://github.com/mage-ai/mage-ai/issues) for small to medium complexity projects and bugs that need fixing.

> **🤹 Loremaster**
>
> ### Suggest new features
>
> We *love* your suggestions. First take a look at our [roadmap](/about/roadmap) to see if your idea is already planned. You might also consider checking out open issues on [GitHub](https://github.com/mage-ai/mage-ai).
>
> Otherwise, drop a line in our [Slack](https://mage.ai/chat) and let us know what you're thinking. We recommend the `#feature-requests` channel.
>
> ### Write documentation
>
> We have *a lot* of docs. We really hope they're fun to read, but we know they can be improved.
>
> The great news is that they're fun to write, too! The *easiest* way to contribute is by scrolling to the bottom of the page you'd like to edit (maybe this one) and clicking the "Edit this page" link. You'll be taken to GitHub where you can make your changes and submit a pull request.
>
> ![](https://mintcdn.com/mage/KCVtYQD46PIylqWZ/media/edit-docs.png?fit=max\&auto=format\&n=KCVtYQD46PIylqWZ\&q=85\&s=d7203cb15c020b1c930cb499473bd53d)
>
> If you like working in code, that's great too! All of our docs are written in markdown, so you can just pull Mage from [GitHub](https://github.com/mage-ai/mage-ai) and start writing.

> **🗣️ Bard/Herald**
>
> ### Community Support
>
> If you're already an adept Mager, we need your help! We're looking for community members to help answer questions in our [Slack](https://mage.ai/chat). Check out the `#bugs-troubleshoot-questions`, `#general`, and `#feature-requests` channels to get started. Anything helps!
>
> If you want to take it up a notch, check out our GitHub issues! We have a lot of [open issues](https://github.com/mage-ai/mage-ai/issues) that need your help.

> **🧱 Stonemason**
>
> ### Backend development
>
> We're always looking for help with the Mage backend— that might look like refactoring code, adding new features, or fixing bugs.
>
> [Here's](https://github.com/mage-ai/mage-ai/pull/3497) an example of an *amazing* backend PR that drastically improved dbt functionality in Mage.
>
> Check out our [roadmap](https://airtable.com/shrJctaGt4tvU27uF/tbl5bIlFkk2KhCpL4) for small to large complexity projects that need your help!
>
> ### Data integration
>
> Mage supports [data integration](/data-integrations/overview) pipelines via the Singer spec. We're always looking for help adding/updating our sources and destinations.
>
> Here's an example of a [great community PR](https://github.com/mage-ai/mage-ai/pull/2277) & here's our [docs on adding a source](/contributing/data-integrations/adapt-existing-source).
>
> ### IO operations (SQL block) and templates
>
> Want to help build *awesome* tools *inside* the pipeline builder page? Help out with our IO operations by adding new SQL blocks! Here's a PR that implements a [DuckDB IO Class](https://github.com/mage-ai/mage-ai/pull/3463).
>
> The simplest way you can help is by adding data loader/exporter templates. Here's an example PR that adds a few for [DuckDB](https://github.com/mage-ai/mage-ai/pull/3553).

> **👨‍🎨 Illuminator**
>
> ### Frontend development
>
> Want to help build the *future* of the Mage UI? Help out with our frontend development! We're always looking for help with UI/UX, accessibility, and more. Checkout our [frontend](/contributing/frontend/overview) page for more info.

Got questions? Live chat with us in [Slack](https://www.mage.ai/chat)

Anything you contribute, the Mage team and community will maintain. We’re in it
together!


# Styling
Source: https://docs.mage.ai/contributing/styling



## Python

Ruff is the repository's only canonical Python formatter. It uses single quotes and
a 100-character line length. Run the pinned version from the repository environment
only on the Python files changed by your PR:

```bash theme={"system"}
isort <changed-python-files>
pre-commit run ruff-format --files <changed-python-files>
python -m ruff check <changed-python-files>
flake8 <changed-python-files>
pre-commit run --files <changed-python-files>
```

Do not use Black or format whole directories as part of a feature or bug-fix PR.
Repository-wide formatting should be isolated in a dedicated formatting PR.

## Frontend

```bash theme={"system"}
eslint_d --fix --ext .js,.jsx,.ts,.tsx
git diff --name-status master | grep '^[AM?]' | awk '{$1=""; sub("^[ \t]+", ""); print}' | grep '\.\(js\|jsx\|ts\|tsx\)$' | xargs eslint_d --config mage_ai/frontend/.eslintrc.js --fix --ext .js,.jsx,.ts,.tsx
```


# Using data integrations in batch pipelines (in Beta)
Source: https://docs.mage.ai/data-integrations/batch-pipelines

Load data from data integration sources and export to data integration destinations as blocks inside batch pipelines.

![Data integrations combined with batch pipelines](https://img.wattpad.com/5ea336f507cf05c821ffbba96cf6c7739fe8dc3e/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f776174747061642d6d656469612d736572766963652f53746f7279496d6167652f4672554e706c504b46744b7469413d3d2d313038313931343836362e313638396434663466313232653937363936373434323438363033362e676966)

> [!WARNING]
>
> Available in version **`0.9.35`** and greater.

## Overview

You can use a [data integration source](/data-integrations/sources/overview) (e.g. Stripe)
and/or a [data integration destination](/data-integrations/destinations/overview) (e.g. Trino)
as a block in batch pipelines.

### Why is this important?

Combine the low-code aspect and capabilities of data integrations with the flexibility and
dynamic capabilities of a batch pipeline.

### Example

1. Use a Python data loader block to write custom code for fetching Stripe credentials.
2. Use a SQL data loader block to fetch data from a table in PostgreSQL.
3. Add a data integration source block to fetch data from Stripe.
4. Use the output from the Python data loader block and interpolate the Stripe credentials into the data integration source block.
5. Add a Python transformer block to merge, clean, and aggregate data fetched from PostgreSQL and Stripe.
6. Add a data integration destination block that writes the output data from the Python transformer block and export it to SalesForce.

> [!TIP]
>
> The image below doesn’t reflect the above example exactly.
>
> The image shows a different set of blocks than the example above.

![Example](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/block-runs.png)

***

## Features

* Use any data integration source.
* Use any data integration destination.
* Sources and destinations have capabilities of a normal Python block in a batch pipeline.
* Dynamically calculate the source and destination credentials at runtime.
* Dynamically build the source and destination catalog schema at runtime.
* Source blocks can output its fetched data to any downstream block.
* Any block can output its data to a destination block and have it exported to that destination.
* Support incremental sync
* `[Coming soon]` *Support log based sync*.
* `[Coming soon]` *Replicate source and destination blocks*.
* `[Coming soon]` *Support source and destination blocks as dynamic blocks*.

***

## How to use

1. Go to the [project preferences](http://localhost:6789/settings/workspace/preferences) and
   enable the feature labeled **`Data integration in batch pipeline`**.
   > [!TIP]
   >
   > This feature requires the `Add new block v2` feature to be enabled.

2. Create a new batch pipeline.

3. Add a new Data loader block by selecting:

   1. **Templates** →
   2. **Data loader** →
   3. **Sources** →
   4. Select the source to add to the pipeline.

   ![Add source block](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/add-source-block.png)

4. Configure the source:

   1. Add credentials.
   2. Select 1 or more streams to sync.
   3. Setup stream settings.
   4. Select 1 or more columns to sync from stream.

   ![Source block](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/source-block.png)

5. Add a new Data exporter block by selecting:

   1. **Templates** →
   2. **Data exporter** →
   3. **Destinations** →
   4. Select the source to add to the pipeline.

   ![Add destination block](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/add-destination-block.png)

6. Configure the destination:

   1. Add credentials.
   2. Select 1 or more stre.ams to export.
   3. Setup stream settings.
   4. Select 1 or more columns to include in the destination table when exporting stream.

   ![Destination block](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/destination-block.png)

***

## Configure source and destination

For more information on how to configure and setup a data integration source and destination,
refer back to the original [data integration guide](/guides/data-integration-pipeline).

### Credentials

![Configure credentials](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/configuration-credentials.png)

### Upstream block settings

![Configure upstream block settings](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/configuration-upstream-block-settings.png)

### Select streams

![Select streams](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/streams.png)

### Streams overview with bulk editing

![Streams overview with bulk editing](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/overview.png)

### Stream detail overview

For more information on how to configure settings for a stream,
refer back to the original [data integration guide](/guides/data-integration-pipeline).

![Stream detail overview](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/stream-detail-overview.png)

### Stream detail schema properties with bulk editing

For more information on how to configure schema properties for a stream,
refer back to the original [data integration guide](/guides/data-integration-pipeline).

![Stream detail schema properties with bulk editing](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/stream-detail-schema-properties.png)

### Stream detail sample data

![Stream detail sample data](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/stream-detail-sample-data.png)

***

## Python blocks

When adding a data integration source or destination block to a batch pipeline,
you can choose the language to be YAML or Python (default option is YAML).

If you choose Python as the language, you programmatically define the following settings of the
data integration:

| Configuration | Example | Decorator | Description |
| - | - | - | - |
| Source | `stripe` | `@data_integration_source` | The source to load data from. |
| Destination | `salesforce` | `@data_integration_destination` | The destination to export data to. |
| Config | `database='...'` | `@data_integration_config` | A dictionary containing the credentials and other settings specific to the source or destination. |
| Selected streams | `['account']` | `@data_integration_selected_streams` | A list of stream names to be selected from the catalog. |
| Catalog | `{...}` | `@data_integration_catalog` | A dictionary containing a list of streams and the stream’s settings, schema properties, and metadata. |

### `@data_integration_source`

> [!TIP]
>
> This decorated function is required.

```python theme={"system"}
@data_integration_source
def source(*args, **kwargs) -> str:
    return 'postgresql'
```

### `@data_integration_destination`

> [!TIP]
>
> This decorated function is required.

```python theme={"system"}
@data_integration_destination
def destination(*args, **kwargs) -> str:
    return 'postgresql'
```

### `@data_integration_config`

> [!TIP]
>
> This decorated function is required.

```python theme={"system"}
@data_integration_config
def config(*args, **kwargs) -> Dict:
    destination: str = kwargs.get('destination', None)
    return {
        'database': '',
        'host': '',
        'password': '',
        'port': 5432,
        'schema': '',
        'table': '',
        'username': '',
    }
```

#### Keyword arguments inside the decorated function

The following keyword arguments are only available in the decorated function’s body for sources.

| | |
| - | - |
| Keyword | `source` |
| Sample code | `kwargs['source']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the source |
| Available in | Sources only |

| | |
| - | - |
| Keyword | `destination` |
| Sample code | `kwargs['destination']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the destination |
| Available in | Destinations only |

### `@data_integration_selected_streams`

> [!NOTE]
>
> This decorated function is optional.

```python theme={"system"}
@data_integration_selected_streams(discover_streams: bool = False)
def selected_streams(*args, **kwargs) -> List[str]:
    config: Dict = kwargs.get('config', None)
    discover_streams_func: Callable = kwargs.get('discover_streams_func', None)
    source: str = kwargs.get('source', None)

    return []
```

#### Decorator arguments

The following arguments are only available in the decorator `@data_integration_selected_streams`
for sources.

| | |
| - | - |
| Argument | `discover_streams` |
| Argument values | `False`, `True` |
| Description | If `True`, there will be a list of stream names available in the function’s body via the keyword argument `selected_streams`; e.g. `kwargs['selected_streams']`. This list of stream names are retrieved by the source’s underlying code in the [Mage Integrations library](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources). |
| Available in | Sources only |

#### Keyword arguments inside the decorated function

The following keyword arguments are only available in the decorated function’s body.

| | |
| - | - |
| Keyword | `source` |
| Sample code | `kwargs['source']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the source |
| Available in | Sources only |

| | |
| - | - |
| Keyword | `destination` |
| Sample code | `kwargs['destination']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the destination |
| Available in | Destinations only |

| | |
| - | - |
| Keyword | `config` |
| Sample code | `kwargs['config']` |
| Return value type | `dict` |
| Sample return value | `{ 'database': '...' }` |
| Description | The dictionary containing the credentials and other configurations. |
| Available in | Sources and destinations |

| | |
| - | - |
| Keyword | `selected_streams` |
| Sample code | `kwargs['selected_streams']` |
| Return value type | `List[str]` |
| Sample return value | `['account', 'users']` |
| Description | A list of strings containing all the available stream names for the source. This is only available if `discover_streams` is `True`. |
| Available in | Sources only |

| | |
| - | - |
| Keyword | `discover_streams_func` |
| Sample code | `kwargs['discover_streams_func']` |
| Return value type | `Callable` |
| Sample return value | If function is invoked, the return value is the same as `selected_streams` above. |
| Description | A list of stream names are retrieved by the source’s underlying code in the [Mage Integrations library](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources). |
| Available in | Sources only |

### `@data_integration_catalog`

> [!NOTE]
>
> The returned dictionary will override the catalog setup through the
> user interface.

```python theme={"system"}
@data_integration_catalog(discover: bool = False, select_all: bool = True)
def catalog(*args, **kwargs) -> Dict:
    config: Dict = kwargs.get('config', None)
    selected_streams: List[str] = kwargs.get('selected_streams', None)
    source: str = kwargs.get('source', None)

    # catalog_from_discover is None unless discover=True
    catalog_from_discover: Dict = kwargs.get('catalog', None)

    # Executing this function will fetch and return the catalog
    discover_func: Callable = kwargs.get('discover_func', None)

    return {
        'streams': [],
    }
```

#### Decorator arguments

The following arguments are only available in the decorator `@data_integration_catalog`
for sources.

| | |
| - | - |
| Argument | `discover` |
| Argument values | `False`, `True` |
| Description | If `True`, the source will fetch the catalog and all stream schema properties then pass the returned value to the decorated function’s keyword arguments. |
| Available in | Sources only |

| | |
| - | - |
| Argument | `select_all` |
| Argument values | `True`, `False` |
| Description | If `False`, the streams in the catalog won’t be selected by default and you’ll have to manually select the stream via the stream’s metadata. |
| Available in | Sources only |

#### Keyword arguments inside the decorated function

The following keyword arguments are only available in the decorated function’s body.

| | |
| - | - |
| Keyword | `source` |
| Sample code | `kwargs['source']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the source |
| Available in | Sources only |

| | |
| - | - |
| Keyword | `destination` |
| Sample code | `kwargs['destination']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the destination |
| Available in | Destinations only |

| | |
| - | - |
| Keyword | `config` |
| Sample code | `kwargs['config']` |
| Return value type | `dict` |
| Sample return value | `{ 'database': '...' }` |
| Description | The dictionary containing the credentials and other configurations. |
| Available in | Sources and destinations |

| | |
| - | - |
| Keyword | `selected_streams` |
| Sample code | `kwargs['selected_streams']` |
| Return value type | `List[str]` |
| Sample return value | `['account', 'users']` |
| Description | A list of strings containing the available stream names returned from the decorated function under `@data_integration_selected_streams`. |
| Available in | Sources and destinations |

| | |
| - | - |
| Keyword | `catalog` |
| Sample code | `kwargs['catalog']` |
| Return value type | `dict` |
| Sample return value | `{ 'streams': [] }` |
| Description | A dictionary with a key named `streams` that is a list of dictionaries containing the stream schema properties and metadata. This is only available if `discover` is `True`. |
| Available in | Sources only |

| | |
| - | - |
| Keyword | `discover_func` |
| Sample code | `kwargs['discover_func']` |
| Return value type | `Callable` |
| Sample return value | If function is invoked, the return value is the same as `catalog` above. |
| Description | If function is invoked, the source will fetch the catalog and all stream schema properties. |
| Available in | Sources only |

***

## Data from upstream blocks

The data integration source and destination block can depend on upstream blocks.

Those upstream blocks can pass its output data into the data integration source and destination,
which then can be used as:

* Input arguments to interpolate information in the credentials
  (or other decorated functions if source or destination block is a Python block).
* Data to be exported to a destination.

### Input arguments

Follow these steps to use the output of an upstream block as an input:

1. Open the data integration block’s configuration and click on the tab
   labeled **Upstream block settings**.

   ![Configure upstream block settings](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/configuration-upstream-block-settings.png)
2. Toggle the name of the block you want as an input.
3. Check the box labeled **Use the block’s output data as an input**.

> [!TIP]
>
> Inputs are available as positional arguments for all decorated functions in Python blocks.

#### Other options

> **Only use this block’s output data as an input**
>
> If checked, then the upstream block’s output data is only used as inputs and won’t be ingested
> and exported to the destination. This is only applicable to destination blocks.

> **Use catalog as an input**
>
> If checked, then the upstream block’s catalog will be included as part of the input argument(s)
> for the data integration block.

#### Examples using upstream data output as inputs

If you have an upstream block, named `pg_credentials`, that has the following code:

```python theme={"system"}
@custom
def get_credentials(*args, **kwargs):
    return dict(
        database='dangerous',
        host='host.docker.internal',
        password='postgres',
        port=5432,
        schema='mage',
        username='postgres',
    )
```

Here are examples on how to use the upstream block’s output to configure credentials at runtime:

> **YAML blocks**
>
> If you have a data integration source or
> destination block that depends on the above upstream block,
> you can interpolate the output of the upstream block in the source or destination block’s
> credentials like this:
>
> ```yaml theme={"system"}
> database: |
>   {{ block_output('pg_credentials', parse='lambda x: x["database"]') }}
> host: |
>   {{ block_output('pg_credentials', parse='lambda x: x["host"]') }}
> password: |
>   {{ block_output('pg_credentials', parse='lambda x: x["password"]') }}
> port: |
>   {{ block_output('pg_credentials', parse='lambda x: x["port"]') }}
> schema: |
>   {{ block_output('pg_credentials', parse='lambda x: x["schema"]') }}
> username: |
>   {{ block_output('pg_credentials', parse='lambda x: x["username"]') }}
> ```

> **Python blocks**
>
> Depending on how many upstream blocks your source or destination block uses as an input,
> it’ll determine how many positional arguments are available in any of your decorated functions.
>
> ```python theme={"system"}
> @data_integration_config
> def config(credentials, *args, **kwargs) -> Dict:
>     return {
>         'database': credentials['database'],
>         'host': credentials['host'],
>         'password': credentials['password'],
>         'port': credentials['port'],
>         'schema': credentials['schema'],
>         'username': credentials['username'],
>     }
> ```

#### Examples using upstream block’s catalog as input

If you have an upstream block, named `titanic_survival`, that has the following code:

```python theme={"system"}
import io
import pandas as pd
import requests


@data_loader
def load_data_from_api(*args, **kwargs):
    url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'
    response = requests.get(url)
    return pd.read_csv(io.StringIO(response.text), sep=',')
```

And its output data has the following Pandas data types:

| Column | Data type | Sample data |
| - | - | - |
| PassengerId | `int64` | 2 |
| Survived | `int64` | 1 |
| Pclass | `int64` | 1 |
| Name | `object` | Mrs. John Bradley (Florence Briggs Thayer) |
| Sex | `object` | female |
| Age | `float64` | 38 |
| SibSp | `int64` | 1 |
| Parch | `int64` | 0 |
| Ticket | `object` | PC 17599 |
| Fare | `float64` | 71.2833 |
| Cabin | `object` | C85 |
| Embarked | `object` | C |

Here are examples on how to use the upstream block’s catalog as an input to the data integration’s
decorated functions:

> **Python blocks**
>
> Depending on how many upstream blocks your source or destination block uses as an input,
> it’ll determine how many positional arguments are available in any of your decorated functions.
>
> ```python theme={"system"}
> @data_integration_catalog
> def catalog(titanic_survival_data_and_catalog, *args, **kwargs) -> Dict:
>     _data, catalog = titanic_survival_data_and_catalog
>
>     return catalog
> ```
>
> The positional argument correlated to the upstream block that you’re using as an input
> will be either a single object or a tuple. It’s a tuple if you checked the box
> labeled **Use catalog as an input** in the data integration block’s configuration on
> the tab labeled **Upstream block settings**.
>
> In this example, the last item in the tuple from the 1st positional argument (
> e.g. `titanic_survival_data_and_catalog`) contains a Python dictionary that represents
> the catalog from the upstream block named `titanic_survival`.
>
> This catalog contains a key named `streams` that is a list of dictionaries containing
> stream settings, metadata, and schema properties of the output data from the upstream
> block `titanic_survival`.
>
> Here is an example of that catalog (a lot of the metadata and schema properties are omitted
> from this example so that it’s not too lengthy):
>
> ```python theme={"system"}
> {
>   'streams': [
>     {
>       'metadata': [
>         {
>           'breadcrumb': [
>             'properties',
>             'Embarked',
>           ],
>           'metadata': {
>             'inclusion': 'available',
>             'selected': True,
>           },
>         },
>       ],
>       'schema': {
>         'properties': {
>           'PassengerId': {
>             'type': [
>               'null',
>               'integer',
>             ],
>           },
>         },
>         'type': 'object',
>       },
>       'stream': 'titanic',
>       'type': 'SCHEMA',
>       'replication_method': 'FULL_TABLE',
>     },
>   ],
> }
> ```

### Data to be exported

> [!NOTE]
>
> Outputs from upstream blocks, as data to be exported, can only be used by destination blocks.

The outputs from upstreams blocks will not only be available as an input,
but it’ll also be used as a data stream when ingesting and exporting to the destination.

You can control which upstream block’s data is ingested and exported by configuring the
destination block’s streams.

![Select streams](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/streams.png)

***

## Output data from source blocks

> [!NOTE]
>
> Destination blocks don’t output any data to downstream blocks.

A data integration source block will output the data fetched from the source and pass that
data to its downstream blocks.

A downstream data integration destination block will automatically know how to ingest the data
and process it for exporting.

### Shape of output data

All other types of blocks will receive the data integration source block’s output data either as
a dictionary or as a list containing 2 or more dictionaries.

The output data will be a dictionary if the source block is only fetching 1 stream.

The output data will be a list of dictionaries if the source block is fetching 2 or more streams.
The number of items in the list corresponds to the number of streams that the source block is
configured to sync.

#### Shape of dictionary

Each item in the list is a dictionary with the following keys:

| Key | Value data type | Sample value | Description |
| - | - | - | - |
| `stream` | `dict` | `{ 'stream': 'accounts' }` | The dictionary for key `stream` contains information about the stream’s name (via the `stream` key), settings (e.g. replication method), metadata, and schema properties. |
| `columns` | `List[str]` | `['id', 'power', 'created_at']` | A list of column names from the data that was fetched for the stream. |
| `rows` | `List[List[Any]]` | `[[1, 3.5, '3000-01-01']]` | A list of lists where each item in the outer list is a row that was fetched from the stream. Each item in the inner list is a value from a column. The column that the value corresponds to is determined by the index of that value in its list and the column name at that same index in the `columns` list. |

### Sample output data

> **JSON format**
>
> ```json theme={"system"}
> {
>   "columns": [
>     "passengerid",
>     "survived",
>     "pclass",
>     "name",
>     "sex",
>     "age",
>     "sibsp",
>     "parch",
>     "ticket",
>     "fare",
>     "cabin",
>     "embarked",
>     "_mage_created_at",
>     "_mage_updated_at"
>   ],
>   "stream": {
>     "tap_stream_id": "titanic",
>     "replication_method": "FULL_TABLE",
>     "key_properties": [],
>     "schema": {
>       "properties": {
>         "passengerid": {
>           "type": [
>             "null",
>             "integer"
>           ]
>         }
>       },
>       "type": "object"
>     },
>     "stream": "titanic",
>     "metadata": [
>       {
>         "breadcrumb": [],
>         "metadata": {
>           "table-key-properties": [],
>           "forced-replication-method": "FULL_TABLE",
>           "valid-replication-keys": [],
>           "inclusion": "available",
>           "schema-name": "titanic",
>           "selected": true
>         }
>       },
>       {
>         "breadcrumb": [
>           "properties",
>           "passengerid"
>         ],
>         "metadata": {
>           "inclusion": "available",
>           "selected": true
>         }
>       }
>     ],
>     "auto_add_new_fields": false,
>     "unique_conflict_method": "UPDATE",
>     "run_in_parallel": false
>   },
>   "rows": [
>     [
>       1,
>       0,
>       3,
>       "Braund, Mr. Owen Harris",
>       "male",
>       22.0,
>       1,
>       0,
>       "A/5 21171",
>       7.25,
>       null,
>       "S",
>       "2023-09-23T04:14:44.850136-07:00",
>       "2023-09-23T04:14:44.850136-07:00"
>     ],
>     [
>       1,
>       0,
>       3,
>       "Braund, Mr. Owen Harris",
>       "male",
>       22.0,
>       1,
>       0,
>       "A/5 21171",
>       7.25,
>       null,
>       "S",
>       "2023-09-23T04:37:30.366570-07:00",
>       "2023-09-23T04:37:30.366570-07:00"
>     ],
>     [
>       2,
>       1,
>       1,
>       "Cumings, Mrs. John Bradley (Florence Briggs Thayer)",
>       "female",
>       38.0,
>       1,
>       0,
>       "PC 17599",
>       71.2833,
>       "C85",
>       "C",
>       "2023-09-23T04:14:44.850153-07:00",
>       "2023-09-23T04:14:44.850153-07:00"
>     ],
>     [
>       2,
>       1,
>       1,
>       "Cumings, Mrs. John Bradley (Florence Briggs Thayer)",
>       "female",
>       38.0,
>       1,
>       0,
>       "PC 17599",
>       71.2833,
>       "C85",
>       "C",
>       "2023-09-23T04:37:30.366592-07:00",
>       "2023-09-23T04:37:30.366592-07:00"
>     ],
>     [
>       3,
>       1,
>       3,
>       "Heikkinen, Miss. Laina",
>       "female",
>       26.0,
>       0,
>       0,
>       "STON/O2. 3101282",
>       7.925,
>       null,
>       "S",
>       "2023-09-23T04:14:44.850155-07:00",
>       "2023-09-23T04:14:44.850155-07:00"
>     ],
>     [
>       3,
>       1,
>       3,
>       "Heikkinen, Miss. Laina",
>       "female",
>       26.0,
>       0,
>       0,
>       "STON/O2. 3101282",
>       7.925,
>       null,
>       "S",
>       "2023-09-23T04:37:30.366594-07:00",
>       "2023-09-23T04:37:30.366594-07:00"
>     ],
>     [
>       4,
>       1,
>       1,
>       "Futrelle, Mrs. Jacques Heath (Lily May Peel)",
>       "female",
>       35.0,
>       1,
>       0,
>       "113803",
>       53.1,
>       "C123",
>       "S",
>       "2023-09-23T04:14:44.850157-07:00",
>       "2023-09-23T04:14:44.850157-07:00"
>     ]
>   ]
> }
> ```

***

## Load/Export stream data in parallel

For each stream in a data integration source block,
you can enable that stream to have its data fetched in parallel
while other streams for that source is being fetched.

For each stream in a data integration destination block,
you can enable that stream to have its data exported in parallel
while other streams for that destination is being exported.

This option can be turned on by editing a specific stream’s settings.

![Stream detail overview](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/stream-detail-overview.png)

This option can also be turned on or off for multiple streams at once by bulk editing all the
streams at once in the **Overview** section:

![Streams overview with bulk editing](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/overview.png)

***

## Incremental sync

Synchronize your data incrementally after the 1st sync.

When you choose 1 or more bookmark values in a stream, the value from that property will be
saved after each successful sync. The next sync will use that value to compare to the next
batch of records that need to be synced.

### Override bookmark values

If you want to change the last saved bookmark value for the next sync, you can do this in 2 ways:

> **Trigger pipeline to run once**
>
> <video />

> **New trigger or edit an existing trigger**
>
> <video />

### Example of bookmark values editor

![Override bookmark values.](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/incremental/bookmark-values.png)


# Configure data integration pipelines
Source: https://docs.mage.ai/data-integrations/configuration

There are many ways to configure your data integration pipeline.

![Configure data integrations](https://media3.giphy.com/media/v1.Y2lkPTc5MGI3NjExNnFzZnZudjd1OGNnbmgxMmZjZm1pdHBoeTRmdzFwaWhmN2VicjRuaCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/rMGDB5lVFWg9O/giphy.gif)

## Interpolation

You can interpolate pipeline variables, environment variables, secrets, and more into the source
config file, schema settings, and destination config file.

### Pipeline variables

```
{{ variables('variable_name') }}
```

#### Variable names

You can add your own custom variables to the pipeline. For more information,
read the [documentation](/getting-started/runtime-variable#using-runtime-variables).

Here are the default variables available to interpolate:

| Name | Description | Sample value |
| - | - | - |
| `pipeline.name` | Name of the current pipeline. | `ETL demo` |
| `pipeline.uuid` | UUID of the current pipeline. | `etl_demo` |
| `ds` | Execution date string. | `2023-12-25` |
| `hr` | Execution hour string from `00` to `23`. | `12` |
| `env` | Environment Mage is running in. | `dev`, `prod` |
| `execution_partition` | The current partition of the execution: datetime in this format `%Y%m%dT%H%M%S`. | `20231225T122520` |

### Common prefixes

Mage supports table prefixes for integration sources. To add a prefix, add the following to the source configuration for a data integration pipeline:

```yaml theme={"system"}
_patterns:
  destination_table: MY_PREFIX_{{ variables('stream') }}
```

For example, configuring a PostgreSQL source:

![Configure data integrations](https://mage-ai.github.io/assets/integration-prefix.png)

### Environment variables

```
{{ env_var('environment_variable_name') }}
```

### Secrets

Use the following syntax to interpolate secrets:

```
{{ mage_secret_var('your_secret_name') }}
```

For more information on secrets, read the [documentation](/development/variables/secrets).


# Airtable
Source: https://docs.mage.ai/data-integrations/destinations/airtable

How to configure and authenticate the Airtable destination for loading data into Airtable via the API.

![Airtable Logo](https://1000logos.net/wp-content/uploads/2022/05/Airtable-Logo.png)

Connect Mage to Airtable as a **destination** to load data into your bases. This guide explains how to configure authentication using a Personal Access Token and the required base ID.

***

## Configuration Parameters

To use Airtable as a destination, provide the following configuration values:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `token` | Airtable Personal Access Token with the required scopes | `pat1234abcd...` | ✅ |
| `base_id` | The unique ID of the Airtable base you're writing to (starts with `app`) | `app5678efgh...` | ✅ |

***

## Generating Credentials

To generate your Airtable Personal Access Token:

1. Visit the [Airtable Developer Hub](https://airtable.com/developers/web/guides/personal-access-tokens).
2. Click **Create token**.
3. Assign the appropriate scopes (listed below).
4. Save the token securely and include it in your Mage destination configuration.

***

## Required Scopes

Make sure your token includes the following scopes:

* `schema.bases:write` — Required for schema access
* `data.records:write` — Required to insert, update, or delete records in Airtable

***

## FAQs

### What is `base_id` in Airtable?

The `base_id` is a unique identifier for your Airtable base. It typically starts with `app` and can be found in the base’s URL or retrieved using the Airtable API.

### How should I store my Airtable token securely?

Use environment variables or Mage’s secrets manager to store your token. Avoid hardcoding tokens in your configuration files or version control.

### Can Mage write data to multiple Airtable bases?

Yes, but you must configure each destination with its corresponding `base_id` and ensure the access token has permission for each base.

***

## Related Resources

* [Airtable API Docs](https://airtable.com/developers/web/api/introduction)
* [Schema Guide](https://airtable.com/developers/web/guides/schema)
* [Personal Access Token Guide](https://airtable.com/developers/web/guides/personal-access-tokens)


# Amazon S3
Source: https://docs.mage.ai/data-integrations/destinations/amazon_s3

How to configure Amazon S3 as a destination in Mage to write data to your S3 bucket in CSV or Parquet format.

Configure **Amazon S3** as a Mage destination to write data files in CSV or Parquet format directly into your AWS S3 bucket.

***

## Configuration Parameters

The following parameters are required to set up Amazon S3 as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `aws_access_key_id` | Your AWS access key ID. | `AKIA...` | ✅ |
| `aws_secret_access_key` | Your AWS secret access key. | `xyz456...` | ✅ |
| `aws_region` | AWS region where your S3 bucket is located. | `us-west-2` | ✅ |
| `role_arn` | (Optional) ARN of an IAM role to assume for access. Useful for temporary or cross-account access. | `arn:aws:iam::111111:role/example-role` | ❌ |
| `bucket` | The name of the S3 bucket where data will be written. | `user_generated_content` | ✅ |
| `file_type` | The file format to use when writing to S3. Supported values: `csv`, `parquet`. | `parquet` | ✅ |
| `object_key_path` | The S3 object key prefix (path) to write files under. Do **not** include `s3://`, the bucket name, or table name in this path. | `users/ds/20221225` | ✅ |
| `column_header_format` | (Optional) Format for column headers. Will be applied before writing to S3. Options: `upper`, `lower`, or `null` (no formatting). | `lower` | ❌ |
| `date_partition_format` | (Optional) Format for date-based partitioning in file paths. If null, partitioning is skipped. Examples: `%Y%m%d`, `%Y%m%dT%H`. | `%Y%m%d` | ❌ |
| `aws_endpoint` | (Optional) Custom endpoint URL for non-AWS S3-compatible services (e.g., MinIO). | `https://play.min.io` | ❌ |

***

## Credential Requirements

To use Amazon S3, you need an AWS IAM user or role with permissions to:

* Put objects (`s3:PutObject`)
* Create folders (`s3:PutObject`)
* Optionally assume roles (`sts:AssumeRole`) if `role_arn` is used

Ensure the credentials provided have appropriate access to the target bucket and path.

***

## FAQs

### Can I write to MinIO or other S3-compatible storage?

Yes. Set the `aws_endpoint` field to your custom endpoint URL (e.g., `https://play.min.io`). Ensure your credentials and permissions are configured accordingly.

### How are files named in S3?

Mage automatically handles file naming using table names and batch metadata. Your `object_key_path` acts as the prefix under which the files are written.

### What happens if I don’t set `date_partition_format`?

Files will be written directly into the specified path without date-based folder partitioning.

### What if both `role_arn` and access keys are set?

Mage will attempt to assume the IAM role using the provided access key credentials. This is helpful in cross-account setups or for short-lived access via STS.

***

## Related Resources

* [Amazon S3 Documentation](https://docs.aws.amazon.com/s3/)
* [IAM Policy Reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html)
* [MinIO Documentation](https://min.io/docs/minio/linux/index.html)


# BigQuery
Source: https://docs.mage.ai/data-integrations/destinations/bigquery

How to configure BigQuery as a destination in Mage for loading data into your Google Cloud BigQuery datasets.

Use **BigQuery** as a Mage destination to load structured data into Google Cloud’s fully-managed, serverless data warehouse.

***

## Configuration Parameters

When configuring BigQuery as a destination, provide the following parameters:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `path_to_credentials_json_file` | Absolute path to your Google Cloud service account credentials file (JSON). If Mage is running on a GCP instance, leave this as `null` to use the instance's service account. | `/path/to/credentials.json` | ✅ |
| `project_id` | The GCP project ID where your BigQuery dataset exists. | `example_project` | ✅ |
| `dataset` | Name of the BigQuery dataset where data will be written. | `example_dataset` | ✅ |
| `location` | (Optional) The region of your BigQuery dataset. If not set, BigQuery may infer it automatically. | `us-west1` | ❌ |
| `disable_update_column_types` | If `false`, Mage will alter existing BigQuery column types to match the schema when necessary. If `true`, existing types will not be modified. | `false` (default) | ❌ |
| `use_batch_load` | Use BigQuery **load jobs** instead of the streaming API. Recommended for better performance and cost efficiency. | `true` (default) | ❌ |
| `credentials_info` | Alternative to `path_to_credentials_json_file`. Directly embed your credentials JSON as a dictionary (useful for environment-based configs). | See structure below | ❌ |

### `credentials_info` Structure

If you're passing credentials inline instead of a path, structure it like this:

```yaml theme={"system"}
auth_provider_x509_cert_url: str
auth_uri: str
client_email: str
client_id: str
client_x509_cert_url: str
private_key: str
private_key_id: str
project_id: str
token_uri: str
type: str
```

***

## Optional Configuration

| Key | Description | Example Value | Default |
| - | - | - | - |
| `skip_schema_creation` | If `true`, Mage will **not** run a `CREATE SCHEMA` command. Useful if the schema already exists. See [issue #3416](https://github.com/mage-ai/mage-ai/issues/3416). | `true` | `false` |
| `lower_case` | If `true`, column names will be automatically converted to lowercase before writing to BigQuery. | `true` | `true` |

## Mage Pro Features

The following features are available only in [**Mage Pro**](https://cloud.mage.ai):

### Advanced Configuration (Mage Pro only)

| Key | Description | Default |
| - | - | - |
| `truncate_table_before_full_table_sync` | If `true`, Mage will truncate (empty) the destination table before performing a full table sync. This ensures no old data remains. | `false` |

***

## Required Permissions

Your GCP service account or user must have the following permissions to use BigQuery as a destination:

```text theme={"system"}
bigquery.datasets.create
bigquery.jobs.create
bigquery.readsessions.create
bigquery.readsessions.getData
```

Additionally, the account must have the **BigQuery Data Editor** role on the target dataset:

* Role: `roles/bigquery.dataEditor`

***

## Related Resources

* [BigQuery API Overview](https://cloud.google.com/bigquery/docs/reference/rest)
* [Google Cloud IAM Roles](https://cloud.google.com/bigquery/docs/access-control)
* [Mage GitHub Issue #3416](https://github.com/mage-ai/mage-ai/issues/3416)


# ClickHouse
Source: https://docs.mage.ai/data-integrations/destinations/clickhouse

How to configure ClickHouse as a destination in Mage to write data to a ClickHouse database.

Configure **ClickHouse** as a Mage destination to write tabular data into a ClickHouse database using the SQLAlchemy driver.

***

## Configuration Parameters

Provide the following parameters to use ClickHouse as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `sqlalchemy_url` | SQLAlchemy connection string used to connect to ClickHouse. Includes host, user, password, etc. | `clickhouse://default:@localhost/default` | ✅ |
| `table_name` | (Optional) Name of the destination table to write data into. If omitted, the stream name will be used. | `abc123` | ❌ |

***

## SQLAlchemy URL Format

The `sqlalchemy_url` should follow this structure:

```
clickhouse://<username>:<password>@<host>:<port>/<database>
```

**Examples:**

* Without password (using default database):\
  `clickhouse://default:@localhost/default`

* With credentials:\
  `clickhouse://user:pass123@127.0.0.1:9000/analytics_db`

Ensure the database exists in ClickHouse before running the Mage pipeline.

***

## Requirements

* The ClickHouse database must be reachable from the Mage environment.
* Mage requires the [`clickhouse-connect`](https://pypi.org/project/clickhouse-connect/) Python package to be installed.
* The target table should match the schema or allow dynamic schema creation depending on your setup.

***

## FAQs

### Can I use a secure connection?

Yes. To use HTTPS or TLS-secured connections, ensure your SQLAlchemy URL includes the appropriate protocol and port, and that your ClickHouse server supports it.

### What happens if `table_name` is not provided?

Mage will automatically use the stream name as the destination table name.

***

## Related Resources

* [ClickHouse SQLAlchemy Docs](https://clickhouse.com/docs/en/integrations/sqlalchemy)
* [ClickHouse Official Documentation](https://clickhouse.com/docs/en/)
* [clickhouse-connect PyPI Package](https://pypi.org/project/clickhouse-connect/)


# Databricks
Source: https://docs.mage.ai/data-integrations/destinations/databricks

How to configure Databricks as a destination in Mage to write pipeline data to Databricks SQL warehouses using Unity Catalog.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

Use **Databricks** as a destination in Mage to load structured data into your Databricks workspace. Mage integrates with Databricks SQL warehouses and uses **Unity Catalog** for table management.

This destination is ideal for exporting transformed pipeline data to Databricks for analytics, ML workloads, or data lakehouse operations.

***

## Required Configuration

Provide the following credentials when configuring Databricks as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `access_token` | Personal access token to authenticate with Databricks | `dapi123abc...` | ✅ |
| `server_hostname` | Hostname of your Databricks workspace | `dbc-123456.cloud.databricks.com` | ✅ |
| `http_path` | HTTP path of the Databricks SQL warehouse or cluster | `/sql/1.0/warehouses/abc123` | ✅ |
| `schema` | Name of the schema (database) within the catalog | `analytics` | ✅ |
| `table` | Name of the table to write data into | `user_events` | ✅ |

***

## Optional Configuration

| Key | Description | Example Value | Default |
| - | - | - | - |
| `catalog` | Name of the Unity Catalog to use | `workspace` | - |
| `skip_schema_creation` | If `true`, Mage won't run `CREATE SCHEMA`. Useful if the schema already exists. | `true` | `false` |
| `lower_case` | If `true`, all column names will be lowercased. | `true` | `true` |
| `allow_reserved_words` | If `true`, Mage will allow use of SQL reserved words as column names. | `false` | `false` |

***

## Notes

* **Unity Catalog**: Set `catalog` to your Unity Catalog name. This is an optional field for table management in Databricks.
* **Access Token**: Generate a personal access token from your Databricks workspace settings.
* **HTTP Path**: Find the HTTP path in your SQL warehouse or cluster connection details.
* **Permissions**: Ensure your access token has permissions to create schemas and write to tables in the specified catalog and schema.

***

## Related Resources

* [Databricks SQL Documentation](https://docs.databricks.com/sql/)
* [Unity Catalog Overview](https://docs.databricks.com/data-governance/unity-catalog/index.html)
* [Databricks Personal Access Tokens](https://docs.databricks.com/dev-tools/auth/pat.html)


# Delta Lake (Azure)
Source: https://docs.mage.ai/data-integrations/destinations/delta_lake_azure

How to configure Delta Lake on Azure as a destination in Mage to write data to a Delta table using ABFS.

## Configuration Parameters

To use **Delta Lake on Azure** as a destination, provide the following parameters:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `account_name` | Your Azure Storage account name. This is the name of the account that hosts your Delta table. | `myazurestorage` | ✅ |
| `access_key` | The access key for your Azure Storage account. Used to authenticate writes to your Delta table. | `ABC123...XYZ` | ✅ |
| `table_uri` | URI for the Delta table. Must follow the `abfs://<container>/<path>` format. For more options, refer to the [delta-rs docs](https://delta-io.github.io/delta-rs/python/usage.html). | `abfs://my-container/delta/my-table` | ✅ |
| `mode` | Write mode. - `append`: Adds rows to the table. Errors if schema changes. - `overwrite`: Replaces all existing rows and recreates the table. | `append` (default) or `overwrite` | ✅ |

***

## Notes

* The table URI must point to a valid **Delta Lake table path** accessible via Azure Data Lake Storage (Gen2) using the **ABFS** protocol.
* Mage uses the [delta-rs](https://delta-io.github.io/delta-rs/python/usage.html) engine for interacting with Delta Lake.
* If using `overwrite` mode, ensure downstream systems tolerate table recreation.

***

## Related Resources

* [Delta Lake Python API (`delta-rs`)](https://delta-io.github.io/delta-rs/python/usage.html)
* [Delta Lake Overview](https://delta.io/)
* [Azure Storage ABFS URI Format](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri)


# Delta Lake (S3)
Source: https://docs.mage.ai/data-integrations/destinations/delta_lake_s3

How to configure Delta Lake on AWS S3 as a destination in Mage to write data to a Delta table using S3 URIs.

## Configuration Parameters

To use **Delta Lake on AWS S3** as a destination in Mage, provide the following parameters:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `aws_access_key_id` | Your AWS access key ID. Required to authenticate with S3. | `AKIAIOSFODNN7EXAMPLE` | ✅ |
| `aws_secret_access_key` | Your AWS secret access key. Used for writing to the S3 bucket. | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` | ✅ |
| `aws_region` | AWS region where your S3 bucket is located. | `us-west-2` *(default)* | ✅ |
| `bucket` | Name of the S3 bucket where the Delta table is stored. | `mage-spark-data` | ✅ |
| `table` | Logical name of the Delta table that Mage will write to. | `dim_profiles_v1` | ✅ |
| `mode` | Write mode: - `append`: Adds rows to the table. Fails if schema changes. - `overwrite`: Replaces all existing rows and recreates the table. | `append` (default) or `overwrite` | ✅ |
| `object_key_path` | Path inside the S3 bucket where the Delta table is written. Exclude `s3://`, bucket name, and table name from this path. | `users/ds/20250708` | ✅ |

***

## Notes

* The object key path is relative to the S3 bucket and should not include the scheme (`s3://`), bucket name, or table name.
* Mage uses the [delta-rs](https://delta-io.github.io/delta-rs/python/usage.html) engine for writing data to Delta Lake on S3.
* Use `overwrite` mode with caution: it deletes all existing rows and recreates the table.

## Preview in Mage Pro

Mage Pro Data Preview, Data Explorer, AI Sidekick exploration, and Analytics
chart datasets can read a bounded window from a Delta Lake table on S3 when the
source is configured as `delta_lake_s3`.

Preview supports either a full table URI:

```yaml theme={"system"}
provider: delta_lake_s3
uri: s3://analytics-lake/warehouse/orders
```

Or a destination-style config:

```yaml theme={"system"}
provider: delta_lake_s3
bucket: analytics-lake
object_key_path: warehouse
table: orders
```

Mage reads AWS credentials from the selected `io_config.yaml` profile when one
is provided. Use a profile with read access to the target table path.

***

## Related Resources

* [Delta Lake Python API (`delta-rs`)](https://delta-io.github.io/delta-rs/python/usage.html)
* [Delta Lake Overview](https://delta.io/)
* [Working with S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html)


# Doris
Source: https://docs.mage.ai/data-integrations/destinations/doris

How to configure Doris as a destination in Mage to write data to a Doris database table, including SSH tunnel and optional connection settings.

## Configuration Parameters

To use **Doris** as a destination in Mage, provide the following parameters:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `database` | Name of the Doris database to export data to. | `demo` | ✅ |
| `host` | Hostname or IP address of your Doris database server. | `mage.abc.us-west-2.rds.amazonaws.com` | ✅ |
| `port` | Port number on which Doris is listening. The default is usually `9030`. | `9030` | ✅ |
| `username` | Username used to connect to the Doris database. Must have permission to read and write data. | `root` | ✅ |
| `password` | Password for the above user. | `abc123...` | ✅ |
| `table` | Name of the destination table in Doris that Mage will create or write to. | `dim_users_v1` | ✅ |
| `connection_method` | Method used to connect to Doris. Choose between `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` | ✅ |
| `ssh_host` | *(Optional)* Bastion host IP or DNS if using SSH tunnel. | `123.45.67.89` | ❌ |
| `ssh_port` | *(Optional)* Port of the bastion host. Defaults to `22`. | `22` | ❌ |
| `ssh_username` | *(Optional)* Username to authenticate with the bastion server. | `bastion_user` | ❌ |
| `ssh_password` | *(Optional)* Password for SSH authentication. Required if not using a private key. | `password` | ❌ |
| `ssh_pkey` | *(Optional)* Path to or contents of private key file used for SSH authentication. | `/path/to/private/key.pem` | ❌ |
| `conn_kwargs` | *(Optional)* Additional [MySQL connection options](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) as a dictionary. | `{"ssl_ca": "CARoot.pem"}` | ❌ |
| `use_lowercase` | *(Optional)* Whether to convert column names to lowercase. | `true` or `false` | ❌ |

***

## Optional Settings

| Key | Description | Example Value | Required |
| - | - | - | - |
| `skip_schema_creation` | If `true`, Mage will not run the `CREATE SCHEMA` command. Useful if the schema already exists. [Related issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` | ❌ |
| `lower_case` | If `true`, Mage will enforce lowercase column names. Default is `true`. | `true` | ❌ |

***

## Notes

* **SSH tunneling** is useful for secure access to Doris behind firewalls or private networks.
* `conn_kwargs` is especially helpful if you need to enable SSL or set advanced connection options.
* Doris uses a MySQL-compatible protocol, so many standard MySQL configurations also apply.

***

## Frequently Asked Questions

### Can I connect to Doris using an SSH tunnel?

Yes. Set `connection_method` to `ssh_tunnel` and provide the appropriate `ssh_host`, `ssh_username`, and either `ssh_password` or `ssh_pkey` depending on your authentication method.

### Can I use a private key for SSH authentication?

Yes. Use the `ssh_pkey` parameter to provide the file path or content of the SSH private key. This is recommended over using `ssh_password`.

### Does Doris support SSL connections?

Yes. Use the `conn_kwargs` parameter to pass MySQL-compatible SSL options like `ssl_ca`, `ssl_cert`, and `ssl_key`.

### What happens if I use `overwrite` mode?

Mage will drop the existing table (if it exists) and recreate it, replacing all existing data. Make sure your downstream systems can handle this reset.

### What’s the difference between `use_lowercase` and `lower_case`?

Both settings control whether column names are converted to lowercase. `use_lowercase` applies during export, while `lower_case` can be used to enforce this behavior globally. We recommend keeping them consistent.

***

## Related Resources

* [Doris Official Documentation](https://doris.apache.org/)
* [MySQL Connector Python Docs](https://dev.mysql.com/doc/connector-python/en/)
* [How to Configure SSH Tunnel Connections](https://www.ssh.com/academy/ssh/tunneling)


# DuckDB
Source: https://docs.mage.ai/data-integrations/destinations/duckdb

How to configure DuckDB as a destination in Mage to write pipeline data to local DuckDB files for fast, embedded analytics.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

![DuckDB Logo](https://miro.medium.com/v2/resize:fit:1400/format:webp/0*Qcgc4cmJk5lFGBwi.png)

***

## Overview

Use **DuckDB** as a destination in Mage to write structured data to local DuckDB database files. DuckDB is an embedded analytical database that runs in-process, making it ideal for fast local analytics, data staging, or lightweight data processing workflows.

This destination is perfect for scenarios where you need to quickly analyze data locally before pushing to a larger data warehouse, or for building lightweight analytics pipelines.

***

## Required Configuration

You must enter the following configuration when writing to a DuckDB destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `file_path` | Absolute file path to the `.duckdb` file where data will be saved. | `/home/src/default_repo/mage_output.duckdb` | ✅ |
| `if_exists` | What to do if the table already exists: `replace`, `append`, or `fail`. | `replace` | ✅ |

***

## Configuration Options

### `if_exists` Values

| Value | Description |
| - | - |
| `replace` | Overwrites the existing table with new data. The table will be recreated on each pipeline run. |
| `append` | Adds new data to the existing table. Existing rows remain unchanged. |
| `fail` | Raises an error if the table already exists. Useful for preventing accidental overwrites. |

***

## Notes

* **Embedded Database**: DuckDB is an embedded database. The file is local to the machine or container where Mage is running.
* **Pandas Integration**: Supports writing pandas DataFrames directly to tables in a `.duckdb` file.
* **File Path**: Use absolute paths to ensure the file is created in the correct location.
* **Performance**: DuckDB is optimized for analytical queries and can handle large datasets efficiently.
* **Use Cases**: Ideal for fast, local analytics or staging data before pushing to a warehouse.

***

## Sample Configuration

```yaml theme={"system"}
file_path: /home/src/default_repo/mage_output.duckdb
if_exists: replace
```

***

## Related Resources

* [DuckDB Documentation](https://duckdb.org/docs/)
* [DuckDB Python API](https://duckdb.org/docs/api/python/overview)
* [DuckDB GitHub Repository](https://github.com/duckdb/duckdb)


# Elasticsearch
Source: https://docs.mage.ai/data-integrations/destinations/elasticsearch

How to configure Elasticsearch as a destination in Mage to stream or batch index records into an Elasticsearch cluster using custom index metadata, authentication, and bulk operations.

![Elasticsearch logo](https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt280217a63b82a734/6202d3378b1f312528798412/elastic-logo.svg)

***

## Configuration Parameters

By default, the Mage `table` name is used to set the Elasticsearch **index** name.

| Key | Description | Default | Required |
| - | - | - | - |
| `scheme` | HTTP scheme used to connect to Elasticsearch (`http` or `https`). | `http` | ✅ |
| `host` | Elasticsearch host. Can be a domain or IP address. | `localhost` | ✅ |
| `port` | Elasticsearch port. | `9200` | ✅ |
| `username` | Username for basic authentication. | `None` | ❌ |
| `password` | Password for basic authentication. | `None` | ❌ |
| `bearer_token` | Bearer token for token-based authentication. | `None` | ❌ |
| `api_key_id` | API key ID for key-based authentication. | `None` | ❌ |
| `api_key` | API key for key-based authentication. | `None` | ❌ |
| `ssl_ca_file` | Path to SSL certificate file used for TLS verification. | `None` | ❌ |
| `verify_certs` | Whether to verify SSL certificates when using HTTPS. | `True` | ❌ |
| `index_schema_fields` | JSONPath mapping for selecting values from the record to generate index names dynamically. | `None` | ❌ |
| `_op_type` | Operation type for the Elasticsearch request. Typically `index`, but can be `create`, `update`, or `delete`. | `index` | ❌ |
| `metadata_fields` | Dictionary used to map fields in the input record to metadata (e.g. `_id`) in the Elasticsearch index request. | `None` | ❌ |
| `bulk_kwargs` | Settings that control Elasticsearch `bulk` behavior. Useful for tuning performance and retries. See table below. | `None` | ❌ |

***

## Metadata Field Mapping

Use `metadata_fields` to extract values from incoming records and apply them to Elasticsearch document metadata (e.g., `_id`):

```yaml theme={"system"}
metadata_fields:
  <stream_name>:
    <field>: <jsonpath>
```

**Example:**

```yaml theme={"system"}
metadata_fields:
  stream_name:
    _id: my_id
```

> This will set the `_id` of each indexed document to the value of the `my_id` field in the input record.

***

## Bulk Operation Settings (`bulk_kwargs`)

All options below are optional, but useful for controlling indexing performance and fault tolerance.

| Key | Description | Default |
| - | - | - |
| `use_parallel` | If `true`, enables parallel bulk indexing using multiple threads. Can significantly improve throughput. | `false` |
| `chunk_size` | Max number of records per bulk request. | `500` |
| `max_chunk_bytes` | Max size in bytes of each bulk request payload. | `104857600` |
| `max_retries` | *(Only with `use_parallel: false`)* Number of retry attempts on HTTP 429 (too many requests) errors. | `0` |
| `initial_backoff` | *(Only with `use_parallel: false`)* Initial wait time (in seconds) before retry. Each subsequent retry backs off exponentially. | `2` |
| `max_backoff` | *(Only with `use_parallel: false`)* Maximum time to wait before retrying. | `600` |
| `thread_count` | *(Only with `use_parallel: true`)* Number of threads to use in the bulk operation thread pool. | `4` |
| `queue_size` | *(Only with `use_parallel: true`)* Size of the task queue buffering chunks between the main thread and processing threads. | `4` |

***

## Example Configuration

```yaml theme={"system"}
scheme: http
host: localhost
port: 9200
metadata_fields:
  stream_name:
    _id: my_id
bulk_kwargs:
  use_parallel: true
  chunk_size: 500
```

***

## Notes

* You can authenticate with **basic auth**, **bearer tokens**, or **API keys**.
* Set `_op_type: create` if you want to avoid overwriting documents with the same `_id`.
* For production environments, it's recommended to use `https`, enable `verify_certs`, and configure `ssl_ca_file`.
* Performance tuning with `bulk_kwargs` is especially helpful for large datasets or real-time indexing.

***

## Related Resources

* [Elasticsearch Bulk API Docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)
* [Elasticsearch Python Client (elasticsearch-py)](https://elasticsearch-py.readthedocs.io/en/v8.11.1/)
* [Elasticsearch Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)


# Google Cloud Storage
Source: https://docs.mage.ai/data-integrations/destinations/google_cloud_storage

How to configure Google Cloud Storage (GCS) as a destination in Mage to write data files in Parquet or CSV format with optional date partitioning.

## Overview

Use **Google Cloud Storage (GCS)** as a destination in Mage to write data from your pipelines directly into a GCS bucket. This is ideal for exporting pipeline outputs in formats like **Parquet** or **CSV** for downstream analytics, storage, or ML workflows.

Mage supports native integration with GCS using either a service account key file or automatic authentication via GCP instance metadata.

***

## Configuration Parameters

You must provide the following parameters when configuring GCS as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `bucket` | Name of the target GCS bucket where files will be stored. | `user_generated_content` | ✅ |
| `google_application_credentials` | Path to the GCP service account JSON key file. Leave as `null` if Mage is running on GCP and should use the default instance service account. | `/path/to/key.json` | ✅ |
| `file_type` | Format of the files to be written. Supported values: `parquet`, `csv`. | `parquet` or `csv` | ✅ |
| `object_key_path` | Relative path inside the bucket where files will be stored. Do **not** include `gs://`, bucket name, or table name. | `users/ds/20221225` | ✅ |
| `date_partition_format` | *(Optional)* Datetime format to use for partitioned folders. If `null`, partitioning will be skipped. Examples: `%Y%m%d`, `%Y%m%dT%H`. | `null` or `%Y%m%d` | ❌ |

***

## Notes

* The `object_key_path` defines where in the GCS bucket the data will be written. Mage will automatically append file extensions based on the selected `file_type`.
* If you're running Mage inside a GCP service (like Cloud Run, GKE, or Compute Engine), and you leave `google_application_credentials` as `null`, Mage will use the instance's IAM identity for authentication.
* Use `date_partition_format` to enable directory-based partitioning by time, which is useful for managing time-series datasets or incremental loads.

***

## Related Resources

* [Google Cloud Storage Documentation](https://cloud.google.com/storage/docs/introduction)
* [GCS URI Naming Guidelines](https://cloud.google.com/storage/docs/gsutil/addlhelp/HowNamingWorks)
* [GCP Service Account Authentication](https://cloud.google.com/docs/authentication/getting-started)

***

## Frequently Asked Questions

### Can Mage write to a GCS bucket if it's running inside GCP?

Yes. If Mage is running inside a GCP environment (like GKE, Cloud Run, or Compute Engine), you can leave `google_application_credentials` as `null`. Mage will use the default service account attached to the instance for authentication.

***

### What file formats are supported when exporting data to GCS?

Mage currently supports `parquet` and `csv` as the `file_type` options when writing data to Google Cloud Storage.

***

### How does `object_key_path` work?

`object_key_path` specifies the path inside the bucket where your files will be stored. It should exclude the bucket name and should not include `gs://`. For example, `users/2025_07_08`.

***

### What does `date_partition_format` do?

When provided, this setting tells Mage to create date-based subfolders (e.g., `20250708`, `20250708T13`) based on the run time. If set to `null`, partitioning is disabled and all files will go into the specified `object_key_path`.

***

### Do I need to manually set file extensions?

No. Mage automatically adds the appropriate extension (`.parquet` or `.csv`) to each file based on the `file_type` you select.

***

### Can I write to a multi-region or dual-region GCS bucket?

Yes. Mage can write to any standard, multi-region, or dual-region GCS bucket as long as the configured credentials have write access.


# Iceberg S3
Source: https://docs.mage.ai/data-integrations/destinations/iceberg_s3

How to configure Apache Iceberg on S3 as a destination in Mage to write data to ACID-compliant Iceberg tables stored in AWS S3.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

Use **Apache Iceberg on S3** as a destination in Mage to write data to ACID-compliant Iceberg tables stored in AWS S3.

***

## Overview

This destination writes data to Apache Iceberg tables stored in AWS S3. Iceberg provides:

* **ACID Properties**: Full ACID compliance for data consistency
* **Schema Evolution**: Add, remove, or modify columns without breaking existing queries
* **Time Travel**: Query data as it was at any point in time
* **Partitioning**: Automatic partitioning for better query performance
* **Compaction**: Automatic file compaction to optimize storage

***

## Required Configuration

| Parameter | Description | Required | Default |
| - | - | - | - |
| `aws_access_key_id` | S3 access key ID | ✅ | - |
| `aws_secret_access_key` | S3 secret access key | ✅ | - |
| `bucket` | S3 bucket name | ✅ | - |
| `namespace` | Iceberg namespace | ✅ | - |
| `table_name` | Iceberg table name | ✅ | - |

***

## Optional Configuration

| Parameter | Description | Required | Default |
| - | - | - | - |
| `aws_region` | AWS region | ❌ | `us-west-2` |

***

## Configuration Example

```yaml theme={"system"}
aws_access_key_id: your_access_key
aws_region: us-west-2
aws_secret_access_key: your_secret_key
bucket: your-bucket-name
namespace: your_namespace
table_name: your_table_name
```

***

## Table Structure

Iceberg tables are created with the following structure:

```
s3://bucket/namespace.db/table_name/
├── metadata/
│   ├── v1.metadata.json
│   ├── v2.metadata.json
│   └── ...
├── data/
│   ├── 00000-0-abc123.parquet
│   ├── 00001-0-def456.parquet
│   └── ...
└── snapshots/
    ├── 1234567890.avro
    └── ...
```

***

## Usage Examples

### Creating a New Table

The destination automatically creates the table if it doesn't exist, using the schema from the incoming data.

### Appending Data

New data is automatically appended to the existing Iceberg table.

### Schema Evolution

If the incoming data has new columns, the table schema is automatically updated.

***

## Troubleshooting

### Common Issues

1. **Access Denied**: Check your S3 credentials and bucket permissions
2. **Region Mismatch**: Ensure the region matches your S3 bucket configuration

### Debugging

Enable debug logging to see detailed information about table creation and data writing operations.

***

## Related Resources

* [Apache Iceberg Documentation](https://iceberg.apache.org/docs/)
* [Iceberg S3 Configuration](https://iceberg.apache.org/docs/latest/aws/)
* [AWS S3 Documentation](https://docs.aws.amazon.com/s3/)


# Kafka
Source: https://docs.mage.ai/data-integrations/destinations/kafka

How to configure Apache Kafka as a destination in Mage to publish JSON records to a Kafka topic using optional message keys.

![Kafka logo](https://upload.wikimedia.org/wikipedia/commons/thumb/5/53/Apache_kafka_wordtype.svg/2560px-Apache_kafka_wordtype.svg.png)

***

## Overview

Use **Apache Kafka** as a destination in Mage to stream records from your pipeline into Kafka topics in real time. Each record is serialized as a JSON object and sent to the configured `bootstrap_server`.

If a `Key Property` is defined in the Source Schema, Mage will use the corresponding field value as the Kafka message key—enabling partitioned message routing and supporting downstream systems that rely on message keys.

This is ideal for:

* Real-time data streaming
* Event-driven microservices
* Kafka-based data pipelines and messaging queues

***

## Configuration Parameters

| Key | Description | Default | Required |
| - | - | - | - |
| `bootstrap_server` | Comma-separated list of Kafka brokers with ports (e.g., `kafka:9093`, `broker1:9092,broker2:9092`). | None | ✅ |
| `topic` | Kafka topic to publish messages to. | None | ✅ |
| `api_version` | *(Optional)* Kafka API version to use. If unspecified, Mage will auto-detect the server version. | None | ❌ |

***

## Notes

* Records are serialized as **JSON** using the [Kafka Python client](https://kafka-python.readthedocs.io/en/master/).
* To set Kafka message keys, define a `Key Property` in your source schema.
* You can explicitly set the `api_version` to match your Kafka broker version if needed.
* Compatible with self-hosted Kafka, Confluent Cloud, Amazon MSK, and other Kafka-compatible platforms.

***

## Example Configuration

```yaml theme={"system"}
bootstrap_server: broker1.kafka.internal:9092,broker2.kafka.internal:9092
topic: mage_data_stream
api_version: (0, 10, 1)
```

***

## Related Resources

* [Apache Kafka Documentation](https://kafka.apache.org/documentation/)
* [Kafka Python Client (kafka-python)](https://kafka-python.readthedocs.io/en/master/)
* [Confluent Kafka Topic Design](https://developer.confluent.io/learn/kafka/topic-design/)


# MongoDB
Source: https://docs.mage.ai/data-integrations/destinations/mongodb

How to configure MongoDB as a destination in Mage to write data to a MongoDB database using a connection string.

## Overview

Use **MongoDB** as a destination in Mage to export records from your pipelines directly into a MongoDB collection. This is ideal for persisting semi-structured JSON data, powering APIs, or integrating with operational data systems.

Each record from your pipeline will be inserted as a document in the specified MongoDB collection. If no collection name is provided, Mage will default to using the stream name as the collection name.

***

## Configuration Parameters

| Key | Description | Example Value | Required |
| - | - | - | - |
| `connection_string` | MongoDB connection URI string, including hostname and port. | `mongodb://mongodb0.example.com:27017` | ✅ |
| `db_name` | Name of the MongoDB database where documents will be stored. | `admin` | ✅ |
| `table_name` | *(Optional)* Name of the MongoDB collection to write to. If omitted, the stream name is used. | `user_profiles` | ❌ |

***

## Notes

* The `connection_string` should include any authentication, cluster details, or replica set info as needed (e.g., `mongodb+srv://user:pass@cluster0.mongodb.net/`).
* If `table_name` is not set, Mage automatically uses the name of the stream that produced the data.
* Mage inserts each record as a single MongoDB document in the target collection.

***

## Example Configuration

```yaml theme={"system"}
connection_string: mongodb://mongodb0.example.com:27017
db_name: analytics
```

***

## Related Resources

* [MongoDB Connection String URI Format](https://www.mongodb.com/docs/manual/reference/connection-string/)
* [MongoDB Python Driver (`pymongo`)](https://pymongo.readthedocs.io/en/stable/)
* [MongoDB Collections vs Tables](https://www.mongodb.com/docs/manual/core/databases-and-collections/)


# MSSQL (Microsoft SQL Server)
Source: https://docs.mage.ai/data-integrations/destinations/mssql

How to configure Microsoft SQL Server (MSSQL) as a destination in Mage to write pipeline data into a SQL Server database using SQL authentication or Azure AD.

## Overview

Use **Microsoft SQL Server (MSSQL)** as a destination in Mage to export data from your pipelines into relational tables for analytics, reporting, or operational workloads. Mage supports both on-premise SQL Server deployments and cloud-hosted options like **Azure SQL Database**.

***

## Configuration Parameters

You must provide the following credentials to configure the MSSQL destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `database` | Name of the target database where data will be written. | `msdb` | ✅ |
| `host` | Hostname or IP address of your MSSQL server. | `172.20.0.2` | ✅ |
| `port` | Port used by the MSSQL service. Typically `1433`. | `1433` | ✅ |
| `username` | Username with permission to read and write to the specified schema and table. | `guest` | ✅ |
| `password` | Password for the MSSQL user. | `abc123...` | ✅ |
| `schema` | Target schema inside the database. | `public` | ✅ |
| `table` | Name of the destination table Mage will create or write to. | `users` | ✅ |

***

## Optional Parameters

| Key | Description | Example Value |
| - | - | - |
| `authentication` | Authentication method. Set to `ActiveDirectoryServicePrincipal` to use [Azure AD authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver16). | `ActiveDirectoryServicePrincipal` |
| `driver` | Name of the [ODBC driver for SQL Server](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16). | `ODBC Driver 18 for SQL Server` |
| `skip_schema_creation` | If `true`, Mage will skip the `CREATE SCHEMA` command during initialization. [See GitHub issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, all column names will be converted to lowercase. Default is `true`. | `true` |
| `connection_method` | Connection method. Set to `ssh_tunnel` to connect via SSH tunnel. Default: `direct` | `ssh_tunnel` |
| `ssh_host` | (SSH tunnel only) The host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (SSH tunnel only) The port of the intermediate bastion server. Default: `22` | `22` |
| `ssh_username` | (SSH tunnel only) The username used to connect to the bastion server. | `username` |
| `ssh_password` | (SSH tunnel only, optional) The password used to connect to the bastion server. Required if not using `ssh_pkey`. | `password` |
| `ssh_pkey` | (SSH tunnel only, optional) The path to the private key used to connect to the bastion server. Required if not using `ssh_password`. | `/path/to/private/key` |

> [!NOTE]
>
> SSH tunneling is a Mage Pro only feature.

***

## Notes

* Mage uses the **SQLAlchemy MSSQL dialect** to connect and write data into Microsoft SQL Server.
* Ensure the user has `INSERT` and `CREATE TABLE` permissions if writing to new tables.
* Azure-hosted SQL Servers may require SSL or Azure AD authentication depending on your network configuration.
* Make sure any required firewall rules or virtual network settings are properly configured for external access.

***

## Related Resources

* [Microsoft SQL Server Docs](https://learn.microsoft.com/en-us/sql/sql-server/)
* [ODBC Driver for SQL Server](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server)
* [Azure Active Directory Authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver16)


# MySQL
Source: https://docs.mage.ai/data-integrations/destinations/mysql

How to configure MySQL as a destination in Mage to write pipeline data to a MySQL table with optional SSH tunneling and connection parameters.

![MySQL logo](https://user-images.githubusercontent.com/78053898/198754276-a4524211-aafa-40b2-a95a-1d8f08ba6835.png)

***

## Overview

Use **MySQL** as a destination in Mage to write pipeline data into structured tables within a MySQL database. This integration supports both direct connections and **SSH tunneling** for secure access to remote or private databases.

Each record from the Mage pipeline is inserted as a row in the configured table. Mage supports optional connection customization and SSL settings through connection keyword arguments.

***

## Configuration Parameters

You must provide the following parameters when configuring the MySQL destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `database` | Name of the MySQL database where data will be written. | `demo` | ✅ |
| `host` | Hostname or IP address of the MySQL server. | `mage.abc.us-west-2.rds.amazonaws.com` | ✅ |
| `port` | Port number used by MySQL (typically `3306`). | `3306` | ✅ |
| `username` | Username with permission to access the database and target schema. | `root` | ✅ |
| `password` | Password for the MySQL user. | `abc123...` | ✅ |
| `table` | Name of the destination table that Mage will write to. | `dim_users_v1` | ✅ |
| `connection_method` | Method used to connect to the MySQL server: `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` | ✅ |
| `ssh_host` | *(Optional)* Host of the bastion or jump server for SSH tunnel access. | `123.45.67.89` | ❌ |
| `ssh_port` | *(Optional)* SSH port on the bastion host. Default is `22`. | `22` | ❌ |
| `ssh_username` | *(Optional)* SSH username for authentication. | `username` | ❌ |
| `ssh_password` | *(Optional)* SSH password. Required if not using a private key. | `password` | ❌ |
| `ssh_pkey` | *(Optional)* Path to or content of the private key file used for SSH authentication. | `/path/to/private/key.pem` | ❌ |
| `conn_kwargs` | *(Optional)* Additional [MySQL connection parameters](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) in dictionary format. | `{"ssl_ca": "CARoot.pem"}` | ❌ |
| `use_lowercase` | *(Optional)* Whether to convert all column names to lowercase. | `true` or `false` | ❌ |

***

## Optional Configs

| Key | Description | Example Value |
| - | - | - |
| `skip_schema_creation` | If `true`, Mage will skip running the `CREATE SCHEMA` command. Useful when the schema already exists. [See related issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, all column names will be forced to lowercase. Default is `true`. | `true` |

***

## Notes

* For cloud-hosted MySQL (e.g., Amazon RDS, Azure Database for MySQL), ensure proper firewall rules and access control.
* Use `ssh_tunnel` mode when connecting to databases hosted in private networks or VPCs.
* The `conn_kwargs` parameter is useful for enabling SSL/TLS and advanced options (e.g., `ssl_ca`, `ssl_cert`, `connect_timeout`).
* Mage automatically creates the destination table unless `skip_schema_creation` is set to `true`.

***

## Related Resources

* [MySQL Documentation](https://dev.mysql.com/doc/)
* [MySQL Connector/Python Connect Arguments](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html)
* [SSH Tunneling Explained](https://www.ssh.com/academy/ssh/tunneling)


# OpenSearch
Source: https://docs.mage.ai/data-integrations/destinations/opensearch

How to configure OpenSearch as a destination in Mage to stream pipeline data into OpenSearch indices using JSON records with support for authentication, ECS metadata, and SSL.

## Overview

Use **OpenSearch** as a destination in Mage to publish JSON-formatted records into an OpenSearch index. This is ideal for search indexing, log aggregation, and real-time analytics.

By default, Mage uses the pipeline’s `table` name to set the OpenSearch **index name**. You can also define dynamic index patterns using JSONPath expressions.

***

## Configuration Parameters

| Key | Description | Default | Required |
| - | - | - | - |
| `scheme` | HTTP scheme used for connecting to OpenSearch. Use `http` or `https`. | `http` | ✅ |
| `host` | Hostname or IP address of the OpenSearch node or cluster. | `localhost` | ✅ |
| `port` | Port number for the OpenSearch HTTP endpoint. | `9200` | ✅ |
| `username` | *(Optional)* Username for basic authentication. | `None` | ❌ |
| `password` | *(Optional)* Password for basic authentication. | `None` | ❌ |
| `bearer_token` | *(Optional)* Bearer token for token-based authentication. | `None` | ❌ |
| `api_key_id` | *(Optional)* API key ID for key-based authorization. | `None` | ❌ |
| `api_key` | *(Optional)* API key secret for key-based authorization. | `None` | ❌ |
| `ssl_ca_file` | *(Optional)* Path to the SSL CA certificate file for verifying HTTPS connections. | `None` | ❌ |
| `index_schema_fields` | *(Optional)* JSONPath mapping used to dynamically generate the index name based on record values. | `None` | ❌ |
| `metadata_fields` | *(Optional)* Dictionary mapping fields (using JSONPath) to populate ECS-style metadata fields in the index request (e.g., `_id`, `@timestamp`). | `None` | ❌ |

***

## Notes

* Mage automatically converts each record into JSON format and indexes it into the specified OpenSearch index.
* If `index_schema_fields` is configured, Mage will use the JSONPath-extracted values from each record to generate dynamic index names.
* `metadata_fields` is useful for setting OpenSearch ECS-compatible fields like `_id`, `@timestamp`, etc., from the incoming data stream.
* Authentication can be configured using **basic auth**, **bearer tokens**, or **API key pairs**.
* For production use, we recommend enabling `https` and providing a trusted `ssl_ca_file`.

***

## Related Resources

* [OpenSearch Documentation](https://opensearch.org/docs/)
* [OpenSearch Python Client](https://opensearch-project.github.io/opensearch-py/)
* [ECS (Elastic Common Schema) Fields](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html)
* [OpenSearch Security Configuration](https://opensearch.org/docs/latest/security-plugin/)


# OracleDB
Source: https://docs.mage.ai/data-integrations/destinations/oracledb

How to configure Oracle Database (OracleDB) as a destination in Mage to write data to a specified schema using the thin or thick client.

## Overview

Use **OracleDB** as a destination in Mage to write pipeline data directly into a table within an Oracle database. Mage supports both **thin** and **thick** connection modes, allowing flexibility for local clients, instant clients, or full Oracle installations.

This destination is ideal for enterprise data integration with existing Oracle systems, including Oracle XE, Oracle Cloud Infrastructure (OCI), and on-premise Oracle servers.

***

## Configuration Parameters

You must provide the following credentials when configuring the OracleDB destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `host` | Hostname or IP address of the Oracle database server. | `oracledb.example.com` | ✅ |
| `port` | Port on which the Oracle listener accepts client connections. Typically `1521`. | `1521` | ✅ |
| `service` | Oracle **service name** (not SID). The listener uses this to direct connections to the correct pluggable database. | `xepdb1` | ✅ |
| `user` | Username with permission to connect and write to the target schema. | `oracle_user` | ✅ |
| `password` | Password for the Oracle user. | `xyz123` | ✅ |
| `database` | Name of the Oracle database where the table will be created or updated. | `orcl` | ✅ |
| `mode` | Oracle client mode. Set to either `thin` (default) or `thick` depending on the installed driver. | `thin` or `thick` | ✅ |

***

## Optional Parameters

| Key | Description | Example Value |
| - | - | - |
| `lower_case` | If `true`, all column names will be converted to lowercase. Default is `true`. | `true` |

***

## Notes

* The `mode` parameter determines how the Oracle client connects:
  * Use `thin` for lightweight connections (e.g., with the Oracle Instant Client).
  * Use `thick` when a full Oracle client is installed and native libraries are required.
* Ensure the Oracle listener is correctly configured to accept service name–based connections.
* Use of the `service` name is preferred over SID in modern Oracle setups, especially with pluggable databases (PDBs).

***

## Related Resources

* [Oracle Net Services Overview](https://docs.oracle.com/en/database/oracle/oracle-database/19/netag/introduction-to-oracle-net-services.html)
* [Oracle Instant Client Installation Guide](https://www.oracle.com/database/technologies/instant-client.html)
* [Oracle Database Connection Strings](https://docs.oracle.com/en/database/oracle/oracle-database/19/jjdbc/data-sources-and-url-connection-strings.html)


# Destinations technical documentation
Source: https://docs.mage.ai/data-integrations/destinations/overview

A destination is a system where you want the data to be exported. A destination can be a 3rd party API, SaaS, database, data warehouse, or a data lake.

## Singer spec

Mage uses the data engineering community standard for data integrations called the
[Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).

In addition,
Mage further standardizes the spec and provides common classes and methods to make implementing them easier and faster.

## Folder structure

Destinations are defined in the `mage_integrations/destinations/` directory.

All destinations should be a subclass of the `Destination` class defined in
`mage_integrations/destinations/base.py`.

## Configs

* `maximum_batch_size_mb`: Destination processes records the batches. The maximum batch size can be configured.

## Contribute

Want to learn about contributing a destination? Read on [here](/contributing/data-integrations/add-new-destination)!


# PostgreSQL
Source: https://docs.mage.ai/data-integrations/destinations/postgresql

How to configure PostgreSQL as a destination in Mage to write pipeline data into a PostgreSQL table using direct database credentials and optional schema/table controls.

![PostgreSQL logo](https://user-images.githubusercontent.com/78053898/198754309-2ef713a7-62c8-4ea8-9ebb-8c24ed038cb3.png)

***

## Overview

Use **PostgreSQL** as a destination in Mage to export structured data from your pipelines directly into a PostgreSQL table. Mage supports both self-hosted and cloud-hosted PostgreSQL instances such as **bit.io**, **Supabase**, **Render**, and **AWS RDS**.

Mage will automatically create the target table and insert pipeline output rows into it, with support for schema selection, lowercase column naming, and reserved keyword handling.

***

## Configuration Parameters

Provide the following credentials when setting up PostgreSQL as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `database` | Name of the PostgreSQL database where data will be written. | `demo` | ✅ |
| `host` | Hostname or IP address of the PostgreSQL server. | [`db.bit.io`](https://bit.io/) | ✅ |
| `port` | Port used by the PostgreSQL service. Default is `5432`. | `5432` | ✅ |
| `username` | Username with access to the target database and schema. | `guest` | ✅ |
| `password` | Password for the PostgreSQL user. | `abc123...` | ✅ |
| `schema` | Name of the schema where the table will be created. | `public` | ✅ |
| `table` | Name of the table Mage will write records into. | `dim_users_v1` | ✅ |

***

## Optional Parameters

| Key | Description | Example Value |
| - | - | - |
| `skip_schema_creation` | If `true`, Mage will skip running `CREATE SCHEMA`. Use this if the schema already exists. [See issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will convert all column names to lowercase. Default is `true`. | `true` |
| `allow_reserved_words` | If `true`, Mage will allow column names that are PostgreSQL reserved words. Default is `false`. | `true` or `false` |

***

## Notes

* PostgreSQL is a widely supported open-source relational database—ideal for analytics, app backends, and transformations.
* Ensure the user has `CREATE TABLE`, `INSERT`, and optionally `CREATE SCHEMA` privileges.
* Mage supports exporting data to **existing tables** or **creating new tables** based on the pipeline output schema.
* If your database is hosted on platforms like Heroku, AWS RDS, or bit.io, make sure the network settings allow inbound connections from Mage.

***

## Related Resources

* [PostgreSQL Documentation](https://www.postgresql.org/docs/)
* [bit.io Free PostgreSQL Hosting](https://bit.io/)
* [PostgreSQL Reserved Keywords](https://www.postgresql.org/docs/current/sql-keywords-appendix.html)
* [SQLAlchemy PostgreSQL Dialect](https://docs.sqlalchemy.org/en/20/dialects/postgresql.html)


# Redshift
Source: https://docs.mage.ai/data-integrations/destinations/redshift

How to configure Amazon Redshift as a destination in Mage to write data into Redshift tables using user credentials or IAM-based authentication.

![Amazon Redshift Logo](https://user-images.githubusercontent.com/78053898/198753538-2d606c3a-f6b0-472a-b0b3-c16086f256fc.png)

***

## Overview

Use **Amazon Redshift** as a destination in Mage to export pipeline data into a highly scalable, cloud-based data warehouse. Mage supports both **standard username/password authentication** and **IAM database authentication** using AWS access keys.

Mage will automatically create and write to the specified table, with support for optional schema management and merge-based updates.

***

## Configuration Parameters

You must provide the following credentials when configuring Redshift as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `database` | Name of the Redshift database where data will be written. | `demo` | ✅ |
| `host` | Hostname of the Redshift cluster endpoint. | `mage-prod.3.us-west-2.redshift.amazonaws.com` | ✅ |
| `port` | Redshift port (default is `5439`). | `5439` | ✅ |
| `user` | Redshift username with access to the database and schema. | `awsuser` | ✅ (if not using IAM) |
| `password` | Password for the Redshift user. | `abc123...` | ✅ (if not using IAM) |
| `schema` | Schema within the database where the table will be created or written to. | `public` | ✅ |
| `table` | Name of the destination table to store data from your pipeline. | `dim_users_v1` | ✅ |
| `region` | AWS region where the Redshift cluster is hosted. | `us-west-2` | ✅ |
| `use_merge_load` | If `true`, Mage will attempt to merge incoming data with existing records (UPSERT-like behavior). | `false` *(default)* | ❌ |
| `use_s3_copy` | If `true`, Mage will upload data to S3 and load into Redshift using the COPY command (Pro feature). | `false` *(default)* | ❌ |
| `type_mapping` | Custom mapping to override default column type conversions. Useful for specifying custom Redshift data types like SUPER for JSON data (Pro feature). | see below | ❌ (Pro) |

> **Note**: `use_s3_copy` and `type_mapping` are Mage **Pro-only** features. `use_s3_copy` enables faster bulk loads via S3 and supports IAM roles or access key authentication.

***

## IAM-Based Authentication (Alternative to User/Password)

If you prefer to use **IAM credentials** instead of traditional authentication, you can use the following fields:

| Key | Description | Example Value | Required (if using IAM) |
| - | - | - | - |
| `access_key_id` | AWS access key ID for an IAM user or role with Redshift database access. | `AKIA...` | ✅ |
| `secret_access_key` | AWS secret access key corresponding to the above access key ID. | `xyz123...` | ✅ |
| `cluster_identifier` | Identifier of the Amazon Redshift cluster. | `mage-prod` | ✅ |
| `db_user` | IAM-authenticated Redshift user. | `admin` | ✅ |

***

## S3 Configuration (Required if `use_s3_copy` is true)

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=redshift)

| Key | Description | Required | Sample Value |
| - | - | - | - |
| `aws_access_key_id` | AWS access key for uploading data to S3. Ignored if using IAM role. | ✅ (unless using IAM role) | `AKIA...` |
| `aws_secret_access_key` | AWS secret key for uploading data to S3. Ignored if using IAM role. | ✅ (unless using IAM role) | `xyz...` |
| `aws_region` | Region where the S3 bucket is located. | ✅ | `us-west-2` |
| `bucket` | S3 bucket to temporarily stage Parquet files for Redshift COPY. | ✅ | `mage-feature-sets` |
| `redshift_iam_role` | IAM role ARN with S3 read access attached to your Redshift cluster. | Optional | `arn:aws:iam::123456789012:role/MyRedshiftRole` |

Configure these inside a nested `s3_creds` config object:

```yaml theme={"system"}
use_s3_copy: true
s3_creds:
  aws_access_key_id: AKIA...
  aws_secret_access_key: xyz...
  aws_region: us-west-2
  bucket: mage-feature-sets
  redshift_iam_role: arn:aws:iam::123456789012:role/MyRedshiftRole
```

***

## Type Mapping

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=redshift)

The `type_mapping` configuration allows you to override default column type conversions. This is useful for advanced cases like storing JSON in Redshift’s `SUPER` type or adjusting column widths.

### Example Configuration

**YAML Format:**

```yaml theme={"system"}
type_mapping:
  object: SUPER
  array: SUPER
  string: VARCHAR(1000)
```

**Complete YAML Example:**

```yaml theme={"system"}
database: demo
host: mage-prod.3.us-west-2.redshift.amazonaws.com
password: abc123...
port: 5439
region: us-west-2
schema: public
user: awsuser
use_merge_load: false
use_s3_copy: false
lower_case: true
type_mapping:
  object: SUPER
  array: SUPER
  string: VARCHAR(1000)
  datetime: TIMESTAMPTZ
  number: BIGINT
```

### Supported Overrides

* **`object`** → Redshift type for JSON objects (e.g., `SUPER`)
* **`array`** → Redshift type for JSON arrays (e.g., `SUPER`)
* **`string`** → Custom VARCHAR or other string type
* **`datetime`** → Custom timestamp type (e.g., `TIMESTAMPTZ`)
* **`number`** → Custom numeric type (e.g., `BIGINT`)

> When using `SUPER`, Mage automatically applies `JSON_PARSE()` for correct Redshift insertion.

***

## Optional Parameters

| Key | Description | Example Value |
| - | - | - |
| `skip_schema_creation` | If `true`, Mage will skip running the `CREATE SCHEMA` command. Useful if the schema already exists. [See issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will automatically convert column names to lowercase. Default is `true`. | `true` |

***

## Grant Permissions

To allow Mage to create schemas and insert data, grant the following permissions to your Redshift user or IAM role:

```sql theme={"system"}
GRANT CREATE ON DATABASE your_database TO your_user;
GRANT CREATE ON ALL TABLES IN SCHEMA your_schema TO your_user;
```

Refer to the [Redshift GRANT command documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) for details.

***

## Related Resources

* [Amazon Redshift Documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html)
* [Redshift IAM Database Authentication](https://docs.aws.amazon.com/redshift/latest/mgmt/guid-iam-db-authentication.html)
* [Redshift SQL GRANT Syntax](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html)
* [Redshift JDBC and ODBC Drivers](https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html)


# Salesforce
Source: https://docs.mage.ai/data-integrations/destinations/salesforce

How to configure Salesforce as a destination in Mage to write records using OAuth or username/password authentication with support for insert, update, upsert, and bulk actions.

![Salesforce logo](https://user-images.githubusercontent.com/78053898/198753571-7c033641-75a2-4338-b257-ee37a159915e.png)

***

## Overview

Use **Salesforce** as a destination in Mage to export records into standard or custom Salesforce objects. Mage supports both **OAuth 2.0** and **username/password/token** authentication methods, and enables `insert`, `update`, `upsert`, `delete`, and `hard_delete` actions.

You can also configure behavior for bulk failures, define external ID fields for upserts, and optionally override the source-to-object mapping using `table_name`.

***

## Configuration Parameters

| Key | Description | Required |
| - | - | - |
| `client_id` | OAuth client ID from your Salesforce connected app. | Optional |
| `client_secret` | OAuth client secret from your connected app. | Optional |
| `refresh_token` | OAuth refresh token. Used to obtain new access tokens. | Optional |
| `username` | Username used for Salesforce **username/password** authentication. | Optional |
| `password` | Corresponding Salesforce password. | Optional |
| `security_token` | Salesforce security token (needed with password-based login). | Optional |
| `domain` | Salesforce domain to authenticate against. Use `login` (default), `test` (sandbox), or a custom My Domain. | ✅ |
| `action` | Default operation to perform on incoming records: `insert`, `update`, `upsert`, `delete`, or `hard_delete`. | ✅ |
| `external_id_name` | External ID field for `upsert` operations. Default is `"Id"`. | Optional |
| `allow_failures` | If `true`, failed records will be skipped and processing will continue. Default is `false`. | Optional |
| `table_name` | Override the source stream name with a specific Salesforce object name (e.g., `Account`, `Contact`, `Opportunity`). | Optional |

***

## Authentication Options

You can authenticate to Salesforce using either of the following methods:

### Option 1: OAuth 2.0

* Set `client_id`, `client_secret`, and `refresh_token`.
* Recommended for connected apps and secure access with refreshable tokens.
* [How to obtain credentials →](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/salesforce/README.md#how-to-get-your-client_id-client_secret-and-refresh_token)

### Option 2: Username / Password

* Set `username`, `password`, and `security_token`.
* Recommended for sandbox/testing environments.
* Ensure that IP restrictions are disabled or add your Mage IP to the trusted IP list.

***

## Limitations

To map a **non-default source stream** to a specific Salesforce object, you must explicitly set the `table_name` parameter in your configuration.

* The value of `table_name` **must** match the API name of a Salesforce object (e.g., `Account`, `Lead`, `Opportunity`, or custom objects like `MyObject__c`).

***

## Allow Failures Behavior

When `allow_failures` is set to `true`:

* Records that are **eligible** (i.e., match the Salesforce object schema and are valid for the selected action) but **fail to commit** will be skipped.
* The pipeline will continue processing other records.

When `allow_failures` is set to `false`:

* Any failed record will raise an error and halt the pipeline.

***

## Related Resources

* [Salesforce API Documentation](https://developer.salesforce.com/docs)
* [Salesforce Connected App Setup](https://help.salesforce.com/s/articleView?id=sf.connected_app_create.htm)
* [Salesforce Bulk API](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/)
* [Salesforce Object Reference](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_concepts.htm)


# Snowflake
Source: https://docs.mage.ai/data-integrations/destinations/snowflake

How to configure Snowflake as a destination in Mage to write pipeline data to Snowflake using batch loads, optional key-pair auth, and custom warehouse settings.

![Snowflake logo](https://user-images.githubusercontent.com/78053898/198754338-a8aeb12e-6e23-45e5-b130-7a1979a2b31d.png)

***

## Overview

Use **Snowflake** as a destination in Mage to load structured data into your cloud data warehouse. Mage supports both **row-by-row insertion** and **batch upload** modes, and integrates with Snowflake using **username/password** or **key-pair authentication**.

This destination is ideal for exporting transformed pipeline data for analytics, BI tools, or data lakehouse modeling.

***

## Required Configuration

Provide the following credentials when configuring Snowflake as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `account` | Your Snowflake [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). | `abc1234.us-east-1` | ✅ |
| `database` | Name of the Snowflake database to write to. | `DEMO_DB` | ✅ |
| `schema` | Schema within the database where the table will be created. | `PUBLIC` | ✅ |
| `table` | Target table that Mage will create or insert into. | `dim_users_v1` | ✅ |
| `username` | Snowflake user with `INSERT` and `CREATE TABLE` permissions. | `guest` | ✅ |
| `warehouse` | Virtual warehouse used to execute queries and ingest data. | `COMPUTE_WH` | ✅ |
| `use_batch_load` | If `true`, enables batch loading instead of row-by-row inserts. Recommended for performance. | `true` *(default)* | ✅ |
| `disable_double_quotes` | If `true`, column and table names will not be wrapped in double quotes. Disabling may reduce case-sensitivity issues. | `false` *(default)* | ✅ |

***

## Optional Connection Parameters

| Key | Description | Example Value |
| - | - | - |
| `password` | Password for the Snowflake user. | `abc123...` |
| `private_key_file` | Path to the private key file for key-pair authentication (requires version ≥ 0.9.76). | `/path/to/key.p8` |
| `private_key_file_pwd` | Passphrase for the private key file. | `abc123...` |
| `role` | Snowflake role to use for the session. | `ANALYST_ROLE` |

🔐 [Enable key-pair authentication →](https://docs.snowflake.com/en/user-guide/key-pair-auth)

***

## Optional Mage Settings

| Key | Description | Example Value |
| - | - | - |
| `skip_schema_creation` | If `true`, Mage will skip running the `CREATE SCHEMA` command. Useful if the schema already exists. [See issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will convert all column names to lowercase. Default is `true`. | `true` |

***

## Notes

* Set `use_batch_load: true` for large datasets to improve performance and reduce API calls.
* When using **key-pair authentication**, ensure the private key is generated and stored securely.
* If `disable_double_quotes` is `false`, column names will retain their case. Otherwise, names will be treated as case-insensitive.
* Make sure your user has permission to create tables and write to the specified schema in the selected database and warehouse.

***

## Related Resources

* [Snowflake Account Identifier Format](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html)
* [Snowflake Key-Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth)
* [Snowflake Warehouses Overview](https://docs.snowflake.com/en/user-guide/warehouses.html)
* [Snowflake SQL Reference](https://docs.snowflake.com/en/sql-reference/)


# Teradata
Source: https://docs.mage.ai/data-integrations/destinations/teradata

How to configure Teradata as a destination in Mage to write pipeline data to a Teradata database using standard host, port, and user credentials.

## Overview

Use **Teradata** as a destination in Mage to write structured pipeline data into a Teradata database table. This is ideal for analytics, reporting, or operational workloads in enterprise environments using Teradata’s scalable architecture.

Mage connects using direct credentials and supports exporting to tables in any schema the user has access to.

***

## Configuration Parameters

Provide the following credentials when configuring Teradata as a **destination**:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `database` | Name of the Teradata database where the table will be created or written to. | `demo` | ✅ |
| `host` | Hostname or IP address of the Teradata server. | `test.clearscape.teradata.com` | ✅ |
| `port` | Port used by the Teradata database (commonly `1025`, depending on environment). | `1025` | ✅ |
| `username` | Teradata username with write access to the specified database and schema. | `demo_user` | ✅ |
| `password` | Password for the Teradata user. | `abc123...` | ✅ |

***

## Notes

* Mage will automatically create the target table if it doesn't exist, unless configured otherwise.
* Ensure the user has `CREATE TABLE`, `INSERT`, and `SELECT` privileges in the target schema.
* Teradata uses ANSI SQL but has some syntax differences (e.g., date handling, identity columns) to be aware of.
* This integration is well-suited for batch ingestion pipelines and bulk exports into data marts.

***

## Related Resources

* [Teradata Documentation](https://docs.teradata.com/)
* [Teradata SQL Reference](https://docs.teradata.com/r/Teradata-SQL)
* [Teradata Vantage Overview](https://www.teradata.com/Products/Vantage)
* [Clearscape Analytics](https://www.teradata.com/Products/Clearscape-Analytics)


# Trino
Source: https://docs.mage.ai/data-integrations/destinations/trino

How to configure Trino as a destination in Mage to write pipeline data to Trino-compatible catalogs using supported connectors such as PostgreSQL, Iceberg, Delta Lake, and more.

![Trino logo](https://trino.io/assets/trino-og.png)

***

## Overview

Use **Trino** as a destination in Mage to write data to a variety of underlying data sources including **PostgreSQL**, **Delta Lake**, **Iceberg**, **S3**, and more—via Trino’s supported catalog and connector system.

Trino enables federated query and write access across databases, data warehouses, and data lakes, all through a unified SQL engine.

***

## Configuration Parameters

You must provide the following parameters when configuring Trino as a destination:

| Key | Description | Example Value | Required |
| - | - | - | - |
| `catalog` | Trino catalog used to access the target data source (e.g., PostgreSQL, Iceberg, Delta Lake). | `my_prod_pg` | ✅ |
| `connector` | Trino connector name. Determines how Trino interacts with the catalog's backend. | `postgresql` | ✅ |
| `host` | Hostname or IP address of the Trino coordinator. | `127.0.0.1` | ✅ |
| `port` | Port where the Trino coordinator is running. Default is `8080`. | `8080` | ✅ |
| `username` | Username for authenticating with the Trino coordinator. | `admin` | ✅ |
| `password` | *(Optional)* Password for Trino authentication. | `abc123...` | ❌ |
| `schema` | Schema name where the target table will be created or written to. | `public` | ✅ |
| `table` | Name of the destination table to write data into. | `dim_users_v1` | ✅ |
| `query_max_length` | Maximum character length allowed for Trino SQL query payloads. | `1000000` | ❌ |
| `ssl` | Whether to disable SSL certificate verification. Set `false` to disable. | `false` | ❌ |
| `location` | (Delta Lake only) URI location of the target table or storage bucket. | `s3://my-delta-bucket/` | ❌ |
| `ignore_location_for_temp_tables` | Prevents Trino from setting `WITH LOCATION` for temp tables. Recommended when using Delta Lake with Glue metastore. | `false` *(default)* | ❌ |

***

## Notes on Delta Lake + Glue Integration

When using the **Delta Lake connector** with a **Glue metastore**, tables created with the `WITH LOCATION` property will **not delete underlying data** when dropped.

To avoid residual files when Mage creates temp tables, set:

```yaml theme={"system"}
ignore_location_for_temp_tables: true
```


# Overview
Source: https://docs.mage.ai/data-integrations/overview

Data integration is the process of synchronizing data between two systems.

![Data integrations](https://www.radicalcompliance.com/wp-content/uploads/2021/02/data-meme.gif)

## Requirements

> [!WARNING]
>
> Data integrations are only supported when running Mage using [Docker](/getting-started/setup#using-docker).

## Use cases

### Syncing external data → internal systems

If you have data in 3rd party systems (e.g other SaaS tools) and you want that
data available in your database, data warehouse, or data lake, you can use Mage
to sync data between those systems.

### Syncing internal data → external systems (reverse)

Your internal database or data warehouse may contain application data generated
by your users or customers. This data can be valuable if used in conjunction
with 3rd party SaaS tools.

You can use Mage to sync data from your internal systems, and have that data
appear in various 3rd party SaaS tools.

***

## How to build a data integration pipeline

[Documentation here](/guides/data-integration-pipeline)

***

## Sources

A source is a system that you want to load data from and synchronize it into
another system. A source can be a 3rd party API, SaaS, database, data warehouse,
or a data lake.

Read more about Sources:

* [Technical documentation](/data-integrations/sources/overview)
* [Add a new source](/contributing/data-integrations/add-new-source)

### Available sources

* [Airtable](/data-integrations/sources/airtable)
* [Amazon S3](/data-integrations/sources/amazon_s3)
* [Amplitude](/data-integrations/sources/amplitude)
* [API](/data-integrations/sources/api)
* [Azure Blob Storage](/data-integrations/sources/azure_blob_storage)
* [BigQuery](/data-integrations/sources/bigquery)
* [Chargebee](/data-integrations/sources/chargebee)
* [Commercetools](/data-integrations/sources/commercetools)
* [Couchbase](/data-integrations/sources/couchbase)
* [Datadog](/data-integrations/sources/datadog)
* [Dremio](/data-integrations/sources/dremio)
* [DynamoDB](/data-integrations/sources/dynamodb)
* [Facebook Ads](/data-integrations/sources/facebook_ads)
* [Freshdesk](/data-integrations/sources/freshdesk)
* [Front](/data-integrations/sources/front)
* [Google Ads](/data-integrations/sources/google_ads)
* [Google Analytics](/data-integrations/sources/google_analytics)
* [Google Cloud Storage](/data-integrations/sources/google_cloud_storage)
* [Google Search Console](/data-integrations/sources/google_search_console)
* [Google Sheets](/data-integrations/sources/google_sheets)
* [HubSpot](/data-integrations/sources/hubspot)
* [Intercom](/data-integrations/sources/intercom)
* [Knowi](/data-integrations/sources/knowi)
* [LinkedIn Ads](/data-integrations/sources/linkedin_ads)
* [Microsoft SQL Server](/data-integrations/sources/mssql)
* [Mode](/data-integrations/sources/mode)
* [Monday](/data-integrations/sources/monday)
* [MongoDB](/data-integrations/sources/mongodb)
* [MySQL](/data-integrations/sources/mysql)
* [OracleDB](/data-integrations/sources/oracledb)
* [Outreach](/data-integrations/sources/outreach)
* [Paystack](/data-integrations/sources/paystack)
* [Pipedrive](/data-integrations/sources/pipedrive)
* [PostgreSQL](/data-integrations/sources/postgresql)
* [Postmark](/data-integrations/sources/postmark)
* [PowerBI](/data-integrations/sources/powerbi)
* [Redshift (Amazon Web Services)](/data-integrations/sources/redshift)
* [Salesforce](/data-integrations/sources/salesforce)
* [SFTP](/data-integrations/sources/sftp)
* [Snowflake](/data-integrations/sources/snowflake)
* [Stripe](/data-integrations/sources/stripe)
* [Tableau](/data-integrations/sources/tableau)
* [Twitter Ads](/data-integrations/sources/twitter_ads)
* [Zendesk](/data-integrations/sources/zendesk)
* *Shopify (coming soon)*

### Sources available in Mage Pro

* [Facebook Pages](/data-integrations/sources/facebook_pages)
* [Google Campaign Manager 360](/data-integrations/sources/google_campaign_manager_360)
* [Google Display & Video 360](/data-integrations/sources/google_display_video_360)
* [Instagram](/data-integrations/sources/instagram)
* [Microsoft Ads](/data-integrations/sources/microsoft_ads)
* [QuickBooks](/data-integrations/sources/quickbooks)
* [Salesforce Marketing Cloud](/data-integrations/sources/salesforce_marketing_cloud)
* [TikTok Ads](/data-integrations/sources/tiktok_ads)
* [Instagram](/data-integrations/sources/instagram)
* [QuickBooks](/data-integrations/sources/quickbooks)
* [Xero](/data-integrations/sources/xero)
* [Youtube Analytics](/data-integrations/sources/youtube_analytics)

Mage adheres to the data engineering standard for data integrations called the
[Singer Spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).

All sources (aka taps) that has been developed by the open-source community can
be adapted to use in Mage. Here is a
[short list of available ones](https://www.singer.io/#taps).

> [!NOTE]
>
> Don’t see a source you need?
>
> Check out the next section on **Add a new source** or tell us on slack and we
> can help you build it.

[Join us on Slack](https://www.mage.ai/chat)

***

## Destinations

A destination is a system that you want to export data into. A destination can
be a 3rd party API, SaaS, database, data warehouse, or a data lake.

Read more about destinations:

* [Technical documentation](/data-integrations/destinations/overview)
* [Add a new destination](/contributing/data-integrations/add-new-destination)

### Available destinations

* [Amazon Web Services (AWS) Redshift](/data-integrations/destinations/redshift)
* [Amazon Web Services (AWS) S3](/data-integrations/destinations/amazon_s3)
* [BigQuery](/data-integrations/destinations/bigquery)
* [Clickhouse](/data-integrations/destinations/clickhouse)
* [Delta Lake (S3)](/data-integrations/destinations/delta_lake_s3)
* [Delta Lake (Azure)](/data-integrations/destinations/delta_lake_azure)
* [Elasticsearch](/data-integrations/destinations/elasticsearch)
* [Google Cloud Storage](/data-integrations/destinations/google_cloud_storage)
* [Kafka](/data-integrations/destinations/kafka)
* [Microsoft SQL Server](/data-integrations/destinations/mssql)
* [MongoDB](/data-integrations/destinations/mongodb)
* [MySQL](/data-integrations/destinations/mysql)
* [Opensearch](/data-integrations/destinations/opensearch)
* [OracleDB](/data-integrations/destinations/oracledb)
* [PostgreSQL](/data-integrations/destinations/postgresql)
* [Salesforce](/data-integrations/destinations/salesforce)
* [Snowflake](/data-integrations/destinations/snowflake)
* [Trino](/data-integrations/destinations/trino)
* *Azure Blob Storage (coming soon)*
* *HubSpot (coming soon)*

### Destinations available in Mage Pro

* [Databricks](/data-integrations/destinations/databricks)
* [DuckDB](/data-integrations/destinations/duckdb)
* [Iceberg S3](/data-integrations/destinations/iceberg_s3)

Mage adheres to the data engineering standard for data integrations called the
[Singer Spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).

All destinations (aka targets) that has been developed by the open-source
community can be adapted to use in Mage. Here is a
[short list of available ones](https://www.singer.io/#targets).

> Don’t see a destination you need?
>
> Check out the next section on **Add a new destination** or tell us on
> [Slack](https://www.mage.ai/chat). and we can help you build it.

[Join us on Slack](https://www.mage.ai/chat)


# Airtable
Source: https://docs.mage.ai/data-integrations/sources/airtable



![Airtable](https://1000logos.net/wp-content/uploads/2022/05/Airtable-Logo.png)

## Configuration

To set up the Airtable source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `token` | Airtable personal access token with access to your desired tables. | `pat...` | ✅ |
| `base_id` | The ID of the Airtable base (application) you want to access. | `app...` | ✅ |
| `table_name` | The name or ID of the table to sync. If not specified, all tables will sync. | `tbl...` | |

> **Find your Personal Access Token:**\
> See Airtable’s [guide](https://airtable.com/developers/web/guides/personal-access-tokens) for generating a personal access token.

## Required Scopes

Ensure your Airtable personal access token has the following scopes:

* `data.records:read`
* `schema.bases:read`


# Amazon S3
Source: https://docs.mage.ai/data-integrations/sources/amazon_s3



![aws\_s3\_logo](https://github.com/mage-ai/mage-ai/assets/78053898/a44c1c95-5aff-4fa4-8653-8939ccf12d09)

## Configuration

To set up the Amazon S3 source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `aws_access_key_id` | AWS access key ID. | `abc123` | ✅ |
| `aws_secret_access_key` | AWS secret access key. | `xyz456` | ✅ |
| `aws_region` | Region where the S3 bucket is located. | `us-west-2` | |
| `role_arn` | (Optional) ARN of the IAM role to assume for accessing the S3 bucket. | `arn:aws:iam::111111:role/example-role` | |
| `bucket` | Name of the AWS S3 bucket to read data from. | `user_generated_content` | ✅ |
| `file_type` | Type of files in the bucket. Supported values: `parquet`, `csv`, `infer`. If `infer`, the type is detected based on file extension. | `infer` | |
| `prefix` | Path within the bucket where files are located. Do not include `s3://`, the bucket name, or the table name in this path. | `users/ds/20221225` | |
| `search_pattern` | Regular expression to match files within the prefix. | `test_table\\/.*\\.csv` | |
| `single_stream_in_prefix` | If `true`, all files in the prefix are treated as a single stream. Defaults to `false`. | `false` | |
| `table_configs` | A list of table configurations to configure multiple streams. Each table config must include `prefix`, `search_pattern`, and `table_name`. | `[{"prefix": "users/", "search_pattern": "test_table\\/.*\\.csv", "table_name": "test_table"}]` | |
| `aws_endpoint` | (Optional) Custom S3-compatible endpoint (e.g., for MinIO). | `https://play.min.io` | |


# Amplitude
Source: https://docs.mage.ai/data-integrations/sources/amplitude



![Amplitude](https://user-images.githubusercontent.com/78053898/198753477-f08a7424-00de-4ae3-9d04-e1c1429a7d28.png)

## Configuration

To set up the Amplitude source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `api_key` | Amplitude project's unique API key. | `73bb...` | ✅ |
| `secret_key` | Amplitude project's unique secret key. | `ABC1...` | ✅ |

> **Find your API credentials:**\
> See Amplitude’s [documentation](https://www.docs.developers.amplitude.com/analytics/find-api-credentials/).

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `host` | Amplitude host URL. Standard server: `https://amplitude.com`. EU Residency server: `https://analytics.eu.amplitude.com`. Defaults to `https://amplitude.com`. | `https://analytics.eu.amplitude.com` |

## Query Behavior

Loading data from Amplitude requires a start date and an end date.

Mage automatically passes the start date and end date based on the scheduled trigger you create.\
For example, if your pipeline runs daily, the start date will be 1 day ago, and the end date will be today.

If you prefer to hardcode and customize these dates, follow the [instructions to override runtime variables](https://docs.mage.ai/production/configuring-production-settings/runtime-variable).

Add the following keys and values to your pipeline’s runtime variables:

| Key | Description | Sample Value |
| - | - | - |
| `_start_date` | Date to start fetching events. | `2022-10-01` |
| `_end_date` | Date to stop fetching events. | `2022-10-03` |

## Schema

The following resources are synced:

* [Events](https://www.docs.developers.amplitude.com/analytics/apis/export-api/#response-schema)


# API Source Configuration
Source: https://docs.mage.ai/data-integrations/sources/api

Learn how to configure API sources in Mage to extract data from RESTful and file-based endpoints. This guide covers authentication, pagination, response parsing, and advanced options available in Mage Pro.


Mage supports integrating with any RESTful or file-based API. This allows you to extract data from JSON, CSV, TSV, XLSX responses, with support for pagination, authentication, and schema inference.

📚 Sample public APIs for testing: [https://apipheny.io/free-api/](https://apipheny.io/free-api/)

## Basic Features (OSS)

### Configuration Fields

You must enter the following credentials when configuring this source:

| Key | Description | Required or optional |
| - | - | - |
| `url` | API URL | Required |
| `method` | `GET` or `POST` | Optional: `GET` is default |
| `query` | URL query parameters. | Optional |
| `payload` | When method is `POST`, this payload is used in the request body. | Optional |
| `headers` | Request headers. | Optional |
| `response_parser` | Parse the API response using dot notation. The final result must be an array. | Optional |
| `columns` | If the final data returned from the API or from the `response_parser` is not a JSON object (e.g. it’s an array of strings or an array of array of strings), then you must define the columns. | Required (conditionally) |
| `separator` | If the file extension is TSV, XLSX (or CSV with specific separator), then you can define it | Optional: ',' is default |
| `has_header` | If the file extension is TSV, XLSX or CSV, and contains a header, then you can define it as True | Optional: False is default |

***

## Mage Pro Features

The following features are available only in [**Mage Pro**](https://cloud.mage.ai) and are designed for advanced API integrations at scale:

### Advanced Configuration (Mage Pro only)

These configuration parameters give you greater control over how data is fetched, parsed, and validated from APIs—especially useful for large files, custom encodings, or complex schemas.

| Key | Description | Default |
| - | - | - |
| `encoding` | Specifies the character encoding of the API response, such as `"utf-8"` or `"iso-8859-1"`. Use this to avoid decoding issues with special characters. | `"utf-8"` |
| `verify` | Whether to verify SSL certificates for HTTPS requests. Disabling this is useful in internal or staging environments with self-signed certs. | `True` |
| `schema_discovery_sample_rows` | The number of rows to sample when inferring the schema from API responses or files. A higher value improves schema accuracy for inconsistent data. | `100` |
| `force_all_columns_type` | Forces all columns to be interpreted as a single type: `"string"`, `"integer"`, or `"number"`. Useful for normalizing schema across variable data sources. | *None* |
| `infer_schema_when_syncing` | Enables schema inference at the time of sync (rather than preview), which is ideal for streaming or large file responses. | `False` |

These options enhance your ability to handle diverse data formats and edge cases in production environments, giving you full control over data ingestion quality and reliability.

### Multi-Stream Configuration (Mage Pro only)

Mage Pro supports defining multiple independent API configs using the `streams` key. Each key in the `streams` map represents a separate stream and can define its own URL, headers, parser, and pagination settings.

```yaml theme={"system"}
streams:
  users:
    url: https://api.example.com/users
    headers:
      x-api-key: user-key
    response_parser: data.users

  orders:
    url: https://api.example.com/orders
    headers:
      x-api-key: order-key
    response_parser: data.orders
```

Each stream entry can independently use any configuration field supported in the single-stream format.

### Pagination Configuration

To sync paginated API responses, the `pagination_config` config block supports two types:

* **`"cursor"`** — follows a token provided in the API response (e.g., `next_cursor`)
* **`"page"`** — iterates using a page number or offset (e.g., `page=1`, `page=2`, …)

***

#### Cursor-Based Pagination

Use when the API returns a **next cursor/token** in the response body.

| Key | Description | Required |
| - | - | - |
| `type` | Must be `"cursor"` | Yes |
| `cursor_param` | Name of the query parameter for the cursor (single cursor only) | Yes\* |
| `cursor_params` | List of query parameter names used for multiple cursors | Yes\* |
| `initial_cursor_value` | Initial value(s) to start pagination. Can be a string, list, or dictionary. | No |
| `next_cursor_path` | Dot path(s) in response to extract next cursor. String for single, dictionary for multiple cursors. | Yes |

> \*Either `cursor_param` or `cursor_params` must be provided.
> If `cursor_params` is used, `next_cursor_path` must be a dictionary mapping each param to its response path.

**Example: Single cursor**

```yaml theme={"system"}
pagination_config:
  type: cursor
  cursor_param: next
  initial_cursor_value: null
  next_cursor_path: meta.next_cursor
```

**Example: Multiple cursors**

```yaml theme={"system"}
pagination_config:
  type: cursor
  cursor_params:
    - start
    - end
  initial_cursor_value:
    start: null
    end: null
  next_cursor_path:
    start: meta.start_cursor
    end: meta.end_cursor
```

#### Page-Based Pagination

Use when the API accepts **page numbers** or **offsets** and optionally supports a `limit` parameter.

| Key | Description | Required |
| - | - | - |
| `type` | `"page"` | ✅ |
| `param` | Name of the query parameter that controls page or offset (e.g., `"page"`, `"offset"`) | ✅ |
| `initial_value` | Starting value for the page or offset | ✅ |
| `increment_by` | Value to increase per request (e.g., `1` for pages, `100` for offset) | ✅ |
| `limit_param` | Name of the limit parameter (e.g., `"limit"`, `"per_page"`) | Optional |
| `limit_value` | Limit value per page | Optional |
| `stop_condition.type` | When to stop: `"empty"` (no results) or `"max_value"` (based on total pages) | ✅ |
| `stop_condition.max_value_path` | (If using `max_value`) Dot-notation path to total pages or count (e.g., `meta.totalPages`) | Required for `max_value` |

***

**Example: Page + Total Pages (stop when last page reached)**

```json theme={"system"}
pagination_config:
  type: page
  param: page
  initial_value: 1
  increment_by: 1
  stop_condition:
    type: max_value
    max_value_path: total_pages
```

**Example: Offset + Limit (stop when response is empty)**

```json theme={"system"}
pagination_config:
  type: page
  param: offset
  initial_value: 0
  increment_by: 100
  limit_param: limit
  limit_value: 100
  stop_condition:
    type: empty
```

## Other Example Configs

### Example GET API 1

```json theme={"system"}
url: https://api.plos.org/search
query:
  q: "title:DNA"
headers:
  Content-Type: application/json
response_parser: response.docs[0].author_display
columns:
  - author_display
```

The response from the above endpoint is:

```json theme={"system"}
{
  "response": {
    "numFound": 5669,
    "start": 0,
    "maxScore": 6.7217336,
    "docs": [
      {
        "id": "10.1371/journal.pone.0000290",
        "journal": "PLoS ONE",
        "eissn": "1932-6203",
        "publication_date": "2007-03-14T00:00:00Z",
        "article_type": "Research Article",
        "author_display": [
          "Rayna I. Kraeva",
          "Dragomir B. Krastev",
          "Assen Roguev",
          "Anna Ivanova",
          "Marina N. Nedelcheva-Veleva",
          "Stoyno S. Stoynov"
        ],
        "abstract": [
          "Nucleic acids, due to their structural and chemical properties, can form double-stranded secondary structures that assist the transfer of genetic information and can modulate gene expression. ",
          "However, the nucleotide sequence alone is insufficient in explaining phenomena like intron-exon recognition during RNA processing. This raises the question whether nucleic acids are endowed with other attributes that can contribute to their biological functions. ",
          "In this work, we present a calculation of thermodynamic stability of DNA/DNA and mRNA/DNA duplexes across the genomes of four species in the genus Saccharomyces by nearest-neighbor method. ",
          "The results show that coding regions are more thermodynamically stable than introns, 3′-untranslated regions and intergenic sequences. Furthermore, open reading frames have more stable sense mRNA/DNA duplexes than the potential antisense duplexes, a property that can aid gene discovery. ",
          "The lower stability of the DNA/DNA and mRNA/DNA duplexes of 3′-untranslated regions and the higher stability of genes correlates with increased mRNA level. These results suggest that the thermodynamic stability of DNA/DNA and mRNA/DNA duplexes affects mRNA transcription."
        ],
        "title_display": "Stability of mRNA/DNA and DNA/DNA Duplexes Affects mRNA Transcription",
        "score": 6.7217336
      }
    ]
  }
}
```

However, with the `response_parser` value of `response.docs[0].author_display`,
the data that is extracted from the API’s response is:

```json theme={"system"}
[
  "Rayna I. Kraeva",
  "Dragomir B. Krastev",
  "Assen Roguev",
  "Anna Ivanova",
  "Marina N. Nedelcheva-Veleva",
  "Stoyno S. Stoynov"
]
```

Since each item in the final data is not a JSON object, the `columns` configuration value is required.

The final data is converted into a JSON object before being outputted to its destination:

```json theme={"system"}
[
  {
    "author_display": "Rayna I. Kraeva"
  },
  {
    "author_display": "Dragomir B. Krastev"
  },
  {
    "author_display": "Assen Roguev"
  },
  {
    "author_display": "Anna Ivanova"
  },
  {
    "author_display": "Marina N. Nedelcheva-Veleva"
  },
  {
    "author_display": "Stoyno S. Stoynov"
  }
]
```

### Example GET API 2

```json theme={"system"}
url: https://api.coingecko.com/api/v3/coins/markets
query:
  vs_currency: usd
```

The response from the above endpoint is:

```json theme={"system"}
[
  {
    "id": "bitcoin",
    "symbol": "btc",
    "name": "Bitcoin",
    "image": "https://assets.coingecko.com/coins/images/1/large/bitcoin.png?1547033579",
    "current_price": 17154.23,
    "market_cap": 329331527834,
    "market_cap_rank": 1,
    "fully_diluted_valuation": 359638803581,
    "total_volume": 14339246353,
    "high_24h": 17227.56,
    "low_24h": 17107.75,
    "price_change_24h": 18.57,
    "price_change_percentage_24h": 0.10839,
    "market_cap_change_24h": -425260465.15130615,
    "market_cap_change_percentage_24h": -0.12896,
    "circulating_supply": 19230300.0,
    "total_supply": 21000000.0,
    "max_supply": 21000000.0,
    "ath": 69045,
    "ath_change_percentage": -75.16662,
    "ath_date": "2021-11-10T14:24:11.849Z",
    "atl": 67.81,
    "atl_change_percentage": 25185.95387,
    "atl_date": "2013-07-06T00:00:00.000Z",
    "roi": null,
    "last_updated": "2022-12-11T00:32:01.695Z"
  }
]
```

Because there is no `response_parser`, the final data matches the exact response from the API.

Since each item in the final data is a JSON object, the `columns` configuration value isn’t required.

### Example POST API

```yaml theme={"system"}
url: https://api.something.com/users
method: POST
payload:
  user:
    first_name: Urza
    power: 10
headers:
  "Content-Type": "application/json"
  token: abc123
response_parser: "user"
```

### Example XLSX or CSV

```yaml theme={"system"}
url: https://docs.google.com/spreadsheets/d/e/2PACX-1vTLcLUBAJAWf-8NQSjlbB3E4LR6DWk5QIZC-KtRb1j2CXXcgY6mE6vOJAW8PoJ1BAOgjXYpE4tY1LAD/pub?output=xlsx
method: GET
has_header: True
```

The response from the above endpoint is:

```
teste,first_name,second_name,email
1,Sadella,Tythacott,stythacott0@sina.com.cn
2,Melessa,Flaune,mflaune1@si.edu
3,Caroljean,Filipowicz,cfilipowicz2@guardian.co.uk
4,Doll,Wannan,dwannan3@people.com.cn
5,Nancy,Giraudy,ngiraudy4@pagesperso-orange.fr
6,Dominic,Bimson,dbimson5@vinaora.com
7,Kikelia,Bishopp,kbishopp6@cdc.gov
8,Andrus,Pomfrett,apomfrett7@wikipedia.org
9,Wildon,Fillingham,wfillingham8@google.co.uk
10,Alfonse,Leechman,aleechman9@jalbum.net
11,Phil,Emblem,pemblema@opera.com
12,Eyde,Brewer,ebrewerb@istockphoto.com
....
```

Data can also be checked by clicking on the "Load Sample data" button.


# Azure Blob Storage
Source: https://docs.mage.ai/data-integrations/sources/azure_blob_storage



![Azure Blob Storage](https://user-images.githubusercontent.com/80284865/212203409-7f9660ba-abf1-4a1c-9e86-0d699ed04381.png)

## Configuration

To set up the Azure Blob Storage source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `connection_string` | Storage account connection string. | `BlobEndpoint=https://xxx.blob.core.windows.net/;yyyyyy&sig=testsig` |
| `container_name` | The name of the Blob storage container where the data is stored. | `testcontainer` |
| `prefix` | Path within the container where files are located. Do not include the container name or table name in this path. | `users/ds/20221225` |

## How to Find Connection String

* **Option 1:** Get the connection string from your Storage Account's Access Keys:\
  [Microsoft documentation](https://learn.microsoft.com/en-us/answers/questions/1071173/where-can-i-find-storage-account-connection-string)

* **Option 2:** If you want to restrict access to the storage account, generate a **Shared Access Signature (SAS)** token and build the connection string:\
  [Microsoft documentation](https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/create-sas-tokens?view=form-recog-3.0.0#create-your-sas-tokens)


# BigQuery
Source: https://docs.mage.ai/data-integrations/sources/bigquery



## Configuration

To set up the BigQuery source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. If Mage is running on GCP, you can leave this `null` to use the instance’s service account. | `/path/to/service_account_credentials.json` |
| `dataset` | BigQuery dataset you want to read data from. | `example_dataset` |
| `credentials_info` | An alternative way to specify Google service account credentials directly as a dictionary. | Structure shown below |

`credentials_info` structure:

```yaml theme={"system"}
auth_provider_x509_cert_url: str
auth_uri: str
client_email: str
client_id: str
client_x509_cert_url: str
private_key: str
private_key_id: str
project_id: str
token_uri: str
type: str
```

### Optional Configs

| Key | Description | Sample value |
| - | - | - |
| `skip_schema_creation` | If `true`, Mage won't run CREATE SCHEMA command. For more information, see this [issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will set all columns name as lower case. Default is `true` | `true` |

### Required permissions

Your BigQuery account should at least have the below permissiont to use the BigQuery destination

```
bigquery.datasets.create
bigquery.jobs.create
bigquery.readsessions.create
bigquery.readsessions.getData
```

Your account should also have "BigQuery Data Editor" role for the BigQuery dataset you specified in the config.


# Chargebee
Source: https://docs.mage.ai/data-integrations/sources/chargebee



## Configuration

To set up the Chargebee source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `api_key` | API key for your Chargebee site. | `fake_api_key...` |
| `site` | Name of your Chargebee site. (For `mage.chargebee.com`, enter `mage`.) | `your-site-name` |
| `start_date` | The date in ISO (`YYYY-MM-DDTHH:MM:SSZ`) format to start pulling data (for resources that support this). | `2010-01-01T00:00:00Z` |
| `include_deleted` | (Optional) Whether to include deleted records in all streams. Defaults to `True`. | `True` |
| `request_timeout` | (Optional) Timeout for requests to the Chargebee API (in seconds). Defaults to `300`. | `300` |

## Streams

The following streams are supported in the Chargebee source:

👉 [List of Supported Streams](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/chargebee/streams)


# Commercetools
Source: https://docs.mage.ai/data-integrations/sources/commercetools



![Commercetools](https://commercetools.com/_build/images/logos/commercetools-logo-desktop.svg)

## Configuration

To set up the Commercetools source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `client_id` | `client_id` of the API client. Used to obtain the access token. | `abcdefg` |
| `client_secret` | `client_secret` of the API client. Used to obtain the access token. | `abcdefghijklmnopqrstuvwxyz` |
| `host` | The host in your Commercetools URL, e.g., `aws`, `gcp`. | `gcp` |
| `project_key` | The project key associated with your project. | `projectkey` |
| `region` | The region in your Commercetools URL, e.g., `us-central1`, `us-east-2`. | `us-central1` |

## How to Get Your `client_id` and `client_secret`

Follow [Commercetools's documentation](https://docs.commercetools.com/api/authorization):

* Create an API client.
* Retrieve `client_id`, `client_secret`, and `project_key` from the API client credentials page.


# Couchbase
Source: https://docs.mage.ai/data-integrations/sources/couchbase



![Couchbase](https://www.couchbase.com/wp-content/uploads/2022/08/CB-logo-R_B_B.png)

## Configuration

To set up the Couchbase source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `connection_string` | Connection string for your Couchbase database. [More info](https://docs.couchbase.com/kotlin-sdk/current/howtos/connecting.html#connection-string-scheme). | `couchbase://my_instance.cloud.couchbase.com` |
| `username` | Username to access the database (must have permissions to read from the specified bucket and scope). | `username` |
| `password` | Password for the user. | `password` |
| `bucket` | Name of the Couchbase bucket containing your data. | `my_bucket` |
| `scope` | Name of the Couchbase scope. Only collections within this scope will be available in Mage. | `my_scope` |
| `strategy` | (Optional) Strategy for schema inference. See below. | `infer` |

## Strategy Details

By default, Mage will attempt to infer your data’s schema from a sample of documents.

* `strategy: infer` — Mage will try to infer a unified schema from the data. If multiple schemas exist, Mage will pick the one covering the most data points.
* `strategy: combine` — Mage will combine the data into a single `_document` column as an object.


# Custom source
Source: https://docs.mage.ai/data-integrations/sources/custom_source

Easily plug in your own data integration sources and destinations with Mage Pro.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

Mage Pro makes it easy to integrate your own custom data integration sources and destinations.

To get started, follow the [adapt an existing source](/contributing/data-integrations/adapt-existing-source) guide or the [add a new source](/contributing/data-integrations/add-new-source) guide to create your custom integration.

Once your custom source or destination is ready, simply include it in your Mage Pro project by adding it to the appropriate folder as shown below.

## Folder structure

To register custom sources and destinations in your Mage Pro project, organize your project directory as follows:

```
.
├── mage_data
└── project_name
    ├── charts
    ├── custom
    ├── data_exporters
    ├── data_loaders
    ├── dbt
    ├── extensions
    ├── pipelines
    ├── data_integrations              # ← important
    │   └── sources
    │   │   ├── custom_source_1
    │   │   │   └── templates
    │   │   │   │   └── config.json
    │   │   │   └── __init__.py
    │   │   │   └── README.md
    │   │   ├── custom_source_2
    │   │   └── metadata.yaml          # Register all source definitions here
    │   └── destinations
    │       ├── custom_destination_1
    │       ├── custom_destination_2
    │       └── metadata.yaml          # Register all destination definitions here
    ├── scratchpads
    ├── transformers
    ├── utils
    ├── __init__.py
    ├── io_config.yaml
    ├── metadata.yaml
    └── requirements.txt
```

**Instructions**

1. Create a `data_integrations` folder inside your project directory.
2. Inside data\_integrations, create two subfolders:
   * `sources`: for your custom data source integrations
   * `destinations`: for your custom data destination integrations
3. Each source/destination should have its own folder (e.g., `custom_source_1`) containing:
   * An `__init__.py` file with the integration class
   * A `README.md` for documentation
   * A `templates/` directory with a `config.json` file for default configuration
4. Define a `metadata.yaml` file in each of the sources/ and destinations/ folders to register the integrations with Mage Pro.

This structure allows Mage Pro to discover and load your custom integrations dynamically.

## Register custom sources and destinations

To register your custom sources or destinations, create or update the `metadata.yaml` file inside the
respective `sources/` or `destinations/` subfolders under `data_integrations`.

📘 Example: `sources/metadata.yaml`

```yaml theme={"system"}
sources:
- name: Source Name
  module_name: SourceModuleName
  uuid: source_uuid
```

📘 Example: `destinations/metadata.yaml`

```yaml theme={"system"}
destinations:
- name: Source Name
  module_name: SourceModuleName
  uuid: source_uuid
```

Field descriptions

* `name`: Display name shown in the Mage UI dropdown.
* `module_name` (optional): The class name defined in the source's `__init__.py` file. Defaults to the `name` field with spaces removed.
* `uuid` (optional): Folder name of the source. Defaults to the lowercased `name`, with spaces replaced by underscores.

## Code modification

Unlike the [add a new source guide](/contributing/data-integrations/add-new-source) (which adds sources to the Mage core),
using custom sources in your project requires updating import paths to point to your project-level modules.

Update any imports from `mage_integrations` to `data_integrations` like so:

```python theme={"system"}
# Before
from mage_integrations.sources.custom_source.module import SomeClass

# After
from data_integrations.sources.custom_source.module import SomeClass
```

This change ensures that Mage loads and uses your custom integration from the project folder instead of the built-in Mage package.

## Testing Your Custom Source

To test your custom data source integration:

1. Create a new data integration pipeline.
2. **Select your custom source** from the source dropdown.
   * Custom sources appear at the **bottom** of the list in the UI.
3. **Enter your source configuration**.
4. Click "**Test connection**" if your custom source class implements the test\_connection method.
5. Click "**View and select stream**" to verify that the expected streams are returned.
6. **Add a destination** and run the pipeline using a trigger to test end-to-end data flow.

### Troubleshooting: Source Not Appearing in UI?

If your custom source isn't showing up in the Data Integration source dropdown, follow these steps to debug:

1. Create a pipeline and add a **scratchpad block**.
2. **Run the following code** in the block to inspect the custom sources:

```python theme={"system"}
# Inspect custom source registration
from mage_ai.server.api.integration_sources import get_custom_collection

custom_sources = get_custom_collection('sources', debug=True)
print(f'Custom sources detected: {custom_sources}')
```

3. **Try importing your custom module manually** to ensure it's discoverable and error-free:

```python theme={"system"}
from data_integrations.sources.source_uuid import SourceModuleName
```

Replace `source_uuid` and `SourceModuleName` with the actual folder name and class name of your custom source.

### Troubleshooting: "View and select stream" Not Working?

If clicking "**View and select stream**" in the UI doesn’t return any streams, you can manually debug the discovery step using a **scratchpad block** in your pipeline.

Run the following code to test your custom source's `discover_streams` method:

```python theme={"system"}
from data_integrations.sources.source_uuid import SourceModuleName

# Replace with your actual config keys and values
config = dict(
    key1='value1',
    key2='value2',
)

# Initialize the source class with config
source = SourceModuleName(config=config)

# Run stream discovery
streams = source.discover_streams()

# Print discovered streams
print(f'Discovered streams: {streams}')
```

**What to check**

* Ensure the `config` you provide contains all required fields expected by your source class.
* If `discover_streams()` returns an empty list or raises an error, check your source implementation or credentials.


# Datadog
Source: https://docs.mage.ai/data-integrations/sources/datadog



![Datadog](https://imgix.datadoghq.com/img/about/presskit/logo-v/dd_vertical_white.png?auto=format\&fit=max\&w=847)

## Configuration

To set up the Datadog source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `api_key` | Datadog API key. | `abc123...` |
| `application_key` | Datadog application key. | `def456...` |
| `start_date` | The date in ISO (`YYYY-MM-DDTHH:MM:SSZ`) format to start pulling data (for streams that support this). If a bookmark property is used, it will prioritize the latest bookmark date. | `2010-01-01T00:00:00Z` |
| `query` | Some streams support a `query` parameter to narrow down the data (currently `audit_logs` and `logs` streams). | `query string...` |

## Query Configuration

To set up a query for a specific stream, provide the stream name as a key in the `query` field, and the query string as the value.

Example:

```yaml theme={"system"}
query:
  audit_logs: your query here...
```


# Doris
Source: https://docs.mage.ai/data-integrations/sources/doris



## Configuration

To set up the Doris source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | The name of the database you want to read data from. | `demo` |
| `host` | Host name of your Doris database. | `mage.abc.us-west-2.rds.amazonaws.com` |
| `port` | Port where the database is running (typically 9030). | `9030` |
| `username` | Username with permissions to read and write to the specified schema. | `root` |
| `password` | Password for the user. | `abc123...` |
| `connection_method` | Connection method to Doris server: `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` |
| `conn_kwargs` | (Optional) Extra [connection keyword arguments](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) in dictionary format. | `{"ssl_ca": "CARoot.pem", "ssl_cert": "certificate.pem", "ssl_key": "key.pem"}` |
| `ssh_host` | (Optional) Host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (Optional) Port of the bastion server. Default is 22. | `22` |
| `ssh_username` | (Optional) Username used to connect to the bastion server. | `username` |
| `ssh_password` | (Optional) Password used to connect to the bastion server (if authenticating with password). | `password` |
| `ssh_pkey` | (Optional) Path to the private key used to connect to the bastion server (if authenticating with private key). | `/path/to/private/key` |

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch in each batch (defaults to 50,000). Adjust higher if your instance has more memory. | `50000` |


# Dremio
Source: https://docs.mage.ai/data-integrations/sources/dremio



![Dremio](https://www.dremio.com/wp-content/uploads/2023/02/CompanyDremio_centered.png)

## Configuration

This source uses the Dremio Arrow Flight Client to connect and execute SQL queries and shares most of its settings.\
For more details, refer to the [Dremio Python Client documentation](https://docs.dremio.com/current/sonar/client-applications/clients/python).

> **Note:** Mage's implementation of the Dremio source does **not** allow the `query` connection parameter.

### Required Parameters

| Key | Description | Sample Value |
| - | - | - |
| `username` | Username of the Dremio account used for authentication. | `ABC123` |
| `schema` | Dremio source table schema. | `ABC123` |
| `source_backend` | SQL source backend used to query data from Dremio (see section below). | `postgresql` |
| `password` | Password for the Dremio account. (Required if no token is provided.) | `ABC123` |
| `token` | Personal Access Token or OAuth2 Token. (Required if no password is provided.) | `ABC123` |

### Optional Parameters

| Key | Description | Sample Value |
| - | - | - |
| `hostname` | Hostname or IP address of the coordinator node (defaults to `localhost`). | `localhost` |
| `port` | Dremio's Arrow Flight server port (defaults to `32010`). | `32010` |
| `tls` | Enables TLS encryption for the connection. | `false` |
| `disable_certificate_verification` | Disables TLS server certificate verification. | `true` |
| `path_to_certs` | Path to trusted certificates for TLS connections. | `/path/to/certs` |
| `session_properties` | Key-value pairs for session properties. | `{ "key": "value" }` |

### Source Backend

To maintain consistent data extraction quality, this source allows specifying a preferred SQL source backend.\
Currently supported backend sources:

* `postgresql`
* `mysql`
* `mssql`

> **Note:**\
> OracleDB and Snowflake do **not** require a `source_backend`.\
> Internal tables created by S3 or other object storage also do **not** require a backend source.


# Amazon DynamoDB
Source: https://docs.mage.ai/data-integrations/sources/dynamodb

Use Mage to extract data from Amazon DynamoDB tables using AWS credentials or IAM roles.

## Overview

Amazon DynamoDB is a fully managed NoSQL database service that delivers single-digit millisecond performance at scale. With Mage, you can integrate DynamoDB as a source to extract and transform structured data from your tables into your pipelines.

## Configuration

To connect Mage to DynamoDB, add a source block with the following configuration in your data integration pipeline.

| Key | Description | Example Value |
| - | - | - |
| `aws_access_key_id` | Your AWS access key ID. *(Not required if using `role_arn` and Mage is running on AWS.)* | `AKIA...` |
| `aws_secret_access_key` | Your AWS secret access key. *(Not required if using `role_arn` and Mage is running on AWS.)* | `abc123xyz789...` |
| `aws_region` | AWS region where your DynamoDB tables are located. Defaults to `us-west-2`. | `us-west-2` |
| `role_arn` | (Optional) ARN of an IAM role Mage should assume when accessing DynamoDB. | `arn:aws:iam::123456789012:role/mage-role` |

If running Mage on EC2, EKS, or ECS, you can skip `aws_access_key_id` and `aws_secret_access_key` by granting access via IAM roles (via instance profiles or IRSA).

## Permissions

Ensure your AWS IAM user or role has the following permissions:

* `dynamodb:Scan`
* `dynamodb:Query`
* `dynamodb:DescribeTable`
* `dynamodb:GetItem` *(if using key-based lookups)*
* `dynamodb:ListTables` *(optional — for dynamic table discovery)*

These permissions are required for Mage to:

* Read data from DynamoDB tables
* Apply filters or projections
* Automatically detect schema or validate table access


# Facebook Ads
Source: https://docs.mage.ai/data-integrations/sources/facebook_ads



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=facebook-ads)

![Facebook](https://upload.wikimedia.org/wikipedia/commons/thumb/8/89/Facebook_Logo_%282019%29.svg/2560px-Facebook_Logo_%282019%29.svg.png)

## Configuration

To set up the Facebook Ads source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `account_id` | The Facebook Ad account ID used when extracting data. | `"abc123..."` (double quotes required) | ✅ |
| `access_token` | OAuth token to access API endpoints. | `abc123...` | ✅ |
| `start_date` | Fetch data newer than the specified `start_date`. Format: ISO8601. | `2022-01-01T00:00:00Z` | ✅ |

## Streams

Facebook Ads includes the following insight streams:

* `ads_insights`
* `ads_insights_age_and_gender`
* `ads_insights_country`
* `ads_insights_device_platform`
* `ads_insights_platform_and_device`
* `ads_insights_region`
* `ads_insights_dma`
* `ads_insights_hourly_advertiser`

### `account_id`

Visit [Facebook Ads Manager Accounts](https://www.facebook.com/adsmanager/manage/accounts) to find your account ID.\
It should look like: `13261011608127020`.

> **Note**\
> Your account ID is **not** the same as your App ID.

### `access_token`

1. [Create a Facebook App](https://developers.facebook.com/docs/development/create-an-app/).
2. Get the access token:
   1. Go to: `https://developers.facebook.com/apps/[app_id]/marketing-api/tools`
      * Replace `[app_id]` with the ID of the app you created.
   2. Under **Select Token Permissions**, check the following:
      * `ads_management`
      * `ads_read`
      * `read_insights`
   3. Click **Get Token**.
3. Copy the generated `access_token`.

## Errors

### API Rate Limit

If you encounter API rate limits, refer to this [Facebook Community Thread](https://developers.facebook.com/community/threads/158172292247085/) for more information.


# Facebook Pages
Source: https://docs.mage.ai/data-integrations/sources/facebook_pages



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

![Facebook Pages](https://upload.wikimedia.org/wikipedia/commons/thumb/8/89/Facebook_Logo_%282019%29.svg/2560px-Facebook_Logo_%282019%29.svg.png)

## Configuration

Connect to **Facebook Pages** and sync posts, insights, and engagement metrics. Provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `access_token` | OAuth access token for Facebook Graph API with the required permissions (`pages_read_engagement`, `pages_read_user_content`). | `EAABsbCS1i...` | ✅ |
| `page_ids` | List of Facebook Page IDs you want to extract data from. | `["1234567890", "9876543210"]` | ✅ |
| `start_date` | Start date (ISO 8601 format) to begin fetching posts and engagement data. | `2025-01-01T00:00:00Z` | ✅ |

## How to Get Your `access_token`

You have two options to generate an access token:

### 1. Use the OAuth Button (Recommended)

* Click the **OAuth Authorize** button in your Mage workspace.
* Approve the required Facebook permissions:
  * `pages_show_list`
  * `pages_read_engagement`
  * `pages_read_user_content`
  * `read_insights`
* The access token will be securely generated and auto-filled in the configuration.

> **Tip**: Use the OAuth flow for a smoother, safer authentication process without manually handling tokens.

### 2. Manually Generate Access Token

* Create a Facebook App via [Facebook Developer Portal](https://developers.facebook.com/).
* Configure the app and get the permissions listed above.
* Generate a **User Access Token** or **Page Access Token** via [Graph API Explorer](https://developers.facebook.com/tools/explorer/).
* (Optional) Exchange for a **long-lived token** for production environments.

## What is Facebook Pages API?

The [Facebook Pages API](https://developers.facebook.com/docs/pages) allows programmatic access to posts, insights, page metadata, and engagement data for Facebook Pages. Businesses use it to monitor social presence, measure engagement, and automate reporting workflows.

## Why Integrate Facebook Pages with Mage?

* **Automate data extraction**: Fetch page posts, reactions, and comments for analysis.
* **Engagement tracking**: Monitor likes, shares, comments, and post performance.
* **Insights at scale**: Integrate page metrics into BI dashboards for better reporting.
* **Marketing analytics**: Correlate Facebook engagement with other marketing activities.

## Supported Streams

The Facebook Pages source extracts the following resources:

* **page**: Basic page details like name, ID, and metadata.
* **page\_insights**: Aggregated metrics like page impressions, engagement, and follower growth.
* **post\_attachments**: Media and links attached to posts.
* **post\_insights**: Detailed insights for each post, including reach and engagement breakdown.
* **post\_tagged\_profile**: Profiles tagged in the posts.
* **posts**: Content posted on the page — text, images, videos, and links.


# Freshdesk
Source: https://docs.mage.ai/data-integrations/sources/freshdesk



![Freshdesk](https://cdn.worldvectorlogo.com/logos/freshdesk-logo.svg)

## Configuration

To set up the Freshdesk source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `api_key` | Freshdesk API key. | `abc123` |
| `domain` | Your Freshdesk domain. For `https://mage.freshdesk.com`, the domain is `mage`. | `mage` |
| `start_date` | Fetch data newer than the specified `start_date`. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2021-01-01T00:00:00Z` |

## How to Get Your `api_key`

Follow [Freshdesk’s documentation](https://support.freshdesk.com/en/support/solutions/articles/215517-how-to-find-your-api-key)\
to locate your API key.


# Front
Source: https://docs.mage.ai/data-integrations/sources/front



![Front](https://user-images.githubusercontent.com/80284865/215037433-f8b9e0f6-01e8-4fab-ac2e-f1a191c78103.png)

## Configuration

To set up the Front source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `token` | The API token of the Front app. | `abc123` |
| `start_date` | Used for the `analytics` stream. Fetch data newer than this `start_date`. Format: ISO8601. | `2023-01-01T00:00:00Z` |
| `end_date` | Used for the `analytics` stream. Fetch data older than this `end_date`. If `null`, current date is used. | `2023-01-05T00:00:00Z` |
| `incremental_range` | Used for the `analytics` stream. Defines the incremental extract range: `daily` or `hourly`. | `daily`, `hourly` |

## How to Get Your `token`

Follow [Front’s documentation](https://dev.frontapp.com/docs/create-and-revoke-api-tokens#create-an-api-token)\
to create and retrieve your API token.


# GitHub
Source: https://docs.mage.ai/data-integrations/sources/github



![GitHub](https://github.githubassets.com/images/modules/logos_page/GitHub-Logo.png)

## Configuration

To set up the GitHub source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `access_token` | A GitHub access token containing credentials to access your desired repository. | `abcdefghijklmnopqrstuvwxyz1234567890ABCD` | ✅ |
| `repository` | The repository path (everything after `github.com`). | `mage-ai/mage-ai` | ✅ |
| `start_date` | A sync start date. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2021-01-01T00:00:00Z` | ✅ |
| `request_timeout` | Request timeout value in seconds. | `300` | ✅ |
| `base_url` | The base URL before the repository path (defaults to `https://api.github.com`). | `https://api.github.com` | ✅ |

## Generating Credentials

Follow [this guide](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) to generate a Personal Access Token (PAT).

## Supported Endpoints

This source pulls raw data from the GitHub REST API and extracts the following resources for a single repository:

* Assignees
* Collaborators
* Commits
* Commit Comments
* Events
* Issues
* Issue Events
* Issue Milestones
* Projects
* Project Cards
* Project Columns
* Pull Requests
* PR Commits
* Releases
* Comments
* Reviews
* Review Comments
* Stargazers
* Teams
* Team Members
* Team Memberships

It outputs the schema for each resource.

> **Note:**\
> Incremental sync is currently **not supported** for any endpoint.\
> To add support, the `IncrementalStream` class in `./tap_github/streams.py` needs modifications to correctly handle bookmark values.


# Google Ads
Source: https://docs.mage.ai/data-integrations/sources/google_ads

Sync campaign, ad group, keyword, and performance report data from the Google Ads API v23 using OAuth 2.0 authentication.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

![Google Ads](https://hellomidia.com.br/wp-content/uploads/2018/06/google-ads.png)

## Overview

This source syncs campaign, ad group, ad, keyword, and performance report data from the [Google Ads API](https://developers.google.com/google-ads/api/docs/start) (**v23**). Authentication uses user OAuth (not service accounts) — the connector requires a `refresh_token` alongside a `developer_token`.

Incremental state is handled inside the connector; **stream-level INCREMENTAL selection is not available** in the UI, but bookmarks still resume progress across syncs.

## Authentication

This connector uses **two kinds** of credentials:

* **Application credential** (`developer_token`) — identifies your app to the Google Ads API. Issued per manager account via [API Center](https://ads.google.com/aw/apicenter). This is not an OAuth token.
* **User OAuth** (`refresh_token`, `oauth_client_id`, `oauth_client_secret`) — proves the user has consented to read their Ads data. The required OAuth scope is `https://www.googleapis.com/auth/adwords`.

### Manual setup vs Mage OAuth Authorize

| Method | What to set |
| - | - |
| **Mage OAuth Authorize** (recommended) | Use the **OAuth Authorize** button on the source block. Mage injects `refresh_token` automatically. You only need `developer_token` and `login_customer_ids`. You may omit `oauth_client_id` and `oauth_client_secret`. |
| **Manual** | Create an OAuth 2.0 client in Google Cloud (Web or Desktop app type), obtain a `refresh_token` for the Ads scope, and fill in all three OAuth fields. |

## Service accounts and Application Default Credentials are **not** supported.

![Google](https://hellomidia.com.br/wp-content/uploads/2018/06/google-ads.png)

## INCREMENTAL Sync

This source handles state internally, meaning **INCREMENTAL sync selection is not supported**.

## Configuration

| Key | Required | Default | Description |
| - | - | - | - |
| `developer_token` | ✅ | — | Google Ads API application token from [API Center](https://ads.google.com/aw/apicenter). Not a user OAuth token. |
| `oauth_client_id` | Conditional | — | Google Cloud OAuth client ID. Required for manual OAuth setup; optional when using Mage OAuth Authorize. |
| `oauth_client_secret` | Conditional | — | Google Cloud OAuth client secret. Same conditions as `oauth_client_id`. |
| `refresh_token` | ✅ | — | Long-lived user refresh token for the `https://www.googleapis.com/auth/adwords` scope. Injected automatically by Mage OAuth Authorize. |
| `login_customer_ids` | ✅ | — | List of `{ customerId, loginCustomerId }` pairs. `customerId` is the account to read; `loginCustomerId` is the manager (MCC) account used to access it, or `null` for direct access. |
| `start_date` | ✅ | — | Fetch data newer than this date. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). |
| `end_date` | | now | Upper bound for report syncs. |
| `request_timeout` | | `900` | HTTP timeout in seconds for API calls. |
| `conversion_window` | | `30` | Conversion attribution window in days. Allowed: `1`–`30`, `60`, `90`. |
| `query_limit` | | `1000000` | Max rows per query for core streams. |

## Sample configs

**Using Mage OAuth Authorize** (recommended — no OAuth client credentials needed):

```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
refresh_token: YOUR_REFRESH_TOKEN
start_date: '2023-01-01T00:00:00Z'
login_customer_ids:
  - customerId: '1234567890'
    loginCustomerId: null
```

**Manual OAuth setup with manager account**:

```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
oauth_client_id: YOUR_CLIENT_ID.apps.googleusercontent.com
oauth_client_secret: YOUR_CLIENT_SECRET
refresh_token: YOUR_REFRESH_TOKEN
start_date: '2023-01-01T00:00:00Z'
login_customer_ids:
  - customerId: '1234567890'
    loginCustomerId: '0987654321'
```

## How to get `login_customer_ids`

`customerId` is the account whose data you want to sync. `loginCustomerId` is the manager (MCC) account you use to access it — leave it `null` when there is no manager hop.

1. Sign in to [Google Ads](https://ads.google.com/).
2. Open the account switcher in the top bar.
3. Copy the numeric ID for each account (strip dashes: `123-456-7890` → `'1234567890'`).

## How to get a `developer_token`

A developer token belongs to a [manager (MCC) account](https://support.google.com/google-ads/answer/6139186). Client-only accounts cannot open API Center.

1. Sign in to a manager account.
2. Open [API Center](https://ads.google.com/aw/apicenter) (**Tools & Settings → Setup → API Center**).
3. Complete the API sign-up. Your token may start as **Test** tier; request **Basic** or **Standard** access when you are ready for production volume.
4. Set the token as `developer_token` in your config.

## Related links

* [Google Ads API — Get Started](https://developers.google.com/google-ads/api/docs/start)
* [OAuth with Google Ads API](https://developers.google.com/google-ads/api/docs/oauth/overview)
* [Developer token access levels](https://developers.google.com/google-ads/api/docs/access-levels)
* [Manager accounts overview](https://support.google.com/google-ads/answer/6139186)
  To set up the Google Ads source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `login_customer_ids` | A list containing both `customerId` and `loginCustomerId` from Google Ads. | `"login_customer_ids": [{"customerId": "1234567890", "loginCustomerId": "0987654321"}]` | ✅ |
| `oauth_client_id` | OAuth client ID from Google Cloud. | `XXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYY.apps.googleusercontent.com` | ✅ |
| `oauth_client_secret` | OAuth client secret from Google Cloud. | `abc...` | ✅ |
| `refresh_token` | Google API refresh token, obtainable from [Google OAuth Playground](https://developers.google.com/oauthplayground). | `abc...` | ✅ |
| `developer_token` | Google Ads developer token. | `abc...` | ✅ |
| `start_date` | Fetch data newer than the specified `start_date`. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2022-01-01T00:00:00Z` | ✅ |

## Generating Credentials

Check out [this article](https://articles.wesionary.team/how-to-implement-google-ads-api-ff69f628d4ac)\
by **Sudeep Timalsina** for a great guide on generating credentials for Google Ads API access.


# Google Analytics
Source: https://docs.mage.ai/data-integrations/sources/google_analytics



## Configuration

To set up the Google Analytics source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. | `/path/to/service_account_credentials.json` |
| `property_id` | Google Analytics GA4 Property ID. [Find your Property ID here](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id). | `12345678` |
| `start_date` | The start date for the Google Analytics query. Format: `YYYY-MM-DD`. | `2022-01-01` |
| `end_date` | The end date for the Google Analytics query. Format: `YYYY-MM-DD`. | `2022-01-01` |

### Prerequisites

1. Enable the **Google Analytics API** in your Google Cloud Console:\
   [Google Cloud API Dashboard](https://console.cloud.google.com/apis/dashboard)
2. The service account must have access to the Google Analytics property:
   * Go to the [Google Analytics Admin Dashboard](https://analytics.google.com/analytics)
   * Click **Admin** in the bottom left corner.
   * Click **Account Access Management**.
   * Click **Property Access Management**.
   * Add the service account email address (from your JSON credentials) with the appropriate permissions.

## Schema

Google Analytics data includes two types of columns:

* [Dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema#custom_dimensions)
* [Metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema#custom_dimensions)

### Limitations

* **Maximum number of dimensions:** 9\
  ([See more on Dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/rest/v1beta/Dimension))
* **Maximum number of metrics:** 10\
  ([See more on Metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/rest/v1beta/Metric))


# Google Campaign Manager 360
Source: https://docs.mage.ai/data-integrations/sources/google_campaign_manager_360

Sync entity data and Report Builder exports from Campaign Manager 360 using the DFA Reporting & Trafficking API v5 and a service account.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

## Overview

This source syncs entity data and report exports from [Campaign Manager 360](https://campaignmanager.google.com/) using the [Campaign Manager 360 API (dfareporting v5)](https://developers.google.com/doubleclick-advertisers). API v4 is sunset; this connector targets **v5 only**.

There are two categories of streams:

* **Entity streams** (`user_profiles`, `advertisers`, `campaigns`, `reports`) — list resources via the DFA Reporting & Trafficking API. Each row includes a stable `entity_id`, flattened top-level fields, and `_raw` (full API object as JSON).
* **Report export stream** (`report_rows`) — runs saved **Report Builder** reports by ID, waits for completion, downloads the CSV file via the API, and emits one record per data row. Columns vary by report definition and are packed into a `row_data` JSON string.

Authentication uses a **Google Cloud service account** JSON key. Domain-wide delegation is supported via the optional `email` field.

All streams use **full table** replication.

## Configuration

| Key | Required | Default | Description |
| - | - | - | - |
| `path_to_credentials_json_file` | ✅ | — | Path to a Google Cloud **service account** JSON key with access to Campaign Manager 360. |
| `email` | | — | Google user to impersonate when using **domain-wide delegation**. Leave blank for direct service-account access. |
| `profile_id` | ✅ | — | Campaign Manager **user profile ID**. Required for `advertisers`, `campaigns`, `reports`, and `report_rows`. Run the `user_profiles` stream (or use **Load sample data**) to discover available profile IDs. |
| `advertiser_ids` | | auto-discover | Comma-separated DCM advertiser IDs. If omitted, the `campaigns` stream auto-discovers advertisers by calling `advertisers.list` for `profile_id`. |
| `report_ids` | ✅ (`report_rows` only) | — | Comma-separated IDs of saved **Report Builder** reports. Required for `report_rows`; ignored by all other streams. Each report must be configured to output **CSV** (not Excel). |
| `report_use_latest_file` | | `false` | When `true`, reads the most recently completed (`REPORT_AVAILABLE`) file for each report instead of triggering a new execution. Use this for idempotent syncs or when you lack permission to trigger new runs. Raises an error if no completed file exists for a report ID. |
| `report_run_synchronous` | | `false` | Passes `synchronous=true` to [`reports.run`](https://developers.google.com/doubleclick-advertisers/rest/v5/reports/run). May complete faster for small reports when supported. Has no effect when `report_use_latest_file` is `true`. |
| `report_poll_interval_seconds` | | `10` | Seconds between [`files.get`](https://developers.google.com/doubleclick-advertisers/rest/v5/files/get) status polls while the report is `QUEUED` or `PROCESSING`. Has no effect when `report_use_latest_file` is `true`. |
| `report_poll_timeout_seconds` | | `3600` | Maximum seconds to wait for `REPORT_AVAILABLE` before failing the sync. Has no effect when `report_use_latest_file` is `true`. |

## Sample configs

**Entity streams only** — user profiles, advertisers, campaigns, reports (no report download):

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
profile_id: "1234567"
```

**Report rows — trigger a new run and wait for completion**:

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
profile_id: "1234567"
report_ids: "987654321,876543210"
report_poll_interval_seconds: 30
report_poll_timeout_seconds: 1800
```

**Report rows — read the most recently completed file** (no new execution triggered):

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
profile_id: "1234567"
report_ids: "987654321"
report_use_latest_file: true
```

**Campaigns scoped to specific advertisers with domain-wide delegation**:

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
email: analyst@company.com
profile_id: "1234567"
advertiser_ids: "111111111,222222222"
```

## Streams

| Stream | Description |
| - | - |
| `user_profiles` | DCM user profiles accessible to the credential. A useful starting point — run this stream to discover your `profile_id` value. |
| `advertisers` | Advertisers visible to `profile_id` via [`advertisers.list`](https://developers.google.com/doubleclick-advertisers/rest/v5/advertisers/list). |
| `campaigns` | Campaigns per advertiser. Uses `advertiser_ids` if set; otherwise auto-discovers advertisers first. |
| `reports` | Saved Report Builder report definitions for `profile_id`. Returns report metadata only — use `report_rows` to execute and download data. |
| `report_rows` | Executes each configured `report_id`, waits for the file to reach `REPORT_AVAILABLE`, downloads the CSV, and emits one record per data row. Empty rows (e.g. summary/footer rows) are skipped. Each record contains `_report_id`, `_file_id`, `_row_number`, and `row_data` (a JSON string mapping CSV header names to cell values). |

**`report_rows` and sample data:** When loading sample data, only the first `report_id` is run and at most 25 CSV rows are returned, so the UI preview does not trigger every report or download large files.

## How `report_rows` works

1. For each `report_id`, the connector calls [`reports.run`](https://developers.google.com/doubleclick-advertisers/rest/v5/reports/run) to start a new execution (or, if `report_use_latest_file: true`, looks up the most recent completed file via `reports.files.list`).
2. It polls [`files.get`](https://developers.google.com/doubleclick-advertisers/rest/v5/files/get) every `report_poll_interval_seconds` until the status reaches `REPORT_AVAILABLE`, `FAILED`, or `CANCELLED`, or until `report_poll_timeout_seconds` is exceeded.
3. Once available, the file is downloaded in streaming chunks via the `files.get_media` endpoint and written to a local temp file.
4. The temp file is parsed as a UTF-8 CSV. Blank rows are skipped. Each data row is emitted as a record.

## Prerequisites

1. Enable **Campaign Manager 360 API** (`dfareporting.googleapis.com`) in [Google Cloud Console](https://console.cloud.google.com/apis/dashboard).
2. Create a service account and download its JSON key. Set `path_to_credentials_json_file` to the local path.
3. In Campaign Manager 360, grant the service account access to the account (via **Admin → User Profiles**). Use **domain-wide delegation** and set `email` if your organization requires impersonation.
4. For `report_rows`: create and save the report in Campaign Manager's [Report Builder](https://support.google.com/campaignmanager/answer/2823849) with **CSV** output format. Copy the numeric report ID from the URL or Report Builder UI and add it to `report_ids`.

### OAuth scopes used

| Scope | Used by |
| - | - |
| `https://www.googleapis.com/auth/dfareporting` | All streams (reporting API) |
| `https://www.googleapis.com/auth/dfatrafficking` | Entity streams (trafficking API — required for advertisers, campaigns) |

## Related links

* [Campaign Manager 360 API (v5) overview](https://developers.google.com/doubleclick-advertisers)
* [reports.run reference](https://developers.google.com/doubleclick-advertisers/rest/v5/reports/run)
* [File status lifecycle](https://developers.google.com/doubleclick-advertisers/rest/v5/File#Status)
* [Service account authentication](https://cloud.google.com/iam/docs/service-account-overview)


# Google Cloud Storage
Source: https://docs.mage.ai/data-integrations/sources/google_cloud_storage



## Configuration

To set up the Google Cloud Storage source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. If Mage is running on GCP, you can leave this `null` to use the instance’s service account. | `/path/to/service_account_credentials.json` |
| `bucket` | Name of the Google Cloud Storage bucket to read data from. | `my_bucket` |
| `file_type` | Type of the files in the bucket. Supported values: `parquet`, `csv`. | `parquet` |
| `credentials_info` | Alternative way to specify Google service account credentials directly as a dictionary. | Structure shown below |
| `prefix` | A prefix used to filter blobs in the bucket. Only blobs whose names start with this prefix will be included. | `sales_data_` |

**`credentials_info` structure:**

```yaml theme={"system"}
type: str
project_id: str
private_key_id: str
private_key: str
client_email: str
client_id: str
auth_uri: str
token_uri: str
auth_provider_x509_cert_url: str
client_x509_cert_url: str
universe_domain: str
```


# Google Display & Video 360
Source: https://docs.mage.ai/data-integrations/sources/google_display_video_360

Sync DV360 entity metadata and Bid Manager performance report data from Google's Display & Video 360 platform using a service account.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

## Overview

This source syncs entity metadata from [Display & Video 360](https://displayvideo.google.com/) using the [Display Video API v4](https://developers.google.com/display-video/api/reference/rest/v4) and performance report data using the [DoubleClick Bid Manager API v2](https://developers.google.com/bid-manager/reference/rest).

**Entity streams** cover the full DV360 resource hierarchy: partners → advertisers → campaigns, insertion orders, line items, creatives, channels, ad groups, and more, including nested resources such as assigned targeting options and YouTube asset associations.

**Bid Manager streams** (`bid_manager_queries`, `bid_manager_reports`, `bid_manager_report_rows`) list saved queries and their completed reports, and download CSV report data directly from the GCS path returned by the Bid Manager API.

Authentication uses a **Google Cloud service account** JSON key (not user OAuth). Domain-wide delegation is supported via the optional `email` field.

All streams use **full table** replication. Each entity row includes `entity_id`, `_raw` (full API object as JSON), and flattened top-level fields.

## Configuration

| Key | Required | Default | Description |
| - | - | - | - |
| `path_to_credentials_json_file` | ✅ | — | Path to a Google Cloud **service account** JSON key with access to Display & Video 360. |
| `email` | | — | Google user to impersonate when using **domain-wide delegation**. Leave blank for direct service-account access. |
| `partner_id` | ✅ | — | DV360 **partner ID**. Required for `advertisers` and any advertiser-scoped stream when `advertiser_ids` is not set. Not required for `partners` or `bid_manager_*` streams. Optional for **Test connection** (falls back to `partners.list` without it). |
| `advertiser_ids` | | auto-discover | Comma-separated DV360 advertiser IDs. If set, all advertiser-scoped streams use only these IDs and `partner_id` is not needed for those streams. If omitted, advertiser IDs are discovered by calling `advertisers.list` under `partner_id`. |
| `bid_manager_query_ids` | | all queries | Comma-separated Bid Manager query IDs. If omitted, all queries from `queries.list` are used — this can be slow when there are many queries. |

## Sample configs

**DV360 entity streams only** (no Bid Manager):

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
partner_id: "123456"
```

**Scoped to specific advertisers** (`partner_id` not required for advertiser-scoped streams):

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
advertiser_ids: "111111111,222222222"
```

**Bid Manager streams with explicit query IDs** (disable entity streams in catalog):

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
bid_manager_query_ids: "123456789,987654321"
```

**Full config — DV360 entities + Bid Manager + domain-wide delegation**:

```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
email: analyst@company.com
partner_id: "123456"
advertiser_ids: "111111111,222222222"
bid_manager_query_ids: "123456789"
```

## Streams

### DV360 entity streams

| Stream | API resource |
| - | - |
| `partners` | [`partners.list`](https://developers.google.com/display-video/api/reference/rest/v4/partners/list) — partners visible to the credential. No `partner_id` required. |
| `advertisers` | [`advertisers.list`](https://developers.google.com/display-video/api/reference/rest/v4/advertisers/list) — advertisers under `partner_id`. |
| `campaigns` | `advertisers.campaigns.list` |
| `insertion_orders` | `advertisers.insertionOrders.list` |
| `line_items` | `advertisers.lineItems.list` |
| `ad_groups` | `advertisers.adGroups.list` |
| `ad_group_ads` | `advertisers.adGroupAds.list` |
| `ad_assets` | `advertisers.adAssets.list` |
| `creatives` | `advertisers.creatives.list` |
| `channels` | `advertisers.channels.list` |
| `channel_sites` | `advertisers.channels.sites.list` — walks each channel per advertiser. |
| `invoices` | `advertisers.invoices.list` |
| `location_lists` | `advertisers.locationLists.list` |
| `location_list_assigned_locations` | `advertisers.locationLists.assignedLocations.list` |
| `negative_keyword_lists` | `advertisers.negativeKeywordLists.list` |
| `negative_keyword_list_negative_keywords` | `advertisers.negativeKeywordLists.negativeKeywords.list` |
| `advertiser_assigned_targeting_options` | `advertisers.targetingTypes.assignedTargetingOptions.list` — one request per targeting type enum value. |
| `ad_group_assigned_targeting_options` | `advertisers.adGroups.targetingTypes.assignedTargetingOptions.list` — one request per ad group per targeting type. |
| `ad_group_youtube_asset_associations` | `advertisers.adGroups.youtubeAssetTypes.youtubeAssetAssociations.list` |
| `line_item_assigned_targeting_options` | `advertisers.lineItems.targetingTypes.assignedTargetingOptions.list` |
| `line_item_youtube_asset_associations` | `advertisers.lineItems.youtubeAssetTypes.youtubeAssetAssociations.list` |

### Bid Manager streams

| Stream | Description |
| - | - |
| `bid_manager_queries` | Saved Bid Manager query definitions from [`queries.list`](https://developers.google.com/bid-manager/reference/rest/v2/queries/list). |
| `bid_manager_reports` | Report metadata per query from [`queries.reports.list`](https://developers.google.com/bid-manager/reference/rest/v2/queries.reports/list). |
| `bid_manager_report_rows` | For each completed (`DONE`) CSV report, streams rows from the GCS path in `metadata.googleCloudStoragePath`. Each record contains `entity_id`, `_query_id`, `_report_id`, `_row_number`, and `row_data` (JSON map of CSV column → value). Reports with status other than `DONE`, XLSX format, or no GCS path are skipped. |

**Sample data limits:** `bid_manager_queries` returns at most 25 rows; `bid_manager_reports` uses only the first query and one report; `bid_manager_report_rows` returns at most 25 CSV rows from a single report.

## Prerequisites

1. Enable the **Display & Video 360 API** (`displayvideo.googleapis.com`) in [Google Cloud Console](https://console.cloud.google.com/apis/dashboard).
2. For Bid Manager streams, also enable the **DoubleClick Bid Manager API** (`doubleclickbidmanager.googleapis.com`) and grant the service account read access to the GCS buckets where report files are delivered.
3. Create a service account and download its JSON key. Set `path_to_credentials_json_file` to the local path.
4. In the DV360 UI, grant the service account access at the partner or advertiser level as appropriate.
5. If your organization uses **domain-wide delegation**: configure it in Google Workspace Admin for the service account, include the DV360 and Bid Manager scopes, and set `email` to the user to impersonate.

### OAuth scopes used

| Scope | Used by |
| - | - |
| `https://www.googleapis.com/auth/display-video` | All DV360 entity streams |
| `https://www.googleapis.com/auth/doubleclickbidmanager` | Bid Manager streams |
| `https://www.googleapis.com/auth/devstorage.read_only` | Bid Manager report file download from GCS |

## Related links

* [Display & Video 360 API (v4) reference](https://developers.google.com/display-video/api/reference/rest/v4)
* [DoubleClick Bid Manager API (v2) reference](https://developers.google.com/bid-manager/reference/rest)
* [Service account authentication](https://cloud.google.com/iam/docs/service-account-overview)
* [Domain-wide delegation](https://developers.google.com/workspace/guides/create-credentials#optional_set_up_domain-wide_delegation_for_a_service_account)


# Google Search Console
Source: https://docs.mage.ai/data-integrations/sources/google_search_console



## Configuration

To set up the Google Search Console source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. | `/path/to/service_account_credentials.json` |
| `email` | If using domain-wide delegation, enter the user email to impersonate. | `test@xyz.com` |
| `site_urls` | Website URLs separated by commas (e.g., `https://www.example.com`, `sc-domain:example.com`). | `https://www.mage.ai, sc-domain:example.com` |
| `start_date` | Start date for the Google Search Console query. Format: `YYYY-MM-DD`. | `2022-01-01` |

## Prerequisites

* [Enable the Google Search Console API](https://console.cloud.google.com/apis/dashboard).
* Grant the service account access to your Search Console property:
  * Go to [Google Search Console](https://search.google.com/search-console).
  * Navigate to **Settings** → **Users and permissions**.
  * Add your service account email with the appropriate permissions.


# Google Sheets
Source: https://docs.mage.ai/data-integrations/sources/google_sheets



## Configuration

To set up the Google Sheets source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. | `/path/to/service_account_credentials.json` |
| `spreadsheet_id` | The unique identifier of a spreadsheet (found in the spreadsheet URL after `/spreadsheets/d/`). | `abcdefg123456` |
| `selected_sheet_names` | A list of sheet names to filter and extract. | `["sheet1", "sheet2"]` |

## Prerequisites

* [Enable the Google Sheets API](https://console.cloud.google.com/apis/library/sheets.googleapis.com).
* Share your spreadsheet with the service account:
  * Open the spreadsheet.
  * Click **Share**.
  * Add the service account’s email with **Editor** or **Viewer** permission.

## Sheet Requirements

Each sheet must meet the following requirements:

* Sheet is not empty.
* The first row must contain column names (no duplicate column names).
* The following rows must contain only valid cell values:
  * No [errorType](https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/other#ExtendedValue) or formulaType cells.


# HubSpot
Source: https://docs.mage.ai/data-integrations/sources/hubspot



![HubSpot](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3f/HubSpot_Logo.svg/2560px-HubSpot_Logo.svg.png)

## Configuration

To set up the HubSpot source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_token` | Secret access token used to make authenticated API requests. | `my_token` |
| `disable_collection` | If `false`, disables the collection of anonymous usage metrics. | `false` |
| `request_timeout` | Timeout (in seconds) for API requests. | `300` |
| `start_date` | Cutoff date for syncing historical data. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2023-01-01T00:00:00Z` |

## How to Get `access_token`

Follow HubSpot’s [documentation](https://developers.hubspot.com/docs/api/private-apps) to create a Private App and generate your `access_token`.

### Scopes

Select all the **Read** scopes (except `crm.objects.feedback_submissions`) under the **CRM** section.

| Scope | Read |
| - | - |
| `crm.lists` | ✅ |
| `crm.objects.companies` | ✅ |
| `crm.objects.contacts` | ✅ |
| `crm.objects.custom` | ✅ |
| `crm.objects.deals` | ✅ |
| `crm.objects.line_items` | ✅ |
| `crm.objects.marketing_events` | ✅ |
| `crm.objects.owners` | ✅ |
| `crm.objects.quotes` | ✅ |
| `crm.schemas.companies` | ✅ |
| `crm.schemas.contacts` | ✅ |
| `crm.schemas.custom` | ✅ |
| `crm.schemas.deals` | ✅ |
| `crm.schemas.line_items` | ✅ |
| `crm.schemas.quotes` | ✅ |


# Instagram
Source: https://docs.mage.ai/data-integrations/sources/instagram



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

![Instagram](https://upload.wikimedia.org/wikipedia/commons/a/a5/Instagram_icon.png)

## Configuration

Connect to **Instagram Graph API** to sync media, stories, and insights for business accounts. Provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `access_token` | OAuth access token with Instagram Graph API permissions. | `EAAGm0PX4ZCpsBA...` | ✅ |
| `ig_user_ids` | List of Instagram Business Account IDs to fetch data for. | `["1234567890", "9876543210"]` | ✅ |
| `media_insights_lookback_days` | Number of days to look back for media insights. | `60` | Optional |
| `start_date` | Start date (ISO 8601 format) to begin fetching media and insights data. | `2025-01-01T00:00:00Z` | ✅ |
| `metrics_log_level` | Log level for API metrics (e.g., `INFO`, `DEBUG`, `WARNING`). | `INFO` | Optional |

## How to Get Your `access_token`

You have two options to generate an access token:

### 1. Use the OAuth Button (Recommended)

* Click the **OAuth login** button in your Mage workspace.
* Approve the required permissions for Instagram:
  * `instagram_basic`
  * `instagram_manage_insights`
  * `pages_show_list`
* The access token will be securely generated and automatically populated in the configuration.

> **Tip**: Use the OAuth flow for a smoother and safer authentication process without manually handling tokens.

### 2. Manually Generate Access Token

* Create a Facebook App via [Facebook Developer Portal](https://developers.facebook.com/).
* Add Instagram Basic Display API and Instagram Graph API products to your app.
* Generate a **User Access Token** via [Graph API Explorer](https://developers.facebook.com/tools/explorer/).
* Make sure to select the correct Instagram-related permissions.
* Convert the short-lived token to a **long-lived token** for production use.

## What is Instagram Graph API?

The [Instagram Graph API](https://developers.facebook.com/docs/instagram-api) is designed for businesses and creators to access media, insights, and metrics on Instagram accounts. It enables data-driven marketing and content analysis.

## Why Integrate Instagram with Mage?

* **Automate content extraction**: Sync media posts and stories.
* **Measure engagement**: Track likes, comments, saves, impressions, and reach.
* **Audience insights**: Understand follower growth and demographics.
* **Marketing analytics**: Integrate with other platforms for a 360-degree marketing view.

## Supported Streams

The Instagram source extracts the following resources:

* **media**: All published media objects (images, videos, carousels).
* **media\_children**: Children of carousel media posts.
* **media\_insights**: Metrics like reach, impressions, engagement for each media post.
* **stories**: Instagram Stories posted by the account.
* **story\_insights**: Insights for each Story, including taps forward, exits, impressions.
* **user\_insights\_28day**: Aggregated user insights over a 28-day window.
* **user\_insights\_daily**: Daily user insights.
* **user\_insights\_followers**: Follower count and demographic data.
* **user\_insights\_online\_followers**: Times of day when followers are online.
* **user\_insights\_weekly**: Weekly user insights.
* **users**: Profile data of the Instagram Business accounts.


# Intercom
Source: https://docs.mage.ai/data-integrations/sources/intercom



![Intercom](https://cdn.cdnlogo.com/logos/i/76/intercom.svg)

## Configuration

To set up the Intercom source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_token` | The access token used to authenticate your Intercom account. | `abcdefg123456` |
| `start_date` | Filters results to fetch only records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |

### How to Get `access_token`

Follow Intercom’s [instructions](https://developers.intercom.com/building-apps/docs/authentication-types#section-how-to-get-your-access-token)\
to obtain your `access_token`.


# Knowi
Source: https://docs.mage.ai/data-integrations/sources/knowi



![Knowi](https://images.g2crowd.com/uploads/product/image/social_landscape/social_landscape_7ef98d665734f3cc8618df9149c0f5fc/knowi.jpg)

## Configuration

To set up the Knowi source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_token` | (REQUIRED) The access token used to authenticate your Knowi account. | `abcdefg123456` |
| `request_timeout` | (OPTIONAL) The amount of time before the request times out (default: 300 seconds). | `300` |
| `user_agent` | (OPTIONAL) The User Agent string to send in the request header. | `my-app-v1.0` |
| `start_date` | (OPTIONAL) Filters results to fetch only records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |

### How to Get Your `access_token`

Follow Knowi’s [instructions](https://www.knowi.com/docs/managementAPI.html#APIAuthentication)\
to generate and retrieve your `access_token`.


# LinkedIn Ads
Source: https://docs.mage.ai/data-integrations/sources/linkedin_ads



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=linkedin-ads)

![LinkedIn Ads](https://i0.wp.com/winatlinkedin.com/wp-content/uploads/2019/07/Linkedin-ad-management-2.png)

## Configuration

To set up the LinkedIn Ads source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `accounts` | Comma-separated list of account IDs. Required if syncing `accounts` or `account_users` streams. | `"id1, id2, id3"` | Required (conditional) |
| `access_token` | Access token for authentication. | `def789...` | ✅ |
| `request_timeout` | Timeout (in seconds) for API requests. | `300` | |
| `start_date` | Fetch data newer than this `start_date`. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2023-01-01T00:00:00Z` | ✅ |
| `insights_lookback_window` | Days to rewind from the saved analytics bookmark for ad analytics streams. | `7` | |
| `user_agent` | User agent string (usually your email address). | `your_email@your_domain.com` | ✅ |

Instead of providing an `access_token`, you can also use the following OAuth credentials:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `client_id` | OAuth client ID. | `abc123...` | ✅ |
| `client_secret` | OAuth client secret. | `xyz456...` | ✅ |
| `refresh_token` | OAuth refresh token. | `def789...` | ✅ |

### How to Get `access_token`

1. Create a LinkedIn App: [LinkedIn Developers](https://www.linkedin.com/developers/apps).
2. Enable the **Marketing Developer Platform** product:\
   [Request Access](https://learn.microsoft.com/en-us/linkedin/shared/authentication/getting-access?context=linkedin%2Fcontext#marketing).
3. Submit the access form and wait for approval (a few business days).
4. Once approved, follow these [instructions](https://www.linkedin.com/developers/tools/oauth) to generate an `access_token`.


# Microsoft Ads
Source: https://docs.mage.ai/data-integrations/sources/microsoft_ads

Sync performance reports from the Microsoft Advertising API (Bing Ads v13) using OAuth 2.0 and the official bingads Python SDK.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

## Overview

This source syncs performance reports from the [Microsoft Advertising API](https://learn.microsoft.com/en-us/advertising/guides/get-started) (Bing Ads API **v13**) using the official [`bingads`](https://github.com/BingAds/BingAds-Python-SDK) Python SDK.

Reports are requested asynchronously: the connector submits a report request, polls until the result is ready, downloads the data, and emits one record per row. The three performance-report streams use **daily aggregation** by default and sync **incrementally** on `TimePeriod` per ad account, so only new or updated days are fetched on subsequent runs. The `ad_labels` stream is a full-table stream built from current Campaign Management ad label associations.

Authentication uses **OAuth 2.0** with a `refresh_token` (no service accounts). You provide an Azure AD application's `client_id` (and optionally `client_secret`) alongside a `developer_token` that identifies your API application.

## Configuration

| Key | Required | Default | Description |
| - | - | - | - |
| `developer_token` | ✅ | — | Microsoft Advertising developer token. Identifies your application to the API. Obtained from [Microsoft Advertising API Center](https://developers.ads.microsoft.com/Account). |
| `client_id` | ✅ | — | Azure AD **Application (client) ID** for your registered app. |
| `client_secret` | | — | Azure AD client secret. Required for **confidential/web app** OAuth flows. Leave empty for **public/desktop app** flows. |
| `refresh_token` | ✅ | — | OAuth refresh token scoped to `https://ads.microsoft.com/msads.manage`. See [Authentication with OAuth](https://learn.microsoft.com/en-us/advertising/guides/authentication-oauth). |
| `oauth_redirection_uri` | | `https://login.microsoftonline.com/common/oauth2/nativeclient` | Redirect URI registered in Azure AD. Must match the URI used when obtaining the refresh token. Required when `client_secret` is set. |
| `customer_id` | ✅ | — | Microsoft Advertising **manager account (customer) ID** the user operates under. |
| `account_ids` | ✅ | — | Comma-separated **advertiser account IDs** to sync. Find these in the Microsoft Advertising UI under account settings. |
| `start_date` | ✅ | — | Reporting lower bound. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). |
| `end_date` | | today (UTC) | Optional reporting upper bound. |
| `insights_lookback_window` | | `0` | Days to rewind from the saved report bookmark when syncing performance reports. Never goes earlier than `start_date`. |
| `ad_labels_max_ads` | | `10000` | Maximum current ads to scan per account when syncing `ad_labels`; increase only when large-account sync time and API quota are acceptable. |
| `ad_labels_max_hierarchy_requests` | | `1000` | Maximum campaign/ad group/ad hierarchy API requests per account when syncing `ad_labels`. |
| `environment` | | `production` | API environment: `production` or `sandbox`. |
| `aggregation` | | `Daily` | Report aggregation period. Common values: `Daily`, `Weekly`, `Monthly`. |
| `report_time_zone` | | — | [ReportTimeZone](https://learn.microsoft.com/en-us/advertising/reporting-service/reporttimezonetype) value. Leave empty to use the account's default time zone. |
| `poll_interval_ms` | | `5000` | Milliseconds between status polls while the async report is generating. |
| `report_timeout_ms` | | `3600000` | Maximum milliseconds to wait for report download before failing (default 1 hour). |

> **Legacy key:** `accounts` is accepted as an alias for `account_ids` for backwards compatibility.

## Sample configs

**Minimal — production environment, daily reports**:

```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
refresh_token: YOUR_REFRESH_TOKEN
customer_id: "12345678"
account_ids: "111111111,222222222"
start_date: "2025-01-01T00:00:00Z"
```

**Confidential app flow (with client secret)**:

```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
client_secret: YOUR_CLIENT_SECRET
refresh_token: YOUR_REFRESH_TOKEN
oauth_redirection_uri: https://login.microsoftonline.com/common/oauth2/nativeclient
customer_id: "12345678"
account_ids: "111111111"
start_date: "2025-01-01T00:00:00Z"
```

**Sandbox testing**:

```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
refresh_token: YOUR_SANDBOX_REFRESH_TOKEN
customer_id: "12345678"
account_ids: "111111111"
start_date: "2025-01-01T00:00:00Z"
environment: sandbox
```

## Streams

| Stream | Report type | Description |
| - | - | - |
| `campaign_performance_report` | [CampaignPerformanceReportRequest](https://learn.microsoft.com/en-us/advertising/reporting-service/campaignperformancereportrequest) | Daily campaign-level impressions, clicks, spend, and conversions. |
| `ad_group_performance_report` | [AdGroupPerformanceReportRequest](https://learn.microsoft.com/en-us/advertising/reporting-service/adgroupperformancereportrequest) | Daily ad group-level performance metrics. |
| `ad_performance_report` | [AdPerformanceReportRequest](https://learn.microsoft.com/en-us/advertising/reporting-service/adperformancereportrequest) | Daily ad-level performance metrics. |
| `ad_labels` | [Campaign Management label associations](https://learn.microsoft.com/en-us/advertising/guides/labels) | Full-table rows for current ad label associations. |

Performance report streams sync **incrementally** on `TimePeriod` (one bookmark per ad account). On the first run the full history from `start_date` is fetched; subsequent runs fetch only new or updated days. The `ad_labels` stream syncs as **full table** from the current label associations returned by the Campaign Management API.

## How to get credentials

### 1. Developer token

1. Sign in to [Microsoft Advertising](https://ads.microsoft.com/) as the account owner or a user with API access.
2. Go to **Tools → API Center** (or visit the [Microsoft Advertising API Center](https://developers.ads.microsoft.com/Account) directly).
3. Request and copy your developer token.

### 2. Azure AD app (`client_id` / `client_secret`)

1. Open [Azure Portal → App registrations](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade) and click **New registration**.
2. Choose **Public client / native** for desktop flows (no `client_secret` required) or **Web** for server flows.
3. Add `https://login.microsoftonline.com/common/oauth2/nativeclient` as a redirect URI (or your own URI for web flows).
4. Copy the **Application (client) ID** → `client_id`.
5. For web flows: go to **Certificates & secrets** and create a client secret → `client_secret`.

### 3. Refresh token

Use the [Microsoft Advertising OAuth guide](https://learn.microsoft.com/en-us/advertising/guides/authentication-oauth) to complete the authorization code flow with the scope `https://ads.microsoft.com/msads.manage offline_access`. Copy the resulting `refresh_token` into your config.

### 4. `customer_id` and `account_ids`

Find these in the [Microsoft Advertising UI](https://ui.ads.microsoft.com):

* **Customer ID** — shown in the top-right account selector next to your account name.
* **Account IDs** — visible in **Accounts Summary** or each account's settings page.

## Related links

* [Microsoft Advertising API — Get Started](https://learn.microsoft.com/en-us/advertising/guides/get-started)
* [Authentication with OAuth](https://learn.microsoft.com/en-us/advertising/guides/authentication-oauth)
* [Reporting Service overview](https://learn.microsoft.com/en-us/advertising/reporting-service/reporting-service-reference)
* [ReportTimeZone values](https://learn.microsoft.com/en-us/advertising/reporting-service/reporttimezonetype)


# Mode
Source: https://docs.mage.ai/data-integrations/sources/mode



![Mode](https://ta-relay-public-files-prod.s3.us-east-2.amazonaws.com/icp/product_images/163aa82596261bf2ce6b6d88e9b68a4f.png)

## Configuration

To set up the Mode source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_token` | Access token used to authenticate your Mode account. | `abcdefg123456` |
| `password` | Mode token password. | `supersecurepassword` |
| `workspace` | Workspace ID in Mode. | `ws-12345` |
| `user_agent` | User agent string to send in the request header. | `my-app-v1.0` |
| `start_date` | Filters results to only fetch records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |

### How to Get Your `access_token` and `password`

Follow Mode’s [instructions](https://mode.com/developer/api-reference/authentication/)\
to generate your `access_token` and `password`.


# Monday
Source: https://docs.mage.ai/data-integrations/sources/monday



![Monday](https://user-images.githubusercontent.com/80284865/216693464-961b160f-d987-41a6-85e6-a216f040e897.png)

## Configuration

To set up the Monday source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `api_token` | API token used for authentication. | `abcdefghijklmnopqrstuvwxyz` |
| `board_id` | ID of the board to query board-related data. | `12345678` |
| `board_limit` | Number of boards returned per page for the `boards` stream. | `25` |

## How to Get Your `api_token`

Follow [Monday’s documentation](https://developer.monday.com/api-reference/docs/authentication#accessing-api-tokens)\
to retrieve your API token.


# MongoDB
Source: https://docs.mage.ai/data-integrations/sources/mongodb



![MongoDB](https://upload.wikimedia.org/wikipedia/commons/thumb/9/93/MongoDB_Logo.svg/2560px-MongoDB_Logo.svg.png)

## Configuration

To set up the MongoDB source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | Name of the database you want to read data from. | `company` |
| `host` | Hostname of your MongoDB server. | `127.0.0.1` |
| `password` | Password for the user to access the database. | `abc123...` |
| `port` | Port on which MongoDB is running (typically `27017`). | `27017` |
| `user` | Username that will access the database. | `admin` |
| `replica_set` | (Optional) Name of the replica set, if any. | `optional` |
| `ssl` | (Optional) Use SSL for the connection. Default is `false`. | `false` |
| `authSource` | (Optional) Authentication source database name. | `the_database` |
| `authMechanism` | (Optional) Authentication mechanism. Default is none. Example: `SCRAM-SHA-256`. | `SCRAM-SHA-256` |

## Run MongoDB Locally

You can run a local instance of MongoDB using Docker:

```bash theme={"system"}
docker run -p 27017:27017 \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=password \
  --name mongo \
  mongo:latest
```


# MSSQL (Microsoft SQL Server)
Source: https://docs.mage.ai/data-integrations/sources/mssql



## Configuration

To set up the MSSQL source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | Name of the database you want to read data from. | `msdb` |
| `host` | Hostname of the MSSQL database, including port number at the end (e.g., `host,port`). | `172.20.0.2,1433` |
| `password` | Password for the MSSQL user to access the database. | `abc123...` |
| `port` | Port where the database is running (typically `1433`). | `1433` |
| `schema` | Schema of the database you want to read data from. | `public` |
| `username` | Username that will access the database (must have read and write permissions to the specified schema). | `guest` |

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `authentication` | Authentication mode for connecting to MSSQL with [Azure Active Directory authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver16). | `ActiveDirectoryServicePrincipal` |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Adjust if your instance has more memory. | `50000` |
| `driver` | ODBC [driver](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16) for SQL Server. | `ODBC Driver 18 for SQL Server` |
| `connection_method` | Connection method. Set to `ssh_tunnel` to connect via SSH tunnel. Default: `direct` | `ssh_tunnel` |
| `ssh_host` | (SSH tunnel only) The host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (SSH tunnel only) The port of the intermediate bastion server. Default: `22` | `22` |
| `ssh_username` | (SSH tunnel only) The username used to connect to the bastion server. | `username` |
| `ssh_password` | (SSH tunnel only, optional) The password used to connect to the bastion server. Required if not using `ssh_pkey`. | `password` |
| `ssh_pkey` | (SSH tunnel only, optional) The path to the private key used to connect to the bastion server. Required if not using `ssh_password`. | `/path/to/private/key` |

> [!NOTE]
>
> SSH tunneling is a Mage Pro only feature.


# MySQL
Source: https://docs.mage.ai/data-integrations/sources/mysql



![MySQL](https://user-images.githubusercontent.com/78053898/198753513-4a149790-853a-4dcd-8c93-388f84ef6aeb.png)

## Configuration

To set up the MySQL source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | Name of the database you want to read data from. | `demo` |
| `host` | Hostname of your MySQL server. | `mage.abc.us-west-2.rds.amazonaws.com` |
| `port` | Port on which the database is running (typically `3306`). | `3306` |
| `username` | Username with permissions to read and write to the specified schema. | `root` |
| `password` | Password for the user to access the database. | `abc123...` |
| `connection_method` | Method used to connect to MySQL server, either `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` |
| `conn_kwargs` | (Optional) Extra [connection keyword arguments](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) in dictionary format. | `{"ssl_ca": "CARoot.pem", "ssl_cert": "certificate.pem", "ssl_key": "key.pem"}` |
| `ssh_host` | (Optional) Host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (Optional) Port of the bastion server. Default is `22`. | `22` |
| `ssh_username` | (Optional) Username used to connect to the bastion server. | `username` |
| `ssh_password` | (Optional) Password for bastion authentication (if using password). | `password` |
| `ssh_pkey` | (Optional) Path to the private key file for bastion authentication (if using a private key). | `/path/to/private/key` |

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Adjust higher if your instance has more memory. | `50000` |


# OracleDB
Source: https://docs.mage.ai/data-integrations/sources/oracledb



## Configuration

To set up the OracleDB source, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `host` | OracleDB host. | `oracledb.example.com` |
| `port` | OracleDB port (defaults to `1521`). | `1521` |
| `service` | Listener service name on the database server for client connections. | `xepdb1` |
| `password` | Password for the user. | `xyz123` |
| `user` | Username with permission to connect and query the database. | `xyz123` |
| `mode` | Connection mode for the Oracle client. Accepted values are `thin` or `thick`. | `thick` |

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Adjust higher if your instance has more memory. | `50000` |


# Outreach
Source: https://docs.mage.ai/data-integrations/sources/outreach



![Outreach](https://lever-client-logos.s3.amazonaws.com/a8587b80-ef40-42f9-bc60-90ed6d34321d-1533317789481.png)

## Config

You must enter the following credentials when configuring this source:

| Key | Description | Sample value | Required |
| - | - | - | - |
| `client_id` | ... | `abc123...` | ✅ |
| `client_secret` | ... | `xyz456...` | ✅ |
| `page_size` | ... | ... | |
| `quota_limit` | ... | ... | |
| `redirect_uri` | ... | `https://www.mage.ai` | ✅ |
| `refresh_token` | ... | `def789...` | ✅ |
| `start_date` | A cutoff date for syncing historical data. | `2023-01-01T00:00:00Z` | ✅ |

### How to get `refresh_token`

Follow [Outreach’s instructions](https://api.outreach.io/api/v2/docs).

#### Get `code`

```
https://api.outreach.io/oauth/authorize?client_id=<Application_Identifier>&redirect_uri=<Application_Redirect_URI>&response_type=code&scope=<Scope1+Scope2+Scope3>
```

Sample URL:

```
https://accounts.outreach.io/oauth/authorize?client_id=...&redirect_uri=https%3A%2F%2Fwww.mage.ai&response_type=code&scope=accounts.read+calls.read+duties.read+events.read+mailboxes.read+mailings.read+opportunities.read+personas.read+prospects.read+sequences.read+stages.read+tasks.read+teams.read+users.read
```

#### Exchange `code` for `refresh_token`

```curl theme={"system"}
curl https://api.outreach.io/oauth/token \
  -X POST \
  -d client_id=<Application_Identifier> \
  -d client_secret=<Application_Secret> \
  -d redirect_uri=<Application_Redirect_URI> \
  -d grant_type=authorization_code \
  -d code=<Authorization_Code>
```

Sample response:

```json theme={"system"}
{
  "access_token": <Access_Token>,
  "token_type": "bearer",
  "expires_in": 7200,
  "refresh_token": <Refresh_Token>,
  "scope": <Scope1+Scope2+Scope3>,
  "created_at": 1503301100
}
```

## Streams

### Prospects

The column `contactHistogram` in the stream `prospects` have been removed because the
column type (array of arrays) isn’t supported in destinations at the moment.


# Sources technical documentation
Source: https://docs.mage.ai/data-integrations/sources/overview

A source is a system that you want to load data from and synchronize it into another system. A source can be a 3rd party API, SaaS, database, data warehouse, or a data lake.

![mage sources](https://mage-ai.github.io/assets/mage-sources.png)

## Singer spec

Mage uses the data engineering community standard for data integrations called the
[Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).

In addition,
Mage further standardizes the spec and provides common classes and methods to make implementing them easier and faster.

## Folder structure

Sources are defined in the `mage_integrations/sources/` directory.

All sources should be a subclass of the `Source` class defined in
`mage_integrations/sources/base.py`.

## Contribute

Want to learn about contributing a source? Read on [here](/contributing/data-integrations/add-new-source)!


# Paystack
Source: https://docs.mage.ai/data-integrations/sources/paystack



## Configuration

Connect to **Paystack**, a leading payments gateway in Africa, and automate your payment data extraction with the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `start_date` | Start date in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) to begin pulling transaction and payment data. If bookmarks are available, Mage prioritizes the latest bookmark. | `2010-01-01T00:00:00Z` |
| `secret_key` | Secret key from your Paystack dashboard used to authenticate API requests. | `fake_api_key...` |
| `request_timeout` | (Optional) Timeout (in seconds) for API requests to Paystack (default: 300 seconds). | `300` |

## What is Paystack?

[Paystack](https://paystack.com/) is a modern payments platform helping businesses in Africa collect online and offline payments seamlessly. With Paystack, businesses can accept payments via credit card, debit card, bank transfers, and mobile money.

## Why Integrate Paystack with Mage?

* **Automated financial data ingestion**: Pull transaction records, payment events, and customer data directly into your data warehouse.
* **Real-time analytics**: Enable up-to-date reporting and insights on payment performance, transaction trends, and customer behavior.
* **Secure and compliant**: OAuth and API key authentication ensure secure access to sensitive payment data.
* **Scalable extraction**: Handle large volumes of payment data efficiently for growing businesses.


# Pipedrive
Source: https://docs.mage.ai/data-integrations/sources/pipedrive



## Configuration

To connect to the Pipedrive CRM API and set up the Pipedrive source in Mage, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `start_date` | The start date in ISO format (`YYYY-MM-DDTHH:MM:SSZ`) to begin syncing data from Pipedrive (for supported resources). | `2010-01-01T00:00:00Z` |
| `api_key` | API key used to authenticate your Pipedrive account and access CRM data. | `fake_api_key...` |

## What is Pipedrive?

[Pipedrive](https://www.pipedrive.com/) is a leading customer relationship management (CRM) platform designed to help sales teams manage their pipelines efficiently. Integrating Pipedrive with Mage allows you to automate the extraction of CRM data like deals, leads, contacts, and activities for reporting and analytics.

## How to Get Your `api_key`

1. Log in to your [Pipedrive account](https://app.pipedrive.com/).
2. Click on your profile picture (top right) → **Company Settings** → **Personal Preferences** → **API**.
3. Copy your personal API token.

For more information, refer to [Pipedrive's API documentation](https://pipedrive.readme.io/docs/core-api-concepts-authentication).

## Why Integrate Pipedrive with Mage?

* **Automate data extraction**: Schedule and sync CRM data without manual intervention.
* **Custom analytics**: Build advanced reports using Pipedrive data.
* **Sales performance insights**: Analyze deals, stages, activities, and forecasts.
* **Unified data warehouse**: Combine Pipedrive data with other business data for holistic reporting.

## Supported Streams in Pipedrive

Mage’s Pipedrive integration extracts data from the following objects (streams):

* Activities
* Activity Types
* Currency
* Deal Products
* Deals
* Dealflow
* Filters
* Files
* Notes
* Organizations
* Persons
* Pipelines
* Products
* Stages
* Users


# PostgreSQL
Source: https://docs.mage.ai/data-integrations/sources/postgresql



![PostgreSQL](https://user-images.githubusercontent.com/78053898/198754309-2ef713a7-62c8-4ea8-9ebb-8c24ed038cb3.png)

## Configuration

To set up the PostgreSQL source, provide the following connection parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | Name of the database you want to extract data from. | `demo` |
| `host` | Hostname of your PostgreSQL database. | `db.bit.io` |
| `password` | Password for the PostgreSQL user. | `abc123...` |
| `port` | Port number where the database is accessible (typically `5432`). | `5432` |
| `schema` | Schema containing the tables you want to extract. | `public` |
| `username` | Username with permissions to read and write from the specified schema. | `guest` |
| `replication_slot` | Logical replication slot name for Change Data Capture (CDC). | `mage_slot` |
| `publication_name` | Logical replication publication name for CDC. | `mage_pub` |

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch per batch (default is 50,000). Adjust if your instance has sufficient memory. | `50000` |

## Why Use PostgreSQL as a Source in Mage?

* **Real-time change data capture (CDC)**: Capture and sync only changed rows.
* **Reliable large-scale data ingestion**: Load millions of rows efficiently.
* **Flexible schema support**: Work with multiple schemas in your PostgreSQL instance.
* **Secure and scalable**: Support for SSL connections, logical replication slots, and publication management.

## Change Data Capture (CDC)

Mage supports 2 types of change data capture with PostgreSQL:

1. Batch query
2. Log replication

![](https://user-images.githubusercontent.com/78053898/198754309-2ef713a7-62c8-4ea8-9ebb-8c24ed038cb3.png)

### Batch query

Mage will query PostgreSQL in batches using `SELECT`, `WHERE`, and `ORDER BY`
statements.

### Log replication

Mage supports **Change Data Capture (CDC)** for PostgreSQL using **logical replication**,
enabling real-time tracking of inserts, updates, and deletes without requiring full table reloads.
By reading PostgreSQL’s Write-Ahead Logs (WAL), Mage captures changes efficiently and applies them to the
destination system — inserting new records, updating existing ones, or deleting as needed —
ensuring low-latency, incremental data synchronization.

#### Setup in PostgreSQL

1. Enable logical replication in Postgres config
   1. Local Postgres
      1. Open the `postgresql.conf` file. Here is an example location on Mac OSX:
         `/Users/Mage/Library/Application Support/Postgres/var-14/postgresql.conf`.
      2. Under the settings section, change the value of `wal_level` to `logical`. The line in your
         `postgresql.conf` file should look like this:
         ```text theme={"system"}
         wal_level = logical
         ```
      3. Restart the PostgreSQL service or database. You can do this via the PostgreSQL app or if you’re
         on Linux, run the following commands:
         ```bash theme={"system"}
         sudo service postgresql stop
         sudo service postgresql start
         ```
      4. Run the following query in your PostgreSQL database: `SHOW wal_level`. The result should be:

         | `wal_level` |
         | - |
         | `logical` |
   2. AWS RDS/Aurora
      1. Follow this [doc](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.html#AuroraPostgreSQL.Replication.Logical.Configure)
         to set up PostgreSQL logical replication for an Aurora PostgreSQL DB cluster
      2. Logical replication for AWS RDS (PostgreSQL) can be set up in a similar way with the Aurora cluster.
   3. Azure Database for PostgreSQL
      1. Azure Database for PostgreSQL supports logical replication for **Flexible Server** and **Single Server** deployments.
      2. Follow this [Azure guide](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical-replication) to enable logical replication on Azure PostgreSQL Flexible Server.
      3. Make sure the following parameters are set in the Azure portal:
         * `wal_level` = `logical`
         * `max_replication_slots` ≥ 1
         * `max_wal_senders` ≥ 1
      4. Grant replication privileges to your user:

         ```sql theme={"system"}
         ALTER ROLE <username> WITH REPLICATION;
         ```
      5. Azure also requires that you configure firewall rules to allow your Mage server or IP range to access the PostgreSQL server.
      6. For more detailed instructions, see: [Set up logical replication on Azure PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical-replication).

2. Run the following command in PostgreSQL to create a replication slot:

   ```sql theme={"system"}
   SELECT pg_create_logical_replication_slot('mage_slot', 'pgoutput');
   ```

   <sub>The replication slot name `mage_slot` quoted is used as an example in the guide. The actual replication slot name should be unique for each pipeline source. </sub>

   The result should looking something like this:

   | `pg_create_logical_replication_slot` |
   | - |
   | `(mage_slot,0/51A80778)` |

3. Create a publication for all tables or for 1 specific table using the following commands:

   ```sql theme={"system"}
   CREATE PUBLICATION mage_pub FOR ALL TABLES;
   ```

   <sub>`mage_pub` is used in Mage’s [code](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/postgresql/__init__.py#L126)</sub>

   or for 1 table:

   ```sql theme={"system"}
   CREATE PUBLICATION mage_pub FOR TABLE some_schema.some_table_name;
   ```

   <sub>Replace `some_schema` with the schema of the table and `some_table_name` with the name
   of the table you want to replicate.</sub>

   Run the following command to add or remove a table from publication.

   ```sql theme={"system"}
   ALTER PUBLICATION mage_pub ADD/DROP TABLE table_name;
   ```

4. Verify that the publication was created successfully by running the following command in PostgreSQL:

   ```sql theme={"system"}
   SELECT * FROM pg_publication_tables;
   ```

   The result should looking something like this:

   | `pubname` | `schemaname` | `tablename` |
   | - | - | - |
   | `mage_pub` | `public` | `users` |

5. Grant the database user permission to read the replication slot:
   ```sql theme={"system"}
   ALTER ROLE <username> WITH REPLICATION;
   ```

#### Create data integration pipeline in Mage

Follow this [guide to create a data integration pipeline](/guides/data-integration-pipeline)
in Mage.

However, choose **PostgreSQL** as the source and choose **`LOG_BASED`** as the
replication method.

#### Testing pipeline end-to-end

Once you’ve created the pipeline, add a few rows into your PostgreSQL table that you just
created a logical replication for.

You can use the `INSERT` command to add rows. For example:

```sql theme={"system"}
INSERT INTO some_schema.some_table_name
VALUES (1, 2, 3)
```

<sub>
  Replace `some_schema` with the schema of the table and `some_table_name` with the name
  of the table you want to replicate.
</sub>

<sub>
  Change the `VALUES` to match the columns in your table.
</sub>

##### Verify replication logs being created

Run the following commands in PostgreSQL to check for new logs:

```sql theme={"system"}
SELECT
  *
FROM pg_logical_slot_peek_binary_changes('mage_slot', null, null, 'proto_version', '1', 'publication_names', 'mage_pub');
```

##### Run sync

After you added a few new rows, [create a trigger](/guides/triggering-pipelines#trigger)
to start running your pipeline and begin syncing data.

## What is PostgreSQL?

[PostgreSQL](https://www.postgresql.org/) is an open-source, enterprise-grade relational database system. Known for its robustness, extensibility, and SQL compliance, PostgreSQL powers thousands of applications worldwide in both small businesses and large enterprises.

Common use cases include:

* Transactional applications
* Analytical workloads
* Hybrid OLTP + OLAP systems
* Data warehousing
* Business intelligence (BI) and reporting pipelines


# Postmark
Source: https://docs.mage.ai/data-integrations/sources/postmark



![Postmark](https://assets.getshieldsecurity.com/onedollarplugin.com/uploads/2020/12/postmarkapp-logo-1024x576.jpg)

## Configuration

To connect to the Postmark API and extract email event data, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `postmark_server_token` | Your Postmark server's API token used to authenticate requests. | `abc123...` | ✅ |
| `start_date` | Cutoff date to start syncing historical data (format: `YYYY-MM-DD`). | `2023-01-01` | ✅ |

## What is Postmark?

[Postmark](https://postmarkapp.com/) is a reliable transactional email service for sending and tracking important application-generated emails, such as password resets, receipts, welcome emails, and more.\
Integrating Postmark with Mage allows you to automate the extraction of email delivery events, bounce information, and open/click tracking into your data warehouse for advanced analytics.

## How to Get Your `postmark_server_token`

1. Log in to your [Postmark account](https://postmarkapp.com/).
2. Navigate to **Servers** → **API Tokens**.
3. Copy the **Server API Token** you want to use.

For more information, see Postmark's [API token documentation](https://postmarkapp.com/support/article/1008-what-are-the-account-and-server-api-tokens).

## Why Integrate Postmark with Mage?

* **Automated email tracking**: Extract delivery, open, click, and bounce events.
* **Centralized analytics**: Consolidate Postmark email data with other sources in your data warehouse.
* **Improve deliverability**: Analyze bounce reasons and optimize your sending practices.
* **Real-time monitoring**: Stay updated on the performance of transactional emails.
* **Scalable architecture**: Sync millions of email events without manual intervention.

## Supported Streams from Postmark

When integrated with Mage, the following Postmark data streams are available:

* **messages\_opens** — Track when recipients open your emails.
* **messages\_outbound** — Retrieve details of outbound messages sent via Postmark.
* **stats\_outbound\_bounces** — Statistics on email bounces (hard, soft bounces).
* **stats\_outbound\_clients** — Email client analytics (e.g., Gmail, Outlook, Apple Mail).
* **stats\_outbound\_overview** — High-level summary of outbound email performance.
* **stats\_outbound\_platform** — Platform analytics showing devices used by recipients.


# Power BI
Source: https://docs.mage.ai/data-integrations/sources/powerbi



![Power BI](https://icon-library.com/images/power-bi-icon/power-bi-icon-21.jpg)

## Configuration

To set up the Power BI source and connect to your workspace, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_token` | (REQUIRED) Access token used to authenticate with your Power BI account. | `abcdefg123456` |
| `request_timeout` | (OPTIONAL) Timeout (in seconds) for API requests (default: 300). | `300` |
| `user_agent` | (OPTIONAL) User Agent string to send in the request header. | `my-app-v1.0` |
| `start_date` | (OPTIONAL) Filter results to only sync records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |

## What is Power BI?

[Power BI](https://powerbi.microsoft.com/) is Microsoft’s leading business intelligence (BI) platform, enabling organizations to visualize data, create interactive dashboards, and generate real-time insights.\
Integrating Power BI with Mage allows you to automate data extraction, sync analytics reports, and merge insights across your enterprise data ecosystem.

## How to Get Your `access_token`

Follow Power BI’s [official instructions](https://community.fabric.microsoft.com/t5/Developer/REST-API-Get-Access-Token/m-p/1895937/) to generate an OAuth `access_token` via the Azure Active Directory (Azure AD) app registration process.

## Why Integrate Power BI with Mage?

* **Automate report extraction**: Fetch dashboards, reports, and datasets seamlessly.
* **Centralize business analytics**: Combine Power BI data with data warehouses and lakes.
* **Real-time decision-making**: Access up-to-date analytics without manual refreshes.
* **Enhanced scalability**: Manage large-scale BI deployments with streamlined data syncs.
* **Boost productivity**: Eliminate manual export processes for business users and analysts.

## Supported Streams from Power BI

When integrated with Mage, Power BI enables access to the following data streams:

* **apps** — Metadata about published Power BI apps in your organization.
* **dashboards** — Details of dashboards created and shared within workspaces.
* **datasets** — Information about datasets, including refresh schedules and models.
* **groups** — Workspaces and group information for collaborative projects.
* **reports** — Metadata on reports built and published in Power BI.
* **tiles** — Tiles on dashboards representing visuals and pinned content.


# QuickBooks
Source: https://docs.mage.ai/data-integrations/sources/quickbooks



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

## Configuration

To set up the QuickBooks source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `client_id` | Client ID associated with your QuickBooks app. | `123456` | ✅ |
| `client_secret` | Client secret generated when you create your QuickBooks app. | `abcdefg` | ✅ |
| `start_date` | The start date for syncing data. Format: `YYYY-MM-DDTHH:MM:SSZ`. | `2021-01-01T00:00:00Z` | ✅ |
| `user_agent` | User agent name registered for your QuickBooks app. | `agent_name` | ✅ |
| `realm_id` | Realm ID associated with your QuickBooks company account. | `123456` | ✅ |
| `refresh_token` | Refresh token used to renew the access token after expiration (requires `offline_access` scope). | `abcdefg` | ✅ |

## How to Generate Credentials

Follow QuickBooks' [OAuth 2.0 Authentication Playground guide](https://developer.intuit.com/app/developer/qbo/docs/develop/authentication-and-authorization/oauth-2.0-playground) to generate the required credentials:

1. Register a new QuickBooks app to get your `client_id` and `client_secret`.
2. Configure the OAuth scopes required by your app.
3. Use the OAuth 2.0 playground to complete authentication and retrieve your `refresh_token` and `realm_id`.

## Additional Notes

* The `start_date` determines how far back data will be synced.
* Ensure that the `offline_access` scope is enabled to obtain a `refresh_token`.
* Tokens expire after a set period; use the `refresh_token` to obtain new access credentials automatically.


# Amazon Redshift
Source: https://docs.mage.ai/data-integrations/sources/redshift



![Amazon Redshift](https://user-images.githubusercontent.com/78053898/198753538-2d606c3a-f6b0-472a-b0b3-c16086f256fc.png)

## Configuration

To connect to your Amazon Redshift cluster and extract data, provide the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | Name of the database you want to load data from. | `demo` |
| `host` | Hostname of your Redshift cluster. | `mage-prod.3.us-west-2.redshift.amazonaws.com` |
| `password` | Password for the database user. | `abc123...` |
| `port` | Port where Redshift is running (typically `5439`). | `5439` |
| `region` | AWS region where your Redshift cluster is located. | `us-west-2` |
| `schema` | Schema containing the tables you want to extract. | `public` |
| `user` | Username with permissions to read and write to the specified schema. | `awsuser` |

### IAM Authentication (Optional)

Alternatively, use IAM database authentication by providing the following parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_key_id` | AWS access key for the IAM role or IAM user. | `abc123...` |
| `cluster_identifier` | Cluster identifier of the Amazon Redshift cluster. | `mage-prod` |
| `db_user` | Database user for Redshift IAM authentication. | `admin` |
| `secret_access_key` | AWS secret access key for the IAM role or IAM user. | `xyz123` |

### Optional Configuration

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Tune higher for larger clusters. | `50000` |

## What is Amazon Redshift?

[Amazon Redshift](https://aws.amazon.com/redshift/) is a fully managed, petabyte-scale cloud data warehouse solution designed for large-scale data storage and analytics. Redshift delivers fast query performance by using columnar storage, massively parallel processing (MPP), and advanced compression.

Integrating Amazon Redshift with Mage allows you to automate data extraction from Redshift clusters, streamline ETL pipelines, and centralize data for analytics and business intelligence workloads.

## Why Integrate Amazon Redshift with Mage?

* **High-performance data extraction**: Load massive datasets at scale.
* **Flexible authentication**: Connect using either traditional credentials or secure IAM authentication.
* **Automated ETL pipelines**: Move data from Redshift to your data lake or warehouse with zero manual effort.
* **Cost-efficient analytics**: Optimize Redshift query performance and storage.
* **Unified data workflows**: Combine Redshift data with other sources to power real-time dashboards and reports.


# Salesforce
Source: https://docs.mage.ai/data-integrations/sources/salesforce



![](https://user-images.githubusercontent.com/78053898/198753571-7c033641-75a2-4338-b257-ee37a159915e.png)

## Config

You must enter the following settings when configuring this source:

| Key | Description | Sample value |
| - | - | - |
| `api_type` | The `api_type` is used to switch the behavior of the tap between using Salesforce's "REST" and "BULK" APIs. When new fields are discovered in Salesforce objects, the `select_fields_by_default` key describes whether or not the tap will select those fields by default. | `REST`, `BULK` |
| `domain` | Your Salesforce instance domain. Use 'login' (default) or 'test' (sandbox), or Salesforce My domain. | `test`, `login` |
| `select_fields_by_default` | If `true`, the fields in a schema of a stream will all be selected by default when setting up a synchronization. | `true`, `false` |
| `start_date` | The `start_date` is used by the tap as a bound on SOQL queries when searching for records.  This should be this exact format `YYYY-mm-ddTHH:MM:SSZ`. | `2022-11-30T21:31:20Z` |

If using `OAuth` based authentication:

| Key | Description | Sample value |
| - | - | - |
| `client_id` | OAuth Salesforce App secrets. | `ABC1...` |
| `client_secret` | OAuth Salesforce App secrets. | `ABC1...` |
| `refresh_token` | The `refresh_token` is a secret created during the OAuth flow | `ABC1...` |

If using `Password` based authentication:

| Key | Description | Sample value |
| - | - | - |
| `username` | Salesforce account username | `ABC1...` |
| `password` | Salesforce account password | `ABC1...` |
| `security_token` | Salesforce account security token | `ABC1...` |

Optional settings:

| Key | Description | Sample value |
| - | - | - |
| `threshold` | When running `INCREMENTAL` sync runs, threshold is used to throttle how often STATE messages are generated (in `REST` api\_type).This can be useful to minimize the amount of `STATE` messages being generated. | Defaults to 1000 |
| `streams` | List of stream names to be discovered inside the salesforce tap. if none is given, the tap will search for all available streams, which can take several minutes. | \["Account"] |

### How to get your `client_id`, `client_secret`, and `refresh_token`

#### Find your `client_id` and `client_secret`

1. Sign into your Salesforce account.
2. Create a **Connected App** and **Enable OAuth Settings for API Integration**
   by following this [Salesforce documentation](https://help.salesforce.com/s/articleView?id=sf.connected_app_create_api_integration.htm\&type=5).
   1. Use the following URL as the callback URL: [https://login.salesforce.com/services/oauth2/success](https://login.salesforce.com/services/oauth2/success)
3. When you are at the step to choose which OAuth Scopes to authorize, you must give your
   Connected App at least these 2 scopes:
   1. Manage (`api`)
   2. Perform requests at any time (`refresh_token`)
4. After you created a **Connected App**, go to your **Setup** and click on the **Home** tab.
5. On the left sidebar under the **Platform tools > Apps > App Manager**, click on **App Manager**.
6. Find the row that contains the app you just created. On the furthest right column, click the dropdown caret icon and click **View**.
7. Under the **API (Enable OAuth Settings)** section, find the subsection labeled **Consumer Key and Secret** and click the button **Manage Consumer Details**.
8. A new page will open, displaying your **Consumer Key** (which is your `client_id`) and **Consumer Secret** (which is your `client_secret`).

#### Authorize your **Connected App**

1. Construct a URL with the following format:
   `https://[your_salesforce_domain].my.salesforce.com/services/oauth2/authorize?client_id=[client_id]&redirect_uri=https://login.salesforce.com/services/oauth2/success&response_type=code`

| Variable | Description | Sample value |
| - | - | - |
| `[your_salesforce_domain]` | The domain of your Salesforce account. | `mage-dev-ed` |
| `[client_id]` | Your **Consumer Key** | `ABC123...` |

2. Open that URL in a browser.
3. Authorize your **Connected App** to have API access to your Salesforce account.
4. Once you authorize, you’ll be redirect to a URL like this:
   `https://login.salesforce.com/services/oauth2/success?code=[code]`.
5. Note the value of the `code` URL parameter, it’ll be used to get a `refresh_token`.

#### Get a `refresh_token`

Open a terminal and make the following POST request:

```bash theme={"system"}
curl -X POST https://[your_salesforce_domain].my.salesforce.com/services/oauth2/token \
   -H "Content-Type: application/x-www-form-urlencoded"  \
   -d "grant_type=authorization_code&code=[code]&client_id=[client_id]&client_secret=[client_secret]&redirect_uri=https://login.salesforce.com/services/oauth2/success"
```

Change the following URL parameters to match your credentials:

| URL parameter | Description | Sample value |
| - | - | - |
| `[your_salesforce_domain]` | The domain of your Salesforce account. | `mage-dev-ed` |
| `[code]` | The code you received after authorizing your **Connected App** | `ABC123...` |
| `[client_id]` | Your **Consumer Key** | `ABC123...` |
| `[client_secret]` | Your **Consumer Secret** | `ABC1...` |

Once you execute the above CURL command successfully,
you’ll receive a response that could look like this:

```json theme={"system"}
{
  "access_token": "...",
  "refresh_token": "...",
  "signature": "...",
  "scope": "...",
  "id_token": "...",
  "instance_url": "https://[your_salesforce_domain].my.salesforce.com",
  "id": "https://login.salesforce.com/id/.../...",
  "token_type": "Bearer",
  "issued_at": ""
}
```

Take note of the `refresh_token`, it’ll be used when you configure this source.

### Additional resources in case you get lost

* [https://stackoverflow.com/questions/12794302/salesforce-authentication-failing/29112224#29112224](https://stackoverflow.com/questions/12794302/salesforce-authentication-failing/29112224#29112224)
* [https://developer.salesforce.com/docs/atlas.en-us.api\_rest.meta/api\_rest/intro\_oauth\_and\_connected\_apps.htm](https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_oauth_and_connected_apps.htm)
* [https://help.salesforce.com/s/articleView?id=sf.remoteaccess\_oauth\_web\_server\_flow.htm\&type=5](https://help.salesforce.com/s/articleView?id=sf.remoteaccess_oauth_web_server_flow.htm\&type=5)

## Schema

The available schema depends on the objects in your Salesforce account.

Some example schema can include the following (depending on your account):

* AIApplication
* AIApplicationConfig
* AIInsightAction
* AIInsightFeedback
* AIInsightReason
* AIInsightValue
* AIPredictionEvent
* AIRecordInsight
* AcceptedEventRelation
* Account
* ...
* WorkTypeFeed
* WorkTypeGroup
* WorkTypeGroupFeed
* WorkTypeGroupHistory
* WorkTypeGroupMember
* WorkTypeGroupMemberFeed
* WorkTypeGroupMemberHistory
* WorkTypeGroupShare
* WorkTypeHistory
* WorkTypeShare
* *and more...*


# Salesforce Marketing Cloud
Source: https://docs.mage.ai/data-integrations/sources/salesforce_marketing_cloud

Sync campaigns, journeys, assets, subscribers, and engagement events from Salesforce Marketing Cloud using the REST and SOAP APIs.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

![Salesforce Marketing Cloud](https://upload.wikimedia.org/wikipedia/commons/f/f9/Salesforce.com_logo.svg)

## Overview

This source syncs marketing data from [Salesforce Marketing Cloud](https://www.salesforce.com/products/marketing-cloud/overview/) (SFMC) using both the **Marketing Cloud Engagement REST API** and the **SOAP API**. Authentication uses the OAuth 2.0 `client_credentials` flow via an installed package.

## Authentication

Create an **Installed Package** in Marketing Cloud:

1. In Marketing Cloud, go to **Setup → Platform Tools → Apps → Installed Packages**.
2. Click **New** and give the package a name.
3. Under **Components**, click **Add Component → API Integration → Server-to-Server**.
4. Grant the scopes listed below and save.
5. Copy the **Client ID**, **Client Secret**, and the tenant subdomain from the **Authentication Base URI** (e.g. `mc563885gzs27c5t9-63k636ttgm` from `https://mc563885gzs27c5t9-63k636ttgm.auth.marketingcloudapis.com/`).

### Required scopes

| Area | Scope |
| - | - |
| Campaigns | `campaigns_read` |
| Content Builder (assets) | `documents_and_images_read`, `saved_content_read` |
| Journey Builder | `journeys_read` |
| Automation Studio | `automations_read` |
| Email Studio / SOAP | `list_and_subscribers_read`, `tracking_events_read`, `data_extensions_read` |

## Configuration

| Key | Required | Default | Description |
| - | - | - | - |
| `client_id` | ✅ | — | OAuth 2.0 client ID from the installed package. |
| `client_secret` | ✅ | — | OAuth 2.0 client secret from the installed package. |
| `subdomain` | ✅ | — | Tenant-specific subdomain. Accepts a bare subdomain (`mc563885gzs27c5t9-63k636ttgm`) or a full URL — the connector parses the subdomain automatically. |
| `account_id` | | — | MID of the child business unit to target. Omit to use the top-level account. |
| `start_date` | | — | ISO 8601 lower bound for incremental streams. **Required** for event streams (`sent_events`, `open_events`, `click_events`, `bounce_events`, `unsub_events`). Example: `2024-01-01T00:00:00Z`. |

## Sample configs

**Minimal — REST streams only:**

```yaml theme={"system"}
client_id: abcdef1234567890abcdef1234567890
client_secret: ABCDEF1234567890abcdef1234567890
subdomain: mc563885gzs27c5t9-63k636ttgm
```

**With child business unit and event streams:**

```yaml theme={"system"}
client_id: abcdef1234567890abcdef1234567890
client_secret: ABCDEF1234567890abcdef1234567890
subdomain: mc563885gzs27c5t9-63k636ttgm
account_id: "7281234"
start_date: "2024-01-01T00:00:00Z"
```

## Streams

### REST streams

| Stream | Replication | Key | Description |
| - | - | - | - |
| `campaigns` | Full table | `entity_id` | Marketing campaigns. |
| `assets` | Incremental (`modifiedDate`) | `entity_id` | Content Builder assets — emails, images, and templates. |
| `journeys` | Full table | `entity_id` | Journey Builder definitions including triggers, goals, and activities. |
| `journey_statistics` | Full table | `entity_id` | Aggregate send/open/click/bounce metrics per journey, plus per-activity breakdowns. |
| `automations` | Full table | `entity_id` | Automation Studio definitions including schedule and steps. |

### SOAP streams

| Stream | Replication | Key | Description |
| - | - | - | - |
| `subscriber_lists` | Full table | `entity_id` | Email Studio subscriber lists. |
| `data_extensions` | Full table | `entity_id` | Data Extension definitions — name, key, sendable flag, and retention settings. |
| `subscribers` | Full table | `entity_id` | All subscribers with email address and status. |
| `sends` | Full table | `entity_id` | Email send job summaries with delivered/open/click/bounce counts. |
| `sent_events` | Incremental (`EventDate`) | `entity_id` | Individual email send records per subscriber. Requires `start_date`. |
| `open_events` | Incremental (`EventDate`) | `entity_id` | Email open tracking records per subscriber. Requires `start_date`. |
| `click_events` | Incremental (`EventDate`) | `entity_id` | Link click records including URL. Requires `start_date`. |
| `bounce_events` | Incremental (`EventDate`) | `entity_id` | Bounce records with category, type, and SMTP details. Requires `start_date`. |
| `unsub_events` | Incremental (`EventDate`) | `entity_id` | Unsubscribe records per subscriber. Requires `start_date`. |

Event streams use **30-day date windowing** to avoid SFMC server-side timeouts on large datasets. The `start_date` config value (or the `EventDate` bookmark from a previous sync) is used as the window start.

## Related links

* [SFMC Installed Packages](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/packages-overview.html)
* [Marketing Cloud Engagement REST API](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/rest-api-overview.html)
* [Marketing Cloud SOAP API](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/web_service_guide.html)
* [OAuth 2.0 Server-to-Server Integration](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/integration-s2s-client-credentials.html)


# SFTP
Source: https://docs.mage.ai/data-integrations/sources/sftp



## Configuration

Connect securely to an SFTP server and load files into your Mage pipelines by providing the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `host` | Hostname or IP address of the SFTP server. | `sftp.example.com` |
| `port` | Port number for the SFTP server. | `22` |
| `username` | Username to authenticate with the SFTP server. | `myuser` |
| `password` | Password for authentication (optional if using private key). | `mypassword` |
| `private_key_file` | Path to the private key file for key-based authentication (optional). | `/home/user/.ssh/id_rsa` |
| `start_date` | Earliest file date to synchronize (`YYYY-MM-DD` format). | `2021-01-28` |
| `tables` | An array defining table configurations for loading files (required). | See table config below |
| `decryption_config` | Configuration for decrypting GPG-encrypted files (optional). | See decryption config below |

## Tables Configuration

Define how files are processed and structured into tables:

### Required Parameters

| Key | Description | Sample Value |
| - | - | - |
| `table_name` | Name assigned to the table (stream). | `customer_orders` |
| `search_prefix` | Directory path where the target files are located. | `/Export/SubFolder` |
| `search_pattern` | Regex pattern to match filenames for ingestion. | `MyExportData.*\\.zip.gpg$` |
| `delimiter` | Delimiter used to separate fields (default is `,`). | `,` |

### Optional Parameters

| Key | Description | Sample Value |
| - | - | - |
| `key_properties` | Array of unique keys for the table. Defaults to `['_sdc_source_file', '_sdc_source_lineno']`. | `["_sdc_source_file"]` |
| `encoding` | Character encoding of the file (default: `utf-8`). | `utf-8` |
| `sanitize_header` | Clean up header names for SQL compatibility (default: `False`). | `True` or `False` |
| `skip_rows` | Number of header or metadata rows to skip at the top of the file (default: `0`). | `0` |
| `auto_generate_field_names` *(Pro only)* | Automatically generate column names (`col_0`, `col_1`, etc.) if headers are missing (default: `false`). | `true` or `false` |
| `schema_discovery_sample_rows` *(Pro only)* | Number of rows to sample to infer schema (default: `100`). Not applicable for encrypted files. | `100` |

## Decryption Configuration (Optional)

If your files are encrypted with GPG, specify the following:

| Key | Description | Sample Value |
| - | - | - |
| `SSM_key_name` | AWS SSM parameter key name storing the GPG private key passphrase. | `SSM_PARAMETER_KEY_NAME` |
| `gnupghome` | Path to your `.gnupg` directory containing GPG keys. | `/home/.gnupg` |
| `passphrase` | Passphrase for decrypting GPG files. | `mysecretpassphrase` |

## Mage Pro Enhancements 🚀

With **Mage Pro**, the SFTP source is enhanced for improved performance, usability, and scalability.\
The following features are **available in Mage Pro only**:

* **Auto-Generate Field Names**:\
  Automatically generate column names (`col_0`, `col_1`, ..., `col_N`) when headers are missing by setting `auto_generate_field_names: true`.
* **Schema Discovery Sample Rows**:\
  Speed up schema inference by limiting how many rows are sampled with `schema_discovery_sample_rows`.
* **Improved Logging**:\
  Enhanced, structured logging for better traceability and debugging.
* **TableSpec Schema Support**:\
  Define detailed table specifications for schema generation and validation.
* **Selective Column Extraction**:\
  Write only the selected columns to the schema to reduce size and improve performance.
* **Better Code Quality and Documentation**:\
  Cleaner, more robust code and fully updated documentation for enterprise use.

## What is SFTP?

**SFTP (Secure File Transfer Protocol)** is a secure method for transferring files over an encrypted SSH connection.\
It provides encrypted file access, transfer, and management across a reliable, secure channel.

## Why Use SFTP with Mage?

* **Secure and encrypted**: Safely ingest sensitive files over SSH-secured channels.
* **Seamless ETL workflows**: Automate file ingestion into your data warehouse or lake.
* **Support for encrypted files**: Easily decrypt GPG-protected files during ingestion.
* **High performance**: Faster schema discovery, selective column loading, and large file handling.
* **Enterprise-ready**: Built for scale with advanced logging, security, and performance features.


# Snowflake
Source: https://docs.mage.ai/data-integrations/sources/snowflake



![Snowflake](https://user-images.githubusercontent.com/78053898/198754338-a8aeb12e-6e23-45e5-b130-7a1979a2b31d.png)

## Configuration

Connect to your Snowflake data warehouse and extract data by providing the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `account` | Your [Snowflake account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). | `abc1234.us-east-1` |
| `database` | The name of the database you want to query data from. | `DEMO_DB` |
| `schema` | Schema containing the tables you want to extract data from. | `PUBLIC` |
| `username` | Username for accessing the database (must have read/write permissions). | `guest` |
| `warehouse` | Name of the warehouse for executing queries. | `COMPUTE_WH` |

## Optional Connection Configs

| Key | Description | Sample Value |
| - | - | - |
| `password` | Password for the database user. | `abc123...` |
| `private_key_file` | Path to the Snowflake private key file (supported in version >= 0.9.76). | `/path/to/snowflake_key` |
| `private_key_file_pwd` | Passphrase for the private key file (supported in version >= 0.9.76). | `abc123...` |
| `role` | Role of the user to access the database. | `ROLE` |

> For setting up key-pair authentication, follow [this guide](https://docs.snowflake.com/en/user-guide/key-pair-auth).

## Other Optional Configs

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch per batch (default: 50,000). Increase for high-memory systems. | `50000` |

## What is Snowflake?

[Snowflake](https://www.snowflake.com/) is a leading cloud-based data warehouse platform that provides fast, scalable, and flexible data storage and analytics. It separates compute and storage for cost efficiency and performance, supporting structured and semi-structured data with SQL-based querying.

## Why Integrate Snowflake with Mage?

* **Seamless ETL pipelines**: Automate extraction of large datasets from Snowflake into your data lakes or warehouses.
* **Scalable performance**: Handle millions of rows with ease using high-speed, cloud-native architecture.
* **Flexible authentication**: Use either password-based or secure key-pair authentication.
* **Cost-effective**: Optimize compute and storage separately for your data workflows.
* **Enterprise-ready security**: Built-in role-based access control and encryption.


# Stripe
Source: https://docs.mage.ai/data-integrations/sources/stripe



![Stripe](https://user-images.githubusercontent.com/78053898/198754241-6581cb14-eddc-425b-b90f-e2a11ef22782.png)

## Configuration

Connect to Stripe’s powerful payment platform and automate your data extraction by providing the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `account_id` | Your Stripe account’s unique identifier. | `73bb...` |
| `client_secret` | Secret key used to authenticate API requests. | `ABC1...` |
| `start_date` | Date to start syncing data from (`YYYY-MM-DD`). | `2022-01-01` |

> **Find your API keys**:\
> Refer to Stripe’s [official documentation](https://stripe.com/docs/keys) to locate your account’s API keys.

## What is Stripe?

[Stripe](https://stripe.com/) is a leading payment processing platform that enables businesses of all sizes to accept online payments, manage subscriptions, send invoices, and handle financial operations with powerful APIs and a seamless developer experience.

## Why Integrate Stripe with Mage?

* **Automated data ingestion**: Sync financial and customer data from Stripe into your data warehouse or lake.
* **Comprehensive financial insights**: Access detailed records of transactions, payouts, subscriptions, and more.
* **Real-time analytics**: Stay updated with payment activities without manual exports.
* **Scalable extraction**: Handle high volumes of transactions efficiently with robust API integrations.
* **Secure and compliant**: Stripe’s infrastructure ensures PCI compliance and high security for payment data.

## Supported Schemas

The following Stripe data objects are extracted and loaded:

* Balance transactions
* Charges
* Coupons
* Customers
* Disputes
* Events
* Invoice items
* Invoice line\_items
* Invoices
* Payment intents
* Payout transactions
* Payouts
* Plans
* Products
* Subscription items
* Subscriptions
* Transfers


# Tableau
Source: https://docs.mage.ai/data-integrations/sources/tableau



![Tableau](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSWG60Loiu8KeR9TgUNjopJ59A9R0-pp0fJCitooXRz4tsn-5erVYtB-CjilCrRAZSkUn4\&usqp=CAU)

## Configuration

Connect to your Tableau Cloud environment and extract data using the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `access_token` | (REQUIRED) Access token used to authenticate with Tableau API. | `abcdefg123456` |
| `base_url` | (REQUIRED) Base API URL including your `<servername>` and `<siteid>`. | `https://<servername>/api/3.20/sites/<siteid>` |
| `request_timeout` | (OPTIONAL) Timeout for API requests (default: 300 seconds). | `300` |
| `user_agent` | (OPTIONAL) Custom User-Agent string for API requests. | `my-app-v1.0` |
| `start_date` | (OPTIONAL) Fetch records updated after this date for incremental sync. | `2023-01-01` |

> **Note**:\
> This connector has been tested with **Tableau Cloud**. It may not work with Tableau Server deployments.

### How to get your `access_token`

Follow Tableau’s [official documentation](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_auth.htm) to generate an API `access_token` for authentication.

## What is Tableau?

[Tableau](https://www.tableau.com/) is a leading business intelligence and data visualization platform. It helps organizations turn raw data into interactive dashboards and actionable insights without requiring deep technical expertise.

## Why Integrate Tableau with Mage?

* **Automate data extraction**: Streamline the process of pulling analytics data from Tableau Cloud.
* **Enable real-time updates**: Keep your datasets synced with the latest visualizations and reports.
* **Enhance data workflows**: Integrate Tableau data into broader ETL pipelines for advanced analytics.
* **Scalable and secure**: Handle large datasets securely via Tableau’s API with token-based authentication.


# Teradata
Source: https://docs.mage.ai/data-integrations/sources/teradata



## Configuration

Connect to your Teradata database and extract data by providing the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `database` | The name of the database you want to read data from. | `demo` |
| `host` | Hostname or IP address of your Teradata database. | `test.clearscape.teradata.com` |
| `port` | Port number for the database connection (default is 1025 for Teradata). | `1025` |
| `username` | Username with permissions to access the database and schema. | `demo_user` |
| `password` | Password for the user to authenticate. | `abc123...` |

### Optional Configs

| Key | Description | Sample Value |
| - | - | - |
| `batch_fetch_limit` | Number of rows to fetch per batch (default: 50,000). Adjust based on available memory. | `50000` |

## What is Teradata?

[Teradata](https://www.teradata.com/) is a powerful enterprise-grade relational database management system (RDBMS) optimized for high-performance data warehousing and large-scale analytics. It supports complex queries and advanced analytics across massive datasets.

## Why Integrate Teradata with Mage?

* **High-performance analytics**: Leverage Teradata’s parallel processing to extract large volumes of data efficiently.
* **Scalable ETL workflows**: Integrate Teradata data into your data pipelines for advanced transformation and analysis.
* **Enterprise-ready**: Built for high security, reliability, and compliance standards.
* **Batch optimization**: Customize batch fetch limits to tune performance based on workload and memory.


# TikTok Ads
Source: https://docs.mage.ai/data-integrations/sources/tiktok_ads

Sync ad management objects and performance insight reports from the TikTok Marketing API using a long-lived access token.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

## Overview

This source syncs ad management objects, audience objects, and performance insight reports from the [TikTok Marketing API](https://ads.tiktok.com/marketing_api/). It covers advertisers, campaigns, ad groups, ads, audiences, and a set of insight breakdowns (by age/gender, country, platform, and province).

Authentication is via a **long-lived access token** issued by the TikTok Marketing API portal. The token is scoped to one or more advertiser accounts, which you list in `accounts`.

Insight streams (`ad_insights`, `ad_insights_by_*`) sync **incrementally** from `start_date`. Entity streams (`advertisers`, `campaigns`, `adgroups`, `ads`, `audiences`) sync **full table** by default.

## Configuration

| Key | Required | Default | Description |
| - | - | - | - |
| `access_token` | ✅ | — | TikTok Marketing API access token. Obtained from the TikTok for Business API portal. |
| `accounts` | ✅ | — | Comma-separated advertiser account IDs, or a YAML list. All streams sync data for these accounts. |
| `start_date` | ✅ | — | Start of incremental extraction. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). |
| `end_date` | | now | Optional upper bound for extraction. |
| `sandbox` | | `"false"` | Set to `"true"` to use TikTok sandbox API hosts instead of production. |
| `user_agent` | | — | User-Agent header for API calls. TikTok recommends setting this to identify your integration. |
| `request_timeout` | | `300` | Request timeout in seconds. |
| `page_size` | | `1000` | Number of results per API page. |
| `include_deleted` | | `"false"` | Set to `"true"` to include deleted rows in the `ads`, `adgroups`, and `campaigns` streams. |
| `insights_lookback_window` | | `0` | Days to rewind from the saved reporting bookmark for insight streams. Never goes earlier than `start_date`. |

## Sample configs

**Minimal — entity and insight streams for one account**:

```yaml theme={"system"}
access_token: YOUR_ACCESS_TOKEN
accounts: "1234567890123456789"
start_date: "2024-01-01T00:00:00Z"
```

**Multiple accounts with deleted rows included**:

```yaml theme={"system"}
access_token: YOUR_ACCESS_TOKEN
accounts: "1234567890123456789,9876543210987654321"
start_date: "2024-01-01T00:00:00Z"
include_deleted: "true"
insights_lookback_window: 7
```

**Sandbox testing**:

```yaml theme={"system"}
access_token: YOUR_SANDBOX_TOKEN
accounts: "1234567890123456789"
start_date: "2024-01-01T00:00:00Z"
sandbox: "true"
```

## Streams

| Stream | Type | Description |
| - | - | - |
| `advertisers` | Full table | Advertiser account details for the configured `accounts`. |
| `campaigns` | Full table | Campaigns per advertiser. Set `include_deleted: "true"` to include deleted campaigns. |
| `adgroups` | Full table | Ad groups per campaign. |
| `ads` | Full table | Individual ads per ad group. |
| `audiences` | Full table | Custom audiences per advertiser, including `ad_type`. Requires TikTok Audience Management read permission. |
| `ad_insights` | Incremental | Daily ad-level performance metrics from `start_date`. |
| `ad_insights_by_age_and_gender` | Incremental | Ad performance broken down by age and gender. |
| `ad_insights_by_country` | Incremental | Ad performance broken down by country. |
| `ad_insights_by_platform` | Incremental | Ad performance broken down by platform (iOS, Android, etc.). |
| `campaign_insights_by_province` | Incremental | Campaign performance broken down by province/region. |

## How to get an access token

1. Go to the [TikTok for Business Marketing API portal](https://ads.tiktok.com/marketing_api).
2. Create a new app (use the **Marketing API** app flow, not the consumer "TikTok for Developers" Login Kit/Share Kit flow).
3. Under app permissions, request **read** access for:
   * **Ad Account Management**
   * **Ads Management**
   * **Audience Management**
   * **Reporting**
4. Optionally add a **sandbox advertiser** for testing without spending budget.
5. Authorize the app against your TikTok Ads account and generate an access token.
6. Copy the `access_token` and your advertiser ID(s) into the source config.

> **Token expiry:** TikTok Marketing API access tokens are long-lived but can expire. If syncs fail with auth errors, regenerate the token in the portal and update your config.

## Related links

* [TikTok Marketing API documentation](https://ads.tiktok.com/marketing_api/docs)
* [TikTok Ads Manager](https://ads.tiktok.com/)
* [Reporting API reference](https://ads.tiktok.com/marketing_api/docs?id=1738864835805185)


# Twitter Ads
Source: https://docs.mage.ai/data-integrations/sources/twitter_ads



![Twitter Ads](https://user-images.githubusercontent.com/80284865/233933127-e360397e-5997-4b16-9df3-6363a62809a7.png)

## Configuration

Connect to the Twitter Ads API and extract campaign and ad performance data by providing the following configuration parameters:

| Key | Description | Sample Value |
| - | - | - |
| `start_date` | Absolute beginning date for syncing bookmarked endpoints (`YYYY-MM-DDTHH:MM:SSZ`). | `2023-01-01T00:00:00Z` |
| `consumer_key` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) consumer key. | `YOUR_TWITTER_ADS_CONSUMER_KEY` |
| `consumer_secret` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) consumer secret. | `YOUR_TWITTER_ADS_CONSUMER_SECRET` |
| `access_token` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) access token. | `YOUR_TWITTER_ADS_ACCESS_TOKEN` |
| `access_token_secret` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) access token secret. | `YOUR_TWITTER_ADS_ACCESS_TOKEN_SECRET` |
| `account_ids` | Comma-delimited list of Twitter Ad Account IDs. | `id1, id2, id3` |
| `attribution_window` | Number of days for latency look-back period to stabilize analytics reporting. | `14` |
| `with_deleted` | Include logically deleted records in the results (`true` or `false`). | `true` |
| `country_codes` | Comma-delimited list of ISO 2-letter country codes for targeting and segmentation. | `US, CA, MX, DE` |
| `page_size` | Optional: custom `page_size` for pagination. | `1000` |
| `reports` | List of reports with `name`, `entity`, `segment`, and `granularity` details. | `[{"name": "campaigns_genders_hourly_report", "entity": "CAMPAIGN", "segment": "GENDER", "granularity": "HOUR"}]` |
| `request_timeout` | Timeout for the Twitter Ads API client (default: 300 seconds). | `300` |

## How to Get Access to the Twitter Ads API

To use the Twitter Ads source, you must first get access to the Twitter Ads API. Follow [this guide](https://developer.twitter.com/en/docs/twitter-ads-api/getting-started) to apply for access and configure your app credentials.

## What is Twitter Ads?

[Twitter Ads](https://ads.twitter.com/) is an advertising platform that allows businesses to promote content, grow followers, and drive engagement through highly targeted ad campaigns across the Twitter network.

## Why Integrate Twitter Ads with Mage?

* **Automated marketing insights**: Extract detailed campaign, ad group, and ad-level performance metrics.
* **Advanced segmentation**: Access granular breakdowns by gender, location, device, and more.
* **Real-time reporting**: Keep your analytics and BI dashboards updated with the latest ad performance data.
* **Scalable API extraction**: Efficiently handle large ad accounts and high-volume data reporting.
* **Secure and reliable**: OAuth 1.0a authentication ensures secure API access.

## Supported Streams

The following Twitter Ads objects are extracted:

* account\_media
* accounts
* advertiser\_business\_categories
* campaigns
* cards
* cards\_image\_conversation
* cards\_poll
* cards\_video\_conversation
* content\_categories
* funding\_instruments
* iab\_categories
* line\_item\_apps
* line\_items
* media\_creatives
* preroll\_call\_to\_actions
* promotable\_users
* promoted\_accounts
* promoted\_tweets
* scheduled\_promoted\_tweets
* tailored\_audiences
* targeting\_app\_store\_categories
* targeting\_conversations
* targeting\_criteria
* targeting\_devices
* targeting\_events
* targeting\_interests
* targeting\_languages
* targeting\_locations
* targeting\_network\_operators
* targeting\_platform\_versions
* targeting\_platforms
* targeting\_tv\_markets
* targeting\_tv\_shows
* tracking\_tags
* tweets


# Xero
Source: https://docs.mage.ai/data-integrations/sources/xero

Sync accounting data from Xero including invoices, contacts, payments, and more. Supports incremental syncing with automatic OAuth token refresh.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

## Configuration

To set up the Xero source, provide the following configuration parameters:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `client_id` | Client ID issued when you create your Xero app. | `abcdefg` | ✅ |
| `client_secret` | Client secret generated when you create your Xero app. | `abcdefg` | ✅ |
| `tenant_id` | Tenant ID associated with your Xero organization. | `abcdefg` | ✅ |
| `start_date` | The start date for syncing data. Format: `YYYY-MM-DDTHH:MM:SSZ`. | `2021-01-01T00:00:00Z` | ✅ |
| `refresh_token` | OAuth refresh token (required if `refresh_token_secret_name` is not provided). | `abcdefg` | ⚠️ |
| `refresh_token_secret_name` | Name of the Mage secret containing the refresh token (recommended for production). | `XERO_REFRESH_TOKEN` | ⚠️ |

> [!NOTE]
>
> You must provide either `refresh_token` or `refresh_token_secret_name`. Using `refresh_token_secret_name` is recommended because Xero uses rotating refresh tokens (each token can only be used once), and this option automatically persists the new token after each refresh.

### Option 1: Direct Refresh Token

```yaml theme={"system"}
client_id: YOUR_CLIENT_ID
client_secret: YOUR_CLIENT_SECRET
tenant_id: YOUR_TENANT_ID
start_date: "2021-01-01T00:00:00Z"
refresh_token: YOUR_REFRESH_TOKEN
```

> [!WARNING]
>
> Xero refresh tokens are single-use. Each time the token is refreshed, a new one is issued and the old one is invalidated. With this option, the new token is only stored in memory during the sync, so subsequent pipeline runs may fail if the token has been rotated.

### Option 2: Mage Secret (Recommended)

```yaml theme={"system"}
client_id: YOUR_CLIENT_ID
client_secret: YOUR_CLIENT_SECRET
tenant_id: YOUR_TENANT_ID
start_date: "2021-01-01T00:00:00Z"
refresh_token_secret_name: XERO_REFRESH_TOKEN
```

First, create a secret in Mage. You can choose any name for your secret:

1. Open a pipeline edit page
2. Expand the right side panel
3. Click the **"Secrets"** tab
4. Click **"Add new secret"**
5. Enter your chosen secret name (e.g., `XERO_REFRESH_TOKEN`) and your initial refresh token as the value
6. Copy the secret name to your Xero config's `refresh_token_secret_name` field

> [!TIP]
>
> The tap will automatically update the secret with the new refresh token after each sync, ensuring reliable long-term operation.

## Supported Streams

The Xero source supports the following streams:

### Incremental Streams

These streams support incremental syncing based on the `start_date` configuration and bookmark state:

| Stream | Primary Key | Replication Key |
| - | - | - |
| `accounts` | AccountID | UpdatedDateUTC |
| `bank_transactions` | BankTransactionID | UpdatedDateUTC |
| `bank_transfers` | BankTransferID | CreatedDateUTC |
| `contacts` | ContactID | UpdatedDateUTC |
| `credit_notes` | CreditNoteID | UpdatedDateUTC |
| `employees` | EmployeeID | UpdatedDateUTC |
| `expense_claims` | ExpenseClaimID | UpdatedDateUTC |
| `invoices` | InvoiceID | UpdatedDateUTC |
| `items` | ItemID | UpdatedDateUTC |
| `journals` | JournalID | JournalNumber |
| `linked_transactions` | LinkedTransactionID | UpdatedDateUTC |
| `manual_journals` | ManualJournalID | UpdatedDateUTC |
| `overpayments` | OverpaymentID | UpdatedDateUTC |
| `payments` | PaymentID | UpdatedDateUTC |
| `prepayments` | PrepaymentID | UpdatedDateUTC |
| `purchase_orders` | PurchaseOrderID | UpdatedDateUTC |
| `quotes` | QuoteID | UpdatedDateUTC |
| `receipts` | ReceiptID | UpdatedDateUTC |
| `users` | UserID | UpdatedDateUTC |

### Full Table Streams

These streams are fully synced on each run (no incremental support):

| Stream | Primary Key |
| - | - |
| `branding_themes` | BrandingThemeID |
| `contact_groups` | ContactGroupID |
| `currencies` | Code |
| `organisations` | OrganisationID |
| `repeating_invoices` | RepeatingInvoiceID |
| `tax_rates` | TaxType |
| `tracking_categories` | TrackingCategoryID |

## How to Generate Credentials

To use this source, you need to create a Xero OAuth app and complete the OAuth flow.

### 1. Create a Xero App

Go to the [Xero Developer Portal](https://developer.xero.com/app/manage) and create a new app. Note down your **Client ID** and **Client Secret**.

### 2. Configure OAuth2 Scopes

When setting up your Xero app, grant the following OAuth2 scopes:

| Scope | Required For |
| - | - |
| `accounting.settings.read` | Organisation settings, currencies, etc. |
| `accounting.transactions.read` | Invoices, bank transactions, payments |
| `accounting.contacts.read` | Contacts |
| `accounting.reports.read` | Reports |
| `accounting.journals.read` | Journals |
| `accounting.attachments.read` | Attachments |
| `offline_access` | Long-lived access via refresh tokens |

> [!NOTE]
>
> * The minimum scope required for discovery is `accounting.settings.read`.
> * You only need scopes for the streams you plan to sync.
> * Missing scopes for a selected stream will result in `401` or `403` errors.

**Example Scope String:**

```
accounting.settings.read accounting.transactions.read accounting.contacts.read accounting.reports.read accounting.journals.read accounting.attachments.read offline_access
```

### 3. Complete the OAuth2 Authorization Flow

Follow [Xero's OAuth2 Auth Flow Guide](https://developer.xero.com/documentation/guides/oauth2/auth-flow/) to obtain your initial `access_token` and `refresh_token`.

> [!TIP]
>
> Make sure to request the `offline_access` scope to receive a refresh token.

### 4. Get Your Tenant ID

The `tenant_id` uniquely identifies your Xero organization. After completing OAuth authentication:

1. Call the connections endpoint:

```bash theme={"system"}
curl -X GET "https://api.xero.com/connections" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json"
```

2. Extract the `tenantId` from the response:

```json theme={"system"}
[
  {
    "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "tenantType": "ORGANISATION",
    "tenantName": "Your Company Name"
  }
]
```

> [!NOTE]
>
> If your app is connected to multiple Xero organizations, the response will contain multiple entries. Choose the `tenantId` for the organization you want to sync data from.

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `access_token` | OAuth access token used to authenticate API requests. | `abcdefg` | ✅ |
| `id_token` | Token containing user identity details (if OpenID Connect scopes are requested). | `abcdefg` | ✅ |
| `start_date` | The start date for syncing data. Format: `YYYY-MM-DDTHH:MM:SSZ`. | `2021-01-01T00:00:00Z` | ✅ |
| `client_id` | Client ID issued when you create your Xero app. | `abcdefg` | ✅ |
| `client_secret` | Client secret generated when you create your Xero app. | `abcdefg` | ✅ |
| `tenant_id` | Tenant ID associated with your Xero organization. | `abcdefg` | ✅ |
| `refresh_token` | Refresh token used to renew the access token after expiration (requires `offline_access` scope). | `abcdefg` | ✅ |

## Required Xero OAuth2 Scopes

To use this source, you must grant your Xero app the following OAuth2 scopes.

### Minimum Scope for Discovery

* `accounting.settings.read`
  Required for the `/Currencies` endpoint, which is called during discovery to verify:
  * Your `access_token` is valid
  * Your `tenant_id` is authorised
  * The app has the correct scopes

If this scope is missing, discovery will fail before any sync begins.

### Scopes for Syncing All Streams

To sync all available streams, grant the following scopes:

| Scope | Required For |
| - | - |
| `accounting.settings.read` | Organisation settings, currencies, etc. |
| `accounting.transactions.read` | Invoices, bank transactions, payments |
| `accounting.contacts.read` | Contacts |
| `accounting.reports.read` | Reports |
| `accounting.journals.read` | Journals |
| `accounting.attachments.read` | Attachments |
| `offline_access` | Long-lived access via refresh tokens |

**Note:**

* You only need scopes for the streams you plan to sync.
* Missing scopes for a selected stream will result in `401` or `403` errors.

### Example Scope String

```
accounting.settings.read
accounting.transactions.read
accounting.contacts.read
accounting.reports.read
accounting.journals.read
accounting.attachments.read
offline_access
```

## How to Generate Credentials

Follow Xero's [OAuth 2.0 Authentication Flow guide](https://developer.xero.com/documentation/guides/oauth2/auth-flow/) to generate the required credentials:

1. Register a new Xero app to get your `client_id` and `client_secret`.
2. Set up the OAuth scopes your app requires.
3. Complete the OAuth 2.0 authorization flow to obtain your `access_token`, `refresh_token`, `id_token`, and `tenant_id`.
4. Store these securely in your configuration.

## Additional Notes

* The `start_date` determines how far back data will be synced.
* Ensure that the `offline_access` scope is enabled to receive a `refresh_token`.
* Tokens expire after a set period; use the `refresh_token` to obtain new `access_token` and `id_token` values automatically.
* During setup, the connector calls the `/Currencies` endpoint to validate platform access before syncing.


# YouTube Analytics API Source
Source: https://docs.mage.ai/data-integrations/sources/youtube_analytics

This guide explains how to configure the YouTube Analytics API as a data source in Mage. Learn how to authenticate with Google Cloud, request the necessary OAuth scopes, and sync data from specific channels.


Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

Mage supports ingesting data from YouTube Analytics by connecting to the YouTube Analytics API and YouTube Data API v3 using OAuth 2.0 credentials.

This integration allows you to extract rich channel analytics, video performance, and viewer engagement metrics directly into your Mage pipelines.

***

## 🔧 Configuration

To connect to the YouTube Analytics API, provide the following parameters in your source configuration:

| Key | Description | Sample Value | Required |
| - | - | - | - |
| `client_id` | OAuth 2.0 Client ID from Google Cloud Console | `123456789-abcdef.apps.googleusercontent.com` | ✅ |
| `client_secret` | OAuth 2.0 Client Secret from Google Cloud Console | `abcdefghijklmnopqrstuvwxyz` | ✅ |
| `refresh_token` | OAuth 2.0 refresh token for authenticating requests | `1//abcdefghijklmnopqrstuvwxyz` | ✅ |
| `start_date` | Start date for pulling analytics data | `2021-01-01T00:00:00Z` | ✅ |
| `user_agent` | User agent string identifying the integration | `example@example.com` | ✅ |
| `channel_ids` | Comma-separated list of channel IDs to retrieve analytics for | `UC1234..., UC5678...` | ✅ |

***

## 🔐 How to Generate Credentials

Follow these steps to generate the required credentials from the Google Cloud Console:

### 1. Set Up OAuth Credentials

1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project (or use an existing one)
3. Enable the following APIs:
   * **YouTube Data API v3**
   * **YouTube Analytics API v2**
4. Navigate to “APIs & Services > Credentials”
5. Create a new **OAuth 2.0 Client ID** (select "Web application" or "Desktop app")

### 2. Generate a Refresh Token

Use [OAuth 2.0 Playground](https://developers.google.com/oauthplayground/) to get a `refresh_token`:

* Click the gear icon and check **"Use your own OAuth credentials"**, then enter your Client ID and Secret.
* Select the following scopes:
  * `https://www.googleapis.com/auth/youtube.readonly`
  * `https://www.googleapis.com/auth/yt-analytics.readonly`
* Follow the steps to **authorize**, **exchange code for tokens**, and **copy the refresh token**.

## How It Works

This integration connects to two key Google APIs to extract YouTube Analytics data and video metadata:

### APIs Used

* **[YouTube Reporting API](https://developers.google.com/youtube/reporting/v1/reports)**\
  Used to retrieve bulk analytics reports for YouTube channels and content owners.

  * [Jobs](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/jobs/list): List available report jobs
  * [Reports](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/jobs.reports/list): Retrieve generated reports
  * [Report Types](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/reportTypes/list): View available report categories
  * Report examples:
    * [Channel Reports](https://developers.google.com/youtube/reporting/v1/reports/channel_reports)
    * [Content Owner Reports](https://developers.google.com/youtube/reporting/v1/reports/content_owner_reports)

* **[YouTube Data API v3](https://developers.google.com/youtube/v3/docs)**\
  Used to fetch real-time metadata and performance metrics for channels, videos, and playlists.

  * [Channels](https://developers.google.com/youtube/v3/docs/channels/list)
  * [Playlists](https://developers.google.com/youtube/v3/docs/playlists/list)
  * [Playlist Items](https://developers.google.com/youtube/v3/docs/playlistItems/list)
  * [Videos](https://developers.google.com/youtube/v3/docs/videos/list)

## Supported Streams

The following streams are available in the YouTube Analytics integration:

| Stream | Description |
| - | - |
| `channels` | Metadata and statistics for authenticated YouTube channels |
| `playlist_items` | Videos and items within playlists |
| `playlists` | Metadata about playlists created by the authenticated user |
| `reports` | Bulk analytics data from YouTube Reporting API |
| `videos` | Video metadata, engagement, and performance statistics |

## Sync Behavior

* **Incremental Syncs**\
  The integration supports stateful incremental loading, so only new or modified records are fetched during each sync run.

* **Schema Output**\
  Mage automatically generates the output schema for each stream, making it easier to understand and work with the resulting data.


# Zendesk
Source: https://docs.mage.ai/data-integrations/sources/zendesk



![Zendesk](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Zendesk_logo.svg/2560px-Zendesk_logo.svg.png)

## Config

You must enter the following credentials when configuring this source:

| Key | Description | Sample value |
| - | - | - |
| `access_token` | OAuth token to access API endpoints. | `abc123...` |
| `request_timeout` | Number of milliseconds until the API request should timeout and raise an error. | 300 |
| `start_date` | Fetch data that is newer than the `start_date`. | `2022-01-01T00:00:00Z` |
| `subdomain` | If you access Zendesk from a URL like this [https://magesupport.zendesk.com/](https://magesupport.zendesk.com/), then the `domain` is `magesupport`. | `magesupport` |

## How to get your `access_token`

Follow [Zendesk’s documentation](https://support.zendesk.com/hc/en-us/articles/4408845965210)
to get your access token.

Example Admin Center URL: [https://magesupport.zendesk.com/admin/home](https://magesupport.zendesk.com/admin/home)
(subdomain in example is `magesupport`).

### Example values for getting access token

**Zendesk authorization page**

[https://magesupport.zendesk.com/oauth/authorizations/new?client\_id=mage-ai\&response\_type=code\&redirect\_uri=https://www.mage.ai\&scope=read](https://magesupport.zendesk.com/oauth/authorizations/new?client_id=mage-ai\&response_type=code\&redirect_uri=https://www.mage.ai\&scope=read)

**Authorize redirect URL with code**

[https://www.mage.ai/?code=6b21972c79e12cbda37947df4ba22d4cb394da752e955c6dac81361d5f453695](https://www.mage.ai/?code=6b21972c79e12cbda37947df4ba22d4cb394da752e955c6dac81361d5f453695)

**Example cURL command to exchange code for access token**

```curl theme={"system"}
curl --request POST \
  --url https://magesupport.zendesk.com/oauth/tokens \
  --header 'Content-Type: application/json' \
  --data '{
  "grant_type": "authorization_code",
  "code": "6b21972c79e12cbda37947df4ba22d4cb394da752e955c6dac81361d5f453695",
  "client_id": "mage-ai",
  "client_secret": "your_oauth_client_secret",
  "redirect_uri": "https://www.mage.ai",
  "scope": "read"
}'
```

**Example OAuth token exchange response**

```json theme={"system"}
{
  "access_token": "cdaae5613efca304972cd8175adf51b560691f51e9a6331a986a772801a5d898",
  "token_type": "bearer",
  "scope": "read"
}
```


# Project Structure
Source: https://docs.mage.ai/design/abstractions/project-structure

Details about how the Mage file directory works and how typical projects are structured.

## Overview

A Mage project lives inside it's own folder. It's there where mage data is stored (`mage_data`) and individual projects are housed. You can think of a project as an environment: it contains configurations, pipelines, and other project-specific data.

## Project structure

Here is a [sample project](https://github.com/mage-ai/demo%5Fetl%5Fpipeline) and a sample folder structure:

```
.
├── mage_data
└── my-first-project
    ├── charts
    ├── custom
    ├── data_exporters
    ├── data_loaders
    ├── dbt
    ├── extensions
    ├── pipelines
    │   └── demo
    │       ├── __init__.py
    │       └── metadata.yaml
    ├── scratchpads
    ├── transformers
    ├── utils
    ├── __init__.py
    ├── io_config.yaml
    ├── metadata.yaml
    └── requirements.txt
```

Let's walk through each folder:

* Each [Block](/design/core-abstractions#block) type has it's own directory:
  * `charts`
  * [`custom`](/guides/blocks/custom-blocks)
  * [`data_loaders`](/design/blocks/data-loader)
  * [`data_exporters`](/design/blocks/data-exporter)
  * [`extensions`](/design/blocks/extension)
  * [`sensors`](/design/blocks/sensor)
  * [`scratchpads`](/design/blocks/scratchpad)
  * [`transformers`](/design/blocks/transformer)
* dbt assets are stored in the [dbt directory](/dbt/overview).
* Each [pipeline](/design/core-abstractions#pipeline) is represented by a YAML file in the `pipelines` folder under the Mage project directory: `[project_dir]/pipelines/[pipeline_name]/metadata.yaml`
* The `utils` folder is meant to hold custom utilities for your project. For example, [Python scripts](/development/dependencies/custom-files).
* The `extensions` folder is used for Mage extensions that integrate other data tools, like [Great Expectations](/development/testing/great-expectations).
* Data loader and exporter [configurations](/development/io_config) are stored in the `io_config.yaml` file.
* The `metadata.yaml` file contains project-level metadata. There's a metadata file in each pipeline as well.

Code can be shared across the entire project.

## Project organization

We know that you're hard at work creating blocks and building pipelines, that's why we've made it easy to organize your project.

Blocks can be organized in subdirectories within their respective folders— no additional configuration required. For example, you might have a `data_loaders` folder with subdirectories for `api`, `database`, and `file` data loaders.

You can do the same when creating blocks via the UI— simply name the block `subfolder_name/block_name` and the subfolder will be automatically created.

This might look like:

```
.
└── data_loaders
    ├── api
    │   ├── load_bamboo.py
    │   ├── load_recurly.py
    │   └── load_stripe.py
    ├── database
    |   ├── load_duckdb.py
    │   └── load_influxdb.py
    └── file
        ├── load_avro.py
        └── load_parquet.py
```

## Project data

Project-level data is stored in the `mage_data` directory— located at the same level as your project folder.

`mage_data` holds project-level cache data and stores the result of Block runs, which are then returned to the user via the UI. Here are the locations for various components of `mage_data`:

* SQLite DB: `mage_data/[project_folder]/mage-ai.db`

  * The SQLite DB stores project-level metadata, such as pipeline and block run data. Here's a sample tree tables in the `tables` schema:

  ```
   └── tables
       ├── backfill
       ├── block_run
       ├── event_matcher
       ├── oauth2_access_token
       ├── oauth2_application
       ├── permission
       ├── pipeline_run
       ├── pipeline_schedule
       ├── pipeline_schedule_event_matcher_association
       ├── role
       ├── secret
       ├── sqlite_master
       ├── user
       └── user_role
  ```

  * You can connect to the SQLite DB like any other database. The JDBC URL `jdbc:sqlite:PATH`, where PATH is `/home/src/mage_data/[project_folder]/mage-ai.db` for Docker installs or `~/.mage_data/[project_folder]/mage-ai.db` for pip installs.
* Block output: `mage_data/[project_folder]/pipelines/[pipeline_uuid]/.variables/`
* Pipeline and block logs: `mage_data/[project_folder]/pipelines/[pipeline_uuid]/.logs/`
* Cache: `mage_data/[project_folder]/.cache`

To clear project block output, cache data, or log data, consider performing routine operations on this folder. This might be helpful for reducing storage size, optimizing cost, or otherwise maintaining your Mage project.

## Changing the name of your project

### Docker

If you’re running Mage in Docker, change the environment variable `USER_CODE_PATH` to the absolute path where your project lives.

For example, if your original project name was `demo_magic` and the directory is located at `/home/src/demo_magic`,
then the value of `USER_CODE_PATH` should be: `/home/src/demo_magic`.

### Not using Docker

When you run `mage start [project_name]` on the command line to start running Mage,
the 1st command line argument value is used for your project’s name.


# Overview
Source: https://docs.mage.ai/design/blocks

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

## Unique block paradigm

### Modularity

Mage breaks down complex data engineering tasks into smaller, manageable units (blocks). Each block in a pipeline is a standalone file that can be executed independently.

Modular design makes it easier to maintain and update data pipelines by modifying a specific block or creating new ones without affecting other parts of the pipeline.

This isolation reduces the risk of introducing new bugs into the system when making changes. The modularity of blocks leads to cleaner, more maintainable codebases.

Modular design enables fault-tolerant pipelines so that when one or more blocks fail, the rest can keep running. Debugging modular blocks is faster than debugging a monolithic code base. Once resolved, failed blocks can be retried without re-running the entire pipeline.

### Reusability

This design encapsulates each step of a data pipeline into reusable blocks that perform a specific function; such as data loading, transformation, or exporting.

Developers can build complex pipelines faster by leveraging existing blocks across different projects, different pipelines, or within the same pipeline.

Designing blocks to be reusable significantly reduces the time and effort required to develop new data pipelines, as developers can reuse blocks instead of writing new code from scratch.

### Scalability

Developers can optimize individual blocks for handling large data volumes or more complex processing tasks efficiently, without the need for extensive redesigns or refactoring of the entire pipeline.

Dynamic blocks adapt their behavior based on input data or runtime conditions, enabling the creation of flexible and complex data pipelines that can easily accommodate varying scalability requirements all without the need to write duplicate code.

***

## Block types

* [Data loader](/design/blocks/data-loader)
* [Transformer](/design/blocks/transformer)
* [Data exporter](/design/blocks/data-exporter)
* [Scratchpad](/design/blocks/scratchpad)
* [Sensor](/design/blocks/sensor)
* [dbt](/dbt/overview)
* [Extensions](/design/blocks/extension)
* [Callbacks](/design/blocks/callbacks)

![Blocks](https://user-images.githubusercontent.com/97557538/198362650-6a61dfb2-8afb-40d4-b5ce-fb9f145f23a6.png)


# Callback blocks
Source: https://docs.mage.ai/design/blocks/callbacks

A callback block is associated to another block. When the parent block succeeds or fails, the callback block functions are executed.

> [!NOTE]
>
> Requires version `0.8.61` or greater.

![Callbacks](https://i.imgflip.com/3s2icl.jpg)

Callback blocks are part of your pipeline but don’t run as individual steps in your pipeline
like data loader blocks, transformer blocks, etc.
However, callback blocks are associated to other blocks within the pipeline
(e.g. data loaders, transformers, data exporters, etc).

After those other blocks are executed,
any associated callback blocks will also be executed depending on the status of the parent block.

***

## Example

1. Your pipeline has the following blocks:
   1. `load_data_from_api` (data loader)
   2. `clean_column_names` (transformer)
   3. `save_data` (data exporter)

2. Then, you add the following callback block to your pipeline:

   ```python theme={"system"}
   @callback('success')
   def hello_world(parent_block_data, **kwargs):
       # some code...


   @callback('failure')
   def alert_me(parent_block_data, **kwargs):
       # some code...

   @callback('cancelled')
   def notify_cancelled(parent_block_data, **kwargs):
       # some code...
   ```

3. You associate the above callback block to the following 2 blocks:

   * `load_data_from_api` (data loader)
   * `clean_column_names` (transformer)

4. When you run the pipeline, it’ll execute the `load_data_from_api` (data loader) block first.
   It successfully completes. Then, the callback block will run and execute the function `hello_world`.

5. Next, the pipeline will execute the `clean_column_names` (transformer) block.
   It fails due to some error. The callback block will run and execute the function `alert_me`.

***

## How to add callbacks to your pipeline

1. Create a new pipeline or open an existing pipeline.
2. Edit the pipeline.
3. On the right side of the page, click the **Add-ons** icon in the navigation.
   If you don’t see it, try expanding the right area of the page.
4. Click the button **Callbacks**.
5. Click the button **+ Callback block**.
6. In the callback block’s code, add a function and decorate it with `@callback`.
   You can have as many functions in a callback block as you want.
7. Use `@callback` or `@callback('success')` to decorate a function that should only run when the
   parent block successfully completes.
8. Use `@callback('failure')` to decorate a function that should only run when the
   parent block fails.

### Supported callbacks

#### `success`

Execute callback function when block runs successfully.

```python theme={"system"}
@callback('success')
def only_run_this_function_on_success(parent_block_data, **kwargs):
    pass
```

#### `failure`

Execute callback function when block fails.

```python theme={"system"}
@callback('failure')
def only_run_this_function_on_failure(parent_block_data, **kwargs):
    pass
```

#### `cancelled`

\[!NOTE] Available in versions `>= 0.9.77`.
Execute callback function when block is cancelled (manually or by system interruption).

```python theme={"system"}
@callback('cancelled')
def only_run_this_function_on_cancelled(parent_block_data, **kwargs):
    pass
```

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=pipeline-callbacks)

## Mage Pro callback features

### Pipeline-level callbacks

In **Mage Pro**, callback blocks can be attached to the pipeline itself instead of only to
individual parent blocks. Pipeline-level callbacks run once after a pipeline run reaches one of
these terminal statuses:

* `completed`
* `failed`
* `cancelled`

Block-level callbacks continue to run from their parent block status. Pipeline-level callbacks are
separate and wait for the full pipeline run outcome.

To configure pipeline-level callbacks in the editor:

1. Open the pipeline editor.
2. Create at least one callback block from the **Add-ons** panel.
3. In the callback block footer selector, choose the current pipeline UUID under **Pipeline**.

The selector groups targets under **Pipeline** and **Blocks**. The **Pipeline** group
contains only the current pipeline UUID.
If a callback block still uses block-level-only function signatures with required positional block
input, such as `def success_callback(parent_block_data, **kwargs)`, the editor still shows the
current pipeline target. Saving that pipeline target shows a compatibility error because pipeline
callbacks do not pass `parent_block_data` or other block inputs. To use the callback at the pipeline
level, make block input arguments optional, for example
`def success_callback(parent_block_data=None, **kwargs)`, use documented pipeline context keyword
arguments, or use a kwargs-only signature such as `def success_callback(**kwargs)`. The original
required-argument signature remains valid for block-level callbacks.

You can also configure the same behavior in the pipeline `metadata.yaml` file or through the
pipeline create and update API:

```yaml theme={"system"}
callbacks:
- name: notify stakeholders
  type: callback
  uuid: notify_stakeholders

pipeline_callbacks:
- callback_block_uuid: notify_stakeholders
  enabled: true
```

The association controls whether the callback block runs once for the pipeline. The callback block
code controls terminal-status-specific behavior by defining matching callback functions:

```python theme={"system"}
@callback('success')
def notify_pipeline_completed(**kwargs):
    pipeline_run_id = kwargs.get('pipeline_run_id')
    terminal_status = kwargs.get('terminal_status')
    print(f'Pipeline run {pipeline_run_id} finished with {terminal_status}.')


@callback('failure')
def notify_pipeline_failed(**kwargs):
    error_context = kwargs.get('terminal_status')


@callback('cancelled')
def notify_pipeline_cancelled(**kwargs):
    pipeline_run_id = kwargs.get('pipeline_run_id')
```

Pipeline-level callbacks receive a safe pipeline-run context instead of upstream block output.
Available keyword arguments are intentionally minimal:

| Name | Description |
| - | - |
| `pipeline_run_id` | ID of the pipeline run that triggered the callback. |
| `terminal_status` | Terminal pipeline run status: `completed`, `failed`, or `cancelled`. |

Use `pipeline_run_id` with existing authorized APIs or models if callback code needs additional
pipeline-run details.

Run details show each dispatched pipeline callback as a block run with its status and any
user-code error captured by the normal block-run execution path. Callback statuses can be
`initial`, `queued`, `running`, `completed`, or `failed`.

### Advanced parameters

In **Mage Pro**, you can pass additional parameters to the `@callback` decorator.

#### `fetch_inputs`

By default, Mage will fetch upstream inputs before executing a callback.
If your callback doesn’t need input data (e.g. sending alerts, posting logs), you can **disable input fetching** to improve performance.

```python theme={"system"}
@callback('failure', fetch_inputs=False)
def alert_me_without_inputs(*args, **kwargs):
    # Runs faster since upstream block inputs are not fetched
    send_alert("Parent block failed!")
```

* **`fetch_inputs=True`** (default) → Fetch parent block inputs before executing callback
* **`fetch_inputs=False`** → Skip fetching inputs, speeding up execution

### Positional arguments

The 1st positional argument is the data output of the callback block’s parent block.

> [!NOTE]
>
> Positional arguments are only supported in Mage versions `>= 0.8.81`.

### Keyword arguments

Here are the keyword arguments that are  available in each callback function:

| Name | Description | Sample value |
| - | - | - |
| `block_uuid` | Callback block UUID. | `'fireball_callback'` |
| `parent_block_uuid` | The block UUID of the parent block. | `'parent_block'` |
| `ds` | Date string when the parent block started executing. | `'2023-12-25'` |
| `event` | A dictionary containing metadata from an event triggered pipeline. | `{}` |
| `execution_date` | Python datetime object for when the parent block started executing. | `datetime.datetime(2023, 4, 26, 20, 28, 17, 335254, tzinfo=datetime.timezone.utc)` |
| `execution_partition` | Partition used for the parent block when it was executed. | `'207/20230426T202817'` |
| `hr` | Hour string when the parent block started executing. | `'20'` |
| `pipeline_run` | Python pipeline run object associated to the current run. | `PipelineRun(id=2357, pipeline_uuid=fire_etl, execution_date=2023-04-26 20:28:17.335254+00:00)` |
| `pipeline_uuid` | UUID of the current pipeline. | `'fire_etl'` |
| `__input` | For version `>= 0.8.81`. See below for more details. | `{ 'some_block_uuid': {} }` |

#### `__input` keyword argument

> [!NOTE]
>
> For versions `>= 0.8.81`.

If your callback function is defined to only accept keyword arguments and no positional arguments,
you can access the callback block’s parent block’s data output by accessing the key
labeled `__input` in the callback function’s keyword arguments.

The value of the `__input` key is a dictionary. The keys in that dictionary are named after the
callback block’s parent block’s UUID.

For example, if the callback block’s parent block’s UUID is `load_api_data`, then this is how
you would access `load_api_data`’s data output:

```python theme={"system"}
@callback('success')
def callback_for_load_api_data(**kwargs):
    data_from_parent_block = kwargs['__input']['load_api_data']
```

#### Data integration pipelines only

The following keyword arguments are only available in callback functions with parent blocks in a
data integration pipelines:

| Name | Description | Sample value |
| - | - | - |
| `destination_table` | The table name that the source data is being exported to. | `'dim_users_v1'` |
| `index` | The order in which the parent block executed compared to other blocks of the same type. | `0` |
| `is_last_block_run` | Boolean value for whether or not the parent block was the last block ran for the block type in the pipeline. | `True` or `False` |
| `stream` | The name of the source stream (or table) that the parent block is syncing for. | `'stripe_transactions'` |

***

## Legacy

> [!NOTE]
>
> For versions `< 0.8.61`.

### Adding a callback to your block

1. Create a block as normal.
2. In the top right of the block, click the triple dot icon which should open up a menu
   with more options. Click "Add callback".
3. You should now see another code editor under the main editor for your block.
   ![Block callback](https://mage-ai.github.io/assets/block-callback.png)
4. You can add an `on_success` callback and/or a `on_failure` callback. These callbacks
   will run after your block run completes or fails. When you run your block in the
   pipeline edit page, it will also run the callback depending on the status of the block.
5. The callback function will be passed your pipeline's runtime variables as keyword
   arguments, so you can use the same `kwargs.get('<variable>')` syntax in your callback.


# Conditional blocks
Source: https://docs.mage.ai/design/blocks/conditionals

A conditional block is an 'Add-on Block' associated with another block. The condition will be evaluated before the parent block is executed, determining if the parent block gets executed.

> [!NOTE]
>
> Requires version `0.8.93` or greater.

Conditional blocks are part of your pipeline but don't run as individual steps in your pipeline like data loader blocks, transformer blocks, etc.
However, conditional blocks are associated to other blocks within the pipeline (e.g. data loaders, transformers, data exporters, etc).

Before those other blocks are executed, any associated conditional blocks will be executed. If any conditional block returns False, the parent block will not be executed, and its status will be set as `condition_failed`.

### An example condition

Take the following example:

![](https://mage-ai.github.io/assets/blocks/conditionals-example.png)

The `morpheus` block returns a variable— `pill`, which is either red or blue. Both downstream blocks have conditionals associated with them. The `red_pill` block will only execute if `pill` is red, and the `blue_pill` block will only execute if `pill` is blue.

Here's what the conditionals look like for each:

![](https://mage-ai.github.io/assets/blocks/conditional-red-pill.png)

![](https://mage-ai.github.io/assets/blocks/conditional-blue-pill.png)

Because `pill=red`, the `red_pill` block will execute, and the `blue_pill` block will not. Additionally, any *downstream* blocks of the `blue_pill` block will not execute.

## Add conditionals to your pipeline

Conditional blocks are "add-on blocks" in Mage— that means they're added through the side panel on the right in the pipeline editor.

![](https://mage-ai.github.io/assets/blocks/add-conditionals.gif)

To add a condition to your block:

1. On the right side of the page, expand the side panel by hovering, click the **Add-ons** icon in the navigation.
2. Click the button **Conditionals**.
3. Click the button **+ Conditional block**.
4. In the conditional block's code, add a function that returns a boolean value. Be sure it's decorated with `@condition`.
5. Select the magnifiying class icon at the bottom of the block to associate your conditional with a block.

Looking for a video walkthrough? Check out our [tips and tricks](/guides/tips-and-tricks) page!

### Positional arguments

The positional arguments to the conditional block will be the same as the positional
arguments to the parent block. You can use the positional arguments to get the output
of the upstream blocks.

### Keyword arguments

Here are the keyword arguments that are available in each conditional function:

| Name | Description | Sample value |
| - | - | - |
| `block_uuid` | Conditional block UUID. | `'fireball_conditional'` |
| `ds` | Date string when the parent block started executing. | `'2023-12-25'` |
| `event` | A dictionary containing metadata from an event triggered pipeline. | `{}` |
| `execution_date` | Python datetime object for when the trigger run started executing. | `datetime.datetime(2023, 4, 26, 20, 28, 17, 335254, tzinfo=datetime.timezone.utc)` |
| `execution_partition` | Partition used for the trigger run when it was executed. | `'207/20230426T202817'` |
| `hr` | Hour string when the trigger run started executing. | `'20'` |
| `pipeline_run` | Python pipeline run object associated to the current run. | `PipelineRun(id=2357, pipeline_uuid=fire_etl, execution_date=2023-04-26 20:28:17.335254+00:00)` |
| `pipeline_uuid` | UUID of the current pipeline. | `'fire_etl'` |


# Data exporter
Source: https://docs.mage.ai/design/blocks/data-exporter

After completing data transformations, utilize the data exporter blocks to either load the processed  data or store a machine learning model in an external data storage system.

![exporter](https://mage-ai.github.io/assets/blocks/exporter/data-exporter.gif)

Mage natively supports integration with a variety of data storage systems. However, these integrations often
require specific configurations in both the exporter block and the `io_config.yml` file to ensure seamless operation.
The `io_config.yml` file typically includes connection details such as host, port, database name, username, and password.
Meanwhile, the exporter block needs to be configured with the appropriate export parameters, such as target table names,
schema details, and conflict resolution strategies.

### Technical Details

1. **Data Exporter Blocks**:
   * These blocks are designed to facilitate the movement of transformed data or trained models to external systems.
   * Configuration parameters might include destination paths, file formats, table names, schemas, and update strategies.
   * Most data exporters include a `config_profile` parameter set to `'default'` by default. This parameter can be customized
     to use different configuration profiles if you have multiple profiles or have renamed them.
2. **Supported Data Storage Systems**:
   * Mage supports a wide range of storage systems including PostgreSQL, MySQL, AWS S3, Google Cloud Storage, Azure Blob Storage, and many more.
   * Each system may have unique requirements and configurations to ensure compatibility and optimal performance.
3. **Configuration in `io_config.yml`**:
   * This file serves as the central configuration hub for defining connection parameters.
   * Typical parameters include:
     * `host`: The server address of the storage system.
     * `port`: The port number for the connection.
     * `database`: The name of the target database or data storage container.
     * `username` and `password`: Authentication credentials.
     * Additional parameters as required by specific storage systems (e.g., SSL settings, API tokens).

### Examples

> **Data Warehouse**
>
> Configure the `io_config.yml` file to connect your Mage pipeline to a snowflake data warehouse. While optional,
> depending on how your Snowflake DW is configured you may need to enter all information into the .yml file.
> It’s recommended to store sensitive information as Secrets. See the general [Secrets documentation](https://docs.mage.ai/development/variables/secrets)
> for more information.
>
> ```YAML theme={"system"}
>   SNOWFLAKE_USER: username
>   SNOWFLAKE_PASSWORD: password
>   SNOWFLAKE_ACCOUNT: account_id.region
>   SNOWFLAKE_DEFAULT_WH: null                  # Optional default warehouse
>   SNOWFLAKE_DEFAULT_DB: null                  # Optional default database
>   SNOWFLAKE_DEFAULT_SCHEMA: null              # Optional default schema
>   SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null      # Optional private key passphrase
>   SNOWFLAKE_PRIVATE_KEY_PATH: null            # Optional private key path
>   SNOWFLAKE_ROLE: null                        # Optional role name
>   SNOWFLAKE_TIMEOUT: null                     # Optional timeout in seconds
> ```
>
> Enter information for the following in the Data Exporter block
>
> 1. table\_name - requires developers enter the name of their destination table
> 2. database - requires developers enter the name of their destination data base
> 3. schema - requires developers enter the name of their destination schema
>
> All other information is handled in the 'io\_config.yml' file.
>
> Example Code:
>
> ```Python theme={"system"}
> from mage_ai.settings.repo import get_repo_path
> from mage_ai.io.config import ConfigFileLoader
> from mage_ai.io.snowflake import Snowflake
> from pandas import DataFrame
> from os import path
>
> if 'data_exporter' not in globals():
>     from mage_ai.data_preparation.decorators import data_exporter
>
>
> @data_exporter
> def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
>     """
>     Template for exporting data to a Snowflake warehouse.
>     Specify your configuration settings in 'io_config.yaml'.
>
>     Docs: https://docs.mage.ai/design/data-loading#snowflake
>     """
>     table_name = 'your_table_name'
>     database = 'your_database_name'
>     schema = 'your_schema_name'
>     config_path = path.join(get_repo_path(), 'io_config.yaml')
>     config_profile = 'default'
>
>     with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
>         loader.export(
>             df,
>             table_name,
>             database,
>             schema,
>             if_exists='replace',  # Specify resolution policy if table already exists
>         )
> ```

> **Data Lake**
>
> Configure the `io_config.yml` file to connect your Mage pipeline to a Azure Blob Storage. Configure some Secrets and enter them into `io_config.yml` file.
> If you need more information on entering secrets see this [documentation](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure).
>
> ```YAML theme={"system"}
>   AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
>   AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
>   AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
>   AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
> ```
>
> Enter information for the following in the Data Exporter block
>
> 1. container\_name - requires developers enter the name of their destination container
> 2. blob\_path - requires developers enter the name of their destination blob path
>
> All other information is handled in the `io_config.yml` file.
>
> Example Code:
>
> ```Python theme={"system"}
> from mage_ai.settings.repo import get_repo_path
> from mage_ai.io.azure_blob_storage import AzureBlobStorage
> from mage_ai.io.config import ConfigFileLoader
> from pandas import DataFrame
> from os import path
>
> if 'data_exporter' not in globals():
>     from mage_ai.data_preparation.decorators import data_exporter
>
>
> @data_exporter
> def export_data_to_azure_blob_storage(df: DataFrame, **kwargs) -> None:
>     """
>     Template for exporting data to a Azure Blob Storage.
>     Specify your configuration settings in 'io_config.yaml'.
>
>     Docs: https://docs.mage.ai/design/data-loading
>     """
>     config_path = path.join(get_repo_path(), 'io_config.yaml')
>     config_profile = 'default'
>
>     container_name = 'your_container_name'
>     blob_path = 'your_blob_path'
>
>     AzureBlobStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
>         df,
>         container_name,
>         blob_path,
>     )
> ```

> **Database**
>
> Configure the `io_config.yml` file to connect your Mage pipeline to a PostgreSQL database. Configure some Secrets and enter them into `io_config.yml` file.
> If you need more information on entering secrets see this [documentation](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure).
>
> ```YAML theme={"system"}
>   POSTGRES_CONNECT_TIMEOUT: 10
>   POSTGRES_DBNAME: postgres
>   POSTGRES_SCHEMA: public # Optional
>   POSTGRES_USER: username
>   POSTGRES_PASSWORD: password
>   POSTGRES_HOST: hostname
>   POSTGRES_PORT: 5432
> ```
>
> > [!NOTE]
> >
> > If exporting from Docker to an external machine use the host.docker.internal for `POSTGRES_HOST:`
>
> Enter information for the following in the Data Exporter block
>
> 1. schema\_name - requires developers enter the name of their destination container
> 2. table\_name - requires developers enter the name of their destination blob path
>
> All other information is handled in the `io_config.yml` file.
>
> Example Code:
>
> ```Python theme={"system"}
> from mage_ai.settings.repo import get_repo_path
> from mage_ai.io.config import ConfigFileLoader
> from mage_ai.io.postgres import Postgres
> from pandas import DataFrame
> from os import path
>
> if 'data_exporter' not in globals():
>     from mage_ai.data_preparation.decorators import data_exporter
>
>
> @data_exporter
> def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
>     """
>     Template for exporting data to a PostgreSQL database.
>     Specify your configuration settings in 'io_config.yaml'.
>
>     Docs: https://docs.mage.ai/design/data-loading#postgresql
>     """
>     schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
>     table_name = 'your_table_name'  # Specify the name of the table to export data to
>     config_path = path.join(get_repo_path(), 'io_config.yaml')
>     config_profile = 'default'
>
>     with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
>         loader.export(
>             df,
>             schema_name,
>             table_name,
>             index=False,  # Specifies whether to include index in exported table
>             if_exists='replace',  # Specify resolution policy if table name already exists
>         )
> ```

> **Delta Lake**
>
> Unlike other data exporters, delta lake exporters do not currently configure through the `io_config.yml` file.
> They contain the necessary configurations within the exporter block itself. Lets break that down.
>
> **Storage Options**
>
> * `'AWS_ACCESS_KEY_ID'`: Your AWS access key ID.
>
> * `'AWS_SECRET_ACCESS_KEY'`: Your AWS secret access key.
>
> * `'AWS_REGION'`: The AWS region where your S3 bucket is located.
>
> * `'AWS_S3_ALLOW_UNSAFE_RENAME'`: This option allows unsafe rename operations on S3, which might be necessary for some workflows.
>
> > [!NOTE]
> >
> > Remember Secrets can be stored in Mage’s internal Secrets Manager, .YAML files, or sync directly with Cloud Secret Managers.
>
> **Additional Configurations**
>
> * `uri`: The S3 URI where the Delta Table is stored.
>
> Example Code:
>
> ```Python theme={"system"}
> from deltalake.writer import write_deltalake
> if 'data_exporter' not in globals():
>     from mage_ai.data_preparation.decorators import data_exporter
>
>
> @data_exporter
> def export_data(df, *args, **kwargs):
>     """
>     Export data to a Delta Table
>
>     Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
>     """
>     storage_options = {
>         'AWS_ACCESS_KEY_ID': '',
>         'AWS_SECRET_ACCESS_KEY': '',
>         'AWS_REGION': '',
>         'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
>     }
>
>     uri = 's3://[bucket]/[key]'
>
>     write_deltalake(
>         uri,
>         data=df,
>         mode='append',          # append or overwrite
>         overwrite_schema=False, # set True to alter the schema when overwriting
>         partition_by=[],
>         storage_options=storage_options,
>     )
> ```

By correctly configuring these components, you can effectively streamline the data loading process into your chosen storage system,
whether it be a relational database, a data lake, or a machine learning model repository.


# Data loader
Source: https://docs.mage.ai/design/blocks/data-loader

Write your code for fetching data from a remote source or loading it from disk.

## Example

```python theme={"system"}
import io
import pandas as pd
import requests


@data_loader
def load_data_from_api(*args, **kwargs):
    url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'

    response = requests.get(url)
    return pd.read_csv(io.StringIO(response.text), sep=',')
```

## How data is passed to other blocks

The object that is returned from the decorated function (e.g. `load_data_from_api`)
will be made available in all blocks that depend on this block.

## Templates

To read about Mage's data loading clients that interface with popular data
storage systems, see docs on [Data loading](/design/data-loading).


# Dynamic blocks
Source: https://docs.mage.ai/design/blocks/dynamic-blocks

Use the output of a block to dynamically create more blocks.

[YouTube video player](https://www.youtube.com/embed/jQZb3lrAb8w?si=VF28Os4XbinyVOi8)

A dynamic block will create multiple downstream blocks at runtime.

The number of blocks it creates is equal to the number of items in the output data of the
dynamic block multiplied by the number of its downstream blocks.

For example, the following data loader block will output a list of 2 lists.
The 1st item in the list contains a list of 3 dictionaries containing user information.

```python theme={"system"}
from typing import Dict, List


@data_loader
def load_data(*args, **kwargs) -> List[List[Dict]]:
    users = []
    metadata = []

    for i in range(3):
        i += 1
        users.append(dict(id=i, name=f'user_{i}'))
        metadata.append(dict(block_uuid=f'for_user_{i}'))

    return [
        users,
        metadata, # optional
    ]
```

If the above data loader has 2 downstream blocks (e.g. 2 blocks that depend on this data loader),
then this dynamic block will create 6 total downstream blocks at runtime
(3 users multiplied by 2 downstream blocks).

> **Krichkorn Oparad**
>
> > This feature saves me when I need to securely transfer multiple media files from one location to another (SCP), especially when the files are spread across different directories and across the network. Typically, one would have to loop through each file, but secure copy (SCP) operations can fail for various reasons, and dealing with exceptions can be error-prone.
> >
> > An alternative approach would be to use an API trigger (like the one provided by Mage AI) to receive POST requests for each file that needs to be securely copied. However, this method can be highly resource-intensive, particularly when dealing with thousands of pipeline runs.
> >
> > Dynamic block, which allows me to manage source and destination specifications in a DataFrame and then dynamically execute SCP operations without having to handle exceptions manually (thanks to the "retry incomplete block" feature), is a game-changer. Not only does it save me from the hassle of exception handling, but it also keeps computational resources in check by avoiding unnecessary pipeline triggers.
> >
> > Thank you for your world-class project. Keep up the excellent work.
> >
> > ![](https://yt3.googleusercontent.com/ytc/AIdro_kcyWKgnvWDUDD6iOcxByIKb4E4zXJGYlT52Mtorrljo98f=s176-c-k-c0x00ffffff-no-rj)

### Data output shape of a dynamic block

A dynamic block must return a list of 2 lists of dictionaries
(e.g. `List[List[Dict]]`).

For example, a data loader block returns the following output:

```json theme={"system"}
[
    [
        {
            "id": 1,
            "name": "user_1"
        },
        {
            "id": 2,
            "name": "user_2"
        },
        {
            "id": 3,
            "name": "user_3"
        }
    ],
    [
        {
            "block_uuid": "for_user_1"
        },
        {
            "block_uuid": "for_user_2"
        },
        {
            "block_uuid": "for_user_3"
        }
    ]
]
```

#### Input data for downstream blocks

The 1st item in the list is a list of dictionaries.
These dictionaries contain data that will be passed down to all its downstream blocks.

```json theme={"system"}
[
    {
        "id": 1,
        "name": "user_1"
    },
    {
        "id": 2,
        "name": "user_2"
    },
    {
        "id": 3,
        "name": "user_3"
    }
]
```

The number of items in this list of dictionaries will correspond to how many downstream blocks get
created at runtime.

For example, if this dynamic block has a downstream transformer block with the following code:

```python theme={"system"}
from typing import Dict, List


@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
    data['id'] = int(data['id']) * 100
    return [data]
```

The return output data of this downstream transformer block will be:

```json theme={"system"}
[
    {
        "id": 100,
        "name": "user_1"
    }
]
```

> Note
>
> The transformer block is returning a list as its output data.
> The list of items is used as positional arguments in any downstream block of this transformer block.

#### Metadata for downstream blocks (optional)

The 2nd item in the list is a list of dictionaries.
These dictionaries contain metadata for each downstream block.

```json theme={"system"}
[
    {
        "block_uuid": "for_user_1"
    },
    {
        "block_uuid": "for_user_2"
    },
    {
        "block_uuid": "for_user_3"
    }
]
```

This metadata is used to uniquely identify and create each dynamically created downstream block.
The keys in each metadata dictionary are:

| Key | Description | Required |
| - | - | - |
| `block_uuid` | This value is used in combination with the downstream block’s original UUID to construct a unique UUID across all dynamic blocks. This value must be unique within the same list of metadata dictionaries. | Yes |

If the above transformer block UUID is `anonymize_user_data` and the metadata for the 1st
dynamically created block is `{ "block_uuid": "for_user_1" }`, then the dynamically created block’s
UUID is `anonymize_user_data:for_user_1`.

The convention is `[original_block_uuid]:[metadata_block_uuid]`.

## Dynamically created blocks

Every downstream block from a dynamic block is referred to as a dynamically created block.

The number of these blocks created are determined by the return output data of a dynamic block.

For example, if a dynamic block returns the following data:

```json theme={"system"}
[
    [
        {
            "id": 1,
            "name": "user_1"
        },
        {
            "id": 2,
            "name": "user_2"
        },
        {
            "id": 3,
            "name": "user_3"
        }
    ],
    [
        {
            "block_uuid": "for_user_1"
        },
        {
            "block_uuid": "for_user_2"
        },
        {
            "block_uuid": "for_user_3"
        }
    ]
]
```

Then there will be 3 dynamically created blocks for every downstream block of that dynamic block.

If the dynamic block had 1 downstream block, then 3 blocks will be dynamically created.

For example, if the downstream block was a transformer with UUID `anonymize_user_data`
and the following code:

```python theme={"system"}
from typing import Dict, List


@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
    data['id'] = int(data['id']) * 100
    return [data]
```

There will be 3 dynamically created blocks with the following UUID and return output data:

| Dynamic block UUID | Return output data |
| - | - |
| `anonymize_user_data:for_user_1` | `[{ "id": 100, "name": "user_1" }]` |
| `anonymize_user_data:for_user_2` | `[{ "id": 200, "name": "user_2" }]` |
| `anonymize_user_data:for_user_3` | `[{ "id": 300, "name": "user_3" }]` |

### Downstream blocks of dynamically created blocks

If a dynamically created block has downstream blocks,
those downstream blocks will be created multiple times. The number of times those are created
correspond to how many blocks were created dynamically from the dynamic block.

In the current example, 3 blocks were created dynamically
(3 users, 1 downstream block from the dynamic block).
If the above transformer block has 2 downstream blocks, then 6 more blocks will be created.

For example, if the transformer block `anonymize_user_data` has 2 downstream blocks with UUID
`clean_column_names` and `compute_engagement_score`, the following blocks will also be
dynamically created:

1. `anonymize_user_data:for_user_1`

```
→ clean_column_names:for_user_1
→ compute_engagement_score:for_user_1
```

2. `anonymize_user_data:for_user_2`

```
→ clean_column_names:for_user_2
→ compute_engagement_score:for_user_2
```

3. `anonymize_user_data:for_user_3`

```
→ clean_column_names:for_user_3
→ compute_engagement_score:for_user_3
```

The data that is passed into each of these downstream blocks
(e.g. `clean_column_names`, `compute_engagement_score`) will come from its upstream block
(e.g. `anonymize_user_data:for_user_1`, `anonymize_user_data:for_user_2`, `anonymize_user_data:for_user_3`).

For example, the block `clean_column_names:for_user_1` and `compute_engagement_score:for_user_1`
will have input data with the following shape and value:

```json theme={"system"}
{
    "id": 100,
    "name": "user_1"
}
```

### Reduce output

By default, dynamically created blocks will create more dynamically created blocks from their own
downstream blocks.
A dynamically created block will pass its return output data to its downstream blocks and so on.

However, a dynamically created block can be configured to reduce all the outputs across each
dynamically created block and combine them into a single list of items.

If we configure the above transformer block `anonymize_user_data` to reduce its output, then
there will only be 2 downstream blocks instead of 6.

The 2 downstream blocks will be `clean_column_names` and `compute_engagement_score`.
The input data to each of these 2 downstream blocks will be:

```json theme={"system"}
[
    {
        "id": 100,
        "name": "user_1"
    },
    {
        "id": 200,
        "name": "user_2"
    },
    {
        "id": 300,
        "name": "user_3"
    }
]
```

***

## Dynamic SQL blocks

SQL blocks can be dynamic or be a child of a dynamic block.

Here is an example pipeline with code:

```
    [data_loader]
          ↓
 [dynamic_sql_loader]
          ↓
    [transformer]
```

**`data_loader.py`**: dynamic block

```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
    return [[dict(uuid=1), dict(uuid=2)]]
```

**`dynamic_sql_loader`**: dynamic block, dynamic child

```sql theme={"system"}
SELECT * FROM {{ df_1 }}
UNION ALL
SELECT 2 AS uuid
```

**`transformer`**: dynamic child

```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
    return data
```

When this pipeline is triggered, the following block runs will be created:

```
                                [data_loader]
                                 ↓         ↓
              [dynamic_sql_loader:0]     [dynamic_sql_loader:1]
                ↓                ↓         ↓                ↓
 [transformer:0:0]  [transformer:0:1]   [transformer:1:0]  [transformer:1:1]
```

***

## Tutorial guide

Follow this [guide](/guides/blocks/dynamic-blocks) to create a simple pipeline with a dynamic block,
dynamically created blocks, and a block that reduces its output.


# Remote blocks
Source: https://docs.mage.ai/design/blocks/remote-blocks

Use outputs from blocks in another pipeline run as inputs to downstream blocks.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=remote-blocks)

## Overview

Remote blocks let one pipeline pass references to block outputs from another pipeline run.
Mage stores a reference to the source block, source pipeline, source repository path, and execution
partition, then resolves that reference when a downstream block runs.

Remote blocks are used by [global data products](/extensibility/global-data-products/overview) and
can also be returned when triggering a pipeline from code.

Use remote blocks when a downstream pipeline needs data that was already produced by an upstream
pipeline run. This avoids copying the data into runtime variables and lets Mage read the saved block
variables from the source pipeline partition.

## How it works

A remote block reference contains:

| Field | Description |
| - | - |
| `block_uuid` | The block whose saved output variables should be read. |
| `pipeline_uuid` | The pipeline that owns the block. |
| `repo_path` | The repository path for the workspace that owns the source pipeline. Mage Pro supports one project repository per workspace. |
| `execution_partition` | The partition for the source pipeline run. Mage reads the block variables from this partition. |

When the receiving block executes, Mage replaces each reference with that remote block's saved
outputs and makes them available in `kwargs['remote_blocks']`.

The resulting shape is a list of remote block output groups:

```python theme={"system"}
[
    [
        output_from_first_remote_block,
        another_output_from_first_remote_block,
    ],
    [
        output_from_second_remote_block,
    ],
]
```

Each inner list contains the saved output variables from one referenced block.

## Use with global data products

When a pipeline uses a global data product block, Mage triggers or reuses the source pipeline run
for that data product. After the source run is available, Mage returns remote block references for
the configured output blocks in that global data product.

Downstream blocks can read those outputs from `kwargs['remote_blocks']`:

```python theme={"system"}
import pandas as pd


@transformer
def combine_remote_outputs(*args, **kwargs):
    remote_blocks = kwargs.get('remote_blocks') or []
    frames = []

    for remote_block_outputs in remote_blocks:
        for output in remote_block_outputs:
            if output is not None:
                frames.append(output)

    if not frames:
        return pd.DataFrame()

    return pd.concat(frames, ignore_index=True)
```

If the source block returns more than one value, each value appears in the inner list for that
remote block.

## Pass remote blocks to another pipeline

Remote blocks do not require a global data product. If the current pipeline has already produced
outputs that another pipeline should consume, build remote block references from the current
pipeline context and pass them to `trigger_pipeline`.

For example, this data exporter triggers another pipeline and passes outputs from five blocks in
the current pipeline run:

```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline


if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def trigger_downstream_pipeline(*args, **kwargs):
    current_pipeline_uuid = kwargs['pipeline_uuid']
    execution_partition = kwargs.get('execution_partition')

    remote_blocks = [
        {
            'block_uuid': 'load_orders',
            'pipeline_uuid': current_pipeline_uuid,
            'execution_partition': execution_partition,
        },
        {
            'block_uuid': 'summarize_orders',
            'pipeline_uuid': current_pipeline_uuid,
            'execution_partition': execution_partition,
        },
        {
            'block_uuid': 'load_customers',
            'pipeline_uuid': current_pipeline_uuid,
            'execution_partition': execution_partition,
        },
        {
            'block_uuid': 'load_products',
            'pipeline_uuid': current_pipeline_uuid,
            'execution_partition': execution_partition,
        },
        {
            'block_uuid': 'build_order_details',
            'pipeline_uuid': current_pipeline_uuid,
            'execution_partition': execution_partition,
        },
    ]

    trigger_pipeline(
        'downstream_pipeline_uuid',
        variables={},
        check_status=False,
        error_on_failure=False,
        poll_interval=60,
        poll_timeout=None,
        verbose=True,
        remote_blocks=remote_blocks,
    )
```

In the downstream pipeline, add a block that reads the resolved outputs from
`kwargs['remote_blocks']`. The order matches the order of the references passed to
`trigger_pipeline`.

```python theme={"system"}
if 'custom' not in globals():
    from mage_ai.data_preparation.decorators import custom


@custom
def read_remote_outputs(*args, **kwargs):
    remote_blocks = kwargs.get('remote_blocks')
    if not remote_blocks:
        return {}

    (
        load_orders,
        summarize_orders,
        load_customers,
        load_products,
        build_order_details,
    ) = remote_blocks

    return {
        'orders': load_orders[0],
        'order_summary': summarize_orders[0],
        'customers': load_customers[0],
        'products': load_products[0],
        'order_details': build_order_details[0],
    }
```

Each item in `remote_blocks` is a list because a source block can save multiple output variables.
Use index `0` when the source block returns a single output.

## Trigger a pipeline and return remote blocks

You can also request remote block references when triggering a pipeline from code:

```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline


remote_block_payload = trigger_pipeline(
    'source_pipeline_uuid',
    check_status=True,
    error_on_failure=True,
    return_remote_blocks=True,
)
```

When `return_remote_blocks=True`, Mage returns a payload containing `remote_blocks` for the blocks in
the triggered pipeline run. Pass that payload into another pipeline trigger to make those outputs
available to the downstream pipeline:

```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline


remote_block_payload = trigger_pipeline(
    'source_pipeline_uuid',
    check_status=True,
    error_on_failure=True,
    return_remote_blocks=True,
)

trigger_pipeline(
    'downstream_pipeline_uuid',
    check_status=True,
    error_on_failure=True,
    remote_blocks=remote_block_payload[0]['remote_blocks'],
)
```

In blocks inside `downstream_pipeline_uuid`, read the resolved outputs from
`kwargs['remote_blocks']`.

## Notes and limitations

* Remote blocks read saved block variables. The source block must have completed and saved outputs
  for the referenced execution partition.
* `pipeline_uuid` and `execution_partition` are not filled in automatically when you build remote
  block references manually. Use the current block context, such as `kwargs['pipeline_uuid']` and
  `kwargs.get('execution_partition')`, when passing outputs from the current pipeline run.
* `execution_partition` pins the downstream read to the source pipeline run that produced the
  referenced outputs. If it is omitted, Mage reads from the unpartitioned variable path.
* Remote blocks pass data into block code through keyword arguments, not positional upstream
  arguments.
* Global data product blocks pass remote block references while the global data product block
  itself is executing. Downstream blocks receive the resolved outputs.
* The `partitions` setting for a global data product controls how many completed source run
  partitions are included when reading outputs through the global data product API. Remote block
  references point to the source run partition returned by the trigger path.


# Scratchpad
Source: https://docs.mage.ai/design/blocks/scratchpad

Use these blocks to experiment and write throw away code.

Scratchpad blocks aren't used when executing a pipeline.

![Blocks](https://mage-ai.github.io/assets/blocks/scratchpad.gif)

## Getting data from other blocks

If you want to read the output data from another block, use the following sample code:

```python theme={"system"}
from mage_ai.data_preparation.variable_manager import get_variable


df = get_variable('pipelined_uuid', 'block_uuid', 'output_0')
```

* Change the value of `pipelined_uuid` to match your current pipeline UUID.
* Change the value of `block_uuid` to match the UUID of the block you want to read data from.


# Sensor
Source: https://docs.mage.ai/design/blocks/sensor

A sensor is a block that continuously evaluates a condition until it’s met or until a period of time has elapsed.

![Python mecha](https://mage-ai.github.io/assets/blocks/grandwizard_a_big_mythical_spirit_like_eye_scanning_a_bunch_of__b030824d-801c-4f8d-a9bd-51f0681eaa78.png)

In your pipeline, you can add sensors that run continuously and only complete
when an external pipeline or external block is successfully ran for a particular
partition.

Then, you can add a block in your pipeline that depends on that sensor.

If there is a block with a sensor as an upstream dependency, that block won’t start
running until the sensor has evaluated its condition successfully.

***

## Setup

1. [Add a sensor to your pipeline](/design/blocks/sensor)
2. [Configuring a sensor](/design/core-abstractions#sensor)

***

## Example

You can add sensors to a pipeline the same way you add other types of blocks.

```python theme={"system"}
from mage_ai.orchestration.run_status_checker import check_status


@sensor
def check_condition(*args, **kwargs) -> bool:
    return check_status(
        'pipeline_uuid',
        kwargs['execution_date'],
        block_uuid='block_uuid',
        hours=24,
    )
```

In the above code, change `pipeline_uuid` to the actual pipeline UUID you want this sensor to keep
checking whether or not it’s finished running successfully.

The `block_uuid` positional argument is optional. If you add a value for the `block_uuid`
positional argument, then the sensor will check the status of the block in that pipeline.
Only when that specific block is finished running successfully, then the sensor will be complete.

The 'hours' positional argument is optional. If you add a value for the 'hours' positional argument,
then the sensor will check the status of the pipeline for that many hours in past from execution\_date.
The default value is 24 hours.

Sensors can also have upstream block dependencies. The output of the upstream blocks will be passed
into the sensor in the `args` parameter like in other Mage block types.

Currently, the sensor makes a check every 60 seconds.

## Templates

When adding a sensor block, here are the available templates to choose from:

1. Google BigQuery
2. MySQL
3. PostgreSQL
4. Amazon Redshift
5. Amazon S3
6. Snowflake

Below are the code examples in those templates.

### Google BigQuery

Check the results of a BigQuery query.

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path


@sensor
def query_bigquery_and_check_condition(*args, **kwargs) -> bool:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your BigQuery query'  # Specify your SQL query here

    loader = BigQuery.with_config(ConfigFileLoader(config_path, config_profile))
    df = loader.load(query)

    # Add your checks here
    if df.empty:
        return False

    return True
```

### MySQL

Check the results of a MySQL query.

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path


@sensor
def query_mysql_and_check_condition(*args, **kwargs) -> bool:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your MySQL query'  # Specify your SQL query here

    with MySQL.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```

### PostgreSQL

Check the results of a PostgreSQL query.

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path


@sensor
def query_postgres_and_check_condition(*args, **kwargs) -> bool:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your Postgres query'  # Specify your SQL query here

    with Postgres.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```

### Amazon Redshift

Check the results of a Redshift query.

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path


@sensor
def query_redshift_and_check_condition(*args, **kwargs) -> bool:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your Redshift query'  # Specify your SQL query here

    with Redshift.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```

### Amazon S3

Check if a file or folder exists in a S3 bucket.

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path


@sensor
def check_condition(*args, **kwargs) -> bool:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    s3_path = 'path/to/folder/or/file'

    config_file_loader = ConfigFileLoader(config_path, config_profile)
    return S3.with_config(config_file_loader).exists(
        bucket_name, s3_path
    )
```

### Snowflake

Check the results of a Snowflake query.

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path


@sensor
def query_snowflake_and_check_condition(*args, **kwargs) -> bool:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your Snowflake query'  # Specify your SQL query here

    with Snowflake.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```


# Transformer
Source: https://docs.mage.ai/design/blocks/transformer

Use these blocks to clean, transform, and enhance data from other blocks.

## Example

```python theme={"system"}
from pandas import DataFrame


@transformer
def transform_df(df: DataFrame, *args, **kwargs) -> DataFrame:
    return df.iloc[:10]
```

## How data is received from upstream blocks

All blocks (except Scratchpads and Sensors) pass their data from the return statement in their
decorated function to all their downstream blocks.

In order to access the data from an upstream block within a current block,
use the positional arguments of the decorated function.

For example, in the above function named `transform_df`, the output data from its upstream block
is accessed by the positional argument named `df`.

## Getting data from multiple upstream blocks

If a block has more than 1 upstream block,
then each upstream block’s output is accessible via positional arguments.

For example, if you have 3 upstream blocks,
then there will be 3 positional arguments available within the `transform_df` function.

See below for an example:

```python theme={"system"}
from pandas import DataFrame
import pandas as pd


@transformer
def transform_df(df1, df2, df3, **kwargs) -> DataFrame:
    return pd.concat([df1, df2, df3])
```

## Handling column renames in data integration pipeline

Since data passed to transform blocks is a pandas DataFrame, many operations are available on each row of your data.

It's possible to do a few dangerous things with this, for example if you have a target destination that needs it's columns renamed to match existing schemas.

In this case, there can be edge case mismatches between the data types in the source columns, and the expected data types in the destination. This is particularly important for SQL columns where the INSERT commands generated by the pipelines will use CAST expressions to try and make sure the writes succeed. However, if a column is renamed in the transformer, we will use our best guess by looking at values to decide what to CAST the column values to.

For example, a TIMESTAMPTZ value of `2023-01-09T20:01:00`, once it's renamed, can look like a string, but using an INSERT with `CAST('2023-01-09T20:01:00' AS TEXT)` to a column with data type `TIMESTAMPTZ` will fail.

### Using the `rename_columns` decorator parameter (Recommended)

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=transformer-rename-columns)

The recommended way to rename columns in data integration pipelines is to use the `rename_columns` parameter in the `@transformer` decorator. This approach automatically preserves original column types and updates schema metadata including `unique_constraints` and `key_properties`.

**Benefits:**

* **Type preservation**: Original column types are maintained after renaming
* **Schema integrity**: `unique_constraints` and `key_properties` are automatically updated
* **Declarative**: Column renaming is specified at the decorator level, making intent clear
* **Only renames existing columns**: Columns that don't exist in the DataFrame are silently skipped

**Example:**

```python theme={"system"}
from pandas import DataFrame

@transformer(rename_columns={
    'impressions': 'field_5627',
    'clicks': 'field_5628',
    'new_free_signups': 'field_5629',
    'new_paid_signups': 'field_5626',
    'event_timestamp': 'field_5630'
})
def transform(data, *args, **kwargs):
    # Your transformation logic here
    # Columns are automatically renamed after this function executes
    data.insert(0, 'trashed', False)
    return data
```

The `rename_columns` parameter accepts a dictionary where:

* **Keys** are the original column names (must exist in the DataFrame)
* **Values** are the new column names

**Note:** If a column in `rename_columns` doesn't exist in the DataFrame, it will be silently skipped. All target column names must be unique.

### Using manual column renaming with aliases (Legacy approach)

If you need more control or are using an older version, you can manually rename columns in your transformer function and add column "aliases" in the pipeline's data integration catalog.

1. Have a transformer block in your integration pipeline that does some column renaming:

```python theme={"system"}

from pandas import DataFrame
import pandas as pd

@transformer
def transform(data, *args, **kwargs):
    # keys = source column names, values = destination column names
    COLUMN_MAP = {
        'impressions': 'field_5627', 
        'clicks': 'field_5628', 
        'new_free_signups': 'field_5629',
        'new_paid_signups': 'field_5626', 
        'event_timestamp': 'field_5630'
    }
    # the pandas rename in place to match destination schema
    data.rename(columns=COLUMN_MAP, inplace=True)
    data.insert(0, 'trashed', False)
    return data

```

2. Add 'aliases' property to each renamed field in your pipeline's `data_integration_catalog.json` file.

```json theme={"system"}

{
    "catalog": {
        "streams": [
            {
                "tap_stream_id": "marketing_data",
                "replication_method": "INCREMENTAL",
                "key_properties": [
                    "id"
                ],
                "schema": {
                    "properties": {
                        "id": {
                            "type": [
                                "integer"
                            ]
                        },
                        "impressions": {
                            "type": [
                                "null",
                                "number"
                            ],
                            "aliases": [
                                "field_5627"
                            ]
                        },
                        "clicks": {
                            "type": [
                                "null",
                                "number"
                            ],
                            "aliases": [
                                "field_5628"
                            ]
                        },
                        "new_free_signups": {
                            "type": [
                                "null",
                                "number"
                            ],
                            "aliases": [
                                "field_5629"
                            ]
                        },
                        "new_paid_signups": {
                            "type": [
                                "null",
                                "number"
                            ],
                            "aliases": [
                                "field_5626"
                            ]
                        },
                        "event_timestamp": {
                            "format": "date-time",
                            "type": [
                                "null",
                                "string"
                            ],
                            "aliases": [
                                "field_5630"
                            ]
                        }
                    },
                    "type": "object"
                },
                "stream": "marketing_data",
                "metadata": [...],
                ...
            }
        ]
    }
}

```

Now, pipeline runs will use the correct data types for INSERT commands, thanks to transformer column aliases defined in the schema.

**Note:** When using the `rename_columns` decorator parameter, you don't need to manually add aliases to the catalog - the schema metadata is automatically updated. The manual approach with aliases is only needed when renaming columns inside the transformer function code.


# Transformer blocks
Source: https://docs.mage.ai/design/blocks/transformer-blocks

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

## Core Concepts

The core object of any transformer action is the **payload**, a JSON object that describes the transformation to perform alongside settings for that transformation. A sample transformer action payload is shown below:

```json theme={"system"}
{
    "action_type": "impute",
    "action_arguments": ["age", "name"],
    "action_options": {
        "strategy": "constant",
        "value": "32"
    },
    "axis": "column",
    "action_variables": {
        "age": {
            "uuid": "age",
            "column_type": "number"
        },
        "name": {
            "uuid": "hourly_salary",
            "column_type": "text"
        }
    }
}
```

The above transformer action imputes missing values in the "age" and "hourly\_salary" columns with the constant value `32`. Here is a list of the keys used in transformer action payloads:

| Key | Description |
| - | - |
| axis | Axis to perform the transformation on. Either 'row' or 'column'. |
| action\_type | Type of action to perform. See [Transformer Action Reference](#transformer-action-reference) for the list of action types. |
| action\_code | Text field for specifying any code such as filters. |
| action\_arguments | The columns to perform the transformation on. |
| action\_options | Other settings to provide to the transformer action. See [Transformer Action Reference](#transformer-action-reference) for specific details on which actions require which options. |
| action\_variables | Variable metadata for each input argument (column). Specifies column name and type. |
| outputs | Variable metadata for each output column. Specifies column name and type. See [Column Types](#appendix-column-types) for information on possible column types to specify. |

This JSON formatted payload is difficult to generate so it is recommended to use the `build_transformer_action` factory to generate the payload. The above snippet generates the same payload as above:

```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action

df = pd.DataFrame({...}) # your input data frame
payload = build_transformer_action(
    df,
    action_type=ActionType.IMPUTE,
    arguments=['age'],
    axis=Axis.COLUMN,
    options={'strategy': 'constants', 'values': 32},
)
```

See [Building Transformer Actions](#building-transformer-actions) for detailed information on using `build_transformer_action`.

The transformer action payload, once built, can be used to construct a `BaseAction` object. This object processes the payload to identify the transformations to apply a data frame. Through this action object, the transformer action can be applied to a data frame using the `execute` function:

```python theme={"system"}
payload = build_transformer_action(...)
transformed_df = BaseAction(payload).execute(df)
```

# Building Transformer Actions

***build\_transformer\_action*** - `build_transformer_action(df: DataFrame, action_type: ActionType | str, arguments: List[str], action_code: str, options: Dict, axis: Axis | str, outputs: List[Dict]) -> Dict:`

Source: `mage_ai.data_cleaner.transformer_actions.utils.build_transformer_action`

Builds transformer action payload from arguments. The output of this function can be passed
as input to `transformer_actions.base.BaseAction` in order to perform the requested transformation.

Note: Action variables are inferred from `arguments` and `df`.

* **Args**:
  * `df (DataFrame)`: The data frame to build a transformer action payload for.
  * `action_type (ActionType | str)`: Transformer action to perform.
  * `arguments (List[str], optional)`: Columns to perform this action on. Defaults to `[]`.
  * `action_code (str, optional)`: Special code or query to execute with action. Defaults to `''`.
  * `options (Dict, optional)`: Options specifying behavior of action. Defaults to `{}`.
  * `axis (Union[Axis, str], optional)`: Axis of the data frame to apply the action to. Defaults to `Axis.COLUMN`.
  * `outputs (List[Dict], optional)`: Specifies metadata of newly created columns. Defaults to `[]`.
* **Returns**: (`Dict`) Transformer action payload to use with `transformer_actions.base.BaseAction`
* **Example**:
  ```python theme={"system"}
    action = build_transformer_action(
        df,
        action_type=ActionType.FIRST,
        action_code='retailer != null and retail_price >= 0',  # Enter filtering condition on rows before aggregation
        arguments=['retail_price', 'review'],  # Enter the columns to compute aggregate over
        axis=Axis.COLUMN,
        options={'groupby_columns': ['retailer']},  # Enter columns to group by
        outputs=[
            # The number of outputs below must match the number of arguments
            {'uuid': 'first_retail_price', 'column_type': 'number_with_decimals'},
            {'uuid': 'first_review', 'column_type': 'category'},
        ],
    )
  ```

# Transformer Action Reference

* [Building Transformer Actions](#building-transformer-actions)
* [Transformer Action Reference](#transformer-action-reference)
* [Column Actions](#column-actions)
  * [Aggregation Actions](#aggregation-actions)
  * [Formatting Actions](#formatting-actions)
    * [Clean Column Names](#clean-column-names)
    * [Fix Syntax Errors](#fix-syntax-errors)
    * [Reformat Values](#reformat-values)
  * [Column Removal Actions](#column-removal-actions)
    * [Select Columns](#select-columns)
    * [Remove Columns](#remove-columns)
  * [Row Shifting Actions](#row-shifting-actions)
    * [Shift Up](#shift-up)
    * [Shift Down](#shift-down)
  * [Feature Scaling Actions](#feature-scaling-actions)
    * [Normalize Data](#normalize-data)
    * [Standardize Data](#standardize-data)
  * [Other Column Actions](#other-column-actions)
    * [Difference](#difference)
    * [Fill In Missing Values](#fill-in-missing-values)
    * [Remove Outliers](#remove-outliers)
* [Row Actions](#row-actions)
  * [Drop Duplicates](#drop-duplicates)
  * [Filter](#filter)
  * [Remove Rows](#remove-rows)
  * [Sort](#sort)
* [Appendix: Filter Syntax](#appendix-filter-syntax)
* [Appendix: Column Types](#appendix-column-types)

# Column Actions

## Aggregation Actions

Applies some aggregation function over groups determined by values in some columns. A set of grouping columns are provided, whose unique combinations of values are used to form groups. For example, consider the dataset below:

| Paid Dues | Due Amount |
| - | - |
| Yes | \$5.00 |
| No | \$15.20 |
| Yes | \$3.50 |
| Yes | \$1.28 |
| No | \$25.03 |

The data could be grouped by "Paid Dues" to form the following groups:

| Paid Dues | Due Amount |
| - | - |
| Yes | \$5.00 |
| Yes | \$3.50 |
| Yes | \$1.28 |

| Paid Dues | Due Amount |
| - | - |
| No | \$15.20 |
| No | \$25.03 |

Then an aggregation function is applied to each group over the input columns. In the example, we can compute the total dues by group by aggregating over "Due Amount":

| Paid Dues | Total Due Amount |
| - | - |
| Yes | \$9.78 |
| No | \$40.23 |

Then the output of this aggregation is used to create a new column for each input column, where each row receives the corresponding aggregate of the group it falls into. In the example above, the final output data frame would be:

| Paid Dues | Due Amount | Total Due Amount by Payment Status |
| - | - | - |
| Yes | \$5.00 | \$9.78 |
| No | \$15.20 | \$40.23 |
| Yes | \$3.50 | \$9.78 |
| Yes | \$1.28 | \$9.78 |
| No | \$25.03 | \$40.23 |

All of these transformer actions have the same set of arguments when using `build_transformer_action`, but have different types corresponding to the different aggregation functions that could be applied to the groups.

**Example**

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.AVERAGE,
    action_code='brand != null',
    arguments=['revenue', 'cost'],
    axis=Axis.COLUMN,
    options={'groupby_columns': ['brand']},
    outputs=[
        {'uuid': 'avg_rev_per_brand', 'column_type': 'number_with_decimals'},
        {'uuid': 'avg_cost_per_brand', 'column_type': 'number_with_decimals'},
    ],
)
```

**Args**

* ***action\_code:*** Filter for selecting rows before aggregation is performed. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.
* ***arguments:*** Columns to compute aggregate values over. For each argument a new column will be created to store the aggregates values over the column.
  * If no arguments are provided, no aggregation is performed
* ***options*:**
  * `groupby_columns` (optional) - list of columns to form groups with when aggregating. A group consists of each unique combination of values chosen from each of the columns specified in this option.
    * If this is empty, aggregation is performed over the first column specified in *arguments* (only if *arguments* is nonempty - if empty no aggregation is performed) and only one column is added

```
The following options are used as a shortcut to simplify filtering over time windows.
Let $A$ and $B$ be datetime columns, and let $\text{window}$ be an integer describing the length of
a time window in seconds. If the following options are specified, rows for which the time value for
$B \in [A, A + \text{window}]$ are selected (All rows where value for $B$ must (a) be greater than
or equal to the value for $A$ and (b) less than or equal to the value $A+window$).
```

* `timestamp_feature_a` (optional) - time feature A (a column in the data frame)
* `timestamp_feature_b` (optional) - time feature B (a column in the data frame)
* `window` (optional) - max time window between feature A and feature B in seconds

If some (or none) of these options are specified, then the filtering is not performed. Regardless if this filtering is performed, *action\_code* filtering is still performed.

* ***outputs*:** Specifies the metadata for each output column of the aggregation (corresponding to the aggregation created from input argument in *arguments*). This array must be the same length as *arguments*, and each entry must contain the following metadata:
  * `uuid (string)`: Name of the new column
  * `column_type (string | ColumnType)`: Type of the new column ( See `mage_ai.data_cleaner.column_types.constants.ColumnType` for options)

**Aggregation Functions**
These are the possible aggregations that can be applied to each group per each input column. To use the aggregation function specify the corresponding action type in `build_transformer_action`.

| Name | Action Type Enum | Description |
| - | - | - |
| ***Average*** | `ActionType.AVERAGE` | Averages input column value over each group |
| ***Count Distinct*** | `ActionType.COUNT_DISTINCT` | Counts the number of distinct rows in each group per input column |
| ***Count*** | `ActionType.COUNT` | Counts the number of rows in each group per input column |
| ***First*** | `ActionType.FIRST` | For each input column, selects first item in each group (as appearing in the order of the original data frame) |
| ***Last*** | `ActionType.LAST` | For each input column, selects last item in each group (as appearing in the order of the original data frame) |
| ***Maximum*** | `ActionType.MAX` | Gets the maximum value per group for each input column |
| ***Median*** | `ActionType.MEDIAN` | Gets the median value per group for each input column |
| ***Minimum*** | `ActionType.MIN` | Gets the maximum value per group for each input column |
| ***Sum*** | `ActionType.SUM` | Sums over all values in each group per input column |

## Formatting Actions

These transformer actions involve reformatting column names or the values in columns

### Clean Column Names

Cleans column names according to the following rules:

1. Names are converted to snake case
2. All wrapping whitespace or underscores are removed
3. All non-alphanumeric characters (except "\_") are removed
4. Numbers are prefixed with "number\_"
5. All Python keywords are postfixed by "\_"

Snake case is chosen as

1. all characters are lower case
2. no whitespace is used (which makes referring to columns in code easier)

| Original Name | Cleaned Name | Notes |
| - | - | - |
| good\_name | good\_name | columns in snake case are left unchanged |
| bad case | bad\_case | |
| %@#342%34@@#342 | number\_34234342 | Symbols are removed first, then number is cleaned |
| PascalCaseColumn | pascal\_case\_column | |
| camelCaseText | camel\_case\_text | |
| \_\_\_snake\_case\_\_\_ | snake\_case | Surrounding underscores are stripped |

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.CLEAN_COLUMN_NAME,
    arguments=df.columns,
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** Columns whose name to clean. If empty, no columns names are cleaned.

### Fix Syntax Errors

Marks syntax errors in column values. Syntax errors are defined as values that are improperly formatted or of the incorrect type. For example:

* A number in a text column
* An improperly formatted email, phone number, or zip code
* An improperly formatted date

Values that break these syntax errors are currently marked as:

* `"invalid"` if column is of categorical or string type
* `pandas.NaT` if column is a datetime
* `np.nan` if column is a number type

*WIP*: Intelligently fix syntax errors in these columns automatically

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.FIX_SYNTAX_ERRORS,
    arguments=df.columns,
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** Columns to mark syntax errors for. If empty no columns are checked.

### Reformat Values

Reformats values in column based on requested action. The currently supported reformats are:

* Standardize capitalization (`reformat = 'caps_standardization'`): Forces text column to follow a single capitalization strategy (either lowercase or uppercase)
* Convert currencies to a number (`reformat = 'currency_to_num'`): Converts a currency value (by default stored as a string) to a decimal number. If unable to convert, doesn't perform conversion
  * Supported currencies: **\$, CAD, £, €, ¥, Rs, 元**
    | Original String | Parsed Decimal | Notes |
    | - | - | - |
    | "  \$ 10000" | 10000 | |
    | "- ¥ 22.324523" | -22.324523 | Negation in front of symbol supported |
    | " 10000 元   " | 10000 | Currencies with postfixed symbols are supported |
    | "0.42 €" | 0.42 | |
    | "-  3.42032 CAD" | -3.42032 | |
    | "Rs - 100000.23" | -100000.23 | Negation after symbol is supported |
* Convert string to datetime (`reformat = 'date_time_conversion'`): Converts a string value representing a datetime to a `pandas.Timestamp` object if correctly formatted, else converts to `None`.
* Trim whitespace (`reformat = 'trim'`): Removes leading and trailing whitespace from text columns.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REFORMAT,
    arguments=[],
    axis=Axis.COLUMN,
    options={'reformat': 'caps_standardization', 'capitalization': 'lowercase'},
)
```

**Args**

* ***arguments:*** Columns to apply the reformat action to
* ***options:***
  * `reformat` (optional): The reformat action to apply to the specified columns. Can be:
    * `None` - no reformat is applied
    * `'caps_standardization'` - standardize capitalization
    * `'currency_to_num'` - convert currency string to number
    * `'date_format_conversion'` - convert datetime string to `pandas.Timestamp`
    * `'trim'` - remove leading and trailing whitespace from text columns
  * `capitalization` (optional): Specifies the capitalization strategy to use when standardizing capitalization. This argument is ignored unless `reformat = "caps_standardization"`. Options are `['lowercase', 'uppercase']`.

## Column Removal Actions

### Select Columns

Keeps only the specified columns; removes all other columns from data frame.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SELECT,
    arguments=[],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to keep in the data frame. If empty, all columns are removed.

### Remove Columns

Drops the specified columns.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REMOVE,
    arguments=[],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to remove in the data frame. If empty, no columns are removed.

## Row Shifting Actions

These actions copy a column with all rows shifted up or down by a certain amount. When rows are shifted, one or more values are deleted and some new values have to be filled in. The fill-in data types are:

| Type | Fill Value |
| - | - |
| Numeric | `np.nan` |
| Datetime | `pandas.NaT` |
| Other Types | `None` |

### Shift Up

Shifts all rows in a column up by one. The first entry is deleted and the last entry is turned to a null type.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SHIFT_UP,
    arguments=['Name'],
    axis=Axis.COLUMN,
    outputs=[{'uuid': 'shifted_name', 'type': 'text'}],
)
```

**Args**

* ***arguments:*** The column to shift up. At most one column can be shifted at once:
  * If multiple columns are provided as arguments, only the first is shifted up
  * If no columns are provided then no shifting is performed
* ***outputs:*** Metadata for the newly added upshifted column. Must have at most a single entry containing the following information:
  * `uuid`: Name of the new upshifted column
  * `type`: Data type of the new upshifted column

### Shift Down

Shifts all rows up by *periods* rows. The last *periods* entries are deleted and the first *periods* are converted to a null type.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SHIFT_DOWN,
    arguments=['created_time'],
    axis=Axis.COLUMN,
    options={'periods': 3},
    outputs=[{'uuid': 'downshifted_created_time', 'type': 'datetime'}],
)
```

**Args**

* ***arguments:*** The column to shift down. At most one column can be shifted at once:
  * If multiple columns are provided as arguments, only the first is shifted down
  * If no columns are provided then no shifting is performed
* ***options:***
  * `periods` (optional): Specifies how many rows to shift down. Defaults to 1.
  * `groupby_columns` (optional): Specifies columns to group by. If specified, shifts each *group* by *periods* rows instead of the entire data frame. The resulting column is then returned. If unspecified no grouping is performed. Multiple columns can be specified to group by.
* ***outputs:*** Metadata for the newly added downshifted column. Must have at most a single entry containing the following information:
  * `uuid`: Name of the new downshifted column
  * `type`: Data type of the new downshifted column

## Feature Scaling Actions

These actions are used to normalize the range of independent variables or features of data. These actions are  generally performed during the data preprocessing step.

| Type | Formula |
| - | - |
| Normalization | `(X-Xmin)/(Xmax-Xmin)` |
| Standardization | `(X-Xmean)/Xstd` |

### Normalize Data

Perform Normalization on the desired columns of a dataset.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.NORMALIZE,
    arguments=['Column1','Column2'],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to Normalize.
  * If no columns are provided then no Normalization is performed

### Standardize Data

Perform Standardization on the desired columns of a dataset.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.STANDARDIZE,
    arguments=['Column1','Column2'],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to Standardize.
  * If no columns are provided then no Standardization is performed

## Other Column Actions

### Difference

Adds a new column where each row is the difference between every consecutive two rows in the original column.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.DIFF,
    arguments=['position'],
    axis=Axis.COLUMN,
    outputs=[{'uuid': 'delta_position', 'column_type': 'number_with_decimals'}],
)
```

**Args**

* ***arguments:*** The column to calculate difference on. At most one column can be specified:
  * If multiple columns are provided as arguments, only the first is used in calculated
  * If no columns are provided then no calculation
* ***outputs:*** Metadata for the newly added difference column. Must have at most a single entry containing the following information:
  * `uuid`: Name of the difference column
  * `type`: Data type of the difference column (most often will be a number)

### Fill In Missing Values

Imputes missing values using different strategies:

| Strategy | Description |
| - | - |
| Average | Fill missing values with the average per column. |
| Column | Fill missing values with values from another column. |
| Constant | Fill missing values with a placeholder constant. Can fill with a specific value; otherwise a default placeholder is chosen by column type. |
| Median | Fill missing values with the median per column. |
| Mode | Fill missing values with the most frequent value per column. |
| Random | Fill missing values with a randomly sampled non-null value from same column. |
| Sequential | Fill missing values with the previous row. Should only be used with timeseries columns. |

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.IMPUTE,
    arguments=df.columns,  # Specify columns to impute
    axis=Axis.COLUMN,
    options={'strategy': ImputationStrategy.CONSTANT},  # Specify imputation strategy
)
```

**Args**

* ***arguments:*** The columns to impute (fill in the missing values).
* ***options:***
  * `strategy` (required): Specifies strategy to use. See table above or enum `ImputationStrategy`
  * `value` (optional): Value to fill in missing value with. This argument is ignored unless `strategy = 'constant'`, in which case the constant value used to impute is `value`.
  * `timeseries_cols = []` (optional): Columns which specify the time series in the data frame. This argument is ignored unless `strategy = 'sequential'`, in which case the data frame is sorted along these columns to sequentially impute data in the order of the time series.

### Remove Outliers

Removes multidimensional outliers in the data frame by analyzing numerical columns. If there are no numerical columns, no check is performed. Null values are ignored when checking for outliers.

The following methods are supported for detecting outliers:

| Method | Description |
| - | - |
| `'itree'` | Uses the [Isolation Tree](https://scikit-learn.org/stable/modules/outlier_detection.html#isolation-forest) algorithm. Number of trees is set to 100. |
| `'lof'` | Uses the [Local Outlier Factor](https://scikit-learn.org/stable/modules/outlier_detection.html#local-outlier-factor) algorithm.  Number of neighbors for computing local density scales with amount of data and contamination rate. |
| `'auto'` | Automatically determines from above methods based on number of data points and dimensionality of the data. |

Notes:

* The contamination rate for both algorithms is estimated via the per-dimension IQR (which discounts multidimensional interactions but still estimates how many outliers there likely are)
* If the input data has more than 20 dimensions, the dimensionality is reduced using PCA

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REMOVE_OUTLIERS,
    arguments=df.columns,  # Specify columns to remove outliers from
    axis=Axis.COLUMN,
    options={'method': 'auto'},  # Specify algorithm to use for outlier removal
)
```

**Args**

* ***arguments:*** The set of columns to consider when checking for outliers. Of these columns, only numerical columns will be used to calculate outliers.
* ***options:***
  * `method` (optional): Specifies outlier detection method to use:

    * `'itree'`
    * `'lof'`
    * `'auto'`

    Defaults to `'auto'`.

# Row Actions

These are transformer actions that can be applied to each row of a data frame, and so have `axis = Axis.ROW`.

### Drop Duplicates

Drops duplicate rows from the data frame.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.DROP_DUPLICATE,
    arguments=[],
    axis=Axis.ROW,
    options={'keep': 'first'},
)
```

**Args**

* ***arguments:*** the columns along which comparisons will be made to check for duplicate rows. If empty, all columns will be used to perform comparison.
* ***options:***
  * `keep` (optional): specifies policy on keeping duplicates:

    * `'first'` - keeps the first occurrence of the row
    * `'last'` - keeps the last occurrence of the row
    * `False` - removes occurrences of duplicates

    Defaults to `'last'`.

### Filter

Filters row by boolean condition. Selects all rows that meets the filter condition.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.FILTER,
    axis=Axis.ROW,
    action_code='column1 != null and column2 <= 5'
)
```

**Args**

* ***action\_code:*** filter to apply to select rows. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.

### Remove Rows

Removes rows specified by their index.
**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REMOVE,
    axis=Axis.ROW,
    options={'rows': []},
)
```

**Args**

* ***options:***
  * `rows`: List of the indices corresponding to the rows to remove. If empty, no rows are removed.

### Sort

Sorts rows in the specified order.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SORT,
    arguments=[],
    axis=Axis.ROW,
    options={'ascending': True},
)
```

**Args**

* ***arguments:*** the columns whose value to consider while sorting rows. If empty then sort is not performed.
* ***options:***
  * `ascending` (optional): Specifies how to sort rows. If `True`, then rows are sorted in ascending order; if `False` sorted in descending order. Defaults to `True`.
    * If sorted in ascending order, null values are placed at the top of the data frame in the same order as they appeared in the original data frame
    * If sorted in descending order, null values are placed at the bottom of the data frame in the same order as they appeared in the original data frame
  * `ascendings` (optional): A list of booleans specifying if each column specified in *arguments* should be sorted in ascending order (must be the same length). If specified, this list will take precedence over the `ascending` option.

# Appendix: Filter Syntax

Valid filters are composed of clauses of the form:

```
[column_name] [op] [value or column_name]
```

* List of valid operations: `==, !=, <=, <, >, >=, contains, not contains`
  * `contains` (and `not contains`) checks if the value is a substring (or not a substring) of values in the column specified.
* The right hand side of each clause can either be a literal value (boolean, number, string) or another column to compare to.

Clauses can be connected using the `and` and `or` operators and grouped with parentheses.

```
(brand != null and value > 60) or (discounted == null)
```

**Rules on Naming Columns**

* If column names contain only alphanumeric characters and the special character '\_', then the raw column name can used to filter code.
  Example: `age != null and price_of_item >= 50.23`
* If columns contain whitespace, newlines, and some of the following special characters: `+=-*&^%$! ?~|<>(){}[],.`, then the quoted column name can be used to filter code. Any quote character can be used.
  Example: `("+= -*&^%$! ?~ |<>" == False and ' kas22d fe ($)' > 50.23) or ("dis>>> ??cou nted" contains "partial")`
* If columns contain any other special characters apart from those listed above, the column cannot be filtered. This is a fundamental limitation of pandas, see [here](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html?highlight=query#pandas.DataFrame.query:~:text=During%20parsing%20a,not%20be%20escaped) for more information.

# Appendix: Column Types

These are list of column types used in the Mage tool. One of these column types should be specified for every entry in the *options* argument of a transformer action. The enum source is `mage_ai.data_cleaner.column_types.constants.ColumnType`.

| Type | Enum | String Value | Description |
| - | - | - | - |
| Category | `ColumnType.CATEGORY` | "category" | Categorical variable that takes one of some finite set of values. This set has at most 255 entries. |
| High Cardinality Category | `ColumnType.CATEGORY_HIGH_CARDINALITY` | "category\_high\_cardinality" | Categorical variable that takes one of some finite set of values. This set has more than 255 entries. |
| Datetime | `ColumnType.DATETIME` | "datetime" | Datetime objects or strings representing a datetime. |
| Email | `ColumnType.EMAIL` | "email" | Strings that contains emails. |
| List | `ColumnType.LIST` | "list" | Lists of primitives (integers, strings, booleans, floats) of variable length. |
| Number | `ColumnType.NUMBER` | "number" | Integer numbers. Can be a Python integer, NumPy integer or a stringified integer. |
| Floating Point Number | `ColumnType.NUMBER_WITH_DECIMALS` | "number\_with\_decimals" | Floating point numbers. Can be a Python float, NumPy float, or a stringified float. |
| Phone Number | `ColumnType.PHONE_NUMBER` | "phone\_number" | Phone numbers (either in string form or integer form). |
| Text | `ColumnType.TEXT` | "text" | Text values, such as sentences or paragraphs. |
| Binary | `ColumnType.TRUE_OR_FALSE` | "true\_or\_false" | A variable that takes on two different values (including booleans). |
| Zip Code | `ColumnType.ZIP_CODE` | "zip\_code" | Zip codes (either in string form or integer form). |


# Abstractions
Source: https://docs.mage.ai/design/core-abstractions

These are the fundamental concepts that Mage uses to operate.

![Core abstractions](https://user-images.githubusercontent.com/78053898/198752755-ed44ffd6-7e6c-45d3-a4ca-91f2810ce46c.svg)

## Project

A Mage project forms the basis for all the work you do in Mage. It contains the code for your data pipelines, blocks, and other project-specific data. You can think of it like a GitHub repo— it can even track your [file history, too](https://docs.mage.ai/guides/version-control/file-versions#file-versioning-and-history).

Check out our detailed [project structure](/design/abstractions/project-structure) docs for more info.

## Pipeline

A pipeline contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.

Each pipeline is represented by a YAML file (here's an [example](https://github.com/mage-ai/demo_etl_pipeline/blob/master/pipelines/etl_demo/metadata.yaml)) in the `pipelines` folder under the Mage project directory.

![Pipeline](https://mage-ai.github.io/assets/overview/mage-build.gif)

Pipelines can be filtered, tagged, and grouped by various properties. See our [docs page](/pipelines/pipeline-tagging) for more info.

## Block

A block is a file with code that can be executed independently or within a
pipeline. Together, blocks form a Directed Acyclic Graph (DAG), which we call pipelines. A block won't start running in a pipeline until all its upstream dependencies are met.

There are 8 types of blocks.

1. Data loader

2. Transformer

3. Data exporter

4. Scratchpad

5. Sensor

6. dbt

7. Extensions

8. Callbacks

For more information, please see the [documentation on blocks](/design/blocks)

Here is an example of a
[data loader block](https://github.com/mage-ai/demo%5Fetl%5Fpipeline/blob/master/data%5Floaders/load%5Fdataset.py)
and a snippet of its code:

```bash theme={"system"}
@data_loader
def load_data_from_api() -> DataFrame:
    url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'

    response = requests.get(url)
    return pd.read_csv(io.StringIO(response.text), sep=',')
```

Each block file is stored in a folder that matches its respective type (e.g.
transformers are stored in `[project_name]/transformers/`.

## Sensor

A sensor is a block that continuously evaluates a condition until it's met or
until a period of time has elapsed.

If there is a block with a sensor as an upstream dependency, that block won't
start running until the sensor has evaluated its condition successfully.

Sensors can check for anything. Examples of common sensors check for:

* Does a table exist (e.g. `mage.users_v1`)?

* Does a partition of a table exist (e.g. `ds = 2022-12-31`)?

* Does a file in a remote location exist (e.g. `S3`)?

* Has another pipeline finished running successfully?

* Has a block from another pipeline finished running successfully?

* Has a pipeline run or block run failed?

Here is an example of a sensor that will keep checking to see if pipeline
`transform_users` has finished running successfully for the current execution
date:

```bash theme={"system"}
from mage_ai.orchestration.run_status_checker import check_status

@sensor
def check_condition(**kwargs) -> bool:
    return check_status(
        'pipeline_uuid',
        kwargs['execution_date'],
    )
```

> [!NOTE]
>
> This example is using a helper function called `check_status` that handles the
> logic for retrieving the status of a pipeline run for `transform_users` on the
> current execution date. You can optionally pass block uuid and hours
> parameters to `check_status` to check the status of a block run or within a
> interval of time.

## Data product

Every block produces data after it's been executed. These are called **data
products** in Mage.

Data validation occurs whenever a block is executed.

Additionally, each data product produced by a block can be automatically
partitioned, versioned, and backfilled.

Some examples of data products produced by blocks:

* 📋 Dataset/Table in a database, data warehouse, etc.

* 🖼️ Image

* 📹 Video

* 📝 Text file

* 🎧 Audio file

## Trigger

A trigger is a set of instructions that determine when or how a pipeline should
run. A pipeline can have 1 or more triggers.

New trigger creation supports 2 trigger types:

1. Schedule

2. API

Event triggers are deprecated and may only appear for existing legacy
configurations.

### Schedule

A schedule-type trigger will instruct the pipeline to run after a start date and
on a set interval.

Currently, the frequency pipelines can be scheduled for include:

* Run exactly once

* Hourly

* Daily

* Weekly

* Monthly

* Every N minutes (coming soon)

### Deprecated Event

Event triggers are deprecated for new trigger creation. Use API triggers for new
event-driven or webhook-style workflows. Existing legacy event triggers may
still appear in older projects.

### API

An API-type trigger will instruct the pipeline to run after a specific API call
is made.

You can make a POST request to an endpoint provided in the UI when creating or
editing a trigger. You can optionally include runtime variables in your request
payload.

## Run

A run record stores information about when it was started, its status, when it
was completed, any runtime variables used in the execution of the pipeline or
block, etc.

Every time a pipeline or a block is executed (outside of the notebook while
building the pipeline and block), a run record is created in a database.

There are 2 types of runs:

### Pipeline run

This contains information about the entire pipeline execution.

### Block run

Every time a pipeline is executed, each block in the pipeline will be executed
and potentially create a block run record.

## Log

A log is a file that contains system output information.

It's created whenever a pipeline or block is ran.

Logs can contain information about the internal state of a run, text that is
outputted by loggers or `print` statements in blocks, or errors and stack traces
during code execution.

Here is an example of a log in the
[**Data pipeline management**](/about/features) UI:

![Log detail](https://mage-ai.github.io/assets/logs/log-detail.png)

Logs are stored on disk wherever Mage is running. However, you can configure
where you want log files written to (e.g. Amazon S3, Google Storage, etc).

## Backfill

A backfill creates 1 or more pipeline runs for a pipeline.


# Core Design Principles
Source: https://docs.mage.ai/design/core-design-principles

Every user experience and technical design decision adheres to these principles.

![Core design principles](https://user-images.githubusercontent.com/78053898/198752891-1e823231-f5eb-48ea-8a6d-91e60ec368c9.svg)

## 💻 Easy developer experience

Open-source engine that comes with a custom notebook UI for building data
pipelines.

* Mage comes with a specialized notebook UI for building data pipelines.

* Use Python and SQL (more languages coming soon) together in the same pipeline
  for ultimate flexibility.

* Set up locally and get started developing with a single command.

* Deploying to production is fast using native integrations with major cloud
  providers.

## 🚢 Engineering best practices built-in

Build and deploy data pipelines using modular code. No more writing throwaway
code or trying to turn notebooks into scripts.

* Writing reusable code is easy because every block in your data pipeline is a
  standalone file.

* Data validation is written into each block and tested every time a block is
  run.

* Operationalizing your data pipelines is easy with built-in observability, data
  quality monitoring, and lineage.

* Each block of code has a single responsibility: load data from a source,
  transform data, or export data anywhere.

## 💳 Data is a first class citizen

Designed from the ground up specifically for running data-intensive workflows.

* Every block run produces a data product (e.g. dataset, unstructured data,
  etc.)

* Every data product can be automatically partitioned.

* Each pipeline and data product can be versioned.

* Backfilling data products is a core function and operation.

## 🪐 Scaling is made simple

Analyze and process large data quickly for rapid iteration.

* Transform very large datasets through a native integration with Spark.

* Handle data intensive transformations with built-in distributed computing
  (e.g. Dask, Ray) \[coming soon].

* Run thousands of pipelines simultaneously and manage transparently through a
  collaborative UI.

* Execute SQL queries in your data warehouse to process heavy workloads.


# Data loader utilities
Source: https://docs.mage.ai/design/data-loading



Mage provides data loading clients that simplify loading and exporting data in your pipelines, allowing you to spend more time analyzing and transforming your data for ML tasks. Currently, Mage includes clients for the following data sources:

**File Storage:**

* [AWS S3](#s3)
* [Azure Blob Storage](#azureblobstorage)
* [Azure Data Lake Storage](#azure-data-lake-storage) (Pro only)
* [Filesystem](#fileio)
* [Google Cloud Storage](#googlecloudstorage)

**Databases:**

* [AWS Redshift](#redshift)
* [ClickHouse](#clickhouse)
* [Databricks SQL](#databricks-sql) (Pro only)
* [DuckDB](#duckdb)
* [MySQL](#mysql)
* [MSSQL](#mssql)
* [MongoDB](#mongodb)
* [OracleDB](#oracledb)
* [PostgreSQL](#postgresql)
* [SQLite](#sqlite)
* [Snowflake](#snowflake)
* [Spark](#spark)
* [Trino](#trino)

**Data Warehouses:**

* [Google BigQuery](#bigquery)
* [Microsoft Fabric Warehouse](#microsoft-fabric-warehouse) (Pro only)

**APIs & Services:**

* [Google Sheets](#googlesheets)

Mage's data loading clients fall into two categories:

* **File-Loading Clients** - import/export data between files and your pipeline. Includes both local filesystem storage and external filesystem storage (like AWS S3)
* **Database Clients** - imports data from an external database into your pipeline and exports data frames back into that database. These database clients include the `execute` method which execute your queries in the connected database. The Google BigQuery client follows this structure.
  * A subcategory of database clients are **connection-based clients**, which wrap a connection to the database. This connection is used to execute transactions (sets of queries) on the database, which are either committed (saved to database) or rolled-back (deleted). Clients for PostgreSQL, Redshift, and Snowflake follow this structure.

## Example: Loading data from a file

While traditional Pandas IO procedures can be utilized to load files into your pipeline, Mage provides the `mage_ai.io.file.FileIO` client as a convenience wrapper.

The following code uses the `load` function of the `FileIO` client to load the Titanic survival dataset from a CSV file into a Pandas DataFrame for use in your pipeline. All data loaders can be initialized with the `verbose = True` parameter, which will print the current action the data loading client is performing. This parameter defaults to `False`.

```python theme={"system"}
loader = FileIO(verbose=True)
df = loader.load(
    'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
)
```

Then the `export` function is used to save this data frame back to file, this time in JSON format

```python theme={"system"}
loader.export(df, './titanic_survival.json', orient='records')
```

Any formatting settings (such as specifying the data frame *orient*) can be passed as keyword arguments to `load` and `export`. These arguments are passed to Pandas IO procedures for the requested file format, enabling fine-grained control over how your data is loaded and exported.

As the data loader was constructed with the verbose parameter set to `True`, the above operations would print the following output describing the actions of the data loader.

```bash theme={"system"}
FileIO initialized
├─ Loading data frame from 'https://raw.githubusercontent.com/datasciencedojodatasets/master/titanic.csv'...DONE
└─ Exporting data frame to './titanic_survival.json'...DONE
```

See the [FileIO](#fileio) API to learn more about loading data from filesystem.

## Example: Loading data from Snowflake warehouse

Loading data from a Snowflake data warehouse is made easy using the `mage_ai.io.snowflake.Snowflake` data loading client. In order to authenticate access to a Snowflake warehouse, the client requires the associated Snowflake credentials:

* Account Username
* Account Password
* Account ID (including your region) (Ex: *example.us-west-2*)

These parameters can be manually specified as input to the data loading client (see [Snowflake](#snowflake) API). However we recommend using a [Configuration Loader](#configuration-settings) to handle loading these secrets. If you used `mage init` to create your project repository, you can store these values in your `io_config.yaml` file and use `mage_ai.io.config.ConfigFileLoader` to construct the data loader client.

For detailed information about setting up `io_config.yaml`, see the [IO Config Setup documentation](/development/io_config_setup).

An example `io_config.yaml` file in this instance would be:

```yaml theme={"system"}
default:
    SNOWFLAKE_USER: my_username
    SNOWFLAKE_PASSWORD: my_password
    SNOWFLAKE_ACCOUNT: example.us-west-2
```

with which a Snowflake data loading client can be constructed as:

```python theme={"system"}
config = ConfigFileLoader('io_config.yaml', 'default')
loader = Snowflake.with_config(config, verbose=True)
```

To learn more about using configuration settings, see [Configuration Settings](#configuration-settings).

Then the following code uses the functions:

* `execute()` - executes an arbitrary query on your data warehouse. In this case, the warehouse, database, and schema to use are selected.
* `load()` - loads the results of a `SELECT` query into a Pandas DataFrame.
* `export()` - stores data frame as a table in your data warehouse. If the table exists, then the data is appended by default (and can be configured with other behavior, see [Snowflake](#snowflake) API). If the table doesn't exist, then the table is created with the given schema name and table name (the table data types are inferred from the Python data type).

```python theme={"system"}
with loader:
    loader.execute('USE WAREHOUSE my_warehouse;')
    loader.execute('USE DATABASE my_database;')
    loader.execute('USE SCHEMA my_schema;')
    df = loader.load('SELECT * FROM test_table;')
    loader.export(df, 'my_schema', 'test_table')
```

The `loader` object manages a direct connection to your Snowflake data warehouse, so it is important to make sure that your connection is closed once your operations are completed. You can manually use `loader.open()` and `loader.close()` to open and close the connection to your data warehouse or automatically manage the connection with a context manager.

To learn more about loading data from Snowflake, see the [Snowflake](#snowflake) API for more details on member functions and usage.

# Client APIs

This section covers the API for using the following data loaders.

## FileIO

`mage_ai.io.file.FileIO`

Handles data transfer between the filesystem and the Mage app. The `FileIO` client currently supports importing and exporting with the following file formats:

* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel

### &#x20;Constructor&#x20;

`__init__(self, verbose: bool)`

Initializes the FileIO data loading client.

* **Args**
  * `verbose (bool)`: Enables verbose output printing. Defaults to False.

### &#x20;Methods&#x20;

***export*** - `export(df: DataFrame, filepath: os.PathLike, format: FileFormat | str = None, **kwargs) -> None:`

Exports the input data frame to the file specified.

If the format is HDF5, the default key under which the data frame is stored is the stem of the filename. For example, if the file to write the data frame to is 'storage/my\_data\_frame.hdf5', the key would be 'my\_data\_frame'. This can be overridden using the `key` keyword argument.

* **Args**:

  * `df (DataFrame)`: Data frame to export.
  * `filepath (os.PathLike)`: Filepath to export data frame to.
  * `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Writer |
    | - | - |
    | CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
    | JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
    | Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
    | HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
    | XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
    | Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |

***load*** - `load(filepath: os.PathLike, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT **kwargs) -> DataFrame`

Loads the data frame from the filepath specified. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).

* **Args**:

  * `filepath (os.PathLike)`: Filepath to load data frame from.
  * `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  * `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Reader |
    | - | - |
    | CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
    | JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
    | Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
    | HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
    | XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
    | Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |

* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.

## S3

`mage_ai.io.s3.S3`

Handles data transfer between an S3 bucket and the Mage app. The `S3` client supports importing and exporting with the following file formats:

* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel

If an IAM profile is not set up using `aws configure`, then AWS credentials for accesses the S3 bucket can be manually specified through either Mage's [configuration loader](#configuration-settings) system or through keyword arguments in the constructor (see constructor).

If using configuration settings, specify the following keys:

```yaml theme={"system"}
AWS_ACCESS_KEY_ID: AWS Access Key ID credential
AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
AWS_REGION: AWS Region
```

### &#x20;Constructor&#x20;

`__init__(self, verbose: bool)`

Initializes the S3 data loading client.

* **Args**
  * `verbose (bool)`: Enables verbose output printing. Defaults to False.
  * If IAM profile is not set up using `aws configure` and Mage's configuration loader is not used, then specify your AWS credentials through the following keyword arguments:
    * `aws_access_key_id (str)`: AWS Access Key ID credential
    * `aws_secret_access_key (str)`: AWS Secret Access Key credential
    * `aws_region (str)`: Region associated with AWS IAM profile

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> S3`

Creates S3 data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`S3`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***export*** - `export(df: DataFrame, bucket_name: str, object_key: str, format: FileFormat | str = None, **kwargs) -> None:`

Exports data frame to an S3 bucket.

If the format is HDF5, the default key under which the data frame is stored is the stem of the filename. For example, if the file to write the data frame to is 'storage/my\_data\_frame.hdf5', the key would be 'my\_data\_frame'. This can be overridden using the `key` keyword argument.

* **Args**:

  * `data (Union[DataFrame, str])`: Data frame or file path to export.
  * `bucket_name (str)`: Name of the bucket to export data frame to.
  * `object_key (str)`: Object key in S3 bucket to export data frame to.
  * `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Writer |
    | - | - |
    | CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
    | JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
    | Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
    | HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
    | XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
    | Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |

***load*** - `load(bucket_name: str, object_key: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT **kwargs) -> DataFrame`

Loads data from object in S3 bucket into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).

* **Args**:

  * `bucket_name (str)`: Name of the bucket to load data from.
  * `object_key (str)`: Key of the object in S3 bucket to load data from.
  * `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  * `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Reader |
    | - | - |
    | CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
    | JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
    | Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
    | HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
    | XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
    | Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |

* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.

## FileIO

`mage_ai.io.file.FileIO`

Handles data transfer between the filesystem and the Mage app. The `FileIO` client currently supports importing and exporting with the following file formats:

* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel

### &#x20;Constructor&#x20;

`__init__(self, verbose: bool)`

Initializes the FileIO data loading client.

* **Args**
  * `verbose (bool)`: Enables verbose output printing. Defaults to False.

### &#x20;Methods&#x20;

***export*** - `export(df: DataFrame, filepath: os.PathLike, format: FileFormat | str = None, **kwargs) -> None:`

Exports the input data frame to the file specified.

If the format is HDF5, the default key under which the data frame is stored is the stem of the filename. For example, if the file to write the data frame to is 'storage/my\_data\_frame.hdf5', the key would be 'my\_data\_frame'. This can be overridden using the `key` keyword argument.

* **Args**:

  * `df (DataFrame)`: Data frame to export.
  * `filepath (os.PathLike)`: Filepath to export data frame to.
  * `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Writer |
    | - | - |
    | CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
    | JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
    | Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
    | HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
    | XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
    | Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |

***load*** - `load(filepath: os.PathLike, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT **kwargs) -> DataFrame`

Loads the data frame from the filepath specified. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).

* **Args**:

  * `filepath (os.PathLike)`: Filepath to load data frame from.
  * `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  * `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Reader |
    | - | - |
    | CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
    | JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
    | Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
    | HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
    | XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
    | Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |

* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.

## GoogleCloudStorage

`mage_ai.io.google_cloud_storage.GoogleCloudStorage`

Handles data transfer between a Google Cloud Storage bucket and the Mage app. Supports loading files of any of the following types:

* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel

If `GOOGLE_APPLICATION_CREDENTIALS` environment is set, no further arguments are needed other than those specified below. Otherwise, use the factory method `with_config` to construct the data loader using manually specified credentials.

To authenticate (and authorize) access to Google Cloud Storage, credentials must be provided.
Below are the different ways to access those credentials:

* Define the `GOOGLE_APPLICATION_CREDENTIALS` environment variable holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY_FILEPATH` key with your [configuration loader](#configuration-settings) or the `path_to_credentials` keyword argument with the client constructor holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY` key with your [configuration loader](#configuration-settings) or the `credentials_mapping` keyword argument with the client constructor holding a mapping sharing the same contents as your service key
  * if using a configuration file, be careful to wrap your service key values in quotes so the YAML parser reads the settings correctly
* Manually pass the `google.oauth2.service_account.Credentials` object with the keyword argument `credentials`

### &#x20;Constructor&#x20;

`__init__(self, verbose: bool = False, **kwargs)`

Initializes the GoogleCloudStorage data loading client.

* **Args**

  * `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
  * `credentials_mapping (Mapping[str, str])` - Mapping object corresponding to your service account key. See instructions above on when to use this keyword argument
  * `path_to_credentials (str)` - Filepath to service account key. See instructions above on when to use this keyword argument.

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> GoogleCloudStorage`

Creates GoogleCloudStorage data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`GoogleCloudStorage`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***export*** - `export(df: DataFrame, bucket_name: str, object_key: str, format: FileFormat | str = None, **kwargs) -> None:`

Exports data frame to a GoogleCloudStorage bucket.

* **Args**:

  * `data (Union[DataFrame, str])`: Data frame or file path to export.
  * `bucket_name (str)`: Name of the bucket to export data frame to.
  * `object_key (str)`: Object key in GoogleCloudStorage bucket to export data frame to.
  * `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Writer |
    | - | - |
    | CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
    | JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
    | Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
    | HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
    | XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
    | Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |

***load*** - `load(bucket_name: str, object_key: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT, **kwargs) -> DataFrame`

Loads data from object in GoogleCloudStorage bucket into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).

* **Args**:

  * `bucket_name (str)`: Name of the bucket to load data from.
  * `object_key (str)`: Key of the object in GoogleCloudStorage bucket to load data from.
  * `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  * `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Reader |
    | - | - |
    | CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
    | JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
    | Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
    | HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
    | XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
    | Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |

* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.

## AzureBlobStorage

`mage_ai.io.azure_blob_storage.AzureBlobStorage`

Handles data transfer between an Azure Blob Storage container and Mage. Supports loading files of the following types:

* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel

To authenticate access to Azure Blob Storage, the following environment variables must be set:

* `AZURE_CLIENT_ID`
* `AZURE_CLIENT_SECRET`
* `AZURE_TENANT_ID`
* `AZURE_STORAGE_ACCOUNT_NAME`

By default, these correspond to variables of the same name in `io_config.yaml`.

### &#x20;Constructor&#x20;

Initializes data loader from an Azure Blob Storage container.

* **Args**

  * `verbose (bool)`: Enables verbose output printing. Defaults to `False`.

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> AzureBlobStorage`

Creates AzureBlobStorage data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`AzureBlobStorage`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***export*** - `export(df: DataFrame, bucket_name: str, object_key: str, format: FileFormat | str = None, **kwargs) -> None:`

Exports data frame to an Azure Blob Storage container.

* **Args**:

  * `df (DataFrame)`: Data frame to export.
  * `container_name (str)`: Name of the Azure container to export data to.
  * `blob_path (str)`: The desired output path of the data in your Azure Blob
  * `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    | Format | Pandas Writer |
    | - | - |
    | CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
    | JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
    | Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
    | HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
    | XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
    | Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |

***load*** - `load(container_name: str, blob_path: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT, **kwargs) -> DataFrame`

Loads data from object in Azure Blob Storage into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).

## GoogleSheets

`mage_ai.io.google_sheets.GoogleSheets`

Handles data transfer between a Google Sheets spreadsheet and the Mage app.

Authentication is the same for our other Google Cloud data loaders. Read more about configuration [here](integrations/databases/GoogleSheets#configuration).

The Google Sheets class is defined [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/google_sheets.py).

## BigQuery

`mage_ai.io.bigquery.BigQuery`

Handles data transfer between a BigQuery data warehouse and the Mage app.

Authentication with a Google BigQuery warehouse requires specifying the service account key for the service account that has access to the BigQuery warehouse. There are four ways to provide this service key:

* Define the `GOOGLE_APPLICATION_CREDENTIALS` environment variable holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY_FILEPATH` key with your [configuration loader](#configuration-settings) or the `path_to_credentials` keyword argument with the client constructor holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY` key with your [configuration loader](#configuration-settings) or the `credentials_mapping` keyword argument with the client constructor holding a mapping sharing the same contents as your service key
  * if using a configuration file, be careful to wrap your service key values in quotes so the YAML parser reads the settings correctly
* Manually pass the `google.oauth2.service_account.Credentials` object with the keyword argument `credentials`

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the BigQuery data loading client.

* **Args**

  * `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
  * `credentials_mapping (Mapping[str, str])` - Mapping object corresponding to your service account key. See instructions above on when to use this keyword argument
  * `path_to_credentials (str)` - Filepath to service account key. See instructions above on when to use this keyword argument.

  All other keyword arguments can be found in the [Google BigQuery Python Client docs](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html?highlight=client#google-cloud-bigquery-client-client)

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> BigQuery`

Creates BigQuery data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`BigQuery`) BigQuery data loading client constructed using this method

***with\_credentials\_file*** - `with_credentials_file(cls, path_to_credentials: str, **kwargs) -> BigQuery`

Constructs BigQuery data loader using the file containing the service account key.

* **Args**:
  * `path_to_credentials (str)`: Path to the credentials file.
  * `**kwargs`: Additional parameters to pass to BigQuery client constructor.
* **Returns**: (`BigQuery`) BigQuery data loading client constructed using this method

***with\_credentials\_object*** - `with_credentials_object(cls, credentials: Mapping[str, str], **kwargs) -> BigQuery`

Constructs BigQuery data loader using manually specified service account key credentials.

* **Args**:
  * `credentials (Mapping[str, str])`: Mapping containing same key-value pairs as a service account key.
  * `**kwargs`: Additional parameters to pass to BigQuery client constructor.
* **Returns**: (`BigQuery`) BigQuery data loading client constructed using this method

### &#x20;Methods&#x20;

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected BigQuery warehouse.

* **Args**:
  * `query_string (str)`: Query to execute on the BigQuery warehouse.
  * `**kwargs`: Additional arguments to pass to query, such as query configurations. See [*Client.query()*](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html?highlight=client#google.cloud.bigquery.client.Client.query) docs for additional arguments.

***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, **kwargs) -> None`

Exports a data frame to a Google BigQuery warehouse. If table doesn't exist, the table is automatically created.

* **Args**:
  * `df (DataFrame)`: Data frame to export
  * `table_id (str)`: ID of the table to export the data frame to. If of the format
    `"your-project.your_dataset.your_table_name"`. If this table exists, the table schema must match the data frame schema. If this table doesn't exist, the table schema is automatically inferred.
  * `if_exists (str)`: Specifies export policy if table exists. Either - `'fail'`: throw an error. - `'replace'`: drops existing table and creates new table of same name. - `'append'`: appends data frame to existing table. In this case the schema must match the original table.
    Defaults to `'replace'`. If `write_disposition` is specified as a keyword argument, this parameter
    is ignored (as both define the same functionality).
  * `**configuration_params`: Configuration parameters for export job. See valid configuration parameters at [*LoadJobConfig*](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.LoadJobConfig.html#google.cloud.bigquery.job.LoadJobConfig) docs.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

When a select query is provided, this function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional parameters to pass to the query. See [Google BigQuery Python client](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html?highlight=client#google.cloud.bigquery.client.Client.query) docs for additional arguments.

***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`

Sample data from a table in the BigQuery warehouse. Sample is not guaranteed to be random.

* **Args**:

  * `schema (str)`: The schema to select the table from.
  * `size (int)`: The number of rows to sample. Defaults to 10,000,000.
  * `table (str)`: The table to sample from in the connected database.

* **Returns**: (`DataFrame`) Sampled data from the table.

## PostgreSQL

`mage_ai.io.postgres.Postgres`

Handles data transfer between a PostgreSQL database and the Mage app. The `Postgres` client utilizes the following keys to connect the PostgreSQL database.

```yaml theme={"system"}
POSTGRES_DBNAME: PostgreSQL database name
POSTGRES_USER: PostgreSQL database login username
POSTGRES_PASSWORD: PostgreSQL database login password
POSTGRES_HOST: PostgreSQL database hostname
POSTGRES_PORT: PostgreSQL database port
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the Postgres data loading client.

* **Args**:
  * `dbname (str)`: The name of the database to connect to.
  * `user (str)`: The user with which to connect to the database with.
  * `password (str)`: The login password for the user.
  * `host (str)`: Host address for database.
  * `port (str)`: Port on which the database is running.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
  * `**kwargs`: Additional settings for creating psycopg2 connection

### &#x20;Attributes&#x20;

* *conn* (`psycopg2.connection.Connection`) - underlying [psycopg2 Connection](https://www.psycopg.org/docs/connection.html) object

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Postgres`

Creates Postgres data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Postgres`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to PostgreSQL database.

***commit*** - `commit()`

Saves all changes made to the database since the previous transaction.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected PostgreSQL database. Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the PostgreSQL database.
  * `**kwargs`: Additional parameters to pass to the query. See [`psycopg2` docs](https://www.psycopg.org/docs/usage.html#query-parameters) for configuring query parameters.

***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, index: bool, **kwargs) -> None`

Exports data frame to the PostgreSQL database from a Pandas data frame. If table doesn't exist, the table is automatically created. If the schema doesn't exist, the schema is also created.

Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:

  * `df (DataFrame)`: Data frame to export to the PostgreSQL database.

  * `table_name (str)`: Name of the table to export data to (excluding database and schema).

  * `database (str)`: Name of the database in which the table is located.

  * `schema (str)`: Name of the schema in which the table is located.

  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either

    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.

    Defaults to `'replace'`.

  * `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.

  * `**kwargs`: Additional arguments to pass to writer. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for details on UPSERT (`unique_constraints`, `unique_conflict_method`) and other advanced options like `overwrite_types`, `query_string`, `auto_clean_name`, etc.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional parameters to pass to the query. See [`psycopg2` docs](https://www.psycopg.org/docs/usage.html#query-parameters) for configuring query parameters.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to PostgreSQL database.

***rollback*** - `rollback()`

Rolls back (deletes) all database changes made since the last transaction.

***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`

Sample data from a table in the PostgreSQL database. Sample is not guaranteed to be random.

* **Args**:

  * `schema (str)`: The schema to select the table from.
  * `size (int)`: The number of rows to sample. Defaults to 10,000,000.
  * `table (str)`: The table to sample from in the connected database.

* **Returns**: (`DataFrame`) Sampled data from the table.

## Advanced Export Parameters for SQL Databases

The following advanced export parameters are available for all SQL database clients (PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, ClickHouse, Trino, Databricks SQL, SQLite, etc.) that inherit from `BaseSQL`. These parameters provide fine-grained control over how data is exported.

> **Note:** UPSERT (UPDATE or INSERT) functionality is only supported on a subset of SQL databases (e.g., PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, SQLite). It is **not** supported on Redshift, ClickHouse, Trino or Databricks SQL. See below for details.

### UPSERT (UPDATE or INSERT)

UPSERT allows you to update existing rows when they match unique constraints, or insert new rows if they don't exist. This is useful for maintaining data integrity and avoiding duplicate entries.

To enable UPSERT functionality, you need to specify both `unique_constraints` and `unique_conflict_method` parameters:

* `unique_constraints (List[str])`: A list of column names that form the unique constraint. These columns are used to identify existing rows for updates.
* `unique_conflict_method (str)`: Specifies how to handle conflicts when unique constraints are violated. Can be either:
  * `'UPDATE'` (or use constant `UNIQUE_CONFLICT_METHOD_UPDATE`): Updates existing rows that match the unique constraints with new values from the data frame, and inserts new rows that don't match.
  * `'IGNORE'` (or use constant `UNIQUE_CONFLICT_METHOD_IGNORE`): Ignores rows that would violate unique constraints (does not update or insert).

**Example: UPSERT with PostgreSQL**

```python theme={"system"}
from mage_ai.io.postgres import Postgres
from mage_ai.io.constants import UNIQUE_CONFLICT_METHOD_UPDATE

loader = Postgres(...)
with loader:
    loader.export(
        df,
        table_name='users',
        schema='public',
        if_exists='append',
        unique_constraints=['user_id', 'email'],  # Columns that form unique constraint
        unique_conflict_method=UNIQUE_CONFLICT_METHOD_UPDATE,  # Update existing, insert new
    )
    loader.commit()
```

In this example:

* If a row with the same `user_id` and `email` already exists, it will be updated with the new values from the data frame.
* If no matching row exists, a new row will be inserted.

**Example: Ignore duplicates**

```python theme={"system"}
from mage_ai.io.postgres import Postgres
from mage_ai.io.constants import UNIQUE_CONFLICT_METHOD_IGNORE

loader = Postgres(...)
with loader:
    loader.export(
        df,
        table_name='users',
        schema='public',
        if_exists='append',
        unique_constraints=['user_id'],
        unique_conflict_method=UNIQUE_CONFLICT_METHOD_IGNORE,  # Skip rows that already exist
    )
    loader.commit()
```

**Note**: UPSERT functionality is supported on PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, and SQLite. The exact SQL syntax used may vary by database (e.g., PostgreSQL uses `ON CONFLICT`, MySQL uses `ON DUPLICATE KEY UPDATE`, MSSQL uses `MERGE`), but the interface remains consistent across all supported databases.

UPSERT functionality is **not supported** on Redshift, ClickHouse, Trino and Databricks SQL.

### Additional Export Parameters

The following parameters provide additional control over the export process:

* `allow_reserved_words (bool)`: If `True`, allows column names that are SQL reserved words. Defaults to `False`.
* `auto_clean_name (bool)`: If `True`, automatically cleans column and table names (removes special characters, handles reserved words). Defaults to `True`.
* `case_sensitive (bool)`: If `True`, preserves case sensitivity for identifiers. Defaults to `False`.
* `cascade_on_drop (bool)`: If `True`, adds `CASCADE` when dropping tables. Defaults to `False`.
* `drop_table_on_replace (bool)`: If `True`, drops the table when using `if_exists='replace'` instead of deleting all rows. Defaults to `False`.
* `overwrite_types (Dict[str, str])`: A dictionary mapping column names to SQL data types to override automatic type inference. Example: `{'price': 'DECIMAL(10,2)', 'description': 'TEXT'}`.
* `query_string (str)`: A custom SQL query string to use for creating the table. If provided, the table is created using `CREATE TABLE AS SELECT ...` syntax.
* `skip_check_table_exists (bool)`: If `True`, skips checking if the table exists before export. Defaults to `False`.
* `skip_create_schema (bool)`: If `True`, skips creating the schema if it doesn't exist. Defaults to `False`.
* `skip_semicolon_at_end (bool)`: If `True`, omits the semicolon at the end of SQL statements. Defaults to `False`.

**Example: Using overwrite\_types**

```python theme={"system"}
loader.export(
    df,
    table_name='products',
    schema='public',
    overwrite_types={
        'price': 'DECIMAL(10,2)',
        'description': 'TEXT',
        'created_at': 'TIMESTAMP',
    }
)
```

**Example: Using query\_string**

```python theme={"system"}
loader.export(
    df,
    table_name='filtered_data',
    schema='public',
    query_string='SELECT * FROM source_table WHERE status = \'active\'',
)
```

## OracleDB

```yaml theme={"system"}
ORACLEDB_USER: OracleDB username
ORACLEDB_PASSWORD: OracleDB password
ORACLEDB_HOST: OracleDB host
ORACLEDB_PORT: OracleDB port
ORACLEDB_SERVICE: OracleDB service
ORACLEDB_MODE: OracleDB mode
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the OracleDB data loading client.

* **Args**:
  * `user (str)`: The user for connecting to the database with.
  * `password (str)`: The login password for the user.
  * `host (str)`: Host address for database.
  * `port (int)`: Port on which the database is running Defaults to 3306.
  * `service (str)`: OracleDB service
  * `mode (str)`: switch between OracleDB mode of `thin` or `thick`
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating psycopg2 connection

To switch OracleDB mode to `thick`, it is required to use the customized oracle Dockerfile in `integrations/oracle/Dockerfile`.

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Postgres`

Creates OracleDB data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`OracleDB`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected OracleDB database. Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the OracleDB database.
  * `**kwargs`: Additional parameters to pass to the query.

***export*** - `export(df: DataFrame, schema_name: str, table_name: str, if_exists: str, index: bool, **kwargs) -> None`

Exports data frame to the OracleDB database from a Pandas data frame. If table doesn't exist, the table is automatically created. The `schema_name` can be set as `None` since it is not required for OracleDB loaders.

Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:

  * `df (DataFrame)`: Data frame to export to the OracleDB database.

  * `schema_name (str)`: Not required for OracleDB loaders. Set to `None`.

  * `table_name (str)`: Name of the table to export data to (excluding database).

  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either

    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.

    Defaults to `'replace'`.

  * `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.

  * `**kwargs`: Additional arguments to pass to writer.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to OracleDB database.

## MySQL

`mage_ai.io.mysql.MySQL`

Handles data transfer between a MySQL database and the Mage app. The `MySQL` client utilizes the following keys to connect the MySQL database.

```yaml theme={"system"}
MYSQL_DATABASE: MySQL database name
MYSQL_HOST: MySQL database hostname
MYSQL_PASSWORD: MySQL database login password
MYSQL_PORT: MySQL database port
MYSQL_USER: MySQL database login username
MYSQL_ALLOW_LOCAL_INFILE: Pass boolean value to `allow_local_infile` param during connection to enable the capability to load local data file.
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the MySQL data loading client.

* **Args**:
  * `database (str)`: The name of the database to connect to.
  * `user (str)`: The user for connecting to the database with.
  * `password (str)`: The login password for the user.
  * `host (str)`: Host address for database.
  * `port (int)`: Port on which the database is running Defaults to 3306.
  * `allow_local_infile (bool)`: Enables verbose capability to load local file. Defaults to `False`.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating psycopg2 connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MySQL`

Creates MySQL data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MySQL`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to MySQL database.

***commit*** - `commit()`

Saves all changes made to the database since the previous transaction.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected MySQL database. Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the MySQL database.
  * `**kwargs`: Additional parameters to pass to the query.

***export*** - `export(df: DataFrame, schema_name: str, table_name: str, if_exists: str, index: bool, **kwargs) -> None`

Exports data frame to the MySQL database from a Pandas data frame. If table doesn't exist, the table is automatically created. The `schema_name` can be set as `None` since it is not required for MySQL loaders.

Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:

  * `df (DataFrame)`: Data frame to export to the MySQL database.

  * `schema_name (str)`: Not required for MySQL loaders. Set to `None`.

  * `table_name (str)`: Name of the table to export data to (excluding database).

  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either

    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.

    Defaults to `'replace'`.

  * `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.

  * `**kwargs`: Additional arguments to pass to writer. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for details on UPSERT (`unique_constraints`, `unique_conflict_method`) and other advanced options.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to MySQL database.

***sample*** - `sample(database: str, table: str, size: int, **kwargs) -> DataFrame`

Sample data from a table in the MySQL database. Sample is not guaranteed to be random.

* **Args**:

  * `database (str)`: The database to select the table from.
  * `table (str)`: The table to sample from in the connected database.
  * `size (int)`: The number of rows to sample. Defaults to 10,000,000.

* **Returns**: (`DataFrame`) Sampled data from the table.

## DuckDB

`mage_ai.io.duckdb`

Handles data transfer between DuckDB database and the Mage app.
DuckDB client utilizes the following keys to connect:

```yaml theme={"system"}
DUCKDB_DATABASE: database # Database name
DUCKDB_SCHEMA: main # Schema name. Default is main.
```

### &#x20;Constructor&#x20;

```
__init__(self,
         database: str
         schema: str = None,
         verbose: bool = True,
         **kwargs
        )
```

Initializes the DuckDB data loading client.

* **Args**:
  * `database (str)`: The name of the database to connect to.
  * `schema`: Schema to use.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating psycopg2 connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> DuckDB`

Creates DuckDB data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`DuckDB`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to DuckDB database.

***open*** - `open()`

Opens a connection to DuckDB database.

***table\_exists*** - `table_exists()`

Build query to check if a table existing in the DB.

* **Args**:
  * `schema_name`: Name of the schema to use.
  * `table_name`: Name of the table.
* **Returns**: true or false.

***upload\_dataframe*** - `upload_dataframe()`

Build query to insert data into table.

* **Args**:
  * `df`: Data in dataframe type.

***get\_type*** - `get_type()`

Convert data from input type into DuckDB type.

* **Args**:
  * `column`: Metadata associated with the column.
  * `dtype`: Input data type.
* **Returns**: DuckDB data type in string format.

## Snowflake

`mage_ai.io.snowflake.Snowflake`

Handles data transfer between a Snowflake data warehouse and the Mage app. The `Snowflake` client utilizes the following keys to authenticate access and connect to Snowflake servers.

```yaml theme={"system"}
SNOWFLAKE_USER: username
SNOWFLAKE_PASSWORD: password
SNOWFLAKE_ACCOUNT: account_id.region
SNOWFLAKE_DEFAULT_WH: null                  # Optional default warehouse
SNOWFLAKE_DEFAULT_DB: null                  # Optional default database
SNOWFLAKE_DEFAULT_SCHEMA: null              # Optional default schema
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null      # Optional private key passphrase
SNOWFLAKE_PRIVATE_KEY_PATH: null            # Optional private key path
SNOWFLAKE_ROLE: null                        # Optional role name
SNOWFLAKE_TIMEOUT: null                     # Optional timeout in seconds
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes settings for connecting to Snowflake data warehouse.
The following arguments must be provided to the connector, all other
arguments are optional.

*Required Arguments*:

* `user (str)`: Username for the Snowflake user.
* `password (str)`: Login Password for the user.
* `account (str)`: Snowflake account identifier (including region, excluding `snowflake-computing.com` suffix).

*Optional Arguments*:

* `verbose (bool)`: Specify whether to print verbose output.
* `database (str)`: Name of the default database to use. If unspecified no database is selected on login.
* `schema (str)`: Name of the default schema to use. If unspecified no schema is selected on login.
* `warehouse (str)`: Name of the default warehouse to use. If unspecified no warehouse is selected on login.

### &#x20;Attributes&#x20;

* *conn* (`snowflake.connector.Connection`) - underlying [Snowflake Connection](https://docs.snowflake.com/en/user-guide/python-connector-api.html#object-connection) object

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Snowflake`
Creates Snowflake data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `verbose (bool)`: Enables verbose output printing. Defaults to False.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Snowflake`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to Snowflake server.

***commit*** - `commit()`

Saves all changes made to the warehouse since the previous transaction.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected Snowflake warehouse. Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the Snowflake warehouse.
  * `**kwargs`: Additional parameters to pass to the query. See [Snowflake Connector Docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#execute) for additional parameters.

***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, **kwargs) -> None`

Exports a Pandas data frame to a Snowflake warehouse based on the table name. If table doesn't exist, the table is automatically created.

Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:

  * `df (DataFrame)`: Data frame to export to a Snowflake warehouse.

  * `table_name (str)`: Name of the table to export data to (excluding database and schema).

  * `database (str)`: Name of the database in which the table is located.

  * `schema (str)`: Name of the schema in which the table is located.

  * `if_exists (str, optional)`: Specifies export policy if table exists. Either

    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.

    Defaults to `'append'`.

  * `**kwargs`: Additional arguments to pass to writer. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for details on UPSERT (`unique_constraints`, `unique_conflict_method`) and other advanced options.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from Snowflake into a Pandas data frame based on the query given. This will fail unless a `SELECT` query is provided.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse using `execute`.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `*args, **kwargs`: Additional parameters to pass to the query. See [Snowflake Connector Docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#execute) for additional parameters.
* **Returns**: (`DataFrame`) Data frame containing the queried data

***open*** - `open()`

Opens a connection to Snowflake servers.

***rollback*** - `rollback()`

Rolls back (deletes) all database changes made since the last transaction.

***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`

Sample data from a table in the Snowflake warehouse. Sample is not guaranteed to be random.

* **Args**:

  * `schema (str)`: The schema to select the table from.
  * `size (int)`: The number of rows to sample. Defaults to 10,000,000.
  * `table (str)`: The table to sample from in the connected database.

* **Returns**: (`DataFrame`) Sampled data from the data frame.

## Redshift

`mage_ai.io.redshift.Redshift`

Handles data transfer between a Redshift cluster and the Mage app. Mage uses temporary credentials to authenticate access to a Redshift cluster. There are two ways to specify these credentials:

1. Pre-generate temporary credentials and specify them in the configuration settings. Add the following keys to the configuration settings for `Redshift` to use the temporary credentials:
   ```yaml theme={"system"}
   REDSHIFT_DBNAME: Name of Redshift database to connect to
   REDSHIFT_HOST: Redshift Cluster hostname
   REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
   REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
   REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
   ```
2. Provide an IAM Profile to automatically generate temporary credentials for connection. The IAM profile is read from `~/.aws/` and is used with the `GetClusterCredentials` endpoint to generate temporary credentials. Add the following keys to the configuration settings for `Redshift` to generate temporary credentials

   ```yaml theme={"system"}
   REDSHIFT_DBNAME: Name of Redshift database to connect to
   REDSHIFT_DBUSER: Redshift database user to generate credentials for
   REDSHIFT_CLUSTER_ID: Redshift cluster ID
   REDSHIFT_IAM_PROFILE: Name of the IAM profile to generate temp credentials with
   ```

   If an IAM profile is not setup using `aws configure`, manually specify the AWS credentials in the configuration settings as well.

   ```yaml theme={"system"}
   AWS_ACCESS_KEY_ID: AWS Access Key ID credential
   AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
   AWS_SESSION_TOKEN: AWS Session Token (used to generate temp DB credentials)
   AWS_REGION: AWS Region
   ```

### &#x20;Constructor&#x20;

`__init__(**kwargs)`

* **Args**

  * `verbose`: Enables verbose output. Defaults to `False`.

  See other keyword arguments in [Redshift's Python connector](https://docs.aws.amazon.com/redshift/latest/mgmt/python-configuration-options.html) docs.

### &#x20;Attributes&#x20;

* *conn* (`redshift_connector.Connection`) - the underlying [Redshift Connection](https://docs.aws.amazon.com/redshift/latest/mgmt/python-api-reference.html#python-api-connection) object.

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Redshift`

Initializes Redshift client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor.
* **Returns**: (`Redshift`) Redshift data loading client constructed using this method

***with\_temporary\_credentials*** - `with_temporary_credentials(database: str, host: str, user: str, password: str, port: int = 5439, **kwargs) -> Redshift`

Creates a Redshift data loader with temporary database credentials.

* **Args**:
  * `database (str)`: Name of the database to connect to.
  * `host (str)`: The hostname of the Redshift cluster which the database belongs to.
  * `user (str)`: Temporary credentials username for use in authentication.
  * `password (str)`: Temporary credentials password for use in authentication.
  * `port (int, optional)`: Port number of the Redshift cluster. Defaults to 5439.
  * `**kwargs`: Additional parameters passed to the loader constructor.
* **Returns**: (`Redshift`) Redshift data loading client constructed using this method

***with\_iam*** - `with_iam(cluster_identifier: str, database: str, db_user: str, profile: str, **kwargs) -> Redshift`

Creates a Redshift data loader using an IAM profile from `~/.aws`.

The IAM Profile settings can also be manually specified as keyword arguments to this constructor, but is not recommended. If credentials are manually specified, the region of the Redshift cluster must also be specified.

* **Args**:
  * `cluster_identifier (str)`: Identifier of the cluster to connect to.
  * `database (str)`: The database to connect to within the specified cluster.
  * `db_user (str)`: Database username
  * `profile (str, optional)`: The profile to use from stored credentials file.
    Defaults to 'default'.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Redshift`) Redshift data loading client constructed using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to the Redshift cluster specified in the loader configuration.

***commit*** - `commit()`

Saves all changes made to the database since the last transaction.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected Redshift cluster. Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the Redshift cluster.
  * `**kwargs`: Additional parameters to pass to the query. See [`redshift-connector` docs](https://github.com/aws/amazon-redshift-python-driver#configuring-cursor-paramstyle) for configuring query parameters.

***export*** - `export(df: DataFrame, table_name: str) -> None`

Exports a Pandas data frame to a Redshift cluster under the specified table. The changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `df (DataFrame)`: Data frame to export to database in Redshift cluster.
  * `table_name (str)`: Name of the table to export the data to. Table must already exist.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse using `execute`.

* **Args**:

  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `*args, **kwargs`: Additional parameters to send to query, including parameters
    for use with format strings. See [`redshift-connector` docs](https://github.com/aws/amazon-redshift-python-driver#configuring-cursor-paramstyle) for configuring query parameters.

* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to the Redshift cluster specified in the loader configuration.

***rollback*** - `rollback()`

Rolls back (deletes) all database changes made since the last transaction.

***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`

Sample data from a table in the selected database in the Redshift cluster. Sample is not guaranteed to be random.

* **Args**:

  * `schema (str)`: The schema to select the table from.
  * `size (int)`: The number of rows to sample. Defaults to 10,000,000.
  * `table (str)`: The table to sample from in the connected database.

* **Returns**: (`DataFrame`) Sampled data from the table.

## MSSQL

`mage_ai.io.mssql.MSSQL`

Handles data transfer between a Microsoft SQL Server database and the Mage app. The `MSSQL` client utilizes the following keys to connect to the database.

```yaml theme={"system"}
MSSQL_DATABASE: MSSQL database name
MSSQL_USER: MSSQL database login username
MSSQL_PASSWORD: MSSQL database login password
MSSQL_HOST: MSSQL database hostname
MSSQL_PORT: MSSQL database port (defaults to 1433)
MSSQL_SCHEMA: MSSQL database schema (defaults to 'dbo')
MSSQL_AUTHENTICATION: Authentication method (optional)
MSSQL_CONNECTION_METHOD: Connection method ('direct' or 'ssh_tunnel')
MSSQL_SSH_HOST: SSH tunnel host (if using SSH tunnel)
MSSQL_SSH_PORT: SSH tunnel port
MSSQL_SSH_USERNAME: SSH tunnel username
MSSQL_SSH_PASSWORD: SSH tunnel password
MSSQL_SSH_PKEY: SSH tunnel private key path
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the MSSQL data loading client.

* **Args**:
  * `database (str)`: The name of the database to connect to.
  * `user (str)`: The user for connecting to the database with.
  * `password (str)`: The login password for the user.
  * `host (str)`: Host address for database.
  * `port (int)`: Port on which the database is running. Defaults to 1433.
  * `schema (str)`: Schema name. Defaults to 'dbo'.
  * `authentication (str, optional)`: Authentication method.
  * `connection_method (str)`: Connection method ('direct' or 'ssh\_tunnel'). Defaults to 'direct'.
  * `ssh_host (str, optional)`: SSH tunnel host.
  * `ssh_port (int, optional)`: SSH tunnel port.
  * `ssh_username (str, optional)`: SSH tunnel username.
  * `ssh_password (str, optional)`: SSH tunnel password.
  * `ssh_pkey (str, optional)`: SSH tunnel private key path.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MSSQL`

Creates MSSQL data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MSSQL`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to MSSQL database.

***commit*** - `commit()`

Saves all changes made to the database since the previous transaction.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected MSSQL database. Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the MSSQL database.
  * `**kwargs`: Additional parameters to pass to the query.

***export*** - `export(df: DataFrame, schema_name: str, table_name: str, if_exists: str, index: bool, **kwargs) -> None`

Exports data frame to the MSSQL database from a Pandas data frame. If table doesn't exist, the table is automatically created.

Any changes made to the database will not be saved unless `commit()` is called afterward.

* **Args**:
  * `df (DataFrame)`: Data frame to export to the MSSQL database.
  * `schema_name (str)`: Name of the schema in which the table is located. Defaults to 'dbo'.
  * `table_name (str)`: Name of the table to export data to.
  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.
      Defaults to `'replace'`.
  * `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
  * `**kwargs`: Additional arguments to pass to the writer. See the [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) section for details on using options such as `unique_constraints`, `unique_conflict_method`, and other advanced export options.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to MSSQL database.

***rollback*** - `rollback()`

Rolls back (deletes) all database changes made since the last transaction.

## ClickHouse

`mage_ai.io.clickhouse.ClickHouse`

Handles data transfer between a ClickHouse data warehouse and the Mage app. The `ClickHouse` client utilizes the following keys to authenticate access and connect to ClickHouse servers.

```yaml theme={"system"}
CLICKHOUSE_HOST: ClickHouse hostname
CLICKHOUSE_PORT: ClickHouse port
CLICKHOUSE_USERNAME: ClickHouse username
CLICKHOUSE_PASSWORD: ClickHouse password
CLICKHOUSE_DATABASE: ClickHouse database name
CLICKHOUSE_INTERFACE: ClickHouse interface (optional)
CLICKHOUSE_SSL_CA_CERT: Path to CA certificate for SSL (optional)
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the ClickHouse data loading client.

* **Args**:
  * `host (str)`: ClickHouse hostname.
  * `port (int)`: ClickHouse port.
  * `username (str)`: ClickHouse username.
  * `password (str)`: ClickHouse password.
  * `database (str)`: Database name. Defaults to 'default'.
  * `interface (str, optional)`: ClickHouse interface.
  * `secure (bool, optional)`: Enable SSL.
  * `ca_cert (str, optional)`: Path to CA certificate.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings passed to ClickHouse client

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> ClickHouse`

Creates ClickHouse data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`ClickHouse`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***execute*** - `execute(command_string: str, **kwargs) -> None`

Sends command to the connected ClickHouse warehouse.

* **Args**:
  * `command_string (str)`: Command to execute on the ClickHouse warehouse.
  * `**kwargs`: Additional arguments to pass to command.

***load*** - `load(query_string: str, limit: int, **kwargs) -> DataFrame`

Loads data from ClickHouse into a Pandas data frame based on the query given. This will fail if the query returns no data from the database. When a select query is provided, this function will load at maximum 10,000,000 rows of data. To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded dataframe to. Defaults to 10,000,000.
  * `**kwargs`: Additional arguments to pass to query.
* **Returns**: (`DataFrame`) Data frame associated with the given query.

***export*** - `export(df: DataFrame, table_name: str, database: str, if_exists: str, **kwargs) -> None`

Exports a Pandas data frame to a ClickHouse warehouse. If table doesn't exist, the table is automatically created.

* **Args**:
  * `df (DataFrame)`: Data frame to export to ClickHouse warehouse.
  * `table_name (str)`: Name of the table to export data to.
  * `database (str)`: Name of the database in which the table is located.
  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.
      Defaults to `'replace'`.
  * `**kwargs`: Advanced export parameters. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases). (UPSERT not supported.)

## SQLite

`mage_ai.io.sqlite.SQLite`

Handles data transfer between a SQLite database and the Mage app.

### &#x20;Constructor&#x20;

`__init__(self, database: str, verbose: bool = True, **kwargs)`

Initializes the SQLite data loading client.

* **Args**:
  * `database (str)`: Path to the SQLite database file.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> SQLite`

Creates SQLite data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`SQLite`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to SQLite database.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected SQLite database.

* **Args**:
  * `query_string (str)`: The query to execute on the SQLite database.
  * `**kwargs`: Additional parameters to pass to the query.

***export*** - `export(df: DataFrame, table_name: str, if_exists: str, index: bool, **kwargs) -> None`

Exports data frame to the SQLite database from a Pandas data frame. If table doesn't exist, the table is automatically created.

* **Args**:
  * `df (DataFrame)`: Data frame to export to the SQLite database.
  * `table_name (str)`: Name of the table to export data to.
  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.
      Defaults to `'replace'`.
  * `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
  * `**kwargs`: Advanced export options such as `unique_constraints`, `unique_conflict_method`, `overwrite_types`, `auto_clean_name`, etc. See the [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) section for details.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to SQLite database.

## Trino

`mage_ai.io.trino.Trino`

Handles data transfer between a Trino data warehouse and the Mage app. The `Trino` client utilizes the following keys to authenticate access and connect to Trino servers.

```yaml theme={"system"}
TRINO_CATALOG: Trino catalog name
TRINO_HOST: Trino hostname
TRINO_PORT: Trino port (defaults to 8080)
TRINO_USER: Trino username
TRINO_PASSWORD: Trino password
TRINO_SCHEMA: Trino schema name
```

### &#x20;Constructor&#x20;

`__init__(self, catalog: str, host: str, user: str, password: str = None, port: int = 8080, schema: str = None, verbose: bool = True, **kwargs)`

Initializes the Trino data loading client.

* **Args**:
  * `catalog (str)`: Trino catalog name.
  * `host (str)`: Trino hostname.
  * `user (str)`: Trino username.
  * `password (str, optional)`: Trino password.
  * `port (int)`: Trino port. Defaults to 8080.
  * `schema (str, optional)`: Trino schema name.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Trino`

Creates Trino data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Trino`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to Trino warehouse.

***commit*** - `commit()`

Saves all changes made to the warehouse since the previous transaction.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected Trino warehouse. Any changes made to the warehouse will not be saved unless `commit()` is called afterward.

* **Args**:
  * `query_string (str)`: The query to execute on the Trino warehouse.
  * `**kwargs`: Additional parameters to pass to the query.

***export*** - `export(df: DataFrame, table_name: str, schema: str, if_exists: str, **kwargs) -> None`

Exports a Pandas data frame to a Trino warehouse. If table doesn't exist, the table is automatically created.

Any changes made to the warehouse will not be saved unless `commit()` is called afterward.

* **Args**:
  * `df (DataFrame)`: Data frame to export to Trino warehouse.
  * `table_name (str)`: Name of the table to export data to.
  * `schema (str)`: Name of the schema in which the table is located.
  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.
      Defaults to `'replace'`.
  * `**kwargs`: Advanced export parameters. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases). (UPSERT not supported.)

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from Trino into a Pandas data frame based on the query given. This will fail if the query returns no data from the warehouse.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to Trino warehouse.

***rollback*** - `rollback()`

Rolls back (deletes) all warehouse changes made since the last transaction.

## MongoDB

`mage_ai.io.mongodb.MongoDB`

Handles data transfer between a MongoDB database and the Mage app. The `MongoDB` client utilizes the following keys to connect to the database.

```yaml theme={"system"}
MONGODB_CONNECTION_STRING: MongoDB connection string (alternative to individual settings)
MONGODB_HOST: MongoDB hostname
MONGODB_PORT: MongoDB port (defaults to 27017)
MONGODB_USER: MongoDB username
MONGODB_PASSWORD: MongoDB password
MONGODB_DATABASE: MongoDB database name
MONGODB_COLLECTION: MongoDB collection name
```

### &#x20;Constructor&#x20;

`__init__(self, connection_string: str = None, host: str = None, port: int = 27017, user: str = None, password: str = None, database: str = None, collection: str = None, verbose: bool = True, **kwargs)`

Initializes the MongoDB data loading client.

* **Args**:
  * `connection_string (str, optional)`: MongoDB connection string. If provided, other connection parameters are ignored.
  * `host (str, optional)`: MongoDB hostname.
  * `port (int, optional)`: MongoDB port. Defaults to 27017.
  * `user (str, optional)`: MongoDB username.
  * `password (str, optional)`: MongoDB password.
  * `database (str, optional)`: MongoDB database name.
  * `collection (str, optional)`: MongoDB collection name.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MongoDB`

Creates MongoDB data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MongoDB`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***load*** - `load(collection: str = None, query: Dict = None, **kwargs) -> DataFrame`

Loads the data frame from the MongoDB collection.

* **Args**:
  * `collection (str, optional)`: MongoDB collection name. Defaults to collection specified in constructor.
  * `query (Dict, optional)`: Filter the result by using a query object. Examples:
    `{ "address": "Park Lane 38" }`, `{ "address": { "$gt": "S" } }`
* **Returns**: (`DataFrame`) Data frame object loaded from the MongoDB collection.

***export*** - `export(data: Union[DataFrame, List[Dict]], collection: str = None, **kwargs) -> None`

Exports the input dataframe to the MongoDB collection.

* **Args**:
  * `data (Union[DataFrame, List[Dict]])`: Data frame or List of Dictionary to export.
  * `collection (str, optional)`: MongoDB collection name. Defaults to collection specified in constructor.

## Databricks SQL

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=databricks)

`mage_ai.io.databricks_sql.DatabricksSQL`

Handles data transfer between a Databricks SQL warehouse and the Mage app. The `DatabricksSQL` client utilizes the following keys to authenticate access and connect to Databricks.

```yaml theme={"system"}
DATABRICKS_ACCESS_TOKEN: Databricks access token
DATABRICKS_HOST: Databricks workspace hostname
DATABRICKS_HTTP_PATH: Databricks SQL warehouse HTTP path
DATABRICKS_DATABASE: Databricks database/catalog name
DATABRICKS_SCHEMA: Databricks schema name
```

### &#x20;Constructor&#x20;

`__init__(self, verbose: bool = True, **kwargs)`

Initializes the Databricks SQL data loading client.

* **Args**:
  * `access_token (str)`: Databricks access token (required).
  * `host (str)`: Databricks workspace hostname (required).
  * `http_path (str)`: Databricks SQL warehouse HTTP path (required).
  * `database (str, optional)`: Database/catalog name.
  * `schema (str, optional)`: Schema name.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating connection

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> DatabricksSQL`

Creates Databricks SQL data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`DatabricksSQL`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***close*** - `close()`

Closes connection to Databricks SQL warehouse.

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected Databricks SQL warehouse.

* **Args**:
  * `query_string (str)`: The query to execute on the Databricks SQL warehouse.
  * `**kwargs`: Additional parameters to pass to the query.

***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, **kwargs) -> None`

Exports a Pandas data frame to a Databricks SQL warehouse. If table doesn't exist, the table is automatically created.

* **Args**:
  * `df (DataFrame)`: Data frame to export to Databricks SQL warehouse.
  * `table_name (str)`: Name of the table to export data to.
  * `database (str)`: Name of the database/catalog in which the table is located.
  * `schema (str)`: Name of the schema in which the table is located.
  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.
      Defaults to `'replace'`.
  * `**kwargs`: See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for available options.

***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`

Loads data from Databricks SQL into a Pandas data frame based on the query given. This will fail if the query returns no data from the warehouse.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***open*** - `open()`

Opens a connection to Databricks SQL warehouse.

## Spark

`mage_ai.io.spark.Spark`

Handles data transfer between a Spark session and the Mage app. The `Spark` client utilizes the following keys to connect to Spark.

```yaml theme={"system"}
SPARK_HOST: Spark host/master URL
SPARK_METHOD: Spark connection method
SPARK_SCHEMA: Spark database/schema name
```

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the Spark data loading client.

* **Args**:
  * `host (str)`: Spark master URL (e.g., 'local', 'spark://host:port').
  * `method (str, optional)`: Spark connection method.
  * `database (str, optional)`: Database/schema name.
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings for creating Spark session

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Spark`

Creates Spark data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Spark`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***execute*** - `execute(query_string: str, **kwargs) -> None`

Sends query to the connected Spark session.

* **Args**:
  * `query_string (str)`: Query to execute on the Spark session.
  * `**kwargs`: Additional arguments to pass to query.

***load*** - `load(query_string: str, limit: int, **kwargs) -> DataFrame`

Loads data from Spark into a Pandas data frame based on the query given. This will fail if the query returns no data.

This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in Spark.

* **Args**:
  * `query_string (str)`: Query to fetch a table or subset of a table.
  * `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs`: Additional arguments to pass to query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.

***export*** - `export(df: DataFrame, table_name: str, database: str, if_exists: str, **kwargs) -> None`

Exports a Pandas data frame to a Spark session. If table doesn't exist, the table is automatically created.

* **Args**:
  * `df (DataFrame)`: Data frame to export to Spark session.
  * `table_name (str)`: Name of the table to export data to.
  * `database (str)`: Name of the database in which the table is located.
  * `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
    * `'fail'`: throw an error.
    * `'replace'`: drops existing table and creates new table of same name.
    * `'append'`: appends data frame to existing table.
      Defaults to `'replace'`.
  * `**kwargs`: Additional arguments to pass to writer.

## Azure Data Lake Storage

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=azure-data-lake-storage)

`mage_ai.io.azure_data_lake_storage.AzureDataLakeStorage`

Handles data transfer between an Azure Data Lake Storage Gen2 container and the Mage app. Supports loading files of the following types:

* CSV
* JSON
* Parquet

To authenticate access to Azure Data Lake Storage, the following environment variables must be set:

* `AZURE_CLIENT_ID`
* `AZURE_CLIENT_SECRET`
* `AZURE_TENANT_ID`
* `AZURE_STORAGE_ACCOUNT_NAME`

By default, these correspond to variables of the same name in `io_config.yaml`.

### &#x20;Constructor&#x20;

`__init__(self, verbose: bool = True, **kwargs)`

Initializes data loader from an Azure Data Lake Storage Gen2 container.

* **Args**:
  * `storage_account_name (str)`: Azure Storage account name (required).
  * `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
  * `**kwargs`: Additional settings. If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are provided, they will be used for authentication. Otherwise, DefaultAzureCredential will be used.

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> AzureDataLakeStorage`

Creates AzureDataLakeStorage data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`AzureDataLakeStorage`) The constructed dataloader using this method

### &#x20;Methods&#x20;

***export*** - `export(df: DataFrame, container_name: str, file_path: str, format: FileFormat | str = None, **kwargs) -> None`

Exports data frame to an Azure Data Lake Storage Gen2 container.

* **Args**:
  * `df (DataFrame)`: Data frame to export.
  * `container_name (str)`: Name of the Azure container to export data to.
  * `file_path (str)`: The desired output path of the data in your Azure Data Lake Storage.
  * `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
  * `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats in [FileIO](#fileio) section.

***load*** - `load(container_name: str, file_path: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT, **kwargs) -> DataFrame`

Loads data from object in Azure Data Lake Storage Gen2 into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).

* **Args**:
  * `container_name (str)`: Name of the container to load data from.
  * `file_path (str)`: Path of the file in Azure Data Lake Storage to load data from.
  * `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  * `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  * `**kwargs` - Additional keyword arguments to pass to the file reader. See possible arguments by file formats in [FileIO](#fileio) section.
* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.

## Microsoft Fabric Warehouse

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=microsoft-fabric-warehouse)

`mage_ai.io.microsoft_fabric_warehouse.MicrosoftFabricWarehouse`

Handles data transfer between a Microsoft Fabric Warehouse and the Mage app. This client extends the MSSQL client and uses the same connection methods.

### &#x20;Constructor&#x20;

`__init__(self, **kwargs)`

Initializes the Microsoft Fabric Warehouse data loading client. Uses the same parameters as [MSSQL](#mssql).

### &#x20;Factory Methods&#x20;

***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MicrosoftFabricWarehouse`

Creates Microsoft Fabric Warehouse data loading client from configuration settings.

* **Args**:
  * `config (BaseConfigLoader)`: Configuration loader object.
  * `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MicrosoftFabricWarehouse`) The constructed dataloader using this method

# Configuration Settings

Connections to third-party data storage require you to specify confidential information such as login information or access keys. While you can manually specify this information code while constructing data loading clients, it is recommended to not store the secrets directly in code.

Instead, Mage provides **configuration loaders** which allow data loading clients to use your secrets without explicitly writing them in code.

> [!WARNING]
>
> Currently Mage only supports AWS Secrets Manager as a configuration loader. If you need to use secrets from Azure Key Vault
> or GCP Secret Manager, you can reference the following docs:
>
> * [Azure Key Vault](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure)
> * [GCP Secret Manager](https://docs.mage.ai/production/deploying-to-cloud/secrets/GCP)

Currently, the following sources (and their corresponding configuration loader) can be used to load configuration settings:

* Configuration File - `ConfigFileLoader`
* Environment Variables - `EnvironmentVariableLoader`
* AWS Secrets Manager - `AWSSecretLoader`

For example, the code below constructs a Redshift data loading client using secrets stored in AWS Secrets Manager

```python theme={"system"}
from mage_ai.io.config import AWSSecretLoader
from mage_ai.io.redshift import Redshift

config = AWSSecretLoader()
loader = Redshift.from_config(config)
```

The following are the set of allowed key names that you must name your secrets with in order for Mage's configuration loaders to recognize your secrets. In code you can refer to these keys by their string name or using the `mage_ai.io.config.ConfigKey` enum. Not all keys need be specified at once - only use the keys related to the services you utilize.

| Key Name | Service | Client Constructor Parameter | Description | Notes |
| - | - | - | - | - |
| **AWS General** | | | | |
| AWS\_ACCESS\_KEY\_ID | AWS General | - | AWS Access Key ID credential | Used by [Redshift](#redshift) and [S3](#s3) |
| AWS\_SECRET\_ACCESS\_KEY | AWS General | - | AWS Secret Access Key credential | Used by [Redshift](#redshift) and [S3](#s3) |
| AWS\_SESSION\_TOKEN | AWS General | - | AWS Session Token (used to generate temporary DB credentials) | Used by Redshift |
| AWS\_REGION | AWS General | - | AWS Region | Used by [Redshift](#redshift) and [S3](#s3) |
| AWS\_ENDPOINT | AWS General | - | AWS endpoint URL | Used by [S3](#s3). Optional |
| **AWS Redshift** | | | | |
| REDSHIFT\_DBNAME | [AWS Redshift](#redshift) | database | Name of Redshift database to connect to | |
| REDSHIFT\_HOST | [AWS Redshift](#redshift) | host | Redshift Cluster hostname | Use with temporary credentials |
| REDSHIFT\_PORT | [AWS Redshift](#redshift) | port | Redshift Cluster port. Optional, defaults to 5439. | Use with temporary credentials |
| REDSHIFT\_SCHEMA | [AWS Redshift](#redshift) | schema | Redshift schema name | Optional |
| REDSHIFT\_TEMP\_CRED\_USER | [AWS Redshift](#redshift) | user | Redshift temporary credentials username. | Use with temporary credentials |
| REDSHIFT\_TEMP\_CRED\_PASSWORD | [AWS Redshift](#redshift) | password | Redshift temporary credentials password. | Use with temporary credentials |
| REDSHIFT\_DBUSER | [AWS Redshift](#redshift) | db\_user | Redshift database user to generate credentials for. | Use to generate temporary credentials |
| REDSHIFT\_CLUSTER\_ID | [AWS Redshift](#redshift) | cluster\_identifier | Redshift cluster ID | Use to generate temporary credentials |
| REDSHIFT\_IAM\_PROFILE | [AWS Redshift](#redshift) | profile | Name of the IAM profile to generate temporary credentials with | Use to generate temporary credentials |
| **PostgreSQL** | | | | |
| POSTGRES\_DBNAME | [PostgreSQL](#postgresql) | dbname | Database name | |
| POSTGRES\_USER | [PostgreSQL](#postgresql) | user | Database login username | |
| POSTGRES\_PASSWORD | [PostgreSQL](#postgresql) | password | Database login password | |
| POSTGRES\_HOST | [PostgreSQL](#postgresql) | host | Database hostname | |
| POSTGRES\_PORT | [PostgreSQL](#postgresql) | port | PostgreSQL database port | |
| POSTGRES\_SCHEMA | [PostgreSQL](#postgresql) | schema | PostgreSQL schema name | Optional |
| POSTGRES\_CONNECTION\_METHOD | [PostgreSQL](#postgresql) | connection\_method | Connection method ('direct' or 'ssh\_tunnel') | Optional |
| POSTGRES\_SSH\_HOST | [PostgreSQL](#postgresql) | ssh\_host | SSH tunnel host | Optional |
| POSTGRES\_SSH\_PORT | [PostgreSQL](#postgresql) | ssh\_port | SSH tunnel port | Optional |
| POSTGRES\_SSH\_USERNAME | [PostgreSQL](#postgresql) | ssh\_username | SSH tunnel username | Optional |
| POSTGRES\_SSH\_PASSWORD | [PostgreSQL](#postgresql) | ssh\_password | SSH tunnel password | Optional |
| POSTGRES\_SSH\_PKEY | [PostgreSQL](#postgresql) | ssh\_pkey | SSH tunnel private key path | Optional |
| POSTGRES\_SSL\_MODE | [PostgreSQL](#postgresql) | sslmode | SSL mode | Optional |
| POSTGRES\_SSL\_ROOTCERT | [PostgreSQL](#postgresql) | sslrootcert | SSL root certificate path | Optional |
| POSTGRES\_SSL\_CERT | [PostgreSQL](#postgresql) | sslcert | SSL certificate path | Optional |
| POSTGRES\_SSL\_KEY | [PostgreSQL](#postgresql) | sslkey | SSL key path | Optional |
| POSTGRES\_CONNECT\_TIMEOUT | [PostgreSQL](#postgresql) | connect\_timeout | Connection timeout in seconds | Optional |
| **Snowflake** | | | | |
| SNOWFLAKE\_USER | [Snowflake](#snowflake) | user | Snowflake username | |
| SNOWFLAKE\_PASSWORD | [Snowflake](#snowflake) | password | Snowflake password | |
| SNOWFLAKE\_ACCOUNT | [Snowflake](#snowflake) | account | Snowflake account ID (including region) | |
| SNOWFLAKE\_DEFAULT\_DB | [Snowflake](#snowflake) | database | Default database to use. Optional, no database chosen if unspecified. | |
| SNOWFLAKE\_DEFAULT\_SCHEMA | [Snowflake](#snowflake) | schema | Default schema to use. Optional, no schema chosen if unspecified. | |
| SNOWFLAKE\_DEFAULT\_WH | [Snowflake](#snowflake) | warehouse | Default warehouse to use. Optional, no warehouse chosen if unspecified. | |
| SNOWFLAKE\_PRIVATE\_KEY\_PASSPHRASE | [Snowflake](#snowflake) | private\_key\_passphrase | Private key passphrase for key pair authentication | Optional |
| SNOWFLAKE\_PRIVATE\_KEY\_PATH | [Snowflake](#snowflake) | private\_key\_path | Path to private key file for key pair authentication | Optional |
| SNOWFLAKE\_ROLE | [Snowflake](#snowflake) | role | Snowflake role name | Optional |
| SNOWFLAKE\_TIMEOUT | [Snowflake](#snowflake) | timeout | Query timeout in seconds | Optional |
| **Google BigQuery** | | | | |
| GOOGLE\_SERVICE\_ACC\_KEY | [Google BigQuery](#bigquery) | credentials\_mapping | Service account key | |
| GOOGLE\_SERVICE\_ACC\_KEY\_FILEPATH | [Google BigQuery](#bigquery) | path\_to\_credentials | Path to service account key | |
| GOOGLE\_LOCATION | [Google BigQuery](#bigquery) | location | Google Cloud location | Optional |
| **MySQL** | | | | |
| MYSQL\_DATABASE | [MySQL](#mysql) | database | MySQL database name | |
| MYSQL\_USER | [MySQL](#mysql) | user | MySQL username | |
| MYSQL\_PASSWORD | [MySQL](#mysql) | password | MySQL password | |
| MYSQL\_HOST | [MySQL](#mysql) | host | MySQL hostname | |
| MYSQL\_PORT | [MySQL](#mysql) | port | MySQL port. Defaults to 3306. | |
| MYSQL\_CONNECTION\_METHOD | [MySQL](#mysql) | connection\_method | Connection method ('direct' or 'ssh\_tunnel') | Optional |
| MYSQL\_SSH\_HOST | [MySQL](#mysql) | ssh\_host | SSH tunnel host | Optional |
| MYSQL\_SSH\_PORT | [MySQL](#mysql) | ssh\_port | SSH tunnel port | Optional |
| MYSQL\_SSH\_USERNAME | [MySQL](#mysql) | ssh\_username | SSH tunnel username | Optional |
| MYSQL\_SSH\_PASSWORD | [MySQL](#mysql) | ssh\_password | SSH tunnel password | Optional |
| MYSQL\_SSH\_PKEY | [MySQL](#mysql) | ssh\_pkey | SSH tunnel private key path | Optional |
| **MSSQL** | | | | |
| MSSQL\_DATABASE | [MSSQL](#mssql) | database | MSSQL database name | |
| MSSQL\_USER | [MSSQL](#mssql) | user | MSSQL username | |
| MSSQL\_PASSWORD | [MSSQL](#mssql) | password | MSSQL password | |
| MSSQL\_HOST | [MSSQL](#mssql) | host | MSSQL hostname | |
| MSSQL\_PORT | [MSSQL](#mssql) | port | MSSQL port. Defaults to 1433. | |
| MSSQL\_SCHEMA | [MSSQL](#mssql) | schema | MSSQL schema name. Defaults to 'dbo'. | |
| MSSQL\_AUTHENTICATION | [MSSQL](#mssql) | authentication | Authentication method | Optional |
| MSSQL\_CONNECTION\_METHOD | [MSSQL](#mssql) | connection\_method | Connection method ('direct' or 'ssh\_tunnel') | Optional |
| MSSQL\_SSH\_HOST | [MSSQL](#mssql) | ssh\_host | SSH tunnel host | Optional |
| MSSQL\_SSH\_PORT | [MSSQL](#mssql) | ssh\_port | SSH tunnel port | Optional |
| MSSQL\_SSH\_USERNAME | [MSSQL](#mssql) | ssh\_username | SSH tunnel username | Optional |
| MSSQL\_SSH\_PASSWORD | [MSSQL](#mssql) | ssh\_password | SSH tunnel password | Optional |
| MSSQL\_SSH\_PKEY | [MSSQL](#mssql) | ssh\_pkey | SSH tunnel private key path | Optional |
| MSSQL\_DRIVER | [MSSQL](#mssql) | driver | ODBC driver name | Optional |
| **ClickHouse** | | | | |
| CLICKHOUSE\_HOST | [ClickHouse](#clickhouse) | host | ClickHouse hostname | |
| CLICKHOUSE\_PORT | [ClickHouse](#clickhouse) | port | ClickHouse port | |
| CLICKHOUSE\_USERNAME | [ClickHouse](#clickhouse) | username | ClickHouse username | |
| CLICKHOUSE\_PASSWORD | [ClickHouse](#clickhouse) | password | ClickHouse password | |
| CLICKHOUSE\_DATABASE | [ClickHouse](#clickhouse) | database | ClickHouse database name | |
| CLICKHOUSE\_INTERFACE | [ClickHouse](#clickhouse) | interface | ClickHouse interface | Optional |
| CLICKHOUSE\_SSL\_CA\_CERT | [ClickHouse](#clickhouse) | ca\_cert | Path to CA certificate for SSL | Optional |
| **Trino** | | | | |
| TRINO\_CATALOG | [Trino](#trino) | catalog | Trino catalog name | |
| TRINO\_HOST | [Trino](#trino) | host | Trino hostname | |
| TRINO\_PORT | [Trino](#trino) | port | Trino port. Defaults to 8080. | |
| TRINO\_USER | [Trino](#trino) | user | Trino username | |
| TRINO\_PASSWORD | [Trino](#trino) | password | Trino password | Optional |
| TRINO\_SCHEMA | [Trino](#trino) | schema | Trino schema name | Optional |
| **Databricks SQL** | | | | |
| DATABRICKS\_ACCESS\_TOKEN | [Databricks SQL](#databricks-sql) | access\_token | Databricks access token | |
| DATABRICKS\_HOST | [Databricks SQL](#databricks-sql) | host | Databricks workspace hostname | |
| DATABRICKS\_HTTP\_PATH | [Databricks SQL](#databricks-sql) | http\_path | Databricks SQL warehouse HTTP path | |
| DATABRICKS\_DATABASE | [Databricks SQL](#databricks-sql) | database | Databricks database/catalog name | Optional |
| DATABRICKS\_SCHEMA | [Databricks SQL](#databricks-sql) | schema | Databricks schema name | Optional |
| **Spark** | | | | |
| SPARK\_HOST | [Spark](#spark) | host | Spark master URL | |
| SPARK\_METHOD | [Spark](#spark) | method | Spark connection method | Optional |
| SPARK\_SCHEMA | [Spark](#spark) | database | Spark database/schema name | Optional |
| **MongoDB** | | | | |
| MONGODB\_CONNECTION\_STRING | [MongoDB](#mongodb) | connection\_string | MongoDB connection string | Alternative to individual settings |
| MONGODB\_HOST | [MongoDB](#mongodb) | host | MongoDB hostname | |
| MONGODB\_PORT | [MongoDB](#mongodb) | port | MongoDB port. Defaults to 27017. | |
| MONGODB\_USER | [MongoDB](#mongodb) | user | MongoDB username | |
| MONGODB\_PASSWORD | [MongoDB](#mongodb) | password | MongoDB password | |
| MONGODB\_DATABASE | [MongoDB](#mongodb) | database | MongoDB database name | |
| MONGODB\_COLLECTION | [MongoDB](#mongodb) | collection | MongoDB collection name | Optional |
| **OracleDB** | | | | |
| ORACLEDB\_USER | [OracleDB](#oracledb) | user | OracleDB username | |
| ORACLEDB\_PASSWORD | [OracleDB](#oracledb) | password | OracleDB password | |
| ORACLEDB\_HOST | [OracleDB](#oracledb) | host | OracleDB hostname | |
| ORACLEDB\_PORT | [OracleDB](#oracledb) | port | OracleDB port | |
| ORACLEDB\_SERVICE | [OracleDB](#oracledb) | service | OracleDB service name | |
| ORACLEDB\_MODE | [OracleDB](#oracledb) | mode | OracleDB mode ('thin' or 'thick') | Optional |
| **DuckDB** | | | | |
| DUCKDB\_DATABASE | [DuckDB](#duckdb) | database | DuckDB database path | |
| DUCKDB\_SCHEMA | [DuckDB](#duckdb) | schema | DuckDB schema name. Defaults to 'main'. | Optional |
| MOTHERDUCK\_TOKEN | [DuckDB](#duckdb) | - | MotherDuck authentication token | Optional |
| **SQLite** | | | | |
| SQLITE | [SQLite](#sqlite) | database | SQLite database file path | |
| **Azure** | | | | |
| AZURE\_CLIENT\_ID | Azure Services | client\_id | Azure AD application client ID | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| AZURE\_CLIENT\_SECRET | Azure Services | client\_secret | Azure AD application client secret | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| AZURE\_TENANT\_ID | Azure Services | tenant\_id | Azure AD tenant ID | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| AZURE\_STORAGE\_ACCOUNT\_NAME | Azure Services | storage\_account\_name | Azure Storage account name | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| **Microsoft Fabric** | | | | |
| MICROSOFT\_FABRIC\_WAREHOUSE\_NAME | [Microsoft Fabric](#microsoft-fabric-warehouse) | warehouse\_name | Microsoft Fabric warehouse name | |
| MICROSOFT\_FABRIC\_WAREHOUSE\_ENDPOINT | [Microsoft Fabric](#microsoft-fabric-warehouse) | endpoint | Microsoft Fabric warehouse endpoint | |
| MICROSOFT\_FABRIC\_WAREHOUSE\_SCHEMA | [Microsoft Fabric](#microsoft-fabric-warehouse) | schema | Microsoft Fabric warehouse schema | Optional |

## Configuration Loader APIs

This section contains the exact APIs and more detailed information on the configuration loaders. Every configuration loader has two functions:

* `contains` - checks if the configuration source contains the requested key. Commonly, the `in` operation is used to check for setting existence (but is not always identical as `contains` can accept multiple parameters while the `in` keyword only accepts the key).

  ```python theme={"system"}
  if config.contains(ConfigKey.POSTGRES_PORT):
      ...
  # alternatively
  if ConfigKey.POSTGRES_PORT in config:
      ...
  ```

* `get` - gets the configuration setting associated with the given key. If the key doesn't exist, returns None. Commonly, the data model overload `__getitem__` is used to fetch a configuration setting (but is not always identical as `get` can accept multiple parameters while `__getitem__` does not).

  ```python theme={"system"}
  user = config.get(ConfigKey.REDSHIFT_DBUSER)
  # alternatively
  user = config[ConfigKey.REDSHIFT_DBUSER]
  ```

These functions are shared among all configuration loaders, but depending on the source some function signatures may differ.

### Configuration File

Loads configuration settings from a configuration file.

For detailed information about creating and managing `io_config.yaml` files, see the [IO Config Setup documentation](/development/io_config_setup).

For detailed information about creating and managing `io_config.yaml` files, see the [IO Config Setup documentation](/development/io_config_setup).

**Example**:

```python theme={"system"}
from mage_ai.io.config import ConfigKey, ConfigFileLoader

config = ConfigFileLoader('path/to/my/io_config.yaml', 'my_profile')
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
```

**Constructor**: `__init__(filepath: os.PathLike, profile: str)`
Initializes IO Configuration loader. Input configuration file can have two formats:

* *Standard*: contains a subset of the configuration keys specified in `ConfigKey`. This
  is the default and recommended format. Below is an example configuration file using this format.
  ```yaml theme={"system"}
  version: 0.1.0
  default:
      AWS_ACCESS_KEY_ID: AWS Access Key ID credential
      AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
      AWS_REGION: AWS Region
      REDSHIFT_DBNAME: Name of Redshift database to connect to
      REDSHIFT_HOST: Redshift Cluster hostname
      REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
      REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
      REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
  ```
  The above configuration file has a single **profile** named `'default'`. Each profile organizes a set of keys to use (for example, distinguishing production keys versus development keys). A configuration file can have multiple profiles.
* *Verbose*: Instead of configuration keys, each profile stores an object of settings associated with
  each data migration client. This format was used in previous versions of this tool, and exists
  for backwards compatibility. Below is an example configuration file using this format.
  ```yaml theme={"system"}
  version: 0.0.0
  default:
      AWS:
          Redshift:
              database: Name of Redshift database to connect to
              host: Redshift Cluster hostname
              port: Redshift Cluster port. Optional, defaults to 5439
              user: Redshift temp credentials username
              password: Redshift temp credentials password
          access_key_id: AWS Access Key ID credential
          secret_access_key: AWS Secret Access Key credential
          region: AWS Region
  ```

Use handlebars and `env_var` syntax to reference environment variables in either configuration file format.

```yaml theme={"system"}
version: 0.1.0
default:
    GOOGLE_SERVICE_ACC_KEY_FILEPATH: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
```

**Args**:

* `filepath (os.PathLike, optional)`: Path to IO configuration file. Defaults to `'[repo_path]/io_config.yaml'`
* `profile (str, optional)`: Profile to load configuration settings from. Defaults to `'default'`.

**Methods**

***contains*** - `contains(self, key: ConfigKey | str) -> Any`

Checks if the configuration setting stored under `key` is contained.

* **Args**:
  * `key (str)`: Name of the configuration setting to check.
* **Returns** (`bool`) Returns `True` if configuration setting exists, otherwise returns `False`.

***get*** - `get(self, key: ConfigKey | str) -> Any`

Loads the configuration setting stored under `key`.

* **Args**:
  * `key (str)`: Key name of the configuration setting to load
* **Returns**: (`Any`) Configuration setting corresponding to the given key

### Environment Variables

Loads configuration settings from environment variables in your current environment.

**Example**:

```python theme={"system"}
from mage_ai.io.config import ConfigKey, EnvironmentVariableLoader

config = EnvironmentVariableLoader()
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
```

**Constructor** : `__init__(self)` - no parameters for construction.

**Methods**:

***contains*** - `contains(env_var: ConfigKey | str) -> bool`

Checks if the environment variable given by `env_var` exists.

* **Args**:

  * `key (ConfigKey | str)`: Name of the configuration setting to check existence of.

* **Returns** (`bool`) Returns `True` if configuration setting exists, otherwise returns `False`.

***get*** - `get(env_var: ConfigKey | str) -> Any`

Loads the config setting stored under the environment variable `env_var`.

* **Args**:

  * `env_var (str)`: Name of the environment variable to load configuration setting from

* **Returns**: (`Any`) The configuration setting stored under `env_var`

### AWS Secret Loader

Loads secrets from AWS Secrets Manager. To authenticate access to AWS Secrets Manager, either

* Configure your AWS profile using the AWS CLI
  ```bash theme={"system"}
  aws configure
  ```
* Manually specify your AWS Credentials when constructing the configuration loader
  `python config = AWSSecretLoader( aws_access_key_id = 'your access key id', aws_secret_access_key = 'your secret key', region_name = 'your region' ) `

**Example**:

```python theme={"system"}
from mage_ai.io.config import ConfigKey, AWSSecretLoader

config = AWSSecretLoader()
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
# with finer control on version
postgres_db = config.get(ConfigKey.POSTGRES_DBNAME, version_id='my_version_id')
```

**Constructor** : `__init__(self, **kwargs)`:

* **Keyword Arguments**:
  * `aws_access_key_id (str, Optional)`: AWS access key ID credential.
  * `aws_secret_access_key (str, Optional)`: AWS secret access key credential.
  * `region_name (str, Optional)`: AWS region which Secrets Manager is created in.

**Methods**:

***contains*** - `contains( secret_id: ConfigKey | str, version_id: str, version_stage_label : str) -> bool`
Check if there is a secret with ID `secret_id` contained. Can also specify the version of the
secret to check. If

* both `version_id` and `version_stage_label` are specified, both must agree on the secret version.
* neither of `version_id` or `version_stage_label` are specified, any version is checked.
* one of `version_id` and `version_stage_label` are specified, the associated version is checked.

When using the `in` operator, comparisons to specific versions are not allowed.

* **Args**:
  * `secret_id (str)`: ID of the secret to load
  * `version_id (str, Optional)`: ID of the version of the secret to load. Defaults to `None`.
  * `version_stage_label (str, Optional)`: Staging label of the version of the secret to load. Defaults to `None`.
* **Returns**: (`bool`) Returns true if secret exists, otherwise returns false.

***get*** - `get(secret_id: ConfigKey | str, version_id: str, version_stage_label : str) -> bytes | str`
Loads the secret stored under `secret_id`. Can also specify the version of the
secret to fetch. If

* both `version_id` and `version_stage_label` are specified, both must agree on the secret version.
* neither of `version_id` or `version_stage_label` are specified, the current version is loaded.
* one of `version_id` and `version_stage_label` are specified, the associated version is loaded.

When using the `__getitem__` overload, comparisons to specific versions are not allowed.

* **Args**:

  * `secret_id (str)`: ID of the secret to load
  * ` version_id (str, Optional)`: ID of the version of the secret to load. Defaults to `None`.
  * `version_stage_label (str, Optional)`: Staging label of the version of the secret to load. Defaults to `None`.

* **Returns**: (`bytes | str`) The secret stored under `secret_id` in AWS secret manager. If secret is a binary value, returns a `bytes` object; else returns a `string` object


# Pipeline Management
Source: https://docs.mage.ai/design/data-pipeline-management



![Mage orchestration](https://mage-ai.github.io/assets/orchestration-overview.gif)

## Setup

If you haven’t setup a project before, check out the
[setup guide](/getting-started/setup) before starting.

***

## Pipelines

<sub>`http://localhost:3000/pipelines`</sub>

This page will show all the pipelines in your project.

> Core abstraction: [Pipeline](/design/core-abstractions#pipeline)
>
> A pipeline contains references to all the blocks of code you want to run,
> charts for visualizing data, and organizes the dependency between each block
> of code.

![Pipelines](https://mage-ai.github.io/assets/pipelines/pipelines-index.jpg)

Learn more about
[**projects and pipelines here**](/design/core-abstractions#project).

From this page, you can also create a new pipeline by clicking the

**`[+ New pipeline]`** button.

#### Creating new pipeline

Creating a new pipeline will take you to the **Pipeline edit page**; a
notebook-like experience for adding blocks, creating dependencies between
blocks, testing code, and visualizing data with charts.

<sub>
  Learn more about the [Notebook for building data
  pipelines](/about/features#notebook-for-building-data-pipelines)
</sub>

***

## Pipeline runs

<sub>
  [`http://localhost:3000/pipeline-runs`](http://localhost:3000/pipeline-runs)
</sub>

View all the runs for every pipeline in your current project.

In Mage Pro, select **Custom UTC range** to filter runs by a specific time
range. Edit the year, month, day, hour, and minute directly in the
`YYYY-MM-DD HH:mm` input, or use the calendar and time picker. The input keeps
the separators in place, shows placeholders for missing digits, and restricts
each segment to its allowed range. Complete valid values update the picker,
and picker selections update the input. Complete both inputs with a valid range
before applying it.
Dates run from 1970 through the end of the current UTC year plus ten years.

> Core abstraction: [Run](/design/core-abstractions#run)
>
> A run record stores information about when it was started, its status, when it
> was completed, any runtime variables used in the execution of the pipeline or
> block, etc.

![Pipeline runs](https://mage-ai.github.io/assets/pipelines/pipeline-runs.jpg)

***

## Pipeline detail

<sub>`http://localhost:3000/pipelines/[uuid]`</sub>

This page contains all the information and history for a single pipeline:

1. [Triggers](/design/core-abstractions#trigger)
2. [Runs](/design/core-abstractions#run)
3. [Logs](/design/core-abstractions#log)

***

## Triggers

<sub>
  [`http://localhost:3000/pipelines/example_pipeline/triggers`](http://localhost:3000/pipelines/example_pipeline/triggers)
</sub>

This page shows all the active and inactive triggers for a single pipeline.

> Core abstraction: [Trigger](/design/core-abstractions#trigger)
>
> A trigger is a set of instructions that determine when or how a pipeline
> should run.

![Pipeline detail](https://mage-ai.github.io/assets/pipelines/pipeline-detail.jpg)

***

#### Create trigger

<sub>`http://localhost:3000/pipelines/[uuid]/triggers/[id]/edit`</sub>

Create a new trigger for this pipeline by clicking the **`[+ Create]`** button
near the top of the page.

You can configure the trigger to run the pipeline on a schedule, when an event
occurs, or when an API is called.

> Core abstraction: [Schedule](/design/core-abstractions#schedule)
>
> A schedule type trigger will instruct the pipeline to run after a start date
> and on a set interval.

***

> Core abstraction: [Event](/design/core-abstractions#event)
>
> An event type trigger will instruct the pipeline to run whenever a specific
> event occurs.

***

> Core abstraction: [API](/design/core-abstractions#api)
>
> An API trigger will instruct the pipeline to run whenever a specific API
> is called.

![Trigger create](https://mage-ai.github.io/assets/pipelines/trigger-create.jpg)

<sub>
  *Example page for creating a schedule type trigger.*
</sub>

***

#### Trigger detail

On this page, you can start or pause the trigger. Starting the trigger will make
it active. Pausing the trigger will prevent it from running the pipeline.

> [!NOTE]
>
> If you have other triggers for this pipeline, pausing 1 trigger may not stop
> the pipeline from running since other triggers can also run the pipeline.

***

![Trigger detail](https://mage-ai.github.io/assets/pipelines/trigger-detail.jpg)

You can also edit the trigger after creating it by clicking the

**`[Edit trigger]`** button.

***

## Runs

<sub>
  [`http://localhost:3000/pipelines/example_pipeline/runs`](http://localhost:3000/pipelines/example_pipeline/runs)
</sub>

View the pipeline runs and block runs for a pipeline.

> Core abstraction: [Run](/design/core-abstractions#run)
>
> A run record stores information about when it was started, its status, when it
> was completed, any runtime variables used in the execution of the pipeline or
> block, etc.

![Pipeline detail runs](https://mage-ai.github.io/assets/pipelines/pipeline-detail-runs.jpg)

#### Retry run

*WIP*

***

## Logs

<sub>
  [`http://localhost:3000/pipelines/example_pipeline/logs`](http://localhost:3000/pipelines/example_pipeline/logs)
</sub>

Browse all logs for a pipeline. You can search and filter logs by log level,
block type, block UUID, and more.

> Core abstraction: [**Log**](/design/core-abstractions#log)
>
> A log is a file that contains system output information.

![Pipeline detail logs](https://mage-ai.github.io/assets/pipelines/pipeline-detail-logs.jpg)

***

## Backfill

*WIP*

***

## Monitor

You can monitor many metrics for each of your pipelines and blocks. Soon, you’ll
be able to monitor aggregate metrics across all pipelines and blocks.

![Monitoring](https://mage-ai.github.io/assets/monitoring.jpg)

Read more [here](/development/observability/monitoring).

## Concurrency

You can limit the concurrency of the block execution to reduce resource consumption.

### Global concurrency

You can configure the maximum number of concurrent block runs in project’s `metadata.yaml` via `queue_config`.

```yaml theme={"system"}
queue_config:
  concurrency: 100
```

The default value of concurrency is 20.

### Global concurrency by block uuid

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=concurrency)

You can configure the global block run concurrency by block uuid in project's `metadata.yaml` via `concurrency_config`.

**Example**

```yaml theme={"system"}
concurrency_config:
  block_run_limit_per_block_uuid:
    block_uuid_1: 1
    block_uuid_2: 5
```

In the example above:

* `block_uuid_1` can run at most 1 concurrent execution.
* `block_uuid_2` can run up to 5 concurrent executions.

**Limitations**

This feature is not supported in the following scenarios:

* Replica blocks: Each replica is assigned a unique UUID for block run, so concurrency limits are not enforced across replicas.
* Dynamic blocks: Each dynamic child block has a unique UUID for block run, preventing concurrency limits from applying across them.
* Single-process mode: When `run_pipeline_in_one_process` is set to `true` in the pipeline’s `metadata.yaml`, global concurrency is not enforced.

### Pipeline level concurrency

You can edit the `concurrency_config` in each pipeline's metadata.yaml file to enforce pipeline level concurrency.
Here is the example:

```yaml theme={"system"}
concurrency_config:
  block_run_limit: 5
  pipeline_run_limit: 3
  pipeline_run_priority: 10
```

The default value of `block_run_limit` and `pipeline_run_limit` can be set via environment variables: `CONCURRENCY_CONFIG_BLOCK_RUN_LIMIT` and `CONCURRENCY_CONFIG_PIPELINE_RUN_LIMIT`

* `block_run_limit`: limit the concurrent blocks runs in one pipeline run.
* `pipeline_run_limit`: limit the concurrent pipeline runs in one pipeline trigger.
* `pipeline_run_limit_all_triggers`: limit the concurrent pipeline runs across all triggers in a pipeline.
* `pipeline_run_priority` (Mage Pro only): set the priority for this pipeline's runs when run limits are reached.
  Higher values start first, and runs with the same priority use execution date order.
  The default value is `0`. You can edit this in Pipeline Settings as **Pipeline priority**.
* `on_pipeline_run_limit_reached`: choose whether to `wait` or `skip` when the pipeline run limit is reached.

## Variable storage

Mage automatically persists the output of block runs on disk. You can specify the path or the storage for block output variables
in the following ways.

* Specify the data directory path via `MAGE_DATA_DIR` environment variable. If you use Mage docker image, this environment
  variable is set to `/home/src/mage_data` by default.
* If the `MAGE_DATA_DIR` environment variable is not set, you can set the `variables_dir` path in project's metadata.yaml. Here is
  an example:
  ```yaml theme={"system"}
  variables_dir: /home/src/mage_data
  ```
* You can also use an external storage to store the block output variables by specifying the `remote_variables_dir` path in project's
  metadata.yaml.
  * AWS S3 storage:
    ```yaml theme={"system"}
    variables_dir: /home/src/mage_data
    remote_variables_dir: s3://bucket/path_prefix
    ```
  * Google Cloud Storage:
    ```yaml theme={"system"}
    variables_dir: /home/src/mage_data
    remote_variables_dir: gs://bucket/path_prefix
    ```
    When using GCS for your remote variables directory, if you run into a "Your default credentials were not found"
    error, you may need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of your
    `application_default_credentials.json` file with your Google Cloud credentials and the `GOOGLE_CLOUD_PROJECT`
    environment variable to your Google Cloud project ID.

## Variable retention

If you want to clean up the old variables in your variable storage, you can set the `variables_retention_period`
config in project's metadata.yaml. The valid period should end with "d", "h", or "w".

Example config:

```yaml theme={"system"}
variables_retention_period: 30d
```

After configuring `variables_retention_period` in project's metadata.yaml, you can run the following command
to clean up old variables:

```bash theme={"system"}
mage clean-cached-variables [project_path]
```

## Cache block output in memory

By default, Mage persists block output on disk. In pipeline's metadata.yaml, you have the option to configure the pipeline to cache
the block output in memory instead of persisting the block output on disk. The feature is only supported in standard batch
pipeline (without dynamic blocks) for now.

Example config:

```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```

***

## Operation history

Keeps track of recently viewed pipelines so you can easily navigate back to them
in the Pipelines Dashboard (`/pipelines`). Adds a `RECENTLY VIEWED` tab to the Pipelines
Dashboard that lists these pipelines.


# Callback templates
Source: https://docs.mage.ai/development/blocks/callbacks/templates



## Base

```python theme={"system"}
if 'callback' not in globals():
    from mage_ai.data_preparation.decorators import callback


@callback('success')
def success_callback(**kwargs):
    pass


@callback('failure')
def failure_callback(**kwargs):
    pass
```


# Conditional templates
Source: https://docs.mage.ai/development/blocks/conditionals/templates



## Base

```python theme={"system"}
if 'condition' not in globals():
    from mage_ai.data_preparation.decorators import condition


@condition
def evaluate_condition(*args, **kwargs) -> bool:
    return True
```


# Data exporter templates
Source: https://docs.mage.ai/development/blocks/data_exporters/templates



## Generic

```python theme={"system"}
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(*args, **kwargs):
    """
    Exports data to some source

    Args:
        args: The input variables from upstream blocks

    Output (optional):
        Optionally return any object and it'll be logged and
        displayed when inspecting the block run.
    """
    # Specify your data exporting logic here
```

## Local file

```python theme={"system"}
from mage_ai.io.file import FileIO
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_file(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to filesystem.

    Docs: /design/data-loading#fileio
    """
    filepath = 'path/to/write/dataframe/to.csv'
    FileIO().export(df, filepath)
```

## Azure Blob Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_azure_blob_storage(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Azure Blob Storage.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    container_name = 'your_container_name'
    blob_path = 'your_blob_path'

    AzureBlobStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        container_name,
        blob_path,
    )
```

## Google BigQuery

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_big_query(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a BigQuery warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#bigquery
    """
    table_id = 'your-project.your_dataset.your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        table_id,
        if_exists='replace',  # Specify resolution policy if table name already exists
    )
```

## Google Cloud Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Google Cloud Storage bucket.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#googlecloudstorage
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    object_key = 'your_object_key'

    GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        bucket_name,
        object_key,
    )
```

## MySQL

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a MySQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#mysql
    """
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            None,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
        )
```

## PostgreSQL

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a PostgreSQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#postgresql
    """
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
        )
```

## Amazon Redshift

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_redshift(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Redshift cluster.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#redshift
    """
    table_name = 'your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            table_name,
            if_exists='replace',  # Specify resolution policy if table already exists
        )
```

## Amazon S3

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_s3(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a S3 bucket.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#s3
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    object_key = 'your_object_key'

    S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        bucket_name,
        object_key,
    )
```

## Snowflake

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Snowflake warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#snowflake
    """
    table_name = 'your_table_name'
    database = 'your_database_name'
    schema = 'your_schema_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            table_name,
            database,
            schema,
            if_exists='replace',  # Specify resolution policy if table already exists
        )
```

## Delta Lake

### Amazon S3

```python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(df, *args, **kwargs):
    """
    Export data to a Delta Table

    Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
    """
    storage_options = {
        'AWS_ACCESS_KEY_ID': '',
        'AWS_SECRET_ACCESS_KEY': '',
        'AWS_REGION': '',
        'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
    }

    uri = 's3://[bucket]/[key]'

    write_deltalake(
        uri,
        data=df,
        mode='append',          # append or overwrite
        overwrite_schema=False, # set True to alter the schema when overwriting
        partition_by=[],
        storage_options=storage_options,
    )
```

### Azure Blob Storage

```python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(df, *args, **kwargs):
    """
    Export data to a Delta Table

    Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
    """
    storage_options = {
        'AZURE_STORAGE_ACCOUNT_NAME': '',
        'AZURE_STORAGE_ACCOUNT_KEY': '',
        'AZURE_STORAGE_ACCESS_KEY': '',
        'AZURE_STORAGE_MASTER_KEY': '',
        'AZURE_STORAGE_CLIENT_ID': '',
        'AZURE_STORAGE_CLIENT_SECRET': '',
        'AZURE_STORAGE_TENANT_ID': '',
        'AZURE_STORAGE_SAS_KEY': '',
        'AZURE_STORAGE_TOKEN': '',
        'AZURE_STORAGE_USE_EMULATOR': '',
    }

    uri = 'az://[container]/[key]'

    write_deltalake(
        uri,
        data=df,
        mode='append',          # append or overwrite
        overwrite_schema=False, # set True to alter the schema when overwriting
        partition_by=[],
        storage_options=storage_options,
    )
```

### Google Cloud Storage

```python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(df, *args, **kwargs):
    """
    Export data to a Delta Table

    Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
    """
    storage_options = {
        'GOOGLE_SERVICE_ACCOUNT': '',
        'GOOGLE_SERVICE_ACCOUNT_PATH': '',
        'GOOGLE_SERVICE_ACCOUNT_KEY': '',
        'GOOGLE_BUCKET': '',
    }

    uri = 'gs://[bucket]/[key]'

    write_deltalake(
        uri,
        data=df,
        mode='append',          # append or overwrite
        overwrite_schema=False, # set True to alter the schema when overwriting
        partition_by=[],
        storage_options=storage_options,
    )
```

## Orchestration

### Trigger pipeline

```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def trigger(*args, **kwargs):
    """
    Trigger another pipeline to run.

    Documentation: /orchestration/triggers/trigger-pipeline
    """

    trigger_pipeline(
        'pipeline_uuid',        # Required: enter the UUID of the pipeline to trigger
        variables={},           # Optional: runtime variables for the pipeline
        check_status=False,     # Optional: poll and check the status of the triggered pipeline
        error_on_failure=False, # Optional: if triggered pipeline fails, raise an exception
        poll_interval=60,       # Optional: check the status of triggered pipeline every N seconds
        poll_timeout=None,      # Optional: raise an exception after N seconds
        verbose=True,           # Optional: print status of triggered pipeline run
    )
```


# Data loader templates
Source: https://docs.mage.ai/development/blocks/data_loaders/templates



## Generic

```python theme={"system"}
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data(*args, **kwargs):
    """
    Template code for loading data from any source.

    Returns:
        Anything (e.g. data frame, dictionary, array, int, str, etc.)
    """
    # Specify your data loading logic here

    return {}


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Local file

```python theme={"system"}
from mage_ai.io.file import FileIO
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_file(*args, **kwargs):
    """
    Template for loading data from filesystem.
    Load data from 1 file or multiple file directories.

    For multiple directories, use the following:
        FileIO().load(file_directories=['dir_1', 'dir_2'])

    Docs: /design/data-loading#fileio
    """
    filepath = 'path/to/your/file.csv'

    return FileIO().load(filepath)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## API

```python theme={"system"}
import io
import pandas as pd
import requests
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_api(*args, **kwargs):
    """
    Template for loading data from API
    """
    url = ''
    response = requests.get(url)

    return pd.read_csv(io.StringIO(response.text), sep=',')


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Azure Blob Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_from_azure_blob_storage(*args, **kwargs):
    """
    Template for loading data from a Azure Blob Storage.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#azureblobstorage
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    container_name = 'your_container_name'
    blob_path = 'your_blob_path'

    return AzureBlobStorage.with_config(ConfigFileLoader(config_path, config_profile)).load(
        container_name,
        blob_path,
    )


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Google BigQuery

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_big_query(*args, **kwargs):
    """
    Template for loading data from a BigQuery warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#bigquery
    """
    query = 'your_gbq_query'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    return BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Google Cloud Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_from_google_cloud_storage(*args, **kwargs):
    """
    Template for loading data from a Google Cloud Storage bucket.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#googlecloudstorage
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    object_key = 'your_object_key'

    return GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).load(
        bucket_name,
        object_key,
    )


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## MySQL

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_mysql(*args, **kwargs):
    """
    Template for loading data from a MySQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#mysql
    """
    query = 'Your MySQL query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## PostgreSQL

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_postgres(*args, **kwargs):
    """
    Template for loading data from a PostgreSQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#postgresql
    """
    query = 'your PostgreSQL query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Amazon Redshift

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_redshift(*args, **kwargs):
    """
    Template for loading data from a Redshift cluster.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#redshift
    """
    query = 'your_redshift_selection_query'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Amazon S3

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_from_s3_bucket(*args, **kwargs):
    """
    Template for loading data from a S3 bucket.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#s3
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    object_key = 'your_object_key'

    return S3.with_config(ConfigFileLoader(config_path, config_profile)).load(
        bucket_name,
        object_key,
    )


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Snowflake

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_snowflake(*args, **kwargs):
    """
    Template for loading data from a Snowflake warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: /design/data-loading#example-loading-data-from-snowflake-warehouse
    """
    query = 'your_snowflake_query'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Delta Lake

### Amazon S3

```python theme={"system"}
from deltalake import DeltaTable
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_table(*args, **kwargs):
    """
    Load a Delta Table

    Docs: https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table
    """
    storage_options = {
        'AWS_ACCESS_KEY_ID': '',
        'AWS_SECRET_ACCESS_KEY': '',
        'AWS_REGION': '',
        'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
    }

    uri = 's3://[bucket]/[key]'

    dt = DeltaTable(uri, storage_options=storage_options)

    return dt.to_pandas()


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

### Azure Blob Storage

```python theme={"system"}
from deltalake import DeltaTable
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_table(*args, **kwargs):
    """
    Load a Delta Table

    Docs: https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table
    """
    storage_options = {
        'AZURE_STORAGE_ACCOUNT_NAME': '',
        'AZURE_STORAGE_ACCOUNT_KEY': '',
        'AZURE_STORAGE_ACCESS_KEY': '',
        'AZURE_STORAGE_MASTER_KEY': '',
        'AZURE_STORAGE_CLIENT_ID': '',
        'AZURE_STORAGE_CLIENT_SECRET': '',
        'AZURE_STORAGE_TENANT_ID': '',
        'AZURE_STORAGE_SAS_KEY': '',
        'AZURE_STORAGE_TOKEN': '',
        'AZURE_STORAGE_USE_EMULATOR': '',
    }

    uri = 'az://[container]/[key]'

    dt = DeltaTable(uri, storage_options=storage_options)

    return dt.to_pandas()


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

### Google Cloud Storage

```python theme={"system"}
from deltalake import DeltaTable
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_table(*args, **kwargs):
    """
    Load a Delta Table

    Docs: https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table
    """
    storage_options = {
        'GOOGLE_SERVICE_ACCOUNT': '',
        'GOOGLE_SERVICE_ACCOUNT_PATH': '',
        'GOOGLE_SERVICE_ACCOUNT_KEY': '',
        'GOOGLE_BUCKET': '',
    }

    uri = 'gs://[bucket]/[key]'

    dt = DeltaTable(uri, storage_options=storage_options)

    return dt.to_pandas()


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Druid

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.druid import Druid
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_druid(*args, **kwargs):
    """
    Template for loading data from a Druid warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#druid
    """
    query = 'your Druid query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Druid.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Pinot

```python theme={"system"}
from mage_ai.data_preparation.repo_manager import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.pinot import Pinot
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_pinot(*args, **kwargs):
    """
    Template for loading data from a Pinot warehouse.
    Specify your configuration settings in 'io_config.yaml'.
    Docs: https://docs.mage.ai/design/data-loading#pinot
    """
    query = 'your Pinot query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Pinot.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

## Orchestration

### Trigger pipeline

```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def trigger(*args, **kwargs):
    """
    Trigger another pipeline to run.

    Documentation: /orchestration/triggers/trigger-pipeline
    """

    trigger_pipeline(
        'pipeline_uuid',        # Required: enter the UUID of the pipeline to trigger
        variables={},           # Optional: runtime variables for the pipeline
        check_status=False,     # Optional: poll and check the status of the triggered pipeline
        error_on_failure=False, # Optional: if triggered pipeline fails, raise an exception
        poll_interval=60,       # Optional: check the status of triggered pipeline every N seconds
        poll_timeout=None,      # Optional: raise an exception after N seconds
        verbose=True,           # Optional: print status of triggered pipeline run
    )
```


# Sensor templates
Source: https://docs.mage.ai/development/blocks/sensors/templates



## Generic

```python theme={"system"}
from mage_ai.orchestration.run_status_checker import check_status

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def check_condition(**kwargs) -> bool:
    """
    Template code for checking if block or pipeline run completed.
    """
    return check_status(
        'pipeline_uuid',
        kwargs['execution_date'],
        block_uuid='block_uuid',  # optional if you want the sensor to wait on a specific block
        hours=24,  # optional if you want to check for a specific number of hours. Default is 24
    )
```

## Google BigQuery

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def query_bigquery_and_check_condition(**kwargs) -> bool:
    """
    Template code for checking the results of a BigQuery query.
    Specify your configuration settings in 'io_config.yaml'.

    Return: True if the sensor should complete, False if it should
    keep waiting
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your BigQuery query'  # Specify your SQL query here

    loader = BigQuery.with_config(ConfigFileLoader(config_path, config_profile))
    df = loader.load(query)

    # Add your checks here
    if df.empty:
        return False

    return True
```

## MySQL

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def query_mysql_and_check_condition(**kwargs) -> bool:
    """
    Template code for checking the results of a MySQL query.
    Specify your configuration settings in 'io_config.yaml'.

    Return: True if the sensor should complete, False if it should
    keep waiting
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your MySQL query'  # Specify your SQL query here

    with MySQL.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```

## PostgreSQL

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def query_postgres_and_check_condition(**kwargs) -> bool:
    """
    Template code for checking the results of a Postgres query.
    Specify your configuration settings in 'io_config.yaml'.

    Return: True if the sensor should complete, False if it should
    keep waiting
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your Postgres query'  # Specify your SQL query here

    with Postgres.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```

## Amazon Redshift

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def query_redshift_and_check_condition(**kwargs) -> bool:
    """
    Template code for checking the results of a Redshift query.
    Specify your configuration settings in 'io_config.yaml'.

    Return: True if the sensor should complete, False if it should
    keep waiting
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your Redshift query'  # Specify your SQL query here

    with Redshift.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```

## Amazon S3

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def check_condition(**kwargs) -> bool:
    """
    Template code for checking if a file or folder exists in a S3 bucket

    You will also need to fill out the following AWS related fields
    in `io_config.yaml`:
        - AWS_ACCESS_KEY_ID
        - AWS_SECRET_ACCESS_KEY
        - AWS_REGION
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    s3_path = 'path/to/folder/or/file'

    config_file_loader = ConfigFileLoader(config_path, config_profile)
    return S3.with_config(config_file_loader).exists(
        bucket_name, s3_path
    )
```

## Google Cloud Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def check_condition(*args, **kwargs) -> bool:
    """
    Template code for checking if a file or folder exists in a Google Cloud Storage bucket.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#googlecloudstorage
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = 'your_bucket_name'
    object_key = 'your_object_key'

    return GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).exists(
        bucket_name,
        object_key,
    )
```

## Snowflake

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path

if 'sensor' not in globals():
    from mage_ai.data_preparation.decorators import sensor


@sensor
def query_snowflake_and_check_condition(**kwargs) -> bool:
    """
    Template code for checking the results of a Snowflake query.
    Specify your configuration settings in 'io_config.yaml'.

    Return: True if the sensor should complete, False if it should
    keep waiting
    """

    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = 'Your Snowflake query'  # Specify your SQL query here

    with Snowflake.with_config(
            ConfigFileLoader(config_path, config_profile)) as loader:
        df = loader.load(query)

        # Add your checks here
        if df.empty:
            return False

    return True
```


# Transformer templates
Source: https://docs.mage.ai/development/blocks/transformers/templates



## Generic

```python theme={"system"}
if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@transformer
def transform(*args, **kwargs):
    """
    Template code for a transformer block.

    Add more parameters to this function if this block has multiple parent blocks.
    There should be one parameter for each output variable from each parent block.

    Args:
        args: The input variables from upstream blocks

    Returns:
        Anything (e.g. data frame, dictionary, array, int, str, etc.)
    """
    # Specify your transformation logic here

    return {}


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```


# CLI Commands Reference
Source: https://docs.mage.ai/development/cli-commands

Complete reference guide for all Mage CLI commands, including usage instructions and OSS vs Pro availability.

Mage provides a command-line interface (CLI) for managing projects, running pipelines, and performing administrative tasks. The CLI is built using [Typer](https://typer.tghoward.com/) and supports both interactive and scripted usage.

## Quick Reference

### Command Availability

| Command | Description | OSS | Pro |
| - | - | - | - |
| [`init`](#init-initialize-mage-project) | Initialize a new Mage project at the specified path | ✅ | ✅ |
| [`start`](#start-start-mage-server-and-ui) | Start the Mage server and web UI | ✅ | ✅ |
| [`run`](#run-run-pipeline) | Execute a pipeline or a specific block within a pipeline | ✅ | ✅ |
| [`clean-cached-variables`](#clean-cached-variables-clean-cached-variables) | Clean up old pipeline run outputs based on retention period | ✅ | ✅ |
| [`clean-old-logs`](#clean-old-logs-clean-old-logs) | Clean up old pipeline run logs based on retention period | ✅ | ✅ |
| [`clean-workspaces`](#clean-workspaces-clean-workspaces) | Clean up old pipeline run outputs and logs inside workspace pods | ❌ | ✅ |
| [`clean-unused-blocks`](#clean-unused-blocks-clean-unused-blocks) | Clean up unused blocks from the project (interactive) | ❌ | ✅ |
| [`create-secret`](#create-secret-create-or-update-secret) | Create or update a secret for the given Mage project | ❌ | ✅ |
| [`run-post-start-hooks`](#run-post-start-hooks-execute-post-start-hooks) | Execute post-start hooks with graceful error handling | ❌ | ✅ |
| [`restart-jobs`](#restart-jobs-restart-failed-or-cancelled-pipeline-runs) | Restart failed or cancelled pipeline runs with optional filters | ❌ | ✅ |
| [`update-workspace`](#update-workspace-update-workspace-configuration) | Update a workspace configuration synchronously | ❌ | ✅ |
| [`update-all-workspaces`](#update-all-workspaces-update-all-workspaces) | Update all workspaces in the cluster synchronously | ❌ | ✅ |
| [`re-add-workspace-routes`](#re-add-workspace-routes-re-add-workspace-routes-to-ingress) | Re-add workspace routes to the ingress (Kubernetes) | ❌ | ✅ |
| [`recover-pipelines-from-file-versions`](#recover-pipelines-from-file-versions-recover-pipelines-from-file-versions) | Recover pipelines from file versions in `.file_versions` folder | ❌ | ✅ |

### Getting Help

To see all available commands:

```bash theme={"system"}
mage --help
```

To get help for a specific command:

```bash theme={"system"}
mage <command> --help
```

***

## Command Categories

* **[Project Management](#project-management)** - Initialize and start Mage projects
* **[Pipeline Execution](#pipeline-execution)** - Run pipelines and blocks
* **[Maintenance Commands](#maintenance-commands)** - Clean up variables and logs
* **[Mage Pro Only Commands](#mage-pro-only-commands)** - Workspace management and advanced features

***

## Available Commands

### Project Management

#### `init` - Initialize Mage Project

**Available in:** ✅ OSS, ✅ Pro

Initialize a new Mage project at the specified path.

**Usage:**

```bash theme={"system"}
mage init <project_path> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project to be created
* `--project-type` (optional): Type of project to create. Options: `main`, `sub`, or `standalone` (default: `standalone`)
* `--cluster-type` (optional): Type of instance to create for workspace management (Pro only)
* `--project-uuid` (optional): Project UUID for the new project

**Example:**

```bash theme={"system"}
mage init my_project --project-type standalone
```

***

#### `start` - Start Mage Server and UI

**Available in:** ✅ OSS, ✅ Pro

Start the Mage server and web UI.

**Usage:**

```bash theme={"system"}
mage start [project_path] [OPTIONS]
```

**Parameters:**

* `project_path` (optional): Path of the Mage project to be loaded (default: current directory)
* `--host` (optional): Specify the host (default: `localhost`)
* `--port` (optional): Specify the port (default: `6789`)
* `--manage-instance` (optional): Enable manage instance (default: `0`)
* `--dbt-docs-instance` (optional): Enable dbt docs instance (default: `0`)
* `--instance-type` (optional): Specify the instance type (default: `server_and_scheduler`)
* `--project-type` (optional): Create project of this type if it does not exist. Options: `main`, `sub`, or `standalone` (default: `standalone`)
* `--cluster-type` (optional): Type of instance to create for workspace management (Pro only)
* `--project-uuid` (optional): Set project UUID for the repo that is being started

**Example:**

```bash theme={"system"}
mage start . --host 0.0.0.0 --port 6789
```

***

### Pipeline Execution

#### `run` - Run Pipeline

**Available in:** ✅ OSS, ✅ Pro

Execute a pipeline or a specific block within a pipeline.

**Usage:**

```bash theme={"system"}
mage run <project_path> <pipeline_uuid> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project that contains the pipeline
* `pipeline_uuid` (required): UUID of the pipeline to be run
* `--test` (optional): Specify if tests should be run (default: `False`)
* `--block-uuid` (optional): UUID of the block to be run (if not specified, runs entire pipeline)
* `--execution-partition` (optional): Execution partition identifier
* `--executor-type` (optional): Executor type to use
* `--callback-url` (optional): Callback URL for block execution
* `--block-run-id` (optional): Block run ID
* `--pipeline-run-id` (optional): Pipeline run ID
* `--runtime-vars` (optional): Runtime variables in JSON format. These will overwrite the pipeline global variables
* `--skip-sensors` (optional): Specify if the sensors should be skipped (default: `False`)
* `--template-runtime-configuration` (optional): Runtime configuration of data integration block runs (JSON format)

**Examples:**

```bash theme={"system"}
# Run entire pipeline
mage run /path/to/project my_pipeline_uuid

# Run specific block
mage run /path/to/project my_pipeline_uuid --block-uuid my_block_uuid

# Run with runtime variables
mage run /path/to/project my_pipeline_uuid --runtime-vars '{"key": "value"}'

# Run with tests
mage run /path/to/project my_pipeline_uuid --test
```

***

### Maintenance Commands

#### `clean-cached-variables` - Clean Cached Variables

**Available in:** ✅ OSS, ✅ Pro

Clean up old pipeline run outputs based on retention period.

**Usage:**

```bash theme={"system"}
mage clean-cached-variables <project_path> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project to clean variables
* `--pipeline-uuid` (optional): UUID of the pipeline to clean (if not specified, cleans all pipelines)
* `--retention-period` (optional): Default retention period, e.g., `7d`, `2w` (days or weeks)

**Example:**

```bash theme={"system"}
mage clean-cached-variables /path/to/project --retention-period 7d
```

***

#### `clean-old-logs` - Clean Old Logs

**Available in:** ✅ OSS, ✅ Pro

Clean up old pipeline run logs based on retention period.

**Usage:**

```bash theme={"system"}
mage clean-old-logs <project_path> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project to clean old logs
* `--pipeline-uuid` (optional): UUID of the pipeline to clean (if not specified, cleans all pipelines)
* `--retention-period` (optional): Default retention period, e.g., `7d`, `2w` (days or weeks)

**Example:**

```bash theme={"system"}
mage clean-old-logs /path/to/project --retention-period 30d
```

***

#### `clean-workspaces` - Clean Workspaces

**Available in:** ❌ OSS, ✅ Pro

Clean up old pipeline run outputs and logs inside Kubernetes workspace pods.

**Usage:**

```bash theme={"system"}
mage clean-workspaces [OPTIONS]
```

**Parameters:**

* `--cluster-type` (optional): Cluster type for workspace cleanup. Currently only `k8s` is supported
* `--workspace-name` (optional): Workspace name to clean. If not specified, cleans all workspaces
* `--pipeline-uuid` (optional): UUID of the pipeline to clean
* `--retention-period` (optional): Default retention period, e.g., `7d`, `2w` (days or weeks)
* `--skip-unavailable / --no-skip-unavailable` (optional): Skip workspaces without a running pod instead of failing the cleanup job

**Example:**

```bash theme={"system"}
mage clean-workspaces --retention-period 30d
```

***

### Mage Pro Only Commands

The following commands are **exclusively available in Mage Pro**:

#### `clean-unused-blocks` - Clean Unused Blocks

**Available in:** ❌ OSS, ✅ Pro

Clean up unused blocks from the given Mage project. This command interactively prompts you to delete blocks that are not referenced by any pipeline.

**Usage:**

```bash theme={"system"}
mage clean-unused-blocks <project_path> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project
* `--block-type` (optional): The type of block to clean up (e.g., `data_loader`, `transformer`, `data_exporter`)

**Example:**

```bash theme={"system"}
mage clean-unused-blocks /path/to/project
```

**Interactive Options:**

* `y`: Yes, delete the block
* `N`: No, skip the block
* `c`: Check content of the block before deciding
* `q`: Quit the operation

***

#### `create-secret` - Create or Update Secret

**Available in:** ❌ OSS, ✅ Pro

Create or update a secret for the given Mage project.

**Usage:**

```bash theme={"system"}
mage create-secret <project_path> <name> <value> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project
* `name` (required): Name of the secret
* `value` (required): Value of the secret
* `--update-exists` (optional): Whether to update the secret if it already exists (default: `True`)

**Example:**

```bash theme={"system"}
mage create-secret /path/to/project my_secret_name "my_secret_value"
```

***

#### `run-post-start-hooks` - Execute Post-Start Hooks

**Available in:** ❌ OSS, ✅ Pro

Execute post-start hooks with graceful error handling. This command reads `POST_START_COMMAND` and `POST_START_HOOK_PATH` from environment variables if not provided as arguments.

**Usage:**

```bash theme={"system"}
mage run-post-start-hooks [OPTIONS]
```

**Parameters:**

* `--post-start-command` (optional): Post-start command to execute (JSON array or shell command string). Can also be set via `POST_START_COMMAND` environment variable
* `--post-start-hook-path` (optional): Path to post-start hook script file. Can also be set via `POST_START_HOOK_PATH` environment variable

**Example:**

```bash theme={"system"}
mage run-post-start-hooks --post-start-command '["pip", "install", "package"]'
```

***

#### `restart-jobs` - Restart Failed or Cancelled Pipeline Runs

**Available in:** ❌ OSS, ✅ Pro

Restart failed or cancelled pipeline runs with optional filters.

**Usage:**

```bash theme={"system"}
mage restart-jobs <project_path> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project to restart jobs from
* `--pipeline-uuid` (optional): UUID of the pipeline to filter jobs by
* `--status` (optional): Status filter: `failed`, `cancelled`, or `both` (default: `failed`)
* `--start-time` (optional): Start time filter (ISO format: `2024-01-01T00:00:00`)
* `--end-time` (optional): End time filter (ISO format: `2024-01-01T23:59:59`)
* `--limit` (optional): Maximum number of jobs to restart (default: `100`)
* `--skip-confirmation` (optional): Skip confirmation prompt (default: `False`)

**Example:**

```bash theme={"system"}
mage restart-jobs /path/to/project --status failed --limit 50
```

***

#### `update-workspace` - Update Workspace Configuration

**Available in:** ❌ OSS, ✅ Pro

Update a workspace configuration synchronously. The cluster type is determined by the `CLUSTER_TYPE` environment variable, defaulting to `k8s` if not set.

**Usage:**

```bash theme={"system"}
mage update-workspace <workspace_name> [OPTIONS]
```

**Parameters:**

* `workspace_name` (required): Name of the workspace to update
* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--container-config` (optional): YAML string for container configuration updates
* `--update-workspace-settings` / `--no-update-workspace-settings` (optional): Whether to update workspace settings with current cluster environment variables (default: `False`)

**Example:**

```bash theme={"system"}
mage update-workspace my_workspace --container-config "memory: 4Gi"
```

***

#### `update-all-workspaces` - Update All Workspaces

**Available in:** ❌ OSS, ✅ Pro

Update all workspaces in the cluster synchronously.

**Usage:**

```bash theme={"system"}
mage update-all-workspaces [OPTIONS]
```

**Parameters:**

* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--container-config` (optional): YAML string for container configuration updates
* `--update-workspace-settings` / `--no-update-workspace-settings` (optional): Whether to update workspace settings with current cluster environment variables (default: `False`)
* `--cluster-type` (optional): Type of cluster: `k8s` or `docker` (default: `k8s`)

**Example:**

```bash theme={"system"}
mage update-all-workspaces --cluster-type k8s --update-workspace-settings
```

***

#### `re-add-workspace-routes` - Re-add Workspace Routes to Ingress

**Available in:** ❌ OSS, ✅ Pro

Re-add workspace routes to the ingress. This is useful for Kubernetes deployments where workspace routes need to be refreshed.

**Usage:**

```bash theme={"system"}
mage re-add-workspace-routes <project_path>
```

**Parameters:**

* `project_path` (required): Path of the Mage project

**Example:**

```bash theme={"system"}
mage re-add-workspace-routes /path/to/project
```

***

#### `recover-pipelines-from-file-versions` - Recover Pipelines from File Versions

**Available in:** ❌ OSS, ✅ Pro

Recover pipelines from file versions. This command lists all pipelines that have file versions in the `.file_versions` folder and allows you to recover pipeline files from those versions.

**Usage:**

```bash theme={"system"}
mage recover-pipelines-from-file-versions <project_path> [OPTIONS]
```

**Parameters:**

* `project_path` (required): Path of the Mage project
* `--pipeline-uuid` (optional): UUID of the pipeline to recover. If not specified, all pipelines are processed
* `--all` / `--no-all` (optional): Recover all files automatically without confirmation. Uses the most recent version for each file (default: `False`)
* `--verbose` / `-v` (optional): Show detailed pipeline information including individual files and blocks (default: `False`)

**Examples:**

```bash theme={"system"}
# Interactive recovery
mage recover-pipelines-from-file-versions /path/to/project

# Recover specific pipeline
mage recover-pipelines-from-file-versions /path/to/project --pipeline-uuid my_pipeline

# Recover all automatically
mage recover-pipelines-from-file-versions /path/to/project --all

# Verbose output
mage recover-pipelines-from-file-versions /path/to/project --verbose
```

***

## Common Use Cases

### Initializing a New Project

```bash theme={"system"}
mage init my_data_project --project-type standalone
cd my_data_project
mage start
```

### Running a Pipeline with Variables

```bash theme={"system"}
mage run /path/to/project my_pipeline_uuid \
  --runtime-vars '{"date": "2024-01-01", "environment": "production"}'
```

### Cleaning Up Old Data

```bash theme={"system"}
# Clean variables older than 7 days
mage clean-cached-variables /path/to/project --retention-period 7d

# Clean logs older than 30 days
mage clean-old-logs /path/to/project --retention-period 30d
```

### Pro: Managing Secrets

```bash theme={"system"}
mage create-secret /path/to/project database_password "secure_password_123"
```

### Pro: Updating Workspace Configuration

```bash theme={"system"}
# Update a single workspace
mage update-workspace dev_workspace --update-workspace-settings

# Update all workspaces
mage update-all-workspaces --cluster-type k8s
```

***

## Environment Variables

Some commands can be configured via environment variables:

* `CLUSTER_TYPE`: Cluster type for workspace operations (default: `k8s`)
* `POST_START_COMMAND`: Post-start command for `run-post-start-hooks`
* `POST_START_HOOK_PATH`: Post-start hook script path for `run-post-start-hooks`
* `ENV_VAR_INSTANCE_TYPE`: Instance type override for `start` command

***

## Additional Resources

* [Mage OSS CLI Reference](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/cli/main.py) - Source code for OSS CLI commands
* [Project Setup Guide](/development/project/setup) - Learn how to set up a Mage project
* [Workspaces Documentation](/guides/developer-ux/workspaces) - Learn about Mage Pro workspaces

***

## Notes

* All paths can be absolute or relative to the current working directory
* JSON parameters should be properly escaped when used in shell commands
* Pro-only commands require appropriate cluster/workspace configuration
* Some commands may require database connections to be properly configured


# Database
Source: https://docs.mage.ai/development/databases/default



By default, Mage uses sqlite to store orchestration data (trigger, pipeline run, and block run). To use a different database, you can set the `MAGE_DATABASE_CONNECTION_URL` environment variable.

In production, you can set the environment variable in the corresponding Terraform script.

## PostgreSQL

`export MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://user:password@host:port/dbname`

To use a specific `schema_name` in PostgreSQL:

`export MAGE_DATABASE_CONNECTION_URL="postgresql+psycopg2://user:password@host:port/dbname?options=-c%20search_path%3Dschema_name"`

### Get credentials from AWS Secret Manager

You can also instruct Mage to fetch your Postgres DB credentials from AWS Secret Manager. You will need to
set the following environment variables:

| Variable | Value |
| - | - |
| `AWS_DB_SECRETS_NAME` | name of secret in AWS Secret Manager |
| `AWS_DEFAULT_REGION` | AWS region the secret is stored in |
| `AWS_ACCESS_KEY_ID` | AWS access key id (not needed if using IAM role to authenticate) |
| `AWS_SECRET_ACCESS_KEY` | AWS secret access key (not needed if using IAM role to authenticate) |

The expected format of the secret is JSON string with the following keys:

* username
* password
* engine (must be `postgres`)
* host
* port
* dbname

## MSSQL (not officially supported)

`export MAGE_DATABASE_CONNECTION_URL="mssql+pyodbc://?odbc_connect=DRIVER={ODBC Driver 18 for SQL Server};SERVER=host;DATABASE=dbname;UID=user;PWD=password;ENCRYPT=yes;TrustServerCertificate=yes;"`

## Troubleshoot database migration failures

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=database)

Mage uses Alembic migrations for orchestration tables such as triggers, pipeline runs, block runs, users, secrets, and audit events. If the server fails during startup with an error like `alembic`, `sqlalchemy.exc`, `DROP INDEX`, `CREATE INDEX`, `alembic_version`, or `Can't locate revision`, use the Mage database CLI before manually editing the database.

Start with a read-only diagnosis:

```bash theme={"system"}
mage db diagnose
```

If your project uses the default SQLite database, run the command from the project directory or pass the project path:

```bash theme={"system"}
mage db diagnose --project-path /path/to/project
```

The command prints the database dialect, redacted database URL, revision stored in the `alembic_version` table, migration head in the installed Mage code, recommendations, and any pending revisions that have not been migrated yet.

When the database revision is not present in the installed migration scripts, `diagnose` also inspects schema evidence registered for a baseline anchor and each subsequent revision in the instrumented migration tail. The anchor verifies selected baseline tables and structures, and each supported revision verifies a structural fingerprint that can include column types and nullability, primary and foreign keys, and index columns and uniqueness. Mage reports each evidence revision as `applied`, `missing`, or `partial`. If the registered evidence is contiguous through the installed head and forms one applied prefix, the output identifies the latest schema-evidenced candidate and the first missing evidenced revision.

The candidate is a diagnostic boundary, not proof that every migration before the baseline anchor succeeded. Mage does not emit a stamp command from schema evidence alone. Inspect the earlier migration files and live schema before changing `alembic_version`; if the evidence has a gap, a partial fingerprint, an applied revision after a missing revision, no applied baseline, or a non-linear migration graph, Mage does not report a candidate at all.

To list every Alembic revision available in the installed Mage code:

```bash theme={"system"}
mage db history
```

This is read-only and does not connect to or change the database.

To inspect the current schema for a specific database table:

```bash theme={"system"}
mage db schema <table>
```

If your database uses a non-default schema, pass it explicitly:

```bash theme={"system"}
mage db schema <table> --schema <schema>
```

This prints the table columns, types, nullability, primary key columns, defaults, indexes, and foreign keys. Use it to confirm whether a failed migration already created or removed the table objects referenced by the error before you decide to roll back or stamp a revision.

### Run pending migrations

If the database is behind the installed Mage code, run:

```bash theme={"system"}
mage db migrate
```

To migrate to a specific Alembic revision:

```bash theme={"system"}
mage db migrate --revision <revision>
```

This runs migration SQL and uses Mage's database migration lock so multiple processes do not migrate the same database at the same time.

### Roll back a failed or incompatible migration

If a migration was applied and you need to revert it, roll back one migration:

```bash theme={"system"}
mage db rollback --revision -1
```

Or roll back to a specific revision:

```bash theme={"system"}
mage db rollback --revision <revision>
```

Review the migration file before rolling back in production. A rollback can drop columns, indexes, tables, or enum values depending on the migration.

Rollback statements often need exclusive locks on orchestration tables. For example, a downgrade that drops columns from the `user` table can wait while the running Mage server has active sessions using that table. The CLI applies a PostgreSQL lock timeout so the command fails instead of waiting forever:

```bash theme={"system"}
mage db rollback --revision -1 --lock-timeout-seconds 10
```

To inspect active PostgreSQL sessions that may be blocking the rollback:

```bash theme={"system"}
mage db locks
```

For an overall statement cap, also pass:

```bash theme={"system"}
mage db rollback --revision -1 --lock-timeout-seconds 10 --statement-timeout-seconds 120
```

If rollback times out waiting for locks, stop the Mage server or other workers connected to the orchestration database, then retry the rollback. After the rollback finishes, run `mage db diagnose` before restarting Mage.

### Stamp a revision without running SQL

Use `stamp` only when the database schema already matches the target revision but the `alembic_version` metadata is missing or wrong:

```bash theme={"system"}
mage db stamp <revision>
```

For non-interactive recovery:

```bash theme={"system"}
mage db stamp <revision> --yes
```

`stamp` changes Alembic metadata without creating, dropping, or altering database objects. It can help when a migration partially completed outside Alembic, but it can also hide real schema drift if used incorrectly. If you see an error such as `index "..." does not exist`, confirm whether the rest of the migration's expected schema changes are already present before stamping past that revision.

After independently verifying that the PostgreSQL schema matches the intended known revision, replace the exact unknown marker with a compare-and-swap recovery command. For example:

```bash theme={"system"}
mage db stamp <independently_verified_revision> \
  --replace-unknown-revision <current_unknown_revision> \
  --yes
```

Back up the database and stop migration-capable Mage processes before running this command in production. `--replace-unknown-revision` is supported only for PostgreSQL and requires `--yes`; the target must exist in the installed migration scripts, the database must contain exactly one `alembic_version` row, that row must still equal the supplied unknown revision, and the supplied current revision must remain unknown to the installed code. Mage performs the replacement transactionally and makes no change if any check fails. This recovery option changes only the revision marker; it does not apply migration SQL.

After replacing the unknown marker, apply the missing migrations and verify the result:

```bash theme={"system"}
mage db migrate
mage db diagnose
```

### Typical recovery flow

1. Back up the database before making changes in production.
2. Run `mage db diagnose`.
3. If `diagnose` reports an unknown revision and a schema-evidenced candidate, verify all earlier migrations and the live schema independently; do not stamp from the candidate alone.
4. If `diagnose` cannot infer a candidate, run `mage db history` and inspect the surrounding migration files; do not guess a revision.
5. If the error names a table, index, or constraint, run `mage db schema <table>` to inspect the live table.
6. If the database is behind, run `mage db migrate`.
7. If the latest migration is bad for your deployment, run `mage db rollback --revision -1`, then pin or deploy a Mage version with a compatible migration chain.
8. If a PostgreSQL schema independently matches the intended known revision but `alembic_version` contains an unknown revision, run `mage db stamp <revision> --replace-unknown-revision <unknown_revision> --yes`. For other marker problems, use the normal `mage db stamp <revision>` command only after verifying the schema.
9. Run `mage db diagnose` again and restart Mage.


# Import your own Python code
Source: https://docs.mage.ai/development/dependencies/custom-files

You can import any Python code from any file that is in your Mage project directory.

![Python mecha](https://mage-ai.github.io/assets/development/python-2.png)

## Importing custom Python code

When you’re writing code in Mage, you can import any other Python files using this format:

```python theme={"system"}
from [project_name].foo import bar
```

For example, if your project name is `demo_project` and you have the following folder structure:

```
demo_project/
|   pipelines/
|   utils/
|   |   helpers/
|   |   |   shared.py
|   |   |   __init__.py
|   |   __init__.py
```

You can write the following import statement to use functions in the
`demo_project/utils/helpers/shared.py` file:

```python theme={"system"}
from demo_project.utils.helpers import shared
```


# Dependency Management
Source: https://docs.mage.ai/development/dependencies/dependency-management

Advanced dependency management with isolated virtual environments, automatic caching, and intelligent package resolution in Mage Pro.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dependency)

Mage Pro includes an advanced dependency management system that provides isolated, reproducible dependency environments for your projects. This system automatically creates and manages virtual environments based on your project's dependencies, ensuring consistent execution across different environments.

## Overview

The dependency management system provides:

* **Isolated environments**: Each unique combination of dependencies gets its own virtual environment
* **Automatic caching**: Environments are reused based on content fingerprinting
* **Fast installation**: Uses `uv` for blazing-fast package installation with `pip` fallback
* **Smart compatibility checking**: Detects and reuses compatible packages from the global environment
* **Safe concurrency**: File-based locking prevents race conditions when multiple processes create environments

## Using Dependency Management

You can manage dependencies in three ways:

### 1. Via UI (Recommended for Quick Updates)

The easiest way to manage your project dependencies is through the Mage UI:

1. Open **Tools** in the left navigation, then select **Editor**

2. **Open your requirements file**: Browse to `/home/src/[project_path]/requirements.txt` and open it in the editor

3. **Edit your dependencies**: Add, remove, or update package requirements in the `requirements.txt` file
   ```txt theme={"system"}
   # Example requirements.txt
   pandas==2.0.0
   numpy>=1.24.0
   scikit-learn==1.3.0
   ```

4. **Click the "Install dependencies" button**: This will automatically:
   * Open a terminal at the bottom of the page
   * Execute `mage deps install` command
   * Show real-time installation progress and output

The system will intelligently install only the packages that need to be added or updated, reusing compatible packages from the global environment when possible.

### 2. Via Mage Terminal

For more control, you can run `mage deps` commands directly in the Mage terminal:

1. Open **Tools** in the left navigation, then select **Terminal**

2. **Run dependency commands**:
   ```bash theme={"system"}
   # Ensure virtual environment exists
   mage deps ensure

   # Install dependencies from requirements.txt
   mage deps install

   # Install from custom requirements file
   mage deps install -r production_requirements.txt

   # List all environments
   mage deps list

   # Clean up old environments
   mage deps cleanup
   ```

See the [CLI Commands](#cli-commands) section below for complete command documentation.

### 3. Via Cluster Restart (Automatic)

When you restart your Mage cluster, the dependency management system automatically:

1. **Reads your latest `requirements.txt`**: Checks for any changes since the last environment was created
2. **Creates or updates the virtual environment**:
   * If the requirements haven't changed, reuses the existing environment
   * If requirements have changed, creates a new environment with updated dependencies
3. **Activates the environment**: Makes it available for all pipeline executions

This ensures your dependencies are always synchronized with your `requirements.txt` after each cluster restart, with no manual intervention required.

## Repair Pipeline Dependency Failures with AI Chat

When a failed pipeline run reports a Python import or package-resolution error,
select **Fix with AI** from the run or its logs. AI Chat investigates using its
existing run-log and log-search tools. When the evidence indicates a dependency
issue, it uses the dependency skill to distinguish missing packages, optional
extras, incompatible versions, and differences between the Mage server and the
pipeline executor. Other errors continue through ordinary failure investigation.
If the skill cannot be loaded, AI Chat discloses that limitation and can continue
an evidence-supported repair using its built-in dependency guidance; it does not
claim to have loaded the skill.

AI Chat's run-log and log-search tools redact sensitive evidence before returning
it to the model. Run diagnostics read bounded excerpts from stored logs. If an
excerpt is incomplete, AI Chat should report that limitation and inspect more
specific evidence before proposing a dependency change.

AI Chat explains each action and reports its result:

1. Review the diagnosis and proposed `requirements.txt` diff. AI Chat states the
   intended change before applying it and follows the environment's action policy.
   If that policy requests confirmation, AI Chat waits for it. After an authorized
   write, Mage records the edit as a file version.
2. Track installation as a separate operation and reported state. Updating
   `requirements.txt` does not install packages by itself; AI Chat states which
   project requirements it will install, follows the environment's action policy,
   and reports the resolver result. The host may request confirmation, but the
   flow does not require a new confirmation between every authorized step.
3. Verify the repair in the environment that executes the pipeline. A successful
   package check or import in the Mage server or terminal does not prove that a
   Kubernetes, streaming, Spark, or other remote executor has the same packages.
   Restart or redeploy an executor separately when its environment requires it.
4. After verification, retry the failed block or pipeline run manually from the
   pipeline UI. The dependency-repair workflow reports its repair and verification
   results but does not automatically retry the run. See
   [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).

If AI Sidekick is disabled for the project, **Fix with AI** is unavailable and
the normal dependency-management controls continue to work without sending a
request to AI Sidekick or an AI model.

## How It Works

### Automatic Server-Side Management

When the Mage server starts, the dependency management system automatically handles virtual environment setup:

* **If no environment exists**: A new virtual environment is automatically created based on your project's `requirements.txt`
* **If an environment exists**: The matching environment is automatically activated for use

This seamless integration means you don't need to manually manage virtual environments for normal server operation. The system ensures your dependencies are always properly configured without manual intervention.

For more control over environment creation, activation, and management, you can use the [CLI commands](#cli-commands) described below.

### Environment Fingerprinting

The system computes a deterministic fingerprint hash for each environment based on:

* Python version (e.g., `3.11.8`)
* Normalized `requirements.txt` content
* `uv` version (if available)

If any of these change, a new environment is created. If they match an existing environment, that environment is reused.

### Directory Structure

Virtual environments are stored under your Mage data directory:

```
/home/src/mage_data/
└── venvs/
    ├── .cleanup.lock             # Global cleanup lock
    └── <python-version>/         # e.g., 3.11.8
        ├── env-<fingerprint>/    # Virtual environment
        │   ├── .mage_env.json    # Environment metadata
        │   ├── bin/              # Python executables
        │   └── lib/              # Installed packages
        └── .locks/               # Lock files for concurrency
            ├── .install.lock     # Installation lock
            ├── .activate.lock    # Activation lock
            ├── global.lock       # Global package versions
            └── venv.lock         # Installed package versions
```

Each virtual environment includes a `.mage_env.json` metadata file that tracks:

* `fingerprint`: Environment fingerprint hash
* `python_version`: Python version
* `created_at`: Creation timestamp (ISO 8601 format)
* `last_used_at`: Last usage timestamp (ISO 8601 format)
* `dependency_source`: Source of dependencies (e.g., `requirements.txt`)

### Package Installation Strategy

The system uses a smart installation strategy to minimize redundant installations:

1. **Global package detection**: Scans globally installed packages
2. **Compatibility filtering**: Filters out requirements already satisfied by global packages
3. **Constrained installation**: Installs remaining packages with global package versions as constraints
4. **Fallback handling**: Automatically retries without constraints if dependency conflicts occur

### Installation Tools

* **Primary**: `uv pip install` - Fast, modern Python package installer
* **Fallback**: `pip install` - Standard Python package installer
* **Virtual environment creation**: `uv venv --system-site-packages`

The `--system-site-packages` flag allows the virtual environment to inherit global packages while still allowing user packages to override them when needed.

## CLI Commands

### `mage deps ensure`

Create or reuse a virtual environment for your project's dependencies.

**Usage:**

```bash theme={"system"}
mage deps ensure [PROJECT_PATH] [OPTIONS]
```

**Parameters:**

* `project-path` (optional): Path to the Mage project (defaults to current directory)
* `--python-version` (optional): Python version to use (defaults to current Python version)
* `--output-file` (optional): File to write environment path to

**Examples:**

```bash theme={"system"}
# Ensure environment for current project
mage deps ensure

# Ensure environment for specific project
mage deps ensure /path/to/project

# Write environment path to file
mage deps ensure --output-file /tmp/venv_path.txt

# Use specific Python version
mage deps ensure --python-version 3.11.8
```

**Output:**

When successful, the command outputs an environment variable assignment:

```bash theme={"system"}
export MAGE_VENV_PATH="/path/to/venv"
```

You can activate this environment in your shell:

```bash theme={"system"}
eval $(mage deps ensure)
source $MAGE_VENV_PATH/bin/activate
```

### `mage deps install`

Install dependencies from a requirements file using the best available installer.

**Usage:**

```bash theme={"system"}
mage deps install [PROJECT_PATH] [OPTIONS]
```

**Parameters:**

* `project-path` (optional): Path to the Mage project (defaults to current directory)
* `-r, --requirements-file` (optional): Path to requirements file (defaults to `requirements.txt` in project)
* `--override-global-version` (optional): Skip global package constraints (default: `False`)

**Examples:**

```bash theme={"system"}
# Install dependencies for current project
mage deps install

# Install from specific requirements file
mage deps install -r custom_requirements.txt

# Install without global package constraints
mage deps install --override-global-version

# Install for specific project
mage deps install /path/to/project
```

**Behavior:**

1. Checks if running in an existing virtual environment (`VIRTUAL_ENV`)
2. If yes: Installs packages using advanced features (filtering, constraints, diff analysis)
3. If no: Installs packages system-wide using standard pip

**Conflict Resolution:**

If installation fails due to conflicting dependencies with global packages, the CLI will prompt:

```
Installation failed due to conflicting dependencies with global package versions.
Would you like to override global package versions and install without constraints? (Y/Yes/yes):
```

Answering "yes" retries the installation without global package constraints.

### `mage deps list`

List all dependency virtual environments with their metadata.

**Usage:**

```bash theme={"system"}
mage deps list [OPTIONS]
```

**Parameters:**

* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--python-version` (optional): Filter by Python version (e.g., `3.11.8`)
* `--verbose` (optional): Show detailed information including full paths

**Examples:**

```bash theme={"system"}
# List all environments
mage deps list

# List environments for specific Python version
mage deps list --python-version 3.11.8

# Show detailed information
mage deps list --verbose
```

**Output:**

```
Virtual Environments:

Python 3.11.8:
  env-abc123def456... (created: 2024-01-15T10:30:00Z, last used: 2024-01-20T14:25:00Z)
  env-789ghi012jkl... (created: 2024-01-10T08:00:00Z, last used: 2024-01-18T16:45:00Z)

Total: 2 environment(s)
```

### `mage deps cleanup`

Clean up stale virtual environments to free disk space.

**Usage:**

```bash theme={"system"}
mage deps cleanup [OPTIONS]
```

**Parameters:**

* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--ttl-seconds` (optional): Time-to-live in seconds (defaults to 604800 = 7 days)
* `--max-retention` (optional): Maximum number of environments to retain across all Python versions
* `--dry-run` (optional): Show what would be deleted without actually deleting (default: `False`)
* `--verbose` (optional): Show detailed information about cleanup process

**Examples:**

```bash theme={"system"}
# Dry run to see what would be deleted
mage deps cleanup --dry-run

# Cleanup with custom TTL (14 days)
mage deps cleanup --ttl-seconds 1209600

# Limit retention to 10 most recent environments
mage deps cleanup --max-retention 10

# Combine options
mage deps cleanup --ttl-seconds 604800 --max-retention 5 --dry-run
```

**How it works:**

1. Identifies environments unused for longer than TTL
2. Skips environments with active installation or activation locks
3. If `max_retention` is set, keeps the N most recently used environments even if they exceed TTL
4. Removes eligible environments and their metadata

**Output:**

```
Cleanup completed:
  - Deleted: 3 environment(s)
  - Skipped: 2 environment(s)
  - Scanned: 5 environment(s)
```

**Configuration:**

You can configure cleanup defaults in your project's `metadata.yaml`:

```yaml theme={"system"}
dependency_venvs_config:
  cleanup_config:
    ttl_seconds: 1209600  # 14 days
    max_retention: 10     # Keep 10 most recent
```

## Features in Detail

### Automatic Package Filtering

The system intelligently filters out packages that are already satisfied by globally installed packages:

```
Checking requirements against global packages...
  → Skipping pandas>=1.3.0 - existing package version 1.5.0 is compatible
  → Installing numpy>=1.22.0 - version 1.21.0 does not satisfy requirement
  → Installing scikit-learn - not found in global packages

Compatibility check complete: 1/3 package(s) satisfied by global packages, 2 will be installed in venv
```

This reduces installation time and disk usage by reusing compatible packages.

### Global Package Constraints

To ensure consistency with the base environment, the system creates a `global.lock` file containing all globally installed package versions and uses it as a constraint file during installation:

```bash theme={"system"}
uv pip install -r requirements.txt -c global.lock
```

This ensures that when a package from `requirements.txt` doesn't specify an exact version, the installer uses the same version as the global environment (if compatible), maintaining consistency.

### Installation Diff Analysis

After installation, the system analyzes the difference between global and virtual environment packages:

```
============================================================
Installation Diff Analysis
============================================================
✅ Reused from global: 45 package(s)
⬆️  Upgraded: 2 package(s)
  - numpy: 1.21.0 → 1.22.0
  - pandas: 1.3.0 → 1.5.0
➕ Newly added: 3 package(s)
  - scikit-learn==1.2.0
  - scipy==1.10.0
  - joblib==1.2.0
============================================================
```

This helps you understand what was installed, upgraded, or reused.

### Concurrency Safety

The system uses cross-platform file locking to prevent race conditions:

* **POSIX systems** (Linux, macOS): Uses `fcntl.flock()`
* **Windows**: Uses `msvcrt.locking()`
* **Lock timeout**: 10 minutes (configurable)

**Lock Types:**

1. **Installation lock** (`.install.lock`): Exclusive lock held during environment creation and dependency installation
2. **Activation lock** (`.activate.lock`): Shared lock held when environment is being used
3. **Cleanup lock** (`.cleanup.lock`): Global lock preventing simultaneous cleanup operations

This ensures that if multiple processes try to create the same environment simultaneously, only one creates it while others wait and then reuse it. Environments with active locks are protected from cleanup.

### Environment Validation

After creation, environments are validated to ensure they're properly initialized:

```python theme={"system"}
def validate_venv(venv_path: str) -> bool:
    """Validate that a virtual environment is properly initialized."""
    python_exe = _get_venv_python_exe(venv_path)
    return os.path.exists(python_exe) and os.path.isfile(python_exe)
```

Invalid environments are automatically cleaned up and recreated.

## Error Handling

The system provides detailed error messages for common issues:

### Lock Acquisition Timeout

```
Could not acquire lock /path/to/lock within 600s timeout

This may indicate:
  - Another process is creating the same environment
  - A previous process crashed while holding the lock
  - System I/O issues

Try waiting a few minutes and retrying, or check for stale lock files.
```

### Installation Failure

```
Failed to install dependencies using uv.
Error output:
[detailed error message]

This may be due to:
  - Network connectivity issues
  - Invalid package specifications in requirements.txt
  - Incompatible package versions
  - Missing system dependencies

Please check your requirements.txt file and try again.
```

### Requirements File Not Found

```
Requirements file not found: /path/to/requirements.txt

Ensure that:
  - The file exists at the specified path
  - The path is correct and accessible
  - You have read permissions for the file
```

## Best Practices

### 1. Pin Package Versions

For reproducibility, pin exact versions in `requirements.txt`:

```
# Good - exact versions
pandas==1.5.0
numpy==1.22.0
scikit-learn==1.2.0

# Less predictable - version ranges
pandas>=1.3.0
numpy>=1.20.0
```

### 2. Use Virtual Environments in Development

Always activate the virtual environment when developing:

```bash theme={"system"}
eval $(mage deps ensure)
source $MAGE_VENV_PATH/bin/activate
```

### 3. Update Dependencies Carefully

When updating dependencies:

1. Update `requirements.txt`
2. Run `mage deps ensure` to create a new environment
3. Test thoroughly before deploying

### 4. Monitor Disk Usage

Virtual environments can consume significant disk space. Use the cleanup command to remove stale environments:

```bash theme={"system"}
# See what would be deleted
mage deps cleanup --dry-run

# Clean up environments unused for 7+ days
mage deps cleanup

# Or configure automatic cleanup in metadata.yaml
```

### 5. Handle Dependency Conflicts

If you encounter dependency conflicts:

1. Try without `--override-global-version` first (uses global constraints)
2. If that fails, retry with `--override-global-version`
3. Review the conflict and adjust `requirements.txt` if needed

## Troubleshooting

### Environment Not Found After Creation

**Symptom**: `mage deps ensure` succeeds but environment doesn't exist

**Solution**: Check for validation failures in logs. The environment may have been created but failed validation and was cleaned up.

### Slow Environment Creation

**Symptom**: Creating environments takes a long time

**Possible causes**:

* Network issues downloading packages
* Many packages to install
* Slow disk I/O

**Solutions**:

* Install `uv` for faster installation: `pip install uv`
* Check network connectivity
* Review `requirements.txt` for unnecessary packages

### Permission Errors

**Symptom**: Cannot create virtual environment due to permission errors

**Solution**: Ensure you have write permissions to the Mage data directory:

```bash theme={"system"}
ls -ld $(python -c "from mage_ai.settings.repo import get_data_dir; print(get_data_dir())")
```

### Stale Lock Files

**Symptom**: Lock acquisition timeout errors persist

**Solution**: Manually remove stale lock files:

```bash theme={"system"}
# Find lock files
find /home/src/mage_data/venvs -name "*.lock"

# Remove stale locks (be careful!)
rm /home/src/mage_data/venvs/*/\.locks/*.lock
```

## Technical Details

### Fingerprint Algorithm

The fingerprint is computed as follows:

```python theme={"system"}
components = [
    f"python:{python_version}",
    f"requirements:{requirements_content or ''}",
    f"uv:{uv_version or 'no-uv'}",
]
fingerprint_input = '\n'.join(components)
fingerprint = hashlib.sha256(fingerprint_input.encode('utf-8')).hexdigest()
```

### Requirements Normalization

Requirements are normalized before fingerprinting:

* Leading/trailing whitespace stripped from each line
* Empty lines removed
* Comments (lines starting with `#`) removed
* Order preserved

This ensures that cosmetic changes to `requirements.txt` don't trigger new environment creation.

### Lock File Semantics

* **.install.lock**: Exclusive lock preventing concurrent installation/creation of environment
* **.activate.lock**: Shared lock indicating environment is actively being used
* **.cleanup.lock**: Global lock preventing simultaneous cleanup operations across all environments
* **global.lock**: Snapshot of globally installed packages at environment creation time (used as constraints)
* **venv.lock**: Snapshot of packages installed in the virtual environment (for diff analysis)

## See Also

* [Python Packages](/development/dependencies/requirements) - Basic requirements.txt usage
* [Custom Files](/development/dependencies/custom-files) - Managing custom Python files
* [CLI Commands](/development/cli-commands) - Complete CLI reference


# Install Python packages using `requirements.txt`
Source: https://docs.mage.ai/development/dependencies/requirements

Mage allows you to easily integrate third-party Python packages into your project and pipelines. This guide will walk you through the process of adding and managing external dependencies.

![Python mecha](https://mage-ai.github.io/assets/development/python-package.png)

> [!NOTE]
>
> **Mage Pro: Advanced Dependency Management**
>
> Mage Pro includes an advanced dependency management system with isolated virtual environments, automatic caching, and intelligent package resolution. Features include:
>
> * **Isolated environments** for each unique combination of dependencies
> * **Automatic caching** with fingerprint-based environment reuse
> * **Fast installation** using `uv` with `pip` fallback
> * **Smart compatibility checking** to reuse global packages
> * **No cluster restarts required** for dependency updates
> * **Easy UI-based installation** via Terminal page
>
> See the [Advanced Dependency Management](/development/dependencies/dependency-management) guide for complete documentation, or the [Quick Start guide](/getting-started/install-dependencies) for UI-based installation.
>
> [Get started with Mage Pro for free](https://cloud.mage.ai/sign-up?source=documentation)

## Add Python packages to `requirements.txt`

### Location

At the root of your Mage project directory, you'll find a file named `requirements.txt`.
This file is used to specify all the Python packages that your project depends on.

### Path

At the root of your project directory, there is a file named `requirements.txt`. Open this file
and add the Python package you want installed and available throughout your entire project.

For example, if your project is named `demo_project` then the file will be located at
`demo_project/requirements.txt`.

### Adding packages

To add a package, simply open the `requirements.txt` file and list each package on a new line. You can also specify version requirements if needed.

### Example `requirements.txt`:

```
matplotlib==3.5.2
tqdm
```

In this example the `requirements.txt` file will install:

* matplotlib version 3.5.2
* tqdm

## Automatic installation

When installing Mage using Docker, Mage provides two main methods for automatically installing third-party packages at runtime.
These methods utilize the `USER_CODE_PATH` environment variable, which defines your project path (e.g. `/home/src/project_name`),
to locate your `requirements.txt` file and install the packages listed there.

> **Using Command Line Flags**
>
> ## Using command line flags
>
> If you installed Mage using Docker, the `USER_CODE_PATH` can be set using command line flags. You will need to add these command line flags at every run when running a new Docker container or installing a new version of Mage.
>
> ### Setting `USER_CODE_PATH` in Docker
>
> The `USER_CODE_PATH` environment variable tells Mage where your project code is located within the Docker container. This is essential for features like automatic package installation from `requirements.txt`.
>
> #### Docker run command
>
> To set the `USER_CODE_PATH` when running a Mage Docker container, use the following command structure:
>
> ```Shell theme={"system"}
> docker run -it -p 6789:6789 -v $(pwd):/home/src \
>   -e USER_CODE_PATH=/home/src/project_name \
>   mageai/mageai /app/run_app.sh mage start [project_name]
> ```
>
> Let's break down this command:
>
> * `it`: Runs the container interactively with a pseudo-TTY.
> * `p 6789:6789`: Maps port 6789 from the container to your host machine.
> * `v $(pwd):/home/src`: Mounts your current directory to `/home/src` in the container.
> * `e USER_CODE_PATH=/home/src/project_name`: Sets the `USER_CODE_PATH` environment variable.
> * `mageai/mageai`: Specifies the Mage Docker image to use.
> * `/app/run_app.sh mage start [project_name]`: The command to run inside the container.
>
> ### Important Notes
>
> 1. **Persistence**: The `USER_CODE_PATH` setting is not persistent across container restarts or when installing new versions of Mage. You must include these flags each time you run a new Docker container.
> 2. **Project Name**: Replace `[project_name]` with your actual Mage project name in both the `USER_CODE_PATH` and the start command.
> 3. **Path Consistency**: Ensure that the path in `USER_CODE_PATH` matches the mount point specified in the `v` flag. In this example, we mount to `/home/src`, so the `USER_CODE_PATH` starts with `/home/src`.
> 4. **Working Directory**: The `$(pwd)` in the `v` flag assumes you're running the Docker command from your project's parent directory. Adjust this path if you're running from a different location.
>
> For more detailed information on environment variables and their impact on Mage's functionality, please refer to our comprehensive environment variable [documentation](https://docs.mage.ai/development/variables/environment-variables).

> **Using Dockerfile**
>
> ## Using Dockerfiles
>
> One important aspect of configuring Mage is setting the `USER_CODE_PATH`, which tells Mage where your project code is located.
> While this can be done using command-line flags when running Docker containers, it's also possible to set the `USER_CODE_PATH`
> directly in your Dockerfile. This approach can be more convenient for consistent deployments and when building custom images.
>
> ```Bash theme={"system"}
> FROM mageai/mageai:latest
>
> ### Set up project-specific arguments
> ARG PROJECT_NAME=[project_name]
> ARG MAGE_CODE_PATH=/home/mage_code
> ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}
>
> ### Set working directory
> WORKDIR ${MAGE_CODE_PATH}
>
> ### Copy project files
> COPY ${PROJECT_NAME} ${PROJECT_NAME}
>
> ### Set the command to run on container startup
> CMD ["/bin/sh", "-c", "/app/run_app.sh"]
> ```
>
> Breakdown of the Dockerfile:
>
> 1. `FROM mageai/mageai:latest`
>    * Specifies the base image as the latest version of the official Mage AI image.
> 2. Project-specific arguments:
>
> ```Bash theme={"system"}
> ARG PROJECT_NAME=[project_name]
> ARG MAGE_CODE_PATH=/home/mage_code
> ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}
> ```
>
> * `PROJECT_NAME`: Should be replaced with your actual project name.
>
> * `MAGE_CODE_PATH`: Sets the base path for Mage code.
>
> * `USER_CODE_PATH`: Constructs the full path to your project, effectively setting the `USER_CODE_PATH` for Mage.
>
> 3. `WORKDIR ${MAGE_CODE_PATH}`
>    * Sets the working directory inside the container to the path specified by `MAGE_CODE_PATH`.
> 4. `COPY ${PROJECT_NAME} ${PROJECT_NAME}`
>    * Copies your project files from the host machine into the container.
> 5. `CMD ["/bin/sh", "-c", "/app/run_app.sh"]`
>    * Specifies the command to run when the container starts, which launches the Mage application.
>
> To use this Dockerfile:
>
> 1. Replace `[project_name]` with your actual project name.
> 2. Ensure your project files are in a directory named after your project.
> 3. Build the Docker image using this Dockerfile.
> 4. Run the container to start your Mage environment.
>
> This Dockerfile demonstrates how to set the `USER_CODE_PATH` using build arguments. By defining it this way, you ensure that Mage will
> consistently know where to find your project code within the container, without needing to specify it each time you run the container.
>
> Follow the [Repository setup](https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup) documentation to deploy this
> Dockerfile to fully automate the installation of third-party Python packages.

If you’re running Mage using pip or conda, your local machine’s environment variables are accessible within the running Mage app.

### Prerequisites

1. Ensure that the environment variable `USER_CODE_PATH` is set to your project path, using
   the command line flags or the Dockerfile methods.

   Example: `/home/src/project_name`

2. Mage uses a script named `run_app.sh` to manage the application startup process.

### Installation process

When you start your Mage server:

1. The `run_app.sh` script checks the `USER_CODE_PATH` environment variable.
2. It locates the `requirements.txt` file within the specified project directory.
3. All packages listed in `requirements.txt` are automatically installed before the server starts.

This ensures that all required packages are available for your project and pipelines.


# Connect a local db in Docker
Source: https://docs.mage.ai/development/docker/connecting-a-database

Here's how you can connect a local database to Mage in Docker for development.

## Overview

When you develop in Docker, you can connect a database to your Mage container. This allows you to use the database in your data pipelines.

A Docker Network is a virtual network that allows containers to communicate with each other.
By creating a network, you can connect database containers to your Mage instances.
Read more about Docker Networks [here](https://docs.docker.com/network/).

The following guides detail various methods for creating databases and connecting them to your Mage project.

We recommend starting with a [docker compose](https://docs.docker.com/compose/) file, as it affords the most flexibility and customizability. You might
choose to use `docker run` instead, as it's a bit simpler.

## Guides

> **PostgreSQL - Docker compose**
>
> If you're using Mage in docker compose, you can easily connect a PostgreSQL database to your Mage container. To do so, first define a `.env` file with the following variables:
>
> * `POSTGRES_DB`
> * `POSTGRES_USER`
> * `POSTGRES_PASSWORD`
> * `POSTGRES_HOST`
> * `PG_HOST_PORT`
>
> Then, add the following to your `docker-compose.yml` file:
>
> ```yaml theme={"system"}
> version: '3'
> services:
>   mage:
>     image: mageai/mageai:latest
>     container_name: magic
>     depends_on:
>       - postgres
>     command: mage start magic
>     env_file:
>       - .env
>     environment:
>       ENV: dev
>       POSTGRES_DB: ${POSTGRES_SCHEMA}
>       POSTGRES_USER: ${POSTGRES_USER}
>       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
>       POSTGRES_HOST: ${POSTGRES_HOST}
>       PG_HOST_PORT: ${PG_HOST_PORT}
>     ports:
>       - 6789:6789
>     volumes:
>       - .:/home/src/
>     restart: on-failure:5
>
>   postgres:
>     image: postgres:14
>     restart: on-failure
>     container_name: postgres-magic
>     build:
>       context: ./db
>     env_file:
>       - .env
>     environment:
>       POSTGRES_DB: ${POSTGRES_DB}
>       POSTGRES_USER: ${POSTGRES_USER}
>       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
>     ports:
>       - "${PG_HOST_PORT}:5432"
> ```

> **PostgreSQL - Docker run**
>
> The following steps will create a PostgreSQL database container and connect it to a Mage container.
>
> **1. Create Docker network**
>
> ```bash theme={"system"}
>  docker network create mage-app
> ```
>
> **2. Start PostgreSQL Docker container**
>
> ```bash theme={"system"}
> docker run --network mage-app --network-alias postgres_db \
>    -it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
>    -e POSTGRES_USER=<username> -e POSTGRES_PASSWORD=<password> \
>    -e POSTGRES_DB=<database> -e PG_DATA=/var/lib/postgresql/data/pgdata \
>    postgres:13-alpine3.17 postgres
> ```
>
> **3. Launch Mage with `MAGE_DATABASE_CONNECTION_URL`**
>
> ```bash theme={"system"}
>    docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
>       -e MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://<username>:<password>@postgres_db:5432/<database> \
>       mageai/mageai \
>       /app/run_app.sh mage start [project_name]
> ```
>
> To use a specific schema in the PostgreSQL database, the connection string could contain the schema name as a parameter:
>
> ```bash theme={"system"}
> docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
>    -e MAGE_DATABASE_CONNECTION_URL="postgresql+psycopg2://<username>:<password>@postgres_db:5432/<database>?options=-c%%20search_path%%3D<schema_name>" \
>    mageai/mageai \
>    /app/run_app.sh mage start [project_name]
> ```

> **MSSQL - Docker run**
>
> **1. Create Docker network**
>
> ```bash theme={"system"}
>  docker network create mage-app
> ```
>
> **2. Start MSSQL Docker container**
>
> ```bash theme={"system"}
> docker run --network mage-app --network-alias mssql_db \
> -it -p 1433:1433 -e "ACCEPT_EULA=Y" \
> -e "MSSQL_SA_PASSWORD=test_Password123" \
> mcr.microsoft.com/mssql/server:2022-latest
> ```
>
> **3. Launch Mage with `MAGE_DATABASE_CONNECTION_URL`**
>
> ```bash theme={"system"}
>    docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
>    -e MAGE_DATABASE_CONNECTION_URL="mssql+pyodbc://?odbc_connect=DRIVER={ODBC Driver 18 for SQL Server};SERVER=mssql_db;DATABASE=master;UID=SA;PWD=test_Password123;ENCRYPT=yes;TrustServerCertificate=yes;" \
>    mageai/mageai \
>    /app/run_app.sh mage start [project_name]
> ```
>
> Here are the parameters we're using in that command:
>
> * `DRIVER={ODBC Driver 18 for SQL Server}`: a formatted string with a version number matching that of the SQL server
> * `SERVER=database_server`: the name or IP address of the database server
> * `DATABASE=database`: the name of the database
> * `UID=user`: the user ID for the database
> * `PWD=password`: the password for the user
> * `ENCRYPT=yes`: whether to use encryption
> * `TrustServerCertificate=yes`: whether to trust the server certificate


# Run docker image as non-root user
Source: https://docs.mage.ai/development/docker/non-root-user

Run Mage docker image as non-root user to restrict access to the system

By default, the official Mage docker images in DockerHub are using the root users. For production deployment,
it's recommended to use non-root users in the docker image to restrict access to the system for better security.
Here is the Dockerfile template that you can use to build your custom docker image that runs Mage as non-root user.

```dockerfile theme={"system"}
ARG VERSION

FROM mageai/mageai:$VERSION

# Add non-root user for Mage service
RUN adduser --disabled-password --gecos '' mage && adduser mage mage

# Grant the user permissions to the Mage related directories
RUN mkdir /home/src/mage_data; chown -R mage /home/src/mage_data
RUN mkdir /home/src/default_repo; chown -R mage /home/src/default_repo

RUN pip3 install -r path/to/requirements.txt

# Set the Mage user
USER mage

ENV PYTHONPATH="${PYTHONPATH}:/home/mage/.local/lib/python3.10/site-packages"

WORKDIR /home/src

CMD ["/bin/sh", "-c", "/app/run_app.sh"]
```


# Mage Pro CI/CD overview
Source: https://docs.mage.ai/development/git/pro-overview

Deploy your code in seconds!

Mage Pro provides integrated CI/CD capabilities to streamline your data pipeline development and deployment workflows. This section covers:

* **Version Control Application** - Built-in Git terminal with authentication and real-time collaboration features
* **Deployment Guide** - Step-by-step instructions for connecting repositories and configuring deployment stages
* **Git Operations** - Standard version control commands and troubleshooting guidance
* **Multi-stage Deployments** - Production-ready deployment workflows with rollback capabilities

Supported Git providers include GitHub and GitLab, with OAuth and SSH authentication options available for secure repository integration.


# Get started with Git
Source: https://docs.mage.ai/development/git/setting-up-git

There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.

![](https://mage-ai.github.io/assets/getting-started/adventure-time.gif)

# Mage Pro

Unlock seamless version control and effortless deployments with Mage Pro!
Check out this [guide](/guides/data-sync/version-control-guide) to experience a more intuitive, user-friendly way to manage and deploy your code.

![Mage Pro deployments page](https://mage-ai.github.io/assets/pro/features/deployments/deployments-page.png)

# Open source

## Choose your adventure

There three different ways to use Git in Mage and *four* ways to version control your files— we'll discussing each of them. We note that our methods for using Git are mutually exclusive.

If you're simply looking for version control, you're in luck! It ships out-of-the-box with Mage. That means you can *always* restore previous versions of your files, even if you haven't configured any version control system. Check out [this page](/getting-started/file-versions) for more info.

## Methods of using Git in Mage

| Recommended | Use | Recommendation | Ease of use | Functionality |
| - | - | - | - | - |
| ✅ | Develop/deploy Mage using GitHub | [Git application](/guides/data-sync/version-control) | Moderate | Full w/ OAuth |
| ✅ | Deploy Mage using Git with any provider | [Integration w/ Git Sync](/guides/data-sync/git-sync) | Advanced | Full w/ SSH/HTTPS Auth |
| ⚠️ | Develop Mage using Git with any provider | [Integration *w/o* Git Sync](/guides/data-sync/git-integration) | Advanced | Full w/ SSH/HTTPS Auth |

Git Sync is particularly useful in production instances, where other Git methods can be used more meaningfully in development instances. Since Git sync is a one-way sync, it's not very useful for collaboration, since changes get overridden. However, this makes it great for a production Mage instance, where verified changes should only be accepted from development environments.

A common pattern is to use development instances to edit and push (Git integration), then have production instances use Git Sync or CI/CD.


# IO Config
Source: https://docs.mage.ai/development/io_config

The `io_config` YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Here's a sample `io_config` file. You can learn more about variable interpolation and referencing secrets via our variables overview [here](/development/variables/overview).

> ℹ️ To learn how to set up your `io_config.yaml` file, check the [setup guide](/development/io_config_setup).
> The `io_config.yaml` file should be placed under your project folder.

```yaml theme={"system"}
version: 0.1.1
default:
  # Default profile created for data IO access.
  # Add your credentials for the source you use, and delete the rest.
  # AWS
  AWS_ACCESS_KEY_ID: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
  AWS_SECRET_ACCESS_KEY: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
  AWS_SESSION_TOKEN: session_token (Used to generate Redshift credentials)
  AWS_REGION: region
  # Azure
  AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
  AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
  AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
  AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
  # ClickHouse
  CLICKHOUSE_DATABASE: default
  CLICKHOUSE_HOST: host.docker.internal
  CLICKHOUSE_INTERFACE: http
  CLICKHOUSE_PASSWORD: null
  CLICKHOUSE_PORT: 8123
  CLICKHOUSE_USERNAME: null
  # Druid
  DRUID_HOST: hostname
  DRUID_PASSWORD: password
  DRUID_PATH: /druid/v2/sql/
  DRUID_PORT: 8082
  DRUID_SCHEME: http
  DRUID_USER: user
  # Google
  GOOGLE_SERVICE_ACC_KEY:
    type: service_account
    project_id: project-id
    private_key_id: key-id
    private_key: "-----BEGIN PRIVATE KEY-----\nyour_private_key\n-----END_PRIVATE_KEY"
    client_email: your_service_account_email
    auth_uri: "https://accounts.google.com/o/oauth2/auth"
    token_uri: "https://accounts.google.com/o/oauth2/token"
    auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/your_service_account_email"
  GOOGLE_SERVICE_ACC_KEY_FILEPATH: "/path/to/your/service/account/key.json"
  GOOGLE_LOCATION: US # Optional
  # MongoDB
  MONGODB_DATABASE: database
  MONGODB_HOST: host
  MONGODB_PASSWORD: password
  MONGODB_PORT: 27017
  MONGODB_COLLECTION: collection
  MONGODB_USER: user
  # MSSQL
  MSSQL_DATABASE: database
  MSSQL_SCHEMA: schema
  MSSQL_DRIVER: "ODBC Driver 18 for SQL Server"
  MSSQL_HOST: host
  MSSQL_PASSWORD: password
  MSSQL_PORT: 1433
  MSSQL_USER: SA
  # MySQL
  MYSQL_DATABASE: database
  MYSQL_HOST: host
  MYSQL_PASSWORD: password
  MYSQL_PORT: 3306
  MYSQL_USER: root
  # Pinot
  PINOT_HOST: hostname
  PINOT_PASSWORD: password
  PINOT_PATH: /query/sql
  PINOT_PORT: 8000
  PINOT_SCHEME: http
  PINOT_USER: user
  # PostgreSQL
  POSTGRES_CONNECT_TIMEOUT: 10
  POSTGRES_DBNAME: postgres
  POSTGRES_SCHEMA: public # Optional
  POSTGRES_USER: username
  POSTGRES_PASSWORD: password
  POSTGRES_HOST: hostname
  POSTGRES_PORT: 5432
  # Redshift
  REDSHIFT_SCHEMA: public # Optional
  REDSHIFT_DBNAME: redshift_db_name
  REDSHIFT_HOST: redshift_cluster_id.identifier.region.redshift.amazonaws.com
  REDSHIFT_PORT: 5439
  REDSHIFT_TEMP_CRED_USER: temp_username
  REDSHIFT_TEMP_CRED_PASSWORD: temp_password
  REDSHIFT_DBUSER: redshift_db_user
  REDSHIFT_CLUSTER_ID: redshift_cluster_id
  REDSHIFT_IAM_PROFILE: default
  # Snowflake
  SNOWFLAKE_USER: username
  SNOWFLAKE_PASSWORD: password
  SNOWFLAKE_ACCOUNT: account_id.region
  SNOWFLAKE_DEFAULT_WH: null                  # Optional default warehouse
  SNOWFLAKE_DEFAULT_DB: null                  # Optional default database
  SNOWFLAKE_DEFAULT_SCHEMA: null              # Optional default schema
  SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null      # Optional private key passphrase
  SNOWFLAKE_PRIVATE_KEY_PATH: null            # Optional private key path
  SNOWFLAKE_ROLE: null                        # Optional role name
  SNOWFLAKE_TIMEOUT: null                     # Optional timeout in seconds
  # Trino
  trino:
    catalog: postgresql                       # Change this to the catalog of your choice
    host: 127.0.0.1
    http_headers:
      X-Something: 'mage=power'
    http_scheme: http
    password: mage1337                        # Optional
    port: 8080
    schema: core_data
    session_properties:                       # Optional
      acc01.optimize_locality_enabled: false
      optimize_hash_generation: true
    source: trino-cli                         # Optional
    user: admin
    verify: /path/to/your/ca.crt              # Optional
    data_type_properties:
      timestamp_precision: 3                              # Optional. Set TIMESTAMP(n) precision
```

# Unified config

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=io_config)

Mage Pro provides a unified framework for configuring connections using `io_config.yaml` across different types of pipelines:

* **Batch pipeline** (Python block, SQL block)
* **Data integration pipeline**
* **Streaming pipeline**

Each pipeline reads from the same io\_config.yaml file, which allows you to centralize connection information and switch between pipeline types with minimal configuration changes.

## Batch pipeline

### Example: Python block (Postgres)

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data_from_postgres(**kwargs):
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

### Example: SQL block

Just select the desired IO Config profile in the UI (under the "Profile" dropdown).
The profile must exist in your io\_config.yaml and contain valid credentials for the source or destination.

## Data integration pipeline

Data integration pipelines use source and destination blocks.
These blocks can reference connection credentials from `io_config.yaml` via the `io_config_profile` key.

The following sources and destinations systems currently support this configuration method:

* BigQuery
* Microsolft SQL Server
* MySQL
* Postgres
* Redshift
* Snowflake

### Example source config (Postgres)

```yaml theme={"system"}
io_config_profile: default  # profile in the io_config.yaml
```

### Example destination config (Postgres)

```yaml theme={"system"}
io_config_profile: default  # profile in the io_config.yaml
schema: test                # Override the schema
```

With this setup, you don’t need to hard-code connection details in your block configs—just use the profile name.

## Streaming pipeline

Streaming pipelines also support `io_config.yaml` for setting up destinations.

### Example: Destination block config (BigQuery)

```yaml theme={"system"}
connector_type: bigquery
profile: default    # profile in the io_config.yaml
config:
  table_id: table_id
  if_exists: "append"
```

The `profile` field points to the profile in your `io_config.yaml`, and additional block-specific settings can still be defined under config.


# IO Config Setup
Source: https://docs.mage.ai/development/io_config_setup

Learn how to set up and configure the io_config file for your Mage project

# IO Config Setup Guide

The `io_config` file is a crucial configuration file in Mage that stores credentials and connection information for accessing various data sources. This guide will help you understand where to store it and how to create it from scratch.

## File Location

The `io_config` file should be stored in your Mage project's root directory. The default path is:

```
your_mage_project/
└── io_config.yaml
```

For detailed information about project structure and file organization, see the [Project Structure documentation](/design/abstractions/project-structure#project-structure).

## Creating a New io\_config File

### 1. Basic Structure

Create a new file named `io_config.yaml` in your project's root directory (e.g. `/home/src/your_mage_project`) with the following basic structure:

```yaml theme={"system"}
version: 0.1.1  # Optional but recommended
default:
  # Your default profile configuration here
```

### 2. Configuration Formats

Mage supports two formats for the `io_config` file:

#### Standard Format (Recommended)

```yaml theme={"system"}
version: 0.1.1
default:
  # PostgreSQL configuration
  POSTGRES_DBNAME: your_database
  POSTGRES_USER: your_username
  POSTGRES_PASSWORD: "{{ env_var('POSTGRES_PASSWORD') }}"
  POSTGRES_HOST: your_host
  POSTGRES_PORT: 5432

  # Snowflake configuration
  SNOWFLAKE_USER: your_username
  SNOWFLAKE_PASSWORD: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
  SNOWFLAKE_ACCOUNT: your_account
  SNOWFLAKE_DEFAULT_WH: your_warehouse
  SNOWFLAKE_DEFAULT_DB: your_database
  SNOWFLAKE_DEFAULT_SCHEMA: your_schema

  # AWS configuration
  AWS_ACCESS_KEY_ID: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
  AWS_SECRET_ACCESS_KEY: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
  AWS_REGION: your_region
```

#### Legacy Format (Verbose)

```yaml theme={"system"}
default:
  PostgreSQL:
    database: your_database
    user: your_username
    password: your_password
    host: your_host
    port: 5432
  AWS:
    region: your_region
    Redshift:
      database: your_redshift_database
      port: 5439
```

### 3. Using Variables and Secrets

For security, it's recommended to use environment variables and other secret management systems for sensitive information. Mage provides several variable syntax options:

#### Variable Syntax

| Syntax | Description |
| - | - |
| `{{ env_var('secret') }}` | Get secret from environment variables |
| `{{ mage_secret_var('secret') }}` | Get secret from Mage secrets |
| `{{ aws_secret_var('secret') }}` | Get secret from AWS Secrets Manager |
| `{{ azure_secret_var('secret') }}` | Get secret from Azure Key Vault |
| `{{ gcp_secret_var('secret') }}` | Get secret from Google Cloud Secret Manager (Mage Pro only) |
| `{{ doppler_secret_var('secret') }}` | Get secret from [Doppler](/production/deploying-to-cloud/secrets/Doppler) (Mage Pro only) |
| `{{ json_value(json_obj, 'key') }}` | Extract value from a JSON string |
| `{{ file('path/to/file.txt') }}` | Load contents of a local file (Mage Pro only) |

#### Example Configurations

**Environment Variables:**

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ env_var('POSTGRES_PASSWORD') }}"
  SNOWFLAKE_PASSWORD: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
```

**Mage Secrets:**

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ mage_secret_var('postgres_password') }}"
  SNOWFLAKE_PASSWORD: "{{ mage_secret_var('snowflake_password') }}"
```

**AWS Secrets Manager:**

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ aws_secret_var('prod/postgres/password') }}"
  SNOWFLAKE_PASSWORD: "{{ aws_secret_var('prod/snowflake/password') }}"
```

**Azure Key Vault:**

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ azure_secret_var('postgres-password') }}"
  SNOWFLAKE_PASSWORD: "{{ azure_secret_var('snowflake-password') }}"
```

**Google Cloud Secret Manager (Mage Pro only):**

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ gcp_secret_var('postgres-password') }}"
  SNOWFLAKE_PASSWORD: "{{ gcp_secret_var('snowflake-password') }}"
```

**Doppler (Mage Pro only):**

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ doppler_secret_var('POSTGRES_PASSWORD') }}"
  SNOWFLAKE_PASSWORD: "{{ doppler_secret_var('SNOWFLAKE_PASSWORD') }}"
```

To use Doppler, configure the Mage runtime with `DOPPLER_TOKEN`, `DOPPLER_PROJECT`, and `DOPPLER_CONFIG`. See [Doppler secrets](/production/deploying-to-cloud/secrets/Doppler) for setup, cache behavior, per-reference project and config overrides, and troubleshooting.

**File Content (Mage Pro only):**

```yaml theme={"system"}
version: 0.1.1
default:
  # Load from local file
  POSTGRES_PASSWORD: "{{ file('secrets/postgres_password.txt') }}"
  # Load JSON field from file
  SNOWFLAKE_PASSWORD: "{{ json_value(file('config/database.json'), 'snowflake_password') }}"
```

**JSON Value Extraction:**

```yaml theme={"system"}
version: 0.1.1
default:
  # Extract specific field from JSON stored in environment variable
  POSTGRES_PASSWORD: "{{ json_value(env_var('DB_CONFIG_JSON'), 'postgres_password') }}"
  # Extract from JSON stored in Mage secret
  SNOWFLAKE_PASSWORD: "{{ json_value(mage_secret_var('database_config'), 'snowflake_password') }}"
```

### 4. Multiple Profiles

You can create multiple profiles for different environments or use cases:

```yaml theme={"system"}
version: 0.1.1
default:
  # Default profile configuration
  POSTGRES_DBNAME: production_db

development:
  # Development environment configuration
  POSTGRES_DBNAME: development_db

staging:
  # Staging environment configuration
  POSTGRES_DBNAME: staging_db
```

## Managing profiles in the UI

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=io_config)

You can create, update, delete, and test `io_config.yaml` profiles from the Mage UI:

1. Open **Settings**.
2. In **Workspace**, select **Data connections**.
3. Select an existing profile or choose **Create New** next to the profile dropdown. You can also open `/settings/workspace/io-config` directly.

Viewers can inspect profiles with masked credentials. Creating, renaming, saving, deleting, and testing requires Editor access plus notebook edit access; otherwise, the page is read-only.

### Create and save profiles

* Enter a profile name before choosing a provider or editing YAML. New profiles start with **Full profile** and an empty YAML mapping.
* Select **Save**, press **Command+S** on macOS, or press **Ctrl+S** on Windows and Linux.
* Profile names can contain up to 128 letters, numbers, periods, underscores, and hyphens. They must start with a letter or number, cannot contain spaces, and cannot be `version`.
* Mage confirms before deleting a profile or discarding an unsaved name or configuration change.
* If another user changes `io_config.yaml`, Mage rejects your stale save or delete. Reload before applying the edit again.

### Edit provider configuration

Select a provider to edit only its keys, or select **Full profile** to edit the complete profile YAML.

* The backend supplies the provider list, YAML templates, and connection-test capabilities.
* Checked providers show their configured-value count. Blank template defaults are not saved or counted.
* Saving one provider preserves unrelated provider keys in the same profile. Removing a key removes it only from the selected provider scope.
* **Clear provider keys** resets that provider's draft; select **Save** to persist the removal.
* Provider-scoped editing rejects mixed or custom keys. Use **Full profile** for those keys.
* Providers can share credential groups, such as AWS or DuckDB keys. Editing or clearing a shared group affects every provider that uses that group but preserves unrelated keys.

Switching providers with an unsaved draft prompts you to **Keep editing** or **Discard changes**.

### Work with masked secrets

Sensitive values appear as `********`.

* Leave the mask unchanged at the same YAML path to preserve the saved value.
* Replace it to update the secret, or remove the key to delete the value.
* Mage rejects a mask when no saved secret exists at that path.
* Lists containing masked secrets must remain unchanged. Replace the masks with explicit values before adding, removing, reordering, or editing list items.

### YAML limitations and recovery

Profiles must be YAML mappings with string keys. A profile can be at most 1 MiB, and the full `io_config.yaml` file can be at most 10 MiB.

The UI rejects YAML tags, sets, binary values, timestamps, anchors, aliases, merge keys, and other non-JSON-safe or cross-profile graph features without rewriting the file. Edit `io_config.yaml` directly to preserve or remove unsupported constructs.

* For invalid YAML, fix the file and select **Retry**.
* For a symbolic link, edit the link target directly; the UI can read the file but cannot save it.

### Test a connection

Select **Test** to check the current editor contents without saving them. The action is available only when the backend advertises testing for the selected provider.

* Mage renders Jinja expressions such as `{{ env_var('POSTGRES_PASSWORD') }}` using current runtime values.
* Tests stop after 10 seconds, and only one test can run per project at a time.
* Most tests open and close a connection without previewing, extracting, or writing data. Success confirms only that lightweight check.
* Trino also runs `SELECT 1` and requires catalog, host, and user configuration.
* Mage shows specific configuration errors after redacting known secrets. Provider and driver exceptions use the generic connection-failure message.

### Workflow checklist

When you validate a workspace manually, check the main profile lifecycle rather than editing `io_config.yaml` directly:

1. Add a profile, choose a provider, edit its YAML, test when available, and save.
2. Reload and confirm the profile persists, credentials are masked, and unchanged masks preserve saved values.
3. Save two providers in one profile and confirm each keeps its own configuration.
4. Clear one provider, save, and confirm unrelated provider keys remain.
5. Confirm invalid YAML, mixed provider keys, stale revisions, read-only access, and delete confirmation block unsafe writes.

## Using io\_config in Your Code

### SQL Block Example

In SQL blocks, you can select the desired IO Config profile from the UI using the "Profile" dropdown menu. For detailed information about configuring SQL blocks, see the [SQL Blocks documentation](/guides/blocks/sql-blocks#configure-sql-block).

### Python Block Examples

You can use `io_config` in your Python code with different profile selection strategies:

**Basic Profile Usage:**

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data_from_postgres(**kwargs):
    query = 'SELECT * FROM your_table'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'  # or any other profile name

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

**Dynamic Profile Selection:**

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data_from_postgres(**kwargs):
    query = 'SELECT * FROM your_table'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    
    # Get profile from pipeline variables or use default
    config_profile = kwargs.get('profile', 'default')
    
    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

**Environment-based Profile Selection:**

```python theme={"system"}
import os
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data(**kwargs):
    query = 'SELECT * FROM your_table'
    config_path = path.join(get_repo_path(), 'io_config.yaml')

    # Automatically select profile based on environment
    environment = os.getenv('ENV', 'development')
    config_profile = environment  # 'development', 'staging', 'production'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

## Databases using io\_config.yaml

The following database and storage integrations use `io_config.yaml` for credentials and connection settings. See each doc for the exact keys and examples:

**Relational & SQL databases**

* [PostgreSQL](/integrations/databases/PostgreSQL)
* [MySQL](/integrations/databases/MySQL)
* [Microsoft SQL Server](/integrations/databases/MicrosoftSQLServer)
* [SQLite](/integrations/databases/SQLite)
* [Redshift](/integrations/databases/Redshift)
* [Trino](/integrations/databases/trino)
* [OracleDB](/integrations/databases/OracleDB)
* [ClickHouse](/integrations/databases/ClickHouse)
* [DuckDB](/integrations/databases/DuckDB)
* [MotherDuck](/integrations/databases/MotherDuck)

**Cloud data warehouses & analytics**

* [BigQuery](/integrations/databases/BigQuery)
* [Snowflake](/integrations/databases/Snowflake)
* [Databricks](/integrations/databases/Databricks)
* [Athena](/integrations/databases/Athena)
* [Microsoft Fabric Warehouse](/integrations/databases/MicrosoftFabricWarehouse)

**Storage & data lakes**

* [S3](/integrations/databases/S3)
* [Google Cloud Storage](/integrations/databases/GoogleCloudStorage)
* [Azure Blob Storage](/integrations/databases/AzureBlobStorage)
* [Azure Data Lake Storage Gen2](/integrations/databases/AzureDataLakeStorage)
* [Iceberg](/integrations/databases/Iceberg)

**Other databases & services**

* [MongoDB](/integrations/databases/MongoDB)
* [Druid](/integrations/databases/Druid)
* [Pinot](/integrations/databases/Pinot)
* [Google Sheets](/integrations/databases/GoogleSheets)
* [Spark](/integrations/databases/Spark)
* [Power BI](/integrations/databases/PowerBI)

**Vector stores & search**

* [Chroma](/integrations/databases/Chroma)
* [Qdrant](/integrations/databases/Qdrant)
* [Weaviate](/integrations/databases/Weaviate)
* [Algolia](/integrations/databases/Algolia)

## Best Practices

1. **Security**:
   * Never commit sensitive credentials directly in the `io_config` file
   * Use environment variables or secret variables for sensitive information
   * Consider using a secrets management service for production environments

2. **Organization**:
   * Use meaningful profile names
   * Group related configurations together
   * Document any non-standard configurations
   * Use the standard format for new configurations

3. **Version Control**:
   * Add `io_config.yaml` to your `.gitignore` file
   * Provide a template file (`io_config.yaml.template`) with dummy values
   * Document the required environment variables

## Troubleshooting

### Common Issues & Solutions

**1. "FileNotFoundError" or "No such file or directory"**

* Make sure your `io_config.yaml` file exists in your project root (e.g., `/home/src/your_mage_project/io_config.yaml`).
* If running in a different environment (e.g., Docker), confirm the file is mounted and accessible.

**2. "KeyError" or "Missing profile"**

* Double-check that the profile name you are referencing in your code (e.g., `'default'`, `'staging'`) exists in your `io_config.yaml`.
* Profile names are case-sensitive.

**3. "Missing required field" or "NoneType" errors**

* Ensure all required fields for your data source/destination are present in the selected profile.

**4. "Environment variable not set"**

* If you use `{{ env_var('VAR_NAME') }}` in your config, make sure the environment variable is set in deployment environment.
* You can check this by running `echo $VAR_NAME` in Mage terminal.

**5. YAML syntax errors**

* Use a YAML linter or validator to check for indentation or formatting issues.
* Strings containing special characters (like `:` or `{}`) should be quoted.

**6. "Invalid credentials" or authentication failures**

* Double-check your credentials and permissions.
* For cloud services, ensure your service account or IAM user has the necessary roles.

**7. "Cannot import ConfigFileLoader" or similar import errors**

* Make sure you are importing from `mage_ai.io.config` (not `mage_ai.io.io_config`).
* Example:
  ```python theme={"system"}
  from mage_ai.io.config import ConfigFileLoader
  ```

**8. "Profile not selected" in SQL blocks**

* In the Mage UI, select the correct profile from the "Profile" dropdown before running your SQL block.

If you continue to have issues, ask for help in the [Mage community Slack](https://mage.ai/chat).

## Additional Resources

* [Variables Overview](/development/variables/overview)
* [Data Integration Guide](/data-integrations/overview)
* [Pipeline Configuration](/design/data-pipeline-management)


# Start page
Source: https://docs.mage.ai/development/project/landing-page

Choose which page opens first when you launch Mage.

By default, when you enter a workspace, Mage takes you to the
**Pipelines page** (`/pipelines`). You can change which page opens
first by picking a different option in Workspace Preferences.

## Available start screens

| Option | Route |
| - | - |
| Pipelines (default) | `/pipelines` |
| Monitoring | `/overview` |
| Runs | `/pipeline-runs` |
| Triggers | `/triggers` |

## Steps to change your start screen

1. Open the account menu, select **Settings**, then select **Workspace** →
   **Preferences**.
2. In the **Workspace** section, find **Start page**.
3. In the dropdown, pick the page you want to
   land on when you next open the workspace.
4. Select **Save Settings**.

The change takes effect the next time you open the workspace home
URL (`/`) — for example by clicking the Mage logo or refreshing.
You can change it back at any time.

> [!NOTE]
>
> This preference is saved in your **browser's local storage**, not in
> your project's `metadata.yaml` file. As a result:
>
> * It applies only to the browser where you set it. Other browsers,
>   other devices, and other team members will keep their own
>   preference (or the default Pipelines page).
> * Clearing your browser data resets it to the default.
> * Open tabs do not sync with each other until they are reloaded.
>
> Instance managers always land on the `/manage` page regardless of
> this setting.


# Local timezone
Source: https://docs.mage.ai/development/project/local-timezone

Display timestamps throughout the app in your local timezone.

You can display dates on various pages (e.g. the Pipelines list and
Triggers/Pipeline runs/Block runs pages) in your local timezone, as
well as configure the start date for triggers and backfills in local
time. In the Version control terminal, this setting also controls new
terminal session timezone behavior: sessions use UTC by default, and use your
browser's local IANA timezone for prompt timestamps and the `TZ` environment
variable when **Use local timezone** is enabled.

## Steps to enable local timezone setting

1. Open the account menu, select **Settings**, then select **Workspace** →
   **Preferences**.
2. Under **Pipeline settings**, turn on **Use local timezone**.
3. Select **Save Settings** to save and persist the
   new settings. If you do not click this button, you will lose any changes when
   leaving the page.

* Alternatively (starting with v0.9.48), you can simply click on the time in the header
  and toggle on the `Display local timezone` setting from the dropdown menu. This may
  require a page refresh for the timestamps to change to UTC or local time.

> [!NOTE]
>
> Please note that certain pages (e.g. a pipeline's Monitor page) or components
> (e.g. Pipeline run bar charts) may still be in UTC time. These UTC dates may
> only display the date (e.g. 2023-09-01) without any time. Dates in local time
> will have a timezone offset (e.g. -07:00) in the timestamp (e.g.
> 2023-09-01T12:30:45-07:00).


# Project metadata.yaml
Source: https://docs.mage.ai/development/project/metadata

Reference for configuring project-level settings in metadata.yaml.

## Overview

Each Mage project has a `metadata.yaml` file at the root of the project
directory. The file controls storage locations, feature flags, compute
configuration, and defaults that apply to all pipelines in the project.

```
your_project/
├─ pipelines/
├─ .../
├─ io_config.yaml
├─ metadata.yaml   <-- project metadata lives here
├─ requirements.txt
```

Use the fields below to customize how your project behaves. For
environment-specific overrides, add an `overrides` section (see
`/extensibility/env-config/project`).

## Sample `metadata.yaml`

```yaml theme={"system"}
project_type: standalone
variables_dir: ~/.mage_data
remote_variables_dir: s3://bucket/path_prefix
help_improve_mage: true

notification_config:
  alert_on:
    - trigger_failure
  slack_config:
    webhook_url: "{{ env_var('MAGE_SLACK_WEBHOOK_URL') }}"

pipelines:
  settings:
    triggers:
      save_in_code_automatically: true

project_uuid: 123456789abcdefg
```

## Top-level fields

**project\_type** `string enum`

Project layout. Common options: `standalone` (default), `main`, or `sub`.

**cluster\_type** `string`

Optional cluster type used by the project (e.g., `k8s`, `docker`). Used by Mage Pro
when running managed workspaces.

**project\_uuid** `string`

Unique identifier for the project. Generated when the project is created.

**variables\_dir** `string`

Local path for storing pipeline variables and outputs. Paths are relative to
the project root unless an absolute path is provided. Default to `/home/src/mage_data`.

**remote\_variables\_dir** `string`

Remote path (e.g., `s3://bucket/prefix`) for storing variables in object
storage instead of the local filesystem.

**variables\_retention\_period** `string`

How long variables are retained (e.g., `90d`).

**workspace\_initial\_metadata** `object`

Default project metadata applied to new workspaces created from this project.
Mage Pro workspaces only.

**workspace\_config\_defaults** `object`

Default workspace configuration applied across environments (e.g., k8s
defaults when running on Kubernetes). Mage Pro workspaces only.

**help\_improve\_mage** `boolean`

Allows Mage to collect limited telemetry to improve the product.

**features** `object`

Feature flags for the project (e.g., `command_center`, `dbt_v2`,
`automatic_kernel_cleanup`). Keys are booleans.

**pipelines.settings.triggers.save\_in\_code\_automatically** `boolean`

Whether new or updated triggers are automatically written to code.

**overrides** `object`

Environment-specific overrides for any top-level field. Mage Pro only. See
`/extensibility/env-config/project`.

## Compute and execution

**emr\_config** `object`

Amazon EMR cluster settings (instance types, security groups, key pair, etc.).

**spark\_config** `object`

Spark configuration shared across pipelines (e.g., `spark_master`,
`executor_env`, `spark_jars`, `use_custom_session`).

**ecs\_config** `object`

Project-level defaults for AWS ECS execution.

**gcp\_cloud\_run\_config** `object`

Project-level defaults for GCP Cloud Run execution.

**azure\_container\_instance\_config** `object`

Project-level defaults for Azure Container Instances execution.

**k8s\_executor\_config** `object`

Kubernetes executor defaults applied to pipelines and blocks.

**concurrency\_config** `object`

Limits and concurrency settings at the project level.

**queue\_config** `object`

Configuration for queueing pipeline runs.

**state\_store\_config** `object`

State store configuration used by pipelines.

## Observability and safety

**notification\_config** `object`

Project-level alerting configuration (alert types, Slack/Teams webhooks, email,
managed delivery in Mage Pro, etc.). You can edit this in `metadata.yaml`.
In Mage Pro, you can also edit it from **Settings > Workspace > Alerts** in the
Mage UI. Pipeline-level alert settings override this project-level configuration
for that pipeline.

### Managed notification delivery

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=managed-notification-delivery)

**notification\_config.managed\_config** `object`

Mage Pro managed alert delivery configuration. Set `enabled` to `true`, use
`channels: [email]`, and optionally add recipient email addresses under
`email_config.to_emails`. Leave `to_emails` empty to use managed API default
recipients when the operator has configured them. You can also set `timeout`,
which is capped at `30` seconds.

```yaml theme={"system"}
notification_config:
  managed_config:
    enabled: true
    channels:
      - email
    email_config:
      to_emails:
        - data-oncall@example.com
    timeout: 10
```

**logging\_config** `object`

Configure log destinations and formats for the project.

**retry\_config** `object`

Default automatic retry behavior applied to block runs at the project level.
In Mage Pro, you can configure these values from **Settings > Workspace >
Automatic retry** or by opening `/settings/workspace/automatic-retry`.
Users with viewer access can open the page, but the retry controls are
read-only.
Mage merges retry settings from least specific to most specific: project
defaults, then pipeline-level `retry_config`, then block-level `retry_config`.
For the same retry field, the most specific configured value wins. Blank or
`null` retry fields are ignored during this merge, so a blank pipeline or
block field inherits the next available value from the less-specific config
instead of clearing it.

```yaml theme={"system"}
retry_config:
  retries: 3
  delay: 5
  max_delay: 60
  exponential_backoff: true
```

* `retries`: number of retry attempts after a block run fails. Use `0` to
  disable global automatic retries. Must be a non-negative integer up to `100`.
* `delay`: seconds to wait before the next retry attempt. Must be a
  non-negative integer up to `86400`.
* `max_delay`: maximum accumulated retry delay in seconds before scheduling
  the next attempt. This cap applies whether or not exponential backoff is
  enabled.
  Must be a non-negative integer up to `86400`.
* `exponential_backoff`: whether to increase the delay between retry attempts.
  If omitted, Mage uses the default enabled backoff behavior.

In Mage Pro, blank project-level retry fields are omitted from saved project
metadata, and `0` is accepted for numeric retry fields.

**ai\_config** `object`

Global AI-related settings (e.g., model providers).

**rag\_config** `object`

Retrieval-augmented generation settings shared across pipelines.

**openai\_api\_key** `string`

Project-level OpenAI API key used by AI features when applicable.

**ldap\_config** `object`

Optional LDAP connection settings for authentication.


# Project setup
Source: https://docs.mage.ai/development/project/setup

How to set up a folder containing your Mage project code and other files.

## Parent folder

Create a parent folder that will contain your Mage project code:

```bash theme={"system"}
mkdir data_monorepo
```

Change directory into the parent folder:

```bash theme={"system"}
cd data_monorepo
```

## Create a Dockerfile

Create an empty Dockerfile:

```bash theme={"system"}
echo "" > Dockerfile
```

Paste the contents from this [Dockerfile](https://github.com/mage-ai/docker/blob/master/Dockerfile)
into the Dockerfile you just created.

## Create a .gitignore file

Create an empty .gitignore file:

```bash theme={"system"}
echo "" > .gitignore
```

Paste the following contents into the .gitignore file you just created:

```
.DS_Store
.file_versions
.gitkeep
.log
.logs/
.mage_temp_profiles
.preferences.yaml
.variables/
__pycache__/
docker-compose.override.yml
logs/
mage-ai.db
mage_data/
secrets/
```

## Create Mage project

Follow these [instructions](/getting-started/setup) to create a Mage project in your
current directory. If you named your project `demo_project`, your current folder structure
will look like this:

```
data_monorepo/
|-- demo_project/
|-- .gitignore
|-- Dockerfile
```


# Configuring Mage settings backend
Source: https://docs.mage.ai/development/settings-backend



## Overview

> [!NOTE]
>
> Available in Mage versions >= 0.9.66
>
> Not all Mage settings can be read through a settings backend. You can find a list of settings
> that can be read through a settings backend [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/settings/keys/__init__.py).
>
> When any settings backend is enabled, Mage will first attempt to read the settings value
> from the settings backend before falling back to environment variables.

By default, many of Mage's configurable settings are read from environment variables. This may make sense
for more settings, but there are some settings that are too sensitive to be stored as environment
variables.

You can configure a settings backend to read settings from a different source. To set this up, you will
need to add a `settings_backend` section to your project's `metadata.yaml` file.

```yaml theme={"system"}

settings_backend:
  backend_type: ...

```

Only the `backend_type` field is required for all settings backends. The other fields will be passed
to the backend as keyword arguments when initializing the backend.

See below sections for the different types of settings backends that are available.

## AWS Secrets Manager

When the AWS Secrets Manager settings backend is enabled, Mage will read the value from the
AWS secret name: `{prefix}{settings_name}`. For example if the setting is `OKTA_CLIENT_SECRET`
and the prefix is `default_repo/settings/`, Mage will fetch the value for `default_repo/settings/OKTA_CLIENT_SECRET`.

```yaml theme={"system"}

settings_backend:
  backend_type: aws_secrets_manager
  prefix: default_repo/settings/  # optional
  use_cache: true  # optional
  cache_config:  # optional
    max_cache_size: 1024
    exception_retry_delay_base: 1
    exception_retry_growth_factor: 2
    exception_retry_delay_max: 3600
    default_version_stage: "AWSCURRENT"
    secret_refresh_interval: 3600  # in seconds

```

Optional fields:

* `prefix`: A prefix to add to the settings name when fetching the value from AWS Secrets Manager.
* `use_cache`: Whether to use a cache for the settings backend. Defaults to `false`.
* `cache_config`: configuration options for the cache if `use_cache` is `true`.
  * `max_cache_size` (int): The maximum number of secrets to cache.
  * `exception_retry_delay_base` (int): The number of seconds to wait after an exception is encountered and before retrying the request.
  * `exception_retry_growth_factor` (int): The growth factor to use for calculating the wait time between retries of failed requests.
  * `exception_retry_delay_max` (int): The maximum amount of time in seconds to wait between failed requests.
  * `default_version_stage` (str): The default version stage to request.
  * `secret_refresh_interval` (int): The number of seconds to wait between refreshing cached secret information.


# Troubleshooting
Source: https://docs.mage.ai/development/troubleshooting

Debugging common issues in Mage and other frequently asked questions.

## Pipeline edit page freezes browser

The data output of a block contains too much information for the browser to render.

To temporarily fix this, open the `/terminal` page in Mage.
Then type the following command press enter to run it:

```bash theme={"system"}
rm -rf mage_data/[project_name]/pipelines/[pipeline_name]/.variables
```

> [!NOTE]
>
> Change `[project_name]` to your Mage project name and change `[pipeline_name]` to the name of the pipeline that is freezing the browser.

The above command will remove the data output files for each block in your pipeline.

***

## Can’t connect to local database from within Mage via Docker

If you’re running Mage via Docker and you’re trying to connect to a local database outside of
Docker, you must use the following `host` or `IP` value when trying to connect to it:
`host.docker.internal`.

***

## Database migration fails during startup

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=troubleshooting)

If Mage fails to start because an Alembic or SQLAlchemy database migration failed, run `mage db diagnose` to inspect the current migration state and get the next recommended command.

See [Troubleshoot database migration failures](/development/databases/default#troubleshoot-database-migration-failures) for `mage db history`, `mage db schema`, `mage db migrate`, `mage db rollback`, and `mage db stamp` recovery steps.

***

## Unpredictable Columns in an Upstream Dataframe

Sometimes, upstream data is out of our control. If we're expecting a column in a dataframe and writing SQL against it, we might see a `SQL compilation` error: `invalid identifier.`

![Oh no](https://media0.giphy.com/media/t1RCtCTlFUhby/200w.gif)

Thankfully, [Mage Community](https://www.mage.ai/chat) member [Josh Pachner](https://mageai.slack.com/team/U05B9DLK0N7) has a solution.

Say you have an upstream block that looks like:

```SQL theme={"system"}
SELECT
  df."col_1",
  df."col_2",
  df."col_3",
FROM {{ df }} df
```

Where the underlying `df` might all three columns... or just a subset. For example, `col_1` isn't always present.

For cases where expected columns *might* not in our dataframe, we can add a transformer block upstream of an exporter:

```python theme={"system"}
# headers = [cols we need in our dataframe]
# df = our DataFrame()

for head in headers:
  if head not in df.columns:
       df[head] = None
```

This is especially helpful in databases like Snowflake, which don't have an `iferror()` function to check for missing columns.

***

## `InvalidToken` error when accessing data in S3

Accessing data located in S3, you may see an error like this:

```text theme={"system"}
An error occurred (InvalidToken) when calling the GetObject operation:
The provided token is malformed or otherwise invalid.
```

This [Stack Overflow](https://stackoverflow.com/questions/54837248/an-error-occurred-invalidtoken-when-calling-the-listbuckets-operation-the-pr) article details the issue. Removing the `AWS_SESSION_TOKEN` line from the `io_config.yaml` should resolve the problem.

## Mage doesn't load properly when running locally

When attempting to connect to Mage with a port *other* than `6789`, the browser may not render the page correctly with `localhost:port`.

Instead, use `127.0.0.1:port` to connect to Mage.

## Using credential files in Docker

If you have files containing secret credentials (e.g. GCP credentials JSON files), you’ll need to mount them as a volume so that those files appear in your running Docker container.

Use the `-v` command in your `docker run` command to mount the directory containing your files:

```bash theme={"system"}
docker run -it -p 6789:6789
    -v /Users/dragon/secrets:/home/secrets \
    -v $(pwd):/home/src mageai/mageai \
    /app/run_app.sh mage start [project_name]
```

or, using `docker compose`, add a volume mount to your Mage service. The following is an example of mounting the `~/.aws` directory for authenticating via the AWS CLI:

```yaml theme={"system"}
services:
  mage:
    image: mageai/mageai:latest
    ...
    volumes:
      - ~/.aws:/root/.aws
```

## "Run block" button doesn't work when running Mage with Nginx

If Mage terminal or "Run block" button doesn't work, it usually means there's a WebSocket connection issue. If Mage server is deployed with Nginx, you can add the below config to Nginx config to enable WebSocket connection.

```
        location / {
          proxy_pass      http://127.0.0.1:6789;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "Upgrade";
          proxy_set_header Host $host;
        }
```

## Block gets stuck in running status

If your block gets stuck in running status in the Mage editor, you can try the following things:

* [Restart the kernel](/getting-started/kernels#restart-kernel) and refresh the page.
* Check whether the CPU and memory usage of the instance is too high. Increase the CPU and memory if the usage is above 50%.

## Server fails to start or keeps restarting due to healthcheck failures

When deploying your Mage server on Cloud (Kubernetes, AWS, GCP, Azure, etc.) with Terraform scripts or Helm chart, there're health checks set up for the
cloud services. If you notice the server keeps restarting, one possible reason is the health check keeps failing due to timeout. You can try increasing the timeout or the initial delay of the health checks to resolve the issue.


# Updating Mage
Source: https://docs.mage.ai/development/updating-mage

How to get the latest and greatest from the Mage team.

> **Docker**
>
> ```bash theme={"system"}
> docker pull mageai/mageai:latest
> ```
>
> > [!TIP]
> >
> > 1. Are you a technologist? *Yes, you are from the future*
> > 2. Do you live on the edge of the galaxy? *Yes you do Sith Lord of the Empire*
> >
> > If you said `Yes`, `No`, or didn’t reply to any of these,
> > you should download the `alpha` build instead:
> >
> > ```bash theme={"system"}
> > docker pull mageai/mageai:alpha
> > ```

> **pip/conda**
>
> ```bash pip theme={"system"}
> pip install -U mage-ai
> ```
>
> ```bash conda theme={"system"}
> conda update mage-ai
> ```


# Mage block variables
Source: https://docs.mage.ai/development/variables/block-variables

Use pipeline-specific variables scoped to an individual block.

> [!NOTE]
>
> Requires version `0.9.23` or greater.

In addition to the global runtime variables that can be used by any block in
a pipeline, there are also variables scoped at the block-level, meaning
they are accessible in a specific block but not other blocks in the same pipeline.
Block variables are not accessible by other pipelines that use the same block.

## Using block variables

Block variables can be accessed via the `kwargs` parameter in your block function.
The variables are stored in a block's configuration, which is accessible through
`kwargs['configuration']`.

```python theme={"system"}
@data_exporter
def export_data(*args, **kwargs):
    demo_block_var = kwargs['configuration'].get('demo_block_var')

```

### Supported block types

Block variables are currently only supported for Python blocks with the
following block types:

1. `data_loader`
2. `transformer`
3. `data_exporter`
4. `custom`
5. `sensor`

## Creating block variables

### In Mage UI

You can create new block variables from the Mage UI through the "Block settings" tab
in the Sidekick of the Pipeline Editor page. The block settings can also be quickly accessed
by clicking the settings icon in a block's header.

In the "Block variables" section of the block settings, click the `+ New` button, enter
the variable's name and value in the input fields and then press `Enter` or `Return` on
your keyboard while still focused on one of the input fields (this step can be repeated
multiple times).

This will add the variable(s), but not save it. You need to also click the blue
`Update block settings` button below the "Block variables" section in order to save
and persist these new variables. You can also edit/delete existing block variables by
hovering over a variable and clicking the edit/trash icons. You can press the `Escape`
key to exit a variable's edit mode.

### In code

You can also create or edit block variables by going to a pipeline's `metadata.yaml` file
(located in the `pipelines/[pipeline_uuid]` directory) and adding them to a block's
`configuration` property.

```yaml theme={"system"}
blocks:
- all_upstream_blocks_executed: true
  color: null
  configuration:
    demo_block_var: block_var_value
    block_var2: val2
    block_var3: val3
  downstream_blocks: []
  executor_config: null
  executor_type: local_python
  has_callback: false
  language: python
  name: billowing snow
  retry_config: {}
  status: updated
  timeout: '60'
  type: data_exporter
  upstream_blocks:
  - muddy_waterfall
  uuid: billowing_snow
```


# Mage environment variables
Source: https://docs.mage.ai/development/variables/environment-variables

Configure Mage settings through environment variables.

## Environment variables in Mage Pro

Environment variables are set from the Mage Pro cluster management portal. Check out the steps and video below on how to create environment variables.

### Adding or modifying environment variables:

* Click the "Edit / add variables" button
* On the Environment Variables screen, you can either input a value directly or add a custom environment variable by specifying its key-value pair
* Click "Save changes" to update the settings.

[YouTube video player](https://www.youtube.com/embed/ZiG67ldzMUE?si=fVCPtZvJfSidpnA_)

> [!NOTE]
>
> After clicking the "Save changes" button the cluster will automatically restart.

## Environment variables in Mage OSS

Environment variables may be set in your Docker container to configure Mage settings. If you’re running Mage using `pip` or `conda`, your local machine’s environment variables
are accessible within the running Mage app.

Environment variables are different from [runtime variables](/getting-started/runtime-variable) or [secrets](/development/variables/secrets).

If you’re running Mage using Docker, you must add the following command line flags:

```bash theme={"system"}
-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2
```

The command to run Mage using Docker could look like this:

```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
    -e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2 \
    mageai/mageai /app/run_app.sh mage start [project_name]
```

## Setting the prefix for Mage URL

You may need to add a prefix to the Mage url. You can configure the Mage server to start
at a prefix to the url by setting the `MAGE_BASE_PATH` environment variable.

Make sure to set the base path without the leading slash. For example, if you want to start Mage at
`http://localhost:6789/base/url/path`, you would configure your environment
variable like this:

`export MAGE_BASE_PATH=base/url/path`

You can also configure the base path separately for client requests and backend routes by setting
the `MAGE_REQUESTS_BASE_PATH` and `MAGE_ROUTES_BASE_PATH` environment variables respectively.

| | |
| - | - |
| `MAGE_BASE_PATH` | Set base path for the entire app. |
| `MAGE_REQUESTS_BASE_PATH` | Set base path for frontend requests and urls, defaults to value of `MAGE_BASE_PATH` |
| `MAGE_ROUTES_BASE_PATH` | Set base path for backend routes, defaults to value of `MAGE_BASE_PATH` |

## Default variables

These are the environment variables used in Mage.

| Variable name | Description | Example values |
| - | - | - |
| `ACTIVE_DIRECTORY_DIRECTORY_ID` | Set this value to your organization's tenant id to enable signing in with Microsoft account. | [More info](/production/authentication/microsoft) |
| `ALLOW_DOWNLOADS_OUTSIDE_PROJECT` | Allow signed file downloads for files outside the current project directory. | `0`, `1` |
| `AUTHENTICATION_MODE` | Set the authentication mode in Mage. Omitting the variable will default to basic password authentication. | `ldap` |
| `AI_SIDEKICK` | **(Mage Pro only.)** Controls AI Sidekick, which defaults to enabled when this variable is unset. Set it to `0` to hide AI chat entry points and reject Sidekick backend requests, including runtime runs, model and proxy session tokens, and voice transcription; set it to `1`, `true`, or `t` (case-insensitive) to enable Sidekick explicitly. | `1` (default), `0` |
| `DATAFRAME_SAMPLE_COUNT_PREVIEW` | Number of lines to display in block sample output preview for dataframes. Not suggested to go over 10 lines (defaults to 10). | 5 |
| `DEBUG` | Turn on "debug" mode. Mage will output more logs that could be useful for debugging issues. | 1 |
| `DEFAULT_EXECUTOR_TYPE` | The default executor type to run blocks | See enum types [here](https://docs.mage.ai/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) |
| `K8S_DEFAULT_CONFIG_PATH` | **(Mage Pro only.)** Path to YAML file for default K8s executor config; merged with block/project config. Prefer configuring via Helm `k8sExecutorConfig`. [K8s executor](/production/executors/k8#default-config-file) · [Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) | — |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_CPU` | **(Mage Pro only.)** Default CPU request for K8s executor jobs (when not specified in config) | `250m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_MEMORY` | **(Mage Pro only.)** Default memory request for K8s executor jobs (when not specified in config) | `512Mi` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_CPU` | **(Mage Pro only.)** Default CPU limit for K8s executor jobs (when not specified in config) | `500m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_MEMORY` | **(Mage Pro only.)** Default memory limit for K8s executor jobs (when not specified in config) | `1Gi` |
| `K8S_EXECUTOR_MAX_RESOURCE_REQUESTS_CPU` | **(Mage Pro only.)** Maximum CPU request guardrail for K8s executor jobs after env/project/pipeline/block config is merged. | `2000m` |
| `K8S_EXECUTOR_MAX_RESOURCE_REQUESTS_MEMORY` | **(Mage Pro only.)** Maximum memory request guardrail for K8s executor jobs after env/project/pipeline/block config is merged. | `4Gi` |
| `K8S_EXECUTOR_MAX_RESOURCE_LIMITS_CPU` | **(Mage Pro only.)** Maximum CPU limit guardrail for K8s executor jobs after env/project/pipeline/block config is merged. | `4000m` |
| `K8S_EXECUTOR_MAX_RESOURCE_LIMITS_MEMORY` | **(Mage Pro only.)** Maximum memory limit guardrail for K8s executor jobs after env/project/pipeline/block config is merged. | `8Gi` |
| `K8S_EXECUTOR_DEFAULT_SERVICE_ACCOUNT_NAME` | **(Mage Pro only.)** Default service account for K8s executor jobs. See [K8s executor docs](/production/executors/k8#environment-variables-reference) for more details | `default` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_URL` | **(Mage Pro only.)** Prometheus base URL for K8s executor resource usage summaries. See [Monitoring with Prometheus](/observability/external/prometheus#kubernetes-resource-usage-in-mage-pro). | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN` | **(Mage Pro only.)** Bearer token for Prometheus resource usage queries. Takes precedence over basic auth when set. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME` | **(Mage Pro only.)** Username for Prometheus basic authentication. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD` | **(Mage Pro only.)** Password for Prometheus basic authentication. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS` | **(Mage Pro only.)** Timeout for each Prometheus resource usage query. | `5` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` | **(Mage Pro only.)** Query range step for Prometheus CPU and memory samples. | `15` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS` | **(Mage Pro only.)** Delay before retrying Prometheus once for completed K8s executor runs with too few samples. | `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS` | **(Mage Pro only.)** Set to `0`, `false`, `no`, or `off` to skip TLS certificate verification for Prometheus resource usage queries. | `true` |
| `WORKSPACE_DEFAULT_CONFIG_PATH` | **(Mage Pro only.)** Path to YAML file for default workspace config; merged with user config. Prefer configuring via Helm `workspaceConfig`. [Workspaces](/guides/developer-ux/workspaces#default-workspace-config-file) · [Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) | — |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU` | **(Mage Pro only.)** Default CPU request for workspace containers (when not specified in container\_config). See [Workspaces docs](/guides/developer-ux/workspaces#default-resource-configuration) | `500m` |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY` | **(Mage Pro only.)** Default memory request for workspace containers (when not specified in container\_config) | `2Gi` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU` | **(Mage Pro only.)** Default CPU limit for workspace containers (when not specified in container\_config) | `4` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY` | **(Mage Pro only.)** Default memory limit for workspace containers (when not specified in container\_config) | `8Gi` |
| `DISABLE_NOTEBOOK_EDIT_ACCESS` | Setting this variable will disable edit access based on the value. [More information](/production/configuring-production-settings/overview#read-only-access) | 1 |
| `DISABLE_STREAMING_PIPELINE_MONITORING` | **(Mage Pro only.)** Set to `1` to disable Streaming health metric collection for streaming pipeline runs. Existing collected metrics remain available. | `0`, `1` |
| `DISABLE_TERMINAL` | Setting this variable will disable access to the terminal. | 1 |
| `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS` | Maximum number of dynamic child block runs a dynamic block may create in one expansion. Leave unset or set to `0` to allow unlimited child runs. | `1000` |
| `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS_BEHAVIOR` | Behavior when a dynamic block exceeds `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS`. Use `fail` to stop the run or `limit` to create only up to the configured maximum. | `fail`, `limit` |
| `DYNAMIC_BLOCKS_MAX_CONCURRENT_CHILD_BLOCKS` | Maximum number of dynamic child block runs that may be queued or running at the same time for one pipeline run. Leave unset or set to `0` to allow unlimited concurrent dynamic child runs. | `25` |
| `ENV` | Set the environment of the Mage app. You can use different code or configurations in different environments. | `dev`, `staging`, `production`/`prod`, `test` |
| `GIT_REPO_LINK` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_REPO_PATH` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_USERNAME` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_EMAIL` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_AUTH_TYPE` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_BRANCH` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SYNC_ON_PIPELINE_RUN` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SYNC_ON_START` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SSH_PUBLIC_KEY` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SSH_PRIVATE_KEY` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_ACCESS_TOKEN` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `HIDE_ENV_VAR_VALUES` | Filter out environment variables in notebook output. | 1 |
| `JUPYTER_KERNEL_IDLE_TIMEOUT_SECONDS` | **(Mage Pro only.)** Number of seconds an inactive per-user Jupyter kernel can stay alive after its websocket clients disconnect. Used when `MULTI_USER_JUPYTER_KERNELS=1`. | `1800` |
| `JUPYTER_KERNEL_MAX_PER_SERVER` | **(Mage Pro only.)** Maximum number of per-user Jupyter kernels to keep on one Mage server. When the limit is reached, Mage evicts idle kernels first. Used when `MULTI_USER_JUPYTER_KERNELS=1`. | `50` |
| `JWT_SECRET` | Modify the json web token secret used for encoding and decoding access tokens. Defaults to `materia`. | `your_jwt_secret` |
| `JWT_DOWNLOAD_SECRET` | Modify the json web token secret used for encoding and decoding download tokens. Defaults to randomly generated string value. Use an explicit value in horizontally scalable deployments, e.g.: kubernetes, cloud deployments | `your_jwt_secret` |
| `LDAP_ADMIN_USERNAME` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_SERVER` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_BIND_DN` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_BIND_PASSWORD` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_BASE_DN` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_AUTHENTICATION_FILTER` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_AUTHORIZATION_FILTER` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_GROUP_FIELD` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_ROLES_MAPPING` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_DEFAULT_ACCESS` | [More information](/production/authentication/overview#ldap) | See link |
| `LOGS_DIR_PATH` | Override the logs directory path. Default path is the variables\_dir path (e.g. `/home/src/mage_data/`) | `/home/src/logs_dir` |
| `MAGE_ACCESS_TOKEN_EXPIRY_TIME` | [More information](/production/authentication/overview#ldap) | See link |
| `MAGE_BASE_PATH` | The base path or prefix of the Mage server url. [More information](/getting-started/setup#setting-prefix-for-mage-url) | `base/path` |
| `MAGE_REQUESTS_BASE_PATH` | [More information](/getting-started/setup#setting-prefix-for-mage-url) | `base/path` |
| `MAGE_ROUTESBASE_PATH` | [More information](/getting-started/setup#setting-prefix-for-mage-url) | `base/path` |
| `MAGE_CACHE_DIRECTORY` | The directory Mage uses for storing and retrieving temporary data used in the UI. | `/root/.mage_data/default_repo/.cache` |
| `MAGE_CODEX_CHAT_ATTACHMENT_MAX_BYTES` | **(Mage Pro only.)** Maximum size, in bytes, for one Mage Pro AI Chat attachment. | `1073741824` |
| `MAGE_CODEX_CHAT_ATTACHMENT_MAX_SESSION_BYTES` | **(Mage Pro only.)** Maximum total attachment size, in bytes, allowed in one Mage Pro AI Chat session. | `2147483648` |
| `MAGE_CODEX_CHAT_ARCHIVE_ATTACHMENT_MAX_BYTES` | **(Mage Pro only.)** Maximum size, in bytes, for one Mage Pro AI Chat browser-zipped folder archive attachment. | `1073741824` |
| `MAGE_CODEX_CHAT_ARCHIVE_ATTACHMENT_MAX_SESSION_BYTES` | **(Mage Pro only.)** Maximum total attachment size, in bytes, allowed in one Mage Pro AI Chat session when the new attachment is a browser-zipped folder archive. | `2147483648` |
| `MAGE_CODEX_CHAT_ATTACHMENT_CONTENT_PREVIEW_BYTES` | **(Mage Pro only.)** Maximum bytes of text content from one Mage Pro AI Chat attachment that Codex receives in the uploaded-file manifest. | `16384` |
| `MAGE_CODEX_CHAT_ATTACHMENT_CONTENT_PREVIEW_TOTAL_BYTES` | **(Mage Pro only.)** Maximum total bytes of text attachment content previews sent to Codex for one chat turn. | `65536` |
| `MAGE_CODEX_CHAT_MAX_CONCURRENT_RUNS` | **(Mage Pro only.)** Maximum number of Codex chat turns that can run at the same time in one Mage process. Leave unset or set to `0` for unlimited. | unset |
| `MAGE_CODEX_CHAT_MODEL` | **(Mage Pro only.)** Default model used by Mage Pro AI Chat when a chat request does not override it. Falls back to `MAGE_CODEX_DEFAULT_MODEL` when unset. | `gpt-5.5` |
| `MAGE_CODEX_CHAT_PROGRESS_HEARTBEAT_SECONDS` | **(Mage Pro only.)** Seconds between Mage Pro AI Chat progress heartbeat events while the agent is running without a new tool, command, or final-response event. | `1` |
| `MAGE_CODEX_CHAT_RATE_LIMIT_RETRY_MAX_SECONDS` | **(Mage Pro only.)** Maximum time Mage Pro AI Chat waits and retries a Codex run after a model rate-limit response before showing the rate-limit error. Set to `0` to disable automatic retry. | `30` |
| `MAGE_CODEX_CHAT_RATE_LIMIT_RETRY_INITIAL_DELAY_SECONDS` | **(Mage Pro only.)** Initial delay before retrying a Codex run after a model rate-limit response. Mage increases the delay on later attempts up to `MAGE_CODEX_CHAT_RATE_LIMIT_RETRY_MAX_DELAY_SECONDS`. | `5` |
| `MAGE_CODEX_CHAT_RATE_LIMIT_RETRY_MAX_DELAY_SECONDS` | **(Mage Pro only.)** Maximum delay between automatic Codex run retry attempts after a model rate-limit response. | `30` |
| `MAGE_CODEX_CHAT_REASONING_EFFORT` | **(Mage Pro only.)** Default reasoning effort used by Mage Pro AI Chat when a chat request does not override it. The AI Chat execution-settings menu takes precedence for that chat. `ultra` is accepted as an alias for Codex CLI's highest supported value, `xhigh`. | `low`, `medium`, `high`, `xhigh`, `ultra` |
| `MAGE_CODEX_CHAT_SPEED` | **(Mage Pro only.)** Default Codex speed used by Mage Pro AI Chat when a chat request does not override it. The AI Chat execution-settings menu takes precedence for that chat. | `standard`, `fast` |
| `MAGE_CODEX_CHAT_SERVICE_TIER` | **(Mage Pro only.)** Default service tier used by Mage Pro AI Chat when a chat request does not override it. | `priority` |
| `MAGE_CODEX_DEFAULT_MODEL` | **(Mage Pro only.)** Default Codex model used by Mage Pro AI Chat and Mage agent MCP sessions when a more specific chat model is not configured. Spawned Codex sessions inherit this value. | `gpt-5.5` |
| `MAGE_SERVER_MAX_BODY_SIZE` | Maximum HTTP request body size, in bytes, accepted by the Mage Tornado server. When unset, Mage sets this to the larger AI Chat attachment limit plus multipart overhead so large uploads can reach the API resource and return normal JSON errors. | `max(MAGE_CODEX_CHAT_ATTACHMENT_MAX_BYTES, MAGE_CODEX_CHAT_ARCHIVE_ATTACHMENT_MAX_BYTES) + 67108864` |
| `MAGE_SERVER_MAX_BUFFER_SIZE` | Maximum HTTP request buffer size, in bytes, accepted by the Mage Tornado server. Defaults to `MAGE_SERVER_MAX_BODY_SIZE`. | `MAGE_SERVER_MAX_BODY_SIZE` |
| `MAGE_SIDEKICK_VOICE_DICTATION_DAILY_MAX_BYTES` | **(Mage Pro only.)** Maximum daily audio bytes accepted per user before Mage Pro refuses new Sidekick voice dictation proxy requests. | `524288000` |
| `MAGE_SIDEKICK_VOICE_DICTATION_LANGUAGE` | **(Mage Pro only.)** Default language code sent with Sidekick voice dictation transcription requests when the browser does not provide one. Invalid values fall back to English. | `en` |
| `MAGE_SIDEKICK_VOICE_DICTATION_MAX_BYTES` | **(Mage Pro only.)** Maximum audio upload size Mage Pro accepts and passes into the scoped OpenAI proxy token budget for Sidekick voice dictation. | `26214400` |
| `MAGE_SIDEKICK_VOICE_DICTATION_MAX_SECONDS` | **(Mage Pro only.)** Maximum client recording duration Mage Pro exposes to Sidekick and passes into the scoped OpenAI proxy token budget for voice dictation. | `120` |
| `MAGE_SIDEKICK_VOICE_DICTATION_MODEL` | **(Mage Pro only.)** Transcription model requested through the Mage Pro API OpenAI proxy for Sidekick voice dictation. | `gpt-4o-transcribe` |
| `MAGE_SIDEKICK_VOICE_DICTATION_MONTHLY_MAX_BYTES` | **(Mage Pro only.)** Maximum monthly audio bytes accepted per user before Mage Pro refuses new Sidekick voice dictation proxy requests. | `10737418240` |
| `MAGE_SIDEKICK_VOICE_DICTATION_PROMPT` | **(Mage Pro only.)** Prompt sent with Sidekick voice dictation transcription requests. Defaults to preserving Mage, Sidekick, Codex, pipeline, Kubernetes, GitHub, API, and other technical terms, and is capped at 1000 characters. | `Transcribe English voice dictation for Mage AI Sidekick...` |
| `MAGE_SIDEKICK_VOICE_DICTATION_RATE_LIMIT_PER_MINUTE` | **(Mage Pro only.)** Maximum Sidekick voice dictation transcription attempts accepted per user per minute before Mage Pro returns a rate-limit response. | `20` |
| `MAGE_DATABASE_CONNECTION_URL` | Specify the database connection url for orchestration DB. Defaults to local sqlite db. | [Example](/production/configuring-production-settings/overview#postgres) |
| `MAGE_DATABASE_APPLICATION_NAME` | Override the PostgreSQL `application_name` used by Mage DB connections. Defaults to a generated `mage:<cluster or project>:<hostname>` value. | `mage:production:server-1` |
| `MAGE_DATABASE_IDLE_IN_TRANSACTION_SESSION_TIMEOUT_MS` | PostgreSQL session timeout, in milliseconds, for idle transactions opened by Mage. Set to `0` to disable the per-session timeout. | `120000` |
| `MAGE_DATABASE_LOCK_TIMEOUT_MS` | Optional PostgreSQL per-session lock wait timeout, in milliseconds, for Mage DB connections. `0` disables this setting. | `0` |
| `MAGE_DATABASE_MAX_OVERFLOW` | Extra PostgreSQL connections a Mage process may open above `MAGE_DATABASE_POOL_SIZE`. | `2` |
| `MAGE_DATABASE_POOL_RECYCLE` | Number of seconds before SQLAlchemy recycles an idle PostgreSQL pooled connection. | `1800` |
| `MAGE_DATABASE_POOL_SIZE` | Base SQLAlchemy connection pool size per Mage process for PostgreSQL orchestration DB connections. Use small values for shared workspaces. | `5` |
| `MAGE_DATABASE_POOL_TIMEOUT` | Number of seconds a Mage request waits for a PostgreSQL pooled connection before failing with a pool exhaustion error. | `10` |
| `MAGE_DATABASE_STATEMENT_TIMEOUT_MS` | Optional PostgreSQL per-session statement timeout, in milliseconds, for Mage DB connections. `0` disables this setting. | `0` |
| `MAGE_PRO_API_API_KEY` | **(Mage Pro only.)** Server-side Mage Pro API key used when Mage Pro proxy mode or other Mage Pro API integrations call the shared Mage Pro API service. Mage Pro AI Chat does not pass this key to the Codex child process or Mage MCP tool environment. | `your_mage_pro_api_key` |
| `MAGE_PRO_API_ENDPOINT` | **(Mage Pro only.)** Mage Pro API host used by Mage Pro proxy mode and other Mage Pro API integrations. Do not include the URL scheme. Hosted production clusters use `pro-api.mage.ai`; staging hosts such as `pro-api-staging.mage.ai` are for non-production environments. | `pro-api.mage.ai` |
| `MAGE_PUBLIC_HOST` | The public host url that can be used to access the Mage app. This value will be used in emails or other notifications. | `http://localhost:6789` |
| `MAGE_RESTART_STREAMING_PIPELINES_ON_REQUIREMENTS_CHANGE` | **(Mage Pro only.)** When true, running streaming pipelines are restarted after the project `requirements.txt` changes. | 1 |
| `MAX_PRINT_OUTPUT_LINES` | The max number of `stdout` lines, such as from `print` statements, included in a message from the websocket server. Defaults to `1000`. | `100` |
| `MEMORY_USAGE_MAXIMUM` | **(Mage Pro only.)** Maximum aggregate memory utilization ratio for the local Python executor guard. At or above the threshold, Mage stops the pipeline run. Values must be greater than `0` and at most `1`; unset or invalid values use `0.95`. Restart the Mage process or workspace after changing it. Container usage is based on raw cgroup memory, including page cache; setting `1` leaves no guard headroom before an out-of-memory failure. | `0.95` |
| `MULTI_USER_JUPYTER_KERNELS` | **(Mage Pro only.)** Enable a separate Python Jupyter kernel for each authenticated user in the pipeline editor so multiple users can run blocks at the same time. | `0` |
| `REQUIRE_USER_AUTHENTICATION` | Enable user authentication in Mage. [More information](/production/authentication/overview) | 1 |
| `RUN_PIPELINE_IN_ONE_PROCESS` | When True, pipelines will run in one process, unless otherwise specified in the settings for a given pipeline. | 1 |
| `SERVER_VERBOSITY` | [More information](/development/observability/logging#server-logging) | See link |
| `SHELL_COMMAND` | Set shell command to use for the Mage terminal. Default command is `bash` for macOS/Unix and `cmd` for Windows. | `bash`, `cmd`, ... |
| `ULIMIT_NO_FILE` | Override the maximum number of open files allowed in Mage processes. | 8192 |
| `USER_CODE_PATH` | Set this path to be `/home/src/{your_project_name}` so that Mage starts your project and installs the packages in requirements.txt when starting the container. | `/home/src/mage_project` |


# Mage variables
Source: https://docs.mage.ai/development/variables/overview

There are a few ways to set up variables in Mage, here's how you can use them to make the best pipelines the world has ever seen. 🌎

There are four types of variable in Mage: Environment variables, runtime variables, block variables, and secrets.

| Variable Type | Scope | Creation | Reference | Encrypted |
| - | - | - | - | - |
| Environment variable | System | Code | `{{ env_var() }}` | 🚫 |
| Runtime variable | Pipeline | Code/UI | `kwargs` or `{{ variables() }}` | 🚫 |
| Block variable | Block | Code/UI | `kwargs['configuration'][key]` | 🚫 |
| Secret | Project | UI | `{{ mage_secret_var() }}` | ✅ |

Here's a brief overview of each:

## Environment variables

Environment variables are a type of variable that is used to store configuration settings and other system-level information for operating systems and applications. These variables are accessible to any process running on the system, which allows multiple programs to share the same configuration information. Environment variables exist in virtually all operating systems, including Linux, macOS, and Windows.

When running Mage in Docker, Mage will inherit the environment variables from the host machine. You can also set environment variables in the Dockerfile. Read more about environment variables in Mage [here](/development/variables/environment-variables) and Docker [here](https://docs.docker.com/compose/environment-variables/set-environment-variables/).

## Runtime variables

Runtime Variables are a set of global variables that can be used by every block in a pipeline. These are useful for storing constants shared by multiple blocks or constants whose value is determined at pipeline runtime (hence runtime variables).

If you're familiar with Airflow, runtime variables are similar to the Airflow "context" object. You can use runtime variables to pass information between blocks in a pipeline. Read more [here](/getting-started/runtime-variable).

## Block variables

In addition to the global runtime variables that can be used by any block in a pipeline, there are also variables scoped at the block-level, meaning they are accessible in a specific block but not other blocks in the same pipeline. Block variables are not accessible by other pipelines that use the same block. Read more [here](/development/variables/block-variables).

## Secrets

Secrets are a way to input sensitive information *via the Mage UI* that can be used by blocks in a pipeline. Secrets are stored in the Mage database and are encrypted. Secrets can come in handy when you have multiple users developing in an environment or are otherwise unable to use environment variables. Read more [here](/development/variables/secrets).


# Accessing variables in Mage
Source: https://docs.mage.ai/development/variables/referencing-variables

You've got all of these great variables and secrets, now let's put them to use. Learn how to access them in your code. 🦸🏻‍♂️

Mage allows users to interpolate variables specific to your [pipeline](/getting-started/runtime-variable) or [project](/development/variables/environment-variables). The following syntax is specific to `yaml` and `SQL` files.

Here's a list of the different variables and functions you can use in your code:

| Syntax | Description |
| - | - |
| `{{ env_var('secret') }}` | Get secret from [environment variables](/development/variables/environment-variables). |
| `{{ variables('secret') }}` | Get secret from [runtime variables](/getting-started/runtime-variable). |
| `{{ mage_secret_var('secret') }}` | Get secret from [Mage secrets](/development/variables/secrets). |
| `{{ aws_secret_var('secret') }}` | Get secret from [AWS Secrets Manager](/production/deploying-to-cloud/secrets/AWS). |
| `{{ azure_secret_var('secret') }}` | Get secret from [Azure Key Vault](/production/deploying-to-cloud/secrets/Azure). |
| `{{ json_value(json_obj, 'key') }}` | Extract value from a JSON string. |
| `{{ n_days_ago(N) }}` | Get the date `N` days ago. |

## Mage Pro only variable syntax

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=variables)

| Syntax | Description |
| - | - |
| `{{ file('path/to/file.txt') }}` | Load the contents of a [local file](/development/variables/referencing-variables#file-content-syntax). |
| `{{ doppler_secret_var('secret') }}` | Get secret from [Doppler](/production/deploying-to-cloud/secrets/Doppler). |

### File content syntax

This is useful for dynamically loading configuration files, secrets, or text assets during pipeline execution.
The path can be either absolute path or relative path to the project.

**Example Usage**

```jinja theme={"system"}
{{ file('secrets.txt') }}`
{{ file('/home/src/default_repo/settings.json') }}`
```

If you want to get a json field from the local file, you can use it together with the `json_value` syntax like

```
{{ json_value(file('path/to/file.json'), 'key') }}`
```

### Doppler secrets syntax

Use `doppler_secret_var` to read a secret directly from Doppler when the template is rendered. See [Doppler secrets](/production/deploying-to-cloud/secrets/Doppler) for complete setup, caching, and troubleshooting guidance.

```jinja theme={"system"}
{{ doppler_secret_var('POSTGRES_PASSWORD') }}
```

Mage reads the default Doppler project and config from these environment variables:

| Variable | Description |
| - | - |
| `DOPPLER_TOKEN` | Doppler token used to read secrets. |
| `DOPPLER_PROJECT` | Doppler project identifier. |
| `DOPPLER_CONFIG` | Doppler config name. |
| `DOPPLER_API_HOST` | Optional API host. Defaults to `https://api.doppler.com`. |
| `DOPPLER_SECRETS_CACHE_TTL_SECONDS` | Optional in-memory cache TTL. Defaults to `10`. Set to `0` to refresh on every lookup. |

You can override the project or config for an individual reference:

```jinja theme={"system"}
{{ doppler_secret_var('POSTGRES_PASSWORD', project='analytics', config='prd') }}
```

When a Doppler secret changes, Mage picks up the new value on the next template render after the cache TTL expires. Already-open database connections or clients may continue using the value they were created with until they are recreated.


# Secrets
Source: https://docs.mage.ai/development/variables/secrets

Storing and using secrets through Mage

In various surfaces in Mage, you may be asked to input config for certain integrations
such as cloud databases or services. In these cases, you may need to input a password
or an api key, but you don't want it to be shown in plain text. To get around this issue,
we created a way to store your secrets in the Mage database.

> [!WARNING]
>
> Your secrets are encrypted before being stored into the Mage database. The encryption key
> will be stored in the Mage data folder which by default will be created at `~/.mage_data`. If
> you need more secure encryption, we recommend using a secrets manager.

## Secret management

### Creating secrets

To manage your secrets in Mage OSS, select **Secrets** in the left navigation. In
Mage Pro, open **Library** and select **Secrets**. This opens the global secrets
management page where you can see all your secrets, create new ones, and delete
existing ones.

You can also manage secrets directly from the pipeline edit page. In the sidekick,
you should see a `Secrets` tab. To create a secret, press the `New` button on this tab.
Input a name for the secret and the value of the secret, and press Enter or Return to save.

![Create Secrets](https://mage-ai.github.io/assets/create-secrets.png)

The secrets can be shared across the project that they are created in.

Pipeline level secrets coming soon...

### Secrets in different environments

Secrets are stored to the Mage database. If you are using the same database across multiple
Mage environments, i.e. development and production, the secrets will not be shared across the environments
unless you shared the same Mage data directory for every environment.

Each secret will be saved to the database with a name and a uuid. The name will be used to identify
the secret, and the uuid is used to identify the environment.

Example:

```
Development environment
project
└───mage_data
│   └───secrets
│   │   │   key: abc123
│   │   │   uuid: dev-uuid

Production environment
project
└───mage_data
│   └───secrets
│   │   │   key: def456
│   │   │   uuid: prod-uuid
```

**Secrets table in database**

| Name | Uuid | Value |
| - | - | - |
| `secret1` | `dev-uuid` | sooper dooper secret string |
| `secret2` | `dev-uuid` | xXxSecreTxXx |
| `secret3` | `dev-uuid` | dev only secret |
| `secret1` | `prod-uuid` | real production secret |
| `secret2` | `prod-uuid` | m4g3r0x! |
| `secret1` | `random` | random secret |

| Command | Development Value | Production Value |
| - | - | - |
| `get_secret_value('secret1')` | sooper dooper secret string | real production secret |
| `get_secret_value('secret2')` | xXxSecreTxXx | m4g3r0x! |
| `get_secret_value('secret3')` | dev only secret | `None` |

Coming soon:

* secrets scoped to a pipeline

## Using Secrets

You can use the following syntax to have Mage interpolate the secret when reading from the config:

```yaml theme={"system"}
field_name: "{{ mage_secret_var('your_secret_name') }}"
```

![Using Secrets](https://mage-ai.github.io/assets/using-secrets.png)

You can also fetch the secret value in a Mage code block by importing a helper method:

```python theme={"system"}
from mage_ai.data_preparation.shared.secrets import get_secret_value

get_secret_value('your_secret_name')
```


# Create agent
Source: https://docs.mage.ai/extensibility/api-reference/agents/create-agent

Create an agent YAML file and backing pipeline.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

```http theme={"system"}
POST /api/agents
```

```json theme={"system"}
{
  "agent": {
    "name": "Customer support agent",
    "description": "Routes support questions.",
    "instructions": "Classify the request.",
    "actions": []
  }
}
```

Creating an agent creates one backing pipeline named after the agent and one YAML block file under `agents/`. If `uuid` is omitted, Mage derives it from the cleaned agent name. If that UUID already exists, the request is rejected.


# Delete agent
Source: https://docs.mage.ai/extensibility/api-reference/agents/delete-agent

Delete an agent YAML file.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

```http theme={"system"}
DELETE /api/agents/{agent_uuid}
```

Deletes the YAML file under `agents/`. The backing pipeline is not deleted by this endpoint.


# Agents API
Source: https://docs.mage.ai/extensibility/api-reference/agents/overview

Manage YAML-backed Mage Pro agents.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

The Agents API manages AI agent YAML files stored under the project `agents/` folder. Each API-created agent has one backing pipeline and one `type: agent` YAML block file.

Agents use `instructions` for saved default instructions. Runtime calls use `message`.

## Resource Shape

```json theme={"system"}
{
  "uuid": "customer_support_agent_a1b2",
  "name": "Customer support agent",
  "description": "Routes support questions.",
  "instructions": "Classify the request.",
  "pipeline_uuid": "customer_support_agent",
  "actions": [
    {
      "pipeline_uuid": "support_workflows",
      "block_uuid": "triage_ticket"
    }
  ],
  "status": "draft",
  "endpoint": {}
}
```

List responses include only valid agent YAML files with `pipeline_uuid`.


# Publish agent
Source: https://docs.mage.ai/extensibility/api-reference/agents/publish-agent

Mark an agent live by upserting an API trigger.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

```http theme={"system"}
POST /api/agents/{agent_uuid}/publish
```

Publishing upserts an API trigger for the backing pipeline, sets `status` to `published`, and writes endpoint metadata to the agent YAML file.

```json theme={"system"}
{
  "agent_publish": {
    "uuid": "customer_support_agent_a1b2",
    "status": "published",
    "endpoint": {
      "enabled": true,
      "pipeline_schedule_id": 42,
      "streaming_run_path": "/api/agents/customer_support_agent_a1b2/runs",
      "token_present": true
    }
  }
}
```


# Read agents
Source: https://docs.mage.ai/extensibility/api-reference/agents/read-agents

List and fetch YAML-backed agents.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

## List Agents

```http theme={"system"}
GET /api/agents
```

Optional query parameters:

* `_limit`: Maximum number of agents to return.
* `_offset`: Number of agents to skip.

```json theme={"system"}
{
  "agents": [],
  "metadata": {
    "count": 0,
    "results": 0,
    "has_more": false
  }
}
```

## Get Agent

```http theme={"system"}
GET /api/agents/{agent_uuid}
```

Returns one `agent` object.


# Run agent
Source: https://docs.mage.ai/extensibility/api-reference/agents/run-agent

Trigger an agent backing pipeline and receive stream metadata.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

## Stream a published agent

The agent's **Endpoint** page shows the streaming POST URL after an API trigger is available. Edit the JSON request body on that page and select **Run agent** to submit it to this endpoint and follow the response stream in the UI:

```http theme={"system"}
POST /api/triggers/{pipeline_schedule_id}/streams
Authorization: Bearer <trigger_token>
Content-Type: application/json
```

```json theme={"system"}
{
  "pipeline_run": {
    "variables": {
      "message": "Summarize yesterday's support tickets."
    }
  }
}
```

Use the active API trigger's schedule ID and token. This request runs the backing pipeline's last agent block and returns server-sent events (SSE), including run metadata and agent output, on the same connection. Include your deployment's URL prefix before `/api` when applicable.

## Start a run with a Mage session

```http theme={"system"}
POST /api/agents/{agent_uuid}/runs
```

```json theme={"system"}
{
  "agent_run": {
    "message": "Summarize yesterday's support tickets."
  }
}
```

The response includes the pipeline run ID, schedule ID, and stream URL.

```json theme={"system"}
{
  "agent_run": {
    "agent_uuid": "customer_support_agent_a1b2",
    "pipeline_run_id": 7,
    "pipeline_schedule_id": 42,
    "run_id": 7,
    "stream_url": "/api/triggers/42/streams/7"
  }
}
```


# Update agent
Source: https://docs.mage.ai/extensibility/api-reference/agents/update-agent

Update the YAML source of truth for an agent.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=agents-api)

```http theme={"system"}
PUT /api/agents/{agent_uuid}
```

```json theme={"system"}
{
  "agent": {
    "name": "Customer support agent",
    "description": "Routes urgent support questions.",
    "instructions": "Classify the request and recommend escalation.",
    "actions": [
      {
        "pipeline_uuid": "support_workflows",
        "block_uuid": "triage_ticket"
      }
    ]
  }
}
```

Updates write directly to the agent YAML file. Agent versions and rollback records are not created.


# Create backfill
Source: https://docs.mage.ai/extensibility/api-reference/backfills/create-backfills

POST /api/pipelines/:pipeline_uuid/backfills

`POST /api/pipelines/:pipeline_uuid/backfills`

**pipeline\_uuid** `string` *required*

Pipeline UUID to create the backfill for.

**Body**

**backfill** `object` *required*

**payload**

**name** `string` *required*

```curl cURL theme={"system"}
curl --request POST \
  --url 'http://localhost:6789/api/pipelines/example_pipeline/backfills' \
  --header 'OAUTH-TOKEN=some_really_long_string' \
  --header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --data '{
    "backfill": {
      "name": "fire wind"
    }
  }'
```

```json Response theme={"system"}
{
  "backfill": {
    "id": 10,
    "block_uuid": null,
    "completed_at": null,
    "created_at": "2023-04-01 01:02:18.411997+00:00",
    "end_datetime": null,
    "failed_at": null,
    "interval_type": null,
    "interval_units": null,
    "metrics": null,
    "name": "fire wind",
    "pipeline_schedule_id": null,
    "pipeline_uuid": "example_pipeline",
    "start_datetime": null,
    "started_at": null,
    "status": null,
    "updated_at": "2023-04-01 01:02:18.411997+00:00",
    "variables": null
  }
}
```


# Delete backfill
Source: https://docs.mage.ai/extensibility/api-reference/backfills/delete-backfill

DELETE /api/backfills/:backfill_id

`DELETE /api/backfills/:backfill_id`

```curl cURL theme={"system"}
curl -X DELETE 
--url http://localhost:6789/api/backfills/1
-H 'Content-Type: application/json' 
-H 'Cookie: oauth_token=some_really_long_string' 
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' 
```

```python Python theme={"system"}
headers = {
"Content-Type": "application/json",
"X-API-KEY": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
"Cookie": "oauth_token=some_really_long_string"
}

r = requests.delete('http://localhost:6789/api/backfills/1', headers=headers)
```

```json Response theme={"system"}
{
    "backfill": {
        "id": 1,
        "block_uuid": null,
        "completed_at": null,
        "created_at": "2023-07-17 21:48:42",
        "end_datetime": null,
        "failed_at": null,
        "interval_type": null,
        "interval_units": null,
        "metrics": null,
        "name": "arwen_starlight_backfill",
        "pipeline_schedule_id": null,
        "pipeline_uuid": "arwen_starlight",
        "start_datetime": null,
        "started_at": null,
        "status": null,
        "updated_at": "2023-07-17 21:48:42",
        "variables": null
    }
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/backfills/overview

Programmatically create multiple pipeline runs to backfill a pipeline.

### Backfill object

```json Response theme={"system"}
{
  "block_uuid": null,
  "completed_at": null,
  "created_at": "2023-03-23 18:27:58.848959+00:00",
  "end_datetime": "2023-03-05 00:00:00+00:00",
  "failed_at": null,
  "id": 9,
  "interval_type": "hour",
  "interval_units": 4,
  "metrics": null,
  "name": "sparkling glitter",
  "pipeline_schedule_id": null,
  "pipeline_uuid": "example_pipeline",
  "start_datetime": "2023-03-01 00:00:00+00:00",
  "started_at": null,
  "status": null,
  "total_run_count": 25,
  "updated_at": "2023-03-23 18:28:22.379725+00:00",
  "variables": null
}
```

**block\_uuid** `string`

If the backfill uses custom code to generate the pipeline runs, this block UUID contains the code
that will be used.

**completed\_at** `datetime`

Date and time the backfill finished running successfully.

**created\_at** `datetime` *required*

Date and time the record was created.

**end\_datetime** `datetime`

The date and time the backfill should stop on.

**failed\_at** `datetime`

Date and time the backfill failed.

**id** `integer` *required*

Unique ID for the record.

**interval\_type** `string`

`minute`, `hour`, `day`, `week`, `month`, `year`

**interval\_units** `integer`

The number of `interval_type` to use in between each pipeline run that the backfill creates.

For example, if `interval_type` is `day` and `interval_units` is 3, then the backfill will create
pipeline runs spanning 3 day increments between the `start_datetime` and `end_datetime`.

**metrics** `object`

Various metrics related to the pipeline run; e.g. number of rows processed in a
data integration pipeline sync.

**name** `string` *required*

Name of the record.

**pipeline\_scheduled\_id** `integer`

Pipeline schedule that this record belongs to.

**pipeline\_uuid** `string` *required*

Pipeline that this record belongs to.

**start\_datetime** `datetime`

The date and time the backfill should start on.

**started\_at** `datetime`

Date and time the backfill started running.

**status** `string` *required*

`initial`, `running`, `completed`, `failed`, `cancelled`

**total\_run\_count** `integer`

Number of pipeline runs that will be created for this backfill.

**updated\_at** `datetime` *required*

Date and time the record was last updated.

**variables** `object`

Object containing variables for the pipeline that is being overwritten by this backfill.

**properties**

**\[key]** `string`

The property name is user defined.


# Read backfills
Source: https://docs.mage.ai/extensibility/api-reference/backfills/read-backfills

GET /api/backfills

`GET /api/backfills`

**\_limit** `integer`

Maximum number of logs to be returned in the API response.

Example: `20`

**\_offset** `integer`

Read logs after N number of logs, where N equals `_offset`.

Example: `10`

**pipeline\_uuid** `string`

The pipeline uuid for which to read backfills.

**include\_run\_count** `boolean`

A boolean which indicates whether to return the run count in the response metadata object.

```curl cURL theme={"system"}
curl --request GET \
  --url 'http://localhost:6789/api/backfills?_limit=20&_offset=0&include_run_count=true&pipeline_uuid=example_pipeline&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'OAUTH-TOKEN=some_really_long_string' \
```

```json Response theme={"system"}
{
  "backfills": [
    {
      "block_uuid": null,
      "completed_at": null,
      "created_at": "2023-03-23 18:27:58.848959+00:00",
      "end_datetime": "2023-03-05 00:00:00+00:00",
      "failed_at": null,
      "id": 9,
      "interval_type": "hour",
      "interval_units": 4,
      "metrics": null,
      "name": "sparkling glitter",
      "pipeline_schedule_id": null,
      "pipeline_uuid": "example_pipeline",
      "start_datetime": "2023-03-01 00:00:00+00:00",
      "started_at": null,
      "status": null,
      "total_run_count": 25,
      "updated_at": "2023-03-23 18:28:22.379725+00:00",
      "variables": null
    }
  ],
  "metadata": {
    "count": 0,
    "next": false
  }
}
```


# Update backfill
Source: https://docs.mage.ai/extensibility/api-reference/backfills/update-backfill

PUT /api/pipelines/:pipeline_uuid/backfills/:backfill_id

`PUT /api/pipelines/:pipeline_uuid/backfills/:backfill_id`

All default attributes for a backfill may be updated in addittion to the following:

**start\_datetime** `string`

A datetime formatted string representing the start time of the backfill, e.g. `"2023-09-05 23:56:05"`.

**end\_datetime**

A datetime formatted string representing the end time of the backfill, e.g. `"2023-09-05 23:56:05"`.

**interval\_type** `string`

The interval type for the backfill. One of: `minute`, `hour`, `day`, `week`, `month`, or `year`.

**interval\_units** `integer`

The number of interval units for the backfill.

**name** `string`

A new name for the backfill.

**status** `status`

The backfill status. One of `initial`, `running`, `completed`, `failed`, `cancelled`.

**variables** `object`

An object containing Global Pipeline Variables to be overridden by the backfill. Any Global Variable may be overridden.

![Global Pipeline Vars](https://mage-ai.github.io/assets/pipeline-variables.png)

```curl cURL theme={"system"}
curl -X PUT
--url http://localhost:6789/api/pipelines/arwen_starlight/backfills/6
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHBpcmVzIjoxNjkyMzE0ODM3LjE1MTkzOSwidG9rZW4iOiJieHlQajlVcHE5N2phTC1nTGtOQTVnM2draWZwRjJwSEZZZkNzLTJ2NmpnIn0.bWqUP7Xg9PmHBW703AGPOE270Cc73F4kb0IKHLppGbM'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' 
-d '{
  "backfill": {
    "start_datetime": "2023-07-17 23:56:05",
    "end_datetime": "2023-07-18 23:56:05",
    "name": "new_name"
  }
}' 

```

```python Python theme={"system"}
import requests

headers = {
  "Content-Type": "application/json",
  "X-API-KEY": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
  "Cookie": "oauth_token=some_really_long_string"
}  
json_data = {
  'backfill': {
          "start_datetime": '2023-07-17 23:56:05',
          "end_datetime": '2023-07-18 23:56:05',
          'name':'new_name'
          }
}

r = r.put('http://localhost:6789/api/pipelines/arwen_starlight/backfills/6', json=json_data, headers=headers)
```

```json Response theme={"system"}
{
  "backfill": {
    "id": 6,
    "block_uuid": null,
    "completed_at": null,
    "created_at": "2023-07-18 23:30:26",
    "end_datetime": "2023-07-18 23:56:05",
    "failed_at": null,
    "interval_type": null,
    "interval_units": null,
    "metrics": null,
    "name": "new_name",
    "pipeline_schedule_id": null,
    "pipeline_uuid": "arwen_starlight",
    "start_datetime": "2023-07-17 23:56:05",
    "started_at": null,
    "status": null,
    "updated_at": "2023-07-18 23:31:57",
    "variables": null
  }
}
```


# Create block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/create-block

POST /api/pipelines/:pipeline_uuid/blocks

`POST /api/pipelines/:pipeline_uuid/blocks`

Create a new block in one or more pipelines.

**name** `string` *required*

Human readable block name. This is used to generate the UUID, e.g. `restless dust`. Names will be slugified to a safe-uuid.

**type** `string` *required*

Type of block:
`callback`, `chart`, `conditional`, `custom`, `data_exporter`, `data_loader`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`, `sensor`, `transformer`

**color** `string`

Block color displayed by the frontend. If omitted or not recognized, the frontend uses the block type's default color.

**config** `object`

An optional parameter specifying a block configuration file For example, `{data_source: "s3"}` will create an s3 data loader.

**configuration** `object`

Miscellaneous configuration settings for the block.

**properties**

**data\_provider** `string`

Database or data warehouse for the SQL block to connect to.

**data\_provider\_database** `string`

Database name to use when saving the output of the SQL block.

**data\_provider\_profile** `string`

Profile target for the dbt block.

**data\_provider\_schema** `string`

Schema name to use when saving the output of the SQL block.

**data\_provider\_table** `string`

Table name to use when saving the output of the SQL block.

**export\_write\_policy** `string`

Whether to `replace` the existing table of the SQL block output, `append`, or
raise an error and `fail`.

**use\_raw\_sql** `string`

Toggle writing raw SQL in the block. Read more [here](/guides/blocks/sql-blocks#using-raw-sql).

**content** `string`

The content of block formatted as a string. Use `/n` for newlines. Omitting will create an empty block.

**extension\_uuid** `string`

UUID of extensions, currently [Great Expectations](/development/testing/great-expectations) is the only extension supported.

**language** `string`

The block language:
`python`, `sql`, `r`, `yaml`, `markdown`

**pipelines** `[string]`

Array of strings containing the pipeline uuids for which the block is a member.

**priority** `integer`

The 0-indexed position of the block in the pipeline, e.g. 0 is the first block location.

**upstream\_blocks** `[string]`

An array of strings representing the uuids of upstream blocks (the blocks on which the new block depends). For example, `["a", "b"]` will create a block that depends on blocks `a` and `b`.

**uuid** `string`

The unique id of the block. This is generated from `name`, so it's not necessary to populate this field.

```bash Request theme={"system"}
curl 'http://localhost:6789/api/pipelines/broken_wave/blocks?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Authorization: Bearer 018Cc8ydDUa6YWoLad2oMdw0EKdWWJuyg85D02GN1ww' \
-H 'Content-Type: application/json' \
--data-raw '
  {
  "block": {
      "name": "silent fire",
      "priority": 2,
      "configuration": {
          "data_source": null
      },
      "language": "python",
      "type": "data_exporter"
  },
  "api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2"
}
' \
--compressed
```

```json Response theme={"system"}
{
    "block": {
        "content": null,
        "name": "silent fire",
        "priority": 2,
        "config": {
            "data_source": null
        },
        "language": "python",
        "type": "data_loader",
        "configuration": {},
        "upstream_blocks": []
    },
}
```


# Delete block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/delete-block

DELETE /api/pipelines/:pipeline_uuid/blocks/:block_uuid

`DELETE /api/pipelines/:pipeline_uuid/blocks/:block_uuid`

Delete a block from a pipeline. By default, this removes the block from the pipeline and keeps the block file in the project.

By default, blocks with downstream dependencies may not be deleted. To force delete blocks, set the `force` query parameter to the boolean `True`.

To also delete the underlying block file, set the `delete_file` query parameter to the boolean `True`. File deletion is only allowed when the block is not a replica and no other pipeline uses the same block file.

The following example deletes a fictional block `winter_wildflower` from the `broken_wave` pipeline.

```bash Request theme={"system"}
curl 'http://localhost:6789/api/pipelines/broken_wave/blocks/winter_wildflower?block_type=data_loader&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2&force=True' \
  -X 'DELETE'
```

The following example removes the block from the pipeline and deletes its block file.

```bash Request theme={"system"}
curl 'http://localhost:6789/api/pipelines/broken_wave/blocks/winter_wildflower?block_type=data_loader&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2&force=True&delete_file=True' \
  -X 'DELETE'
```

```json Response theme={"system"}
{
    "block": {
        "all_upstream_blocks_executed": true,
        "color": null,
        "configuration": {},
        "downstream_blocks": [],
        "executor_config": null,
        "executor_type": "local_python",
        "has_callback": false,
        "name": "winter wildflower",
        "language": "python",
        "retry_config": null,
        "status": "not_executed",
        "type": "data_loader",
        "upstream_blocks": [],
        "uuid": "winter_wildflower"
    }
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/blocks/overview

The Blocks API is used to create, update, and delete blocks objects from a pipeline.

### Block Object

**all\_upstream\_blocks\_executed** `boolean`

Whether or not all upstream blocks have been successfully executed.

**color** `string`

The color of the block in the notebook UI.

**configuration** `object`

Miscellaneous configuration settings for the block.

**properties**

**data\_provider** `string`

Database or data warehouse for the SQL block to connect to.

**data\_provider\_database** `string`

Database name to use when saving the output of the SQL block.

**data\_provider\_profile** `string`

Profile target for the dbt block.

**data\_provider\_schema** `string`

Schema name to use when saving the output of the SQL block.

**data\_provider\_table** `string`

Table name to use when saving the output of the SQL block.

**export\_write\_policy** `string`

Whether to `replace` the existing table of the SQL block output, `append`, or
raise an error and `fail`.

**use\_raw\_sql** `string`

Toggle writing raw SQL in the block. Read more [here](/guides/blocks/sql-blocks#using-raw-sql).

**content** `string`

Block file contents.

**downstream\_blocks** `array of strings`

The block UUIDs that depend on this block.

**executor\_config** `object`

Used when configuring a Kubernetes executor. See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details

**executor\_type** `string`

The type of executor to use for the block:
`ecs`, `gcp_cloud_run`, `azure_container_instance`, `k8s`, `local_python`, `pyspark`

See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details.

**has\_callback** `boolean`

**language** `string`

`python`, `r`, `sql`, `yaml`, `markdown`

**metadata** `object`

**name** `string` *required*

Human readable name of block.

**status** `string`

Status of block: `executed`, `failed`, `not_executed`, `updated`

**type** `string` *required*

Type of block:
`callback`, `chart`, `conditional`, `custom`, `data_exporter`, `data_loader`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`, `sensor`, `transformer`

**upstream\_blocks** `string`

The block UUIDs that this block depends on.

**uuid** `string` *required*

Unique identifier for the block.

```json Response theme={"system"}
{
    "block": {
        "all_upstream_blocks_executed": true,
        "color": null,
        "configuration": {},
        "content": "some_long_block_content_string",
        "downstream_blocks": [],
        "executor_config": null,
        "executor_type": "local_python",
        "has_callback": false,
        "name": "solitary silence",
        "language": "python",
        "retry_config": null,
        "status": "not_executed",
        "type": "data_loader",
        "upstream_blocks": [],
        "uuid": "solitary_silence",
        "content": "some_long_block_content_string",
        "outputs": []
    }
}
```


# Read block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/read-block

GET /api/pipelines/:pipeline_uuid/blocks/:block_uuid

`GET /api/pipelines/:pipeline_uuid/blocks/:block_uuid`

Fetch and return the contents and metadata of a block.

```bash Request theme={"system"}
curl --location 'http://localhost:6789/api/pipelines/broken_wave/blocks/solitary_silence' \
--header 'Cookie: oauth_token=long_token_string' \
--header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
```

```json Response theme={"system"}
{
    "block": {
        "all_upstream_blocks_executed": true,
        "color": null,
        "configuration": {},
        "downstream_blocks": [],
        "executor_config": null,
        "executor_type": "local_python",
        "has_callback": false,
        "name": "solitary silence",
        "language": "python",
        "retry_config": null,
        "status": "not_executed",
        "type": "data_loader",
        "upstream_blocks": [],
        "uuid": "solitary_silence",
        "content": "some_long_block_content_string",
        "outputs": []
    }
}
```


# Update block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/update-block

PUT /api/pipelines/:pipeline_uuid/blocks/:block_uuid

`PUT /api/pipelines/:pipeline_uuid/blocks/:block_uuid`

**name** `string` *required*

Human readable block name. This is used to generate the UUID, e.g. `restless dust`. Names will be slugified to a safe-uuid.

**type** `string` *required*

Type of block:
`callback`, `chart`, `conditional`, `custom`, `data_exporter`, `data_loader`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`, `sensor`, `transformer`

**bookmark\_values** `[string]`

Data integration bookmark values as a list of strings.

**callback\_blocks** `[string]`

Array of [callback](/development/blocks/callbacks) block objects. Same shape as `blocks`.

**color** `string`

Block color displayed by the frontend. If omitted or not recognized, the frontend uses the block type's default color.

**configuration** `object`

Miscellaneous configuration settings for the block.

**properties**

**data\_provider** `string`

Database or data warehouse for the SQL block to connect to.

**data\_provider\_database** `string`

Database name to use when saving the output of the SQL block.

**data\_provider\_profile** `string`

Profile target for the dbt block.

**data\_provider\_schema** `string`

Schema name to use when saving the output of the SQL block.

**data\_provider\_table** `string`

Table name to use when saving the output of the SQL block.

**export\_write\_policy** `string`

Whether to `replace` the existing table of the SQL block output, `append`, or
raise an error and `fail`.

**use\_raw\_sql** `string`

Toggle writing raw SQL in the block. Read more [here](/guides/blocks/sql-blocks#using-raw-sql).

**content** `string`

The content of block formatted as a string. Use `/n` for newlines. Omitting will create an empty block.

**destination\_table** `string`

The destination table for a data integration pipeline.

**downstream\_blocks** `[string]`

Array of blocks downstream from the created block.

**executor\_config** `object`

Used when configuring a Kubernetes executor. See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details

**executor\_type** `string`

Used when configuring a Kubernetes executor. See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details

**extension\_uuid** `string`

UUID of extensions, currently [Great Expectations](/development/testing/great-expectations) is the only extension supported.

**language** `string`

The block language:
`python`, `sql`, `r`, `yaml`, `markdown`

**pipelines** `[string]`

Array of strings containing the pipeline uuids for which the block is a member.

**retry\_config** `object`

The configuration for block retry logic. See the [pipeline runs](/orchestration/pipeline-runs/retrying-block-runs) docs for retry config examples.

**tap\_stream\_id** `string`

The block tap stream ID for data integration pipelines.

**upstream\_blocks** `[string]`

An array of strings representing the uuids of upstream blocks (the blocks on which the new block depends). For example, `["a", "b"]` will create a block that depends on blocks `a` and `b`.

**uuid** `string`

The unique id of the block. This is generated from `name`, so it's not necessary to populate this field.

```bash Request theme={"system"}
curl --location --request PUT 'http://localhost:6789/api/pipelines/broken_wave/blocks/solitary_silence' \
--header 'Cookie: oauth_token=long_token_string' \
--header 'Content-Type: text/plain' \
--header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--data '{
 "block": {
   "color": "teal"
 }
}'
```

```json Response theme={"system"}
{
    "block": {
        "all_upstream_blocks_executed": true,
        "color": "teal",
        "configuration": {},
        "downstream_blocks": [],
        "executor_config": null,
        "executor_type": "local_python",
        "has_callback": false,
        "name": "solitary silence",
        "language": "python",
        "retry_config": null,
        "status": "not_executed",
        "type": "data_loader",
        "upstream_blocks": [],
        "uuid": "solitary_silence",
        "content": "some_long_content_string"
    }
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/logs/overview

Fetch log entries for your Mage instance.

### Log object

```json Response theme={"system"}
{
  "block_run_logs": [
    {
      "name": "export_titanic_clean.log",
      "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/export_titanic_clean.log",
      "content": ""
    }
  ],
  "pipeline_run_logs": [
    {
      "name": "pipeline.log",
      "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/55/20230308T045408/pipeline.log",
      "content": "2023-03-08T04:54:08 {\"cpu\": 0.21, \"cpu_total\": 8, \"cpu_usage\": 0.02625, \"memory\": 4013, \"memory_total\": 17010, \"memory_usage\": 0.23592004703115815, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251248.712125, \"uuid\": \"161125d3c3944d109ce945fa31f8c8a2\"}\n2023-03-08T04:54:08 {\"block_run_id\": 648, \"block_uuid\": \"load_titanic\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 648: pipeline example_pipeline block load_titanic\", \"timestamp\": 1678251248.796509, \"uuid\": \"2223dba30d2d4217828ce031f5aa0f05\"}\n2023-03-08T04:54:09 {\"block_run_id\": 648, \"block_uuid\": \"load_titanic\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 648 (block_uuid: load_titanic) completes.\", \"timestamp\": 1678251249.079394, \"uuid\": \"8c2a6c5ce8d64bbf96b51de463b0e91a\"}\n2023-03-08T04:54:18 {\"cpu\": 0.21, \"cpu_total\": 8, \"cpu_usage\": 0.02625, \"memory\": 4016, \"memory_total\": 17010, \"memory_usage\": 0.23609641387419164, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251258.908166, \"uuid\": \"fd079d501f33478985bf260b92d567f3\"}\n2023-03-08T04:54:18 {\"block_run_id\": 649, \"block_uuid\": \"fill_in_missing_values\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 649: pipeline example_pipeline block fill_in_missing_values\", \"timestamp\": 1678251258.997002, \"uuid\": \"8c4c4d69560a4083a4ff0bf30722e3fa\"}\n2023-03-08T04:54:19 {\"block_run_id\": 649, \"block_uuid\": \"fill_in_missing_values\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 649 (block_uuid: fill_in_missing_values) completes.\", \"timestamp\": 1678251259.224986, \"uuid\": \"44a35d24ae874e77af98cc1e6ab86278\"}\n2023-03-08T04:54:29 {\"cpu\": 0.2, \"cpu_total\": 8, \"cpu_usage\": 0.025, \"memory\": 4016, \"memory_total\": 17010, \"memory_usage\": 0.23609641387419164, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251269.186595, \"uuid\": \"97affe68017a4c65a69fad3492b80ea5\"}\n2023-03-08T04:54:29 {\"block_run_id\": 650, \"block_uuid\": \"export_titanic_clean\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 650: pipeline example_pipeline block export_titanic_clean\", \"timestamp\": 1678251269.292511, \"uuid\": \"97aa24db4b6e4da1b6f98222e1d0cd06\"}\n2023-03-08T04:54:29 {\"block_run_id\": 650, \"block_uuid\": \"export_titanic_clean\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 650 (block_uuid: export_titanic_clean) completes.\", \"timestamp\": 1678251269.383662, \"uuid\": \"d06e3d9dcb2448a583ae323e25e04045\"}\n2023-03-08T04:54:30 {\"block_run_id\": 651, \"block_uuid\": \"falling_glitter\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 651: pipeline example_pipeline block falling_glitter\", \"timestamp\": 1678251270.335112, \"uuid\": \"70b81921cb374e0d9c712ce6379f2148\"}\n2023-03-08T04:54:30 {\"block_run_id\": 651, \"block_uuid\": \"falling_glitter\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 651 (block_uuid: falling_glitter) completes.\", \"timestamp\": 1678251270.571607, \"uuid\": \"f2220d588ca746f49b5fafca1113be16\"}\n2023-03-08T04:54:38 {\"cpu\": 0.2, \"cpu_total\": 8, \"cpu_usage\": 0.025, \"memory\": 4019, \"memory_total\": 17010, \"memory_usage\": 0.23627278071722516, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251278.430712, \"uuid\": \"3de507ebca894100b20ae4a1645724d7\"}\n"
    }
  ],
  "total_block_run_log_count": 1,
  "total_pipeline_run_log_count": 1
}
```

**block\_run\_logs** `array of objects` *required*

Logs for block runs.

**properties**

**name** `string` *required*

File name of log file.

**path** `string` *required*

Full path of log file.

**content** `string` *required*

Contents of log file.

**pipeline\_run\_logs** `array of objects` *required*

Logs for pipeline runs.

**properties**

**name** `string` *required*

File name of log file.

**path** `string` *required*

Full path of log file.

**content** `string` *required*

Contents of log file.

**total\_block\_run\_log\_count** `integer` *required*

Number of block run logs in the API response.

**total\_pipeline\_run\_log\_count** `integer` *required*

Number of pipeline run logs in the API response.


# Read logs
Source: https://docs.mage.ai/extensibility/api-reference/logs/read-logs

GET /api/pipelines/:pipeline_uuid/logs

`GET /api/pipelines/:pipeline_uuid/logs`

**pipeline\_uuid** `string` *required*

Pipeline UUID of the logs.

Example: `example_pipeline`

**\_limit** `integer`

Maximum number of logs to be returned in the API response.

Example: `20`

**\_offset** `integer`

Read logs after N number of logs, where N equals `_offset`.

Example: `10`

**start\_timestamp** `integer`

Read logs on or after a specified timestamp.

Example: `1677648116`

**end\_timestamp** `integer`

Read logs on or before a specified timestamp.

Example: `1678252916`

**level\[]** `array of strings`

Read logs matching one or more log levels.

Example: `INFO`

**source\_category\[]** `array of strings`

Read Mage Pro OpenSearch-backed logs from one or more source categories.
Accepted values are `user_code`, `mage_scheduler`, `executor`,
`connector_library`, `kubernetes`, `system`, and `unknown`.

Example: `user_code`

**exclude\_source\_category\[]** `array of strings`

Exclude Mage Pro OpenSearch-backed logs from one or more source categories.
To hide infrastructure logs, pass `mage_scheduler`, `executor`, `kubernetes`,
and `system`.

Example: `kubernetes`

Source category filters require Mage Pro OpenSearch-backed log search. When the
API falls back to disk logs, responses with active source filters include
`fallback_from_disk: true`, `source_filters_applied: false`, and
`source_filter_fallback_reason: "opensearch_unavailable"`. Legacy OpenSearch
documents are classified from conservative structured fields when possible;
unclassified entries remain `unknown` and are included in unfiltered results.
Unsupported source category values return no OpenSearch-backed rows for that
request instead of being ignored.

**hide\_health\_check** `integer`

Pass `1` to hide scheduler health-check log entries.

Example: `1`

```curl cURL theme={"system"}
curl --request GET \
  --url 'http://localhost:6789/api/pipelines/example_pipeline/logs?_limit=20&_offset=0&start_timestamp=1677648116&exclude_source_category[]=kubernetes&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'OAUTH-TOKEN=some_really_long_string' \
```

```json Response theme={"system"}
{
  "logs": [
    {
      "block_run_logs": [
        {
          "name": "solitary_brook.log",
          "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/solitary_brook.log",
          "content": "very_long_content"
        },
        {
          "name": "load_titanic.log",
          "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/load_titanic.log",
          "content": "very_long_content"
        }
      ],
      "pipeline_run_logs": [
        {
          "name": "pipeline.log",
          "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/55/20230308T045408/pipeline.log",
          "content": "very_long_content \"timestamp\": 1678251278.430712, \"uuid\": \"3de507ebca894100b20ae4a1645724d7\"}\n"
        },
        {
          "name": "pipeline.log",
          "path": "very_long_content"
        }
      ],
      "fallback_from_disk": true,
      "source_filter_fallback_reason": "opensearch_unavailable",
      "source_filters_applied": false,
      "total_block_run_log_count": 2,
      "total_pipeline_run_log_count": 2
    }
  ],
  "metadata": {}
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/oauth-access-tokens/overview

Fetch OAuth access tokens.

### OAuth access token object

```json Response theme={"system"}
{
  "oauth_access_tokens": [
    {
      "expires": "2023-08-18 23:41:52.480598+00:00",
      "token": "JS3Umh826UoZzbuGQUsj36PSBwruVCzQYlRFf3KmJWA",
      "user_id": 2
    }
  ],
  "metadata": {
    "count": 1,
    "next": false
  }
}
```

**expires** `string` *required*

Datetime in UTC when the access token expires and is no longer valid to use.

**token** `string` *required*

The token used to make authenticated API requests to Mage.

**user\_id** `integer` *required*

ID of user that the OAuth access token belongs to.


# Read OAuth tokens
Source: https://docs.mage.ai/extensibility/api-reference/oauth-access-tokens/read-oauth-access-tokens

GET /api/oauth_access_tokens

`GET /api/oauth_access_tokens`

**\_limit** `integer`

Maximum number of logs to be returned in the API response.

Example: `20`

**\_offset** `integer`

Read logs after N number of logs, where N equals `_offset`.

Example: `10`

**show\_all** `integer`

If `show_all` isn’t present, then only show active OAuth access tokens for the current user.

If `show_all` is `1`, then show active and expired OAuth access tokens for the current user.

Example: `1`

```bash Request theme={"system"}
curl --request GET \
  --url 'http://localhost:6789/api/pipelines/oauth_access_tokens/logs?_limit=20&_offset=0&show_all=1&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'OAUTH-TOKEN=some_really_long_string' \
```

```json Response theme={"system"}
{
  "oauth_access_tokens": [
    {
      "expires": "2023-08-18 23:41:52.480598+00:00",
      "token": "JS3Umh826UoZzbuGQUsj36PSBwruVCzQYlRFf3KmJWA",
      "user_id": 2
    },
    {
      "expires": "2023-08-18 23:41:51.418942+00:00",
      "token": "bScSh2uTtnjMlfOjMqyytvbXj1HHQ-CJJt17qIzaP9g",
      "user_id": 2
    },
    {
      "expires": "2023-08-11 18:37:52.832016+00:00",
      "token": "ZBVozeqrPgnF-CoaUHEzMW1BbDJS2o66EJ6T2XOk2xE",
      "user_id": 2
    },
    {
      "expires": "2023-08-07 20:34:40.608736+00:00",
      "token": "dAq4jS5XaPrntFMIcQMNaXpKEUyi2ajAsgqAWXFw1fI",
      "user_id": 2
    },
    {
      "expires": "2023-08-04 20:28:16.966716+00:00",
      "token": "tQYy-W363hpWK7JRHur8gOFb-3BFe6Tp9afWDGtrQh4",
      "user_id": 2
    },
    {
      "expires": "2023-08-01 01:24:37.132478+00:00",
      "token": "dUSVnZc_8JirhcB1QHXqyFTIUFwwpoWLdHT4I8Fuxmc",
      "user_id": 2
    },
    {
      "expires": "2023-08-01 01:24:36.096940+00:00",
      "token": "Uv2C4YklZaKtEPFigzGblFZ8fCx33598ZvZ3JJCyDiw",
      "user_id": 2
    }
  ],
  "metadata": {
    "count": 7,
    "next": false
  }
}
```


# API overview
Source: https://docs.mage.ai/extensibility/api-reference/overview

Our API powers everything in the app— our UI makes calls to the backend, which operates the app. You can learn more about how these work and make your own calls in this section.

Want to jump-in and interact with Mage via API? Follow the [quickstart](#quickstart) below to get going! Otherwise, head to the [components](#components) section to learn about how our API is structured— this will be helpful if you're interested in adding new API endpoints or developing Mage.

You can read more about the components in the *Design* section on the sidebar to the left— it contains pages for [Resources](api-reference/resources), [Policies](api-reference/policies), and [Presenters](api-reference/presenters), which are the three components that make up an API endpoint.

## Quickstart

> **Obtain API Key**
>
> You'll need your API key to authenticate calls. You can find your API key quickly by navigating to the GUI and checking the network request log from the container.
>
> You can read more about how to do this on Chromium based browsers [here](https://developers.google.com/web/tools/chrome-devtools/network/reference#headers).
>
> Once you have your API key, we recommend storing it as a secret and using a solution like [dotenv](https://pypi.org/project/python-dotenv/) to load it into your environment. This will prevent you from exposing it as plain text.

> **Initialize Session**
>
> <Snippet />

> **Authenticate Session**
>
> To authenticate your session, we recommend using the `oauth_token` cookie. This can be done simply in Python requests with the following, where `token` is the value of the `token` key in the response payload:
>
> ```python theme={"system"}
> token = mage_session.get('token')
> s.headers['Cookie'] = f'oauth_token={token}'
> ```

> **Make API calls**
>
> With your API session configured, you can now make API calls! Here's an example of how to list all pipelines:
>
> ```python theme={"system"}
> pipelines_url = base_url + '/api/pipelines'
> r = s.get(pipelines_url)
> list_pipelines = r.json()
>
> print(list_pipelines)
> ```

## Authentication

Most API endpoints require authentication. The API request must include the following:

1. API key
2. OAuth token

### API key

The API request must include the API key in the query parameter as `api_key` or in the request payload body as `api_key`. You can find your API key quickly by navigating to the GUI and checking the request log from the container or going to the "Network" tab in your browser's developer tools.

### OAuth token

The OAuth token is generated by creating a new [session](/api-reference/sessions). Sessions are only required when `REQUIRE_USER_AUTHENTICATION` is enabled for your project— that is, when users must login to access Mage.

The token provided in the session payload is the **raw** value. There are two ways to authenticate with this token:

1. Supply the raw token in the header as:

```json theme={"system"}
{"Cookie": "oauth_token=[RAW-TOKEN-FROM-RESPONSE]"}
```

2. Supply the decoded token in the header as:

```json theme={"system"}
{"OAUTH-TOKEN": "[DECODED-TOKEN-FROM-RESPONSE]"}
```

To decode a token:

```python Python theme={"system"}
from mage_ai.authentication.oauth2 import decode_token

decoded_token = decode_token(token_from_session_response) ['token']
```

### Examples

Decoded token, API key in in URL:

```curl theme={"system"}
curl -X GET "localhost:6789/api/pipelines?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2" \
    -H "OAUTH-TOKEN: [DECODED-TOKEN-FROM-RESPONSE]"
```

Decoded token, API key in body:

```curl theme={"system"}
curl -X POST "localhost:6789/api/pipelines" \
    -H "OAUTH-TOKEN: [DECODED-TOKEN-FROM-RESPONSE]" \
    -d '{
        "api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
        "pipeline": {}
    }'
```

Raw token, API key in body:

```curl theme={"system"}
curl -X POST "localhost:6789/api/pipelines" \
    -H "Cookie: oauth_token=[RAW-TOKEN-FROM-RESPONSE]" \
    -d '{
        "api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
        "pipeline": {}
    }'
```

### Responses

*All* our responses return a `200` status in the main payload. The any error code will be stored in the `error` key. For example, the following is a response from a failed session auth:

```json theme={"system"}
{
    "error": {
        "code": 404,
        "message": "Email/username and/or password invalid.",
        "type": "record_not_found"
    },
    "status": 200
}
```

To check for an error, one might try a solution like the following:

```python Python theme={"system"}
err_code = r.json().get('error').get('code'):
```

## Components

If you're not developing our API, you can jump right in and start querying endpoints using the [quickstart](#quickstart) above. Otherwise, our API is based on three components:

1. [Resources](/api-reference/resources)
2. [Policies](/api-reference/policies)
3. [Presenters](/api-reference/presenters)

If you'd like to contribute a new endpoint, you'll need to create one of each. To update an endpoint, it's important to understand how they work together.

**Resources** are the core of the API. They define the endpoints and the logic behind them. They are the only component that is required to create a new endpoint. They are also the only component that is required to update an existing endpoint.

**Policies** determine what actions can be taken on a resource: what attributes can be read, and which attributes can be written.

**Presenters** determine what attributes of the resource are returned to the client in the response. You can have different sets of attributes be included in the response based on the action (e.g. create, detail, delete, list, update) or a custom format.

To see examples of each, take a look at the `mage-ai/mage_ai/api` directory [here](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/api).

## Testing

Create a test file in `mage_ai/tests/api/operations/test_[resource_name].py`;
replace `resource_name` with the name of your resource.

Paste in the following sample code into your file:

```python theme={"system"}
from mage_ai.orchestration.db.models.oauth import User
from mage_ai.tests.api.operations.base import BaseApiTestCase
from mage_ai.tests.factory import create_user


class SomeResourceOperationTests(BaseApiTestCase):
    model_class = User

    async def test_execute_delete(self):
        user = create_user(email=self.faker.email())

        await self.base_test_execute_delete(user.id)

    async def test_execute_detail(self):
        user = create_user()
        await self.base_test_execute_detail(user.id, dict(
            email=user.email,
            username=user.username,
        ))
```

This is only a sample test file. It’s running tests on the `UserResource`.
You’ll need to update the test logic to unit test your custom API endpoint.


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-runs/overview

Status and metrics for an individual run of a pipeline.

### Pipeline run object

```json Response theme={"system"}
{
  "backfill_id": null,
  "block_runs": [
    {
      "block_uuid": "load_titanic",
      "completed_at": "2023-02-16 06:40:49.769087+00:00",
      "created_at": "2023-02-16 06:40:43.575738+00:00",
      "id": 33,
      "metrics": null,
      "pipeline_run_id": 23,
      "started_at": "2023-02-16 06:40:48.816197+00:00",
      "status": "completed",
      "updated_at": "2023-02-16 06:40:48.873526+00:00"
    },
    {
      "block_uuid": "notify_stakeholders",
      "completed_at": "2023-02-16 06:41:21.249956+00:00",
      "created_at": "2023-02-16 06:41:19.301144+00:00",
      "id": 34,
      "is_pipeline_callback": true,
      "metrics": {
        "block_type": "pipeline_callback"
      },
      "pipeline_run_id": 23,
      "started_at": "2023-02-16 06:41:19.301144+00:00",
      "status": "completed",
      "updated_at": "2023-02-16 06:41:21.249956+00:00"
    }
  ],
  "block_runs_count": 4,
  "completed_at": "2023-02-16 06:41:19.249956+00:00",
  "created_at": "2023-02-16 06:40:38.681910+00:00",
  "event_variables": null,
  "execution_date": "2023-02-16 04:41:03.300124+00:00",
  "id": 23,
  "metrics": null,
  "passed_sla": false,
  "pipeline_schedule_id": 1,
  "pipeline_schedule_name": "late forest",
  "pipeline_schedule_token": "260385a38d62466fb1c88fa8c5ca503f",
  "pipeline_schedule_type": "time",
  "pipeline_uuid": "example_pipeline",
  "status": "completed",
  "updated_at": "2023-02-16 06:41:09.154102+00:00",
  "variables": null
}
```

**backfill\_id** `integer`

Backfill that this run belongs to.

**block\_runs** `array of objects`

The block runs for this specific pipeline run.

**properties**

**block\_uuid** `string` *required*

Block UUID this block run belongs to.

**completed\_at** `datetime`

Date and time the block run finished running successfully.

**created\_at** `datetime` *required*

Date and time the block run was created.

**id** `integer` *required*

ID of block run.

**metrics** `object`

Various metrics related to the block run; e.g. error messages. For failed Mage Pro
pipeline-level callbacks, the redacted callback error is available as `metrics.error.message`.

**is\_pipeline\_callback** `boolean`

Whether this block run represents a Mage Pro pipeline-level callback execution.

**pipeline\_run\_id** `integer` *required*

Pipeline run ID that this block run belongs to.

**started\_at** `datetime`

Date and time the block run started running.

**status** `string` *required*

`initial`, `queued`, `running`, `completed`, `failed`, `cancelled`, `upstream_failed`

**updated\_at** `datetime` *required*

Date and time the block run record was most recently updated.

**block\_runs\_count** `integer` *required*

Number of normal block runs for this pipeline run. Mage Pro pipeline-level callback executions
may also appear in `block_runs`, but they are not counted here.

**completed\_at** `datetime`

Date and time the pipeline run finished running successfully.

**created\_at** `datetime` *required*

Date and time the pipeline run was created.

**event\_variables** `object`

When pipeline run is triggered from an API request, the API request payload is saved in the
pipeline run’s `event_variables` column.

**execution\_date** `datetime`

Date and time the pipeline run started running.

**id** `integer` *required*

ID of pipeline run.

**metrics** `object`

Various metrics related to the pipeline run; e.g. number of rows processed in a
data integration pipeline sync.

**passed\_sla** `boolean`

Whether or not the pipeline run has passed the configured SLA.

**pipeline\_schedule\_id** `integer` *required*

Pipeline schedule ID that this pipeline run belongs to.

**pipeline\_schedule\_name** `string` *required*

Name of the pipeline schedule this pipeline run belongs to.

**pipeline\_schedule\_token** `string` *required*

Unique token of the pipeline schedule that this pipeline run belongs to.

**pipeline\_schedule\_type** `string` *required*

Pipeline schedule type that this pipeline run belongs to.

**pipeline\_uuid** `string` *required*

Pipeline UUID that this pipeline run belongs to.

**status** `string` *required*

`initial`, `running`, `completed`, `failed`, `cancelled`

**updated\_at** `datetime` *required*

Date and time the pipeline run record was most recently updated.

**variables** `object`

Object containing variables for the pipeline that is being overwritten by this pipeline run.

**properties**

**\[key]** `string`

The property name is user defined.


# Read pipeline runs
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-runs/read-pipeline-runs

GET /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs

`GET /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs`

**pipeline\_schedule\_id** `integer` *required*

Pipeline schedule ID that pipeline runs should all belong to.

**\_limit** `integer`

Maximum number of logs to be returned in the API response.

Example: `20`

**\_offset** `integer`

Read logs after N number of logs, where N equals `_offset`.

Example: `10`

```curl cURL theme={"system"}
curl --request GET \
  --url 'http://localhost:6789/api/pipeline_schedules/1/pipeline_runs?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'OAUTH-TOKEN=some_really_long_string' \
```

```json Response theme={"system"}
{
  "pipeline_runs": [
    ...
  ],
  "metadata": {
    "count": 22,
    "next": false
  }
}
```


# Trigger pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-runs/trigger-pipeline

POST /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs/:token

`POST /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs/:token`

Run a pipeline using an API request. For more information,
read this [guide](/orchestration/triggers/trigger-pipeline-api).

**pipeline\_schedule\_id** `integer` *required*

Pipeline schedule ID that pipeline runs should all belong to.

**token** `string` *required*

Unique token of the pipeline schedule that you want to trigger.

```curl cURL theme={"system"}
curl --request POST \
  --url 'http://localhost:6789/api/pipeline_schedules/71/pipeline_runs/4506463a01944ef492f70005996dac1a' \
  --data '{
  "env": "dev",
  "test": 1
}'
```

```json Response theme={"system"}
{
  "pipeline_run": {
    "id": 1249,
    "created_at": "2023-04-01 00:36:23.507282+00:00",
    "updated_at": "2023-04-01 00:36:39.962084+00:00",
    "pipeline_schedule_id": 71,
    "pipeline_uuid": "example_pipeline",
    "execution_date": "2023-04-01 00:36:39.904371+00:00",
    "status": "running",
    "completed_at": null,
    "variables": {},
    "passed_sla": false,
    "event_variables": {
      "env": "dev",
      "test": 1
    },
    "metrics": null,
    "backfill_id": null
  }
}
```


# Create pipeline schedule
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/create-pipeline-schedule

POST /api/pipelines/:pipeline_uuid/pipeline_schedules

`POST /api/pipelines/:pipeline_uuid/pipeline_schedules`

**pipeline\_uuid** `string` *required*

Pipeline UUID that the pipeline schedule should all belong to.

**pipeline\_schedule** `object` *required*

**payload**

**name** `string` *required*

Trigger names are trimmed before validation. The name must include at least one letter or
number, cannot exceed 255 characters, and must be unique within the pipeline.

**description** `string`

**schedule\_interval** `string`

Required when `schedule_type` is `time`.

**schedule\_type** `string`

Trigger type: `api` or `time`.

**settings** `object`

Additional settings for the trigger.

**properties**

**timeout** `integer`

Optional run timeout in seconds. When provided, the value must be a positive integer.
Use `null` or omit this field for no timeout.

**sla** `integer`

Optional SLA threshold in seconds. When provided, the value must be a positive integer.
Use `0`, `null`, or omit this field for no SLA.

**start\_time** `datetime`

Persisted schedule start timestamp. For `@once`, this value determines the single run time;
for recurring time schedules, it anchors when the schedule begins.

**status** `string`

**tags** `array of strings`

Non-empty tag names to associate with the trigger. Tag names are trimmed and duplicate names
are ignored before persistence. Each tag name cannot exceed 70 characters.

**variables** `object`

> [!NOTE]
>
> This endpoint creates supported Schedule and API triggers. Use `time` for Schedule triggers and
> `api` for API triggers. Event trigger creation is deprecated. Creating a trigger accepts the same
> configuration payload used by the trigger creation UI, including tags, settings, runtime
> variables, SLA, and status.

```bash Request theme={"system"}
  curl --request POST
  -H 'Content-Type: application/json'
  -H 'Cookie: oauth_token=some_really_long_string'
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
  -d '{
    "pipeline_schedule": {
      "name": "Example Pipeline Schedule Name",
      "schedule_type": "time",
      "schedule_interval": "@once",
      "start_time": "2023-03-06 04:53:00",
      "variables": {
        "env": "dev112222",
        "test": 11111
      },
      "sla": 1000,
      "settings": {
        "allow_blocks_to_fail": true,
        "create_initial_pipeline_run": false,
        "skip_if_previous_running": true,
        "timeout": 3600
      },
      "tags": ["core data"]
    }
  }'
  --url http://localhost:6789/api/pipelines/example_pipeline/pipeline_schedules
```

```json Response theme={"system"}
{
  "pipeline_schedule": {
    "id": 55,
    "created_at": "2023-03-08 04:52:54.268096+00:00",
    "updated_at": "2023-04-01 00:52:11.753497+00:00",
    "name": "Example Pipeline Schedule Name",
    "next_pipeline_run_date": null,
    "pipeline_uuid": "example_pipeline",
    "schedule_type": "time",
    "start_time": "2023-03-06 04:53:00+00:00",
    "schedule_interval": "@once",
    "status": "inactive",
    "variables": {
      "env": "dev112222",
      "test": 11111
    },
    "sla": 1000,
    "token": "67d62ed3e66143839f58945bb7d16387",
    "settings": {
      "allow_blocks_to_fail": true,
      "create_initial_pipeline_run": false,
      "skip_if_previous_running": true,
      "timeout": 3600
    },
    "tags": ["core data"]
  }
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/overview

Triggers for a pipeline.

### Pipeline schedule object

```json Response theme={"system"}
{
  "created_at": "2023-03-01 20:27:23.645692+00:00",
  "event_matchers": [],
  "id": 41,
  "last_pipeline_run_status": null,
  "name": "dark brook",
  "next_pipeline_run_date": null,
  "pipeline_runs_count": 0,
  "pipeline_uuid": "dry_bird",
  "schedule_interval": "@once",
  "schedule_type": "time",
  "settings": {
    "allow_blocks_to_fail": true,
    "create_initial_pipeline_run": true,
    "skip_if_previous_running": false,
    "timeout": 3600
  },
  "sla": 950400,
  "start_time": "2023-03-01 22:04:00+00:00",
  "status": "inactive",
  "tags": ["core data"],
  "token": "1dca215c01564d48bf0d3db7d5450a20",
  "updated_at": "2023-03-01 22:04:29.761238+00:00",
  "variables": {
    "env": "dev"
  }
}
```

**created\_at** `string` *required*

Date and time the trigger was created.

**event\_matchers** `array of objects`

Deprecated. Legacy event matching logic returned for existing event triggers. New trigger
creation supports `api` and `time` schedule types.

**properties**

**event\_type** `string` *required*

`aws_event`

**id** `string` *required*

Event matcher ID.

**name** `string` *required*

Name of event matcher.

**pattern** `string`

**pipeline\_schedule\_ids** `array of integers`

**id** `integer` *required*

Trigger ID.

**last\_pipeline\_run\_status** `string`

**name** `string` *required*

Name of trigger. Names are trimmed before validation, must include at least one letter or
number, cannot exceed 255 characters, and must be unique within the pipeline.

**next\_pipeline\_run\_date** `datetime`

Next scheduled execution date for the trigger, or `null` when no next run is scheduled.

**pipeline\_runs\_count** `integer`

**pipeline\_uuid** `string` *required*

Pipeline UUID the trigger belongs to.

**schedule\_interval** `string`

`@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`

**schedule\_type** `string` *required*

Trigger type. Supported new trigger creation uses `api` or `time`; `event` may appear for
deprecated legacy event triggers.

**settings** `object`

Additional settings for the trigger.

**properties**

**allow\_blocks\_to\_fail** `boolean`

If `true`, then blocks in the pipeline will continue to run even if unrelated blocks fail.

**create\_initial\_pipeline\_run** `boolean`

If `true`, then an initial pipeline run will be created immediately after enabling the
trigger if the trigger's current execution date is after its start date time but before
current time.

For example, if the current time is `1/4/24 12:00 UTC` and you create a daily trigger, the
current execution date would be `2024-01-04 00:00`. If the start date was set to
`1/3/24 12:00 UTC`, which is before the current execution date, and the `create_initial_pipeline_run`
setting was enabled, a pipeline run with execution date `2024-01-04 00:00` would get created
immediately.

**skip\_if\_previous\_running** `boolean`

If `true`, then new pipeline runs won’t be triggered until the most recent pipeline run is
no longer running.

**timeout** `integer`

Optional run timeout in seconds. When provided, the value is a positive integer. A `null`
or omitted value means no timeout.

**sla** `integer`

Optional SLA threshold in seconds. If the pipeline run has been running for more than the SLA
value, then the pipeline will fail. A positive integer enables SLA enforcement; `0` clears the
SLA on update.

**start\_time** `datetime`

Date and time when the trigger should start running.

**status** `string` *required*

`active`, `inactive`

**tags** `array`

An array of tag names. Each tag name can contain up to 70 characters.

**token** `string`

Unique string used in the URL when triggering pipeline via API request.

**updated\_at** `datetime` *required*

Date and time the trigger was last updated.

**variables** `object`

Object containing variables for the pipeline that is being overwritten by this trigger.

**properties**

**\[key]** `string`

The property name is user defined.


# Read pipeline schedules
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/read-pipeline-schedules

GET /api/pipelines/:pipeline_uuid/pipeline_schedules

`GET /api/pipelines/:pipeline_uuid/pipeline_schedules`

**pipeline\_uuid** `string` *required*

Pipeline UUID that the pipeline schedules should all belong to.

**tag\[]** `string`

Fetch pipeline schedules that have the specified tag.

If you include 2 or more `tag[]` query parameters,
then the pipeline schedules will be fetched if it has at
least 1 of the tags specified.

```curl cURL theme={"system"}
curl --request GET \
  --url 'http://localhost:6789/api/pipelines/:pipeline_uuid/pipeline_schedules?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'Authorization: Bearer ZKXScwLKLZUujbw7vwMeWnhtTTSLqifw8_RnBHYipww' \
```

```json Response theme={"system"}
{
  "pipeline_schedules": [
    {
      "id": 42,
      "created_at": "2023-03-01 22:04:33.609624+00:00",
      "updated_at": "2023-03-01 22:04:35.163834+00:00",
      "name": "dry darkness",
      "pipeline_uuid": "dry_bird",
      "schedule_type": "time",
      "start_time": "2023-03-01 22:04:00+00:00",
      "schedule_interval": "@hourly",
      "status": "inactive",
      "variables": null,
      "sla": null,
      "tags": ["core data"],
      "token": "f176321c4eac425782236acd0eb9886d",
      "settings": {},
      "event_matchers": [],
      "last_pipeline_run_status": null,
      "pipeline_runs_count": 0
    },
    {
      "id": 41,
      "created_at": "2023-03-01 20:27:23.645692+00:00",
      "updated_at": "2023-03-31 05:15:51.171762+00:00",
      "name": "dark brook",
      "pipeline_uuid": "dry_bird",
      "schedule_type": "time",
      "start_time": "2023-03-01 22:04:00+00:00",
      "schedule_interval": "@once",
      "status": "inactive",
      "variables": {
        "env": "dev2"
      },
      "sla": 950400,
      "tags": [],
      "token": "1dca215c01564d48bf0d3db7d5450a20",
      "settings": {
        "allow_blocks_to_fail": true,
        "create_initial_pipeline_run": false,
        "skip_if_previous_running": true
      },
      "event_matchers": [],
      "last_pipeline_run_status": null,
      "pipeline_runs_count": 0
    }
  ],
  "metadata": {
    "count": 3,
    "next": false
  }
}
```


# Update schedule
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/update-pipeline-schedule

PUT /api/pipeline_schedules/:id

**id** `integer` *required*

Pipeline schedule ID

**Body**

**pipeline\_schedule** `object` *required*

**payload**

**event\_matchers** `array of objects`

Deprecated. Used only for maintaining existing legacy event triggers.

**name** `string`

Trigger names are trimmed before validation. The name must include at least one letter or
number, cannot exceed 255 characters, and must be unique within the pipeline.

**schedule\_interval** `string`

Required when `schedule_type` is `time`.

**schedule\_type** `string`

Trigger type. Use `api` or `time` for supported non-legacy triggers. `event` is deprecated
and only used for existing legacy event triggers.

**settings** `object`

Additional settings for the trigger.

**properties**

**timeout** `integer`

Optional run timeout in seconds. When provided, the value must be a positive integer.
Use `null` or omit this field for no timeout.

**sla** `integer`

Optional SLA threshold in seconds. When provided, the value must be a positive integer.
Use `0` or `null` to clear an existing SLA, or omit this field for no SLA change.

**start\_time** `datetime`

Persisted schedule start timestamp. For `@once`, this value determines the single run time;
for recurring time schedules, it anchors when the schedule begins.

**status** `string`

**tags** `array of strings`

Non-empty tag names to associate with the trigger. Tag names are trimmed and duplicate names
are ignored before persistence. Each tag name cannot exceed 70 characters.

**variables** `object`

```curl cURL theme={"system"}
curl --request PUT \
  --url 'http://localhost:6789/api/pipeline_schedules/55?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'OAUTH-TOKEN=some_really_long_string' \
  --data '{
    "pipeline_schedule": {
      "name": "black voice11111",
      "schedule_type": "time",
      "schedule_interval": "@once",
      "start_time": "2023-03-06 04:53:00",
      "variables": {
        "env": "dev112222",
        "test": 11111
      },
      "sla": 1000,
      "settings": {
        "allow_blocks_to_fail": true,
        "create_initial_pipeline_run": false,
        "skip_if_previous_running": true,
        "timeout": 3600
      }
    }
  }'
```

```json theme={"system"}
{
  "pipeline_schedule": {
    "id": 55,
    "created_at": "2023-03-08 04:52:54.268096+00:00",
    "updated_at": "2023-04-01 00:52:11.753497+00:00",
    "name": "black voice11111",
    "pipeline_uuid": "example_pipeline",
    "schedule_type": "time",
    "start_time": "2023-03-06 04:53:00+00:00",
    "schedule_interval": "@once",
    "status": "inactive",
    "variables": {
      "env": "dev112222",
      "test": 11111
    },
    "sla": 1000,
    "token": "67d62ed3e66143839f58945bb7d16387",
    "settings": {
      "allow_blocks_to_fail": true,
      "create_initial_pipeline_run": false,
      "skip_if_previous_running": true,
      "timeout": 3600
    },
    "event_matchers": []
  }
}
```


# Create pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/create-pipeline

POST /api/pipelines

`POST /api/pipelines`

**name** `string` *required*

Human readable name of the pipeline.

**type** `string` *required*

The type of the pipeline:
`integration`, `pyspark`, `python`, `streaming`
Note that `python` is a standard (batch) pipeline with a python backend, while `pyspark` is a batch pipeline with a spark backend.

**tags** `array of strings`

Non-empty tag names to associate with the pipeline. Tag names are trimmed and duplicate names are
ignored before persistence. Each tag name cannot exceed 70 characters.

**clone\_pipeline\_uuid** `string`

If supplied, the uuid of a pipeline to clone.

**extensions** `array of objects`

Array of [extension](/design/blocks/extension) block objects. Same shape as `blocks`.

**callbacks** `array of objects`

Array of [callback](/development/blocks/callbacks) block objects.  Same shape as `blocks`.

**pipeline\_callbacks** `array of objects`

Mage Pro only. Pipeline-level callback configuration for callback blocks that should run once
after the pipeline run reaches a terminal status. Callback decorators inside the callback block
determine the behavior for success, failure, and cancellation.

**properties**

**callback\_block\_uuid** `string` *required*

UUID of a callback block in the pipeline.

**enabled** `boolean`

If `false`, the pipeline-level callback is saved but not dispatched.

**conditionals** `array of objects`

Array of [conditional](/development/blocks/conditionals/) block objects.  Same shape as `blocks`.

```bash Request theme={"system"}
  curl --request POST 
  -H 'Content-Type: application/json' 
  -H 'Cookie: oauth_token=some_really_long_string'
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
  -d '{"pipeline": {"name": "arwen-starlight", "type": "python"}}' 
  --url http://localhost:6789/api/pipelines
```

```json Response theme={"system"}
{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "python",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "pipeline_callbacks": [],
    "conditionals": [],
    "widgets": []
  }
}
```


# Delete pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/delete-pipeline

DELETE /api/pipelines/:uuid

`DELETE /api/pipelines/:uuid`

```bash Request theme={"system"}
  curl -X DELETE 
  --url http://localhost:6789/api/pipelines/arwen_starlight
  -H 'Content-Type: application/json' 
  -H 'Cookie: oauth_token=some_really_long_string' 
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' 
```

```json Response theme={"system"}
{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "integration",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "conditionals": [],
    "widgets": []
  }
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/overview

Data pipeline containing execution settings, resources, block information, and block dependencies.

## Pipeline object

```json Response theme={"system"}
{
  "blocks": [
    {
      "all_upstream_blocks_executed": true,
      "color": null,
      "configuration": {},
      "downstream_blocks": [],
      "executor_config": null,
      "executor_type": "local_python",
      "has_callback": null,
      "name": "load_titanic",
      "language": "python",
      "status": "executed",
      "type": "data_loader",
      "upstream_blocks": [],
      "uuid": "load_titanic",
      "content": "...",
      "metadata": {}
    }
  ],
  "description": null,
  "extensions": {},
  "name": "example_pipeline",
  "pipeline_callbacks": [
    {
      "callback_block_uuid": "notify_stakeholders",
      "enabled": true
    }
  ],
  "schedules": [
    {
      "created_at": "2023-03-14T23:24:17.814478+00:00",
      "id": 59,
      "name": "dry haze",
      "pipeline_uuid": "aged_night",
      "schedule_interval": null,
      "schedule_type": "api",
      "start_time": "2023-03-14T23:25:00+00:00",
      "status": "inactive",
      "updated_at": "2023-03-14T23:25:27.351528+00:00"
    }
  ],
  "type": "python",
  "uuid": "example_pipeline",
  "variables": {
    "env": "prod"
  },
  "widgets": []
}
```

**blocks** `array of objects`

Array of block objects. See the [blocks](/api-reference/blocks/overview) section for more details.

**description** `string`

Description for the pipeline.

**extensions** `array of objects`

Array of extension block objects. Same shape as `blocks`.

**name** `string` *required*

Human friendly name of the pipeline.

**pipeline\_callbacks** `array of objects`

Mage Pro only. Pipeline-level callback configuration for callback blocks that should run once
after the pipeline run reaches a terminal status. Callback decorators inside the callback block
determine the behavior for success, failure, and cancellation.

**properties**

**callback\_block\_uuid** `string` *required*

UUID of a callback block in the pipeline.

**enabled** `boolean`

If `false`, the pipeline-level callback is saved but not dispatched.

**schedules** `array of objects` *required*

Array of trigger objects.

**properties**

**created\_at** `string` *required*

Date and time the trigger was created.

**id** `integer` *required*

Trigger ID.

**name** `string` *required*

Name of trigger.

**pipeline\_uuid** `string` *required*

Pipeline UUID the trigger belongs to.

**schedule\_interval** `string`

`@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`

**schedule\_type** `string` *required*

Trigger type. Supported new trigger creation uses `api` or `time`; `event` may appear for
deprecated legacy event triggers.

**start\_time** `datetime`

Date and time when the trigger should start running.

**status** `string` *required*

`active`, `inactive`

**updated\_at** `datetime` *required*

Date and time the trigger was last updated.

**type** `string` *required*

The type of the pipeline:
`integration`, `pyspark`, `python`, `streaming`
Note that `python` is a standard (batch) pipeline with a python backend, while `pyspark` is a batch pipeline with a spark backend.

**uuid** `string` *required*

Unique identifier for the pipeline.

**variables** `object`

Object containing variables for the pipeline.

**properties**

**\[key]** `string`

The property name is user defined.

**widgets** `array of objects`

Array of widget block objects. Same shape as `blocks`.


# Read pipelines
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/read-pipeline

GET /api/pipelines

To retrieve all pipelines:

`GET /api/pipelines`

To fetch a single pipeline:

`GET /api/pipelines/:uuid`

**include\_schedules** `integer`

If `1`, the pipeline object will include associated triggers.

```bash Request theme={"system"}
curl --request GET \
  --url 'http://localhost:6789/api/pipelines/arwen_starlight?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --header 'OAUTH-TOKEN=some_really_long_string'
```

```json Response theme={"system"}
{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "python",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "pipeline_callbacks": [],
    "conditionals": [],
    "widgets": [],
    "extensions": {}
  }
```


# Update pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/update-pipeline

PUT /api/pipelines/:uuid

`PUT /api/pipelines/:uuid`

All default attributes for a pipeline may be updated in addition to the following:

**add\_upstream\_for\_block\_uuid** `string`

The uuid for a block upstream of the pipeline.

**blocks** `array of objects`

Array of block objects. See the [blocks](/api-reference/blocks/overview) section for more details.

**callbacks** `array of objects`

Array of [callback](/development/blocks/callbacks) block objects. Same shape as `blocks`.

**pipeline\_callbacks** `array of objects`

Mage Pro only. Pipeline-level callback configuration for callback blocks that should run once
after the pipeline run reaches a terminal status. Callback decorators inside the callback block
determine the behavior for success, failure, and cancellation.

**properties**

**callback\_block\_uuid** `string` *required*

UUID of a callback block in the pipeline.

**enabled** `boolean`

If `false`, the pipeline-level callback is saved but not dispatched.

**conditionals** `array of objects`

Array of [conditional](/development/blocks/conditionals/) block objects. Same shape as `blocks`.

**extensions** `array of objects`

Array of [extension](/design/blocks/extension) block objects. Same shape as `blocks`.

**schedules** `array of objects`

Array of [schedule](/guides/triggering-pipelines/) trigger objects.

**tags** `array of strings`

Non-empty tag names to associate with the pipeline. Tag names are trimmed and duplicate names are
ignored before persistence. Each tag name cannot exceed 70 characters.

```bash Request theme={"system"}
curl --request PUT 
  --url http://localhost:6789/api/pipelines/arwen_starlight
  -H 'Content-Type: application/json' 
  -H 'Cookie: oauth_token=some_really_long_string'
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
  -d '{"pipeline": {"type": "integration"}}'
```

```json Response theme={"system"}
{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "integration",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "pipeline_callbacks": [],
    "conditionals": [],
    "widgets": [],
    "extensions": {}
  }
}
```


# API policies
Source: https://docs.mage.ai/extensibility/api-reference/policies

Policies for resources determine what actions can be taken on the resource, what attributes can be read, and which attributes can be written.

Define your policy by creating a file `api/policies/SwordPolicy`:

```python theme={"system"}
from .BasePolicy import BasePolicy

class SwordPolicy(BasePolicy):
    pass
```

## API actions

There are 5 actions you can permit:

1. Create: `operations.CREATE` aka `SwordResource.create` method
2. Delete: `operations.DELETE` aka `SwordResource().delete` method
3. Detail: `operations.DETAIL` aka `SwordResource.member` method
4. List: `operations.LIST` aka `SwordResource.collection` method
5. Update: `operations.UPDATE` aka `SwordResource().update` method

## API scopes

There are 2 scopes you can permit:

1. Private: `OauthScope.CLIENT_PRIVATE`; user must be logged in to have access
2. Public: `OauthScope.CLIENT_PUBLIC`; user must be logged out to have access

## Permit CRUD actions

Define which actions a user can perform:

```python theme={"system"}
from api import operations
from api.oauth_scope import OauthScope

class SwordPolicy(BasePolicy):
    pass

SwordPolicy.allow_actions([
    operations.CREATE,
    operations.LIST,
], scopes=[
    OauthScope.CLIENT_PRIVATE,
])
```

If you want to permit certain actions for a user only if a condition is met, do the following:

```python theme={"system"}
SwordPolicy.allow_actions([
    operations.DELETE,
    operations.DETAIL,
    operations.UPDATE,
], scopes=[
    OauthScope.CLIENT_PRIVATE,
], condition=lambda policy: policy.current_user.is_staff)
```

## Permit query parameters

Define which URL query parameters a user is allowed to use:

```python theme={"system"}
SwordPolicy.allow_query([
    'is_powerful',
], scopes=[
    OauthScope.CLIENT_PRIVATE,
])
```

If you want to permit a different query parameter for a specific action, do the following:

```python theme={"system"}
SwordPolicy.allow_query([
    'is_super_powerful',
], scopes=[
    OauthScope.CLIENT_PRIVATE,
], on_action=[
    operations.LIST,
])
```

You can also add a condition to these permissions:

```python theme={"system"}
SwordPolicy.allow_query([
    'is_super_powerful',
], scopes=[
    OauthScope.CLIENT_PRIVATE,
], on_action=[
    operations.LIST,
], condition=lambda policy: policy.current_user.is_staff)
```

## Permit read attributes

Define which attributes of the resource the current user can read:

```python theme={"system"}
class SwordPolicy(BasePolicy):
    def belongs_to_current_user(self):
        return self.current_user.id == self.resource.user_id

SwordPolicy.allow_read([
    'id',
    'name',
    'strength',
], scopes=[
    OauthScope.CLIENT_PRIVATE,
], on_action=[
    operations.DETAIL,
    operations.LIST,
    operations.UPDATE,
], condition=lambda policy: policy.belongs_to_current_user())
```

## Permit write attributes

Define which attributes of the resource the current user can write to when creating a new
resource or updating an existing resource:

```python theme={"system"}
# Allow writing the name and strength attribute when creating a new resource
SwordPolicy.allow_write([
    'name',
    'strength',
], scopes=[
    OauthScope.CLIENT_PRIVATE,
], on_action=[
    operations.CREATE,
])

# Only allow writing the strength attribute when updating the resource
SwordPolicy.allow_write([
    'strength',
], scopes=[
    OauthScope.CLIENT_PRIVATE,
], on_action=[
    operations.UPDATE,
], condition=lambda policy: policy.belongs_to_current_user())
```


# API presenters
Source: https://docs.mage.ai/extensibility/api-reference/presenters

This will determine what attributes of the resource are returned to the client in the response. You can have different sets of attributes be included in the response based on the action (e.g. create, detail, delete, list, update) or a custom format.

Define your presenter by creating a file `api/presenters/SwordPresenter`:

```python theme={"system"}
from .BasePresenter import BasePresenter

class SwordPresenter(BasePresenter):
    default_attributes = [
        'id',
        'name',
        'strength',
    ]
```

By default, whenever a sword is being returned in an API response, it will return a sword object in
JSON format with the `id`, `name`, and `strength` keys.

## Register different formats for different actions

If you want the resource to return different data, you can register unique formats and the
attributes that format should return:

```python theme={"system"}
from .BasePresenter import BasePresenter
from api import operations

class SwordPresenter(BasePresenter):
    default_attributes = [
        'id',
        'name',
        'strength',
    ]

SwordPresenter.register_formats([
    operations.LIST,
], [
    'id',
])
```

This will only return the `id` attribute when making an API call to collections endpoint
`/swords` to retrieve a list of sword resources.

## Register custom formats

You may want to define a custom format that can return specific attributes when making an API
call by including the `_format` URL parameter. For example, you can present specific fields by
calling the `GET /swords?_format=all_attributes` endpoint.

```python theme={"system"}
from .BasePresenter import BasePresenter
from api import operations

class SwordPresenter(BasePresenter):
    default_attributes = [
        'id',
        'name',
        'strength',
    ]

SwordPresenter.register_formats([
    'all_attributes',
], [
    'created_at',
    'deleted_at',
    'id',
    'name',
    'strength',
    'team',
    'user',
])
```

Note: be sure to permit reading these attributes in the `SwordPolicy`.


# API resources
Source: https://docs.mage.ai/extensibility/api-reference/resources

This file handles the CRUD operations.

When you create a new API called `Swords`, the resource
name is `Sword` and the file name is `SwordResource`. The endpoint for CRUD operations are:

| Action | HTTP method | Endpoint |
| - | - | - |
| Create | `POST` | `/swords` |
| List | `GET` | `/swords` |
| Detail | `GET` | `/swords/:id` |
| Update | `PUT` | `/swords/:id` |
| Delete | `DELETE` | `/swords/:id` |

If your resource belongs to a parent model, for example a sword belongs to a team, then your
endpoints will be the following (note that swords for the create and list endpoints are nested
under the `teams` resource:

| Action | HTTP method | Endpoint |
| - | - | - |
| Create | `POST` | `/teams/:id/swords` |
| List | `GET` | `/teams/:id/swords` |
| Detail | `GET` | `/swords/:id` |
| Update | `PUT` | `/swords/:id` |
| Delete | `DELETE` | `/swords/:id` |

## Collection

This method is executed when you make a request for a list of a specific resource.

Example: `GET /swords`.

Create a file called `SwordResource` that is a subclass of 1 of the following:

1. `GenericResource`: subclass this class if you want to execute arbitrary code
2. `DatabaseResource`: subclass this class only if your resource is directly associated to a
   database model. This subclass has many methods defined by default that handles CRUD operations
   automatically.

### Creating a subclass of `GenericResource`

Create a file called `backend/api/resources/SwordResource.py`:

```python theme={"system"}
from .GenericResource import GenericResource
import requests

class SwordResource(GenericResource):
    @classmethod
    def collection(self, query, meta, user, **kwargs):
        """
        query:
            a dictionary of URL query parameters

        meta:
            a dictionary of meta parameters from the URL; such as _limit and _format

        user:
            a user object that represents the current user who is making the API call

        kwargs:
            a dictionary of other values that are passed down through the API request lifecycle; such
            as parent_model, etc.
        """

        # Fetch a list of Swords from the internet
        response = requests.get('https://www.dropbox.com/s/mnjsxzdsyc39hr3/IkeaStorageJson.json')

        return self.build_result_set(response['swords'], user, **kwargs)
```

### Creating a subclass of `DatabaseResource`

```python theme={"system"}
from .DatabaseResource import DatabaseResource
from examples.models import Sword

class SwordResource(DatabaseResource):
    # Set this class variable to the model that the resource is associated with
    model_class = Sword

    """
    Optionally override collection because the DatabaseResource will handle limits
    and pagination automatically. If you don’t override the collection method, DatabaseResource
    will automatically fetch a list of models from the database for you.
    """
    @classmethod
    def collection(self, query, meta, user, **kwargs):
        """
        Use collection to handle pre-filtering of the results from the database.
        """

        # Filter swords that are weak and belong to the current team
        team = kwargs['parent_model']
        return Sword.objects.filter(team_id=team.id, weak=False)
```

Here is how we would get a list of swords that are weak and belong to team ID 1:

```python theme={"system"}
user = User.objects.get(id=3)
team = Team.objects.get(id=1)

# You typically don’t have to call this function by itself.
# This function is called by making a request to /teams/1/swords.
swords = SwordResource.collection({}, {}, user, parent_model=team)
```

## Create

This method is executed when you make a request to create a new instance of a resource.

Example: `POST /swords`.

### `GenericResource`

```python theme={"system"}
from services.datadog import increment

class SwordResource(GenericResource):
    # code from before

    @classmethod
    def create(self, payload, user, **kwargs):
        """
        payload:
            a dictionary of the request body

            For example, if your request body is { "sword" : { "weak": false } },
            then your payload will be the following dictionary: { 'weak': False }.

            NOTE:
            Payloads must be nested inside a key named after the singular form of the resource name.

        user:
            a user object that represents the current user who is making the API call

        kwargs:
            a dictionary of other values that are passed down through the API request lifecycle;
            such as parent_model, etc.
        """

        # Increment a create metric in DataDog
        increment('swords.create')

        if payload.get('weak'):
            name = 'Weak sword'
        else:
            name = 'Strong sword'

        # Always return a new instance of the resource
        return self({ 'name': name }, user, **kwargs)
```

### `DatabaseResource`

You typically never have to override the `create` class method. The instances where it is overridden
include a scenario when you want to execute a callback function only if the resource was
successfully created.

The `DatabaseResource` `create` method will take the keys and values in the payload argument, assign
those values to a newly instantiated `Sword` object, and then save the model to the database.

```python theme={"system"}
class SwordResource(DatabaseResource):
    # code from before

    @classmethod
    def create(self, payload, user, **kwargs):
        def _upload(resource):
            # upload something only if the resource was successfully created

        self.on_create_callback = _upload

        return super().create(payload, user, **kwargs)
```

## Member

This method is executed when you make a fetch a single resource using it’s ID.
Example: `GET /swords/:id`.

### `GenericResource`

```python theme={"system"}
class SwordResource(GenericResource):
    # code from before

    @classmethod
    def member(self, pk, user, **kwargs):
        """
        pk:
            a string or integer from the URL parameter :id

        user:
            a user object that represents the current user who is making the API call

        kwargs:
            a dictionary of other values that are passed down through the API request lifecycle; such
            as parent_model, etc.
        """

        # Fetch 1 of Sword from the internet
        array = requests.get('https://www.dropbox.com/s/mnjsxzdsyc39hr3/IkeaStorageJson.json')
        sword = filter(lambda x: x['id'] == pk, array)

        return self({ 'name': sword['name'], 'weak': False }, user, **kwargs)
```

### `DatabaseResource`

You almost never have to override the `member` method.
If you do, see `TrainingRunResource` for an example.

## Update

This method is executed when you want to update a single resource using it’s ID.
Example: `PUT /swords/:id`.

This is almost the same as the `create` method except it’s an instance method and the ID of the
resource you want to update must be included in the URL.

```python theme={"system"}
class SwordResource(GenericResource):
    # code from before

    def update(self, payload, **kwargs):
        """
        payload:
            a dictionary of the request body

            For example, if your request body is { "sword" : { "weak": false } },
            then your payload will be the following dictionary: { 'weak': False }.

            NOTE:
            Payloads must be nested inside a key named after the singular form of the resource name.

        kwargs:
            a dictionary of other values that are passed down through the API request lifecycle; such
            as parent_model, etc.
        """

        print('Awesome update!')

        # Always return self
        return self
```

### `DatabaseResource`

You almost never have to override the `update` method.
If you do, see `ModelVersionResource` for an example.

## Delete

This method is executed when you want to delete a single resource using it’s ID.
Example: `DELETE /swords/:id`.

```python theme={"system"}
class SwordResource(GenericResource):
    # code from before

    def delete(self, **kwargs):
        """
        kwargs:
            a dictionary of other values that are passed down through the API request lifecycle; such
            as parent_model, etc.
        """

        print('Deleted!')
```

### `DatabaseResource`

You almost never have to override the `delete` method.

## Registering parent models

If your resource belongs to a parent model, for example a sword belongs to a team, you must do
the following in order for these endpoints to work properly:

1. Create: `POST /teams/:id/swords`
2. List: `GET /teams/:id/swords`

```python theme={"system"}
import importlib

class SwordResource(DatabaseResource):
    # code from before

SwordResource.register_parent_resource(
    getattr(importlib.import_module('api.resources.TeamResource'), 'TeamResource'),
)
```

## Pre-load single associated object

The sword model belongs to a user via the `sword.user_id` column. If you are fetching a list of 100
swords and in your code you are accessing the methods of the `user` object via `sword.user`
(e.g. `sword.user.name`), you will inadvertently make 100 extra API calls to the `users`
database table. Do the following to prevent this:

```python theme={"system"}
import importlib

class SwordResource(DatabaseResource):
    # code from before

SwordResource.register_collective_loader_find(
    getattr(importlib.import_module('api.resources.UserResource'), 'UserResource'),
)
```

## Pre-load multiple associated objects

The sword has many opponents that were defeated by it. You can query all the opponents via the
`swords.opponents.objects.all()` method. This will fetch all opponents from the database where
`opponent.sword_id` equals `sword.id`. If you are accessing the opponents of each sword when
fetching a list of 100 swords, you will be making 100 extra API calls to the `opponents` database.
Do the following to prevent this:

```python theme={"system"}
import importlib

class SwordResource(DatabaseResource):
    # code from before

SwordResource.register_collective_loader_select(
    getattr(importlib.import_module('api.resources.OpponentResource'), 'OpponentResource'),
)
```


# Create session
Source: https://docs.mage.ai/extensibility/api-reference/sessions/create-session

POST /api/sessions

`POST /api/sessions`

```curl cURL theme={"system"}
curl --request POST \
  --url http://localhost:6789/api/sessions \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
  --data '{
  "session": {
    "email": "admin@admin.com",
    "password": "admin"
  }
}'
```

```json Response theme={"system"}
{
  "session": {
    "expires": "2023-04-30 03:02:14.781970+00:00",
    "token": "...",
    "user": {
      "first_name": null,
      "id": 1,
      "last_name": null,
      "owner": true,
      "roles": null,
      "roles_display": "Owner",
      "username": "admin"
    }
  }
}
```


# Overview
Source: https://docs.mage.ai/extensibility/api-reference/sessions/overview

Sign in and retrieve an OAuth token.

### Session object

```json Response theme={"system"}
{
  "expires": "2023-04-30 03:02:14.781970+00:00",
  "token": "...",
  "user": {
    "first_name": null,
    "id": 1,
    "last_name": null,
    "owner": true,
    "roles": null,
    "roles_display": "Owner",
    "username": "admin"
  }
}
```

**expires** `datetime` *required*

The date and time the session is no longer valid.

**token** `string` *required*

OAuth token used to authenticate each API request.

**user** `object` *required*

Object containing variables for the pipeline.

**properties**

**id** `integer` *required*

User ID

**owner** `boolean`

If true, the current user has owner permissions.

**roles** `integer`

Roles a user has, represented as an integer.

**roles\_display** `integer`

Human-readable role.

**username** `string`

Unique username for user.

<Snippet />


# Pipeline configuration environment overrides
Source: https://docs.mage.ai/extensibility/env-config/pipeline

Add environment-specific overrides for your pipeline configuration.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=env-specific-config)

### Location of pipeline configuration

Each pipeline in Mage has a `metadata.yaml` configuration file located in the
pipeline uuid's folder under the `pipelines` directory. For example, if you
had a pipeline with uuid `charismatic_inventor`, the folder structure for that
pipeline might look something like this:

```
your_project/
├─ pipelines/
│  ├─ charismatic_inventor/
│  │  ├─ __init__.py
│  │  ├─ interactions.yaml
│  │  ├─ metadata.yaml
│  │  ├─ triggers.yaml
├ ...
```

### **Override pipeline config based on environment**

In the pipeline's `metadata.yaml` config file, add an `overrides` key at the top-level
(no indentations) with the name of your environment (e.g. `prod`, `dev`, `test`) under
the `overrides` key and indented once. Then under the environment name key, add the
properties of your base pipeline config that you want to override. Make sure the
indentations of the properties match those of the base config. Any environment-specific
overrides will REPLACE the matching property in the base config, so be careful when
overriding properties with nested values.

> [!NOTE]
>
> The environment name should match the environment defined in the `ENV`
> [environment variable](https://docs.mage.ai/development/variables/environment-variables).

### Example pipeline config file with environment overrides

```yaml theme={"system"}
# your_project/pipelines/charismatic_inventor/metadata.yaml
blocks:
- all_upstream_blocks_executed: true
  color: null
  configuration:
    disable_output_preview: false
    file_source:
      path: data_loaders/load_titanic.py
    sample_count_preview: 0
  downstream_blocks:
  - starry_forest
  executor_config: null
  executor_type: local_python
  has_callback: false
  language: python
  name: load_titanic
  retry_config: {}
  status: executed
  timeout: null
  type: data_loader
  upstream_blocks: []
  uuid: load_titanic
- all_upstream_blocks_executed: true
  color: null
  configuration:
    sample_count_preview: 8
  downstream_blocks: []
  executor_config: null
  executor_type: local_python
  has_callback: false
  language: python
  name: starry forest
  retry_config: {}
  status: executed
  timeout: null
  type: custom
  upstream_blocks:
  - load_titanic
  uuid: starry_forest
cache_block_output_in_memory: false
callbacks: []
concurrency_config:
  pipeline_run_limit: 5
  pipeline_run_limit_all_triggers: 5
conditionals: []
created_at: '2024-09-27 23:18:10.554952+00:00'
data_integration: null
description: null
executor_config: {}
executor_count: 1
executor_type: null
extensions: {}
name: charismatic inventor
notification_config: {}
remote_variables_dir: null
retry_config: {}
run_pipeline_in_one_process: false
settings:
  triggers: null
spark_config: {}
state_store_config: {}
tags: []
type: python
uuid: charismatic_inventor
variables_dir: /root/.mage_data/default_repo
widgets: []

overrides:
  prod:
    concurrency_config:
      pipeline_run_limit: 10
      pipeline_run_limit_all_triggers: 20
    description: Overridden description for charismatic inventor in prod
  dev:
    concurrency_config:
      pipeline_run_limit: 1
      pipeline_run_limit_all_triggers: 1
    description: Overridden description for charismatic inventor in dev
```

In the example above when in the `dev` environment, the pipeline's configuration property
of `concurrency_config` will have its `pipeline_run_limit` and `pipeline_run_limit_all_triggers`
properties overridden to be `1` (instead of `5` as defined in the base config).
The `description` property will also be overridden to be
`Overridden description for charismatic inventor in dev` instead of `null`.

Similarly for the `prod` environment, `pipeline_run_limit` will be replaced with `10` and
`pipeline_run_limit_all_triggers` with `20`. The `description` property will be overridden
to be `Overridden description for charismatic inventor in prod`. Other environments
(not `dev` or `prod`) will not utilize the `overrides` section.


# Project configuration environment overrides
Source: https://docs.mage.ai/extensibility/env-config/project

Add environment-specific overrides for your project configuration.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=env-specific-config)

### Location of project configuration

Each project in Mage has a `metadata.yaml` configuration file located in the
project's root directory. For example, if you had a project called `your_project`,
the folder structure for that project might look something like this:

```
your_project/
├─ pipelines/
├─ data_loaders/
├─ transformers/
├─ data_exporters/
├─ .../
├─ io_config.yaml
├─ metadata.yaml
├─ requirements.txt
├─ ...
```

### **Override project config based on environment**

In the project's `metadata.yaml` config file, add an `overrides` key at the top-level
(no indentations) with the name of your environment (e.g. `prod`, `dev`, `test`) under
the `overrides` key and indented once. Then under the environment name key, add the
properties of your base project config that you want to override. Make sure the
indentations of the properties match those of the base config. Any environment-specific
overrides will REPLACE the matching property in the base config, so be careful when
overriding properties with nested values.

> [!NOTE]
>
> The environment name should match the environment defined in the `ENV`
> [environment variable](https://docs.mage.ai/development/variables/environment-variables).

### Example project config file with environment overrides

```yaml theme={"system"}
# your_project/metadata.yaml
project_type: standalone

variables_dir: ~/.mage_data
remote_variables_dir: s3://bucket/path_prefix

variables_retention_period: '90d'

emr_config:
  master_instance_type: 'r5.4xlarge'
  slave_instance_type: 'r5.4xlarge'
  master_security_group: 'sg-xxxxxxxxxxxx'
  slave_security_group: 'sg-yyyyyyyyyyyy'
  ec2_key_name: '[ec2_key_pair_name]'

spark_config:
  app_name: 'my spark app'
  spark_master: 'local'
  executor_env: {}
  spark_jars: []
  spark_home:
  others: {}
  use_custom_session: false
  custom_session_var_name: 'spark'

help_improve_mage: true
notification_config:
  alert_on:
  - trigger_failure
  - trigger_passed_sla
  slack_config:
    webhook_url: "{{ env_var('MAGE_SLACK_WEBHOOK_URL') }}"
  teams_config:
    webhook_url: "{{ env_var('MAGE_TEAMS_WEBHOOK_URL') }}"
project_uuid: 123456781234abcd1234abcde123456
features:
  add_new_block_v2: true
  code_block_v2: true
  command_center: true
  custom_design: true
  data_integration_in_batch_pipeline: true
  dbt_v2: true
  interactions: true
  display_local_timezone: true
  notebook_block_output_split_view: true
  operation_history: true
  polars: true
  automatic_kernel_cleanup: false
pipelines:
  settings:
    triggers:
      save_in_code_automatically: true

overrides:
  dev:
    help_improve_mage: false
    pipelines:
      settings:
        triggers:
          save_in_code_automatically: false
    features:
      automatic_kernel_cleanup: true
      global_hooks: true
```

In the example above when in the `dev` environment, the project's nested configuration
property of `save_in_code_automatically` will be overridden to be `false` (instead of
`true` as defined in the base config). The other properties (i.e. `help_improve_mage`,
`automatic_kernel_cleanup`, and `global_hooks`) will also be overridden.
Other environments (not `dev`) will not utilize the `overrides` section.


# Global data products
Source: https://docs.mage.ai/extensibility/global-data-products/overview

Generate and orchestrate the final output of a pipeline (aka data product). Data products can be globally referenced from any pipeline and its data output can be used in any block.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=global-data-products)

![Global data products](https://j.gifs.com/0grLrN.gif)

## Overview

A data product is any piece of data created by 1 or more blocks in a pipeline.
For example, a block can create a data product that is an in-memory DataFrame,
or a JSON serializable data structure, or a table in a database.

A global data product is a data product that can be referenced and used in any pipeline across the
entire project.
A global data product is entered into the global registry (`global_data_products.yaml`)
under a unique ID (UUID) and it references an existing pipeline.

This feature is important because multiple pipelines can depend on a single global data product,
without having to regenerate the global data product.

In addition, the global data product doesn’t have to run unless something needs its data and
its data is outdated. Its data is outdated if it hasn’t ran for a preset amount of time.
This preset time is configurable.

**Example**

If you have a computationally expensive data pipeline called `users_ltv`
that generates the lifetime value (LTV) of each user, and if you have 2 downstream pipelines
that require the data from the `users_ltv` pipeline, then using a global data product
will make sure that the `users_ltv` pipeline is ran only once as long as the `users_ltv`
data product isn’t outdated.

### Minimize number of runs

A global data product is configured to be outdated after a certain amount of time has passed
since its most recent successful run.

It can also be configured to be outdated starting at a specific time of the
year, month, week, day, hour, minute, etc.

A global data product won’t regenerate unless its been outdated.

### Lazy triggering

A global data product won’t run unless something requires its data. For example, if a block in
another pipeline depends on a global data product, then the global data product will be triggered
to run when that block is complete.

If multiple blocks in the same or different pipelines depend on a global data product and they
all attempt to trigger the global data product, it will only run once no matter how many other
blocks depend on it.

### Global referencing

A global data product can be added as a block in any pipeline.
Once added, any other blocks in that pipeline can depend on it or it can depend on other blocks.

When a downstream block depends on a global data product block, Mage passes the selected source
block outputs through [remote blocks](/design/blocks/remote-blocks). Those
outputs are available in the downstream block as `kwargs['remote_blocks']`.

***

## Register global data product

> [!NOTE]
>
> You must have at least 1 pipeline already created.

1. Go to the global data products list page (e.g. [http://localhost:6789/global-data-products](http://localhost:6789/global-data-products))
   and click the button labeled **+ New global data product**.

   You must enter a unique ID (UUID) that is unique across your entire project.
2. Choose the object type `Pipeline`.
   The object types that are currently supported are:

   * `Pipeline`

   **`Block` object type coming soon.**
3. From the Object UUID dropdown, select the pipeline you want to register as a global data product.
4. Click the button labeled **Create global data product**.

***

## Configure settings

Once you’ve created a global data product, you can configure several different settings.

### Outdated after

After the global data product successfully completes running,
how long after that will the global data product be outdated?

You can set how many seconds, weeks, months, or years will have to pass
after the most recent successful run for the global data product to be considered outdated.

### Outdated starting at

If enough time has passed since the last global data product has ran successfully and
the global data product is determined to be outdated,
then you can configure it to be outdated at a specific date or time.

For example, let’s say the global data product is outdated after 12 hours.
The last successful run was yesterday at 18:00.
The global data product will be outdated today at 06:00.
However, if the Outdated starting at has a value of 17 for Hour of day,
then the global data product won’t run again until today at 17:00.

### Block data to output

The data output from the block(s) you select below will be the data product that is returned
when a downstream entity is requesting data from this global data product.

When requesting data from this global data product,
the selected block(s) will return data from its most recent partition.
You can override this by adding a value in the partitions setting.
For example, if you set the partitions value to 5,
then the selected block will return data from its 5 most recent partitions.
If you set the partitions value to 0, then all the partitions will be returned.

***

## Add to a pipeline

In a pipeline, you can add a block and choose `Global data product`.
The data product browser lets you search by name, UUID, description, pipeline, type, source, or
provider. Group the available data products by type or by pipeline, then select one to
preview its metadata and the blocks or upstream data products it is derived from. Pipeline-backed
data products also show the configured output blocks and partition counts when that information is
available.

Click the **+** button to open the global data products page in a new tab when you need to create or
manage a data product without leaving the pipeline editor.

Click **Add to workflow** to add a selected pipeline data product to the current pipeline as a
block. If its pipeline details can’t be loaded, you can still review the available registry metadata
and add it.

### Override global settings

The default settings for the global data product will be used when it’s triggered from
another pipeline. However, you can override those default settings by configuring the
global data product block settings.

Select the global data product block you added to your pipeline.
Then, in the top right corner of the block, click the settings icon to open the block settings
in the right sidekick. From there, you can override the
**Outdated after**, **Outdated starting at**, and **Block data to output** settings.

## Guides

<video />


# Global Hooks Guide
Source: https://docs.mage.ai/extensibility/global-hooks/guides

Follow this step by step guide on creating a global hook in Mage.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=global-hooks)

![Penguin Fishing](https://mage-ai.github.io/assets/penguin_fishing.gif)

### **Step 1: Navigate to the Global Hooks UI**

1. Log in to your Mage account.

> [!NOTE]
>
> If you are new to Mage, follow these steps to enable Global Hooks:
>
> 1. From the Dashboard UI, click Pipelines from the left navigation pane
> 2. Once in the Pipelines UI, click click Settings from the left navigation pane
> 3. Scroll down to view the features settings and toggle on Global Hooks

2. Click on Global Hooks in the left navigation pane
3. Once on the Global Hooks view, click on the "+ Add new global hook" button.

### **Step 2: Configure the Hook**

1. **Hook UUID**: Enter a unique name for your hook.
2. **Resource Type**: Select the type of resource (API call type) from the resource drop down.
3. **Operation Type**: Select the type of operation needed for your hook.

> [!NOTE]
>
> Once the "Create new global Hook" button is clicked the rest of the configuration settings will populate.

4. **Targeting Conditions**: Enter conditions if you want it to apply the hook to only certain pipelines. Leave this field blank to apply the hook to all pipelines.
5. **Before Operation Starts Toggle**: Toggle to on if you want the hook to run before an operation starts.
6. **After Operation Completes Toggle**: Toggle to on if you want the hook to run after an operation ends.
7. **Run if Operation Succeeds Toggle**: Toggle to on if you want to hook to run only if the operation succeeds.
8. **Run if Operation Fails Toggle**: Toggle to on if you want the hook to run if the operation fails.
9. **Pipeline to Execute**: Select the pipeline you want to execute the hook from.
10. **Valid Code Snapshot**: Click the "Create Snapshot of code" button to validate the pipeline code.

> [!NOTE]
>
> When making a code change to the pipeline, you have to re-validate the global hook by clicking the "Create snapshot" button

11. **Stop Operation if Hook Fails Toggle**: Toggle on if you want to stop the operation from running if the hook fails.
12. **Execute Hook with History and Logging Toggle**: Toggle to on if you want to log the hook's execution, but be aware of potential performance impacts.

> [!NOTE]
>
> Toggling on the history and logging feature will significantly degrade performance of the of the operations resolution time.

13. **Run Hook Asynchronously:** Enable this setting to run hooks asynchronously and not block the current resource operation from resolving
14. **Block Outputs**: To select block output features, Click the "+ Add block outputs" button
15. **Block to Extract Data From**: If needed, select the block data will be merge from into the hook operation’s data.
16. **Object to Merge Block into data**: If needed, select the type of data that will be merged into the hook operation’s data.
17. **Additional Dictionary Keys**: Enter information into this field if you have nested objects in your block.

### **Step 3: Save and Test the Hook**

1. Click the "Save changes" button to update the global hook with the new configuration changes.
2. To execute a global hook, perform the action or operation that matches the hook's configured conditions, and the hook will run the designated code or pipeline based on its defined settings.


# Global Hooks
Source: https://docs.mage.ai/extensibility/global-hooks/overview

Global Hooks are a powerful feature in Mage that allows you to execute custom functions before or after an API request. These hooks can be used to perform additional operations, validate data, or integrate with external systems.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=global-hooks)

![Hook](https://mage-ai.github.io/assets/Hook.gif)

### **How Global Hooks Work**

Global Hooks can be triggered at two different points during the execution of a pipeline:

1. **Pre-completion of a Block**: The hook will run before a specific block in the pipeline is executed.
2. **Post-completion of a Block**: The hook will run after a specific block in the pipeline has completed execution.

### **Configuring Global Hooks**

The Global Hooks UI in Mage provides several configuration options to customize the behavior of your hooks. Here's a breakdown of the available settings:

**Hook UUID:** A unique identifier for the hook. This is used to reference the hook within your pipelines.

**Resource Type:** The type of API resource that the hook is attached to (e.g., Pipeline, Dataset, etc.).

**Operation Type:** The operation that triggers the hook (e.g., Create, Update, Delete).

### **Targeting Conditions**

The Targeting Condition fields in the Global Hooks UI allows you to specify a set of conditions that determine which subset of pipelines the hook should run on. This feature gives you control over the hook’s execution of a subset of features.

### **When to Run a Hook**

**Before/After Operation Toggles:** Toggles to specify whether the hook should run before or after the operation.

**Run if Operation Succeeds/Fails Toggles:** Toggles to specify whether the hook should run only if the operation succeeds or fails.

### Code to Run

When a hook is triggered, it will execute the pipeline that was selected in the pipeline to execute dropdown.

**Pipeline to Execute:** The pipeline that will be executed when the hook is triggered.

**Valid Code Snapshot:** A snapshot of the pipeline code that validates the hook before it runs.

> [!NOTE]
>
> When making a code change to the pipeline, you have to re-validate the global hook by clicking the "Create snapshot" button

### How to Run a Hook

**Stop Operation if Hook Fails Toggle:** If enabled, the operation will be stopped if the hook fails.

**Execute Hook with History and Logging Toggle:** If enabled, the hook's execution will be logged and recorded in the history.

> [!NOTE]
>
> Toggling on the history and logging feature will significantly degrade performance of the of the operations resolution time.

**Run Hook Asynchronously Toggle:** If enabled, the hook will run separately from the main operation, so they don't slow down or block the main task from completing

### Block Outputs

Block outputs are optional features that are used to mutate the hook operation’s input and output features.

**Block to Extract Data From:** The block in the pipeline whose output data will be merged into the resource type operation data.

**Object to Merge Block into Data:** The type of API resource output data (e.g., Payload, Resource, Metadata) into which the block output data will be merged.

**Additional Dictionary Keys:** Additional keys that allow merging the block output data into nested objects or values in the resource type operation data.

With Global Hooks, you can extend the functionality of Mage and integrate it with your existing systems and workflows. By leveraging the various configuration options, you can fine-tune the behavior of your hooks to meet your specific requirements.


# Create Deployment
Source: https://docs.mage.ai/extensibility/pro-api-reference/deployments/overview

POST /deployment_settings/{setting_name}/deployments

Trigger a deployment using a specific deployment setting.

`POST /deployment_settings/{setting_name}/deployments`

**setting\_name** `string` *required*

Name of the deployment setting to use for this deployment. Find the deployment setting name on your Mage Pro cluster's `/apps/deploy` page.

**deployment.source.head.name** `string` *required*

The name of the GitHub branch where the commit is located (e.g., `main`, `feature/xyz`).

**deployment.source.name** `string` *required*

A human-readable description of the commit (e.g., GitHub commit message).

**deployment.source.version** `string` *required*

The GitHub commit SHA that will be deployed.

**deployment.stage.branch.name** `string` *required*

The branch name in the target deployment environment (e.g., `deploy-main`).

**deployment.stage.environment** `string` *required*

The environment where the deployment will be created (e.g., `development`, `staging`, `production`).

**deployment.stage.target\_directory** `string`

Optional. The directory path inside the deployment environment where the code will be deployed.

```bash Request theme={"system"}
curl --request POST \
  -H 'Content-Type: application/json' \
  -H 'OAUTH-TOKEN: <your-oauth-token>' \
  -d '{
        "deployment": {
          "source": {
            "head": {
              "name": "main"
            },
            "name": "Add feature X",
            "version": "abc123def456"
          },
          "stage": {
            "branch": {
              "name": "deploy-main"
            },
            "environment": "production",
            "target_directory": "/home/src/project_name"
          }
        },
        "api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2"
      }' \
  --url https://your-mage-instance.com/api/deployment_settings/prod/deployments
```


# Mage Pro API overview
Source: https://docs.mage.ai/extensibility/pro-api-reference/overview



Mage Pro offers two main APIs for managing your workflows and infrastructure:

* **Mage Cloud API**: Used to manage and update your Mage clusters from the Mage Cloud Portal.
* **Mage Pro Cluster API**: Runs on your Mage Pro cluster and is similar to the Mage OSS API, used to manage pipelines, runs, variables, and other resources.

***

# Mage Cloud API

The Mage Cloud API is a private API used internally by the Mage Cloud Portal to manage Mage Pro clusters. It supports operations such as:

* Starting, stopping, and restarting clusters
* Updating cluster environment variables and configurations

## Base URL

`https://cloud.mage.ai/api/v1`

## Authentication

To interact with the Mage Cloud API, you must first authenticate with your Mage Cloud credentials and obtain a session token.

### Example: Initialize session and get token

```Python theme={"system"}
import requests


API_KEY = 'oMqz2m320NfVERFonfytPFVETfPTHyN5N7db9dLK'
BASE_API = 'https://cloud.mage.ai/api/v1'
SESSION_API = f'{BASE_API}/sessions'


def authenticate_session(username: str, password: str) -> requests.Session:
    """
    Authenticate with the Mage Cloud API using credentials and return an authorized session.

    Args:
        username (str): Mage Cloud login email.
        password (str): Mage Cloud login password.

    Required ENV:
        MAGE_CLOUD_API_KEY

    Returns:
        requests.Session: An authenticated session with token set in cookies.
    """

    headers = {
        "Content-Type": "application/json",
        "X-API-KEY": API_KEY
    }
    payload = {
        "session": {
            "email": username,
            "password": password
        }
    }

    session = requests.Session()
    session.headers = headers

    response = session.post(SESSION_API, json=payload)
    if response.json().get('error'):
        raise ValueError(response.text)

    token = response.json()['session']['token']
    session.headers['Cookie'] = f'oauth_token={token}'

    return session
```

## Start or Stop a Cluster

Once authenticated, you can update the cluster status by modifying the replicaCount and status in the cluster's config.

### Example: Start or Stop a Cluster

```Python theme={"system"}
def update_cluster_status(session: requests.Session, team_id: str, cluster_id: str, start: bool = True):
    """
    Start or stop a Mage Pro cluster by updating replicaCount and status.

    Args:
        session (requests.Session): Authenticated Mage Cloud session.
        team_id (str): Mage Cloud team ID.
        cluster_id (str): Cluster ID.
        start (bool): Set True to start, False to stop the cluster.
    """
    cluster_url = f'{BASE_API}/teams/{team_id}/clusters/{cluster_id}'

    response = session.get(cluster_url)
    response.raise_for_status()

    cluster = response.json()['cluster']
    config = cluster.get('config', {})

    config['replicaCount'] = 1 if start else 0
    cluster['status'] = 'Active' if start else 'Stopped'

    update_payload = {
        "cluster": {
            "config": config,
            "status": cluster['status']
        }
    }

    put_response = session.put(cluster_url, json=update_payload)
    put_response.raise_for_status()

    print(f"Cluster has been {'started' if start else 'stopped'}.")
```

**Example usage**

```Python theme={"system"}
# Pass in credentials from your secret manager or CLI prompt
username = 'your_email@example.com'
password = 'your_password_here'

session = authenticate_session(username, password)

# Start the cluster
update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=True)

# Stop the cluster
# update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=False)
```

## Update Cluster Environment Variables

To add or update environment variables in a Mage Pro cluster, modify the `extraEnvs` field in the cluster config.

### Example: Update Environment Variables

```Python theme={"system"}
def update_cluster_env_vars(session, team_id, cluster_id, updates):
    """
    Add or update environment variables in a Mage Pro cluster.

    Args:
        session (requests.Session): Authenticated session with Mage Cloud API.
        team_id (str): Mage team ID.
        cluster_id (str): Cluster ID.
        updates (dict): Dictionary of environment variable key-value pairs to add or update.
    """
    cluster_url = f'https://cloud.mage.ai/api/v1/teams/{team_id}/clusters/{cluster_id}'

    # Fetch current cluster config
    response = session.get(cluster_url)
    response.raise_for_status()
    cluster = response.json()['cluster']
    config = cluster.get('config', {})

    # Load existing environment variables
    existing_envs = config.get('extraEnvs', [])
    env_map = {env['name']: env['value'] for env in existing_envs}

    # Apply updates
    env_map.update(updates)

    # Convert back to list format
    config['extraEnvs'] = [{'name': k, 'value': v} for k, v in env_map.items()]

    # Build payload and apply update
    update_payload = {
        "cluster": {
            "config": config
        }
    }

    put_response = session.put(cluster_url, json=update_payload)
    put_response.raise_for_status()

    print("Cluster environment variables updated successfully.")
```

**Example usage**

```Python theme={"system"}
session = authenticate_session(username='your_email@example.com', password='your_password')

update_cluster_env_vars(
    session=session,
    team_id='your_team_id',
    cluster_id='your_cluster_id',
    updates={
        'AI_SIDEKICK': '1',
        'DEFAULT_EXECUTOR_TYPE': 'k8s',
        'ENABLE_SOME_FEATURE': 'true'
    }
)
```

***

# Mage Pro cluster API

The Mage Pro Cluster API is the set of HTTP APIs available on your running Mage Pro cluster. It extends the Mage OSS API and allows:

* Create/Get/Update/Delete pipelines/blocks
* Trigger pipelines
* CRUD pipeline schedules/triggers
* CRUD pipeline runs
* CRUD workspaces
* Deploy git commits
* More APIs available in Mage Pro UI

## Base URL

`https://<your-cluster-hostname>/api`

For the full list of cluster API endpoints, see the [Mage Open Source API Reference](/extensibility/api-reference/overview).

> Note: Mage Pro may expose additional admin or authentication APIs depending on your deployment and licensing tier.


# Update Workspace
Source: https://docs.mage.ai/extensibility/pro-api-reference/workspaces/update-workspace

PUT /api/workspaces/{workspace_name}

Update the configuration of an existing workspace in your Mage Pro cluster.

`PUT /api/workspaces/{workspace_name}`

## Path Parameters

**workspace\_name** `string` *required*

Name of the workspace to update. You can find the workspace name on your Mage Pro cluster's `/apps/workspaces` page.

## Request Body Parameters

All request body parameters are nested under the `workspace` object.

**workspace.action** `string` *required*

The action to perform on the workspace. Supported values: `"stop"`, `"resume"`, `"patch"`, `"add_to_ingress"`.
See the [Supported Actions](#supported-actions) section below for details on each action and which parameters they support.

**workspace.container\_config** `string`

Optional. YAML configuration string for the container. Used with the `patch` action.\
Allows you to update container-level settings such as environment variables, resource limits, and other container configurations.
For example, updating environment variables:

```
env:
  - name: ENV
    value: test
  - name: API_KEY
    value: secret-key
```

**workspace.lifecycle\_config** `object`

Optional. Dictionary/object containing lifecycle configuration. Used with the `patch` action.
Defines lifecycle management policies including termination policies, pre-start scripts, and post-start hooks.
Structure:

```json theme={"system"}
{
  "termination_policy": {
    "enable_auto_termination": true,
    "max_idle_seconds": 1800
  },
  "pre_start_script_path": "/path/to/script.sh",
  "post_start": {
    "command": ["echo", "Workspace started"],
    "hook_path": "/path/to/hook.sh"
  }
}
<ParamField body="workspace.action" type="string" required>
The action to perform. Use `"patch"` to apply partial updates to the workspace configuration.
</ParamField>

<ParamField body="workspace.container_config" type="string">
Optional. Configuration string for the container.  
For example, updating the workspace image tag and environment variables:
```

image\_tag: 2026.06.16
env:

* name: ENV
  value: test

````
</ParamField>

<ParamField body="workspace.update_workspace_settings" type="boolean">
Optional. Whether to update workspace settings with the current cluster's environment variables. Used with the `patch` action.
Set to `true` to sync the workspace configuration with the cluster-level environment variables.
When enabled, the workspace will inherit environment variables from the cluster configuration.
</ParamField>

## Supported Actions

The `action` field determines what operation will be performed on the workspace. Each action supports different parameters and cluster types.

### `stop`

Stops a running workspace. The workspace metadata is preserved, but the cloud instance is stopped or paused.

**Supported cluster types:** K8S, Docker

**Required parameters:**
- `workspace.action`: `"stop"`

**Optional parameters:** None

**Request body:**
```json
{
"workspace": {
  "action": "stop"
}
}
````

**Example:**

```bash theme={"system"}
curl --request PUT \
  -H 'Content-Type: application/json' \
  -H 'OAUTH-TOKEN: <your-oauth-token>' \
  -d '{
        "workspace": {
          "action": "stop"
        }
      }' \
  --url https://your-mage-instance.com/api/workspaces/my-workspace
```

### `resume`

Resumes a stopped workspace, restarting the cloud instance.

**Supported cluster types:** K8S, Docker

**Required parameters:**

* `workspace.action`: `"resume"`

**Optional parameters:** None

**Request body:**

```json theme={"system"}
{
  "workspace": {
    "action": "resume"
  }
}
```

**Example:**

```bash theme={"system"}
curl --request PUT \
  -H 'Content-Type: application/json' \
  -H 'OAUTH-TOKEN: <your-oauth-token>' \
  -d '{
        "workspace": {
          "action": "resume"
        }
      }' \
  --url https://your-mage-instance.com/api/workspaces/my-workspace
```

### `patch`

Applies partial updates to the workspace configuration, such as updating environment variables, container settings, or lifecycle configurations.

**Supported cluster types:** K8S, Docker

**Required parameters:**

* `workspace.action`: `"patch"`

**Optional parameters:**

* `workspace.container_config` (string) - YAML configuration string for the container
* `workspace.lifecycle_config` (object) - Dictionary containing lifecycle configuration (termination policy, pre-start scripts, post-start hooks)
* `workspace.update_workspace_settings` (boolean) - Whether to sync workspace settings with cluster-level environment variables

**Request body:**

```json theme={"system"}
{
  "workspace": {
    "action": "patch",
    "container_config": "env:\n  - name: ENV\n    value: test",
    "lifecycle_config": {
      "termination_policy": {
        "enable_auto_termination": true,
        "max_idle_seconds": 1800
      },
      "pre_start_script_path": "/path/to/script.sh"
    },
    "update_workspace_settings": true
  }
}
```

**Example:**

```bash theme={"system"}
curl --request PUT \
  -H 'Content-Type: application/json' \
  -H 'OAUTH-TOKEN: <your-oauth-token>' \
  -d '{
        "workspace": {
          "action": "patch",
          "container_config": "env:\n  - name: ENV\n    value: test",
          "update_workspace_settings": true
        }
      }' \
  --url https://your-mage-instance.com/api/workspaces/my-workspace
```

### `add_to_ingress`

Adds the workspace service to the Kubernetes ingress configuration, making it accessible via the ingress URL.

**Supported cluster types:** Kubernetes (K8S) only

**Required parameters:**

* `workspace.action`: `"add_to_ingress"`

**Optional parameters:** None

**Request body:**

```json theme={"system"}
{
  "workspace": {
    "action": "add_to_ingress"
  }
}
```

**Example:**

```bash theme={"system"}
curl --request PUT \
  -H 'Content-Type: application/json' \
  -H 'OAUTH-TOKEN: <your-oauth-token>' \
  -d '{
        "workspace": {
          "action": "add_to_ingress"
        }
      }' \
  --url https://your-mage-instance.com/api/workspaces/my-workspace
```

Optional. Whether to update workspace settings with the current cluster's environment variables.\
Set to `true` to sync the workspace configuration with the cluster-level environment.

```bash Request theme={"system"}
curl --request PUT \
  -H 'Content-Type: application/json' \
  -H 'OAUTH-TOKEN: <your-oauth-token>' \
  -d '{
        "workspace": {
          "action": "patch",
          "container_config": "env:\n  - name: ENV\n    value: test",
          "container_config": "image_tag: 2026.06.16\nenv:\n  - name: ENV\n    value: test",
          "update_workspace_settings": true
        }
      }' \
  --url https://your-mage-instance.com/api/workspaces/my-workspace
```


# Mage Pro data pipeline
Source: https://docs.mage.ai/getting-started/build-pro-pipeline

Build your first Mage Pro data pipeline

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=build-pro-pipeline)

Mage makes building data pipelines intuitive through its block-based architecture. Instead of writing complex scripts, you create  blocks that handle specific parts of your data workflow. Mage offers a complete data pipeline workflow architecture so you can extract, transform, and load your data to a target in one tool.

**Pipeline Types:**
Mage supports three main pipeline patterns to match your use case:

* **Batch pipelines** for scheduled data processing jobs
* **Data integration pipelines** for syncing data to and from different systems
* **Streaming pipelines** for real-time data processing

**What are blocks?**
Blocks are modular pieces of code that perform specific functions in your pipeline. Each block has a defined purpose:

* **Data loader blocks** bring data into your pipeline from various sources
* **Transformer blocks** clean, filter, and reshape your data
* **Data exporter blocks** send processed data to its final destination

Let's walk through creating your first batch pipeline step by step.

**Steps to build you first pipeline:**

**Step 1**: From the home page click “Pipelines” from the left popout navigation menu

**Step 2**: Click the green “New pipeline” button

**Step 3**: Click “Start from scratch”

You’ll notice that you can select from the three main data engineering use cases as described above: batch, data integration, and streaming. For the purposes of this getting started tutorial we will select the batch pipeline option.

![Create new pipeline](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/create-new-pipeline.png)

After you fill in the Name and optionally complete the Description and Tags, click the blue **Create pipeline** button. Pipeline names must be unique; if the name is already in use, the form shows **A pipeline with this name already exists.** and does not overwrite the existing pipeline. After creation, Mage takes you to the pipeline editor page where we will create our first data loader block.

**Extract data from source systems**

The first block we’ll add to out data pipeline is a data loader block. the data loader block is meant to extract data from source systems, whether its cloud storage, an API or database, Mage Pro has a connection for nearly anything.

**Steps to create your first block:**

**Step 1**: Click the “Blocks” button located in the upper left portion of the UI

![Create new pipeline](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/blocks-blow-up.png)

**Step 2**: A drop down menu will appear, just hover over “Loader“ then select the “Base template (generic)” block under Python.

**Step 3**: The **Block name** dialog will appear. Enter a name such as “fetch data,” then click **Save**.

This will add a new generic Python block to your pipeline. Once the block appears in your pipeline editor UI add the code below and click the “Run code” button located in the top right portion of the block.

```Python theme={"system"}
import pandas as pd
import requests

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data(*args, **kwargs):

    url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/medical.csv'
    df = pd.read_csv(url)
    
    return df
```

Congratulations on adding your first block to a pipeline. This is the first major step in creating a working data pipeline in Mage Pro. Next we’re going to add a transformation step in data pipeline.

**Transform data**

**Step 1**: Add a generic python transformation block to your pipeline by following the same steps above when you added the data loader block

**Step 2**: Give the block a meaningful name such as “transformation risk”

**Step 3**: Add the code below to the transformation block and click the “Run code” button to return a sample output.

```Python theme={"system"}
if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@transformer
def transform(data, *args, **kwargs):

    def classify_health_risk(health_score):
        if health_score <= 3:
            return "Low"
        elif health_score <= 6:
            return "Medium"
        else:
            return "High"
    
    data['heart_health_risk'] = data['health_score'].apply(classify_health_risk)

    return data
```

Finally we’ll want to export our data to a target destination. Mage has many prebuilt templates for writing data to target destinations. From Snowflake to GCS, even Iceberg, Mage can write data to many different target vendors.

To export data complete the following steps:

**Step 1**: Add a generic python exporter block to your pipeline by following the same steps above when you added the data loader and transformer block.

**Step 2**: Give the code block a meaningful name such as “risk data export.”

**Step 3**: Add the code below to the transformation block and click the “Run code” button to return a sample output.

```Python theme={"system"}
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(data, *args, **kwargs):

    return data
```

You did it, you created your first pipeline in Mage. Wasn’t it fun? Mage was created with developers in mind and we pride ourselves on an experience that is both easy and fun for developers. But, we’re not done yet. Next we need to schedule this pipeline so we can automate our workflow.


# Cluster Management
Source: https://docs.mage.ai/getting-started/cluster-management-portal

Manage all your Mage Pro clusters

### Cluster Management Portal

Mage Pro is split into two environments, the Cluster Manager, which supports keeping your clusters and schedulers turned on at all times, and a development environment. The Cluster Manager is where you can see consumption insights into your cluster, add and edit environment variables, and check out estimated pricing for the cluster.

### Add Cluster

Adding a new cluster to your environment is simple in Mage Pro. Just click the “Lanch new cluster” button, provide it a name, and then click “Create cluster.” Your new cluster will take a few minutes to spin up. Once complete you can open the cluster to your development environment or

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/create-new-cluster.png)

### Environment variables

Environment variables are key-value pairs stored in the operating system that can be accessed by applications running on that system. They provide a way to configure application behavior without changing code, which is particularly useful for:

* Storing configuration settings
* Managing credentials securely (without hardcoding them)
* Controlling application behavior across different environments (development, testing, production)

**Add or edit an environment variable**
After accessing the cluster management page by clicking the blue link under the Clusters column, you'll see a blue "Edit / Add variables" button. This section explains how to use this feature to configure your environment variables.

* Navigate to the cluster management page
* Locate and click the blue "Edit / Add variables" button
* In the interface that appears, you can:
  * Add new variables by specifying a name and value
  * Edit existing variables by modifying their values
  * Remove variables when they're no longer needed

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/env-var.png)

### Add users to cluster manager

User management is a component of both the Mage Pro cluster manager and development environment. The cluster manager implements a simple permission system with only three distinct roles:

* **Admin** - Full control to create and delete clusters, modify environment variables, and manage cluster operations (start/stop)
* **Editor** - Limited permissions to modify aspects of the cluster within already created clusters.
* **Viewer** - Read-only access to view cluster configuration and technical details

These role-based permissions help teams maintain appropriate access controls while collaborating on data pipeline projects.

### **How to add or edit user roles**:

Team administrators can manage access permissions for cluster members through the cluster manager interface. Here's how to handle common user management tasks:

**Inviting new users**

1. Navigate to the users page and click "Invite user"
2. Enter the person's email address in the popup box
3. Select the appropriate permission level from the dropdown menu
4. Click "Invite user" to send the invitation

**Modifying existing users**

When you need to adjust permissions or remove someone:

1. Find and click on the user's row in the members list
2. Click the "Actions" button to reveal options
3. Choose "Remove user" to revoke access completely, or
4. Select "Change role" to modify their permissions
5. If changing roles, select the new role from the dropdown
6. Confirm by clicking "Change role"

These straightforward controls help maintain proper access management as your team evolves.

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/user-profile.png)

### Monitor your costs and usage

Keeping track of your resource usage and expenses is straightforward with Mage Pro's built-in monitoring tools. The insights dashboard gives you visibility into each cluster's financial and performance metrics.

**Accessing Usage Metrics**

1. Open your cluster by clicking its blue link in the "Clusters" column
2. Select the "Insights" tab from the navigation menu
3. The default view shows your estimated costs to date
4. Toggle between views by clicking “Realtime cpu usage" or "Realtime memory usage" to check real-time resource utilization

This visibility helps teams manage their infrastructure spending and identify potential performance bottlenecks before they impact operations. Regularly reviewing these metrics can help optimize your cluster configuration for both cost and performance.

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/cluster-insights.png)


# Install dependencies
Source: https://docs.mage.ai/getting-started/install-dependencies

Install dependencies using requirements.txt with automatic environment management

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dependency-installation)

Mage Pro includes an advanced dependency management system that automatically creates and manages isolated virtual environments for your projects. Adding new packages is simple and doesn't require cluster restarts - you can immediately run your code after installation.

## Quick Start: Installing Dependencies via UI

The easiest way to install dependencies is through the Mage UI:
description: "Install dependencies using requirements.txt"
----------------------------------------------------------

Mage Pro makes dependency management simple through the requirements.txt file in your workspace's root folder. Adding new packages doesn't require cluster restarts, so you can immediately run your code after installation.

Installing dependencies in Mage Pro can be done through the requirements.txt file located in the project folder of your cluster or workspace.

![Requirements.txt](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/requirements-txt.png)

### Steps:

1. Open **Tools** in the left navigation, then select **Editor**

2. **Open `requirements.txt`**: Browse to `/home/src/[project_path]/requirements.txt` and click to open it in the editor

3. **Add your packages**: Edit the file to include your dependencies
   ```txt theme={"system"}
   pandas==2.0.0
   numpy>=1.24.0
   scikit-learn==1.3.0
   ```

4. **Click the "Install dependencies" button**: This will automatically:
   * Open a terminal at the bottom of the page
   * Run `mage deps install` command
   * Show real-time installation progress
   * Install only the packages that need to be added or updated

![Install Dependencies](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/pip-install-r.png)

Your dependencies are now installed and ready to use - **no cluster restart required**!

## How It Works

Behind the scenes, Mage Pro's dependency management system:

* **Creates isolated virtual environments** for each unique combination of dependencies
* **Caches and reuses environments** based on content fingerprinting for faster installation
* **Intelligently filters packages** to reuse compatible packages from the global environment
* **Uses `uv` for fast installation** with automatic fallback to `pip`
* **Automatically activates environments** when the server starts

When you restart your cluster, the system automatically creates or updates the virtual environment based on your latest `requirements.txt`, ensuring your dependencies are always synchronized.

## Alternative Methods

### Via Mage Terminal

You can also run dependency commands directly in the Mage terminal:

```bash theme={"system"}
# Install dependencies
mage deps install

# Install from custom requirements file
mage deps install -r production_requirements.txt

# List all virtual environments
mage deps list

# Clean up old environments
mage deps cleanup
```

### Automatic on Cluster Restart

When you restart your Mage cluster, dependencies are automatically installed based on your `requirements.txt` - no manual intervention needed.

## Learn More

For advanced dependency management features including environment fingerprinting, cleanup policies, and programmatic usage, see the complete [Dependency Management](/development/dependencies/dependency-management) guide.
Follow these steps to add and install requirements:

1. Click the requirements.text file to open the file in your editor
2. Add your packages to the file and save it
3. Open **Tools** in the left navigation, then select **Terminal**
4. Run `pip install -r requirements.txt`

![Pip Install](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/pip-install-r.png)

Following this steps will install your requirements and you will not need to restart the cluster after installation. You can run your code as written


# Python kernels
Source: https://docs.mage.ai/getting-started/kernels

Configure Python kernel execution in the pipeline editor.

1. [Python3](#python3)
2. [Multi-user kernels in Mage Pro](#multi-user-kernels)
3. [Restart kernel](#restart-kernel)

## Python3 kernel&#x20;

Python3 is the default kernel. You can prototype and transform small to medium
size datasets with this kernel. Pipelines built with this kernel can be executed
in Python environments.

## Multi-user kernels in Mage Pro&#x20;

Mage Pro can run a separate Python kernel for each authenticated user in the
pipeline editor. This lets multiple users run blocks at the same time without
sharing one notebook kernel, execution queue, or output stream.

Enable the feature by setting the following environment variable on your Mage
Pro cluster:

```bash theme={"system"}
MULTI_USER_JUPYTER_KERNELS=1
```

When this setting is enabled:

* Each authenticated user gets an isolated Python kernel for interactive block
  execution from the pipeline editor.
* Kernel output is routed only to websocket clients for the user that started
  the block run.
* Interrupt and restart actions apply to the current user's kernel.
* Idle kernels without connected websocket clients are cleaned up automatically.

You can tune resource cleanup with:

| Variable | Description | Default |
| - | - | - |
| `JUPYTER_KERNEL_IDLE_TIMEOUT_SECONDS` | Number of seconds an inactive user kernel can stay alive after its websocket clients disconnect. | `1800` |
| `JUPYTER_KERNEL_MAX_PER_SERVER` | Maximum number of user kernels to keep on one Mage server. When the limit is reached, Mage evicts idle kernels first. | `50` |

For debugging, admins can list all active user kernels with:

```bash theme={"system"}
GET /api/kernels?include_all=1
```

By default, `GET /api/kernels` only returns the kernel for the current user.

> [!NOTE]
>
> Multi-user Jupyter kernels are a Mage Pro feature. Mage uses the Python3 kernel
> for this mode; the legacy PySpark notebook kernel is not supported.

## Restart kernel

If your block execution is hanging in the notebook and no block output is
printed, you can try interrupting the block execution and restarting the kernel.
You can find the "Restart kernel" button in the "Run" menu at the top of the
notebook.


# Mage Pro API
Source: https://docs.mage.ai/getting-started/mage-pro-api

Deploy your code in seconds!

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mage-pro-api)

### **Mage Pro API overview**

Mage Pro offers two main APIs for managing your workflows and infrastructure:

* **Mage Cloud API**: Used to manage and update your Mage clusters from the Mage Cloud Portal.
* **Mage Pro cluster API**: Runs on your Mage Pro cluster and is similar to the Mage OSS API, used to manage pipelines, runs, variables, and other resources.

# **Mage Cloud API**

The Mage Cloud API is a private API used internally by the Mage Cloud Portal to manage Mage Pro clusters. It supports operations such as:

* Starting, stopping, and restarting clusters
* Updating cluster environment variables and configurations

## **Base URL**

**`https://cloud.mage.ai/api/v1`**

## **Authentication**

To interact with the Mage Cloud API, you must first authenticate with your Mage Cloud credentials and obtain a session token.

### **Example: Initialize session and get token**

```Python theme={"system"}
import requests


API_KEY = 'oMqz2m320NfVERFonfytPFVETfPTHyN5N7db9dLK'
BASE_API = 'https://cloud.mage.ai/api/v1'
SESSION_API = f'{BASE_API}/sessions'


def authenticate_session(username: str, password: str) -> requests.Session:
    """
    Authenticate with the Mage Cloud API using credentials and return an authorized session.

    Args:
        username (str): Mage Cloud login email.
        password (str): Mage Cloud login password.

    Required ENV:
        MAGE_CLOUD_API_KEY

    Returns:
        requests.Session: An authenticated session with token set in cookies.
    """

    headers = {
        "Content-Type": "application/json",
        "X-API-KEY": API_KEY
    }
    payload = {
        "session": {
            "email": username,
            "password": password
        }
    }

    session = requests.Session()
    session.headers = headers

    response = session.post(SESSION_API, json=payload)
    if response.json().get('error'):
        raise ValueError(response.text)

    token = response.json()['session']['token']
    session.headers['Cookie'] = f'oauth_token={token}'

    return session
```

## **Start or stop a cluster**

Once authenticated, you can update the cluster status by modifying the replicaCount and status in the cluster’s config.

### **Example: Start or stop a cluster**

```Python theme={"system"}
def update_cluster_status(session: requests.Session, team_id: str, cluster_id: str, start: bool = True):
    """
    Start or stop a Mage Pro cluster by updating replicaCount and status.

    Args:
        session (requests.Session): Authenticated Mage Cloud session.
        team_id (str): Mage Cloud team ID.
        cluster_id (str): Cluster ID.
        start (bool): Set True to start, False to stop the cluster.
    """
    cluster_url = f'{BASE_API}/teams/{team_id}/clusters/{cluster_id}'

    response = session.get(cluster_url)
    response.raise_for_status()

    cluster = response.json()['cluster']
    config = cluster.get('config', {})

    config['replicaCount'] = 1 if start else 0
    cluster['status'] = 'Active' if start else 'Stopped'

    update_payload = {
        "cluster": {
            "config": config,
            "status": cluster['status']
        }
    }

    put_response = session.put(cluster_url, json=update_payload)
    put_response.raise_for_status()

    print(f"Cluster has been {'started' if start else 'stopped'}.")
```

Example usage:

```Python theme={"system"}
# Pass in credentials from your secret manager or CLI prompt
username = 'your_email@example.com'
password = 'your_password_here'

session = authenticate_session(username, password)

# Start the cluster
update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=True)

# Stop the cluster
# update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=False)
```

## **Update cluster environment variables**

To add or update environment variables in a Mage Pro cluster, modify the **`extraEnvs`** field in the cluster config.

### **Example: update environment variables**

```Python theme={"system"}
def update_cluster_env_vars(session, team_id, cluster_id, updates):
    """
    Add or update environment variables in a Mage Pro cluster.

    Args:
        session (requests.Session): Authenticated session with Mage Cloud API.
        team_id (str): Mage team ID.
        cluster_id (str): Cluster ID.
        updates (dict): Dictionary of environment variable key-value pairs to add or update.
    """
    cluster_url = f'https://cloud.mage.ai/api/v1/teams/{team_id}/clusters/{cluster_id}'

    # Fetch current cluster config
    response = session.get(cluster_url)
    response.raise_for_status()
    cluster = response.json()['cluster']
    config = cluster.get('config', {})

    # Load existing environment variables
    existing_envs = config.get('extraEnvs', [])
    env_map = {env['name']: env['value'] for env in existing_envs}

    # Apply updates
    env_map.update(updates)

    # Convert back to list format
    config['extraEnvs'] = [{'name': k, 'value': v} for k, v in env_map.items()]

    # Build payload and apply update
    update_payload = {
        "cluster": {
            "config": config
        }
    }

    put_response = session.put(cluster_url, json=update_payload)
    put_response.raise_for_status()

    print("Cluster environment variables updated successfully.")
```

Example usage:

```Python theme={"system"}
session = authenticate_session(username='your_email@example.com', password='your_password')

update_cluster_env_vars(
    session=session,
    team_id='your_team_id',
    cluster_id='your_cluster_id',
    updates={
        'AI_SIDEKICK': '1',
        'DEFAULT_EXECUTOR_TYPE': 'k8s',
        'ENABLE_SOME_FEATURE': 'true'
    }
)
```

# **Mage Pro cluster API**

The Mage Pro Cluster API is the set of HTTP APIs available on your running Mage Pro cluster. It extends the Mage OSS API and allows:

* Create/Get/Update/Delete pipelines/blocks
* Trigger pipelines
* CRUD pipeline schedules/triggers
* CRUD pipeline runs
* CRUD workspaces
* Deploy git commits
* More APIs available in Mage Pro UI

> **Mage Pro /runs api endpoint**
>
> The `/runs` endpoint executes a single block within a pipeline and returns its output in the same HTTP response. No polling required.
>
> This is the right tool when an external application needs the result of a specific block right now, not after a full pipeline run. Common use cases include real-time scoring, record lookups, and enrichment logic triggered per request.
>
> [YouTube video player](https://www.youtube.com/embed/iZzpaO3JumY)
>
> ***
>
> ### What It Does
>
> When you call `/runs`, Mage works through the following steps:
>
> 1. Validates the token against an API-type pipeline trigger
> 2. Resolves the target pipeline and block
> 3. Runs upstream blocks if `run_upstream_blocks` is `true`
> 4. Executes the target block with any variables injected via `kwargs`
> 5. Returns the block output in the same HTTP response
> 6. Writes a `PipelineRun` and `BlockRun` record to the database if `record` is `true`
>
> ***
>
> ### Endpoint
>
> ```
> POST https://<your-cluster-hostname>/api/runs/{token}
> ```
>
> ***
>
> ### Authentication
>
> Every call to `/runs` requires a token from an API-type pipeline trigger.
>
> **Option A: Token in the URL**
>
> ```
> POST /api/runs/your_token_here
> ```
>
> **Option B: Bearer token in the Authorization header**
>
> ```
> POST /api/runs
> Authorization: Bearer your_token_here
> ```
>
> To generate a token:
>
> 1. Open your pipeline in Mage Pro and go to the **Triggers** tab
> 2. Click **Add Trigger** and select type **API**
> 3. Save and copy the token from the trigger detail page
>
> ***
>
> ### Request Body
>
> ```json theme={"system"}
> {
>     "run": {
>         "pipeline_uuid":       "your_pipeline_uuid",
>         "block_uuid":          "your_block_uuid",
>         "variables":           {},
>         "project":             null,
>         "record":              true,
>         "store_variables":     true,
>         "run_upstream_blocks": false,
>         "incomplete_only":     false
>     }
> }
> ```
>
> ***
>
> ### Example: Run a single block
>
> ```python theme={"system"}
> import requests
>
> MAGE_URL = 'https://<your-cluster-hostname>'
> TOKEN    = 'your_trigger_token_here'
>
> response = requests.post(
>     f'{MAGE_URL}/api/runs/{TOKEN}',
>     json={
>         'run': {
>             'pipeline_uuid': 'my_pipeline',
>             'block_uuid':    'my_block'
>         }
>     }
> )
>
> result = response.json()['run']
> print(result)
> ```
>
> ***
>
> ### Example: Run a block with runtime variables
>
> Pass variables in the request body and they get injected into the block via `kwargs`.
>
> ```python theme={"system"}
> response = requests.post(
>     f'{MAGE_URL}/api/runs/{TOKEN}',
>     json={
>         'run': {
>             'pipeline_uuid': 'patient_lookup',
>             'block_uuid':    'transform_medical_data',
>             'variables': {
>                 'patient_id': 'PID00087'
>             }
>         }
>     }
> )
>
> result = response.json()['run']
> print(result)
> ```
>
> Access the variable inside your block:
>
> ```python theme={"system"}
> if 'transformer' not in globals():
>     from mage_ai.data_preparation.decorators import transformer
>
> @transformer
> def transform(df, *args, **kwargs):
>     patient_id = kwargs.get('patient_id')
>     # use patient_id in your block logic
>     return df
> ```
>
> ***
>
> ### Response
>
> A successful call returns HTTP 200 with the block output under the `run` key:
>
> ```json theme={"system"}
> { "run": { ...block output... } }
> ```
>
> ***
>
> ### Request Options Reference
>
> | Option | Default | Description |
> | - | - | - |
> | `pipeline_uuid` | required | The unique identifier of the pipeline containing the target block |
> | `block_uuid` | required | The unique identifier of the block to execute |
> | `variables` | `{}` | Key-value pairs injected into the block at runtime via `kwargs` |
> | `record` | `true` | Write a PipelineRun and BlockRun record to the Mage database |
> | `store_variables` | `true` | Persist the passed variables to disk alongside the block run record |
> | `run_upstream_blocks` | `false` | Run all upstream block dependencies before executing the target block |
> | `incomplete_only` | `false` | When `run_upstream_blocks` is `true`, skip upstream blocks that already completed successfully |
>
> API runs execute pipelines from the current Mage repository. The legacy `project` option is no
> longer supported; send API-run requests to the workspace that owns the pipeline instead.
>
> ***
>
> ### Error Handling
>
> The endpoint raises exceptions directly rather than wrapping them in an error envelope.
>
> | Condition | Behaviour |
> | - | - |
> | Missing or invalid token | 401 Unauthenticated. The request is rejected before any pipeline lookup. |
> | Token does not match a trigger | 404 Resource Not Found |
> | `pipeline_uuid` not found | 404 Resource Not Found. UUIDs are case-sensitive. |
> | `block_uuid` not found | 404 Resource Not Found. The block must exist within the specified pipeline. |
> | Block throws an exception | The exception propagates to the HTTP response. Database records are marked `FAILED` when `record` is `true`. |
>
> ***
>
> ### Audit Trail
>
> When `record` is `true`, Mage writes a `PipelineRun` and `BlockRun` record to the database on every call. You can see these in the Mage Pro UI under pipeline run history. Records are marked `COMPLETED` or `FAILED` depending on the outcome.
>
> If you are calling this endpoint at high frequency and do not need the audit log, set `record` to `false` to skip the database writes.

## **Base URL**

**`https://<your-cluster-hostname>/api`**

For the full list of cluster API endpoints, see the [**Mage Open Source API Reference**](https://docs.mage.ai/extensibility/api-reference/overview).

> Note: Mage Pro may expose additional admin or authentication APIs depending on your deployment and licensing tier.


# Pipeline variables and keyword arguments
Source: https://docs.mage.ai/getting-started/runtime-variable



Runtime Variables are a set of global variables that can be used by every block.
These are useful for storing constants shared by multiple blocks or constants
whose value is determined at pipeline runtime (hence *runtime* variables).

## Using Runtime Variables

Runtime Variables can be accessed via the `**kwargs` parameter in your block
function.

```python theme={"system"}
from pandas import DataFrame
from os import path

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def loader_data(**kwargs) -> DataFrame:
    filepath = kwargs.get('filepath')
    row_limit = kwargs.get('row_limit')
    return pd.read_csv(filepath, nrows=row_limit)
```

Currently, runtime variables must be of primitive Python types or some basic
containers:

* integer
* string
* float
* boolean
* list
* dictionary
* set

Runtime variable names must be a valid Python identifier (i.e., no spaces in
name, can't start with a number).

## Creating Runtime Variables

### In Mage editor

You can create new global variables from the Mage UI through the "Variables" tab
in the sidekick. Click the "New" button, and configure your variable's name and
value, and press Enter to save.

To edit global variables, hover over the variable and click on the edit button.
You can edit the variable name and/or value, and press Enter to save.

![Create Global Variables](https://mage-ai.github.io/assets/configure-runtime-variables.png)

### In code

You can also create pipeline-level variables in code by editing the pipeline's `metadata.yaml` file.

![Create Pipeline Variables](https://mage-ai.github.io/assets/configure-variables-in-code.png)

If you'd like to create environment specific pipeline variables, you can format the `variables` arguments as follows:

```yaml theme={"system"}
variables:
   Sharedvar1: value1
   sharedvar2: value2
   dev:
      var: value1
   staging:
      var: value2
   prod:
      var: value3
```

This create dictionaries of runtime variables as the blocks are executed.

In your execution environment, you may then set variables for each environment `ENV=dev, ENV=staging, ENV=prod` which can be leveraged to indicate the execution environment.

A simple python function can then be used to access the variables:

```python theme={"system"}
import os

def testfunc(**kwargs):
    env = os.environ.get('ENV')
    myyvar = kwargs[env][var]
```

A big thanks to community member [Preston Dotsey](https://mageai.slack.com/team/U061CMVAJH1) for their contribution!

### In Python code

You can also save the variable in Python code. One example usage is to save checkpoint data.

```python theme={"system"}
from mage_ai.data_preparation.variable_manager import set_global_variable
set_global_variable(pipeline_uuid, key, value)
```

## Default Runtime Variables

Mage provides some default variables to give context on the pipeline execution.

* `execution_date`: A datetime object that the pipeline is executed at.
* `event`: If the pipeline is triggered by event, the `event` variable contains
  the event payload.

## Running Pipeline with Runtime Variables

### Run from command line

You can execute your pipeline with runtime variables from the command line.
First, make sure you installed the package.

```bash theme={"system"}
pip install mage-ai
```

Once the package is installed, you can run your pipeline through the command
line.

```bash theme={"system"}
mage run <project_path> <pipeline_uuid> --runtime-vars '{"name": "value"}'...

# example with 2 runtime variables "name" and "ds":
mage run default_repo default_pipeline --runtime-vars '{"name": "default", "ds": "2022-08-18"}'
```

### Run from Python script

If your pipeline is configured to use runtime variables, you can still execute
your pipeline outside the code editor. Provide the runtime variables as keyword
arguments to `mage_ai.run()`:

```python theme={"system"}
mage_ai.run('sample_pipeline', 'repos/default_repo', filepath = 'path/to/my/file.csv', row_limit=1000)
```

## Example - Aggregating Daily Logs

A common use of ETL pipelines is to process and analyze daily events. In the
case of this example, we will create an ETL pipeline to analyze log messages
from a web application. Suppose the following is an example of a log file that
our web application produces.

| log\_date | type | source |
| - | - | - |
| 2022-07-27T20:19:20 | INFO | react-1 |
| 2022-07-27T19:18:45 | WARNING | express-2 |
| 2022-07-27T16:35:28 | DEBUG | react-1 |
| 2022-07-27T10:19:32 | INFO | express-1 |
| 2022-07-27T07:20:26 | ERROR | express-1 |
| 2022-07-27T00:42:37 | ERROR | react-1 |

Suppose we want to know the distribution of log types at the end of every day.
Using Mage's runtime variables this is made a very simple task:

1. Create a data loader to load all log files modified on a specific date. We
   will specify the log folder and the date to load logs from using **runtime
   variables** which are passed to this block via the `**kwargs` parameter.

   ```python theme={"system"}
   from datetime import datetime, timedelta, date
   from pandas import DataFrame, concat, read_csv
   from pathlib import Path
   import os

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader


   @data_loader
   def load_log_data(**kwargs) -> DataFrame:
       start = datetime.fromisoformat(kwargs.get('current_date'))
       end = start + timedelta(days=1)
       logpath = Path(kwargs.get('log_folder'))
       logs = []
       for file in logpath.iterdir():
           print(file)
           modification_time = datetime.fromtimestamp(os.path.getmtime(file))
           if modification_time >= start and modification_time < end:
               df = read_csv(file)
               logs.append(df)
       return concat(logs, axis=0)
   ```

   *Note*: This code ignores the edge case of a log file that spills over
   between days. Since the modification date is used instead of the creation
   date, a log file that is modified between days will only be considered in the
   latter day.

2. Calculate the distribution of log types over this date

   ```python theme={"system"}
   from pandas import DataFrame
   from os import path
   import pandas as pd

   if 'transformer' not in globals():
       from mage_ai.data_preparation.decorators import transformer


   @transformer
   def extract_statistics(df: DataFrame, **kwargs) -> DataFrame:
       now = kwargs.get('current_date')
       count = df['type'].value_counts()
       count = DataFrame({now: count}).T
       return count
   ```

   The result of this transformer is a new data frame that looks like below:

   | | DEBUG | ERROR | INFO | WARNING |
   | - | - | - | - | - |
   | 2022-07-28 | 816 | 765 | 828 | 871 |

This data can then be ingested into a log statistics database.

**Key:** As the current date and log folder are not hardcoded in the pipeline
but instead provided as a runtime variable, your pipeline code remains reusable
without having to change any code. Every day, the pipeline can be ran using
`mage_ai.run()`, providing the current date and log folder as keyword arguments:

```python theme={"system"}
mage_ai.run('log_stats_ingestion', 'repos/default_repo', current_date='2022-07-29', log_folder='logs/webapp')
```

## Example - Model Rockets

Consider the following sample data tracking the launch angle (in degrees) and
vertical velocity (in meters per second) of model rocket tests:

| launch\_id | angle | vertical\_velocity |
| - | - | - |
| 1 | 28.4 | 61.34 |
| 2 | 14.4 | 32.03 |
| 3 | 61.4 | 113.19 |
| 4 | 44.2 | 89.96 |
| 5 | 39.5 | 82.00 |
| 6 | 79.3 | 126.73 |
| 7 | 74.9 | 124.51 |
| 8 | 38.1 | 79.6 |

Suppose we want to convert the launch angle from degrees to radians in our
pipeline. The example below uses the `pi` runtime variable (passed in through
`**kwargs`) to convert the degrees column of some input data frame to a radians.
This allows us to control the precision with which we want to store `pi`.

```python theme={"system"}
from pandas import DataFrame
from os import path

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


@transformer
def convert_to_radians(df: DataFrame, **kwargs) -> DataFrame:
    pi = kwargs.get('pi')
    df['angle_radians'] = df['angle'] * pi / 180
    return df
```

Suppose also want to convert the velocity from meters per second to kilometers
per hour by multiplying by the constant 3.6. Using the runtime variable
`conversion_factor` (again passed in through `**kwargs`), the vertical velocity
in kilometers per hour can be computed:

```python theme={"system"}
from pandas import DataFrame
from os import path

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


@transformer
def convert_velocity(df: DataFrame, **kwargs) -> DataFrame:
    conversion_factor = kwargs.get('conversion_factor')
    df['converted_vertical_velocity'] = df['vertical_velocity'] * conversion_factor
    return df
```

Now that your pipeline is complete, you can run your pipeline using
`mage_ai.run()`. Specify values for runtime variables as keyword arguments to
this function:

```python theme={"system"}
mage_ai.run('model_rocket_data_ingestion', 'repos/default_repo', pi=3.1415, conversion_factor=3.6)
```

Suppose after creating your pipeline, you instead want to store your velocity in
miles per hour instead of kilometers per hour. As the conversion factor is a
runtime variable, you don't have to edit your pipeline - you just have to change
the conversion factor which your pipeline is run with!

```python theme={"system"}
mage_ai.run('model_rocket_data_ingestion', 'repos/default_repo', pi=3.1415, conversion_factor=2.24)
```


# Quickstart
Source: https://docs.mage.ai/getting-started/setup

Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. 🦸‍♀️

## Mage Pro overview

After creating a Mage Pro account or logging into Mage Pro you are directed to the cluster management portal. Here you can manage
your Mage clusters.

### Cluster Pages

1. **Creating a New Cluster**:
   * On the clusters page, you can create a new cluster by clicking the "Launch new cluster" button.
   * Fill in the required field **Cluster Name**. Note that environment variables can be customized after the cluster is created.

![](https://mage-ai.github.io/assets/getting-started/mage-pro/create-new-cluster.png)

> [!NOTE]
>
> If you are in a free trial you are limited to a single cluster.

2. **Viewing Cluster List**:
   * The cluster list page provides an overview of all your clusters, displaying essential information such as the cluster status, ID, UUID, cloud provider, and the number of replicas.
3. **Cluster Details**:
   * Click the "Open cluster" button to view detailed information about the cluster. Here, you can see various metrics and statuses that help you monitor and manage your cluster effectively.
   * Follow the [environment variables](/development/variables/environment-variables) documentation to customize your Mage Pro cluster.

### Cluster login

1. **Accessing the cluster**:
   * Click the "Open cluster" button to navigate to the actual Mage cluster page. Here, you can interact with your cluster and use the Mage tool as you normally would.
   * Currently, we use separate databases for each cluster for enhanced security. The users credentials in the Mage Pro cluster are managed inside the cluster. Use the default owner credentials (email: [admin@admin.com](mailto:admin@admin.com), password: **admin**) to log in. After you login, you should either update the password of the default owner user or create your own owner user.

> [!NOTE]
>
> Credentials may differ from cluster portal. For initial setup use Email or username: [admin@admin.com](mailto:admin@admin.com), Password: admin.

### Run your first pipeline

* **Step 1**: From the pipeline editor page click the "New pipeline" button
* **Step 2**: Select a template for inspiration or start from scratch by clicking the "Start from scratch" button
* **Step 3**: Select your pipeline type, enter a unique name, and optionally add a description and tags
* **Step 4**: Click the **Create pipeline** button
* **Step 5**: Add your blocks by following the Youtube video below

[YouTube video player](https://www.youtube.com/embed/VFFzJFZjuAg?si=GxWtS3f-ovvfw7T1)

## ⛵️ Mage OSS overview

We recommend using Docker to get started.

Docker is a tool that allows you to run Mage in a containerized environment: you can run Mage on any operating system that supports Docker, including Windows, Mac, and Linux. Using Docker means that you don't have to worry about installing dependencies or configuring your environment. If you'd like to install Mage without Docker, you can use `pip` or `conda`.

If you're familiar with [Docker Compose](https://docs.docker.com/compose/) or plan on adding or extending images (e.g. Postgres) in your project, we recommend starting from the *Docker compose template*. Otherwise, we recommend *Docker run*.

## 🪄 Get Mage

> **Docker**
>
> ### Get started
>
> First, be sure [Docker](https://docs.docker.com/engine/install/) is installed. Next, create a new folder for your project, change directory into that folder, and run the following from your terminal:
>
> ```bash Mac/Linux theme={"system"}
> docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]
> ```
>
> ```bash Windows Command Line theme={"system"}
> docker run -it -p 6789:6789 -v "%cd%:/home/src" mageai/mageai /app/run_app.sh mage start [project_name]
> ```
>
> ```bash PowerShell theme={"system"}
> docker run -it -p 6789:6789 -v ${PWD}:/home/src mageai/mageai /app/run_app.sh mage start [project_name]
> ```
>
> ```bash GCP VM Instance theme={"system"}
> sudo docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]
> ```
>
> Once the server is running, open `http://localhost:6789` in your browser and explore! If using any cloud provider use external IP address with the port eg 123.456.78:6789. We recommend one of our [guides](/guides/overview) for inspiration.
>
> ### What's happening?
>
> We're executing `docker run`, which runs a Docker container from the Mage image. additionally, we're *mounting a volume* (the `-v` command) to persist our project files on our container. This mounts the current folder to `/home/src` in the mage container. We're executing our docker image `mageai/mageai` and running a script to start up the container.

> **Docker Compose**
>
> ### Get started
>
> The following is useful if you plan on customizing your image or adding additional services to a Docker network with Mage. Read more about Docker compose [here](https://docs.docker.com/compose/).
>
> Be sure that [Docker](https://docs.docker.com/engine/install/) and [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) are installed and run:
>
> ```bash Mac/Linux theme={"system"}
> git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart \
> && cd mage-quickstart \
> && cp dev.env .env && rm dev.env \
> && docker compose up
> ```
>
> ```bash Windows theme={"system"}
> git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart
> cd mage-quickstart
> cp dev.env .env
> rm dev.env
> docker compose up
> ```
>
> ```Windows powershell/cmd theme={"system"}
> git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart
> cd mage-quickstart
> copy dev.env .env
> del dev.env
> docker compose up
> ```
>
> Once the server is running, open `http://localhost:6789` in your browser and explore! We recommend one of our [guides](/guides/overview) to for inspiration.
>
> ### What's happening?
>
> We're cloning our Docker Compose [quickstart repo](https://github.com/mage-ai/compose-quickstart.git), copying `dev.env` to `.env`, which is ignored by git in our configuration (to avoid exposing secrets), then running `docker compose up` to start a Mage server.

> **pip/conda**
>
> **1. Install Mage**
>
> ```bash pip theme={"system"}
> pip install mage-ai
> ```
>
> ```bash conda theme={"system"}
> conda install -c conda-forge mage-ai
> ```
>
> <sub>If you're installing on Linux and need help, check out this [community provided video walkthrough](https://www.youtube.com/watch?v=WVH7LHqND1I).</sub>
>
> **2. Create new project and launch tool**
>
> ```bash theme={"system"}
> mage start [project_name]
> ```
>
> **3. Open Mage**
>
> Open [http://localhost:6789](http://localhost:6789) in your browser.

> **Kubernetes**
>
> **1. Start a Kubernetes cluster locally**
>
> [Enable Kubernetes](https://docs.docker.com/desktop/kubernetes/#enable-kubernetes) in Docker Desktop to start a Kubernetes cluster locally. Other options for starting a Kubernetes cluster locally are [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and [Minikube](https://minikube.sigs.k8s.io/docs/start/)
>
> **2. Download and update Kubernetes config file**
>
> First, download the Mage Kubernetes config file [here](https://github.com/mage-ai/mage-ai/blob/master/kube/app.yaml).
> Then, replace the `/path/to/mage_project` in the config yaml with the path that you want to use to store your Mage projects.
>
> **3. Run Mage app in Kubernetes cluster**
>
> Install the command line tool, [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), if you haven't already.
> Run the command `kubectl create -f kube/app.yaml` to run Mage in a Kubernetes pod. You can check the pod status with command `kubectl get pods -o wide`.
> Set up port forwarding with command `kubectl port-forward mage-server 6789:6789`.
>
> **4. Open Mage**
>
> Open [http://localhost:6789](http://localhost:6789) in your browser.

> **Cloud**
>
> Our fully managed solution for teams is now available!
>
> [Try Mage Pro for free](https://cloud.mage.ai/sign-up?source=setup)
>
> [Open](https://cloud.mage.ai/sign-up?source=setup)

## Default login credentials

Starting with **Mage OSS v0.9.78 and above**, and in **Mage Pro**,
**user authentication is enabled by default**.

When you first start Mage, the default login credentials are:

| Field | Value |
| - | - |
| Email | `admin@admin.com` |
| Password | `admin` |

⚠️ **Important:** After logging in for the first time, we strongly recommend changing the default password or creating new user accounts for your team. See [User authentication](/production/authentication/overview) for more details.

## 🏃‍♂️ Run your first pipeline

If you haven't already, open a browser to `http://localhost:6789`. From the pipelines page, select `example_pipeline` and open the notebook view by selecting `Edit pipeline` from the left side nav.

![](https://mage-ai.github.io/assets/quickstart/open-pipeline.gif)

Select the first block by clicking it and select the "play" icon in the top right to run the block. You've just ran your first Mage block & loaded data from a dataset!

> **🤔 What's a block?**
>
> <Snippet />

Do the same for the following cells in the pipeline to transform and export the data. Congrats, you're now a Mage ninja! 🥷

## 🧙🏻‍♂️ Install Mage dependencies (optional)

Mage also has the following add-on packages:

| Package | Install | Description |
| - | - | - |
| all | `mage-ai[all]` | install all add-ons |
| azure | `mage-ai[azure]` | install Azure related packages |
| clickhouse | `mage-ai[clickhouse]` | use Clickhouse for data import or export |
| dbt | `mage-ai[dbt]` | install dbt packages |
| google-cloud-storage | `mage-ai[google-cloud-storage]` | use Google Cloud Storage for data import or export |
| hdf5 | `mage-ai[hdf5]` | process data in HDF5 file format |
| mysql | `mage-ai[mysql]` | use MySQL for data import or export |
| postgres | `mage-ai[postgres]` | use PostgreSQL for data import or export |
| redshift | `mage-ai[redshift]` | use Redshift for data import or export |
| s3 | `mage-ai[s3]` | use S3 for data import or export |
| snowflake | `mage-ai[snowflake]` | use Snowflake for data import or export |
| spark | `mage-ai[spark]` | use Spark (EMR) in your Mage pipeline |
| streaming | `mage-ai[streaming]` | use Streaming pipelines |

To install these, run the following command from the Mage terminal:

```bash theme={"system"}
pip install "mage-ai[spark]"
```

or add the following to your `requirements.txt` file:

```
mage-ai[spark]
```

You can access the terminal from the side nav on the right in the pipeline editor page. Read more about installing from `requirements.txt` [here](/development/dependencies/requirements).

![](https://mage-ai.github.io/assets/terminal.png)

## 🧭 Journey on

Navigate to our [tutorials](/guides/overview) to learn more about Mage and how to build your own pipelines or continue exploring our docs for advanced configuration and deployment options.

If you're interested in connecting a database in Docker, check out our [guide](/guides/docker/connecting-a-database) for more information.


# Mage Pro pipeline trigger
Source: https://docs.mage.ai/getting-started/trigger-pipeline

Trigger your first Mage Pro data pipeline

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=pipeline-triggers)

Pipeline automation is a critical component of production data workflows It ensures consistent and reliable execution without manual intervention. Mage Pro provides comprehensive orchestration capabilities through its trigger UI or YAML conde configuration, which initiates pipeline execution based on predefined conditions or external events.

Mage supports two primary orchestration patterns for pipeline automation:

**1. Schedule-Based Orchestration**

Schedule triggers provide time-based automation capabilities with flexible configuration options:

* **Continuous Execution**: Maintains persistent pipeline runs for real-time data processing scenarios
* **Interval-Based Execution**: Configures pipelines to run at specified intervals (hourly, daily, weekly, etc.)
* **One-Time Execution**: Schedules pipelines for single execution at a designated time
* **Custom Timeline Configuration**: Enables complex scheduling patterns with custom cron expressions
* **Deadline Management**: Sets completion time constraints to ensure pipelines finish within specified windows

This method is optimal for batch processing workflows, ETL operations, and scenarios requiring predictable execution patterns.

**2. API-Driven Orchestration**

API triggers enable event-driven pipeline execution through HTTP endpoints:

* **RESTful Integration**: Exposes pipeline execution through standardized POST requests
* **External System Integration**: Allows third-party applications to trigger pipeline runs
* **Programmatic Control**: Enables dynamic pipeline orchestration through application logic

This approach is ideal for real-time processing scenarios, webhook integrations, and microservice architectures requiring on-demand pipeline execution.

**Let’s create a new schedule based trigger:**

**Step 1**: From the pipeline editor, non-viewer users can select **Run pipeline** in the header or **Run** menu to open the Triggers page. Viewer users and users in compact layouts should use the existing left popout navigation menu and select **Triggers**.

**Step 2**: Once on the triggers page click the green “New trigger” button

![Create new pipeline](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/new-trigger-button.png)

**Step 3**: After clicking “New Trigger,” you will be taken to a screen where you can create a trigger. Select the “Schedule” option and the UI pictured below ill populate.

**Step 4**: Complete the text fields below to configure your trigger based on the workflows requirements. You can also configure Run settings, SLA requirements, and Runtime variables in this UI.

**Step 5**: Click Create trigger to save your trigger.

![Create new pipeline](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/trigger-page.png)

**Step 6**: After saving your changes you will be taken to a new screen where you can enable your trigger. Once enabled, the trigger will begin to run based on your previous configuration.

**Step 7**: Click the “Save trigger in code” button to save your trigger to a YAML file.

![Create new pipeline](https://raw.githubusercontent.com/mage-ai/assets/main/pro/cloud/enable-trigger.png)

Once your trigger is enabled, **Frequency** is the trigger plan: it controls the run schedule and actual run time. **Schedule start** only controls when that plan is allowed to begin. Choose **Start at next Frequency time** to let the trigger run at the next scheduled **Frequency** time, or choose **Do not start before date/time** to set the earliest allowed start. That date/time is not the run time; **Frequency** still controls when runs happen.

You have successfully automated your first Mage pipeline. This foundational automation capability enables consistent, reliable data processing without manual intervention. You can now apply these same principles to build additional automated workflows for your data infrastructure.


# Batch pipeline tutorial
Source: https://docs.mage.ai/guides/batch-pipeline



> **[ETL pipeline](/guides/load-api-data)**
>
> Load, transform, and export restaurant data


# Best Practices
Source: https://docs.mage.ai/guides/best-practices

Learn best practices for building, maintaining, and scaling data pipelines in Mage Pro.

This guide provides comprehensive best practices for building, maintaining, and scaling data pipelines in Mage Pro. Following these practices will help you create reliable, performant, and maintainable data engineering solutions.

## Table of Contents

* [Block Design & Architecture](#block-design-architecture)
* [Pipeline Organization](#pipeline-organization)
* [Performance Optimization](#performance-optimization)
* [Testing & Validation](#testing-validation)
* [Error Handling & Reliability](#error-handling-reliability)
* [Resource Management](#resource-management)
* [Collaboration & Version Control](#collaboration-version-control)
* [Monitoring & Observability](#monitoring-observability)
* [Security & Governance](#security-governance)
* [Deployment & CI/CD](#deployment-cicd)

***

## Block Design & Architecture

### Single Responsibility Principle

Each block should have a single, well-defined purpose. This makes blocks easier to understand, test, and maintain.

**✅ Good:**

```python theme={"system"}
# Block: load_customer_data
def load_customer_data():
    """Load customer data from source database."""
    return pd.read_sql("SELECT * FROM customers", connection)

# Block: transform_customer_data
def transform_customer_data(df):
    """Clean and standardize customer data."""
    df = df.dropna(subset=['email'])
    df['email'] = df['email'].str.lower()
    return df
```

**❌ Avoid:**

```python theme={"system"}
# Block: load_and_transform_customers
def load_and_transform_customers():
    """Loads, transforms, and exports customer data."""
    # Loading, transforming, and exporting in one block
    df = pd.read_sql("SELECT * FROM customers", connection)
    df = df.dropna(subset=['email'])
    df.to_parquet('output.parquet')
```

### Modularity & Reusability

Design blocks to be reusable across pipelines and projects. Use clear, descriptive names and avoid hardcoded values.

**✅ Good:**

```python theme={"system"}
# Block: load_data_from_source
def load_data_from_source(source_table: str, connection_string: str):
    """Generic data loader that can be reused for different tables."""
    return pd.read_sql(f"SELECT * FROM {source_table}", connection_string)
```

**❌ Avoid:**

```python theme={"system"}
# Block: load_customers_jan_2024
def load_customers_jan_2024():
    """Hardcoded for specific use case."""
    return pd.read_sql("SELECT * FROM customers WHERE month='2024-01'", conn)
```

### Block Naming Conventions

Use clear, descriptive names that indicate the block's purpose:

* **Data loaders**: `load_<source>_<entity>` (e.g., `load_salesforce_contacts`)
* **Transformers**: `transform_<action>_<entity>` (e.g., `transform_clean_customer_data`)
* **Data exporters**: `export_<entity>_to_<destination>` (e.g., `export_customers_to_warehouse`)

### Organizing Blocks in Subdirectories

Organize blocks in subdirectories within their respective block type folders to improve project structure and maintainability.

**✅ Benefits:**

* Better organization for large projects
* Easier to find related blocks
* Clearer project structure
* Supports team collaboration

**How to Organize:**

1. **Create subdirectories by category:**
   * By source type: `api/`, `database/`, `file/`, `cloud/`
   * By domain: `sales/`, `marketing/`, `finance/`
   * By function: `validation/`, `cleaning/`, `aggregation/`

2. **Using the UI:**
   * When creating a block, name it with the subdirectory: `subfolder_name/block_name`
   * Example: `api/load_stripe` creates `data_loaders/api/load_stripe.py`
   * The subdirectory is automatically created if it doesn't exist

3. **Example structure:**
   ```
   data_loaders/
   ├── api/
   │   ├── load_stripe.py
   │   ├── load_salesforce.py
   │   └── load_hubspot.py
   ├── database/
   │   ├── load_postgres.py
   │   └── load_mysql.py
   └── file/
       ├── load_csv.py
       └── load_parquet.py

   transformers/
   ├── cleaning/
   │   ├── remove_duplicates.py
   │   └── standardize_format.py
   └── aggregation/
       ├── calculate_metrics.py
       └── group_by_category.py
   ```

4. **Best practices:**
   * Use consistent subdirectory naming across block types
   * Keep subdirectory depth to 1-2 levels maximum
   * Group related blocks together logically
   * Document your organization structure for your team

For more details, see the [Project Structure Guide](/design/abstractions/project-structure).

### Block Size Guidelines

Keep blocks focused and reasonably sized:

* **Ideal**: 50-200 lines of code per block
* **Maximum**: 500 lines (consider splitting if larger)
* **Minimum**: Enough code to accomplish one clear task

### Use Block Templates

Create reusable block templates for common patterns:

* Standard ETL patterns
* Data validation patterns
* Error handling templates
* Logging templates

This ensures consistency across your team and reduces duplicate code.

***

## Pipeline Organization

### Choosing the Right Pipeline Type

Mage supports three main pipeline types, each optimized for different use cases. Understanding when to use each type is crucial for building efficient and maintainable data pipelines.

#### Data Integration Pipelines

**Use data integration pipelines when:**

* You need to **sync data between systems** (external to internal, or internal to external)
* You're working with **pre-built connectors** for SaaS tools, databases, or data warehouses
* You want **low-code/no-code** data synchronization
* You need **incremental sync** capabilities with automatic state tracking
* You're doing **reverse ETL** (syncing data from your warehouse to external systems)

**Characteristics:**

* Uses Singer spec-compliant connectors
* Handles schema migrations automatically
* Supports incremental and full sync modes
* Minimal code required - mostly configuration
* Can be combined with batch pipelines for custom transformations

**Example use cases:**

* Syncing Salesforce contacts to your data warehouse
* Loading Stripe transaction data into BigQuery
* Exporting customer segments from Snowflake to HubSpot
* Syncing product data from PostgreSQL to MongoDB

**Learn more:** [Data Integration Overview](/data-integrations/overview)

#### Batch Pipelines

**Use batch pipelines when:**

* You need **scheduled data processing** (hourly, daily, weekly)
* You require **custom transformations** using Python, SQL, or R
* You're building **ETL/ELT workflows** with complex business logic
* You need to **process large volumes** of historical data
* You want to **combine multiple data sources** and apply custom logic
* You're building **analytics pipelines** for reporting and dashboards

**Characteristics:**

* Runs on a schedule, completes, and exits
* Full flexibility with custom code blocks
* Can combine data integration blocks with custom transformers
* Supports backfilling historical data
* Ideal for data warehousing and analytics workloads

**Example use cases:**

* Daily revenue reporting with complex aggregations
* Monthly financial reconciliation
* Customer segmentation analysis
* Data quality checks and validation
* Feature engineering for machine learning

**Learn more:** [Building Batch Pipelines](/getting-started/build-pro-pipeline)

#### Streaming Pipelines

**Use streaming pipelines when:**

* You need **real-time data processing** (sub-second to second latency)
* You're processing **continuous data streams** (Kafka, Kinesis, etc.)
* You require **immediate insights** or alerts
* You're building **real-time analytics** or monitoring systems
* You need **event-driven architectures**
* Low latency is critical (milliseconds to seconds)

**Characteristics:**

* Long-running processes that continuously monitor sources
* Processes data as it arrives (not in batches)
* Event-driven architecture
* Lower latency than batch processing
* Supports stateful processing with window aggregations

**Example use cases:**

* Real-time fraud detection
* Live dashboard updates
* IoT sensor data processing
* Real-time recommendation systems
* Event-driven data transformations
* Real-time data quality monitoring

**Learn more:** [Streaming Pipeline Introduction](/guides/streaming/introduction)

#### Decision Matrix

| Requirement | Data Integration | Batch | Streaming |
| - | - | - | - |
| **Latency** | Minutes to hours | Minutes to hours | Milliseconds to seconds |
| **Code Complexity** | Low (mostly config) | High (custom code) | Medium (transformers) |
| **Use Case** | Data sync | Scheduled processing | Real-time processing |
| **Data Volume** | Small to large | Large | Continuous streams |
| **Custom Logic** | Limited | Full flexibility | Transformers only |
| **State Management** | Automatic | Manual | Automatic (optional) |

**Hybrid Approach:**
You can combine pipeline types:

* Use **data integration pipelines** to sync data into your warehouse
* Use **batch pipelines** to transform and aggregate the synced data
* Use **streaming pipelines** for real-time processing of critical events

### Pipeline Structure

Organize pipelines logically:

1. **Data Integration Pipelines**: Extract and load data from sources
2. **Transformation Pipelines**: Transform and clean data
3. **Analytics Pipelines**: Generate reports and metrics
4. **Orchestration Pipelines**: Coordinate multiple pipelines

### Pipeline Naming

Use consistent naming conventions:

* **Format**: `<domain>_<purpose>_<frequency>` (e.g., `sales_daily_revenue_report`)
* **Examples**:
  * `marketing_campaign_attribution_hourly`
  * `finance_monthly_reconciliation`
  * `product_user_behavior_realtime`

### Pipeline Tagging

Use tags to organize and group related pipelines for easier discovery and management.

When creating a pipeline in Mage Pro, type in the **Tags** field to match existing project tags for the current comma-separated entry, or keep typing to add a new tag.

**✅ Best Practices for Tagging:**

* **Use consistent tag names**: Establish a tagging convention across your team
  * Domain tags: `sales`, `marketing`, `finance`, `product`
  * Type tags: `etl`, `analytics`, `reporting`, `integration`
  * Environment tags: `production`, `staging`, `development`
  * Priority tags: `critical`, `high`, `low`

* **Apply multiple tags**: Pipelines can have multiple tags for flexible grouping
  ```python theme={"system"}
  # Example: A pipeline might have tags: ['sales', 'etl', 'production', 'critical']
  ```

* **Group by tags**: Use the pipeline list view to group and filter by tags
  * In Mage Pro, right-click a pipeline and select "Update tags"
  * Type a new tag, select matching existing tags, or remove selected tags in the modal, then click "Save"
  * Filter and group pipelines by tag on the Pipeline page

* **Tag naming conventions:**
  * Use lowercase with underscores: `data_quality`, `customer_analytics`
  * Keep tags short and descriptive; each tag name can contain up to 70 characters
  * Avoid special characters and spaces

**Benefits of Tagging:**

* Quickly find related pipelines
* Filter pipelines by domain, type, or environment
* Organize pipelines for different teams or projects
* Group pipelines for monitoring and alerting

For more details, see the [Pipeline Tagging Guide](/pipelines/pipeline-tagging).

### Limit Pipeline Complexity

**✅ Recommended:**

* **Block count**: 10-30 blocks per pipeline
* **Depth**: Maximum 5-7 levels of dependencies
* **Branches**: Limit parallel branches to 5-10

**⚠️ If your pipeline exceeds these limits:**

* Split into multiple pipelines
* Use orchestration pipelines to coordinate
* Extract common logic into reusable blocks

### Dependency Management

* **Minimize dependencies**: Only create dependencies where data actually flows
* **Avoid circular dependencies**: Design acyclic graphs
* **Use clear data contracts**: Document expected input/output schemas

***

## Performance Optimization

### Efficient Data Processing

**Batch Processing:**

```python theme={"system"}
# ✅ Process data in batches
def transform_large_dataset(df):
    batch_size = 10000
    results = []
    for i in range(0, len(df), batch_size):
        batch = df.iloc[i:i+batch_size]
        transformed = process_batch(batch)
        results.append(transformed)
    return pd.concat(results)
```

**Use Appropriate Data Types:**

```python theme={"system"}
# ✅ Optimize data types
df['id'] = df['id'].astype('int32')  # Instead of int64
df['category'] = df['category'].astype('category')  # Memory efficient
```

### Leverage Spark for Large Datasets

For datasets larger than memory or requiring distributed processing:

```python theme={"system"}
# ✅ Use Spark for large-scale transformations
from mage_ai.data_preparation.models.block import Block

@block
def transform_with_spark(df):
    spark_df = spark.createDataFrame(df)
    result = spark_df.groupBy('category').agg({'value': 'sum'})
    return result.toPandas()
```

### Optimize API Calls

**Batch API Requests:**

```python theme={"system"}
# ✅ Batch API calls
def load_data_from_api(ids):
    # Process in batches instead of one-by-one
    batch_size = 100
    results = []
    for i in range(0, len(ids), batch_size):
        batch = ids[i:i+batch_size]
        response = api.get_batch(batch)
        results.extend(response)
    return results
```

**Use Connection Pooling:**

```python theme={"system"}
# ✅ Reuse connections
from sqlalchemy import create_engine

# Create connection pool at module level
engine = create_engine('postgresql://...', pool_size=10)

def load_data():
    return pd.read_sql("SELECT * FROM table", engine)
```

### Memory Management

**Stream Large Files:**

```python theme={"system"}
# ✅ Stream processing for large files
def process_large_file(file_path):
    chunk_size = 10000
    for chunk in pd.read_csv(file_path, chunksize=chunk_size):
        process_chunk(chunk)
```

**Clean Up Resources:**

```python theme={"system"}
# ✅ Explicitly close connections and free memory
def load_data():
    conn = create_connection()
    try:
        df = pd.read_sql("SELECT * FROM table", conn)
        return df
    finally:
        conn.close()
        del df  # Help garbage collector
```

### Pipeline Performance

**Choose the Right Executor:**

* **`local_python` executor**: Use for faster execution when you have sufficient local resources. Blocks run in the same process or separate processes on the same machine, which reduces overhead and improves performance for smaller to medium-sized pipelines.

* **`k8s` executor**: Use for scalability when you need more resources or want to distribute block execution across multiple pods. Ideal for large-scale pipelines, resource-intensive transformations, or when you need to scale horizontally.

**Optimize Execution Mode:**

* **`run_pipeline_in_one_process: true`**: Speeds up pipeline execution by running all blocks in a single process. This reduces process creation overhead and improves performance for pipelines with many small blocks or when blocks need to share memory efficiently.

* **`run_pipeline_in_one_process: false`**: Runs blocks in separate processes (for `local_python` executor) or separate pods (for `k8s` executor). Use this when:
  * Blocks need isolation (e.g., different resource requirements)
  * You want to leverage parallel execution across multiple processes/pods
  * Memory isolation is important to prevent one block from affecting others

**Optimize Block Execution:**

* Use dynamic blocks for parallel processing
* Leverage conditional execution for optional steps
* Cache intermediate results when appropriate

**Control Concurrency:**

Configure concurrency limits to optimize resource usage and prevent system overload. You can set limits at both the global (project) and pipeline levels:

* **Global concurrency**: Limit the maximum number of concurrent block runs across all pipelines
* **Pipeline-level concurrency**: Control concurrent block runs and pipeline runs per trigger
* **Per-block concurrency**: Set specific concurrency limits for individual blocks

For more information, see the [Concurrency Control Guide](/design/data-pipeline-management#concurrency).

***

## Testing & Validation

### Write Tests in Blocks

Mage's built-in testing framework makes it easy to write tests alongside your code:

```python theme={"system"}
# ✅ Include tests in your blocks
def transform_data(df):
    # Transform logic
    df['normalized_value'] = df['value'] / df['max_value']
    return df

def test_output(output):
    """Test that transformation produces valid results."""
    assert output is not None, "Output should not be None"
    assert 'normalized_value' in output.columns, "Missing normalized_value column"
    assert (output['normalized_value'] <= 1.0).all(), "Values should be normalized"
    assert output['normalized_value'].notna().all(), "No null values allowed"
```

### Data Quality Checks

**Schema Validation:**

```python theme={"system"}
def validate_schema(df, expected_schema):
    """Validate DataFrame schema matches expected structure."""
    assert set(df.columns) == set(expected_schema.keys()), "Column mismatch"
    for col, dtype in expected_schema.items():
        assert df[col].dtype == dtype, f"Type mismatch for {col}"
```

**Data Quality Rules:**

```python theme={"system"}
def validate_data_quality(df):
    """Comprehensive data quality checks."""
    # Check for nulls in critical columns
    assert df['customer_id'].notna().all(), "No null customer IDs allowed"
    
    # Check value ranges
    assert (df['age'] >= 0).all() and (df['age'] <= 150).all(), "Invalid age range"
    
    # Check for duplicates
    assert df['email'].is_unique, "Duplicate emails found"
    
    # Check referential integrity
    assert df['status'].isin(['active', 'inactive', 'pending']).all(), "Invalid status values"
```

### Test-Driven Development

Write tests before or alongside implementation:

1. **Define expected behavior** in test functions
2. **Implement the logic** to pass tests
3. **Run tests** automatically on each block execution
4. **Refactor** while keeping tests green

***

## Error Handling & Reliability

### Graceful Error Handling

**Handle Expected Errors:**

```python theme={"system"}
# ✅ Handle errors gracefully
def load_data_from_api():
    try:
        response = api.get_data()
        return response
    except APIError as e:
        logger.error(f"API error: {e}")
        # Return empty DataFrame or cached data
        return pd.DataFrame()
    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        raise
```

**Validate Inputs:**

```python theme={"system"}
# ✅ Validate inputs early
def transform_data(df):
    if df is None or df.empty:
        raise ValueError("Input DataFrame is empty")
    
    required_columns = ['id', 'name', 'value']
    missing = set(required_columns) - set(df.columns)
    if missing:
        raise ValueError(f"Missing required columns: {missing}")
    
    # Continue with transformation
    return df
```

### Retry Logic

**Implement Retry for Transient Failures:**

You can implement retry logic in your code for handling transient failures, or use Mage's built-in automatic retry feature for block runs.

**Code-level retry:**

```python theme={"system"}
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
def load_data_with_retry():
    """Retry on transient failures."""
    return api.get_data()
```

**Automatic retry for block runs:**

Mage provides automatic retry functionality for failed block runs. This is especially useful for handling transient network issues, temporary API rate limits, or other recoverable errors. Configure automatic retry settings to automatically retry failed blocks without manual intervention.

For more information, see the [Automatic Retry Guide](/orchestration/pipeline-runs/retrying-block-runs#automatic-retry).

### Idempotency

Design blocks to be idempotent (safe to run multiple times):

```python theme={"system"}
# ✅ Idempotent block
def export_data(df):
    """Export data, safe to run multiple times."""
    # Use upsert instead of insert
    df.to_sql('table', engine, if_exists='replace', index=False)
    # Or use merge/upsert logic
    # merge_data(df, 'table', on='id')
```

### Checkpointing

For long-running pipelines, implement checkpointing:

```python theme={"system"}
# ✅ Save intermediate state
def process_large_dataset(df):
    checkpoint_file = 'checkpoint.pkl'
    
    # Load checkpoint if exists
    if os.path.exists(checkpoint_file):
        processed = pd.read_pickle(checkpoint_file)
        start_idx = len(processed)
    else:
        processed = []
        start_idx = 0
    
    # Process remaining data
    for i in range(start_idx, len(df)):
        result = process_row(df.iloc[i])
        processed.append(result)
        
        # Save checkpoint every 1000 rows
        if i % 1000 == 0:
            pd.DataFrame(processed).to_pickle(checkpoint_file)
    
    return pd.DataFrame(processed)
```

***

## Resource Management

### Compute Resources

**Right-Size Your Resources:**

* Use appropriate compute clusters for workload size
* Monitor CPU and memory usage in Cluster Manager
* Scale up for large transformations, scale down for simple tasks

**Use Spark for Large Datasets:**

* Enable Spark for datasets > 10GB
* Configure Spark resources appropriately
* Monitor Spark job performance

### Connection Management

**Reuse Connections:**

```python theme={"system"}
# ✅ Module-level connection (reused across runs)
import os
from sqlalchemy import create_engine

DATABASE_URL = os.getenv('DATABASE_URL')
engine = create_engine(DATABASE_URL, pool_size=5, max_overflow=10)

def load_data():
    return pd.read_sql("SELECT * FROM table", engine)
```

**Close Connections Properly:**

```python theme={"system"}
# ✅ Context manager for connections
def load_data():
    with create_connection() as conn:
        df = pd.read_sql("SELECT * FROM table", conn)
        return df
    # Connection automatically closed
```

### Memory Optimization

**Process Data in Chunks:**

```python theme={"system"}
# ✅ Chunk processing for large datasets
def process_large_file(file_path):
    chunk_size = 50000
    results = []
    
    for chunk in pd.read_csv(file_path, chunksize=chunk_size):
        processed = transform_chunk(chunk)
        results.append(processed)
        # Clear memory
        del chunk, processed
    
    return pd.concat(results)
```

**Use Efficient Data Structures:**

```python theme={"system"}
# ✅ Use appropriate data types
df['id'] = df['id'].astype('int32')  # 50% memory savings vs int64
df['category'] = df['category'].astype('category')  # Efficient for repeated values
df['date'] = pd.to_datetime(df['date'])  # Proper datetime type
```

***

## Collaboration & Version Control

### Use Workspaces Effectively

**Separate Environments:**

* **Development**: For experimentation and testing
* **Staging**: For pre-production validation
* **Production**: For live data pipelines

**Workspace Best Practices:**

* Use consistent naming across workspaces
* Document workspace purposes
* Limit production workspace access

### Version Control Integration

**Commit Frequently:**

* Commit after completing each block or feature
* Use descriptive commit messages
* Tag releases and important milestones

**Branch Strategy:**

* Use feature branches for new development
* Keep main/master branch stable
* Use pull requests for code review

**Git Best Practices:**

```bash theme={"system"}
# ✅ Good commit messages
git commit -m "feat: Add customer data transformation block"
git commit -m "fix: Resolve null handling in revenue calculation"
git commit -m "refactor: Extract common validation logic to utility"
```

### Code Review

**Review Checklist:**

* [ ] Code follows team standards
* [ ] Tests are included and passing
* [ ] Error handling is appropriate
* [ ] Documentation is clear
* [ ] Performance considerations addressed
* [ ] Security concerns reviewed

### Documentation

**Document Your Blocks:**

```python theme={"system"}
def transform_customer_data(df):
    """
    Transform customer data by standardizing formats and cleaning values.
    
    Args:
        df: DataFrame with customer data containing columns:
            - customer_id: Unique customer identifier
            - email: Customer email address
            - name: Customer full name
            - created_at: Account creation timestamp
    
    Returns:
        DataFrame with transformed customer data:
            - customer_id: Same as input
            - email: Lowercase, validated email addresses
            - first_name: Extracted from name
            - last_name: Extracted from name
            - created_at: Parsed datetime object
    
    Raises:
        ValueError: If required columns are missing
    """
    # Implementation...
```

**Pipeline Documentation:**

* Document pipeline purpose and business logic
* Include data flow diagrams
* Note dependencies and requirements
* Document expected run times and resource needs

***

## Monitoring & Observability

### Logging

**Use Structured Logging:**

```python theme={"system"}
import logging

logger = logging.getLogger(__name__)

def transform_data(df):
    logger.info(f"Starting transformation for {len(df)} rows")
    
    try:
        result = perform_transformation(df)
        logger.info(f"Transformation completed successfully. Output: {len(result)} rows")
        return result
    except Exception as e:
        logger.error(f"Transformation failed: {e}", exc_info=True)
        raise
```

**Log Key Metrics:**

```python theme={"system"}
def process_data(df):
    start_time = time.time()
    logger.info(f"Processing {len(df)} records")
    
    result = transform(df)
    
    duration = time.time() - start_time
    logger.info(f"Processed {len(result)} records in {duration:.2f}s")
    logger.info(f"Processing rate: {len(result)/duration:.0f} records/sec")
    
    return result
```

### Monitoring Pipeline Health

**Set Up Alerts:**

* Pipeline failure alerts
* Performance degradation alerts
* Data quality threshold alerts
* Resource usage alerts

**Monitor Key Metrics:**

* Pipeline run duration
* Block execution times
* Data volume processed
* Error rates
* Resource utilization

### Data Quality Monitoring

**Track Data Quality Metrics:**

```python theme={"system"}
def validate_and_log_quality(df):
    """Validate data and log quality metrics."""
    metrics = {
        'row_count': len(df),
        'null_percentage': df.isnull().sum() / len(df) * 100,
        'duplicate_count': df.duplicated().sum(),
    }
    
    logger.info(f"Data quality metrics: {metrics}")
    
    # Alert if quality thresholds exceeded
    if metrics['null_percentage'].max() > 10:
        logger.warning(f"High null percentage detected: {metrics['null_percentage']}")
    
    return df
```

***

## Security & Governance

### Authentication

**User Authentication is Enabled by Default:**

* **Mage Pro**: User authentication is **always enabled by default** (`REQUIRE_USER_AUTHENTICATION=True`)
* **Mage OSS (0.9.78+)**: User authentication is **enabled by default**
* **Mage OSS (0.8.4-0.9.77)**: Can be enabled by setting `REQUIRE_USER_AUTHENTICATION=1`

Authentication ensures that only authorized users can access the Mage UI and APIs. All users must sign in before creating, managing, or running pipelines.

**IP Whitelisting (Recommended for SaaS and Self-Hosted Mage Pro):**

For additional security, especially for SaaS or self-hosted Mage Pro deployments, it's recommended to set up an IP whitelist to restrict access to your Mage instance.

**Implementation options:**

* **Load Balancer/Reverse Proxy**: Configure IP whitelisting at the load balancer or reverse proxy level (e.g., Nginx, AWS ALB, Azure Application Gateway)
* **Firewall Rules**: Use network-level firewall rules to restrict access
* **VPN/Private Network**: Deploy Mage Pro in a private network accessible only via VPN

**Best Practices:**

* Whitelist only necessary IP ranges (office networks, VPN endpoints)
* Regularly review and update the whitelist
* Use CIDR notation for network ranges
* Document all whitelisted IPs and their purposes
* Consider using a VPN for remote access instead of public IPs

For more information, see the [Authentication Guide](/production/authentication/overview).

### Secrets Management

**Use Mage Secrets:**

* Store API keys, passwords, and tokens in Mage's secrets management
* Never hardcode credentials in blocks
* Rotate secrets regularly

```python theme={"system"}
# ✅ Use secrets
import os

api_key = os.getenv('SALESFORCE_API_KEY')
database_password = os.getenv('DATABASE_PASSWORD')
```

For more information on creating and managing secrets, see the [Secrets Guide](/development/variables/secrets#secrets).

### Access Control

**Follow Principle of Least Privilege:**

* Grant minimum necessary permissions
* Use workspace-level access controls
* Review and audit access regularly

### Data Privacy

**Handle Sensitive Data Appropriately:**

* Mask PII in logs and outputs
* Encrypt sensitive data at rest and in transit
* Comply with data protection regulations (GDPR, CCPA, etc.)

```python theme={"system"}
# ✅ Mask sensitive data in logs
import hashlib

def log_customer_data(customer_id, email):
    # Hash sensitive information
    email_hash = hashlib.sha256(email.encode()).hexdigest()[:8]
    logger.info(f"Processing customer {customer_id} with email {email_hash}...")
```

### Code Security

**Security Best Practices:**

* Avoid executing user input directly
* Validate and sanitize all inputs
* Use parameterized queries for databases
* Keep dependencies updated

```python theme={"system"}
# ✅ Parameterized queries
def load_customer_data(customer_id):
    query = "SELECT * FROM customers WHERE id = :id"
    return pd.read_sql(query, engine, params={'id': customer_id})

# ❌ Avoid SQL injection
# query = f"SELECT * FROM customers WHERE id = {customer_id}"
```

***

## Deployment & CI/CD

### Use CI/CD Deployments (Mage Pro)

**Deployment Workflow:**

1. **Develop** in development workspace
2. **Test** in staging workspace
3. **Deploy** to production using CI/CD

**Best Practices:**

* Automate deployments through CI/CD
* Use deployment pipelines for consistency
* Test deployments in staging first
* Roll back quickly if issues occur

### Environment Management

**Separate Configurations:**

```python theme={"system"}
# ✅ Environment-specific configuration
import os

ENV = os.getenv('ENVIRONMENT', 'development')

if ENV == 'production':
    database_url = os.getenv('PROD_DATABASE_URL')
    api_base_url = 'https://api.production.com'
elif ENV == 'staging':
    database_url = os.getenv('STAGING_DATABASE_URL')
    api_base_url = 'https://api.staging.com'
else:
    database_url = os.getenv('DEV_DATABASE_URL')
    api_base_url = 'https://api.dev.com'
```

### Pipeline Scheduling

**Schedule Appropriately:**

* Match schedule to data freshness requirements
* Consider source system load
* Account for pipeline execution time
* Use appropriate time zones

**Error Handling in Scheduled Pipelines:**

* Set up retry policies using [automatic retry](/orchestration/pipeline-runs/retrying-block-runs#automatic-retry) for block runs
* Configure alerting for failures
* Implement dead letter queues for failed runs

### Backfilling

**Backfill Strategy:**

* Use Mage's built-in backfilling for historical data
* Process in chunks to avoid resource exhaustion
* Monitor backfill progress
* Validate backfilled data

***

## Summary

Following these best practices will help you:

✅ **Build reliable pipelines** with proper error handling and testing\
✅ **Optimize performance** through efficient data processing and resource management\
✅ **Maintain code quality** with modular, reusable blocks\
✅ **Collaborate effectively** using version control and workspaces\
✅ **Monitor and debug** with comprehensive logging and observability\
✅ **Deploy safely** using CI/CD and environment management

Remember: Start with these practices and adapt them to your team's specific needs and constraints. The goal is to build maintainable, scalable, and reliable data pipelines.


# Level 99 dynamic abilities
Source: https://docs.mage.ai/guides/blocks/batch-read-write

Granular block settings for controlling read/write data partitions using output size, number of chunks, and item count.

## Batch read settings

On a block-by-block basis, customize how it reads data from its upstream blocks.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=batch-read-write__read)

### Read specific chunks

Downstream blocks can control the volume of data it loads into memory from its upstream block’s output.

The downstream block will only load the chunk from indigo\_mountain where power equals 5.

**1. Load data**

```python theme={"system"}
import random

import polars as pl

from mage_ai.data.tabular.mocks import create_dataframe


@data_loader
def load_data(*args, **kwargs):
    dfs = []
    for i in range(10):
        df = create_dataframe(n_rows=100_000, use_pandas=False)
        df = df.with_columns(pl.lit(i).alias('power'))
        if i == 5:
            df = df.with_columns(pl.lit(i).cast(pl.Float64).alias('col_0'))
        dfs.append(df)
    return pl.concat(dfs)
```

**2. Transform data**

```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
    return data
```

### Input data types

Downstream blocks can control the strategy it implements when loading an upstream block’s
output data into memory.

#### Batch

Read data in batches.

```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
    print('Batch size:', len(data[0]))
    print('Chunks:', len(data))
```

```
Batch size: 143
Chunks: 21105
```

#### Generator

Batch generator framework to operate and process 1,000+ gigabytes (GB) of data without running out of memory.

```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
    for batch in data:
        df = batch.deserialize()
        print(df.shape)
```

```
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
...
```

**1. Load data**

```python theme={"system"}
import random

import polars as pl

from mage_ai.data.tabular.mocks import create_dataframe


@data_loader
def load_data(*args, **kwargs):
    dfs = []
    for i in range(10):
        df = create_dataframe(n_rows=100_000, use_pandas=False)
        df = df.with_columns(pl.lit(i).alias('power'))
        if i == 5:
            df = df.with_columns(pl.lit(i).cast(pl.Float64).alias('col_0'))
        dfs.append(df)
    return pl.concat(dfs)
```

**2. Transform data**

```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
    for batch in data:
        df = batch.deserialize()
        print(df.shape)
```

#### Reader

Invoke methods on the reader object directly.

```python theme={"system"}
from mage_ai.data.tabular.reader import (
    read_metadata,
    sample_batch_datasets,
    scan_batch_datasets_generator,
)

@transformer
def transform(data, *args, **kwargs):
    print(data.chunks)
    print(data.number_of_outputs)
    print(data.resource_usages)
    print(data.variable_path)
    print(data.variable_type)
    print(data.data_source)

    output = data.read_sync()
    for batch in output:
        df = batch.deserialize()
        print(df.shape)
```

***

## Batch write settings

Customize how data is written when outputting data from a block.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=batch-read-write__write)

```python theme={"system"}
import random

import polars as pl

from mage_ai.data.tabular.mocks import create_dataframe


@data_loader
def load_data(*args, **kwargs):
    dfs = []
    for i in range(10):
        df = create_dataframe(n_rows=100_000, use_pandas=False)
        df = df.with_columns(pl.lit(i).alias('power'))
        if i == 5:
            df = df.with_columns(pl.lit(i).cast(pl.Float64).alias('col_0'))
        dfs.append(df)
    return pl.concat(dfs)
```

### Strategies

Explicitly set the chunking strategy when creating data partitions or batches of data.

Example: create chunks by the value in column `power`

#### Items per chunk

Example: each chunk can only contain a maximum number of 7,777 items.

#### Number of chunks

Example: each chunk must have at least 1,337 items and the total number of chunks cannot exceed 143.

#### Byte size per check

Example: each chunk cannot exceed 100MB in size on disk.

### Modes

Control how data is written.

1. Append
2. Replace


# Custom blocks
Source: https://docs.mage.ai/guides/blocks/custom-blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

## Adding a custom block to the pipeline

1. Create a new pipeline or open an existing pipeline.
2. Add a custom block by clicking on the `Custom` button from the list of block types available.
   The `Custom` button is visible at the bottom of the main section or when hovering over the divider between two blocks.
3. Specify the language for your block (Python and SQL currently supported). For custom Python blocks, use the `@custom` decorator.
4. Specify the color for your block. The colors are a way to visually identify or organize your custom blocks (e.g. The block will have this color in the dependency tree.) and do not affect how they are run.
5. After clicking on the color from the dropdown menu, a new custom block should appear in your pipeline.
6. You can rearrange the blocks by clicking on the block header and dragging it to another position in the pipeline.


# Detaching blocks
Source: https://docs.mage.ai/guides/blocks/detach-blocks

Detach other pipeline associations from a block shared by multiple pipelines.

## Overview

If a block is being used by multiple pipelines, changing the content of that block
will change it for all of the pipelines sharing that block. If you want to create an
isolated copy of that block for a specific pipeline so it doesn't affect other
pipelines sharing it, you can "detach" it.

Detaching a block duplicates it so it is no longer shared with any other pipelines,
removing/detaching other pipeline associations. A new block file is created with the
original block's contents, and the original block's dependencies in the current pipeline
are preserved.

### Supported pipeline types

Detaching a block is supported in `standard` and `streaming` pipelines. It is not supported
in `integration` pipelines.

### Supported block types

* Data loader
* Transformer
* Data exporter
* Sensor
* Custom

### Unsupported block types

* DBT
* Callback
* Conditional
* Extension

## How to detach a block

![](https://mage-ai.github.io/assets/blocks/detach_block.png)

1. If a block is shared by multiple pipelines, there will be a symbol, which looks something
   like `<> X pipelines`, with the number of pipelines in the block's header on the Pipeline
   Editor page. Refer to the screenshot above for an example. All 3 of the example pipeline's
   blocks in the screenshot are shared by multiple pipelines.

2. Click on the shared pipelines symbol (`<>`) or the block settings icon in the block
   header to go to the Block Settings panel in the Sidekick.

3. Click on the `Detach </>` button at the top right of the block settings panel. Note
   that this button will only be visible if the selected block is shared by multiple pipelines.
   Blocks that are not shared by any other pipelines do not have the option to detach itself.

4. The **Block name** dialog appears, allowing you to name the new block that will be
   created to replace the existing block. This new block will not be shared by any other
   pipelines.

5. Click **Save**.


# Dynamic blocks
Source: https://docs.mage.ai/guides/blocks/dynamic-blocks

Create a pipeline with dynamic blocks.

## Build pipeline

1. Add a new data loader block.
2. Paste in the following code:
   ```python theme={"system"}
   from typing import Dict, List


   @data_loader
   def load_data(*args, **kwargs) -> List[List[Dict]]:
       users = []
       metadata = []

       for i in range(3):
           i += 1
           users.append(dict(id=i, name=f'user_{i}'))
           metadata.append(dict(block_uuid=f'for_user_{i}'))

       return [
           users,
           metadata,
       ]
   ```
3. Run the data loader block and the output will be:
   ```python theme={"system"}
   [{'id': 1, 'name': 'user_1'},
    {'id': 2, 'name': 'user_2'},
    {'id': 3, 'name': 'user_3'}]
   ```
4. In the top right corner of the block, click on the triple dot button (`...`) and click the
   dropdown selection labeled `Set block as dynamic`.
   ![Dynamic data loader block](https://mage-ai.github.io/assets/blocks/dynamic-block-data-loader.png)
5. Add a new transformer block.
6. Paste in the following code:
   ```python theme={"system"}
   from typing import Dict, List


   @transformer
   def transform(data: Dict, *args, **kwargs) -> List[Dict]:
       data['id'] = int(data['id']) * 100
       return [data]
   ```
7. Run the transformer block and the output will be:

   ```python theme={"system"}
   {'id': 100, 'name': 'user_1'}
   ```

   ![Dynamic child transformer block 1](https://mage-ai.github.io/assets/blocks/dynamic-block-transformer1.png)
8. Add another transformer block.
9. Paste in the following code:
   ```python theme={"system"}
   from typing import Dict, List
   import uuid


   @transformer
   def transform(data: Dict, *args, **kwargs) -> List[Dict]:
       data['token'] = uuid.uuid4().hex
       return [data]
   ```
10. Change the 2nd transformer block’s upstream parent block to only be the data loader.
    This will require you to remove the existing upstream parent block which is currently pointing to
    the 1st transformer block.
11. In the top right corner of this 2nd transformer block,
    click on the triple dot button (`...`) and click the
    dropdown selection labeled `Reduce output`.
12. Run the transformer block and the output should look something like this:

    ```python theme={"system"}
    {'id': 1, 'name': 'user_1', 'token': '0ea0f28c2de647d997e5c341a857c92f'}
    ```

    ![Dynamic child transformer block 2](https://mage-ai.github.io/assets/blocks/dynamic-block-transformer2.png)
13. Add a new data exporter block.
14. Paste in the following code:
    ```python theme={"system"}
    from typing import Dict, List


    @data_exporter
    def export_data(users: List[Dict], **kwargs):
        print(users)
    ```
15. Run the data exporter block and the output should look something like this:

    ```python theme={"system"}
    [{'id': 1, 'name': 'user_1', 'token': 'bc35e36ecfc24d77a2d910f4b1d6611d'}]
    ```

    When executing the entire pipeline, this output will contain 3 items in this list.

    ![Data exporter block from dynamically created block with reduce output](https://mage-ai.github.io/assets/blocks/dynamic-block-data-exporter.png)

The pipeline’s final dependency graph should look like this:

![Dynamic block dependency graph](https://mage-ai.github.io/assets/blocks/dynamic-block-graph.jpg)

## Run pipeline

1. Open the pipeline's **Triggers** page from the pipeline navigation.
2. Click the button `[Run @once]` in the page's subheader,
   or create a trigger to run once.
3. Click the name of the newly created trigger.
4. The trigger will create 1 pipeline run with 9 block runs.

The block runs should look like this:

![Dynamic block runs](https://mage-ai.github.io/assets/blocks/dynamic-block-runs.png)

## Limit dynamically created blocks

To prevent a dynamic block from creating too many downstream block runs, set a maximum number
of dynamic child block runs. To throttle execution after those rows are created, set a maximum
number of concurrent dynamic child block runs. Unset values and `0` allow unlimited dynamic child
block creation or concurrent dynamic child execution.

Set workspace-wide defaults with the `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS` and
`DYNAMIC_BLOCKS_MAX_CONCURRENT_CHILD_BLOCKS` environment variables. Set
`DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS_BEHAVIOR` to `fail` to stop runs that exceed the creation limit,
or `limit` to create only up to the configured maximum. You can also add this to the project
`metadata.yaml`:

```yaml theme={"system"}
dynamic_blocks:
  max_child_block_runs: 1000
  on_max_child_block_runs_exceeded: fail
  max_concurrent_child_block_runs: 25
```

Set a pipeline-specific limit in the pipeline `metadata.yaml`:

```yaml theme={"system"}
dynamic_blocks:
  max_child_block_runs: 500
  on_max_child_block_runs_exceeded: limit
  max_concurrent_child_block_runs: 10
```

Set a stricter limit for one dynamic block in that block's configuration:

```yaml theme={"system"}
configuration:
  dynamic:
    max_child_block_runs: 100
    on_max_child_block_runs_exceeded: limit
    max_concurrent_child_block_runs: 5
```

## Dynamic blocks 2.0

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dynamic-blocks-2__stream-mode)

In version 1.0 of dynamic blocks,
Mage was loading all the data of an upstream dynamic child block from disk into memory
in order to calculate how many dynamic child blocks should be created.
This can result in using large amounts of memory.

In version 2.0 of dynamic blocks,
Mage leverages the metadata of upstream block outputs
to calculate the number of downstream blocks for dynamic blocks and dynamic child blocks.
The metadata is only several bytes on disk and less than a kilobyte in RAM.

### Stream mode

Here is a sample scenario to describe the previous and current state:

1. Dynamic block A returns a list of 10 items and has 1 direct downstream block B.
2. B is a dynamic child block and has 2 direct downstream block: C and D.
3. C is a dynamic child block and has 1 direct downstream block: D.

```mermaid theme={"system"}
graph TD
  A --> B
  B --> C
  B --> D
  C --> D
```

In the past, this is the sequence of events:

1. Block A finishes executing and returns an output of 10 items.
2. Block B spawns 10 dynamic child blocks.
3. Block B executes all 10 dynamic blocks concurrently.
4. Once all 10 block runs from block B finishes, block C starts.
5. Block C spawns 10 child blocks and executes them.
6. Once Block C completes all 10 runs, block D starts.
7. Block D spawns 100 child blocks and executes them.

With dynamic block 2.0, you can enable stream mode which will allow downstream dynamic child blocks to execute as soon as its upstream parent dynamic child blocks finish. Here is the sequence of events in the new world:

1. Block A executes and can be handled in 2 ways:
   1. Serial: block A executes its code line by line and at the end, the return statement outputs the entire result of 10 items in a list.
   2. Generator: block A executes its code line by line up until a yield block. Once yield is called, the object that is yielded is stored as an output of block A. In this example, block A would yield 10 times, 1 for each item.
2. As soon as 1 output is detected from block A, block B spawns 1 dynamic child that consumes that single output as its input argument.
3. As soon as 1 output is detected from block B, block C spawns 1 dynamic child.
4. As soon as 1 output is detected from both block B and block C, block D spawns 1 dynamic child.


# Level 99 dynamic abilities
Source: https://docs.mage.ai/guides/blocks/dynamic-blocks-2

Execute 100,000+ dynamically created block runs concurrently with upgraded memory management and concurrency.

## Memory optimizations

In version 1.0 of dynamic blocks,
Mage was loading all the data of an upstream dynamic child block from disk into memory
in order to calculate how many dynamic child blocks should be created.
This can result in using large amounts of memory.

In version 2.0 of dynamic blocks,
Mage leverages the metadata of upstream block outputs
to calculate the number of downstream blocks for dynamic blocks and dynamic child blocks.
The metadata is only several bytes on disk and less than a kilobyte in RAM.

## Stream mode

Stream block output data to dynamically generated blocks without waiting for the upstream parent
block to finish executing.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dynamic-blocks-2__stream-mode)

Here is a sample scenario to describe the previous and current state:

1. Dynamic block A returns a list of 10 items and has 1 direct downstream block B.
2. B is a dynamic child block and has 2 direct downstream block: C and D.
3. C is a dynamic child block and has 1 direct downstream block: D.

```mermaid theme={"system"}
graph TD
  A --> B
  B --> C
  B --> D
  C --> D
```

In the past, this is the sequence of events:

1. Block A finishes executing and returns an output of 10 items.
2. Block B spawns 10 dynamic child blocks.
3. Block B executes all 10 dynamic blocks concurrently.
4. Once all 10 block runs from block B finishes, block C starts.
5. Block C spawns 10 child blocks and executes them.
6. Once Block C completes all 10 runs, block D starts.
7. Block D spawns 100 child blocks and executes them.

With dynamic block 2.0, you can enable stream mode which will allow downstream dynamic child blocks to execute as soon as its upstream parent dynamic child blocks finish. Here is the sequence of events in the new world:

1. Block A executes and can be handled in 2 ways:
   1. Serial: block A executes its code line by line and at the end, the return statement outputs the entire result of 10 items in a list.
   2. Generator: block A executes its code line by line up until a yield block. Once yield is called, the object that is yielded is stored as an output of block A. In this example, block A would yield 10 times, 1 for each item.
2. As soon as 1 output is detected from block A, block B spawns 1 dynamic child that consumes that single output as its input argument.
3. As soon as 1 output is detected from block B, block C spawns 1 dynamic child.
4. As soon as 1 output is detected from both block B and block C, block D spawns 1 dynamic child.

## Generated child dependencies

Dynamic block metadata can also define dependencies between generated child block runs.
Use this when a dynamic parent emits many children for the same downstream block, but some
generated children must wait for other generated children before they run.

For example, a dynamic parent can emit five generated children for a downstream transformer:

```python theme={"system"}
from typing import Dict, List


@data_loader
def load_data(*args, **kwargs) -> List[List[Dict]]:
    rows = [
        dict(child_id='root_a', sleep_seconds=4),
        dict(child_id='root_b', sleep_seconds=4),
        dict(child_id='after_a', sleep_seconds=1),
        dict(child_id='after_a_b', sleep_seconds=1),
        dict(child_id='after_c_d', sleep_seconds=1),
    ]

    metadata = [
        dict(block_uuid='root_a'),
        dict(block_uuid='root_b'),
        dict(block_uuid='after_a', upstream_blocks=['root_a']),
        dict(block_uuid='after_a_b', upstream_blocks=['root_a', 'root_b']),
        dict(block_uuid='after_c_d', upstream_blocks=['after_a', 'after_a_b']),
    ]

    return [
        rows,
        metadata,
    ]
```

If the downstream transformer block UUID is `process_child`, Mage creates these block runs:

| Generated block run UUID | Metadata dependency | Scheduling behavior |
| - | - | - |
| `process_child:root_a` | None | Can run as soon as the dynamic parent and static upstreams are ready. |
| `process_child:root_b` | None | Can run as soon as the dynamic parent and static upstreams are ready. |
| `process_child:after_a` | `root_a` | Waits for `process_child:root_a`. |
| `process_child:after_a_b` | `root_a`, `root_b` | Waits for `process_child:root_a` and `process_child:root_b`. |
| `process_child:after_c_d` | `after_a`, `after_a_b` | Waits for `process_child:after_a` and `process_child:after_a_b`. |

The dependency graph for the generated child runs is:

```mermaid theme={"system"}
graph TD
  root_a["process_child:root_a"] --> after_a["process_child:after_a"]
  root_a --> after_a_b["process_child:after_a_b"]
  root_b["process_child:root_b"] --> after_a_b
  after_a --> after_c_d["process_child:after_c_d"]
  after_a_b --> after_c_d
```

### Metadata keys

Add these keys to each metadata dictionary returned by the dynamic parent:

| Key | Type | Required | Description |
| - | - | - | - |
| `block_uuid` | string | Yes | Stable name for the generated child. Mage combines this with the downstream block UUID using the format `[downstream_block_uuid]:[block_uuid]`. |
| `upstream_blocks` | list of strings | No | Generated child dependencies that must finish before this generated child can run. |
| `dynamic_upstream_block_uuids` | list of strings | No | Alternate dependency key for generated child dependencies. Use `upstream_blocks` for new dynamic parent metadata unless you already produce this key. |

Each dependency can be written as a short generated child name, a full generated block run UUID,
or a normal static block UUID:

```python theme={"system"}
dict(block_uuid='after_a', upstream_blocks=['root_a'])
dict(block_uuid='after_a', upstream_blocks=['process_child:root_a'])
dict(block_uuid='after_a', upstream_blocks=['load_shared_inputs'])
```

The first two examples make `process_child:after_a` wait for `process_child:root_a`.
The third example makes `process_child:after_a` wait for the static block run
`load_shared_inputs`.
When the dependency value does not contain `:`, Mage treats it as a generated child name for the
same downstream block if that generated child exists. When the dependency value already contains
`:`, Mage treats it as the exact block run UUID. Use the full UUID form to depend on a generated
child from another downstream block, such as `validate_child:root_a`.

### Scheduling behavior

Generated child dependencies are enforced after the normal pipeline dependency checks.
That means a generated child can run only when both of these are true:

1. The dynamic parent and the downstream block's static upstream chain are ready.
2. Every generated child listed in that generated child's metadata dependencies has finished.

Children with no metadata dependencies can run concurrently. Children with one or more metadata
dependencies remain queued until those generated upstream children finish.

These metadata dependencies control scheduling only. They do not change the input data passed to
the child block. To pass data between generated children, write and read it explicitly using normal
block output, variables, storage, or another shared data location.

Make sure every metadata dependency points to a block run that will exist in the same pipeline run.
If a dependency references a generated child that is never created, the waiting generated child will
not become executable because its declared upstream never finishes.

If a downstream dynamic child has **Reduce output** enabled, Mage still enforces the metadata
dependencies before the generated child block runs. The reduce step happens after the generated
children complete, so downstream reduced blocks receive the combined outputs only after the
dependency-constrained child runs have finished.

## Enabling stream mode

Stream mode allows downstream dynamic child blocks to execute without waiting for every upstream block to finish, improving pipeline performance.

### Accessing block settings

To enable stream mode, access the Dynamic block v2 configuration:

1. Click the **block settings button** next to the "Run code" button
   * Alternatively, use the **Block settings** tab in the right side navigation menu

### Configuration

1. Scroll down to the "Dynamic block v2 configuration" section
2. Select **"Stream"** from the Mode dropdown
3. Enter a **poll interval** in seconds - this is how often the system checks for new data or completed upstream blocks. Lower values (like 1 second) provide faster response times but use more system resources
4. **(Optional)** Toggle **"Combine all output into 1"** to consolidate outputs for downstream processing

Stream mode enables parallel processing and better resource utilization by allowing child blocks to start processing data as it becomes available.

![block-settings-button](https://raw.githubusercontent.com/mage-ai/assets/main/blocks/block-settings-button.png)

## Memory generator

If a dynamic block contains a yield block,
the object yielded will be an output from that block. Once that object is stored,
it is released from memory and the next object in the generator within that block
is processed and loaded into memory.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dynamic-blocks-2__memory-generator)

For example, if you want to loop through a total of 1 billion indexes in 1 million increments
so that you can load 1 million records into memory, you can use a yield block to process a
batch of 1 million, then yield the result as an output, release it from memory automatically,
and then proceed to the next batch.

```python theme={"system"}
@custom
def process(*args, **kwargs):
    for i in range(1_000):
		    yield range(i * 1_000_000, (i + 1) * 1_000_000)
```


# Markdown blocks
Source: https://docs.mage.ai/guides/blocks/markdown-blocks

A markdown block is a standalone text block that supports markdown and can be used to make notes or organize your blocks.

## Adding a markdown block to the pipeline

1. Create a new pipeline or open an existing pipeline.
2. Edit the pipeline.
3. Add a markdown block by clicking on the `Markdown` button from the list of block types available.
   The `Markdown` button is visible at the bottom of the main section or when hovering over the divider between two blocks.
4. You can rearrange the blocks by clicking on the block header and dragging it to another position in the pipeline.

### Shortcuts

* Edit mode: Double-click the markdown block's content area to open the content editor. You can
  also click the "Edit" icon in the block header to edit the content.
* Exit edit mode: Press `Escape` on your keyboard while the content editor is active.
  You may need to press `Escape` twice, once to exit the editor and another time to deselect
  the block. When the block is deselected, the editor will automatically close.

### Directory

The files associated with the markdown blocks are located in the `markdowns` directory
in the File Browser.

### Deleting a markdown block

When you delete a markdown block that is not used in any other pipeline,
the file may also be deleted automatically as well.

## Supported syntax

#### Horizontal breaks

***

#### H1-H5 Headlines

Normal text

**bold text**

*italicized text*

> blockquote

#### Ordered list

1. First item
2. Second item
3. Third item

#### Unordered list

* First item
* Second item
* Third item

You can use inline `test_code` in your sentences.

Links: [Mage homepage](https://www.mage.ai)

#### Tables

| Column1 | Column2 | Column3 |
| - | - | - |
| Text1 | TextA | TextI |
| Text2 | TextB | TextII |
| Text3 | TextC | TextIII |

#### Code blocks

```
{
  "firstName": "John",
  "lastName": "Smith",
  "age": 25
}
```

#### Checkboxes

* [x] Todo 1 checked
* [ ] Todo 2 unchecked
* [ ] Todo 3 unchecked

#### Images

Images are supported with the typical image markdown syntax:

```
![](https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85)
```

or with the `<img>` HTML tag syntax:

```
<img src="https://images.photowall.com/products/57215/golden-retriever-puppy.jpg?h=699&q=85" alt="drawing" width="200"/>
```

With the `<img>` syntax, you can create customizable image sizes with `width` and `height` attributes.


# Native support for Polars
Source: https://docs.mage.ai/guides/blocks/polars

Serialize and deserialize data using Polars DataFrames.

Mage supports using Polars DataFrame in addition to Pandas DataFrame to process the data.

```python theme={"system"}
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
import polars as pl


@data_loader
def load_data(*args, **kwargs):
    df = pl.DataFrame(
        {
            'A': [1, 2, 3, 4, 5] * 10,
            'fruits': ['banana', 'banana', 'apple', 'apple', 'banana'] * 10,
            'ham': ['a', 'b', 'c', 'd', 'e'] * 10,
            'dt': ['2023-03-01', '2023-03-02', '2023-03-03', '2023-03-04', '2023-03-05'] * 10,
            'arr': [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] * 10,
            'obj': [dict(a=[1, 2]), dict(a=[3, 4]), dict(a=[5, 6]), dict(a=[7, 8]), dict(a=[9, 10])] * 10,
        }
    )
    df = df.with_columns(
        pl.col('dt').str.to_datetime('%Y-%m-%d')
    )
    return df
```

If you return a Pandas or Polars DataFrame, it’ll be serialized to disk using PyArrow and stored in Parquet format. When a downstream block loads an upstream block’s output data as an input argument, that data will be deserialized back to the data type before it was serialized.

![](https://media.graphassets.com/FBfwDhuzSnyc2O3pLyMS)

Leveraging Polars and PyArrow in this process improves the memory usage while serializing data to disk and deserializing data from disk by leveraging Polar’s lazy data frames, PyArrow partitions, and Parquet file format features such as accessing metadata and schema information without loading the raw data into memory.


# R blocks
Source: https://docs.mage.ai/guides/blocks/r-blocks

You can write R language to transform data in blocks.

## Requirements

> [!WARNING]
>
> R blocks are only supported when running Mage using [Docker](/getting-started/setup#using-docker).

## Add R block to pipeline

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `R`.

## Example pipeline

1. Data loader
   ```R theme={"system"}
   load_data <- function() {
       # Specify your data loading logic here
       # Return value: loaded dataframe
       df <- read.csv(url('https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'))
       df
   }
   ```
2. Transformer
   ```R theme={"system"}
   library("pacman")
   p_load(dplyr)

   transform <- function(df_1, ...) {
       # Specify your transformation logic here
       # Return value: transformed dataframe.
       df_1 <- filter(df_1, Pclass < 3)
       df_1
   }
   ```
3. Data exporter
   ```R theme={"system"}
   export_data <- function(df_1, ...) {
       # Specify your data exporting logic here
       # Return value: exported dataframe
       write.csv(df_1, "titanic_filtered.csv", row.names = FALSE)
   }
   ```

## Install R packages

Add the following at the start of your code in your R block:

```R theme={"system"}
pacman::p_load(package1, package2, package3)
```

Or

```R theme={"system"}
library("pacman")
p_load(dplyr)
```

> Note
>
> When you run the R block for the 1st time, the package will be installed.
> The 2nd time you run the R block, the package won’t need to be installed again.

### What is `pacman`?

`pacman` is an R package management tool. You can use `p_library()` to view all the available packages.

Here is the documentation for `pacman` where you can find more useful methods: [https://www.rdocumentation.org/packages/pacman/versions/0.5.1](https://www.rdocumentation.org/packages/pacman/versions/0.5.1)

## Runtime variables

Runtime variables can be accessed via `global_vars` vector, like `global_vars['execution_date']`.

Example code:

```R theme={"system"}
load_data <- function() {
    df <- read.csv(file='titanic_clean.csv')
    df['date'] <- global_vars['execution_date']
    df
}
```


# Reordering blocks
Source: https://docs.mage.ai/guides/blocks/reorder-blocks

Reorder blocks in the notebook when editing a pipeline.

## Overview

When you are in the Pipeline Editor page of a standard or streaming pipeline, you might
want to rearrange the order that the blocks appear in the notebook. This could help with
organizing your pipeline or allow you to group your blocks a certain way.

Keep in mind that reordering the blocks from the UI will change the order of
the blocks in your pipeline's `metadata.yaml` configuration file.

## How to reorder blocks

1. Go to your pipeline's Edit page (`/pipelines/[pipeline_uuid]/edit`).

2. Click on the block header of the block you want to move, and drag it
   over the block whose position you want to move it to.

3. Dragging a block from below another block will move it to the position
   above that block. Dragging a block from above another block will move it
   to the position below that block.

![](https://mage-ai.github.io/assets/blocks/reorder-blocks.gif)

4. Once you have the block in the position you want, drop the block.

5. The updated block order should be reflected in your pipeline's
   `metadata.yaml` file.

## Reordering upstream blocks received as positional arguments

If a block has multiple upstream blocks, you can access the upstream blocks' outputs via
the positional arguments of the downstream block's decorated function. You can rearrange
these positional arguments by dragging and dropping the associated blocks using the same
steps as [above](#how-to-reorder-blocks) for reordering blocks.

![](https://mage-ai.github.io/assets/blocks/reorder-upstream-block-args.gif)


# Replicating blocks
Source: https://docs.mage.ai/guides/blocks/replicate-blocks

Reuse the same block multiple times within a single pipeline.

![Replicate blocks](https://64.media.tumblr.com/7354b3e96228024e5ae2924814053f98/tumblr_pcpk4z3lX11sgefjfo9_500.gif)

## What does it mean to replicate a block

Let’s say you have a pipeline named `fire_pipeline` with the following 3 blocks:

```
[load_source_a]            [load_source_b]
      ↓
[transform_data]
```

The transformer block named `transform_data` contains reusable code. Instead of creating a brand
new block in this pipeline that does the exact same thing as `transform_data`, you can replicate
the `transform_data` block.

Once you replicate the `transform_data` block, you can configure it however you want. For example,
you can name the replica block `transform_source_b` or
set the replica block’s upstream block to `load_source_b`.

The `fire_pipeline` will now look like this:

```
[load_source_a]            [load_source_b]
       ↓                          ↓
[transform_data]         [transform_source_b]
```

A replica block’s code and execution logic mirrors the block it replicated.
In this example, the `transform_source_b` block will get its code from the `transform_data` block.

### Why is replicating blocks useful

Replicating blocks make it easier to reuse code in a pipeline while maintaining observability of
each branch in the pipeline because each block, even when replicated, will have its own block
run instance when a pipeline is executed.

These block run instances have monitoring, alerting, and logging built-in as well as producing
partitioned data products when successfully completed.

### Terminology

| Term | Definition |
| - | - |
| Replicated block | An original block in a pipeline that has 1 or more replica blocks. |
| Replica block | A block in a pipeline that uses the code from another block in the same pipeline. |

## How to replicate a block

All block types, except dbt blocks, can be replicated. In the top right corner of the block,
click the triple dot icon to display a dropdown menu. Under the dropdown menu, click the option
labeled **`Replicate block`**.

This will add a new block to your current pipeline. You can optionally name the new replica block.

When the pipeline runs, it’ll create a block run for each block in the pipeline. When a block run
is created for a replica block, the block run will reference that block using the following
block UUID convention: `[replica_block_uuid]:[replicated_block_uuid]`.

For example:

1. Original block’s UUID is `transform_data`.
2. The `transform_data` block is replicated.
3. The newly created replica block’s UUID is `transform_source_b`.
   It’s replicated from the block `transform_data`.

A block run’s `block_uuid` attribute for the replica block `transform_source_b` will appear in
dashboards and logs as `transform_source_b:transform_data`.


# Reuse blocks
Source: https://docs.mage.ai/guides/blocks/reuse-blocks

Learn how to reuse blocks within pipelines, across pipelines, and across projects in Mage for better code organization and efficiency.

## Overview

Mage allows you to reuse blocks across multiple pipelines within the same project, enabling you to build modular, maintainable data pipelines. When you reuse a block, any changes made to that block will automatically apply to all pipelines that use it.

### Benefits of reusing blocks

* **Code efficiency** - Write once, use many times
* **Consistency** - Ensure the same logic is applied across all pipelines
* **Maintenance** - Update logic in one place and it applies everywhere
* **Modularity** - Build complex pipelines from smaller, reusable components
* **Testing** - Test block logic once and reuse with confidence

## Reuse Scenarios and Methods

| Scenario | Method | Description | Best For |
| - | - | - | - |
| **Within Pipeline** | [Replicate blocks](#replicate-blocks-within-a-pipeline) | Right-click to duplicate blocks in the same pipeline | Similar data sources, parallel processing, testing variations |
| **Across Pipelines** | [Drag & drop](#drag-and-drop-from-file-browser) | Drag blocks from file browser to pipeline canvas | Quick reuse of existing blocks |
| **Across Pipelines** | [Search bar](#use-the-search-block-bar) | Search and add blocks from all projects | Finding specific blocks across projects |
| **Code Reuse** | [Utility files](#create-utility-files-and-import-in-blocks) | Create shared Python modules and import them | Small functions, data processing utilities |
| **Code Reuse** | [Block templates](#use-block-templates) | Create reusable templates with placeholders | Common patterns with slight variations |
| **Across Projects** | [Pro templates](#block-templates-across-projects-mage-pro) | Use Mage Pro's template library | Organization-wide standardization |
| **Across Projects** | [Export/import](#cross-project-block-sharing) | Export blocks from one project to another | Sharing between different projects |

## How block sharing works

When you create a block in one pipeline, you can reference that same block in other pipelines. The block file exists in the project's file system and can be imported by any pipeline within the same project.

### Block sharing indicators

When viewing a pipeline, you'll see indicators showing which blocks are shared:

* **Shared blocks** display a symbol like `<> 2 pipelines` in the block header
* **Non-shared blocks** show no additional symbols
* The number indicates how many pipelines are using that block

### Viewing pipelines using a block

To see which pipelines are using a specific block:

1. **Click on a block** in the pipeline editor
2. **Open block settings** - Click the settings icon in the right side panel
3. **View "Pipelines using this block"** - See a list of all pipelines that reference this block
4. **Check project paths** - See which project each pipeline belongs to

This helps you understand the impact of changes to shared blocks across your project.

## Methods to reuse blocks

### Drag and drop from file browser

This is the most intuitive way to reuse blocks across pipelines:

1. **Open the file browser** on the Pipeline Editor page
2. **Navigate to the block** you want to reuse in the file browser
3. **Click and hold** the block file
4. **Drag the block file** onto the center of the page (the Notebook area)
5. **Release the button** and the block will be added to the pipeline

![Drag blocks from the file browser and drop it on the pipeline](https://mage-ai.github.io/assets/dev-ux/code-editor/drag-block.gif)

**Supported block types:**

* `callbacks`, `conditionals`, `custom`, `data_exporters`, `data_loaders`
* `dbt`, `dbts`, `markdowns`, `scratchpads`, `sensors`, `transformers`

> [!NOTE]
>
> **Mage OSS**: Version 0.9.64 or greater is required to add callback and conditional blocks from the file browser.
>
> **Mage Pro**: Callback and conditional blocks are supported by default in all versions.

### Use the search block bar

Search for existing blocks across all your projects:

1. **Click in the search bar** at the bottom of the pipeline editor
2. **Type keywords** to search for existing blocks (e.g., "load titanic")
3. **Browse the search results** showing blocks from different projects
4. **Click on a block** from the dropdown to add it to your pipeline
5. **The block will be added** to your current pipeline

![Search blocks across entire projects](https://mage-ai.github.io/assets/dev-ux/code-editor/block-search.gif)

> [!NOTE]
>
> **Mage OSS**: The "Add new block v2" feature must be enabled in project settings and is available in standard (batch) pipelines. If recently enabled, Mage must be restarted for blocks to be indexed.
>
> **Mage Pro**: This feature is enabled by default and available in all pipeline types.

### Replicate blocks within a pipeline

To reuse a block within the same pipeline (create a copy):

1. **Click the triple dot icon** in the top right corner of the block you want to replicate
2. **Select "Replicate block"** from the dropdown menu
3. **Name the new replica block** (optional)
4. **Customize the new block** as needed for your specific use case
5. **Connect the blocks** in your pipeline flow as required

![Replicate the code of an existing block](https://mage-ai.github.io/assets/dev-ux/code-editor/replicate-block.gif)

**Use cases for block replication:**

* **Similar data sources** - Load from different tables with similar structure
* **Multiple transformations** - Apply similar logic to different data streams
* **Parallel processing** - Process the same data with different parameters
* **Testing variations** - Compare different approaches side by side

Learn more about [replicating blocks](/guides/blocks/replicate-blocks).

## Other ways to reuse code

### Create utility files and import in blocks

For shared functions and logic that don't need to be full blocks:

1. **Create a utility file** in your project (e.g., `default_repo/utils/data_processing.py`)
2. **Write reusable functions** in the utility file
3. **Import the functions** in your blocks using standard Python imports

**Example utility file:**

```python theme={"system"}
# default_repo/utils/data_processing.py
import pandas as pd

def clean_column_names(df):
    """Clean column names by removing spaces and special characters."""
    df.columns = df.columns.str.replace(' ', '_').str.lower()
    return df

def validate_dataframe(df, required_columns):
    """Validate that dataframe has required columns."""
    missing_cols = set(required_columns) - set(df.columns)
    if missing_cols:
        raise ValueError(f"Missing columns: {missing_cols}")
    return True
```

**Using in a block:**

```python theme={"system"}
# In your data loader block
from default_repo.utils.data_processing import clean_column_names, validate_dataframe

@data_loader
def load_data():
    df = pd.read_csv('data.csv')
    df = clean_column_names(df)
    validate_dataframe(df, ['id', 'name', 'email'])
    return df
```

Learn more about [importing custom Python code](/development/dependencies/custom-files#importing-custom-python-code).

### Use block templates

Create reusable block templates for common patterns:

1. **Navigate to Templates** - In Mage OSS, select **Templates** in the left navigation. In Mage Pro, open **Library** and select **Templates**.
2. **Create new template** - Click "New block template" button
3. **Define template properties** - Set template UUID, block type, and language
4. **Edit template content** - Write the template code with placeholders
5. **Save template** - Click "Save template" to create the template

Learn more about [creating and using user-defined templates](/guides/blocks/user-defined-templates).

## Reusing blocks across projects

### Block templates across projects (Mage Pro)

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=block-templates)

> [!NOTE]
>
> **Mage Pro feature**: Mage Pro supports configuring a central template library that shares templates across projects, allowing you to create and share reusable block templates across multiple projects in your organization.

**Creating block templates:**

1. **Navigate to Templates** - Open **Library** in the left navigation and select **Templates**
2. **Create new template** - Click "New block template" button
3. **Define template properties** - Set template UUID, block type, and language
4. **Edit template content** - Write the template code with placeholders
5. **Save template** - Click "Save template" to create the template

**Using block templates:**

1. **Open pipeline editor** - Navigate to the pipeline where you want to add the block
2. **Add new block** - Click the "+" button to add a new block
3. **Browse templates** - Select "Browse templates" from the block creation options
4. **Choose a source** - Select **Core** for Mage's built-in templates or **Team** for templates created in your workspace
5. **Select template** - Browse or search the folders, then select a template to preview its code
6. **Add to workflow** - Click **Add to workflow**. Core templates are added directly; Team templates open the **Block name** dialog so you can enter a name, then click **Save**.

The **dbt** folder contains three workflow starters: select a model or snapshot
from a file, run all models with optional selection or exclusion rules, or
start with a generic dbt command. Selecting the file-based starter opens the
file browser so you can choose the model or snapshot to add.

When you add a callback or conditional block from an existing block's add-on panel,
Mage opens the same template browser scoped to the matching block type.

### Cross-project block sharing

For organizations with multiple projects:

1. **Export blocks** by downloading the block file from the source project
2. **Import blocks** by uploading the block file in the file browser of the target project
3. **Maintain version control** across project boundaries
4. **Share common utilities** and configurations

**Best practices for cross-project sharing:**

* **Document dependencies** - Clearly list required packages and configurations
* **Use relative imports** - Make blocks portable across different project structures
* **Test thoroughly** - Ensure blocks work in different project environments

## Best practices for block reuse

### Design reusable blocks

* **Make blocks generic** - Avoid hardcoded values, use runtime variables instead
* **Clear naming** - Use descriptive names that indicate the block's purpose
* **Documentation** - Add comments explaining what the block does and its inputs/outputs
* **Error handling** - Include proper error handling for different use cases

### Runtime variables for flexibility

Use runtime variables to make blocks adaptable across different pipelines:

```python theme={"system"}
# Instead of hardcoded values
database_name = "production_db"

# Use runtime variables
database_name = kwargs.get("database_name", "default_db")
```

Learn more about [using runtime variables](/getting-started/runtime-variable#using-runtime-variables).

### Block dependencies

When reusing blocks, consider their dependencies:

* **Data loaders** - Ensure data sources are accessible from all pipelines
* **Transformers** - Make sure input data format is consistent
* **Exporters** - Verify output destinations are appropriate for each pipeline

## Managing shared blocks

### Viewing shared block usage

1. **Click on a shared block** in any pipeline
2. **Look for the sharing indicator** (`<> X pipelines`)
3. **Click the indicator** to see which pipelines are using the block
4. **Review the usage** to understand the block's impact

### Updating shared blocks

When you modify a shared block:

1. **Changes apply everywhere** - All pipelines using the block will be affected
2. **Test thoroughly** - Ensure changes work for all use cases
3. **Communicate changes** - Notify team members about significant modifications
4. **Version control** - Use Git to track changes and enable rollbacks if needed

### Deleting shared blocks

When you delete a shared block from the pipeline editor, choose whether to remove only the pipeline reference or remove the block file too:

* **Remove from this pipeline only** - Removes the block from the current pipeline and keeps the shared block file available to other pipelines.
* **Remove and delete the block file** - Permanently deletes the block file only when the block is not a replica and no other pipeline uses that file. This option is disabled when deleting the file would break another pipeline or replica.

### Detaching blocks when needed

If you need to customize a shared block for a specific pipeline:

1. **Use the detach feature** - See [Detaching blocks](/guides/blocks/detach-blocks) guide
2. **Create a copy** - This creates an independent version
3. **Customize freely** - Make changes without affecting other pipelines

## Common use cases

### Data loading blocks

Reuse data loading logic across multiple pipelines:

```python theme={"system"}
# Shared data loader block
@data_loader
def load_customer_data():
    # Load customer data from database
    # Use runtime variables for database connection
    return df
```

### Transformation blocks

Share common data transformations:

```python theme={"system"}
# Shared transformer block
@transformer
def clean_customer_data(df):
    # Standard customer data cleaning logic
    # Works with any customer dataset
    return cleaned_df
```

### Export blocks

Reuse export logic for different destinations:

```python theme={"system"}
# Shared exporter block
@data_exporter
def export_to_warehouse(df):
    # Export to data warehouse
    # Use runtime variables for destination configuration
    pass
```

## Related documentation

* [Block design overview](/design/blocks) - Understanding block architecture
* [Detaching blocks](/guides/blocks/detach-blocks) - Separating shared blocks
* [User-defined templates](/guides/blocks/user-defined-templates) - Creating custom block and pipeline templates
* [Runtime variables](/getting-started/runtime-variable) - Making blocks flexible


# SQL blocks
Source: https://docs.mage.ai/guides/blocks/sql-blocks

Execute SQL commands directly in your database, data warehouse, etc.

![SQL blocks](https://mage-ai.github.io/assets/development/mage-block.png)

## Credentials

Before starting, you need to add credentials so Mage can execute your SQL
commands.

Follow the steps for the database or data warehouse of your choice:

* [BigQuery](/integrations/databases/BigQuery#add-credentials)
* [ClickHouse](/integrations/databases/ClickHouse#add-credentials)
* [Databricks](/integrations/databases/Databricks#add-credentials) (Mage Pro only)
* [Druid](/integrations/databases/Druid#add-credentials)
* [DuckDB](/integrations/databases/DuckDB#add-credentials)
* [MySQL](/integrations/databases/MySQL#add-credentials)
* [Oracle](/design/data-loading#oracledb) (Mage Pro only)
* [PostgreSQL](/integrations/databases/PostgreSQL#add-credentials)
* [Redshift](/integrations/databases/Redshift#add-credentials)
* [Snowflake](/integrations/databases/Snowflake#add-credentials)
* [SQLite](/integrations/databases/SQLite#add-credentials) (Mage Pro only)
* [Microsoft SQL Server](/integrations/databases/MicrosoftSQLServer#add-credentials)
* [Trino](/integrations/databases/Trino#add-credentials)

***

## Add SQL block to pipeline

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.

***

## Configure SQL block

There are 4 - 5 fields that must be configured for each SQL block:

| Field | Required | Description |
| - | - | - |
| Data provider | Yes | The database or data warehouse you want to execute your SQL commands in. |
| Profile | Yes | When you add your credentials to the `io_config.yaml` file, you added them under a key. That key is called the profile. Choose which set of credentials you want this SQL block to use. For detailed information about setting up profiles, see the [IO Config Setup documentation](/development/io_config_setup). |
| Use raw SQL | No | You can write raw SQL and it’ll be executed as written in your data provider. |
| Database | Depends on data provider | Some data warehouses require that we explicitly state the name of the database we want to write to. If this is present, it’s required. The name of the table that is created follows this convention: `[database].[schema].[pipeline UUID]_[block UUID]`. |
| Schema to save | Yes | Every SQL block will save data to your data provider. The name of the table that is created follows this convention: `[schema].[pipeline UUID]_[block UUID]`. |
| Table | No | SQL blocks will automatically name your table for you using a naming convention (see the section `Automatic naming of tables` for more information). You can override this automatic naming convention by filling in a value in this field. |
| Write policy | Yes | How do you want to handle existing data with the same database, schema, and table name? See below for more information. |

#### Write policies

| Policy | Description |
| - | - |
| Append | Add rows to the existing table. |
| Replace | Delete the existing data. |
| Fail | Raise an error during execution. |

#### YAML configuration

You can also modify block configuration in pipeline's `metadata.yaml` file. Each block has a `configuration` field.

Example configuration

```yaml theme={"system"}
  configuration:
    data_provider: sqlserver
    data_provider_profile: default
    data_provider_schema: ''
    export_write_policy: append
    limit: 1000
    limit_in_pipeline_run: 1
    use_raw_sql: false
```

In addition to the fields mentioned in the table above. Here are some extra fields that can be included in the configuration:

| Field | Required | Description |
| - | - | - |
| limit | No | The maximum number of rows to return in notebook. |
| limit\_in\_pipeline\_run | No | The maximum number of rows to return when running the block in the pipeline run. |

#### Dynamic `data_provider_profile`

You can configure the `data_provider_profile` in a SQL block to resolve dynamically at runtime using [Mage variable syntax](/development/variables/referencing-variables).

This is done by modifying the pipeline’s `metadata.yaml` directly.

**Example:**

```yaml theme={"system"}
configuration:
  data_provider: snowflake
  data_provider_profile: '{{ variables("DB_PROFILE") }}'
  data_provider_schema: analytics
  export_write_policy: append
```

When the pipeline runs, the profile name will be resolved from the referenced variable (environment variable, pipeline variable, or secret variable).

***

![SQL block monsters](https://mage-ai.github.io/assets/development/sql-block-monsters.png)

***

## Automatically created tables

Each SQL block will create a table in the data provider of your choice.

When you run a block, it’ll execute your SQL command, then store the results in
a table created in your database or data warehouse.

### Using raw SQL

> [!WARNING]
>
> If you’re using version `0.9.69` or less and interpolating a SQL command from an upstream block that uses raw SQL:
> if the downstream block’s SQL command is referencing the upstream block using a variable such as `{{ df_1 }}` and ends the statement with a semi-colon (`;`), there is an interpolation error.
> To work around the error, remove the semi-colon `;`.
> A fix will be available in `0.9.70`.

If you toggle this setting, you’re responsible for writing the `CREATE TABLE` command and the
`INSERT` command.

For example, if a table already exists then you can write the `INSERT` statement:

```sql theme={"system"}
INSERT INTO mage.users
SELECT 1 AS id, 'Urza' AS username;
```

If the table doesn’t exist yet, you can write both the `CREATE TABLE` statement
and the `INSERT` statement:

```sql theme={"system"}
CREATE TABLE IF NOT EXISTS mage.users (
    id BIGINT
    , username VARCHAR(255)
);

WITH users AS (
    SELECT
        1 AS id
        , 'Urza' AS username
)

INSERT INTO mage.users
SELECT
    *
FROM users;
```

This SQL query will create a table named `mage.users` with 2 columns: `id` as a `BIGINT`
and `username` as a `VARCHAR(255)`.

Then, it’ll insert a single row into that table.

#### Required SQL statements

When writing raw SQL, you must use at least 1 of the following statements:

* `SELECT`
  ```sql theme={"system"}
  SELECT 1;
  ```
* `INSERT`
  ```sql theme={"system"}
  INSERT INTO some_table
  SELECT 1;
  ```
* `CREATE TABLE`
  ```sql theme={"system"}
  CREATE TABLE some_table (id BIGINT);
  ```
* `DROP TABLE`
  ```sql theme={"system"}
  DROP TABLE some_table;
  ```
* `UPDATE`
  ```sql theme={"system"}
  UPDATE some_table
  SET col1 = 'value1'
  WHERE col2 = 'value2'
  ```

#### Multiple SQL statements

You can execute multiple SQL statements in a SQL block. Separate your SQL statements using a semi-colon (`;`).

### Automatic naming of tables

If you don’t choose the setting for using raw SQL,
the name of this automatically created table follows these conventions:

* If `Database` field is configured:
  `[database].[schema].[pipeline UUID]_[block UUID]`
* If no `Database` field is configured: `[schema].[pipeline UUID]_[block UUID]`

Where pipeline UUID is the name of the current pipeline you’re editing.

Where block UUID is the name of the SQL block you are running.

### Upstream blocks

If your SQL block depends on upstream blocks that aren’t SQL blocks (e.g. Python
code blocks), then those blocks will also automatically create tables.

The name of those tables follows the same naming convention mentioned above.

***

## Variables

All SQL blocks have the following variables they can access in their query:

### `{{ execution_date }}`

The date and time the block is run.

**Example**

```sql theme={"system"}
SELECT '{{ execution_date }}' AS today
```

**Result**

| `today` |
| :- |
| `2022-09-24 23:01:08.376057` |

***

If a SQL block has 1 or more upstream blocks, then they have access to their
parent blocks’ output using the following variable:

### `{{ df_1 }}`

Depending on how many upstream blocks there are, the variable name changes. For
example, if there are 3 upstream blocks then there are 3 variables that can be
accessed:

* `{{ df_1 }}`
* `{{ df_2 }}`
* `{{ df_3 }}`

The SQL block UI will display which variable maps to which upstream block. By
convention, the 1st added upstream block will be `{{ df_1 }}`, and every
upstream block added after that will have an incrementing number in the variable
name after the prefix `df_`.

**Example**

```sql theme={"system"}
SELECT a.id , b.username FROM {{ df_1 }} AS a LEFT JOIN {{ df_2 }} AS b ON
a.id = b.user_id LIMIT 1
```

**Result**

| `id` | `username` |
| :- | :- |
| `1` | `Sorcerer supreme` |

***

## Variable interpolation

> [!NOTE]
>
> Available in versions >= `0.9.59`

Interpolate values in the block’s code using data from:

1. Upstream block output
2. Variables
   1. Global variables
   2. Pipeline variables
   3. Runtime variables
3. Environment variables

### Upstream block output

Use the data from 1 or more upstream block’s output by using the `block_output` function.

![Interpolate block output dataz](https://mage-ai.github.io/assets/blocks/interpolate-block-output.png)

#### `block_uuid`

The UUID of the upstream block to get data from.
If argument isn’t present, data from all upstream blocks will be fetched.

| | |
| - | - |
| Optional | ✅ |
| Type | `str` |
| Example | `'data_loader_block'` |

#### `parse`

A lambda function to parse the data from an upstream block’s output.
If the `parse` argument isn’t present, then the fetched data from the upstream block’s output
will be interpolated as is.

| | |
| - | - |
| Optional | ✅ |
| Type | `function` |
| Example | `lambda data, variables: data['runtime'] * variables['tries']` |

**Arguments**

* `data`

  If the `block_uuid` argument isn’t present, then the 1st argument in the lambda function is
  a list of objects.

  The list of objects contain the data from an upstream block’s output.
  The positional order of the data in the list corresponds to the current block’s upstream blocks
  order.

  For example, if the current block has the following upstream blocks with the following output:

  1. `load_api_data`: `[1, 2, 3]`
  2. `load_users_data`: `{ 'mage': 'powerful' }`

  Then the 1st argument in the lambda function will be the following list:

  ```python theme={"system"}
  [
      [1, 2, 3],
      { 'mage': 'powerful' },
  ]
  ```

  | | |
  | - | - |
  | Optional | ❌ |
  | Type | If `block_uuid` argument is present, then the type depends on the output from that block. If `block_uuid` isn’t present, then the type is `list`. |
  | Example | `{ 'mage': 'powerful' }` |

* `variables`
  A dictionary containing pipeline variables and runtime variables.

  | | |
  | - | - |
  | Optional | ❌ |
  | Type | `dict` |
  | Example | `{ 'fire': 40 }` |

#### Example

With `block_uuid`

```sql theme={"system"}
SELECT
    '{{ block_output("magic_config", parse=lambda data, _vars: data["power"]) }}' AS power
    , '{{ block_output("magic_config", parse=lambda data, _vars: data["spell"]) }}' AS spell
FROM mage.core_spells
```

```sql theme={"system"}
SELECT
    '{{ block_output("data_loader_block", parse=lambda data, variables: data["runtime"] * variables["tries"]) }}' AS data
FROM mage.core_spells
```

```sql theme={"system"}
SELECT
    '{{ block_output("load_recent_user", parse=lambda user, _variables: user["id"]) }}'
FROM mage.core_spells
```

Without `block_uuid`

```sql theme={"system"}
SELECT
    '{{ block_output(parse=lambda data, _vars: data[0]["power"]) }}' AS power
    , '{{ block_output(parse=lambda data, _vars: data[0]["spell"]) }}' AS spell
FROM mage.core_spells
```

```sql theme={"system"}
SELECT
    '{{ block_output(parse=lambda data, variables: data[0]["runtime"] * variables[0]["tries"]) }}' AS data
FROM mage.core_spells
```

```sql theme={"system"}
SELECT
    '{{ block_output(parse=lambda user, _variables: user[0]["id"]) }}'
FROM mage.core_spells
```

### Variables

Interpolate values from a dictionary containing keys and values from:

1. Global variables
2. Pipeline variables
3. Runtime variables

#### Example

```sql theme={"system"}
SELECT
    '{{ variables("mana") }}' AS mana
FROM mage.dual_lands
```

### Environment variables

Interpolate values from the environment variables.

#### Example

```sql theme={"system"}
SELECT
    '{{ env_var("ALLOW_POWER_9") }}' AS vintage
FROM mage.mox
```

***

## Executing pure SQL

> [!NOTE]
>
> Available in versions >= `0.9.59`

Mage automatically pre-processes your SQL command before sending it to the final destination.

The pre-process extracts and structures the command so that Mage can build upstream tables and
properly display preview data.

However, there are scenarios where the SQL command you write must be executed as is.
You can disable Mage’s pre-processing by opening a SQL block and checking the box labeled
**Use raw SQL** and **Disable query preprocessing**.

![Execute pure SQl commands](https://mage-ai.github.io/assets/blocks/execute-pure-sql.gif)


# Transformer blocks
Source: https://docs.mage.ai/guides/blocks/transformer-blocks

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

* [Overview](#overview)
  * [Tutorial](#tutorial)
  * [Core Concepts](#core-concepts)
* [Building Transformer Actions](#building-transformer-actions)
* [Transformer Action Reference](#transformer-action-reference)
* [Appendix: Filter Syntax](#appendix-filter-syntax)
* [Appendix: Column Types](#appendix-column-types)

## Tutorial

To introduce how the Transformer Actions library works, we will perform some cleaning actions on the Titanic Survival dataset. Open the Mage app and create a new pipeline. Add a data loader with the following code to load the Titanic Survival dataset:

```python theme={"system"}
from mage_ai.io.file import FileIO
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_file(**kwargs) -> DataFrame:
    """
    Template for loading data from filesystem.
    """
    filepath = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
    return FileIO().load(filepath)
```

Run this block to load the Titanic Survival dataset into your pipeline.

You'll notice that there are three columns that have missing values: "Cabin", "Age", "Embarked". Let's perform imputation to fill in these missing values using the Transformer Action Library. Create a new transformer block below, and select the "Fill in missing values" template.

![Image of transformer block template drop down highlighting "Fill in missing values"](https://mage-ai.github.io/assets/blocks/transformers/transformer_actions_impute.png)

You should see the following code in your new transformer block:

```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis, ImputationStrategy
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
    """
    Execute Transformer Action: ActionType.IMPUTE
    """
    action = build_transformer_action(
        df,
        action_type=ActionType.IMPUTE,
        arguments=df.columns,  # Specify columns to impute
        axis=Axis.COLUMN,
        options={'strategy': ImputationStrategy.CONSTANT},  # Specify imputation strategy
    )

    return BaseAction(action).execute(df)
```

Let's break this code down:

* `build_transformer_action()` creates a transformer action **payload**, which is a dictionary that describes settings for the action to complete. The arguments passed in are
  * `df` - the data frame to transform, used to infer column types
  * `action_type` - specifies the action to perform. In this case, we are performing an `IMPUTE` action.
  * `arguments` - specifies the columns to perform the action on. Currently, the action is configured to impute all columns of the data frame. Since we only want to transform "Cabin", "Age", and "Embarked", we will update this entry to only contain these column names.
  * `axis` - specifies which axis to perform the action on. Imputing values is done on a per column basis, so this is a column transformation.
  * `options` - these are extra settings to specify for this action. For an impute transformation, the main option to specify is `'strategy'`, dictating the imputation strategy to use. See [Fill in Missing Values](#fill-in-missing-values) for details on the valid imputation strategies. For now we will use `MODE`, which specifies to fill empty values with the most frequent value per column.

The modified transformer block is:

```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis, ImputationStrategy
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
    """
    Execute Transformer Action: ActionType.IMPUTE
    """
    action = build_transformer_action(
        df,
        action_type=ActionType.IMPUTE,
        arguments=['Cabin', 'Age', 'Embarked'],  # Specify columns to impute
        axis=Axis.COLUMN,
        options={'strategy': ImputationStrategy.MODE},  # Specify imputation strategy
    )

    return BaseAction(action).execute(df)
```

Execute this transformer - you should notice that all null values are now filled with the mode value of each respective column! After the payload is created by `build_transformer_action()`, the payload is used to construct the `BaseAction` object, representing a reusable instance of the transformation. Then the transformation is executed on the input data frame using `execute()`, returning the transformed data frame when done.

Suppose then you're interested in computing the average fare and age by each location of embarkation - Southampton (S), Cherbourg (C), and Queenstown (Q). Transformer Actions can help us here as well - create a new transformer using the "Aggregate/Aggregate by average value" template.

![Image of transformer block template drop down highlighting "Aggregate/Aggregate by average value"](https://mage-ai.github.io/assets/transformer_actions_agg.png)

You should see the following code in your new transformer block.

```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
    """
    Execute Transformer Action: ActionType.AVERAGE
    """
    action = build_transformer_action(
        df,
        action_type=ActionType.AVERAGE,
        action_code='',  # Enter filtering condition on rows before aggregation
        arguments=[],  # Enter the columns to compute aggregate over
        axis=Axis.COLUMN,
        options={'groupby_columns': []},  # Enter columns to group by
        outputs=[
            # The number of outputs below must match the number of arguments
            {'uuid': 'new_aggregate_column_1', 'column_type': 'number_with_decimals'},
            {'uuid': 'new_aggregate_column_2', 'column_type': 'number_with_decimals'},
        ],
    )

    return BaseAction(action).execute(df)
```

As before, we can fill in the settings we need for performing this aggregation by average value:

* `action_code` is used by aggregation-type transformer actions to filter rows of the dataset before performing the aggregation. For example, if the "Embarked" column had null values, we could filter them out using the filter `'Embarked != null'`. In this case we won't perform any filtering. See [Filter Syntax](#appendix-filter-syntax) for more details on valid filter syntax.
* Since we want to aggregate over "Fare" and "Age", our arguments are `['Fare', 'Age']`
* In `options`, the only parameter is `groupby_columns` which is the list of the columns to group by when performing the aggregation. In this case, the column we want to use to form groups is "Embarked"
* `outputs` is used to specify the name and type of the new columns created by the aggregation. As there are two columns in `arguments` that we want to perform the aggregation over, there should be two corresponding entries in `outputs`.

The resulting template after these edits are:

```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer


@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
    """
    Execute Transformer Action: ActionType.AVERAGE
    """
    action = build_transformer_action(
        df,
        action_type=ActionType.AVERAGE,
        action_code='',  # Enter filtering condition on rows before aggregation
        arguments=['Fare', 'Age'],  # Enter the columns to compute aggregate over
        axis=Axis.COLUMN,
        options={'groupby_columns': ['Embarked']},  # Enter columns to group by
        outputs=[
            # The number of outputs below must match the number of arguments
            {'uuid': 'avg_fare_by_embarked_loc', 'column_type': 'number_with_decimals'},
            {'uuid': 'avg_age_by_embarked_loc', 'column_type': 'number_with_decimals'},
        ],
    )

    return BaseAction(action).execute(df)
```

Run this transformer, and your output data frame should now include two new columns named "avg\_fare\_by\_embarked\_loc" and "avg\_age\_by\_embarked\_loc" which contain the average fare and average age by embarkation location. Each row gets a new value corresponding to the group which that row falls into.

This workflow can be summarized to describe how Transformer Actions can be used effectively:

1. Identify a transformation that may need to be made to your data
2. Create a transformer using the transformer action that corresponds to your template
3. Fill in the parameters for the transformer action to build your transformer action payload
4. Build a `BaseAction` object using the payload and execute the action to get the transformed data frame

## Core Concepts

The core object of any transformer action is the **payload**, a JSON object that describes the transformation to perform alongside settings for that transformation. A sample transformer action payload is shown below:

```json theme={"system"}
{
    "action_type": "impute",
    "action_arguments": ["age", "name"],
    "action_options": {
        "strategy": "constant",
        "value": "32"
    },
    "axis": "column",
    "action_variables": {
        "age": {
            "uuid": "age",
            "column_type": "number"
        },
        "name": {
            "uuid": "hourly_salary",
            "column_type": "text"
        }
    }
}
```

The above transformer action imputes missing values in the "age" and "hourly\_salary" columns with the constant value `32`. Here is a list of the keys used in transformer action payloads:

| Key | Description |
| - | - |
| axis | Axis to perform the transformation on. Either 'row' or 'column'. |
| action\_type | Type of action to perform. See [Transformer Action Reference](#transformer-action-reference) for the list of action types. |
| action\_code | Text field for specifying any code such as filters. |
| action\_arguments | The columns to perform the transformation on. |
| action\_options | Other settings to provide to the transformer action. See [Transformer Action Reference](#transformer-action-reference) for specific details on which actions require which options. |
| action\_variables | Variable metadata for each input argument (column). Specifies column name and type. |
| outputs | Variable metadata for each output column. Specifies column name and type. See [Column Types](#appendix-column-types) for information on possible column types to specify. |

This JSON formatted payload is difficult to generate so it is recommended to use the `build_transformer_action` factory to generate the payload. The above snippet generates the same payload as above:

```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action

df = pd.DataFrame({...}) # your input data frame
payload = build_transformer_action(
    df,
    action_type=ActionType.IMPUTE,
    arguments=['age'],
    axis=Axis.COLUMN,
    options={'strategy': 'constants', 'values': 32},
)
```

See [Building Transformer Actions](#building-transformer-actions) for detailed information on using `build_transformer_action`.

The transformer action payload, once built, can be used to construct a `BaseAction` object. This object processes the payload to identify the transformations to apply a data frame. Through this action object, the transformer action can be applied to a data frame using the `execute` function:

```python theme={"system"}
payload = build_transformer_action(...)
transformed_df = BaseAction(payload).execute(df)
```

# Building Transformer Actions

***build\_transformer\_action*** - `build_transformer_action(df: DataFrame, action_type: ActionType | str, arguments: List[str], action_code: str, options: Dict, axis: Axis | str, outputs: List[Dict]) -> Dict:`

Source: `mage_ai.data_cleaner.transformer_actions.utils.build_transformer_action`

Builds transformer action payload from arguments. The output of this function can be passed
as input to `transformer_actions.base.BaseAction` in order to perform the requested transformation.

Note: Action variables are inferred from `arguments` and `df`.

* **Args**:
  * `df (DataFrame)`: The data frame to build a transformer action payload for.
  * `action_type (ActionType | str)`: Transformer action to perform.
  * `arguments (List[str], optional)`: Columns to perform this action on. Defaults to `[]`.
  * `action_code (str, optional)`: Special code or query to execute with action. Defaults to `''`.
  * `options (Dict, optional)`: Options specifying behavior of action. Defaults to `{}`.
  * `axis (Union[Axis, str], optional)`: Axis of the data frame to apply the action to. Defaults to `Axis.COLUMN`.
  * `outputs (List[Dict], optional)`: Specifies metadata of newly created columns. Defaults to `[]`.
* **Returns**: (`Dict`) Transformer action payload to use with `transformer_actions.base.BaseAction`
* **Example**:
  ```python theme={"system"}
    action = build_transformer_action(
        df,
        action_type=ActionType.FIRST,
        action_code='retailer != null and retail_price >= 0',  # Enter filtering condition on rows before aggregation
        arguments=['retail_price', 'review'],  # Enter the columns to compute aggregate over
        axis=Axis.COLUMN,
        options={'groupby_columns': ['retailer']},  # Enter columns to group by
        outputs=[
            # The number of outputs below must match the number of arguments
            {'uuid': 'first_retail_price', 'column_type': 'number_with_decimals'},
            {'uuid': 'first_review', 'column_type': 'category'},
        ],
    )
  ```

# Transformer Action Reference

* [Building Transformer Actions](#building-transformer-actions)
* [Transformer Action Reference](#transformer-action-reference)
* [Column Actions](#column-actions)
  * [Aggregation Actions](#aggregation-actions)
  * [Formatting Actions](#formatting-actions)
    * [Clean Column Names](#clean-column-names)
    * [Fix Syntax Errors](#fix-syntax-errors)
    * [Reformat Values](#reformat-values)
  * [Column Removal Actions](#column-removal-actions)
    * [Select Columns](#select-columns)
    * [Remove Columns](#remove-columns)
  * [Row Shifting Actions](#row-shifting-actions)
    * [Shift Up](#shift-up)
    * [Shift Down](#shift-down)
  * [Feature Scaling Actions](#feature-scaling-actions)
    * [Normalize Data](#normalize-data)
    * [Standardize Data](#standardize-data)
  * [Other Column Actions](#other-column-actions)
    * [Difference](#difference)
    * [Fill In Missing Values](#fill-in-missing-values)
    * [Remove Outliers](#remove-outliers)
* [Row Actions](#row-actions)
  * [Drop Duplicates](#drop-duplicates)
  * [Filter](#filter)
  * [Remove Rows](#remove-rows)
  * [Sort](#sort)
* [Appendix: Filter Syntax](#appendix-filter-syntax)
* [Appendix: Column Types](#appendix-column-types)

# Column Actions

## Aggregation Actions

Applies some aggregation function over groups determined by values in some columns. A set of grouping columns are provided, whose unique combinations of values are used to form groups. For example, consider the dataset below:

| Paid Dues | Due Amount |
| - | - |
| Yes | \$5.00 |
| No | \$15.20 |
| Yes | \$3.50 |
| Yes | \$1.28 |
| No | \$25.03 |

The data could be grouped by "Paid Dues" to form the following groups:

| Paid Dues | Due Amount |
| - | - |
| Yes | \$5.00 |
| Yes | \$3.50 |
| Yes | \$1.28 |

| Paid Dues | Due Amount |
| - | - |
| No | \$15.20 |
| No | \$25.03 |

Then an aggregation function is applied to each group over the input columns. In the example, we can compute the total dues by group by aggregating over "Due Amount":

| Paid Dues | Total Due Amount |
| - | - |
| Yes | \$9.78 |
| No | \$40.23 |

Then the output of this aggregation is used to create a new column for each input column, where each row receives the corresponding aggregate of the group it falls into. In the example above, the final output data frame would be:

| Paid Dues | Due Amount | Total Due Amount by Payment Status |
| - | - | - |
| Yes | \$5.00 | \$9.78 |
| No | \$15.20 | \$40.23 |
| Yes | \$3.50 | \$9.78 |
| Yes | \$1.28 | \$9.78 |
| No | \$25.03 | \$40.23 |

All of these transformer actions have the same set of arguments when using `build_transformer_action`, but have different types corresponding to the different aggregation functions that could be applied to the groups.

**Example**

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.AVERAGE,
    action_code='brand != null',
    arguments=['revenue', 'cost'],
    axis=Axis.COLUMN,
    options={'groupby_columns': ['brand']},
    outputs=[
        {'uuid': 'avg_rev_per_brand', 'column_type': 'number_with_decimals'},
        {'uuid': 'avg_cost_per_brand', 'column_type': 'number_with_decimals'},
    ],
)
```

**Args**

* ***action\_code:*** Filter for selecting rows before aggregation is performed. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.
* ***arguments:*** Columns to compute aggregate values over. For each argument a new column will be created to store the aggregates values over the column.
  * If no arguments are provided, no aggregation is performed
* ***options*:**
  * `groupby_columns` (optional) - list of columns to form groups with when aggregating. A group consists of each unique combination of values chosen from each of the columns specified in this option.
    * If this is empty, aggregation is performed over the first column specified in *arguments* (only if *arguments* is nonempty - if empty no aggregation is performed) and only one column is added

```
The following options are used as a shortcut to simplify filtering over time windows.
Let $A$ and $B$ be datetime columns, and let $\text{window}$ be an integer describing the length of
a time window in seconds. If the following options are specified, rows for which the time value for
$B \in [A, A + \text{window}]$ are selected (All rows where value for $B$ must (a) be greater than
or equal to the value for $A$ and (b) less than or equal to the value $A+window$).
```

* `timestamp_feature_a` (optional) - time feature A (a column in the data frame)
* `timestamp_feature_b` (optional) - time feature B (a column in the data frame)
* `window` (optional) - max time window between feature A and feature B in seconds

If some (or none) of these options are specified, then the filtering is not performed. Regardless if this filtering is performed, *action\_code* filtering is still performed.

* ***outputs*:** Specifies the metadata for each output column of the aggregation (corresponding to the aggregation created from input argument in *arguments*). This array must be the same length as *arguments*, and each entry must contain the following metadata:
  * `uuid (string)`: Name of the new column
  * `column_type (string | ColumnType)`: Type of the new column ( See `mage_ai.data_cleaner.column_types.constants.ColumnType` for options)

**Aggregation Functions**
These are the possible aggregations that can be applied to each group per each input column. To use the aggregation function specify the corresponding action type in `build_transformer_action`.

| Name | Action Type Enum | Description |
| - | - | - |
| ***Average*** | `ActionType.AVERAGE` | Averages input column value over each group |
| ***Count Distinct*** | `ActionType.COUNT_DISTINCT` | Counts the number of distinct rows in each group per input column |
| ***Count*** | `ActionType.COUNT` | Counts the number of rows in each group per input column |
| ***First*** | `ActionType.FIRST` | For each input column, selects first item in each group (as appearing in the order of the original data frame) |
| ***Last*** | `ActionType.LAST` | For each input column, selects last item in each group (as appearing in the order of the original data frame) |
| ***Maximum*** | `ActionType.MAX` | Gets the maximum value per group for each input column |
| ***Median*** | `ActionType.MEDIAN` | Gets the median value per group for each input column |
| ***Minimum*** | `ActionType.MIN` | Gets the maximum value per group for each input column |
| ***Sum*** | `ActionType.SUM` | Sums over all values in each group per input column |

## Formatting Actions

These transformer actions involve reformatting column names or the values in columns

### Clean Column Names

Cleans column names according to the following rules:

1. Names are converted to snake case
2. All wrapping whitespace or underscores are removed
3. All non-alphanumeric characters (except "\_") are removed
4. Numbers are prefixed with "number\_"
5. All Python keywords are postfixed by "\_"

Snake case is chosen as

1. all characters are lower case
2. no whitespace is used (which makes referring to columns in code easier)

| Original Name | Cleaned Name | Notes |
| - | - | - |
| good\_name | good\_name | columns in snake case are left unchanged |
| bad case | bad\_case | |
| %@#342%34@@#342 | number\_34234342 | Symbols are removed first, then number is cleaned |
| PascalCaseColumn | pascal\_case\_column | |
| camelCaseText | camel\_case\_text | |
| \_\_\_snake\_case\_\_\_ | snake\_case | Surrounding underscores are stripped |

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.CLEAN_COLUMN_NAME,
    arguments=df.columns,
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** Columns whose name to clean. If empty, no columns names are cleaned.

### Fix Syntax Errors

Marks syntax errors in column values. Syntax errors are defined as values that are improperly formatted or of the incorrect type. For example:

* A number in a text column
* An improperly formatted email, phone number, or zip code
* An improperly formatted date

Values that break these syntax errors are currently marked as:

* `"invalid"` if column is of categorical or string type
* `pandas.NaT` if column is a datetime
* `np.nan` if column is a number type

*WIP*: Intelligently fix syntax errors in these columns automatically

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.FIX_SYNTAX_ERRORS,
    arguments=df.columns,
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** Columns to mark syntax errors for. If empty no columns are checked.

### Reformat Values

Reformats values in column based on requested action. The currently supported reformats are:

* Standardize capitalization (`reformat = 'caps_standardization'`): Forces text column to follow a single capitalization strategy (either lowercase or uppercase)
* Convert currencies to a number (`reformat = 'currency_to_num'`): Converts a currency value (by default stored as a string) to a decimal number. If unable to convert, doesn't perform conversion
  * Supported currencies: **\$, CAD, £, €, ¥, Rs, 元**
    | Original String | Parsed Decimal | Notes |
    | - | - | - |
    | "  \$ 10000" | 10000 | |
    | "- ¥ 22.324523" | -22.324523 | Negation in front of symbol supported |
    | " 10000 元   " | 10000 | Currencies with postfixed symbols are supported |
    | "0.42 €" | 0.42 | |
    | "-  3.42032 CAD" | -3.42032 | |
    | "Rs - 100000.23" | -100000.23 | Negation after symbol is supported |
* Convert string to datetime (`reformat = 'date_time_conversion'`): Converts a string value representing a datetime to a `pandas.Timestamp` object if correctly formatted, else converts to `None`.
* Trim whitespace (`reformat = 'trim'`): Removes leading and trailing whitespace from text columns.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REFORMAT,
    arguments=[],
    axis=Axis.COLUMN,
    options={'reformat': 'caps_standardization', 'capitalization': 'lowercase'},
)
```

**Args**

* ***arguments:*** Columns to apply the reformat action to
* ***options:***
  * `reformat` (optional): The reformat action to apply to the specified columns. Can be:
    * `None` - no reformat is applied
    * `'caps_standardization'` - standardize capitalization
    * `'currency_to_num'` - convert currency string to number
    * `'date_format_conversion'` - convert datetime string to `pandas.Timestamp`
    * `'trim'` - remove leading and trailing whitespace from text columns
  * `capitalization` (optional): Specifies the capitalization strategy to use when standardizing capitalization. This argument is ignored unless `reformat = "caps_standardization"`. Options are `['lowercase', 'uppercase']`.

## Column Removal Actions

### Select Columns

Keeps only the specified columns; removes all other columns from data frame.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SELECT,
    arguments=[],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to keep in the data frame. If empty, all columns are removed.

### Remove Columns

Drops the specified columns.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REMOVE,
    arguments=[],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to remove in the data frame. If empty, no columns are removed.

## Row Shifting Actions

These actions copy a column with all rows shifted up or down by a certain amount. When rows are shifted, one or more values are deleted and some new values have to be filled in. The fill-in data types are:

| Type | Fill Value |
| - | - |
| Numeric | `np.nan` |
| Datetime | `pandas.NaT` |
| Other Types | `None` |

### Shift Up

Shifts all rows in a column up by one. The first entry is deleted and the last entry is turned to a null type.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SHIFT_UP,
    arguments=['Name'],
    axis=Axis.COLUMN,
    outputs=[{'uuid': 'shifted_name', 'type': 'text'}],
)
```

**Args**

* ***arguments:*** The column to shift up. At most one column can be shifted at once:
  * If multiple columns are provided as arguments, only the first is shifted up
  * If no columns are provided then no shifting is performed
* ***outputs:*** Metadata for the newly added upshifted column. Must have at most a single entry containing the following information:
  * `uuid`: Name of the new upshifted column
  * `type`: Data type of the new upshifted column

### Shift Down

Shifts all rows up by *periods* rows. The last *periods* entries are deleted and the first *periods* are converted to a null type.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SHIFT_DOWN,
    arguments=['created_time'],
    axis=Axis.COLUMN,
    options={'periods': 3},
    outputs=[{'uuid': 'downshifted_created_time', 'type': 'datetime'}],
)
```

**Args**

* ***arguments:*** The column to shift down. At most one column can be shifted at once:
  * If multiple columns are provided as arguments, only the first is shifted down
  * If no columns are provided then no shifting is performed
* ***options:***
  * `periods` (optional): Specifies how many rows to shift down. Defaults to 1.
  * `groupby_columns` (optional): Specifies columns to group by. If specified, shifts each *group* by *periods* rows instead of the entire data frame. The resulting column is then returned. If unspecified no grouping is performed. Multiple columns can be specified to group by.
* ***outputs:*** Metadata for the newly added downshifted column. Must have at most a single entry containing the following information:
  * `uuid`: Name of the new downshifted column
  * `type`: Data type of the new downshifted column

## Feature Scaling Actions

These actions are used to normalize the range of independent variables or features of data. These actions are  generally performed during the data preprocessing step.

| Type | Formula |
| - | - |
| Normalization | `(X-Xmin)/(Xmax-Xmin)` |
| Standardization | `(X-Xmean)/Xstd` |

### Normalize Data

Perform Normalization on the desired columns of a dataset.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.NORMALIZE,
    arguments=['Column1','Column2'],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to Normalize.
  * If no columns are provided then no Normalization is performed

### Standardize Data

Perform Standardization on the desired columns of a dataset.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.STANDARDIZE,
    arguments=['Column1','Column2'],
    axis=Axis.COLUMN,
)
```

**Args**

* ***arguments:*** The columns to Standardize.
  * If no columns are provided then no Standardization is performed

## Other Column Actions

### Difference

Adds a new column where each row is the difference between every consecutive two rows in the original column.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.DIFF,
    arguments=['position'],
    axis=Axis.COLUMN,
    outputs=[{'uuid': 'delta_position', 'column_type': 'number_with_decimals'}],
)
```

**Args**

* ***arguments:*** The column to calculate difference on. At most one column can be specified:
  * If multiple columns are provided as arguments, only the first is used in calculated
  * If no columns are provided then no calculation
* ***outputs:*** Metadata for the newly added difference column. Must have at most a single entry containing the following information:
  * `uuid`: Name of the difference column
  * `type`: Data type of the difference column (most often will be a number)

### Fill In Missing Values

Imputes missing values using different strategies:

| Strategy | Description |
| - | - |
| Average | Fill missing values with the average per column. |
| Column | Fill missing values with values from another column. |
| Constant | Fill missing values with a placeholder constant. Can fill with a specific value; otherwise a default placeholder is chosen by column type. |
| Median | Fill missing values with the median per column. |
| Mode | Fill missing values with the most frequent value per column. |
| Random | Fill missing values with a randomly sampled non-null value from same column. |
| Sequential | Fill missing values with the previous row. Should only be used with timeseries columns. |

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.IMPUTE,
    arguments=df.columns,  # Specify columns to impute
    axis=Axis.COLUMN,
    options={'strategy': ImputationStrategy.CONSTANT},  # Specify imputation strategy
)
```

**Args**

* ***arguments:*** The columns to impute (fill in the missing values).
* ***options:***
  * `strategy` (required): Specifies strategy to use. See table above or enum `ImputationStrategy`
  * `value` (optional): Value to fill in missing value with. This argument is ignored unless `strategy = 'constant'`, in which case the constant value used to impute is `value`.
  * `timeseries_cols = []` (optional): Columns which specify the time series in the data frame. This argument is ignored unless `strategy = 'sequential'`, in which case the data frame is sorted along these columns to sequentially impute data in the order of the time series.

### Remove Outliers

Removes multidimensional outliers in the data frame by analyzing numerical columns. If there are no numerical columns, no check is performed. Null values are ignored when checking for outliers.

The following methods are supported for detecting outliers:

| Method | Description |
| - | - |
| `'itree'` | Uses the [Isolation Tree](https://scikit-learn.org/stable/modules/outlier_detection.html#isolation-forest) algorithm. Number of trees is set to 100. |
| `'lof'` | Uses the [Local Outlier Factor](https://scikit-learn.org/stable/modules/outlier_detection.html#local-outlier-factor) algorithm.  Number of neighbors for computing local density scales with amount of data and contamination rate. |
| `'auto'` | Automatically determines from above methods based on number of data points and dimensionality of the data. |

Notes:

* The contamination rate for both algorithms is estimated via the per-dimension IQR (which discounts multidimensional interactions but still estimates how many outliers there likely are)
* If the input data has more than 20 dimensions, the dimensionality is reduced using PCA

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REMOVE_OUTLIERS,
    arguments=df.columns,  # Specify columns to remove outliers from
    axis=Axis.COLUMN,
    options={'method': 'auto'},  # Specify algorithm to use for outlier removal
)
```

**Args**

* ***arguments:*** The set of columns to consider when checking for outliers. Of these columns, only numerical columns will be used to calculate outliers.
* ***options:***
  * `method` (optional): Specifies outlier detection method to use:

    * `'itree'`
    * `'lof'`
    * `'auto'`

    Defaults to `'auto'`.

# Row Actions

These are transformer actions that can be applied to each row of a data frame, and so have `axis = Axis.ROW`.

### Drop Duplicates

Drops duplicate rows from the data frame.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.DROP_DUPLICATE,
    arguments=[],
    axis=Axis.ROW,
    options={'keep': 'first'},
)
```

**Args**

* ***arguments:*** the columns along which comparisons will be made to check for duplicate rows. If empty, all columns will be used to perform comparison.
* ***options:***
  * `keep` (optional): specifies policy on keeping duplicates:

    * `'first'` - keeps the first occurrence of the row
    * `'last'` - keeps the last occurrence of the row
    * `False` - removes occurrences of duplicates

    Defaults to `'last'`.

### Filter

Filters row by boolean condition. Selects all rows that meets the filter condition.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.FILTER,
    axis=Axis.ROW,
    action_code='column1 != null and column2 <= 5'
)
```

**Args**

* ***action\_code:*** filter to apply to select rows. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.

### Remove Rows

Removes rows specified by their index.
**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.REMOVE,
    axis=Axis.ROW,
    options={'rows': []},
)
```

**Args**

* ***options:***
  * `rows`: List of the indices corresponding to the rows to remove. If empty, no rows are removed.

### Sort

Sorts rows in the specified order.

**Example**:

```python theme={"system"}
build_transformer_action(
    df,
    action_type=ActionType.SORT,
    arguments=[],
    axis=Axis.ROW,
    options={'ascending': True},
)
```

**Args**

* ***arguments:*** the columns whose value to consider while sorting rows. If empty then sort is not performed.
* ***options:***
  * `ascending` (optional): Specifies how to sort rows. If `True`, then rows are sorted in ascending order; if `False` sorted in descending order. Defaults to `True`.
    * If sorted in ascending order, null values are placed at the top of the data frame in the same order as they appeared in the original data frame
    * If sorted in descending order, null values are placed at the bottom of the data frame in the same order as they appeared in the original data frame
  * `ascendings` (optional): A list of booleans specifying if each column specified in *arguments* should be sorted in ascending order (must be the same length). If specified, this list will take precedence over the `ascending` option.

# Appendix: Filter Syntax

Valid filters are composed of clauses of the form:

```
[column_name] [op] [value or column_name]
```

* List of valid operations: `==, !=, <=, <, >, >=, contains, not contains`
  * `contains` (and `not contains`) checks if the value is a substring (or not a substring) of values in the column specified.
* The right hand side of each clause can either be a literal value (boolean, number, string) or another column to compare to.

Clauses can be connected using the `and` and `or` operators and grouped with parentheses.

```
(brand != null and value > 60) or (discounted == null)
```

**Rules on Naming Columns**

* If column names contain only alphanumeric characters and the special character '\_', then the raw column name can used to filter code.
  Example: `age != null and price_of_item >= 50.23`
* If columns contain whitespace, newlines, and some of the following special characters: `+=-*&^%$! ?~|<>(){}[],.`, then the quoted column name can be used to filter code. Any quote character can be used.
  Example: `("+= -*&^%$! ?~ |<>" == False and ' kas22d fe ($)' > 50.23) or ("dis>>> ??cou nted" contains "partial")`
* If columns contain any other special characters apart from those listed above, the column cannot be filtered. This is a fundamental limitation of pandas, see [here](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html?highlight=query#pandas.DataFrame.query:~:text=During%20parsing%20a,not%20be%20escaped) for more information.

# Appendix: Column Types

These are list of column types used in the Mage tool. One of these column types should be specified for every entry in the *options* argument of a transformer action. The enum source is `mage_ai.data_cleaner.column_types.constants.ColumnType`.

| Type | Enum | String Value | Description |
| - | - | - | - |
| Category | `ColumnType.CATEGORY` | "category" | Categorical variable that takes one of some finite set of values. This set has at most 255 entries. |
| High Cardinality Category | `ColumnType.CATEGORY_HIGH_CARDINALITY` | "category\_high\_cardinality" | Categorical variable that takes one of some finite set of values. This set has more than 255 entries. |
| Datetime | `ColumnType.DATETIME` | "datetime" | Datetime objects or strings representing a datetime. |
| Email | `ColumnType.EMAIL` | "email" | Strings that contains emails. |
| List | `ColumnType.LIST` | "list" | Lists of primitives (integers, strings, booleans, floats) of variable length. |
| Number | `ColumnType.NUMBER` | "number" | Integer numbers. Can be a Python integer, NumPy integer or a stringified integer. |
| Floating Point Number | `ColumnType.NUMBER_WITH_DECIMALS` | "number\_with\_decimals" | Floating point numbers. Can be a Python float, NumPy float, or a stringified float. |
| Phone Number | `ColumnType.PHONE_NUMBER` | "phone\_number" | Phone numbers (either in string form or integer form). |
| Text | `ColumnType.TEXT` | "text" | Text values, such as sentences or paragraphs. |
| Binary | `ColumnType.TRUE_OR_FALSE` | "true\_or\_false" | A variable that takes on two different values (including booleans). |
| Zip Code | `ColumnType.ZIP_CODE` | "zip\_code" | Zip codes (either in string form or integer form). |


# User-defined templates
Source: https://docs.mage.ai/guides/blocks/user-defined-templates

Learn how to create and use custom block templates and pipeline templates in Mage for better code reusability and standardization.

## Overview

User-defined templates allow you to create reusable code patterns for blocks and pipelines, enabling faster development and consistent coding standards across your organization. Templates can include placeholders for customization and can be shared across projects.

## Block Templates

### Creating block templates

Block templates are reusable code patterns that can be customized with variables. They help standardize common patterns across your organization.

**Steps to create a block template:**

1. **Navigate to Block templates** - In Mage OSS, select **Templates** in the left navigation. In Mage Pro, open **Library** and select **Templates**. Then open **Block templates**, or go directly to `/apps/templates/blocks`.
2. **Create new template** - Click "New block template" button
3. **Define template properties**:
   * **Template UUID** - Enter a unique identifier (e.g., `sql_data_loader`)
   * **Block type** - Select the type (Data loader, Transformer, Data exporter, etc.)
   * **Language** - Choose the programming language (Python, SQL, R, YAML, etc.)
4. **Edit template content** - Write the template code with placeholders
5. **Save template** - Click "Save template" to create the template

> [!NOTE]
>
> Template UUIDs can include letters, numbers, underscores, hyphens, and folder separators (`/`). They must start with a letter or number, cannot include empty, `.` or `..` folder segments, and must be 255 characters or fewer. The block type and language are locked after the template is created.

### Template File Structure

When you create a block template, Mage automatically creates the following file structure:

```
custom_templates/
└── blocks/
    └── {template_uuid}/
        ├── metadata.yaml    # Template metadata and configuration
        └── {template_uuid}.py  # Template code file
```

**Template files:**

* **`metadata.yaml`** - Contains template metadata, block type, language, and other configuration
* **`{template_uuid}.py`** - The actual template code file that users will customize

The code file extension follows the selected language. For example, Python templates use `.py`, SQL templates use `.sql`, R templates use `.r`, and YAML templates use `.yaml`.

**Example structure for `test_template`:**

```
custom_templates/
└── blocks/
    └── test_template/
        ├── metadata.yaml
        └── test_template.py
```

### Template Library Interface

The template library provides a centralized location to manage all your block and pipeline templates:

**Template Library Features:**

* **Template categories** - Filter templates by type (Blocks or Pipelines)
* **Block template categories** - Filter block templates by block type
* **Team pipeline templates** - Manage pipeline templates created from pipelines in your project; built-in starting points are available from the pipeline creation flow
* **Pipeline template discovery** - Search and filter the broader built-in catalog from **New pipeline** > **From a template**; the customization page remains focused on Team templates
* **Template preview** - View template details before using
* **Template management** - Edit, delete, or duplicate existing templates

**Accessing Templates:**

1. **From left navigation** - In Mage OSS, select **Templates**. In Mage Pro, open **Library** and select **Templates**. The default Templates page redirects to the block template library at `/apps/templates/blocks`.
2. **From pipeline editor** - Use "Browse templates" when adding new blocks
3. **From pipeline creation** - Select "From a template" when creating new pipelines
4. **Template organization** - Templates are organized by type and category for easy discovery

**Viewing Pipeline Templates:**

* **Open Pipeline templates** - Click the Pipelines tab in the template library or go directly to `/apps/templates/pipelines`
* **Manage team templates** - This customization page lists only reusable pipeline templates created and maintained in your Mage project
* **Browse built-in starting points** - From the Pipelines page, choose **New pipeline**, then **From a template** to browse Basics, General purpose, and Industry solutions alongside Team templates
* **View template details** - See template name, description, and metadata
* **Select a team template** - Click a Team template to inspect or customize it

### Using block templates

**Steps to use a block template:**

1. **Open pipeline editor** - Navigate to the pipeline where you want to add the block
2. **Add new block** - Click the "+" button to add a new block
3. **Browse templates** - Select "Browse templates" from the block creation options
4. **Choose a source** - Select **Core** for Mage's built-in templates or **Team** for templates created in your workspace
5. **Find a template** - Browse the folders or search the selected template source
6. **Preview the code** - Select a template to inspect its code in the read-only preview
7. **Add the template** - Click **Add to workflow**. Core templates are added directly; Team templates open the **Block name** dialog so you can enter a name and click **Save**.
8. **Customize as needed** - Modify the generated block code for your specific use case

> [!NOTE]
>
> The template browser preview is read-only. Add the template to your workflow before
> customizing its generated block code.

When you add a callback or conditional block from an existing block's add-on panel,
Mage opens the same template browser scoped to the matching block type.

The **Core > dbt** folder presents three dbt workflow starters:

* **Single model or snapshot (from file)** opens the file browser so you can choose an existing dbt file.
* **All models (w/ optional exclusion)** creates a dbt block configured to run all models, with optional selection or exclusion rules.
* **Generic dbt command** creates an editable dbt command block.

## Pipeline Templates

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=pipeline-template-catalog)

### Built-in pipeline template catalog

Mage Pro provides three built-in sources alongside templates created by your team:

* **Basics** are remotely provided example pipelines for learning Mage fundamentals and common block patterns.
* **General purpose** templates are starting points for recurring workflows such as ingestion, data quality, streaming, and AI context preparation.
* **Industry solutions** adapt common patterns to outcomes in areas such as retail, financial services, healthcare, manufacturing, and logistics.

Browse these templates from **Pipelines** > **New pipeline** > **Browse templates**. Mage loads lightweight catalog metadata while you browse and downloads a template's blocks only after you select it.

> [!NOTE]
>
> General purpose and Industry solution templates are production-oriented starting points, not zero-configuration demos. Review and adapt them before using them with production data.

Each generated pipeline begins with an **overview and setup** Markdown block that lists its required credentials, identifiers, columns, dependencies, expected output, and limitations.

Configure templates in two places:

* Put non-sensitive settings in each block's **USER CONFIGURATION** dictionary. Runtime variables can override matching lowercase settings when they must differ by run or environment.
* Put credentials in [`io_config.yaml`](/development/io_config_setup). Source and destination blocks select connections by their configured tags, so keep secrets out of block configuration and runtime variables.

Before a first run:

1. Read the **overview and setup** block.
2. Supply the required settings and create the named connection profiles.
3. Start in `dry_run` mode. Loaders return a sample DataFrame and exporters skip destination writes, so you can inspect the transformation without connecting to external systems.
4. Turn off `dry_run`, then run the blocks in order with the smallest representative input supported by the template.
5. Before scheduling, choose an idempotent write policy and add your organization's monitoring, recovery, privacy, and retention controls.

Catalog administrators configure the source with `MAGE_PRO_TEMPLATES_ENDPOINT` and the server-side `MAGE_PRO_TEMPLATES_API_TOKEN`. Catalog updates do not require a new Mage image, and Mage keeps the last successfully fetched catalog available during temporary refresh failures.

### Creating pipeline templates

Pipeline templates define complete pipeline structures that can be customized for different use cases.

**Steps to create a pipeline template:**

1. **Navigate to Team pipeline templates** - Open **Library** in the left navigation, select **Templates**, then **Pipeline templates**, or go directly to `/apps/templates/pipelines`. This customization page lists only templates created and maintained by your team.
2. **Create new template from an existing pipeline** - From the pipeline list, right-click the pipeline you want to reuse and select the action to create a template. This opens `/apps/templates/pipelines/new?pipeline_uuid={pipeline_uuid}`.
3. **Configure template settings** - Set the template UUID, name, description, and other properties.
4. **Save template** - The pipeline structure will be saved as a reusable template.

> [!NOTE]
>
> Pipeline template UUIDs follow the same rules as block template UUIDs: letters, numbers,
> underscores, hyphens, and folder separators (`/`) are allowed, the value must start with a
> letter or number, `.` and `..` folder segments are rejected, and the value must be 255
> characters or fewer.

**Example pipeline template structure:**

```
custom_templates/pipelines/
└── example_pipeline_template/
    ├── metadata.yaml
    └── triggers.yaml
```

### Using pipeline templates

**Steps to use a pipeline template:**

1. **Navigate to the pipelines page** in your project
2. **Click "New"** button to open the pipeline creation menu
3. **Select "From a template"** from the dropdown options
4. **Choose a source** - Start in All templates, or select Basics, General purpose, Industry solutions, or Team
5. **Find a template** - Search the catalog or apply the filters available for the selected source
6. **Choose your template** from the available pipeline templates
7. **Review the configuration guidance** - For a General purpose or Industry solution template, open its first **overview and setup** Markdown block and supply the required credentials, identifiers, schemas, and business rules
8. **Customize the pipeline** - Modify blocks, connections, and settings as needed
9. **Save the pipeline** with your specific configuration

> [!NOTE]
>
> **Note**: Pipeline templates are created from existing pipelines using the right-click context menu. This allows you to reuse successful pipeline patterns across your projects.

## Best Practices

### Template design

* **Keep templates focused** - Each template should serve a specific purpose
* **Use clear variable names** - Make variable purposes obvious
* **Include comprehensive documentation** - Explain usage and customization options
* **Test with different configurations** - Ensure templates work with various inputs
* **Follow coding standards** - Maintain consistent style and practices

### Template maintenance

* **Regular updates** - Keep templates current with latest Mage features
* **User feedback** - Collect and incorporate user suggestions
* **Performance optimization** - Monitor and improve template performance
* **Security review** - Ensure templates follow security best practices


# Community Examples
Source: https://docs.mage.ai/guides/community-examples

A collection of Mage projects and examples created by our community.

## Written Tutorials

### [Simplifying Data Pipelines With Mage Data Integrations](https://ethanbrown.substack.com/p/simplifying-data-pipelines-with-mage)

Ethan Brown introduces Mage Data Integration Blocks, bridging no-code simplicity with standard pipeline capabilities for seamless data integration across various sources. From Postgres to Salesforce, Ethan showcases the effortless process of pulling and leveraging data within standard pipelines, promising streamlined development and efficient syncs. Explore the future of data integration with Mage, where complexity meets accessibility.

### [Mage.ai – Tutorials and Intro](https://janzednicek.cz/en/category/mage-ai-etl-tool/)

Jan provides a series of tutorials from intro and installation to building pipelines and charts.

### [How to Streamline Communication in Data Pipelines Using Mage](https://medium.com/@xiaoxugao/1ab2a885dd73)

In her post, Xiaoxu introduces the concept of a *communication lifecycle* for data pipelines. Proactive feedback to stakeholders is essential for establishing trust and building confidence in data. Xiaoxu demonstrates how Mage's [callbacks](/design/blocks/callbacks) can be used to automate tasks based on block status, reducing the communication lifecycle and establishing a healthy relationship between data owners and consumers.

### [Modern Data Engineering with Mage](https://www.analyticsvidhya.com/blog/2023/06/modern-data-engineering-with-mage-empowering-efficient-data-processing/)

Sri Nikitha walks through a demo of Mage: from installation to a sample pipeline that loads data from a NYC taxi dataset, transforms it, and exports to BigQuery.

### [Postgres to Snowflake Data Transfer](https://sudsy-pail-0c7.notion.site/Data-Migration-Tutorial-using-Mage-46c027b4095d4c9c8216c566eee3c3a4)

Jafar Sharif walks through a tutorial on how to transfer data from Postgres to Snowflake using Mage, including some core ETL functionality. A quick and fun read!

## Video Tutorials

### How To Create Data Pipelines in Mage

Shashank walks through a high-level overview of how to create data pipelines using Mage.

[YouTube video player](https://www.youtube.com/embed/wk9PLudonbg)

### End-to-end Data Engineering Project with Mage

Using Mage and Python for orchestration and ingestion, Darshil walks through how to take his dataset from source to visualization using a combination of popular tools, including Mage, GCS, BigQuery, and Looker Studio.

[YouTube video player](https://www.youtube.com/embed/WpQECq5Hx9g)

### Building an ETL Pipeline with Mage

Arul discusses the process of constructing an ETL (extract-transform-load) pipeline in Mage. Starting from Netflix's top 100 movie dataset, Arul extracts data, transforms it to a useable format, ans loads it into a database for further analysis.

[YouTube video player](https://www.youtube.com/embed/Jvr4Jm97rxY)

### Apache Airflow Vs. Mage.AI! Apache and Mage.AI Compared for Data Orchestration!

The Data Guy compares and contrasts Apache Airflow and Mage.AI!

[YouTube video player](https://www.youtube.com/embed/fbIeCxtdcEg)


# How to build a data integration pipeline
Source: https://docs.mage.ai/guides/data-integration-pipeline

Here are the high level steps to build a data integration pipeline:

## Demo video

<video />

***

## Requirements

You must run Mage using [Docker](/getting-started/setup#using-docker) because there are a lot of
dependencies required for data integrations that can be challenging to install using `pip` or `conda`.

***

## Add new data integration pipeline

1. Open Mage in your browser and click the **`[+ New pipeline]`** button.
2. Click the dropdown menu option **Data integration**.

***

## Use connector documentation in Mage Pro

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration-documentation-sidekick)

Keep connector guidance open without leaving the pipeline editor:

1. Find **Configuration** under the source or destination.
2. Click **Show documentation** to open that connector's guide in the right
   sidekick. You can continue editing the YAML configuration while the guide is
   open.
3. Click **Hide documentation** when you need more room for the editor.

When the documentation sidekick is already open, clicking **View and select
streams** switches the guide back to the source connector. If you previously
hid the documentation, opening stream selection leaves it hidden.

***

## Configure source

1. Click the dropdown menu under **Select source** and choose the option you
   want to load data from (e.g. Amplitude).

2. Depending on the chosen source, you’ll need to enter credentials and options
   into the section labeled **Configuration**. For example, if you chose
   Amplitude, you’ll need to enter credentials like this:

   ```yaml theme={"system"}
   account: ...
   database: ...
   password: "{{ env_var('PASSWORD') }}"
   schema: ...
   table: ...
   username: ...
   warehouse: ...
   ```

   **Best practices**: you can interpolate values in the configuration using the
   following syntax:

   1. `"{{ env_var('SECRET_KEY') }}"`: this will extract the value from the
      `SECRET_KEY` key in your environment variables.
   2. `"{{ variables('SECRET_KEY') }}"`: this will extract the value from the
      `SECRET_KEY` key in your
      [runtime variables](/getting-started/runtime-variable).

   You can also add a prefix to output tables by configuring the following variables in the `source` configuration yaml:

   ```yaml theme={"system"}
   _patterns:
     destination_table: MY_PREFIX_{{ variables('stream') }}
   ```

3. After you enter in all the credentials, click the button
   **`[Fetch list of streams]`** under the section labeled **Select stream**.

4. Shortly after clicking the above button, click the new dropdown menu under
   the section labeled **Select stream**. Then, choose the stream (aka table)
   you want to load data from.

   **Add newly discovered fields in Mage Pro**

   New fields are excluded from an existing catalog by default. If a selected
   stream has new fields:

   1. Find the stream's **Catalog changes** column.
   2. Select **Include in catalog** for that stream.
   3. Confirm the stream selection.

   The option adds only that stream's newly discovered fields. Existing catalog
   settings—including bookmarks, replication methods, unique constraints, and
   selected fields—remain unchanged. Streams you do not opt in keep their current
   catalog unchanged. When the catalog content changes, confirming updates
   `data_integration_catalog.json` and creates a file version that you can inspect
   or restore later.

   **Edit the saved catalog in Mage Pro**

   Open **Edit** → **Data integration catalog file** to view or edit the saved
   JSON. **Format JSON** reformats valid JSON in the editor without saving it.
   Select **Save** to persist your changes and create a file version.

### Configure stream

After selecting a stream (aka table), you’ll need to configure the schema.

Configuring the schema informs your pipeline on which fields to synchronize, how
to determine if a record is unique, and what to do if their are conflicts (aka
duplicate records).

Here are the steps you can optionally go through:

1. Selected field(s):
   * Check the box next to the field name to include the field in your
     synchronization.
   * Uncheck the ones you don’t want to sync.
2. Field type(s)
   * Each field will have a default field type.
   * Add additional field types or remove them if they don’t fit your needs.
3. Unique field(s)
   * On the right of the field names, there is a box you can check that will
     determine which field(s) need to have unique values.
   * If the box is un-checkable, that means you cannot use that field as a
     unique field.
4. Bookmark field(s)
   * Under the column labeled **Bookmark**, check the box to use the field as a
     way to keep track of progress during synchronization.
   * Upon every synchronization, these columns are used to pick up from where
     the previous synchronization left off. In addition, if a synchronization
     fails midway, these bookmark columns are used to track the record that was
     most recently successful.
5. Replication method
   * `FULL_TABLE`: synchronize the entire set of records from the source.
   * `INCREMENTAL`: synchronize the records starting after the most recent
     bookmarked record from the previous synchronization run.
6. Unique conflict method: choose how to handle duplicate records
   * `IGNORE`: skip the new record if it’s a duplicate of an existing record.
   * `UPDATE`: update the existing record with the new record’s properties.

For supported SQL sources, the selected stream configuration surface also shows
**Count rows** next to **Load sample data**. Click **Count rows** after selecting
and configuring a stream to count only that active stream on demand. Mage shows
one context-appropriate result: `FULL_TABLE` streams show the total row count,
and `INCREMENTAL` streams show the row count eligible after the saved bookmark.
If no bookmark exists, Mage warns that the next incremental sync may read all
rows. Log-based SQL replication streams do not support row counts. Slow or
transient failed count requests show an error for the active stream and can be
retried without automatically counting every stream. Permission failures show
the source error without a retry action, since the source credentials or grants
must be fixed first.

Once you've configured the schema for the stream, you can optionally choose to
run the stream in parallel. Parallel streams will be attempted to be synchronized
in parallel if there are enough worker resources available. This may speed up the
sync process if you have a large number of streams.

### Editing bookmark property values

* You can edit the bookmark values for your next sync by editing them
  in the `Bookmark property values` table under the
  **Manually edit bookmark property values** setting in the **Settings** section.
* The **Manually edit bookmark property values** setting will not appear until
  at least one column is selected as a bookmark field.
* In order to enable the toggle that displays the `Bookmark property values`
  table, you need to select a destination first.
* Click the toggle and enter the new value for your bookmark property, and then
  click `Save`. You MUST click the `Save` button in order for the bookmark value
  to be updated.

***

## Configure destination

1. Click the dropdown menu under **Select destination** and choose the option
   you want to export data to (e.g. Snowflake).
2. Depending on the chosen destination, you’ll need to enter credentials and
   options into the section labeled **Configuration**. For example, if you chose
   Snowflake, you’ll need to enter credentials like this:

   ```yaml theme={"system"}
   account: ...
   database: ...
   password: "{{ env_var('PASSWORD') }}"
   schema: ...
   table: ...
   username: ...
   warehouse: ...
   ```

   Best practices: you can interpolate values in the configuration using the
   following syntax:

   1. `"{{ env_var('PASSWORD') }}"`: this will extract the value from the
      `PASSWORD` key in your environment variables.
   2. `"{{ variables('PASSWORD') }}"`: this will extract the value from the
      `PASSWORD` key in your runtime variables.

***

## Run pipeline and start sync

Once you’re done configuring your pipeline, open the pipeline’s **Triggers**
page from the pipeline navigation.

While you’re still in the editor, the breadcrumbs in your header could look like this:
`Pipelines / pipeline name / Editor`.

Once you’re on the pipeline triggers page, create a
[new scheduled trigger](/design/data-pipeline-management#create-trigger)
and choose the `@once` interval. For more schedules, read the
[other options here](/orchestration/triggers/schedule-pipelines).

### Monitoring pipeline

After you create a scheduled trigger, click the **`[Start trigger]`** button at
the top of the page. You’ll see a new pipeline run appear shortly on the screen.

You can [click the logs](/development/observability/logging) for that pipeline
run to view the progress of your synchronization. Any string config values over
8 characters long in the logs will be redacted for security purposes. You'll see
these values hidden as `********` in the logs.

***

## Support

If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat)


# Mage Pro CI/CD workflow
Source: https://docs.mage.ai/guides/data-sync/cicd-workflow

Deploy and manage your pipeline code safely across environments using Mage Pro's integrated deployment application.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=git-terminal)

## Introduction

Mage Pro supports two CI/CD approaches to fit different organizational needs. Both follow the same core workflow: develop → staging → deploy to production using the Git terminal and the Mage Pro deployment application.

This document clarifies the two primary CI/CD approaches available in Mage Pro:

1. **CI/CD by cluster** - Separate clusters for development, staging, and production
2. **CI/CD by workspace** - Multiple workspaces within a single cluster

Both approaches follow the same core principles: develop in isolation, test thoroughly, and deploy safely to production. The choice between cluster based and workspace based workflows depends on your infrastructure requirements, security policies, and team structure.

## 1. CI/CD by cluster

**Setup:** Separate clusters for each environment

* **Development cluster** - Active development
* **Staging cluster** (optional) - Pre-production validation
* **Production cluster** - Live pipelines

## 2. CI/CD by workspace

**Setup:** Single cluster with multiple workspaces (see our [workspaces guide](https://docs.mage.ai/guides/developer-ux/workspaces) for more details on configuring workspaces)

* **Development workspace** - Active development
* **Staging workspace** (optional) - Pre-production validation
* **Production workspace** - Live pipelines

## Git Branch Strategy & Environment Mapping

### Recommended Branch Structure

```python theme={"system"}
main/master     → Production environment
staging         → Staging environment  
development     → Development environment
feature/*       → Development environment (feature branches)
```

### Workflow Best Practices

**Development:**

* Create feature branches from `development`: `feature/new-pipeline`
* Develop pipelines in feature branch and push changes to Github using the Git terminal (for more details on pushing code to Github see Mage Pro’s [Git terminal](https://docs.mage.ai/guides/data-sync/git-terminal) doc)

**Staging:**

* Merge `development` → `staging` when ready for testing
* Staging environment deploys from `staging` branch
* Run full validation and integration tests

**Production:**

* Merge `staging` → `main` after successful testing
* Production environment deploys from `main` branch only
* Use deployment application for controlled production releases (for more on Mage Pro’s deployment application see the [version control guide](https://docs.mage.ai/guides/data-sync/version-control-guide))

### Environment-Specific Configurations

**Same Code, Different Targets:**
The same codebase connects to different systems using environment-specific configurations:

**Environment Variables & Secrets:**

* Each cluster/workspace maintains its own connection settings
* `DATABASE_URL` points to dev/staging/prod databases
* `API_KEYS` use environment-appropriate credentials

**Configuration Examples:**

```python theme={"system"}
Development:   DATABASE_URL = dev-warehouse.company.com
Staging:          DATABASE_URL = staging-warehouse.company.com  
Production:    DATABASE_URL = prod-warehouse.company.com
```

**In Mage Pro:**

* Configure different data source connections per environment
* Use environment variables in pipeline io\_config.yaml file: `{{ env_var('DATABASE_URL') }}`
  * or python interpolation in a block `os.environ.get('DB_USERNAME')`

Using these strategies will allow you to develop the same pipeline code, but select different source and targets for each cluster / workspace environment.

## Conclusion

Both approaches use the same tools and follow identical workflows. The choice depends on your infrastructure and security requirements. Use [Git terminal](https://docs.mage.ai/guides/data-sync/git-terminal) for code management and the deployment application for production deployments in either setup. Use our [version control guide](https://docs.mage.ai/guides/data-sync/version-control-guide) to implement this in your Mage Pro environment.


# Using GitClient in Mage blocks
Source: https://docs.mage.ai/guides/data-sync/git-client

Programmatically perform Git operations (add, commit, push, pull) directly within your Mage pipeline blocks using the GitClient class.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=git-client)

The `GitClient` class provides a simple, Pythonic interface for performing Git operations within your Mage pipeline blocks. This is useful for automating version control workflows, such as:

* Automatically committing and pushing pipeline changes after successful runs
* Syncing code from remote repositories before execution
* Creating automated deployment pipelines

## Prerequisites

Before using `GitClient`, ensure you have configured Git settings in Mage through one of these methods:

1. **Git Sync Settings** (legacy): Configure via Settings → Workspace → Sync data, or see the [Git Sync guide](/development/git/configure)
2. **Deployment Settings** (Mage Pro): Configure via the Deploy app at `/apps/deploy`

## Quick Start

### Basic Usage

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

# Initialize the client (uses legacy git sync settings by default)
git = GitClient()

# Add, commit, and push all changes in one operation
result = git.add_commit_push('Auto-commit from Mage pipeline')

if result.success:
    print(f"✓ Pushed to {result.remote}/{result.branch}")
    print(f"  Commit: {result.commit_hash}")
else:
    print(f"✗ Failed: {result.error}")
```

### Using Deployment Settings (Mage Pro)

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

# Use deployment settings instead of legacy git sync
git = GitClient(use_deployment_settings=True)
result = git.add_commit_push('Deploy changes via Mage Pro')
```

## Using GitClient in Pipeline Blocks

### Data Exporter Block Example

The most common use case is pushing changes at the end of a pipeline:

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(*args, **kwargs):
    """
    Push pipeline changes to Git after processing data.
    """    
    # Initialize GitClient
    git = GitClient()
    
    # Push all changes
    result = git.add_commit_push(
        message='Pipeline run completed - auto commit',
        files='.',  # Stage all files
        add_flags=['-A'],  # Include deletions
    )
    
    if result.success:
        print(f"✓ Successfully pushed to {result.remote}/{result.branch}")
    else:
        print(f"✗ Push failed: {result.error}")
    
    return {
        'git_result': result.to_dict(),
    }
```

### Data Loader Block Example

Pull latest changes before processing:

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data(*args, **kwargs):
    """
    Pull latest changes from remote before loading data.
    """
    git = GitClient()
    
    # Pull latest changes
    result = git.pull()
    
    if result.success:
        print(f"✓ Pulled latest from {result.remote}/{result.branch}")
    else:
        print(f"⚠ Pull failed: {result.error}")
    
    # Your data loading logic here
    return {'status': 'loaded'}
```

## GitClient Methods

### Core Operations

> **add(files, flags)**
>
> Stage files for commit.
>
> **Parameters:**
>
> * `files` (str | List\[str]): File path(s) to stage. Use `'.'` for all files.
> * `flags` (List\[str], optional): Git add flags (e.g., `['-A']` to include deletions).
>
> **Returns:** `GitClient` (for method chaining)
>
> ```python theme={"system"}
> git.add('.')  # Stage all files
> git.add(['file1.py', 'file2.py'])  # Stage specific files
> git.add('.', flags=['-A'])  # Stage all including deletions
> ```

> **commit(message, files)**
>
> Commit staged changes.
>
> **Parameters:**
>
> * `message` (str): The commit message.
> * `files` (List\[str], optional): Files to add before committing.
>
> **Returns:** `str` - The commit hash
>
> ```python theme={"system"}
> commit_hash = git.commit('Update pipeline configuration')
> ```

> **push(remote\_name, branch\_name, remote\_branch\_name)**
>
> Push commits to the remote repository.
>
> **Parameters:**
>
> * `remote_name` (str, optional): Remote name. Defaults to configured remote.
> * `branch_name` (str, optional): Local branch name. Defaults to current branch.
> * `remote_branch_name` (str, optional): Remote branch name if different from local.
>
> **Returns:** `GitOperationResult`
>
> ```python theme={"system"}
> # Push to default remote/branch
> result = git.push()
>
> # Push to specific remote and branch
> result = git.push(remote_name='origin', branch_name='main')
>
> # Push local 'develop' to remote 'staging'
> result = git.push(
>     remote_name='origin',
>     branch_name='develop',
>     remote_branch_name='staging',
> )
> ```

> **pull(remote\_name, branch\_name)**
>
> Pull changes from the remote repository.
>
> **Parameters:**
>
> * `remote_name` (str, optional): Remote name. Defaults to configured remote.
> * `branch_name` (str, optional): Branch name. Defaults to current branch.
>
> **Returns:** `GitOperationResult`
>
> ```python theme={"system"}
> result = git.pull()
> result = git.pull(remote_name='origin', branch_name='main')
> ```

> **fetch()**
>
> Fetch changes from the remote without merging.
>
> **Returns:** `GitOperationResult`
>
> ```python theme={"system"}
> result = git.fetch()
> ```

> **clone(sync\_submodules)**
>
> Clone the remote repository (replaces local content).
>
> **Parameters:**
>
> * `sync_submodules` (bool): Whether to also sync git submodules.
>
> **Returns:** `GitOperationResult`
>
> ```python theme={"system"}
> result = git.clone(sync_submodules=True)
> ```

### Convenience Methods

> **add\_commit\_push(message, files, add\_flags, remote\_name, branch\_name, remote\_branch\_name)**
>
> Add, commit, and push changes in one operation. This is the recommended method for most use cases.
>
> **Parameters:**
>
> * `message` (str): The commit message.
> * `files` (str | List\[str]): File path(s) to stage. Default: `'.'`
> * `add_flags` (List\[str], optional): Git add flags (e.g., `['-A']`).
> * `remote_name` (str, optional): Remote name.
> * `branch_name` (str, optional): Local branch name.
> * `remote_branch_name` (str, optional): Remote branch name if different.
>
> **Returns:** `GitOperationResult`
>
> ```python theme={"system"}
> # Simple push of all changes
> result = git.add_commit_push('Update from pipeline')
>
> # Push specific files
> result = git.add_commit_push(
>     message='Update config',
>     files=['pipelines/', 'data_loaders/'],
> )
>
> # Push including deletions
> result = git.add_commit_push(
>     message='Clean up',
>     files='.',
>     add_flags=['-A'],
> )
>
> # Push to a different remote branch
> result = git.add_commit_push(
>     message='Deploy to staging',
>     branch_name='main',
>     remote_branch_name='staging',
> )
> ```

> **sync(branch)**
>
> Sync local repository with remote (fetch + reset --hard). Discards local changes.
>
> **Parameters:**
>
> * `branch` (str, optional): Branch to sync. Defaults to current branch.
>
> **Returns:** `GitOperationResult`
>
> ```python theme={"system"}
> result = git.sync()  # Reset to match remote
> ```

### Branch Operations

> **switch\_branch(branch, remote)**
>
> Switch to a different branch.
>
> ```python theme={"system"}
> result = git.switch_branch('feature-branch')
> ```

> **merge\_branch(branch, message)**
>
> Merge another branch into the current branch.
>
> ```python theme={"system"}
> result = git.merge_branch('feature-branch', message='Merge feature')
> ```

> **delete\_branch(branch)**
>
> Delete a local branch.
>
> ```python theme={"system"}
> result = git.delete_branch('old-feature')
> ```

### Status Properties

> **status**
>
> Get comprehensive git status information.
>
> ```python theme={"system"}
> status = git.status
> print(status)
> # {
> #     'branch': 'main',
> #     'remote': 'origin',
> #     'remote_url': 'git@github.com:user/repo.git',
> #     'modified_files': ['file1.py', 'file2.py'],
> #     'is_dirty': True,
> #     'status': '...'
> # }
> ```

> **current\_branch**
>
> Get the current branch name.
>
> ```python theme={"system"}
> branch = git.current_branch  # 'main'
> ```

> **is\_dirty**
>
> Check if there are uncommitted changes.
>
> ```python theme={"system"}
> if git.is_dirty:
>     print("There are uncommitted changes")
> ```

> **modified\_files**
>
> Get list of modified files.
>
> ```python theme={"system"}
> files = git.modified_files  # ['file1.py', 'file2.py']
> ```

> **branches**
>
> Get list of local branches.
>
> ```python theme={"system"}
> branches = git.branches  # ['main', 'develop', 'feature-x']
> ```

> **remotes**
>
> Get list of configured remotes.
>
> ```python theme={"system"}
> remotes = git.remotes
> # [{'name': 'origin', 'urls': ['git@github.com:user/repo.git']}]
> ```

> **unpushed\_commits\_count**
>
> Get number of commits not yet pushed to remote.
>
> ```python theme={"system"}
> count = git.unpushed_commits_count  # 3
> ```

## GitOperationResult

All Git operations return a `GitOperationResult` object with the following attributes:

| Attribute | Type | Description |
| - | - | - |
| `success` | bool | Whether the operation succeeded |
| `message` | str | Success or status message |
| `branch` | str | The branch name involved |
| `remote` | str | The remote name involved |
| `commit_hash` | str | The commit hash (for commit/push operations) |
| `error` | str | Error message if operation failed |

```python theme={"system"}
result = git.push()

if result.success:
    print(f"Pushed to {result.remote}/{result.branch}")
    print(f"Commit: {result.commit_hash}")
else:
    print(f"Error: {result.error}")

# Convert to dictionary
result_dict = result.to_dict()
```

## Method Chaining

`GitClient` supports method chaining for fluent API usage:

```python theme={"system"}
git = GitClient()

# Chain add and commit, then push
git.add('.').commit('Update files')
result = git.push()
```

## Advanced Examples

### Conditional Push Based on Changes

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

@data_exporter
def export_data(data, *args, **kwargs):
    git = GitClient()
    
    # Only push if there are actual changes
    if git.is_dirty:
        print(f"Found {len(git.modified_files)} modified files")
        result = git.add_commit_push('Auto-commit changes')
        return {'pushed': result.success, 'commit': result.commit_hash}
    else:
        print("No changes to commit")
        return {'pushed': False, 'message': 'No changes'}
```

### Push to Multiple Branches

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

@data_exporter
def deploy_to_environments(data, *args, **kwargs):
    git = GitClient()
    
    # Push to staging
    staging_result = git.add_commit_push(
        message='Deploy to staging',
        remote_branch_name='staging',
    )
    
    # Push to production (same commit)
    if staging_result.success:
        prod_result = git.push(remote_branch_name='production')
        return {
            'staging': staging_result.to_dict(),
            'production': prod_result.to_dict(),
        }
    
    return {'error': staging_result.error}
```

### Pre-run Sync

```python theme={"system"}
from mage_ai.deployments.git_client import GitClient

@data_loader
def load_with_sync(*args, **kwargs):
    git = GitClient()
    
    # Fetch latest and check for updates
    git.fetch()
    
    if git.unpushed_commits_count < 0:  # We're behind
        print("Local is behind remote, syncing...")
        git.sync()
    
    # Now load your data
    return load_data_from_synced_repo()
```

## Using the Built-in Git Push Template

Mage provides a built-in data exporter template for Git push operations. To add it to your pipeline:

1. Click **Blocks** in the top navigation
2. Select **Exporter**
3. Navigate to **Version control** → **Git push**

![Git push template navigation](https://github.com/mage-ai/assets/blob/main/version-control/git-push-template-nav.png?raw=true)

The template provides a ready-to-use block with configurable options:

```python theme={"system"}
# Parameters available in the template:
commit_message: str  # Commit message
files: str | List[str]  # Files to stage (default: '.')
include_deletions: bool  # Include deleted files (default: True)
remote_name: str  # Remote name (optional)
branch_name: str  # Branch name (optional)
use_deployment_settings: bool  # Use Mage Pro settings (default: False)
```

## Setting Up Automated Git Push Jobs

You can automate Git push operations by creating a pipeline with a Git push block and configuring a trigger to run it on a schedule or in response to events.

### Step 1: Create a Git Push Pipeline

1. Create a new pipeline (e.g., `auto_git_push`)
2. Add a **Data Exporter** block using the Git push template:
   * Click **Blocks** → **Exporter** → **Version control** → **Git push**
3. Configure the block with your desired settings:

```python theme={"system"}
from typing import Dict, List, Optional, Union
from mage_ai.deployments.git_client import GitClient

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_to_git(*args, **kwargs) -> Dict:
    """
    Push changes to a git remote repository.

    Uses the git settings configured via git settings (/settings/workspace/sync-data)
    or deployment settings (/apps/deploy).

    Docs: https://docs.mage.ai/guides/data-sync/git-client
    """

    # Get configuration from kwargs
    commit_message: str = kwargs.get(
        'commit_message',
        'Auto-commit from Mage pipeline'
    )
    files: Union[str, List[str]] = kwargs.get('files', '.')
    include_deletions: bool = kwargs.get('include_deletions', True)
    remote_name: Optional[str] = kwargs.get('remote_name')
    branch_name: Optional[str] = kwargs.get('branch_name')
    use_deployment_settings: bool = kwargs.get('use_deployment_settings', False)

    # Initialize git client
    git = GitClient(use_deployment_settings=use_deployment_settings)

    # Set add flags
    add_flags = ['-A'] if include_deletions else None

    # Add, commit, and push
    result = git.add_commit_push(
        message=commit_message,
        files=files,
        add_flags=add_flags,
        remote_name=remote_name,
        branch_name=branch_name,
    )

    # Print result
    if result.success:
        print(f"✓ Successfully pushed to {result.remote}/{result.branch}")
        if result.commit_hash:
            print(f"  Commit: {result.commit_hash}")
    else:
        print(f"✗ Push failed: {result.error}")

    # Return the result and pass through the input data
    return {
        'result': result.to_dict(),
    }
```

### Step 2: Create a Trigger

Navigate to your pipeline's **Triggers** tab and create a new trigger:

**Schedule Trigger**

For periodic backups (e.g., daily at midnight):

1. Click **+ New trigger**
2. Select **Schedule** as the trigger type
3. Configure the schedule:
   * **Name**: `daily_git_backup`
   * **Frequency**: Daily (or use Custom with cron: `0 0 * * *`)
   * **Start date**: Set your desired start date
4. Click **Save changes**

**Example cron patterns:**

| Pattern | Description |
| - | - |
| `0 * * * *` | Every hour |
| `0 0 * * *` | Daily at midnight |
| `0 9,18 * * 1-5` | Weekdays at 9 AM and 6 PM |
| `*/30 * * * *` | Every 30 minutes |

**API Trigger**

For on-demand pushes from external systems:

1. Click **+ New trigger**
2. Select **API** as the trigger type
3. Configure:
   * **Name**: `git_push_api`
4. Click **Save changes**
5. Copy the API endpoint URL

**Trigger the pipeline via API:**

```bash theme={"system"}
curl -X POST "https://your-mage-instance.com/api/pipeline_schedules/{trigger_id}/pipeline_runs/{token}" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline_run": {
      "variables": {
        "commit_message": "Triggered via API"
      }
    }
  }'
```

> [!NOTE]
>
> For more details on trigger configuration, see the [Pipeline Triggers documentation](/orchestration/triggers/triggering-pipelines).

## Troubleshooting

> **Authentication errors**
>
> If you encounter authentication errors:
>
> 1. Verify your Git settings are configured correctly
> 2. For SSH: Ensure your SSH keys are properly set up
> 3. For HTTPS: Verify your access token has the correct permissions
> 4. Check that the remote URL is accessible from your Mage instance

> **Permission denied**
>
> If you see "Permission denied" errors:
>
> 1. Check that your user has write access to the repository
> 2. Verify the branch is not protected (or you have permission to push to protected branches)
> 3. Ensure your OAuth token hasn't expired

> **No changes to commit**
>
> If `add_commit_push` reports no changes:
>
> 1. Check `git.status` to see the current state
> 2. Verify files are not in `.gitignore`
> 3. Use `git.modified_files` to see what files Git detects as changed

## Related Resources

* [Git Sync Configuration](/development/git/configure) - Configuring Git sync settings
* [Mage Pro Deployment Settings](/guides/data-sync/version-control-guide#1-deployments-app) - Configure deployment settings for `use_deployment_settings=True`


# Git integration for code version control
Source: https://docs.mage.ai/guides/data-sync/git-integration

Mage's Git integration provides a way for you to push changes to a remote repository— for example collaborating on a Mage project with your teammates.

"Git integration" in Mage is a way to collaborate with teammates using Git. It provides a way to push changes to a branch in an external repository. If you're using GitHub, we recommend Mage's direct [GitHub integration](/production/data-sync/github) rather than this method.

First, you'll need to configure Git— see this [guide](/development/git/configure). Once your git settings are configured, you can perform various Git actions in Mage. To start, click the button at the top right with the name of the current branch.

![](https://mage-ai.github.io/assets/version-control/git-integration/branch-button.png)

Once the modal is open, you can start performing actions. Currently, the Git integration supports the following actions:

> **Clone repository**
>
> Clone repository will allow you to clone the specified git repository to your local machine.
> *This operation will overwrite all local changes you have in the local git directory.*

> **Create new branch**
>
> Create new branch will create a new branch in your local git directory. Specify the name of the branch and click `Create`. You will
> automatically be switched to the new branch.
>
> ![](https://mage-ai.github.io/assets/version-control/git-integration/create-new-branch.png)

> **Switch branch**
>
> To switch your local branch, you can click the dropdown menu with the current active branch and select the branch you would like to switch to.
>
> ![](https://mage-ai.github.io/assets/version-control/git-integration/switch-branch.png)

> **Commit & push**
>
> **1. Select files**
>
> First, select the files you would like to commit. You can select all files by clicking the `Include all changes` button.
>
> ![](https://mage-ai.github.io/assets/version-control/git-integration/select-files.png)
>
> **2. Commit your changes**
>
> Then, enter a descriptive commit message and click `Commit`. This will commit your changes to your local git directory.
>
> ![](https://mage-ai.github.io/assets/version-control/git-integration/commit-changes.png)
>
> **3. Push to remote**
>
> Once your changes are committed, you can push your commit to the remote repository by clicking `Push`.
>
> ![](https://mage-ai.github.io/assets/version-control/git-integration/push-to-remote.png)

> **Pull**
>
> Pull your changes from the remote branch to your local branch. This operation is equivalent to `git pull mage-repo <current-branch>`.

> **Hard reset**
>
> Hard reset your local branch with the remote branch. This operation is equivalent to `git reset --hard mage-repo/<current-branch>`.
>
> *This operation will overwrite your local changes.*

For any other operations you may need, you can perform them through the terminal. You can open the terminal by clicking the terminal toggle in the top right of the modal.


# One-way code sync using Git
Source: https://docs.mage.ai/guides/data-sync/git-sync

Enabling one-way Git sync in Mage ensures data will be synced in one direction with a specified git repository.

Git sync is a Git feature that allows you to tie your local project to a remote repo. When Git sync is enabled, you will only be able to pull in changes from the remote repository. Git sync is particularly useful for deployment— for example if you'd like a hosted Mage instance to stay up-to-date with a repo.

Running a Git sync will ***overwrite all local changes*** and replace them with the code in the specified remote branch. Git syncs can be run either from the Git settings page, or before every trigger run if you selected the `Sync before each trigger run` field.

To enable Git Sync, first [configure Git in Mage](/development/git/configure), then navigate to the settings page and tick the `Git Sync` box.

### Syncing git submodules

Repositories synced through Mage will only update 1 level of submodules. If your submodule contains other submodules, you will need to manually update them or reformat your submodules.

If your repo contains git submodules, you can enable syncing for them by ticking the `Include submodules` box. Mage will
use the same credentials that are provided for the main repository to update the submodules.

## Git sync settings as environment variables

> [!NOTE]
>
> For Mage versions `>= 0.9.51`
>
> Mage will prioritize git settings from environment variables when they are set. If you want to be able
> to change git settings from the UI, you will need to set the `GIT_OVERWRITE_WITH_PROJECT_SETTINGS` environment
> variable to `1`. Otherwise, edits to the git settings from the UI will not be applied.

You can choose to pass in Git settings as environment variables. Mage will read from these environment
variables to configure your Git repo.

| Setting | Variable | Description | Required |
| - | - | - | - |
| Git settings | `GIT_REPO_LINK` | Corresponds to `Remote repo url` | ✅ |
| Git settings | `GIT_REPO_PATH` | Corresponds to `Local directory path` | ✅ |
| Git settings | `GIT_USERNAME` | Corresponds to `Username` | ✅ |
| Git settings | `GIT_EMAIL` | Corresponds to `Email` | ✅ |
| Git settings | `GIT_AUTH_TYPE` | The method of authentication. See enum values [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/sync/__init__.py#L13-L16) | ✅ |
| Git sync settings | `GIT_BRANCH` | Corresponds to `Branch name` | 🚫 |
| Git sync settings | `GIT_SYNC_ON_PIPELINE_RUN` | Corresponds to `Sync before each trigger run`, options | 🚫 |
| Git sync settings | `GIT_SYNC_ON_START` | Corresponds to `Sync on server start up`, options | 🚫 |
| Git sync settings | `GIT_SYNC_ON_EXECUTOR_START` | If set to "1", Mage will attempt to run a git sync every time an executor is started, options | 🚫 |
| Git sync settings | `GIT_SYNC_SUBMODULES` | Corresponds to `Include submodules`, options | 🚫 |
| Authentication environment variables | `GIT_SSH_PUBLIC_KEY` | SSH public key encoded in base64, used for ssh auth type | 🚫 |
| Authentication environment variables | `GIT_SSH_PRIVATE_KEY` | SSH private key encoded in base64, used for ssh auth type | 🚫 |
| Authentication environment variables | `GIT_ACCESS_TOKEN` | Git access token, used for https auth type | 🚫 |
| Global git settings | `GIT_ENABLE_GIT_INTEGRATION` | Force git integration module to show in the UI. Primarily useful when the version control app does not work for your repository. | 🚫 |
| Global git settings | `GIT_OVERWRITE_WITH_PROJECT_SETTINGS` | Allow users to overwrite git settings set by environment variables through the Git Settings page in the UI. | 🚫 |


# Git terminal with built-in authentication and shortcuts
Source: https://docs.mage.ai/guides/data-sync/git-terminal

Mage Pro supports Git-based version control, making it easy to  collaborate, track changes, and manage code across environments.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=git-terminal)

### Introduction

The Git Terminal is a browser based terminal with built in Git authentication,
directly integrated into Mage Pro. It provides standard Git functionality with
real-time collaboration features, eliminating the need to switch between tools for
version control operations. Multiple team members can work simultaneously in
the same terminal session, sharing commands and coordinating development work.

### Location

From the home page, open **Tools** in the left navigation and select
**Terminal**. You will then be taken to a terminal screen directly integrated
with Git.

### Integration

The Git Terminal connects with your existing development workflow. Key integration points include:

* GitHub, GitLab, and other Git providers
* Maintains authentication across sessions
* Works within existing Mage Pro workflow

![Mage Pro Git terminal](https://mage-ai.github.io/assets/pro/features/git-terminal-md.jpg)

### Use Cases

Use the Git Terminal in your development environment to push changes to GitHub for source version control after making modifications to pipelines, blocks, or configurations in the Mage UI. Essential use cases include:

* Before deploying changes to save work to version control
* For regular commits to maintain code history and backup

### Authentication

Before using the Git Terminal, authenticate with your Git provider through the Deploy App. This one-time setup enables Git operations within Mage Pro.

**Quick Setup:**

1. Open **Tools** → **Deploy**, then select **Connect repository**
2. Complete OAuth authentication for GitHub, Bitbucket, GitLab, or Azure DevOps
3. Configure your repository connection

*For detailed authentication steps, see the [Version control guide](https://docs.mage.ai/guides/data-sync/version-control-guide).*

Once authenticated, the Git Terminal automatically uses your credentials for all Git operations without requiring additional setup.

For OAuth connections, each terminal session receives a session-scoped Git credential helper for the connected repository's Git provider. HTTPS Git commands such as `git fetch`, `git pull`, and `git push` can use the OAuth token already connected in Mage, and provider CLI commands can read the matching session token environment variable without running a separate CLI login such as `gh auth login`.

If the terminal header shows **Authenticate Git**, Mage has not detected usable Git authentication for the repository, either from the terminal environment such as an environment token, GitHub CLI login, or SSH key, or from a connected Git OAuth token. Select **Authenticate Git** to open the repository connection page for the current project, complete the Deployments Git OAuth flow, and then return to the terminal. If you authenticate Git directly in the terminal, Mage hides the action after it detects that Git authentication is available.

The Mage Pro image includes GitHub CLI (`gh`) for optional GitHub authentication
and status checks. For GitLab, Azure DevOps, and Bitbucket, use standard Git
commands with Mage's credential helper or configured SSH/PAT authentication;
provider-specific CLIs are not required.

### Usage

The Git Terminal supports all standard Git operations you're familiar with. Run standard Git commands such as:

```python theme={"system"}
git status                    # Check current changes
git add .                     # Stage all changes  
git commit -m "message"       # Commit with description
git push origin main          # Push to remote repository
```

### Terminal controls

Use terminal tabs when you need more than one shell context. Each tab runs its
own terminal session, keeps its own working directory, and can be reconnected or
closed without interrupting the other tabs.

The prompt bar shows the current folder, Git branch, and working tree change
counts for the active tab. Autocomplete appears while typing and can suggest
shell commands, Git subcommands, files, folders, executable commands on `PATH`,
and recent command history.

Terminal prompt timestamps and shell commands use UTC by default. If your
workspace has **Use local timezone** enabled, new terminal sessions use your
browser's local IANA timezone for the prompt timestamp and the session `TZ`
environment variable.

Terminal sessions load Mage-managed safeguards for AI and Mage API access. Direct
shell calls from the browser terminal to Mage REST endpoints such as pipelines,
pipeline runs, blocks, block runs, pipeline schedules, and status are blocked;
use the Mage UI, the documented API from outside the workspace terminal, or Mage
agent MCP tools for those operations.

### Conclusion

The Git Terminal streamlines your development workflow by bringing version control directly into the Mage Pro environment.
With built-in authentication, real-time collaboration, and standard Git functionality, teams can maintain professional development
practices without the complexity of managing separate tools. This integrated approach enables faster development cycles while ensuring
code changes are properly tracked and shared across your team.


# Version control application with Git
Source: https://docs.mage.ai/guides/data-sync/version-control

Authenticate with a Git provider then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.

![GitHub authentication](https://wallpapercave.com/dwp2x/wp3082255.jpg)

On the homepage in the left side navigation, click the link labeled **Version control**.
The URL could look something like this: [http://localhost:6789/version-control](http://localhost:6789/version-control).

# Mage Pro

Unlock seamless version control and effortless deployments with Mage Pro!
Check out this [guide](/guides/data-sync/version-control-guide) to experience a more intuitive, user-friendly way to manage and deploy your code.

![Mage Pro deployments page](https://mage-ai.github.io/assets/pro/features/deployments/deployments-page.png)

# Open source

## Demo

<video />

## Setup

![Setup](https://mage-ai.github.io/assets/version-control/setup.png)

### Set Git directory

The Git directory is the local folder that will be linked to your remote repository.

### Authentication

You will be able to authenticate with GitHub by default, but if you need to authenticate with
GitHub Enterprise (GHE) or other providers, you will need to follow the steps below. You can
authenticate with multiple git providers if you need to. Mage will attempt to use the oauth provider
based on the URL of the remote.

> [!WARNING]
>
> Mage authenticates with a default GitHub OAuth application that may not be able to access
> your repository depending on the repository or organization settings. If you encounter issues
> accessing your repository after authenticating with GitHub, you may need to create your own
> OAuth application that has access to the repositories you want to work with. You can follow the
> steps in the `Authenticate with GitHub or GHE` section below to connect Mage with your OAuth
> application.

When creating the OAuth app for GHE, BitBucket, GitLab, you will need to set the authorized
callback URL to `http(s)://<your-mage-url>/oauth`.

> **Authenticate with GitHub or GHE**
>
> **1. Register Mage in GitHub**
>
> In order for Mage to authenticate with your GitHub host, you will need to add Mage as an
> authorized oauth application. You can follow the steps [here](https://docs.github.com/en/enterprise-cloud@latest/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app)
> to set it up.
>
> Make sure you copy the client id and client secret after the app is created.
>
> **2. Update environment variables**
>
> You will need to set the following environment variables in your Mage environment:
>
> 1. GHE\_HOSTNAME: the hostname of your GitHub instance. If you’re using non-enterprise GitHub or GitHub Enterprise Cloud, you should set this to `github.com`
> 2. GHE\_CLIENT\_ID: the client id you retrieved in the previous step
> 3. GHE\_CLIENT\_SECRET: the client secret you retrieved in the previous step
>
> **3. Authenticate with GitHub**
>
> You should now be able to authenticate with GitHub and follow the rest of the steps in this document.

> **Authenticate with BitBucket**
>
> **1. Add Mage as an OAuth consumer**
>
> In order for Mage to authenticate with your BitBucket, you will need to add Mage as an
> OAuth consumer. You can follow the steps [here](https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/)
> to set it up.
>
> Make sure you copy the key and secret after the consumer is created.
>
> **2. Update environment variables**
>
> You will need to set the following environment variables in your Mage environment:
>
> 1. BITBUCKET\_OAUTH\_KEY: the key you retrieved in the previous step
> 2. BITBUCKET\_OAUTH\_SECRET: the secret you retrieved in the previous step
>
> **3. Authenticate Mage with Bitbucket**
>
> You should now be able to authenticate with Bitbucket and follow the rest of the steps in this document.

> **Authenticate with GitLab**
>
> > [!NOTE]
> >
> > Mage will request `read_user`, `write_repository`, and `api` scopes from GitLab in order
> > to perform the necessary git actions.
>
> **1. Add Mage Oauth application**
>
> In order for Mage to authenticate with your GitLab, you will need to add a Mage oauth application.
> You can follow the steps [here](https://docs.gitlab.com/ee/integration/oauth_provider.html)
> to set it up.
>
> Make sure you copy the Application ID and Secret after the oauth app is created.
>
> **2. Update environment variables**
>
> You will need to set the following environment variables in your Mage environment:
>
> 1. GITLAB\_CLIENT\_ID: the application ID you retrieved in the previous step
> 2. GITLAB\_CLIENT\_SECRET: the secret you retrieved in the previous step
> 3. GITLAB\_HOST (optional): hostname for your GitLab instance. Defaults to `https://gitlab.com`
>
> **3. Authenticate Mage with GitLab**
>
> You should now be able to authenticate with GitLab and follow the rest of the steps in this document.

> **Authenticate with Azure DevOps**
>
> **1. Enable Active Directory integration**
>
> First, you will need to enable the [Active Directory](https://docs.mage.ai/production/authentication/microsoft) integration in order for Mage to authenticate your user.
>
> **2. Update environment variables**
>
> Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable.
>
> ```bash theme={"system"}
> export AZURE_DEVOPS_ORGANIZATION=<organization_name>`
> ```
>
> **3. Authenticate Mage with Azure DevOps**
>
> You should now be able to authenticate with Azure DevOps and follow the rest of the steps in this document.

Click the Authenticate button next to your Git Provider to start the authentication process.

![Authenticate](https://mage-ai.github.io/assets/version-control/authentication.png)

After authorizing Mage, you’ll be redirected back to the initial page and an access token will
be in the URL as a parameter. This access token is saved in your application’s database;
it never leaves your application’s database.

### Add remote

Enter a name for your new remote. For the remote URL, you must use the `https` version of the
GitHub repository (as opposed to the `SSH` version).

For example, the remote URL for Mage is `https://github.com/mage-ai/mage-ai.git`.

Note that a remote added via HTTPS/SSH authentication *will not* work— only a remote in the format above is accepted.

Once you fill in both input text fields, click the button labeled **Create new remote**.

### Actions

#### Clone remote repository

> [!WARNING]
>
> Cloning will copy all the files and folders from the remote repository and put them into the Git directory you specified above.
>
> Cloning won’t automatically create a folder that is named after the remote repository.
>
> For example, if you have a file named `magic.py` in a remote repository named `project_romeo`
> then that file will be cloned here `<your git directory>/magic.py`
> as opposed to `<your git directory>/project_romeo/magic.py`.
>
> If you want to clone the content of a remote repository into a folder named after the remote repository, then change the Git init directory above to
> `<your git directory>/[remote repository name]`

1. Under the Actions dropdown, select the option labeled **Clone**.
2. Under the Remote dropdown, select the name of the remote you want to clone from (e.g. origin).

#### Pull remote branch

1. Under the Actions dropdown, select the option labeled **Pull**.
2. Under the Remote dropdown, select the name of the remote you want to pull from (e.g. origin).
3. Under the Branch dropdown, select the name of the branch you want to pull from the remote you chose from the previous step.

***

## Branches

![Branches](https://mage-ai.github.io/assets/version-control/branches.png)

### Switch branch

1. Under the dropdown labeled **Current branch**, select the name of the branch you want to change to.
2. After selecting a branch, the current branch will be updated to what you selected.

### Create branch

1. Type in the name of the new branch in the text input field.
2. Click the button labeled **+ Create new branch**.

### Merge branch

1. Under the Base branch dropdown, select the name of the branch you want to merge into your current branch.
2. Under the Action dropdown, select the option labeled **Merge**.
3. Type a commit message for the merge action.
4. Click the button labeled **Merge**.

### Rebase branch

1. Under the Base branch dropdown, select the name of the branch you want to rebase into your current branch.
2. Under the Action dropdown, select the option labeled **Rebase**.
3. Type a commit message for the rebase action.
4. Click the button labeled **Rebase**.

### Delete branch

1. Under the Base branch dropdown, select the name of the branch you want to delete.
2. Under the Action dropdown, select the option labeled **Delete**.
3. Click the button labeled **Delete**.
4. Confirm the popup dialog box.

***

## Files

![Files](https://mage-ai.github.io/assets/version-control/files.png)

### Add files

1. On the left side under **Not staged**, check the boxes next to the files you want to add.
2. Once you’re done selecting the files, click the button labeled **Add files**.
3. The selected files will be staged.

### Checkout files

1. On the left side under **Not staged**, check the boxes next to the files you want to checkout.
2. Once you’re done selecting the files, click the button labeled **Checkout files**.
3. Confirm the popup dialog box.
4. All changes to the selected files will be undone.

### Reset files

1. On the right side under **Staged files**, check the boxes next to the files you want to reset.
2. Once you’re done selecting the files, click the button labeled **Reset files**.
3. The selected files will be unstaged.

### Commit

1. Enter a commit message.
2. Click the button labeled **Commit N files with message**.

***

## Push

![Push](https://mage-ai.github.io/assets/version-control/push.png)

### Push local changes

1. Under the Remote dropdown, select the name of the remote you want to push to (e.g. origin).
2. Under the Branch dropdown, select the name of the branch you want to push.
3. Click the button labeled **Push \[remote] \[branch]**.

### Create pull request

1. Under the Repository dropdown, select the name of the repository you want to create a pull request for.
2. Under the Base branch dropdown, select the name of the branch you want your pull request to compare against (e.g. main).
3. Under the Compare branch dropdown, select the name of the branch you want your pull request to be for (e.g. feature branch).
4. Type the title of the pull request in the text input field labeled **Title**.
5. Type the description of the pull request in the text input field labeled **Description**.


# Mage Pro Version Control
Source: https://docs.mage.ai/guides/data-sync/version-control-guide

Connect Mage Pro to GitHub, Bitbucket, GitLab, or Azure DevOps using OAuth, HTTPS with a personal access token, or SSH, then deploy commits and manage code from the Version control terminal.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=version-control-guide)

Mage Pro's version control workflow has three parts:

1. **Authentication** — [Connect your Git host](#1-authentication) to Mage (OAuth with GitHub, Bitbucket, GitLab, or Azure DevOps; HTTPS with a personal access token; or SSH). You do this in the **Deployments** app, but it also enables the [Version control terminal](#3-version-control-terminal).
2. **Deployments** — [Deploy](#2-deployments-app) commits to your Mage Pro cluster or workspace from the visual Deployments app.
3. **Version control terminal** — [Run Git in the browser](#3-version-control-terminal) (push, pull, branches) once you are connected.

## 1. Authentication

**Authentication is shared** by the [Deployments app](#2-deployments-app) and the [version control terminal](#3-version-control-terminal). Configure it in **Deployments** → open **Connect repository** or deployment **Setup** → **Connection** → **Authentication type**. Until this is set up, you cannot use Git-based deployment or the terminal.

Mage Pro supports three **authentication types**:

| Type | When to use |
| - | - |
| **OAuth** | Default path: sign in with GitHub, Bitbucket, GitLab, or Azure DevOps and pick org/workspace, repo, and branch from the UI. |
| **HTTPS + PAT** | Your organization requires a **personal access token** instead of OAuth, or you use a provider / URL where OAuth is not an option. |
| **SSH** | You use **deploy keys** or user SSH keys and `git@…` (or `ssh://`) remotes. |

If you change authentication type, **save authentication settings** before continuing with the rest of the connection configuration. Switching types clears the credentials from the other method in your deployment settings. For **HTTPS + PAT** and **SSH**, run **Test connection** after saving. A successful test enables deployment selection from the Deployments list; if the repository URL, username, email, token, or SSH keys change later, Mage marks the connection as needing another test.

### Open the connection UI

From the Home page, open **Tools** in the left navigation and select **Deploy** to access the Deployments interface.

Click the **Setup connection** button to begin syncing with your code repository.
Mage Pro provides two main applications for version control and deployment:

1. **Deployments App**: Deploy code from Git commits to your Mage Pro cluster
2. **Version Control Terminal**: Push/pull code and manage branches using Git commands

## 1. Deployments App

The Deployments App is your primary tool for deploying code from Git commits to your Mage Pro cluster. It provides a visual interface for managing deployments across different environments.

### 1.1 Authentication

First, authenticate with your Git provider in the Deployments application to enable version control and deployment features.

#### Access Deployments

From the Home page, open **Tools** in the left navigation and select **Deploy** to access the Deployments interface.

Click the Setup connection button to begin syncing with your code repository.

![Mage Pro deployments button](https://mage-ai.github.io/assets/pro/features/deployments/setup-connection-091925.png)

### Project settings

Fill in the **Project settings** fields at the top of the Connect repository page, then click **Save settings** before configuring authentication below.

* **Configuration name** — a label for this deployment configuration.
* **Project directory** — the user code path for your project. The default in Mage Pro is `/home/src/default_repo`. If you changed the `USER_CODE_PATH` environment variable in the Mage Cloud Management Portal, set this field to match.
* **Target directory (optional)** — the directory code changes are deployed into. If left empty, the project directory is used. For a nested repository you may want to set this to `/home/src`.

#### Configure Repository Settings

Once you are in the connect repository UI, take the following steps:

**Step 1**: Set your directory as the user code path you intend to use for your project.

* In Mage Pro the default directory will be **`/home/src/default_repo`**.
* If you changed the **`USER_CODE_PATH`** environment variable in the Mage Cloud Management Portal, your directory will need to reflect that change.

**Step 2 (optional)**: Set your target directory for your deployment if it will be different from your top level directory.

* If you have a nested GitHub project, you may want to set the target directory to `/home/src`

![Mage Pro deployments button](https://mage-ai.github.io/assets/pro/features/deployments/connect-repo-091925.png)

After clicking **Save settings**, proceed to the **Connection** section below to set your authentication type and connect your provider.

### HTTPS and personal access token (PAT)

1. Under **Connection**, set **Authentication type** to **HTTPS+PAT**.
2. Enter the **remote repository URL** in HTTPS form (for example `https://github.com/your-org/your-repo.git`). The UI derives owner and repository name from this URL.
3. Enter a **username** and **email** for Git commit authorship. These are also used as HTTP credentials: the username is your Git host account name and the PAT is sent in place of a password.
4. In your Git host, create a **personal access token** (or app password) scoped to at least **clone, fetch, and push**. Scope names differ by provider — for example, GitHub requires the `repo` scope; GitLab requires `read_repository` and `write_repository`; Bitbucket uses the `repository:write` permission.
5. Paste the token into **Personal access token** and click **Save authentication settings**. The token is stored as a **Mage secret** named `mage_git_access_token_{user_id}` (where `{user_id}` is your Mage user ID).
6. Click **Test connection**. After the test passes, Mage shows **Verified for deployment selection** and the Deployments list can show **Select deployment** for this configuration.

HTTPS+PAT uses your stored credentials for Git in the [version control terminal](#3-version-control-terminal) the same way OAuth does.

![Connect with PAT](https://mage-ai.github.io/assets/pro/features/deployments/connect-pat.png)

### SSH key authentication

1. Under **Connection**, set **Authentication type** to **SSH**.
2. Enter the **remote repository URL** in SSH form (for example `git@github.com:your-org/your-repo.git` or a supported `ssh://` URL from your provider).
3. Enter a **username** and **email** for Git **commit** identity (as shown in `git log`), matching what you use for the repository.
4. Add **SSH public and private key** values. The form expects the key material in **base64** (as labeled in the UI). In your Git host, add the public key as a **deploy key** (repo-scoped) or a user key, with permission to read/write the repo as needed.
5. Click **Save authentication settings**.
6. Click **Test connection**. After the test passes, Mage shows **Verified for deployment selection** and the Deployments list can show **Select deployment** for this configuration.

> [!NOTE]
>
> With **SSH** authentication, Mage cannot use the provider’s **Git hosting API** for some deployment flows the same way as with OAuth (for example, **listing commits in the Deployments** UI). For **`git push`**, **`git pull`**, fetches, and other Git wire operations, use the **[version control terminal](#access-the-version-control-terminal)**. A warning is shown in the app when this applies.

![Connect with SSH keys](https://mage-ai.github.io/assets/pro/features/deployments/connect-ssh.png)

### GitHub (OAuth)

Mage Pro supports integration with GitHub repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the GitHub **Connect** OAuth flow below.

**Connect to GitHub using OAuth tokens:**

**Step 1**: Click the **Connect** button to the right of the GitHub connection.

* This will take you to the GitHub sign-in authentication.

![Connect GitHub](https://mage-ai.github.io/assets/pro/features/deployments/connect-oauth-github.png)

**Step 2**: Sign in with your GitHub login credentials.

* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to GitHub.
* Click **Save authentication settings**.

**Step 3**: Select your GitHub organization from the list.

* If your organization isn't appearing, enter the name in the **Organization name** field and click **Set organization**.

> [!NOTE]
>
> To deploy from a **private repository**, click the **Install Mage Pro app on GitHub** button that appears below the Organization section. Once installed, you can verify or update the installation at any time using the **Verify installation** or **Configure installation settings** buttons.

**Step 4**: Select the repository from the dropdown.

* You should see all your GitHub repositories if you authenticated correctly.

**Step 5**: Select the production branch from the dropdown.

* The production branch is the primary branch your production code will mirror.

![Mage Pro install app button](https://mage-ai.github.io/assets/pro/features/deployments/set-branch-09192025.png)

### Bitbucket (OAuth)

Mage Pro supports integration with Bitbucket repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the Bitbucket **Connect** OAuth flow below.

**Connect to Bitbucket using OAuth tokens:**

#### GitHub Authentication

**Step 1**: Click the "Connect" button to the right of the GitHub connection

* This will take you to the GitHub sign-in authentication

**Step 2**: Sign in with your GitHub login credentials

* After authenticating you will return to the Mage-Pro deployment app
* You should now see that you are connected to GitHub

**Step 3**: Select your GitHub organization from the dropdown list.

* If there are no organizations listed, add an organization to the text field below and click the Set organization button

**Step 4**: Select the repo you want to commit code to by clicking the repository drop down box

* You should see all your GitHub repos here if you authenticated correctly

**Step 5**: Select your branch (usually main or master) to commit your code to by clicking the dropdown box and selecting the branch name

**Step 6**: Click the "Deployments" button to see a list of commits from your Git hosting vendor, and deploy from the most recent or any specific commit.

![Mage Pro install app button](https://mage-ai.github.io/assets/pro/features/deployments/set-branch-09192025.png)

> [!NOTE]
>
> If you need to connect deployments to a private GitHub repo, provide an organization then click the Install Mage Pro app on GitHub button.

#### BitBucket Authentication

Mage Pro supports integration with BitBucket repositories for version control, collaboration, and deployments.

**Connect to BitBucket using OAuth tokens:**

**Step 1**: Set your directory as the user code path you intend to use for your project.

* In Mage Pro the default directory will be `/home/src/default_repo`.
* If you changed the `USER_CODE_PATH` environment variable in the Mage Cloud Management Portal, your directory will need to reflect that change.

![Connect repository](https://raw.githubusercontent.com/mage-ai/assets/main/version-control/bitbucket/bitbucket-connect-repo.png)

**Step 1**: Click the **Connect** button to the right of the Bitbucket connection.

* This will take you to the Bitbucket sign-in authentication.

**Step 2**: Sign in with your Bitbucket login credentials.

* After authenticating you will return to the Mage Pro deployment app.

* You should now see that you are connected to Bitbucket.

* Click **Save authentication settings**.
  **Step 2**: Click the "Connect" button to the right of the BitBucket connection

* This will take you to the BitBucket sign-in authentication

**Step 3**: Sign in with your BitBucket login credentials

* After authenticating you will return to the Mage-Pro deployment app
* You should now see that you are connected to BitBucket

![Connect Button](https://raw.githubusercontent.com/mage-ai/assets/main/version-control/bitbucket/bitbucket-connect.png)

**Step 3**: Select your Bitbucket workspace from the dropdown list.

* If there are no workspaces listed, enter a workspace slug in the text field below and click **Set workspace**.

**Step 4**: Select the repository from the dropdown menu.

* You should see a list of all the repositories in that workspace.

**Step 5**: Select the production branch from the dropdown list.

* The production branch is the primary branch your production code will mirror.
  **Step 4**: Select your BitBucket workspace from the dropdown list.

* If there are no organizations listed, add an organization to the text field below and click the "Set organization" button

**Step 5**: Select the repository from the dropdown menu.

* You should see a list of all the repositories that were created in the workspace.

**Step 6**: Finally, select the production branch from the dropdown list

* The production branch is the primary branch that your production code will mirror.

![Bitbucket workspace](https://raw.githubusercontent.com/mage-ai/assets/main/version-control/bitbucket/bitbucket-workspace.png)

### GitLab (OAuth)

Mage Pro supports integration with GitLab repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the **Connect** GitLab OAuth flow below.

**Connect to GitLab using OAuth tokens:**

**Step 1**: Click the **Connect** button to the right of the GitLab connection.

* This will take you to the GitLab sign-in authentication.

**Step 2**: Sign in with your GitLab login credentials.

* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to GitLab.
* Click **Save authentication settings**.

![Connect Button](https://raw.githubusercontent.com/mage-ai/assets/main/version-control/gitlab/gitlab-connect.png)

**Step 3**: Select your GitLab group from the dropdown list.

* If there are no groups listed, enter a group name in the text field below and click **Set group**.

**Step 4**: Select the repository from the dropdown menu.

* You should see a list of all the repositories in that group.

**Step 5**: Select the production branch from the dropdown list.

* The production branch is the primary branch your production code will mirror.

### Azure DevOps (OAuth)

Mage Pro supports integration with Azure DevOps repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the Azure DevOps **Connect** OAuth flow below.

**Connect to Azure DevOps using OAuth tokens:**

**Step 1**: Click the **Connect** button to the right of the Azure DevOps connection.

* This will take you to the Azure DevOps sign-in authentication.

**Step 2**: Sign in with your Azure DevOps credentials.

* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to Azure DevOps.
* Click **Save authentication settings**.

![Connect repository](https://mage-ai.github.io/assets/pro/features/deployments/azuredevops-connect.png)

**Step 3**: Select your Azure DevOps **organization** from the dropdown, or type the organization name directly in the text field and click **Save**.

* If your organization isn't appearing in the dropdown, enter the name in the **Organization name** field and click **Set organization**.

**Step 4**: Select your **Azure DevOps project** from the dropdown.

* This is the project that contains the repository you want to deploy changes from.

**Step 5**: Select the **repository** from the dropdown.

* You should see a list of all repositories in that project.

**Step 6**: Select the **production branch** from the dropdown.

* This is the primary branch your production code will mirror.
* If your branch isn't listed, type the branch name into the text field and click **Set branch**.

## 2. Deployments App

The Deployments app is the control surface for [deploying](#merge-pull-request-and-deploy) merged work from Git to your Mage Pro cluster. It assumes you have already completed [authentication](#1-authentication).

### Merge pull request and deploy

Once you push code to your Git branch, open your Git provider and merge the pull request. When the merge is done, the deployment is available in the Deployments app.

Mage then deploys a chosen commit to your environment:

1. In Mage, go to the **Deployments** view and open the deployments list (highlighted in green at the top when updates are available).
2. Click **Deploy** to ship the version you just merged. Use **Rollback** to deploy an earlier commit if you need to revert.

If a configuration that uses **HTTPS + PAT** or **SSH** shows **Test connection required** instead of **Select deployment**, open **Configure**, run **Test connection**, and return to the Deployments list after the test passes. Mage requires a fresh successful test after saved custom authentication settings change.

#### GitLab Authentication

Mage Pro supports integration with GitLab repositories for version control, collaboration, and deployments.

**Connect to GitLab using OAuth tokens:**

**Step 1**: Set your directory as the user code path you intend to use for your project.

* In Mage Pro the default directory will be `/home/src/default_repo`.
* If you changed the `USER_CODE_PATH` environment variable in the Mage Cloud Management Portal, your directory will need to reflect that change.

**Step 2**: Click the "Connect" button to the right of the GitLab connection

* This will take you to the GitLab sign-in authentication

**Step 3**: Sign in with your GitLab login credentials

* After authenticating you will return to the Mage-Pro deployment app
* You should now see that you are connected to GitLab

![Connect Button](https://raw.githubusercontent.com/mage-ai/assets/main/version-control/gitlab/gitlab-connect.png)

**Step 4**: Select your GitLab group from the dropdown list.

* If there are no groups listed, add a group to the text field below and click the "Set group" button

**Step 5**: Select the repository from the dropdown menu.

* You should see a list of all the repositories that were created in the group.

**Step 6**: Finally, select the production branch from the dropdown list

* The production branch is the primary branch that your production code will mirror.

> [!NOTE]
>
> Currently the deployments application is only available for GitHub, GitLab, and Bitbucket users.

### 1.2 Deploy Code

Deploy code from Git commits to your Mage Pro cluster or workspace using the Deployments application.

#### Merge Pull Request

Once you push code to your Git branch, head to your Git provider and merge the pull request. Once the pull request is merged your deployment will be available in the deployments application interface.

#### Deploy Code

Mage simplifies the deployment of code to production pipelines. Follow these steps to deploy your pipeline's CI/CD Git branches:

1. Once you have merged the pull request in your Git provider, navigate back to the deployments tab and click deployments highlighted in green in the top
2. Click the Deploy button to deploy the current version you just merged
   * To revert to a previous version, simply click the "Rollback" button to deploy the desired version.

![Mage Pro deployments page](https://mage-ai.github.io/assets/pro/features/deployments/deployments-page.png)

## 3. Version control terminal

Once you have completed [Authentication](#1-authentication), you can use standard Git in the browser: status, commit, push, pull, and branch operations against your configured remote.

> [!NOTE]
>
> If you have not set up Git access yet, follow [Authentication](#1-authentication) first: **OAuth**, or **HTTPS+PAT** / **SSH** in the **Deployments** connection settings. All three options enable the version control terminal.

With **OAuth**, the terminal automatically attaches a session-scoped Git credential helper for the connected repository's provider. HTTPS Git commands can use the OAuth token that is already connected in Mage, so you do not need to run a separate provider CLI login in the terminal.

Mage Pro images include GitHub CLI (`gh`) for optional GitHub auth helper and status checks. GitLab, Azure DevOps, and Bitbucket Git operations use the Mage credential helper or configured SSH/PAT authentication and do not require provider-specific CLIs in the runtime image.

### Access the Git terminal

* From the home page, open **Tools** in the left navigation and select **Terminal**.
* When the page loads, type Git commands in the terminal and press **Enter** to run them.
* If the terminal header shows **Authenticate Git**, Mage has not detected usable Git authentication for the repository. Select it to open the repository connection page for the current project, complete Git OAuth in Deployments, and then return to the terminal. If you authenticate Git directly in the terminal, Mage hides the action after it detects that Git authentication is available.
* Use tabs to keep multiple shell sessions open. Each tab keeps its own terminal session and working directory, so closing one tab does not interrupt the other tabs.
* The terminal status row shows the active folder, Git branch, and working tree change counts. Autocomplete appears while typing and suggests shell commands, Git subcommands, files, folders, executable commands, and recent command history from the active working directory.
* Terminal prompt timestamps and shell commands use UTC by default. If **Use local timezone** is enabled for the workspace, new terminal sessions use your browser's local IANA timezone for the prompt timestamp and the session `TZ` environment variable.

### Essential Git commands

* `git checkout -b <branch-name>`: create and switch to a new feature branch.
* `git status`: inspect modified, staged, and untracked files.
* `git add .`: stage changes in the current directory.
* `git commit -m "<commit-message>"`: commit the staged changes.
* `git pull`: fetch and merge remote changes into the current branch.
* `git push origin <branch-name>`: push the current branch to the remote repository.

## Best Practices for Environment Management

### Development Environment

* **Primary Tool - Version Control Terminal**:
  * Create feature branches, develop code, commit changes, push to Git
  * Switch between branches, merge changes
* **Secondary Tool - Deployments App**:
  * Deploy to dev environment for testing and validation
  * Rollback changes or switch to another feature branch (can also be done via terminal)
* **Workflow**: Develop in Mage Pro → Commit to feature branch → Deploy to dev environment for testing
* **Workflow**: Develop locally → Commit to feature branch → Deploy to dev environment for testing

### Staging Environment

* **Primary Tool - Deployments App**:
  * Pull changes from Git (staging/main/master branch)
  * Deploy to staging environment and run testing
* **Secondary Tool - Version Control Terminal**:
  * Make hotfixes and push changes to Git
  * Merge feature branches and prepare releases
* **Workflow**: Pull from Git → Deploy to staging → Run integration tests → Make hotfixes if needed

### Production Environment

* **Primary Tool - Deployments App**:
  * Deploy commits from main/master branch
  * Rollback changes if issues occur
  * Monitor deployment status
* **Secondary Tool - Version Control Terminal**:
  * Emergency patches and hotfixes only
* **Workflow**: Deploy from main/master → Monitor → Rollback if needed


# Add an existing dbt project to Mage
Source: https://docs.mage.ai/guides/dbt/add-existing-dbt

If you have an existing dbt project, you can add it to your Mage project.

For example, if your Mage project is named `demo_project` and your existing dbt
project is named `demo`, add your dbt project in the Mage project under the
following path:

```
demo_project/dbt/demo/
```

You can have multiple dbt projects in a single Mage project under the folder
named `dbt/` at the root location of your Mage project.

For example:

```
demo_project/
|   dbt/
|   |   demo/
|   |   |   ...
|   |   project_2/
|   |   |   ...
```

***

## Required structure

### Connection profiles

For Mage to properly run dbt commands, you must have a `profiles.yml` file at
the root of each of your dbt projects. For example:

```
demo_project/
|   dbt/
|   |   demo/
|   |   |   profiles.yml
|   |   project_2/
|   |   |   profiles.yml
```

For more information on how to configure dbt connection profiles, read this
[documentation](/dbt/connection-profiles).


# dbt connection profiles
Source: https://docs.mage.ai/guides/dbt/connection-profiles



[https://www.youtube.com/embed/rb\_E2Nfm8Ak](https://www.youtube.com/embed/rb_E2Nfm8Ak)

Mage will look for a file named `profiles.yml` located in your dbt project root
folder.

For example, if your Mage project is named `demo_project` and your dbt project
name is `demo`, then your `profiles.yml` file for that dbt project should be
located in `demo_project/dbt/demo/profiles.yml`.

If you have a 2nd dbt project (e.g. named `example`), the connection profiles
for that project should be located in `demo_project/dbt/example/profiles.yml`.

For more information on how to configure your connection profiles, read
[dbt’s documentation](https://docs.getdbt.com/docs/get-started/connection-profiles).

Using Mage, you may also [interpolate variables](/guides/dbt/variable-interpolation) in your connection profiles.
Using Mage, you may also [interpolate variables](/dbt/variable-interpolation) in your connection profiles.

***

## How connection profiles are used

When you run a [single dbt model](/guides/dbt/run-single-model) or
[selected dbt models](/guides/dbt/run-selected-model), there is an input field labeled
When you run a [single dbt model](/dbt/run-single-model) or
[selected dbt models](/dbt/run-selected-model), there is an input field labeled
`dbt profile target`. The value in that field will be the connection profile
target that Mage uses when running a dbt model block.

The command Mage executes is `dbt run` with the flag `--target` and the value of
that flag is the value from the input field labeled `dbt profile target`.


# Develop dbt in Mage
Source: https://docs.mage.ai/guides/dbt/developing-dbt-in-mage

Learn how to develop your favorite dbt models in Mage.

> [!WARNING]
>
> dbt integration is currently only supported when using Mage in
> [Docker](/getting-started/setup#using-docker) or in [Mage Pro](https://cloud.mage.ai/sign-up).

If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat).

[https://www.youtube.com/embed/ESIZXg\_Toxk](https://www.youtube.com/embed/ESIZXg_Toxk)

**1. Set up new Mage project**

Read the [setup guide](/getting-started/setup) to initialize a new project and
start the Mage tool locally.

For the rest of this tutorial, we’ll use the project name `demo_project`.

**2. Set up dbt project**

* Open Mage and go to the terminal page: [http://localhost:6789/terminal](http://localhost:6789/terminal)
* Initiate your dbt project using the `init` command (for this tutorial, we’ll
  use the dbt project name `demo`):

```bash theme={"system"}
cd demo_project/dbt
dbt init -s demo
touch demo/profiles.yml
```

For more information on creating a dbt project, read their
[documentation](https://docs.getdbt.com/reference/commands/init).

**3. Create standard pipeline**

* Go to the Mage dashboard and click the button
  **`+ New pipeline`** and select the option labeled `Standard (batch)`.
* Click the `Pipeline settings` icon in the left pane, and change its name
  to `dbt demo pipeline`, then click the `Save pipeline settings` button.

**4. Create dbt profile for database connections**

* On the left side of the page in the file browser, expand the folder
  `demo_project/dbt/demo/`.
* Click the file named `profiles.yml`.
* Paste the following credentials in that file:

```yaml theme={"system"}
demo:
  target: dev
  outputs:
    dev:
      dbname: xyleviup
      host: queenie.db.elephantsql.com
      password: edSrMWH7Mww-lTKpp-jPHX9sYSNLy7LG
      port: 5432
      schema: dbt_demo
      type: postgres
      user: xyleviup
```

* Save the `profiles.yml` file by pressing `Command (⌘)` + `S`.
* Close the file by pressing the **`X`** button on the right side of the file
  name `dbt/demo/profiles.yml`.

**5. Add data loader block to pipeline**

* Click the **`+ Data loader`** button, select `Python`, then click `API`.
* In the `Block name` dialog, change the name to `load data`, then click `Save`.
* Paste the following code in that block:

  ```python theme={"system"}
  import io
  import pandas as pd
  import requests
  from pandas import DataFrame


  @data_loader
  def load_data_from_api(**kwargs) -> DataFrame:
      url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'

      response = requests.get(url)
      df = pd.read_csv(io.StringIO(response.text), sep=',')
      df.columns = ['_'.join(col.split(' ')) for col in df.columns]
      return df
  ```

**6. Add dbt model block to pipeline**

* Under the data loader block you just added, click the button **`dbt model`**,
  then click the option `Single model`.

* In the file browser that pops up, click the file named
  `my_second_dbt_model.sql` under the folders `demo/models/example/`.

* This will add 2 dbt blocks to your pipeline: 1 for the dbt model named
  `my_first_dbt_model` and the 2nd for the dbt model named `my_second_dbt_model`.

* The model named `my_first_dbt_model` was added to
  the pipeline because `my_second_dbt_model` references it.

![](https://mage-ai.github.io/assets/dbt/add-dbt-model.gif)

**7. Edit my\_first\_dbt\_model**

* In the dbt block named `my_first_dbt_model`, next to the label
  `Target` at the top, choose `dev` in the dropdown list.  You can also
  check `Manually enter target`, and enter `dev` in the input field.
* Paste the following SQL into the dbt model named `my_first_dbt_model`:

```sql theme={"system"}
WITH source_data AS (
    SELECT 1 AS id
    UNION ALL
    SELECT 2 AS id
)

SELECT *
FROM source_data
```

* Run the dbt model block by pressing the play button on the top right of the
  block or by pressing `Command` + `Enter`.
* You should see a preview of the query execution logs. To see the query results,
  click the `Expand table` link at the bottom right corner.

![](https://mage-ai.github.io/assets/dbt/dbt-preview.gif)

* After previewing the results, in the top right corner of the block,
  click on the triple dots to reveal a dropdown menu.
* Under the dropdown menu, click the option **Run model**. This command
  will execute the `dbt run` command and create the table in your data source.

![](https://mage-ai.github.io/assets/dbt/run-model.png)

**8. Edit my\_second\_dbt\_model**

* In the dbt block named `my_second_dbt_model`, next to the label
  `Target` at the top, choose `dev` in the dropdown list.  You can also
  check `Manually enter target`, and enter `dev` in the input field.

* Paste the following SQL into the dbt model named `my_second_dbt_model`:

  ```sql theme={"system"}
  SELECT
      a.*
      , b.*
  FROM {{ ref('my_first_dbt_model') }} AS a

  LEFT JOIN {{ source('mage_demo', 'dbt_demo_pipeline_load_data') }} AS b
  ON 1 = 1

  WHERE a.id = 1
  ```

  > [dbt sources](https://docs.getdbt.com/docs/build/sources)
  >
  > When a dbt model depends on an upstream block that isn’t a dbt model, a
  > source for that block is automatically added to the
  > `demo_project/dbt/demo/models/mage_sources.yml` file.
  >
  > Read more about dbt sources in their
  > [documentation](https://docs.getdbt.com/docs/build/sources).

* Run the dbt model block by pressing the play button on the top right of the
  block or by pressing `Command` + `Enter`.

* You should see a preview of the query execution logs. To see the query results,
  click the `Expand table` link at the bottom right corner.

**9. Add test for dbt model**

* On the right side of the screen, click the tab labeled **`Terminal`**.
* Create a new dbt test file by running the following command:
  ```bash theme={"system"}
  touch demo_project/dbt/demo/tests/test_my_second_dbt_model.sql
  ```
* On the left side of the page in the file browser, expand the folder
  `demo_project/dbt/demo/tests/` and click the file named
  `test_my_second_dbt_model.sql`. If you don’t see it, refresh the page.
* Paste the following SQL in the file:
  ```sql theme={"system"}
  SELECT id
  FROM {{ ref('my_second_dbt_model') }}
  GROUP BY id
  HAVING (id = 0)
  ```
* Read more about dbt tests in their
  [documentation](https://docs.getdbt.com/docs/build/tests).

**10. Execute pipeline end-to-end**

* Open the pipeline's **Triggers** page from the pipeline navigation.

* Create a new trigger with a type `Schedule` and a Frequency `once`.
  For more details, follow these
  [steps](/design/data-pipeline-management#create-trigger).

* After your trigger is created, click the **`Start trigger`** button at the
  top of the page.

* The pipeline will eventually fail because a dbt test failed. This means
  everything is working as expected.

* Open the file `demo_project/dbt/demo/models/example/schema.yml` and remove
  the tests named `unique` under both models. Your file should look like this:

  ```yaml theme={"system"}

  version: 2

  models:
    - name: my_first_dbt_model
      description: "A starter dbt model"
      columns:
        - name: id
          description: "The primary key for this table"
          tests:
            - not_null

    - name: my_second_dbt_model
      description: "A starter dbt model"
      columns:
        - name: id
          description: "The primary key for this table"
          tests:
            - not_null

  ```

* Click on the **`Failed`** button next to the pipeline run and click
  **`Retry run`**. It should complete running successfully after a few
  minutes.

Congratulations! You’ve created a data pipeline that orchestrates your dbt
models.


# Serve dbt docs in production
Source: https://docs.mage.ai/guides/dbt/docs

To serve dbt docs in production, you will need to enable a container to host the dbt docs webserver in the cloud service you are using.

## Mage Pro (auto-generate and serve)

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dbt)

If you're using Mage Pro, dbt docs can be automatically generated and served when the app starts.

* Set the environment variable `ENABLE_DBT_DOCS=true`.
* Mage will discover dbt projects under `/dbt`, generate docs (if missing), and serve them.
* Open a project's docs from the UI:
  * In File Browser, right-click the dbt project folder and click "Open dbt docs".
  * The docs are available at `/dbt_docs/[dbt_project_name]`.
  * The path `/dbt_docs` shows the status for all dbt projects (running port, errors, etc.).

To regenerate docs manually, add a generic dbt block in a pipeline and run `dbt docs generate`.

[https://www.youtube.com/embed/0zB2Sc5BCBM](https://www.youtube.com/embed/0zB2Sc5BCBM)

## Generating docs

Before enabling dbt docs in production, we need to make sure the static files for the docs are generated.

1. Configure the dbt super project.
   * Your dbt projects should be in the `/path/to/<mage_project>/dbt` directory.
     * The `/dbt` directory will serve as a "super-project" for all of your dbt projects in your Mage project.
   * Make sure the `/.../<mage_project>/dbt` directory has the files `dbt_project.yml`, `profiles.yml`, and `packages.yml`. These files are needed for dbt to create docs for all the projects within this directory.
     1. `dbt_project.yml`
        ```yaml theme={"system"}
        name: 'base'
        version: '1.0.0'
        config-version: 2
        profile: 'base'
        target-path: "target"
        clean-targets:
          - "target"
          - "dbt_packages"
        ```
     2. `profiles.yml`: The config in this file is not used, but it needs to still be a valid output target in order for the project to compile.
        ```yaml theme={"system"}
        base:
          target: dev
          outputs:
            dev:
              <output_config>
        ```
     3. `packages.yml`: Add all projects that you want to be included in the documentation.
        ```yaml theme={"system"}
        packages:
          - local: ./project1
          - local: ./project2
        ```
2. Generate the dbt docs
   ```bash theme={"system"}
     dbt deps
     dbt docs generate
   ```

## Terraform

### GCP

In the [mage-ai-terraform-templates](https://github.com/mage-ai/mage-ai-terraform-templates) repository, the terraform templates for GCP have commented out resources at the end of the following files for enabling the dbt docs service.

* [main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf#L184)
* [load\_balancer.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/load_balancer.tf#L81)

Once you uncomment those resources, you should see an output `docs_service_ip` at the end of `terraform apply` with the IP to access the dbt docs in the cloud.

```bash theme={"system"}
Outputs:
docs_service_ip = "<dbt_docs_ip>"
service_ip = "<mage_ip>"
```

### AWS

Coming soon...


# Incremental models
Source: https://docs.mage.ai/guides/dbt/incremental-models



For more information about how to use incremental models, please read dbt’s
[documentation](https://docs.getdbt.com/docs/build/incremental-models).

## How to rebuild incremental models?

You can rebuild 1 or more models using the `--full-refresh` flag
([dbt documentation](https://docs.getdbt.com/docs/build/incremental-models#how-do-i-rebuild-an-incremental-model))
and graph operators ([dbt documentation](https://docs.getdbt.com/reference/node-selection/graph-operators)).

![dbt runtime settings](https://mage-ai.github.io/assets/dbt/dbt-runtime-settings.png)

In order to do this in Mage, follow these steps:

1. Create a new [trigger](/guides/triggering-pipelines).
   > [!NOTE]
   >
   > You may want to create a scheduled trigger with an interval of `@once` because
   > running a full refresh of an incremental model is usually done ad hoc and not
   > on a regular basis.

2. On the right side of the edit trigger page,
   look for the section labeled **dbt runtime settings**.

3. Under the column labeled **Flags**, check the box labeled **`--full-refresh`**.

4. \[Optional] Under the column labeled **Prefix**, write the graph operators of your choice.

5. \[Optional] Under the column labeled **Suffix**, write the graph operators of your choice.


# Use dbt with Mage: Simplify Your Data Transformation Pipelines
Source: https://docs.mage.ai/guides/dbt/overview

Learn how Mage integrates with dbt to orchestrate, schedule, and monitor dbt models as part of your data pipelines.

> [!WARNING]
>
> dbt integration is currently only supported when using Mage in
> [Docker](/getting-started/setup#using-docker) or in [Mage Pro](https://cloud.mage.ai/sign-up).

![](https://c.tenor.com/gbLWPf5HCsYAAAAC/devastator-constructicons.gif)

## What is Mage's dbt Integration?

Mage is a modern data pipeline tool that lets you **build, schedule, and monitor dbt models** alongside your data workflows — all in one platform. With Mage, you can fully automate your dbt workflows without using Airflow or dbt Cloud.

![](https://user-images.githubusercontent.com/59450879/252510713-4fd2ed03-7867-4038-8b0e-24eb1c40c518.png)

## Who Should Use dbt in Mage?

Mage’s dbt integration is designed for both analytics engineers and data engineers who want more control, flexibility, and observability in their data workflows.

### Analytics engineers

If you're transforming raw data into clean, analytics-ready datasets using dbt, Mage helps you:

* A **visual interface** to build and run dbt models
* The ability to **chain dbt steps** with Python or SQL blocks
* Easy scheduling and triggering for model runs
* **Live SQL previews** to speed up development

Mage empowers analytics teams to move faster, test confidently, and collaborate more effectively — all within a unified platform.

### Data engineers

If you're building and orchestrating production-grade data pipelines, Mage lets you:

* Run dbt models as part of larger **end-to-end workflows**
* Replace complex orchestration tools like **Airflow**
* Run dbt across environments — including **Docker**, **Kubernetes**, [**Mage Pro**](https://cloud.mage.ai), or in the **cloud (AWS, GCP, Azure)**
* Monitor, debug, and get alerts on dbt runs with built-in **observability tools**

Mage simplifies pipeline orchestration and lets you focus on data logic — not infrastructure setup or maintenance.

## Key Features

### Schedule dbt model runs

Set dbt models to run:

* On a schedule
* When a file or event triggers
* From an API call or upstream task

This is perfect for productionizing your dbt workflows.

### Run specific dbt models and their dependencies

![](https://mage-ai.github.io/assets/dbt/add-dbt-model.gif)

Use Mage’s visual interface to:

* Select and run individual dbt models
* Automatically include required dependencies

This saves time during development and debugging.

### Exclude Models When Running All

![](https://raw.githubusercontent.com/mage-ai/assets/main/dbt/add-dbt-models.gif)

Need to run everything except one model? No problem. Mage lets you exclude specific dbt models in any run.

### Chain dbt Models with Other Tasks

Build workflows where dbt models run after:

* Pulling data from APIs
* Completing a Python or SQL block
* Finishing a different pipeline

You can treat dbt as a native building block in your DAG.

### Preview dbt model results as you write SQL

![](https://mage-ai.github.io/assets/dbt/dbt-preview.gif)

Mage lets you **preview dbt model output as you write SQL**, helping you test and iterate faster without running full pipelines.

### Build dynamic dbt pipelines using flexible variable interpolation

Take advantage of dynamic variable interpolation like:

* `{{ env_var('...') }}`
* `{{ variables('...') }}`
* `{{ mage_secret_var('...') }}`

### Automatically run dbt tests every time a pipeline runs

Mage will:

* Detect and run your dbt tests during each pipeline run
* Fail the pipeline if any dbt test fails

This enforces quality without extra setup.

### Built-In Monitoring and Alerting

Mage gives you full observability into your dbt runs:

* Track model run history
* See logs and SQL output
* Get alerts when models fail

You don’t need to wire up additional logging systems.

## Supported connectors

1. `dbt-bigquery`
2. `dbt-clickhouse`
3. `dbt-core`
4. `dbt-dremio`
5. `dbt-duckdb`
6. `dbt-mysql`
7. `dbt-postgres`
8. `dbt-redshift`
9. `dbt-snowflake`
10. `dbt-spark`
11. `dbt-sqlserver`
12. `dbt-synapse`
13. `dbt-trino`

## Tutorials

Explore our tutorial to set up and run your first dbt models in Mage:

* [How to Set Up dbt Models & Orchestrate Runs in Mage](/guides/dbt/setup-dbt)


# Run dbt-spark against a pySpark session.
Source: https://docs.mage.ai/guides/dbt/run-dbt-spark



The following procedure demonstrates how to run `dbt-spark` with a pySpark session.

1. Build a Mage docker image with Spark following the instructions given at
   [Build Mage docker image with Spark environment](/integrations/spark-pyspark#1-build-mage-docker-image-with-spark-environment).

2. Run the following command in your terminal to start Mage using docker:

```bash theme={"system"}
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
  /app/run_app.sh mage start demo_project
```

3. Create a new pipeline with a name `dbt_spark`, and add a `Scratchpad` to
   test out the connection with PySpark, with the following code:

```python theme={"system"}
from pyspark.sql import SparkSession
import os

spark = SparkSession.builder.master(os.getenv('SPARK_MASTER_HOST', 'local')).getOrCreate()

spark.sql("show databases;")
```

It should return results similar to the following when running:

```
[Stage 0:>                                                          
(0 + 1) / 1]

    namespace
0 default
```

![PySpark Scratchpad results](https://mage-ai.github.io/assets/pyspark-scratchpad-results.png)

4. Click the `Terminal` icon on the right side of the Mage UI, and create a dbt
   project `spark_demo`, with the following commands:

```bash theme={"system"}
root@488dc9529cf3:/home/src#  cd demo_project/dbt
root@488dc9529cf3:/home/src#  dbt init spark_demo -s
root@488dc9529cf3:/home/src#  touch spark_demo/profiles.yml
```

5. On the left side of the page in the file browser, expand the folder `demo_project/dbt/spark_demo/`.
   Click the file named `profiles.yml`, and add the following settings to this file:

```yaml theme={"system"}
spark_demo:
  target: dev
  outputs:
    dev:
      type: spark
      method: session
      schema: default
      host: local
```

6. Save the `profiles.yml` file by pressing `Command (⌘) + S`, then close the file
   by pressing the X button on the right side of the file name `dbt/spark_demo/profiles.yml`.

7. Click the button `dbt model`, and choose the option `New model`. Enter `model_1`
   as the `Model name`, and `spark_demo/models/example` as the folder location.

8. In the dbt block named `model_1`, next to the label `Target` at the top,
   choose `dev` in the dropdown list. You can also check `Manually enter target`,
   and enter `dev` in the input field.

9. Paste the following SQL into the dbt block `model_1`:

```python theme={"system"}
{{ config(materialized='table') }}

with source_data as (

    select 1 as id
    union all
    select null as id

)

select *
from source_data
```

Click the `Compile & preview` button to execute this new model, which would
generate the results similar to the following:

![dbt-spark testing results](https://mage-ai.github.io/assets/dbt-spark-testing-results.png)


# Run multiple models
Source: https://docs.mage.ai/guides/dbt/run-selected-model

(and optionally exclude others)

![dbt models](https://raw.githubusercontent.com/mage-ai/assets/main/dbt/add-dbt-models.gif)

## Run

1. Under the data loader block you just added, click the button
   **`dbt model`**, then click the option `All models`.
2. In the `dbt project name` input field, enter the name of the dbt project
   that contains the models you want to build and run (e.g. `demo`).
3. In the `dbt profile target` input field, enter the name of the dbt
   connection profile (e.g. `dev`) that you want to use when running the
   selected dbt models.
4. In the text area of the code block, write the models you want to select or
   exclude using dbt’s `--select` and `--exclude` flags and syntax.

   For more information on the `--select` and `--exclude` syntax,
   [read dbt’s documentation](https://docs.getdbt.com/reference/node-selection/syntax#examples).
   For example:

   ```yaml --select theme={"system"}
   $ dbt run --select my_dbt_project_name   # runs all models in your project
   $ dbt run --select my_dbt_model          # runs a specific model
   $ dbt run --select path.to.my.models     # runs all models in a specific directory
   $ dbt run --select my_package.some_model # run a specific model in a specific package
   $ dbt run --select tag:nightly           # run models with the "nightly" tag
   $ dbt run --select path/to/models        # run models contained in path/to/models
   $ dbt run --select path/to/my_model.sql  # run a specific model by its path
   ```

   ```yaml --exclude theme={"system"}
   $ dbt run --exclude my_dbt_project_name
   $ dbt run --exclude my_dbt_model
   $ dbt run --exclude path.to.my.models
   $ dbt run --exclude my_package.some_model
   $ dbt run --exclude tag:nightly
   $ dbt run --exclude path/to/models
   $ dbt run --exclude path/to/my_model.sql
   ```

## Variables

Add additional variables to your `dbt` command by writing the following in your block’s code:

```yaml theme={"system"}
# other optional dbt command line arguments

--vars '{"key": "value"}'
```

## Interpolate values

Interpolate values in the block’s code using data from:

1. Upstream block output
2. Variables
   1. Global variables
   2. Pipeline variables
   3. Runtime variables
3. Environment variables

### Upstream block output

Use the data from 1 or more upstream block’s output by using the `block_output` function.

#### `block_uuid`

The UUID of the upstream block to get data from.
If argument isn’t present, data from all upstream blocks will be fetched.

| | |
| - | - |
| Optional | ✅ |
| Type | `str` |
| Example | `'data_loader_block'` |

#### `parse`

A lambda function to parse the data from an upstream block’s output.
If the `parse` argument isn’t present, then the fetched data from the upstream block’s output
will be interpolated as is.

| | |
| - | - |
| Optional | ✅ |
| Type | `function` |
| Example | `lambda data, variables: data['runtime'] * variables['tries']` |

**Arguments**

* `data`

  If the `block_uuid` argument isn’t present, then the 1st argument in the lambda function is
  a list of objects.

  The list of objects contain the data from an upstream block’s output.
  The positional order of the data in the list corresponds to the current block’s upstream blocks
  order.

  For example, if the current block has the following upstream blocks with the following output:

  1. `load_api_data`: `[1, 2, 3]`
  2. `load_users_data`: `{ 'mage': 'powerful' }`

  Then the 1st argument in the lambda function will be the following list:

  ```python theme={"system"}
  [
      [1, 2, 3],
      { 'mage': 'powerful' },
  ]
  ```

  | | |
  | - | - |
  | Optional | ❌ |
  | Type | If `block_uuid` argument is present, then the type depends on the output from that block. If `block_uuid` isn’t present, then the type is `list`. |
  | Example | `{ 'mage': 'powerful' }` |

* `variables`
  A dictionary containing pipeline variables and runtime variables.

  | | |
  | - | - |
  | Optional | ❌ |
  | Type | `dict` |
  | Example | `{ 'fire': 40 }` |

#### Example

With `block_uuid`

```yaml theme={"system"}
--select models/{{ block_output('data_loader_block') }}.sql
```

```yaml theme={"system"}
--select models/{{ block_output('data_loader_block', parse=lambda data, variables: data['runtime'] * variables['tries']) }}.sql
```

```yaml theme={"system"}
--vars '{"user_id": "{{ block_output('load_recent_user', parse=lambda user, _variables: user['id']) }}"}'
```

Without `block_uuid`

```yaml theme={"system"}
--select models/{{ block_output() }}.sql
```

```yaml theme={"system"}
--select models/{{ block_output(parse=lambda outputs, variables: outputs[1]['runtime'] * variables['tries']) }}.sql
```

```yaml theme={"system"}
--vars '{"user_id": "{{ block_output(parse=lambda outputs, _variables: outputs[0]['id']) }}"}'
```

### Variables

Interpolate values from a dictionary containing keys and values from:

1. Global variables
2. Pipeline variables
3. Runtime variables

#### Example

```yaml theme={"system"}
--select models/{{ variables('some_var_name') }}.sql
```

```yaml theme={"system"}
--vars '{"mage": "{{ variables('power') }}"}'
```

### Environment variables

Interpolate values from the environment variables.

#### Example

```yaml theme={"system"}
--select models/{{ env_var('some_environment_variable_name') }}.sql
```

```yaml theme={"system"}
--vars '{"environment": "{{ env_var('ENV') }}"}'
```


# Run a single model
Source: https://docs.mage.ai/guides/dbt/run-single-model



1. Under the data loader block you just added, click the button **`dbt model`**,
   then click the option `Single model`.
2. In the file browser that pops up, click the file of the dbt model you want to
   add to your pipeline.
3. In the `dbt profile target` input field, enter the name of the dbt connection
   profile (e.g. `dev`) that you want to use when running the selected dbt
   models.

![](https://mage-ai.github.io/assets/dbt/add-dbt-model.gif)

***

## Depending on upstream blocks

If your dbt model references other models, those models will also be added to
the current pipeline as upstream blocks.

Once you’ve added 1 or more dbt models to your pipeline, you can set its
dependencies on other blocks.

The dbt model won’t run until all upstream blocks have successfully been
completed.

***

## Preview dbt model results

You can run a dbt model block and see the results of the SQL query.

Under the hood, Mage is running
[`dbt compile`](https://docs.getdbt.com/reference/commands/compile) for that
single model, then executing the compiled SQL query in the data source from your
dbt project’s
[connection profile](https://docs.getdbt.com/docs/get-started/connection-profiles)
target.

![](https://mage-ai.github.io/assets/dbt/dbt-preview.gif)

***

## Pipeline execution run

When pipeline is triggered and executes, it’ll run each dbt model block using
the [`dbt run`](https://docs.getdbt.com/reference/node-selection/syntax)
command.

***

## Variable interpolation

You can add
[variables](/getting-started/runtime-variable)
specific to your pipeline. These variables are accessible in each block of your
pipeline.

In addition, all the environment variables are accessible within the SQL query
using the `var` syntax.

| Syntax | Description | Example |
| - | - | - |
| `{{ var('...') }}` | Get a value from the environment variables or runtime variables. | `{{ var('ds') }}` |

### SQL statement

Within the SQL query in a dbt model block, you can access these variables using
the `var` syntax. To learn more, read
[dbt’s documentation](https://docs.getdbt.com/reference/dbt-jinja-functions/var).
For example:

```SQL theme={"system"}
SELECT *
FROM users_v1
WHERE created_at >= '{{ var("ds") }}'
```

In the above example, the variable named `ds` is defined as part of the runtime
variables for the pipeline.

### Upstream block output

You can also use the output from an upstream Mage block inside a dbt SQL model
with `block_output`. This is useful when a Python, SQL, or dynamic block decides
which value the dbt model should query, filter, or materialize.

For example, an upstream data loader named `select_city` can return a record:

```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
    return [
        {
            'city': 'seattle',
            'run_label': 'daily_refresh',
        },
    ]
```

Then a dbt SQL model block can interpolate that output before dbt runs:

```sql theme={"system"}
select
  '{{ block_output("select_city", parse=lambda rows: rows[0]["city"]) }}' as city,
  '{{ block_output("select_city", parse=lambda rows: rows[0]["run_label"]) }}' as run_label
```

If the upstream block is dynamic, Mage resolves the output for the current
dynamic run and passes that value to the dbt model. For structured outputs, use
`parse` to return a scalar value that is valid SQL.

### Run one dbt model for each dynamic output

For example, create a data loader named `dynamic_cities` that returns one item
for each dbt model run:

```python theme={"system"}
@data_loader
def load_dynamic_cities(*args, **kwargs):
    outputs = [
        [{'city': 'seattle'}],
        [{'city': 'portland'}],
    ]
    metadata = [{}, {}]

    return [outputs, metadata]
```

In the Pipeline Editor, mark `dynamic_cities` as a dynamic block. Then add a dbt
SQL model directly downstream from it:

```sql theme={"system"}
select
  '{{ block_output(
      "dynamic_cities",
      parse=lambda rows: rows[0]["city"]
  ) }}' as city
```

When the pipeline runs, Mage creates two dynamic runs of the downstream dbt
block. The first run receives the output at index `0` and executes SQL containing
`select 'seattle' as city`. The second receives index `1` and executes SQL
containing `select 'portland' as city`.

These are dynamically created block runs, not copies created with the manual
block replication feature. The optional metadata entries can include
`block_uuid` values when you want stable, descriptive identifiers for the runs.

Mage only hydrates `block_output(...)` before invoking dbt. Other dbt Jinja such
as `{{ config(...) }}`, `{{ ref(...) }}`, and `{{ source(...) }}` is left for dbt
to compile. Hydrated dbt SQL is written to an isolated temporary copy of the dbt
project for each execution, so concurrent runs of the same model do not overwrite
the source model file or read another run's interpolated SQL.

***

## Adding variables when running a YAML dbt block

In addition to interpolating variables described above, you can define variables directly
in the dbt code block if it is a YAML dbt code block (i.e. not a SQL dbt block) by following
these steps:

1. In the Pipeline Editor, add a dbt model by selecting on the `All models (w/option exclusion)`
   option from the dropdown when clicking on the `dbt model` button for adding a block.
2. For the newly added code block, you should see that is `YAML` type indicated in the top left
   corner of the block.
3. On the same line as your `--select` or `--exclude` syntax, add the `--vars` syntax with the
   variables you want to include. Note: Putting the `--vars` syntax on a separate line may cause
   issues.
4. Format the variables like a JSON object, so there should be double quotes around the keys and
   string values. Single quotes surrounding the variables object are optional.
5. You can interpolate global or environment variables using double brackets around the
   variable name. See below for examples.

Examples of YAML dbt code block content (note the syntax for the variables object):

```
--select demo/models --vars '{"demo_key": "demo_value", "date": 20230101}'
--select demo/models --vars {"demo_key":"demo_value","date":20230101}
--select demo/models --vars '{"global_var": {{ test_global_var }}, "env_var": {{ test_env_var }}}'
--select demo/models --vars {"refresh":{{page_refresh}},"env_var":{{env}}}
```


# Running dbt tests
Source: https://docs.mage.ai/guides/dbt/running-dbt-tests



You can write tests for your dbt models by following
[dbt’s documentation](https://docs.getdbt.com/docs/build/tests).

When a pipeline runs, each dbt model block will run and then each test
associated with that dbt model will be ran.

For example, if you add a dbt block to your pipeline for a dbt model named
`my_third_dbt_model` and a test file named `test_my_third_dbt_model.sql` with
the following folder structure:

```
demo_project/
|   dbt/
|   |   demo/
|   |   |   models/
|   |   |   |   example/
|   |   |   |   |   my_third_dbt_model.sql
|   |   |   tests/
|   |   |   |   example/
|   |   |   |   |   test_my_third_dbt_model.sql
```

Then whenever your pipeline runs, the test in `test_my_third_dbt_model.sql`
will also be ran after the dbt run for the model `my_third_dbt_model` is
complete.

Any errors in the dbt test will also fail the pipeline run. The output of the
test will be logged and viewable when inspecting the pipeline’s run and logs.

***

## Folder and file naming conventions

In your tests folder, you must mirror the path of the model you want to test.

For example, if your model is named `my_third_dbt_model` and if its located
under the folder named `example`, then your test file must be located here:

```
models/example/my_third_dbt_model.sql
```

```
tests/example/my_third_dbt_model.sql
```

Note that the model file and test file have the same name (e.g.
`my_third_dbt_model.sql`) and are nested under a folder with identical names
(e.g. `example`).

***

## Example

The content of the dbt model named `my_third_dbt_model` is:

```sql theme={"system"}
SELECT 1 AS n_constant_number
```

### Failed test

The content of the test for `my_third_dbt_model.sql` that will cause a test
failure is:

```sql theme={"system"}
SELECT n_constant_number
FROM {{ ref('my_third_dbt_model') }}
GROUP BY n_constant_number
HAVING (n_constant_number = 1)
```

Here is the sample output of a failed test for the above files:

```
23:57:21  Running with dbt=1.3.0
23:57:21  Found 3 models, 4 tests, 0 snapshots, 0 analyses, 289 macros, 0 operations, 0 seed files, 6 sources, 0 exposures, 0 metrics
23:57:21
23:57:23  Concurrency: 1 threads (target='dev')
23:57:23
23:57:23  1 of 1 START test my_third_dbt_model ........................................... [RUN]
23:57:23  1 of 1 FAIL 1 my_third_dbt_model ............................................... [FAIL 1 in 0.76s]
23:57:24
23:57:24  Finished running 1 test in 0 hours 0 minutes and 3.08 seconds (3.08s).
23:57:24
23:57:24  Completed with 1 error and 0 warnings:
23:57:24
23:57:24  Failure in test my_third_dbt_model (tests/example/my_third_dbt_model.sql)
23:57:24    Got 1 result, configured to fail if != 0
23:57:24
23:57:24    compiled Code at target/compiled/demo/tests/example/my_third_dbt_model.sql
23:57:24
23:57:24  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Traceback (most recent call last):
  File "/home/src/mage_ai/cli/main.py", line 146, in <module>
    main()
  File "/home/src/mage_ai/cli/main.py", line 116, in main
    ExecutorFactory.get_pipeline_executor(pipeline).execute(
  File "/home/src/mage_ai/data_preparation/executors/pipeline_executor.py", line 28, in execute
    asyncio.run(self.pipeline.execute(
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/src/mage_ai/data_preparation/models/pipeline.py", line 233, in execute
    await execution_task
  File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 147, in run_blocks
    await asyncio.gather(*remaining_tasks)
  File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 94, in execute_and_run_tests
    run_dbt_tests(
  File "/home/src/mage_ai/data_preparation/models/block/dbt/utils/__init__.py", line 466, in run_dbt_tests
    raise Exception('dbt test failed.')
Exception: dbt test failed.
```

### Successful test

The content of the test for `my_third_dbt_model.sql` that will yield a passing
test is:

```sql theme={"system"}
SELECT n_constant_number
FROM {{ ref('my_third_dbt_model') }}
GROUP BY n_constant_number
HAVING (n_constant_number = 0)
```

Here is the sample output of a successful test for the above files:

```
23:59:56  Running with dbt=1.3.0
23:59:56  Found 3 models, 4 tests, 0 snapshots, 0 analyses, 289 macros, 0 operations, 0 seed files, 6 sources, 0 exposures, 0 metrics
23:59:56
23:59:57  Concurrency: 1 threads (target='dev')
23:59:57
23:59:57  1 of 1 START test my_third_dbt_model ........................................... [RUN]
23:59:58  1 of 1 PASS my_third_dbt_model ................................................. [PASS in 0.72s]
23:59:59
23:59:59  Finished running 1 test in 0 hours 0 minutes and 3.00 seconds (3.00s).
23:59:59
23:59:59  Completed successfully
23:59:59
23:59:59  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
```


# Make dbt magical
Source: https://docs.mage.ai/guides/dbt/setup-dbt

⏰ Run dbt models in Mage in under a minute.

![dbt](https://www.getdbt.com/ui/img/social/facebook.png)

## About

Our demo dbt repo currently offers two quickstarts: one that runs a simple dbt pipeline and one that runs a dbt pipeline from an external repo. At this time, running dbt in Mage requires Docker, so both of our demos are `docker compose` based.

## Prerequisites

1. [Docker](https://docs.docker.com/engine/install/)
2. [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)

## Configuration

The following command will clone the [demo repo](https://github.com/mage-ai/dbt-quickstart), copy `dev.env` to `.env` and run `docker compose up` to start Mage *and* a Postgres database.

```bash Mac/Linux theme={"system"}
git clone https://github.com/mage-ai/dbt-quickstart mage-dbt-quickstart \
&& cd mage-dbt-quickstart \
&& cp dev.env .env && rm dev.env \
&& docker compose up
```

```bash Windows theme={"system"}
git clone https://github.com/mage-ai/dbt-quickstart mage-dbt-quickstart
cd mage-dbt-quickstart
cp dev.env .env
rm dev.env
docker compose up
```

After running the above command, open the Mage overview page at [http://localhost:6789](http://localhost:6789). Click the pipelines icon on the left to enter the pipelines overview.

![Open Pipelines](https://mage-ai.github.io/assets/dbt/setup-dbt/go-to-pipelines.gif)

## Tutorial

> **A simple dbt pipeline**
>
> First, select the `simple_dbt_python_pipeline` by double-clicking the row from the list of pipelines this will take you directly to the editor. You can also single click,
> then select the "code" icon from the side nav.
>
> Scroll to the bottom-most cell and click *Execute with all upstream blocks*.
>
> ![Run simple pipeline](https://mage-ai.github.io/assets/dbt/setup-dbt/simple-pipeline.png)
>
> You just:
>
> * Seeded a dbt model
> * Performed a dbt transformation
> * Took the transformed data and performed a Python transformation
> * Wrote the data to a Postgres source
>
> To see the output, you can use a querying tool (like [DataGrip](https://www.jetbrains.com/datagrip/) or [psql](https://www.postgresql.org/docs/current/app-psql.html)) to the locally hosted Postgres database and query `public.analytics.cur_customers`.
>
> | customer\_id | first\_name | last\_name | letters\_first\_name | is\_alliterative |
> | - | - | - | - | - |
> | 1 | Michael | P. | 7 | false |
> | 2 | Shawn | M. | 5 | false |
> | 3 | Kathleen | P. | 8 | false |
> | 4 | Jimmy | C. | 5 | false |
> | 5 | Katherine | R. | 9 | false |
> | 6 | Sarah | R. | 5 | false |
> | 7 | Martin | M. | 6 | true |
> | 8 | Frank | R. | 5 | false |
> | 9 | Jennifer | F. | 8 | false |
> | 10 | Henry | W. | 5 | false |

> **Execute dbt from an external repo**
>
> Select the `dynamic_dbt_pipeline` by double-clicking the row from the list of pipelines— this will take you directly to the editor. You can also single click, then select the "code" icon from the side nav.
>
> Scroll to the bottom-most cell and click *Execute and run all upstream blocks*.
>
> ![Compile external pipeline](https://mage-ai.github.io/assets/dbt/setup-dbt/external-pipeline.png)
>
> You just:
>
> * Pulled a GitHub repo and wrote it to your local Mage directory
> * Wrote a demo `profiles.yaml` file that interpolated environment variables from your instance
> * Compiled `dbt build` to write data to your local postgres database.
>
> To run this pipeline, open the pipeline's *Triggers* page from the pipeline navigation and select *Run @once* to trigger a pipeline run. Select the *logs* icon to see the run in realtime!
>
> ![Trigger dbt](https://mage-ai.github.io/assets/dbt/setup-dbt/trigger-dbt.gif)
>
> Now, in addition to pulling the external Jaffle Shop demo and writing the necessary profiles, you'll perform [`dbt build`](https://docs.getdbt.com/reference/commands/build), which will: run models, test tests, snapshot snapshots, and seed seeds. 🥳
>
> To see the output, you can use a querying tool (like DataGrip or psql) to the locally hosted Postgres database. To learn more about the Jaffle Shop demo, [check out the repo](https://github.com/dbt-labs/jaffle_shop).


# Run dbt snapshot
Source: https://docs.mage.ai/guides/dbt/snapshots

Run multiple individual snapshots or bulk snapshots

## Multiple individual snapshots

You can add 1 or more snapshot SQL files into your pipeline as a dbt block.

Add a dbt block as if you were adding a [dbt model](/dbt/run-single-model).
However, select a snapshot file instead of a model file.

## Bulk snapshots

Instead of adding a snapshot file to your pipeline 1 at a time,
you can run the `dbt snapshot` command on an entire folder or exclude certain folders
by doing the following:

1. Add a dbt block and select the option labeled **Generic dbt command**.
2. In the text input field labeled **command**, enter the value `snapshot`.
3. You can optionally add the `--select` or `--exclude` flag in the body of the block.


# dbt sources and upstream dependencies
Source: https://docs.mage.ai/guides/dbt/sources



dbt model blocks can depend on other dbt models and other blocks:
[sensors](/design/blocks/sensor), Python,
[SQL](/guides/blocks/sql-blocks),
[R](/guides/blocks/r-blocks).

***

## dbt models depending on other dbt models

When you add a dbt model block to your pipeline, if it contains any `ref`s,
those dbt models will be added to the current pipeline as upstream dependencies
to the dbt model you initially added.

![](https://mage-ai.github.io/assets/dbt-model-refs.gif)

***

## dbt models depending on other blocks

When a dbt model depends on a block that isn’t another dbt model, a YAML file is
created named `mage_sources.yml`. This file is automatically generated and
updated whenever a new non-dbt model block is an upstream dependency of a dbt
model block.

If you have a Mage project named `demo_project` with a dbt project named `demo`,
here is an example of where the `mage_sources.yml` file could be located:

```
demo_project/
|   dbt/
|   |   demo/
|   |   |   models/
|   |   |   |   mage_sources.yml
|   |   |   |   example/
|   |   |   |   |   my_first_dbt_model.sql
|   |   |   |   |   my_second_dbt_model.sql
```

### Sources

The contents of the `mage_sources.yml` could look like this:

```yaml theme={"system"}
sources:
- loader: mage
  name: mage_demo
  schema: public
  tables:
  - identifier: mage_dbt_demo_pipeline_load_data
    meta:
      block_uuid: load_data
      pipeline_uuid: dbt_demo_pipeline
    name: dbt_demo_pipeline_load_data
version: 2
```

The source name follows this convention: `mage_[DBT_project_name]`.

The table names under the source follow this convention:
`[pipeline_uuid]_[block_uuid]`.

To reference the preceding block in a dbt block use this convention:
`{{ source('[source_name]', '[pipeline_uuid]_[block_uuid]') }`
e.g.
`{{ source('mage_demo', 'dbt_demo_pipeline_load_data') }`

For more information on sources, read
[dbt’s documentation](https://docs.getdbt.com/docs/build/sources).


# dbt variable interpolation
Source: https://docs.mage.ai/guides/dbt/variable-interpolation



## Overview

Mage allows users to reference variables specific to your
[pipeline](/getting-started/runtime-variable) or
[project](/development/variables/environment-variables).
These variables are accessible throughout your dbt profiles, and pipelines by using the following syntax.

| Syntax | Description | Example |
| - | - | - |
| `{{ env_var('...') }}` | Get from environment variables. | `{{ env_var('ENV') }}` |
| `{{ variables('...') }}` | Get from runtime variables (used in YAML). | `{{ variables('execution_date') }}` |
| `{{ var('...') }}` | Get from runtime variables (used in SQL). | `{{ var('execution_date') }}` |

DBT also comes with variable and environment variable support. By default we forward
[pipeline](/getting-started/runtime-variable)
variables as dbt [Project variables](https://docs.getdbt.com/docs/build/project-variables).

## Variable interpolation examples

### dbt models

```sql theme={"system"}
  SELECT
    *
  # get the global environment variable for prod/dev
  FROM {{ env_var('ENV') }}_users.accounts
  WHERE
    # limit query based on the runtime variable, execution_date
    created_at BETWEEN '2023-01-01' AND {{ var('execution_date') }}
```

### dbt profiles

```yaml theme={"system"}
jaffle_shop:
  target: dev
  outputs:
    dev:
      type: postgres
      host: host.docker.internal
      user: postgres
      password: postgres
      port: 5432
      dbname: postgres
      schema: {{ variables('DBT_SCHEMA') }}
      threads: 4
```

### dbt targets

![dbt target](https://mage-ai.github.io/assets/connection-profiles/dbt-target.png)

Here, we've used a manually entered target to specify a postgres database dependent on the environment `postgres_{{ env_var('ENV') }}`.

| Environment | Key | Example string | Result |
| - | - | - | - |
| `prod` | `ENV` | `postgres_{{ env_var('ENV') }}` | `postgres_prod` |
| `dev` | `ENV` | `postgres_{{ env_var('ENV') }}` | `postgres_dev` |


# Block dependency tree
Source: https://docs.mage.ai/guides/developer-ux/block-dependency-tree

Money does grow on trees... if you plant them with magic beans.

## Drag-and-drop blocks to update dependencies

![Drag-and-drop blocks.](https://mage-ai.github.io/assets/dev-ux/dependency-tree/drag-block.gif)

Click a block in the tree, hold down the mouse, and drag it to another block and release.

This will set the dragged block as a downstream block from the one that you dropped it on.

### Drag an entire group of blocks

![Drag an entire group of blocks.](https://mage-ai.github.io/assets/dev-ux/dependency-tree/drag-group.gif)

Blocks in the tree are visually grouped if every block in the group has the same set of
upstream blocks.

When a group is visually formed, you can drag-and-drop the entire group onto another block
and the entire group will be set as the downstream for the block you dropped it on.

***

## Right click menu

![Right click menu.](https://mage-ai.github.io/assets/dev-ux/dependency-tree/right-click.png)

Right click on any block to reveal a dropdown menu full of useful commands.

1. Run block
2. Rename block
3. Show all dependencies
   ![Show all dependencies](https://mage-ai.github.io/assets/dev-ux/dependency-tree/show-all-dependencies.png)
4. Add upstream block
5. Add downstream block
6. Remove upstream dependencies
7. Remove downstream dependencies
8. Add/Edit interactions
9. Delete block
10. Delete block (ignore) dependencies
11. View file versions

***

## Edit the connection between blocks

![Edit the connection between blocks.](https://mage-ai.github.io/assets/dev-ux/dependency-tree/edit-connection.png)

Click on the line between any block and a menu will appear with these options:

1. Add new block between
2. Remove connection

***

## Drag lines to connect blocks

![Drag lines to connect blocks.](https://mage-ai.github.io/assets/dev-ux/dependency-tree/drag-orbs.gif)

Hover your mouse over a block to reveal 2 orbs (aka circles, dots, etc.).
Click and hold down on the top orb and drag it to another block then release.
This will set the 1st block as the downstream dependency of the 2nd block.

If you click on the bottom orb, then the 1st block will be set as the upstream dependency of the 2nd block.

***


# Code editor with enhanced productivity
Source: https://docs.mage.ai/guides/developer-ux/code-browser

Code browser and editor with syntax highlighting, code formatting, and version history.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=code-browser)

![Mage Pro Code editor](https://mage-ai.github.io/assets/pro/features/code-editor.jpg)

1. Multi-column and multi=row layout
2. Support for rendering files of all types across every Mage and non-Mage projects
3. Automatically format code styling and run linters to fix syntax
4. Customizable project and pipeline settings per environment.
5. Manage multiple Mage projects and any other files from a single application
6. Local file edit tracking and version history for restoring changes made in the past
7. Syntax highlighting
8. Extensions


# Pipeline editor canvas
Source: https://docs.mage.ai/guides/developer-ux/code-canvas

Build, run, and manage complex pipelines with the new canvas mode.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=pipeline-editor-canvas)

![Mage Pro Pipeline editor canvas](https://mage-ai.github.io/assets/pro/features/code-editor-canvas.jpg)


# Writing data pipeline code
Source: https://docs.mage.ai/guides/developer-ux/code-editor

Build the future you want to see.

## Adding blocks made easy

You can quickly add blocks of code to your pipeline with these methods:

> **Drag blocks from the file browser and drop it on the pipeline**
>
> > [!NOTE]
> >
> > **Supported block type directories:** `callbacks`, `conditionals`, `custom`, `data_exporters`, `data_loaders`, `dbt`,
> > `dbts`, `markdowns`, `scratchpads`, `sensors`, and `transformers`
> >
> > Version 0.9.64 or greater is required to add `callback` and `conditional`
> > blocks from the file browser.
>
> 1. Open the file browser on the Pipeline Editor page.
> 2. Click a block file and keep holding down.
> 3. Drag the block file onto the center of the page (the Notebook area).
> 4. Release the button and it’ll be added to the pipeline.
>
> ![Drag blocks from the file browser and drop it on the pipeline](https://mage-ai.github.io/assets/dev-ux/code-editor/drag-block.gif)

> **Search blocks across entire projects**
>
> Find existing code and add it to the current pipeline.
>
> > [!NOTE]
> >
> > You must first enable the feature named **Add new block v2** in your
> > [project settings](http://localhost:6789/settings/workspace/preferences).
> > This feature is available in standard (batch) pipelines.
> >
> > If you just recently enabled the feature **Add new block v2**, you must restart Mage
> > so that your blocks can be indexed.
> >
> > Version 0.9.64 or greater is required to add `callback` and `conditional`
> > blocks from the block search.
>
> Instead of clicking a button to add a new block, type in the search bar to find an
> existing block across all your projects.
>
> ![Search blocks across entire projects](https://mage-ai.github.io/assets/dev-ux/code-editor/block-search.gif)
>
> > [!TIP]
> >
> > ⌘ + / (Command + Forward Slash) keyboard shortcut will instantly focus your typing on the
> > search bar from anywhere on the page (as long as you aren’t already typing in a
> > block of code).
>
> > [!TIP]
> >
> > Use the arrow up and arrow down key to browse
> > through the results.

> **Custom code templates for reducing boilerplate work**
>
> Create a template for a block or a pipeline that can be reused throughout your project.
>
> <video />
>
> [Read more here](/custom-templates/overview)

***

## Notes on editing pipelines

> [!WARNING]
>
> Avoid opening and editing the **same pipeline in multiple browser tabs or windows** at the same time.
> Changes from one tab may overwrite changes from another.

Tips to stay safe:

* Edit a pipeline in **one tab only**.
* If you need to monitor the pipeline elsewhere, use another tab in **read-only mode**.
* Close extra editing tabs if opened by mistake.

***

### Collaboration best practices

If multiple users need to develop the **same pipeline**:

* Use **[Workspace mode in Mage Pro](/guides/developer-ux/workspaces)**, which provides:
  * Isolated environments per user or team.
  * Safe collaboration without overwriting each other’s changes.
  * The ability to merge updates back into shared pipelines.

This ensures smooth teamwork and prevents conflicts when more than one person is editing.

***

## Re-arrange and hide or show blocks

> **Re-arrange the order of blocks**
>
> Drag a block and move it up or down by dropping it on another block.
>
> ![Drag a block and move it up or down by dropping it on another block.](https://mage-ai.github.io/assets/dev-ux/code-editor/rearrange-blocks.gif)

> **Filter files to the current pipeline**
>
> When editing a pipeline, open the **Files** tab and use the selector above the search box
> to switch between **All** and **This pipeline**. The
> **This pipeline** filter shows repo-backed files related to the current pipeline,
> including pipeline metadata and configuration files, trigger configuration, referenced block
> source files, and supporting project configuration such as `io_config.yaml`,
> `metadata.yaml`, and `requirements.txt`. Runtime data and log files
> are not included.

> **Block visibility tab**
>
> Use the **Visibility** tab to hide blocks in the notebook. Hidden blocks are collapsed,
> and you can click a hidden block to expand it again. This can help optimize vertical screen
> space while writing code.
>
> ![Hide and show blocks in the current pipeline](https://mage-ai.github.io/assets/dev-ux/file-browser/blocks-in-pipeline.gif)

***

## Pre-built data integrations for 3rd party sources

![Pre-built data integrations for 3rd party sources](https://mage-ai.github.io/assets/batch-pipelines/data-integrations/configuration-credentials.png)

Sync data from 3rd party sources without writing boilerplate code by using pre-built
data integrations (Singer Taps and Targets) provided by the data engineering community.

> [!TIP]
>
> Data integrations to sync your data to 3rd party destinations are also available in the
> same feature set.

> [!TIP]
>
> These sources and destinations can be combined and chained together with
> your existing blocks.

[Read more here](/data-integrations/batch-pipelines)

***

## Use AI to write code for you

A line of code you don’t write is a line of code you never have to debug.
Use AI to generate the code for you.

![Generate code using AI.](https://mage-ai.github.io/assets/dev-ux/code-editor/ai-generate-code.gif)

[Read more here](/guides/ai/overview#generate-block-using-ai)

***

## Document and comment code (AI)

“Work smart, not hard.”

Let the robots document your code so you can focus on the fun and creative side of coding.

> **Document the code from a block**
>
> ![Document the code from a block.](https://mage-ai.github.io/assets/dev-ux/code-editor/ai-document-block.gif)
>
> [Read more here](/guides/ai/overview#add-documentation-for-a-block)

> **Write comments inline with the code**
>
> ![Write comments inline with the code.](https://mage-ai.github.io/assets/dev-ux/code-editor/ai-code-comments.gif)
>
> [Read more here](/guides/ai/overview#add-comments-in-a-block)

> **Document the entire pipeline and each block in it**
>
> ![Document the entire pipeline and each block in it.](https://mage-ai.github.io/assets/dev-ux/code-editor/ai-document-pipeline.gif)
>
> [Read more here](/guides/ai/overview#add-documentation-for-a-pipeline-and-all-its-blocks)

***

## Block upgrades

Enhance your code with these quick power-ups:

> **Recursively dynamic code made simple**
>
> ![Recursively dynamic code made simple.](https://mage-ai.github.io/assets/dev-ux/code-editor/dynamic-blocks.png)
>
> A very `powerful` **ultimate** *spell*.
>
> > [!WARNING]
> >
> > With great power... comes great fun.
>
> [Read more here](/design/blocks/dynamic-blocks)

> **Replicate the code of an existing block**
>
> ![Replicate the code of an existing block](https://mage-ai.github.io/assets/dev-ux/code-editor/replicate-block.gif)
>
> [Read more here](/guides/blocks/replicate-blocks)

> **Conditionally execute code**
>
> ![Conditionally execute code](https://mage-ai.github.io/assets/blocks/conditionals-example.png)
>
> [Read more here](/design/blocks/conditionals)

> **Callbacks**
>
> A callback block is associated to another block.
> When the parent block succeeds or fails, the callback block functions are executed.
>
> [Read more here](/design/blocks/callbacks)

***

## Add new block v2

An improved experience for adding new blocks to your pipeline in the
Pipeline Editor (`/pipelines/[pipeline_uuid]/edit`), which allows you to
search for existing block files and use data integration blocks inside of
a standard (batch) pipeline.

***

## Code and output side-by-side view

Run code and see its output right next to where you’re writing.

This makes it so that you don’t have to keep scrolling down every time you make a code change
and want to see its results.

For table outputs, click a column header to sort the preview or use the filter
icon beside the header to filter rows by column value. Use **Columns** to search
for columns and choose which ones appear in the preview. **Deselect all** hides
every column, while **Reset** restores the initial selection. Wide DataFrame
previews initially show the first 100 columns, and the picker can reveal any of
up to 1,000 sampled columns. These controls are temporary preview tools and don’t
change the underlying block output.

![Code and output side-by-side view](https://mage-ai.github.io/assets/dev-ux/code-editor/split-view.gif)

To enable this feature:

1. Go to your [project settings](http://localhost:6789/settings/workspace/preferences).
2. Enable the feature named **Notebook block output split view**.
3. Open the edit view of a pipeline.
4. Under the header menu named **View**, click the option **Show output next to code**.

> **Check out this demo for a quick overview**
>
> [YouTube video player](https://www.youtube.com/embed/x4VLyIG6fCc)

***


# Browse all files
Source: https://docs.mage.ai/guides/developer-ux/file-browser

Add, edit, move, and delete files or folders in your project.

## How to show the file browser

![File browser](https://mage-ai.github.io/assets/dev-ux/file-browser/overview.gif)

1. When editing a pipeline, expand the left side panel to show the file browser.
2. Go to the Files page [http://localhost:6789/files](http://localhost:6789/files).
3. Launch the Text Editor application.

***

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=file-browser-project-shortcuts)

## Project folder shortcuts

The Pipeline Editor and Apps/Coder file browsers include toolbar and context-menu
shortcuts to help you focus on the current project folder.

* Click the folder icon in the file browser toolbar to focus the tree on the
  current project's root folder. Click it again to return to the full file tree.
  The toolbar button is highlighted while the shortcut is active, exposes its
  pressed state to assistive technology, and remembers the focused view for the
  current project context.
* Right-click a file or folder and choose **Show all project folders** for the
  same focused view. Right-click again and choose **Show all files** to return
  to the full file tree.
* In the Pipeline Editor, the project-folder shortcut is separate from the
  **All** and **This pipeline** selector. Use **This pipeline** to
  filter to files related to the current pipeline, or use the folder shortcut to
  focus the tree on the current project folder. Activating the folder shortcut
  switches the selector back to **All** because the project folder and current
  pipeline filters are alternate scopes.
* Search results follow the active scope, so searches run inside the current
  pipeline files or focused project folder when either filter is selected.

> [!NOTE]
>
> Viewer users can browse folders, search files, and use the file browser scope controls
> in read-only mode. Actions that create, rename, move, upload, delete, or otherwise
> modify files still require edit access.

***

## Right click menu

### File

* Rename file
* Move file
* Download file
* Delete file

![Right clicking a file](https://mage-ai.github.io/assets/dev-ux/file-browser/right-click--file.png)

### Folder

* Open all files in the immediate directory
* New folder
* Rename  folder
* Move folder
* Delete folder
* Upload files
* Expand all subfolders
* Collapse all subfolders
* New Mage project
* New dbt project

![Right clicking a folder](https://mage-ai.github.io/assets/dev-ux/file-browser/right-click-folder.png)

***

## File versions

When you save a file after making a change, a new file version is created and stored locally.

You can undo any change to a file by reverting to a previous version (this is different than git).

![File versions](https://mage-ai.github.io/assets/dev-ux/file-browser/file-versions.gif)

[Read more here](/getting-started/file-versions).

***

## Hidden files

The file browser will hide hidden folders and files by default. You can show them by toggling
the option **Show hidden files** under the **View** menu dropdown options.

![Hidden files](https://mage-ai.github.io/assets/dev-ux/file-browser/hidden-files.gif)

***

## Filter files to the current pipeline

When editing a pipeline, open the **Files** tab and use the selector above the search box to
switch between **All** and **This pipeline**. The **This pipeline**
filter shows repo-backed files related to the current pipeline, including pipeline metadata and
configuration files, trigger configuration, referenced block source files, and supporting project
configuration such as `io_config.yaml`, `metadata.yaml`, and
`requirements.txt`. Runtime data and log files are not included.

## Block visibility tab

Use the **Visibility** tab to hide blocks in the notebook. Hidden blocks are collapsed, and
you can click a hidden block to expand it again. This can help optimize vertical screen space
while writing code.

![Hide and show blocks in the current pipeline](https://mage-ai.github.io/assets/dev-ux/file-browser/blocks-in-pipeline.gif)

***

## Block files used/unused by pipelines

> [!NOTE]
>
> Requires version `0.9.63` or greater.

You can tell if a block is being used by a pipeline (or not used by any pipelines)
by looking at the related block file's icon in the file browser. This is a quick way
for users to see which block files are currently being used by *any* pipeline; the specific
pipeline using a block file is not indicated in the file browser.

You need to be viewing the file browser in the Files (`/files`) and Pipeline Editor
(`/pipelines/[pipeline_uuid]/edit`) pages specifically. Other pages, modals, or
components that have a file browser may not have this feature.

**Applicable block directories:** `callbacks`, `conditionals`, `custom`, `data_exporters`,
`data_loaders`, `dbt`, `extensions`, `markdowns`, `scratchpads`, `sensors`, and `transformers`

![Block files with shared pipeline icons](https://mage-ai.github.io/assets/dev-ux/file-browser/all-block-files.png)

### Block file states

There are three different states for a block file in the file browser that can affect the
icon in front of the block filename and the filename font color. Note that the square icons
can be different colors depending on which block directory it is.

1. **Used by exactly one pipeline** - Filled square icon. For example in the screenshot
   above, `aged_sea.py` and `api_load_data.py` are used by a single pipeline.
2. **Used by more than one pipeline** - Unfilled cyan diamond icon and filename with cyan
   font color. In the screenshot above, `amazon_s3.yaml` and `ancient_pine.yaml` are used by
   multiple pipelines.
3. **Not used by any pipeline** - Unfilled square icon. For example in the screenshot above,
   `bigquery_loader.py` is not used by any pipeline.

### Unused block files

If you specifically want to view the block files that are not being used by any pipeline,
you can apply the "Unused block files" filter found in the file search input at the top of
the file browser. Non-block files may still be visible when this filter is applied.

![Unused block files](https://mage-ai.github.io/assets/dev-ux/file-browser/unused-block-files.png)

***

## Display files by type

> [!NOTE]
>
> The feature is currently only available in the **Text Editor** application

Instead of viewing all files grouped by its parent directory,
you can toggle a different view that shows these files flattened and then grouped
by its file extension type.

This can help browse relevant files faster since you don’t have to dig into each folder
to find the file you’re looking for.

![Display files by type](https://mage-ai.github.io/assets/dev-ux/file-browser/files-flat.gif)


# Personalize your workspace
Source: https://docs.mage.ai/guides/developer-ux/personalize

Customize workspace themes and re-configure UI component layouts.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=personalize)

## Themes

Choose how Mage and its code editors look:

1. Open your account menu and use the **Theme** submenu to select **System**,
   **Light**, or **Dark**. You can also make the same selection under **Color
   mode** on this personalization page. **System** follows your device
   appearance and is the default when you have not chosen a mode.
2. Under **Dark syntax theme** and **Light syntax theme**, choose the editor
   colors Mage should use in each mode.

Mage applies the matching syntax theme automatically when you change color
modes. The setting also applies to the source and destination YAML configuration
editors in data integration pipelines.

When **System** is selected, Mage updates as soon as your operating system or
browser appearance changes. Mage listens for that browser preference instead of
polling or calculating a day/night schedule. Selecting **Light** or **Dark**
keeps that mode active regardless of later device appearance changes.

Your **System**, **Light**, or **Dark** selection is saved in this browser and
preserved when you reload Mage or sign out, so the sign-in page keeps your last
selection. Signing in loads that account's saved preference. If you clear the
site's cookies, the signed-out page defaults to **System**.

***

## Logo

Change the logo that appears in the top left corner.

![Custom logo design](https://mage-ai.github.io/assets/platform/design-header-logo.png)

***

## Layout

### Add block buttons

Customize the buttons that show up by default on the Add new block bar on the notebook page
while building a pipeline.

You can change the order of what appears first as well as which buttons are visible by default
and which are displayed when the dropdown menu is shown after clicking the button **All blocks**.

![Custom button layout](https://mage-ai.github.io/assets/platform/design-add-block_buttons.png)


# Cross-pipeline dependencies
Source: https://docs.mage.ai/guides/developer-ux/pipeline-dependencies

Manage cross-pipeline dependencies and execution flow across every pipeline within a project.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=pipeline-dependencies)

Use pipeline dependency views to understand how pipelines in one project depend
on each other.

![Mage Pro Pipeline dependencies](https://mage-ai.github.io/assets/pro/features/pipeline-dependencies.jpg)

## Explore the project graph

Open `/apps/pipelines/graph` to review the cross-pipeline dependency graph.
Each node represents a pipeline. Each edge points from an upstream pipeline to a
downstream pipeline:

1. **Upstream pipelines** must run first or provide data, state, or execution
   context used by the selected pipeline.
2. **Downstream pipelines** depend on the selected pipeline or run after it.
3. Selecting a pipeline focuses it and highlights the upstream and downstream
   paths that match the current trace direction.

The graph is read-only. Update pipeline relationships from the pipeline or
dependency configuration surfaces that create the underlying upstream and
downstream metadata. When the dependency response includes pipeline colors, the
graph uses those colors for the matching pipeline nodes.

## Focus on one pipeline

Use **Find pipeline** to search by pipeline name or UUID. Selecting a result
centers the graph on that pipeline, selects it, and highlights the dependency
paths that match the selected trace direction.

## Filter the neighborhood

Use the graph controls to narrow large projects:

1. **All pipelines** shows every pipeline node returned by the dependency
   source.
2. **With dependencies** hides isolated pipelines that do not have an upstream or
   downstream relationship.
3. **Both** highlights upstream and downstream pipelines around the focused
   pipeline.
4. **Upstream** highlights only pipelines that lead into the focused pipeline.
5. **Downstream** highlights only pipelines that depend on or follow the focused
   pipeline.

Clear the focus to return to the project-wide graph.

## Review pipeline relationships

Open a pipeline and select **Dependencies** to review the pipelines connected to
the current pipeline. The page groups relationships into upstream dependencies
and downstream pipelines, and shows the relationship type, active trigger count,
edge source, and latest run status when Mage can resolve that information.

Dependency edges can come from configured relationships, runtime lineage, or
pipeline code that calls helpers such as `trigger_pipeline`,
`trigger_pipeline_for_workspace`, `trigger_workspace_pipeline`, and
`check_status`.

When a pipeline is focused, highlighted graph edges show concise relationship
labels such as **Trigger**, **Sensor**, and **Global data product**.

## Trace dependencies between pipeline runs

Open an individual pipeline run and select **Run lineage** in the right-side
inspector. The inspector initially opens to **Dependency tree**; Mage requests
lineage for only the current run when you select **Run lineage**. The tab groups
the recorded relationship into:

1. **Upstream run** shows the direct parent run, when one was recorded.
2. **Downstream runs** shows up to five direct child runs created by the current
   run.

The summary reports the exact direct relationship totals, such as **1 upstream ·
22 downstream**. For a related run in the current workspace, its pipeline name,
run ID, and status are shown only when you can view that pipeline.

When a persisted relationship points to another workspace, the tab shows only a
minimal reference containing the persisted pipeline UUID and run ID. It does not
include that run's status, timestamps, duration, schedule, or trigger metadata.
The reference links to the related run in its workspace; opening that page uses
the destination page's normal authorization and denies access when you cannot
view the target pipeline.

Current-workspace runs that you cannot view are not listed as nodes. Instead,
the affected group reports an aggregate such as **4 additional downstream runs
hidden by permissions.** A persisted relationship whose run or schedule is no
longer available, or that cannot provide usable workspace navigation context,
is also reported only as an unavailable aggregate without exposing its identity.

The tab does not paginate. When more related direct children exist, the single
dependency action reports how many are not displayed and opens the pipeline's
full dependency page. For an unusually large fan-out that cannot be completely
classified by the bounded lineage lookup, the action reports that additional
downstream runs exist without estimating permission-hidden or unavailable
categories. Use the dependency page to continue exploring the pipeline-level
relationship.

### Runtime lineage versus pipeline dependencies

The **Run lineage** tab shows direct, persisted runtime trigger relationships
between specific runs. Mage records these relationships when supported helpers
such as `trigger_pipeline`, `trigger_pipeline_for_workspace`, or
`trigger_workspace_pipeline` create a child run. This view does not infer a run
relationship from the project graph, recurse through later descendants, or
reconstruct relationships for historical runs.

The project graph and pipeline **Dependencies** page describe relationships
between pipeline definitions. They can include configured dependencies,
sensors, global data products, and other evidence that does not identify a
specific parent and child run. Use the graph to understand pipeline-level
impact, and use the individual run view when you need the recorded execution-level
provenance.

Runs started by ordinary schedules or events, direct run creation, external
orchestrators, and older executions without a persisted runtime trigger edge do
not gain inferred lineage in this view. When no direct persisted relationship
exists, the tab reports that no upstream or downstream runs were recorded for
the run.

## Use diagnostics

The graph reports diagnostics when dependency metadata is incomplete or
ambiguous, such as a pipeline referencing a missing pipeline, a self-dependency,
or a cycle. Use diagnostics to decide whether the project needs dependency
metadata cleanup before relying on the graph for release planning or incident
analysis.

## Validate relationship direction

When reviewing a dependency layout, confirm the selected pipeline context before
acting:

1. Select the pipeline you care about.
2. Check whether the highlighted relationships point upstream or downstream from
   the selected pipeline.
3. Use **Upstream** to highlight the pipelines that must run before the selected
   pipeline.
4. Use **Downstream** to highlight the pipelines that can be affected by the
   selected pipeline.
5. Follow the highlighted edge labels to confirm which dependency evidence Mage
   resolved for each relationship.

For dense graphs, keep the focused pipeline selected and move between direction
filters instead of trying to interpret every edge at once.

## Cancel downstream runs

When cancelling a running pipeline run, Mage can also cancel downstream runs that
were triggered by that run. In the cancellation confirmation, select **Cancel
downstream runs triggered by this run** to preview the eligible downstream runs
before confirming.

The preview only counts downstream runs that are still running or queued and
that you have permission to cancel. Runs that are already terminal are skipped,
and runs you cannot edit are shown as blocked. Leaving the option unchecked
cancels only the selected pipeline run.


# Pipeline sidekick
Source: https://docs.mage.ai/guides/developer-ux/sidekick

Have you ever wished you had a sidekick like Batman and Robin? Now you have one.

The Pipeline sidekick appears when you edit a pipeline.

![The Pipeline sidekick appears when you edit a pipeline.](https://mage-ai.github.io/assets/dev-ux/sidekick/open-sidekick.gif)

In the top right corner of the page, click the `Chevron` or `Arrow` looking icon to expand
the right panel and reveal the Sidekick.

***

## Visualize block output data

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=embedded-analytics)

Add 1 or multiple charts to visualize the data produced by a block.
Click the chart icon in a supported block header, or click **Add chart** from
a block output preview, to open AI Sidekick with the pipeline and block context
already filled in. Describe the chart you want, and Sidekick generates Python
Plotly code plus dashboard JSON metadata for the Analytics dashboard.

When Sidekick opens for an Analytics chart, the prompt includes an Analytics
workflow attachment with the target dashboard and source pipeline or block so the
chart is saved in the expected place.

When viewing block output as a table in the Sidekick data view, click a column
header to sort the preview or use the filter icon to narrow rows by column value.
Sorting and filtering only change the current table preview; they don’t modify
the block output saved by the pipeline.

For dynamic blocks and blocks with dynamic upstream dependencies, the Sidekick
data view can show each dynamic child output separately. Use the Dynamic child
selector to switch between child outputs, then use the output tabs to move
between outputs within the selected child. Outputs without displayable data are
skipped.

![Visualize block output data.](https://mage-ai.github.io/assets/dev-ux/sidekick/add-chart.gif)

[Read more](/observability/visualizations/analytics) about how to create AI-generated
Plotly dashboards for an individual pipeline or for the entire project.

***

## Configure pipeline variables

![Configure pipeline variables.](https://mage-ai.github.io/assets/dev-ux/sidekick/variables.png)

Use variables in your code and interpolate them at runtime.
These variables can be defined at the moment the pipeline is triggered or
pre-configured from the Sidekick.

[Read more here](/getting-started/runtime-variable)

***

## Store sensitive secrets

Secrets don’t make friends... but they save companies from oblivion.

![Store sensitive secrets.](https://mage-ai.github.io/assets/create-secrets.png)

[Read more here](/development/variables/secrets)

***

## Customize settings for every individual block

1. Rename a block
2. Set what execution engine the block should use
3. Configure how the block should retry itself in case of failure
4. Set a timeout limit for how long the block can run before it should fail
5. Configure variables specific to a block
6. View which pipelines are currently using a block

![Customize settings for every individual block.](https://mage-ai.github.io/assets/dev-ux/sidekick/block-settings.png)

***


# Develop Mage Pro Code with VS Code or Cursor
Source: https://docs.mage.ai/guides/developer-ux/visual-studio-code

Learn how to develop Mage Pro pipelines and projects using Visual Studio Code or Cursor. This guide walks you through secure remote access via Tailscale and SSH, environment configuration, and integrating your local IDE with Mage Pro.


Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](<https://cloud.mage.ai/sign-up?source=visual studio code>)

[YouTube video player](https://www.youtube.com/embed/pxC8lY3SZ8U?si=M7WLPEbZg00oqVD_)

## Overview

Mage Pro supports remote development using [Visual Studio Code](https://code.visualstudio.com/) or [Cursor](https://www.cursor.so/), so you can build and manage your pipelines in a full-featured local IDE while connecting securely to your Mage Pro environment. This integration leverages [Tailscale VPN](https://tailscale.com/) for secure networking and SSH for remote access.

> ⚠️ This feature is not available by default on Trial clusters.
> To enable access, please contact the Mage team at **[pro@mage.ai](mailto:pro@mage.ai)** to request activation for your Mage Pro account.

## Core components

* **Tailscale VPN**: Secure network connection between local and remote environments
* **Remote SSH Extension**: VS Code's tool for remote development
* **Mage Pro Cluster**: Containerized environment for data pipelines
* **Authentication System**: Secure key-based access control

To integrate with VS Code or Cursor please contact Mage to request an account upgrade. The Mage team will need to enable this feature for your organization.

## Setup guide

### 1. Tailscale VPN setup

Step 1: Navigate to the Tailscale website and download the appropriate version for your operating system.

![Tailscale download ](https://mage-ai.github.io/assets/pro/features/vs-code/Tailscale-download.png)

Step 2: Install Tailscale following the in-app instructions.
Step 3: Access the Tailscale admin console from their main website.
Step 4: Generate an Auth Key by navigating to Settings > Keys and selecting "Generate Auth Key."

![Tailscale keys](https://mage-ai.github.io/assets/pro/features/vs-code/tailscale-keys.jpg)

### 2. Mage Pro configuration

#### Setting Environment Variables for the Cluster

Step 1: Navigate to your Mage Pro management cluster.

Step 2: Select "Edit/Add variable’ to add environment variables.
Step 3: Add a new custom variable:

* Key: `TS_AUTHKEY`
* Key: `TS_AUTH_KEY`
* Value: `YOUR_AUTH_KEY`

Step 4: Click the “Save changes” button to save the TS\_AUTHKEY variable.
(This will restart your cluster).

![TS Authkey Environment Variable](https://mage-ai.github.io/assets/pro/features/vs-code/tailscale-authkey.png)

#### Setting Environment Variables for Workspaces

If you are using Mage Pro **workspaces**, you must configure environment variables separately at the workspace level.

> **Note**: Each workspace requires its own **unique** `TS_AUTHKEY`.
> Generate a separate Tailscale auth key for each workspace to ensure proper network isolation and security.

To configure environment variables for a workspace, follow the guide here:
👉 [Workspace Environment Configuration](/guides/developer-ux/workspaces#environment-configuration)

Here’s an example workspace configuration:

```yaml theme={"system"}
env:
  - name: TS_AUTHKEY
    value: <workspace_specific_tailscale_auth_key>
  - name: TS_EXTRA_ARGS
    value: --accept-routes
  - name: ENV
    value: dev
  - name: USER_CODE_PATH
    value: /home/src/<your_project_folder>
```

### 3. Connection verification

Step 1: Return to the Tailscale admin panel after your cluster restarts.
Step 2: Locate your Mage Pro cluster in your Machine’s list (i.e. mageai-1-tutorials).
Step 3: Note the private IP address from the address column.

![TS Tailscale machines](https://mage-ai.github.io/assets/pro/features/vs-code/tailscale-machines.jpg)

Step 4: Open a terminal on your local machine and run the following command:

```bash theme={"system"}
curl <PRIVATE_IP_ADDRESS>:6789/api/status
```

Step 5: Verify that you receive a successful connection response.

### 4. VS Code setup

Step 1: Open Visual Studio Code on your local machine.
Step 2: Install the "Remote - SSH" extension from the Extensions marketplace (Ctrl+Shift+X or Cmd+Shift+X on macOS).

![VS Code remote SSH](https://mage-ai.github.io/assets/pro/features/vs-code/vs-code-remote-ssh-2.jpg.png)

Step 3: Open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P on macOS).
Step 4: Type "Remote-SSH: Connect to Host" and select it.

![Connect to SSH host](https://mage-ai.github.io/assets/pro/features/vs-code/vs-code-host-connect.png)

Step 5: Enter or select the correct ip address.

![Enter ip address](https://mage-ai.github.io/assets/pro/features/vs-code/vs-code-ip-address.png)

Step 6: Enter the login username and password which will first be provided to you by Mage

### 5. Accessing your code

Step 1: When prompted for the folder to open, enter your user code path.

* Default user code path: `/home/src/default_repo`

Step 3: If you've customized this path, use your specific directory.
Step 4: Click "OK" to open your project files in VS Code.

# Setting up Tailscale on AWS

Connect your AWS instances to your secure network using Tailscale to enable access to your cloud resources. When using Mage Pro in AWS environments, proper route configuration ensures reliable connectivity between your local machine and cloud resources.

## Setting up route acceptance in Tailscale

Route acceptance in Tailscale can be configured in two ways to ensure proper connectivity between your local machine and AWS-hosted Mage Pro environment.

### 1. Using the command line

Run the following command on your local machine to accept routes advertised by Tailscale nodes:

```bash theme={"system"}
tailscale up --accept-routes

```

### 2. Using the admin console

Step 1: Navigate to the Tailscale admin console ([https://login.tailscale.com/admin](https://login.tailscale.com/admin))
Step 2: Go to the "Machines" page
Step 3: Find the machine that's advertising routes (in your case, the "vpn" node)
Step 4: Click on it to view details
Step 5: Under "Subnet routes" section, approve which subnet routes should be advertised to the entire tailnet

### Understanding route configuration

The two configuration methods work together:

* Admin UI settings control which machines can advertise routes (a tailnet-wide setting)
* Command-line option (`-accept-routes`) determines whether your specific machine will accept those advertised routes

### Configuring Mage Pro for route acceptance

To ensure your Mage Pro environment automatically accepts routes when it starts, add the following environment variable:

* Key: `TS_EXTRA_ARGS`
* Value: `-accept-routes`

### Verifying route acceptance

After configuring route acceptance, verify it's working correctly:

```bash theme={"system"}
tailscale status

```

You should no longer see health warnings about routes not being accepted, and you should be able to access the subnet IPs that were being advertised.


# Mage Pro workspaces
Source: https://docs.mage.ai/guides/developer-ux/workspaces

Set up Mage Pro workspaces for organizing your own personal environment.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=workspaces)

[YouTube video player](https://www.youtube.com/embed/O84nC2aOsT4?si=LWWVenWeQDM8YXL_)

## Overview

Mage Pro workspaces provide isolated environments for developing and managing data pipelines. Each workspace operates as a separate container instance, allowing for independent resource allocation and configuration management.

## Core Components

* Runtime Environment: Containerized Python environment
* Storage System: Persistent volume for code and data
* Resource Manager: Kubernetes-based resource allocation
* Access Control: Role-based permission system

> [!NOTE]
>
> To utilize the collaborative workspace environment, please contact Mage to request an account upgrade.
> The Mage team will need to enable this feature for your organization.

## Setup Guide

* **Step 1**: After logging into the Mage Pro environment you will be in the Workspaces interface (url: [https://cluster.mage.ai/\[cluster-uuid\]/manage](https://cluster.mage.ai/\[cluster-uuid]/manage)). Press the Create new workspace button.
* **Step 2**:  Give the workspace a name.
* **Step 3**: Click the dropdown arrow and toggle on the Kubernetes configuration if you need to custom configure your Kubernetes container. (see Kubernetes container configuration section for more details).
* **Step 4**: Click the Lifecycle (optional) dropdown to create lifecycle management policies (see Lifecycle management policies section for more information).
* **Step 5**: Click the create button and wait a few minutes for your workspace to start.
* **Step 6**: Once your workspace is running, click the workspace open button under the open column in the workspace management interface.

### Troubleshoot pending or failed Kubernetes workspaces

For Kubernetes workspaces, administrators can review **Pod diagnostics** on the workspace detail page. The panel shows recent warning events and actionable container states, including image pull failures, scheduling failures caused by node affinity or insufficient resources, and non-zero container exits. Each diagnostic can include the Kubernetes reason, message, occurrence or restart count, and elapsed time. Mage limits the panel to the five most relevant diagnostics and truncates long Kubernetes messages.

The panel is visible only to administrators and appears only while a Kubernetes workspace is not running and Kubernetes returns diagnostics. After the workspace recovers and reaches `RUNNING`, Mage clears the panel instead of presenting retained Kubernetes warning events as current failures. If the Mage control-plane service account cannot list pod events, workspace status remains available but Kubernetes event messages are omitted. Mage Helm chart `0.2.16` and later grant the required `list` verb for the core Kubernetes `events` resource through the existing namespace-scoped `Role`; see [helm-charts #94](https://github.com/mage-ai/helm-charts/pull/94). For manual same-namespace installations, add the following rule and binding in the namespace where Mage manages its workspaces:

```yaml theme={"system"}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: mage-workspace-event-reader
  namespace: <mage-workspace-namespace>
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mage-workspace-event-reader-binding
  namespace: <mage-workspace-namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: mage-workspace-event-reader
subjects:
  - kind: ServiceAccount
    name: <mage-control-plane-service-account>
    namespace: <mage-workspace-namespace>
```

## Workspace Configuration Options

Mage Pro workspaces support comprehensive configuration options that allow you to customize your workspace environment, resources, and behavior. Configuration options vary by cluster type (Kubernetes, Docker).

### Basic Workspace Configuration

All workspace types support these basic fields:

* **`name`** (required): The workspace name. Must be unique within your cluster.
* **`lifecycle_config`** (optional): Lifecycle management policies (see [Lifecycle Management Policies](#lifecycle-management-policies-optional) section below).

### Kubernetes Workspace Configuration

Kubernetes workspaces support the following configuration options:

#### Container Configuration (YAML)

The `container_config` field accepts a YAML string that allows you to configure:

1. **Resource Configuration**: Controls memory and CPU allocation for your workspace container
2. **Environment Variables**: Customize environment variables for your workspace
3. **Other Container Settings**: Any other Kubernetes container configuration options

* **Step 4**: Click the create button and wait a few minutes for your workspace to start.
* **Step 5**: Once your workspace is running, click the workspace open button under the open column in the workspace management interface.

## Kubernetes container configuration

Kubernetes provides a robust container orchestration system for managing your workspace resources and environment. The configuration above covers two key aspects:

1. Resource Configuration: Controls memory and CPU allocation for your workspace container, with both minimum guarantees and upper limits to ensure stable performance.
2. Environment Configuration: Enables customization of environment variables, specifically the project code location if you've modified it from the default path.

These settings help maintain workspace stability and ensure proper resource allocation for development tasks.

### Environment configuration

#### Setting `USER_CODE_PATH`

If you've modified the default `USER_CODE_PATH` environment variable in the base project (Mage Pro management portal), update your Kubernetes configuration:

```yaml theme={"system"}
env:
  - name: USER_CODE_PATH
    value: /home/src/<path_to_folder>
```

This environment variable tells your workspace where to find your project code.

#### Adding Other Environment Variables

You can define additional environment variables in the same env section to further configure your workspace:

```yaml theme={"system"}
env:
  - name: USER_CODE_PATH
    value: /home/src/<your_project_folder>
  - name: ENV
    value: dev
  - name: TS_AUTHKEY
    value: <your_tailscale_auth_key>
  - name: TS_EXTRA_ARGS
    value: --accept-routes
```

**Note on ENV variable**: If you set `WORKSPACE_ENV_PATTERN` at the cluster level, the `ENV` variable will be automatically set based on the workspace name. You can still explicitly set `ENV` in the container configuration to override the pattern. See the [Auto-set ENV variable from workspace name](#auto-set-env-variable-from-workspace-name) section for more details.

### Resource configuration (optional)

**Optionally** configure your workspace's computing resources in Kubernetes:

```yaml theme={"system"}
resources:
  requests: # Minimum resources guaranteed
    memory: "2Gi" # 2 gigabytes of RAM
    cpu: "1" # 1 full CPU core
  limits: # Maximum resources allowed
    memory: "4Gi" # 4 gigabytes RAM ceiling
    cpu: "2" # Can use up to 2 CPU cores
```

This configuration ensures your workspace has adequate resources for development while preventing resource overconsumption. The requests represent guaranteed minimums, while limits prevent your workspace from using more than its allocated share.

#### Default Resource Configuration

These environment variables set default resource requests and limits for Kubernetes workspace containers. They are applied when a workspace's `container_config` does not specify resources (or when specific fields like `requests.cpu` or `limits.memory` are omitted). User-specified values in `container_config` always take precedence.

| Variable | Purpose | Example Value |
| - | - | - |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU` | Default CPU request for workspace containers | `500m` |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY` | Default memory request for workspace containers | `2Gi` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU` | Default CPU limit for workspace containers | `4` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY` | Default memory limit for workspace containers | `8Gi` |

**Example:**

```bash theme={"system"}
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU="500m"
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY="2Gi"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU="4"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY="8Gi"
```

#### Complete Kubernetes Container Configuration Example

Here's a complete example of a Kubernetes `container_config` YAML:

```yaml theme={"system"}
env:
  - name: USER_CODE_PATH
    value: /home/src/<path_to_folder>
  - name: ENV
    value: dev
  - name: TS_AUTHKEY
    value: <your_tailscale_auth_key>
  - name: TS_EXTRA_ARGS
    value: --accept-routes

resources:
  requests: # Minimum resources guaranteed
    memory: "2Gi" # 2 gigabytes of RAM
    cpu: "1" # 1 full CPU core
  limits: # Maximum resources allowed
    memory: "4Gi" # 4 gigabytes RAM ceiling
    cpu: "2" # Can use up to 2 CPU cores
```

**Resource Configuration Details:**

* **`requests`**: Minimum resources guaranteed to your workspace. Kubernetes will ensure these resources are available.
* **`limits`**: Maximum resources your workspace can use. Prevents resource overconsumption and ensures fair resource allocation across workspaces.

**Note**: You can include any valid Kubernetes container configuration in the `container_config` YAML, including:

* Container image (`image`)
* Environment variables (`env`)
* Resource requests and limits (`resources`)
* Volume mounts (`volumeMounts`)
* Security context (`securityContext`)
* And other Kubernetes container spec fields

#### Container Image Configuration

When creating or updating a workspace, you can specify the container image in the `container_config` YAML:

```yaml theme={"system"}
image: [image_repo]/mage-pro:20260112-1200
```

**Important behavior when updating workspaces:**

* **If `image` is specified**: The workspace will use the specified image.
* **If `image` is NOT specified**: The system will use the image from the main cluster pod configuration template (`pod_config`), **not** the current workspace's image.

This means that when updating a workspace without specifying an `image` in `container_config`, the workspace image may change to match the main cluster pod configuration, even if the workspace currently uses a different image. To preserve the current image, explicitly include it in your `container_config` when updating.

### Pod Scheduling Configuration

You can control which nodes your workspace pods are scheduled on using `nodeSelector`, `affinity`, and `tolerations`.

#### Node Selector

Use `nodeSelector` to schedule workspace pods on nodes with specific labels. This is the simplest way to target specific nodes.

**Example:**

```yaml theme={"system"}
nodeSelector:
  disktype: ssd
  instance-type: gpu
```

#### Affinity

Use `affinity` for more advanced scheduling rules, including node affinity, pod affinity, and pod anti-affinity.

**Node Affinity Example:**

```yaml theme={"system"}
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/os
          operator: In
          values:
          - linux
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
        - key: node-type
          operator: In
          values:
          - compute-optimized
```

**Pod Anti-Affinity Example (spread workspaces across nodes):**

```yaml theme={"system"}
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - workspace
        topologyKey: kubernetes.io/hostname
```

#### Tolerations

Use `tolerations` to allow workspace pods to be scheduled on nodes with taints.

**Example:**

```yaml theme={"system"}
tolerations:
- key: "workspace"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
- key: "gpu"
  operator: "Exists"
  effect: "NoSchedule"
```

#### Complete Example with Scheduling Configuration

Here's a complete example combining resource configuration with pod scheduling:

```yaml theme={"system"}
env:
  - name: USER_CODE_PATH
    value: /home/src/my-project
  - name: ENV
    value: production

resources:
  requests:
    memory: "4Gi"
    cpu: "2"
  limits:
    memory: "8Gi"
    cpu: "4"

nodeSelector:
  instance-type: gpu

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: zone
          operator: In
          values:
          - us-west-2a
          - us-west-2b

tolerations:
- key: "gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
```

**Note**: These pod-level scheduling configurations (`nodeSelector`, `affinity`, `tolerations`) are applied to the workspace pod, not the container. They control where Kubernetes schedules your workspace pods in the cluster.

## Share secrets across workspaces

To share secrets across different workspaces, you need to set the `SHARE_SECRETS_ACROSS_WORKSPACES` environment variable at the **cluster level** (not at the workspace level) and set the value to `1` to enable this feature. This will allow developers to share their secrets across different workspaces within a cluster.

**Note**: The `SHARE_SECRETS_ACROSS_WORKSPACES` environment variable must be set on the **main cluster container**, not on individual workspaces.

## Share secrets across workspaces

To share secrets across different workspaces developers need to create a custom environment variable
called `SHARE_SECRETS_ACROSS_WORKSPACES` then set the value to `1` to enable this feature. This will
allow developers to share their secrets across different workspaces within a cluster.

![Add key value pairs at the bottom of the add/edit env var page](https://mage-ai.github.io/assets/guides/workspaces/add-env-var-secrets-workspaces.png)

## Inherit environment variables

Configuring your Mage Pro workspaces to inherit the main cluster’s environment
variables is essential for efficiently configuring your workspace container.
The following options allow you to control how variables are shared within your environment.

To enable environment variable inheritance in workspaces, you must set the `INHERIT_ENV_VARS` environment variable at the **cluster level** (not at the workspace level):
To enable environment variable inheritance in workspaces, you must set:

```bash theme={"system"}
INHERIT_ENV_VARS=1
```

**Note**: The `INHERIT_ENV_VARS` environment variable must be set on the **main cluster container**, not on individual workspaces. This applies to **both Managed Cloud** and **Self-hosted Mage Pro**.

### Managed Cloud

The `USER_DEFINED_VARS` environment variable (set at the **cluster level**) lets you control which environment variables are inherited by workspaces.
This applies to **both Managed Cloud** and **Self-hosted Mage Pro**.

### Managed Cloud

The USER\_DEFINED\_VARS environment variable lets you control which environment variables are inherited by workspaces.
It accepts a comma-separated list of variable names, giving you flexibility to include only the configuration values you need across workspaces.

* If `USER_DEFINED_VARS` is **not set**: all environment variables will be inherited automatically.
* If `USER_DEFINED_VARS` is **set**: only the specified variables will be inherited.

**Example – inherit specific variables:**

```bash theme={"system"}
USER_DEFINED_VARS=VAR1,VAR2
```

**Example – inherit by pattern (wildcard/regex):**

```bash theme={"system"}
USER_DEFINED_VARS=DEV_ENV_*
```

This will inherit all environment variables matching the pattern (e.g., any starting with `DEV_ENV_*`).

### Self-hosted Mage Pro (Hybrid Cloud, Private Cloud, On-prem)

You **must** set `USER_DEFINED_VARS` at the **cluster level** to define which variables to inherit:
You **must** set `USER_DEFINED_VARS` to define which variables to inherit:

```bash theme={"system"}
USER_DEFINED_VARS=VAR1,VAR2
```

Wildcard/regex-style patterns are supported here as well.

### Auto-set ENV variable from workspace name

You can automatically set the `ENV` environment variable for each workspace based on its name using the `WORKSPACE_ENV_PATTERN` cluster-level environment variable. This eliminates the need to manually set `ENV` in each workspace's container configuration.

**Configuration:**
Set `WORKSPACE_ENV_PATTERN` at the **cluster level** (not at the workspace level):

```bash theme={"system"}
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
```

* `{workspace_name}`: Placeholder that will be replaced with the actual workspace name
* The pattern supports any string with the `{workspace_name}` placeholder

**How it works:**

* If `ENV` is not explicitly set in a workspace's container configuration, it will be automatically set using the pattern
* If `ENV` is explicitly set in the workspace's container configuration, that value takes precedence over the pattern
* The pattern is only applied when `WORKSPACE_ENV_PATTERN` is defined

**Examples:**

**Example 1: Simple prefix pattern**

```bash theme={"system"}
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
```

For a workspace named `my-workspace`, this sets `ENV=dev_my-workspace`.

**Example 2: Environment prefix with suffix**

```bash theme={"system"}
WORKSPACE_ENV_PATTERN=env-{workspace_name}-prod
```

For a workspace named `analytics`, this sets `ENV=env-analytics-prod`.

**Example 3: Just workspace name**

```bash theme={"system"}
WORKSPACE_ENV_PATTERN={workspace_name}
```

For a workspace named `production`, this sets `ENV=production`.

**Integration with environment-specific variable inheritance:**

When `ENV` is automatically set via `WORKSPACE_ENV_PATTERN`, it enables the environment-specific variable inheritance feature. You can then use:

```bash theme={"system"}
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
USER_DEFINED_VARS__ENV__dev_my-workspace=DEV_ENV_*
```

This automatically sets `ENV=dev_my-workspace` for a workspace named `my-workspace`, and that workspace will inherit all variables matching `DEV_ENV_*` from the cluster.

### Inherit variables by workspace environment

You can define inheritance rules per workspace environment using cluster-level environment variables with the `ENV` variable:

### Inherit variables by workspace environment (new)

You can define inheritance rules per workspace environment using the `ENV` variable:

```bash theme={"system"}
USER_DEFINED_VARS__ENV__{env_value}={pattern}
```

* `{env_value}`: value of the `ENV` variable in the workspace.
* `{pattern}`: wildcard/regex pattern for matching variables.

**Example:**

```bash theme={"system"}
USER_DEFINED_VARS__ENV__dev=DEV_ENV_*
```

Any workspace with `ENV=dev` inherits all variables matching `DEV_ENV_*`.

**Combined example with WORKSPACE\_ENV\_PATTERN:**

```bash theme={"system"}
# Automatically set ENV based on workspace name
WORKSPACE_ENV_PATTERN=dev_{workspace_name}

# Inherit variables based on the auto-set ENV value
USER_DEFINED_VARS__ENV__dev_workspace1=DEV_ENV_*
USER_DEFINED_VARS__ENV__dev_workspace2=PROD_ENV_*
```

In this example:

* A workspace named `workspace1` automatically gets `ENV=dev_workspace1` and inherits variables matching `DEV_ENV_*`
* A workspace named `workspace2` automatically gets `ENV=dev_workspace2` and inherits variables matching `PROD_ENV_*`

### Updating workspaces after environment variable changes

When you add or modify environment variables in your base cluster, existing workspaces will not automatically inherit these changes. To ensure your workspaces reflect the updated environment configuration, you must restart the affected workspaces.

To update a workspace:

1. Open the workspace settings by clicking into the workspace you want to update
2. Click the update button for your workspace

![Update workspace button](https://raw.githubusercontent.com/mage-ai/assets/main/guides/workspaces/update-workspace-button.png)

3. Click the dropdown arrow on the Kubernetes configuration section
4. Toggle the "update workspace settings" option to true
5. Click update

![Update workspace configuration](https://raw.githubusercontent.com/mage-ai/assets/main/guides/workspaces/workspace-update-configuration-2.png)

The container will take up to several minutes to update and restart with the new environment configuration.

**Note on container image during updates:**

When updating a workspace, if you don't explicitly specify the `image` field in your `container_config` YAML, the system will use the image from the main cluster pod configuration template, not the workspace's current image. This means:

* If your workspace currently uses a custom image (e.g., `my-custom-image:v1`) and you update the workspace without specifying `image` in `container_config`, the image may change to match the main cluster pod configuration (e.g., `mageai/mageai:latest`).
* To preserve your current workspace image during updates, explicitly include it in your `container_config`:

```yaml theme={"system"}
image: my-custom-image:v1
env:
  - name: USER_CODE_PATH
    value: /home/src/my-project
```

## Lifecycle Management Policies (Optional)

Mage Pro workspaces support various lifecycle management policies that control how your workspace behaves throughout its runtime. These policies include automated termination settings, pre-initialization scripts, and post-initialization procedures, giving you complete control over your workspace's behavior from startup to shutdown.

Lifecycle configuration is specified in the `lifecycle_config` field as a YAML object:

```yaml theme={"system"}
termination_policy:
  enable_auto_termination: true
  max_idle_seconds: 1800
pre_start_script_path: /path/to/script.sh
post_start:
  command: ["echo", "Workspace started"]
  hook_path: /path/to/hook.sh
```

### Termination Policy

The termination policy controls how your workspace shuts down when inactive.

**Configuration:**

* **`enable_auto_termination`** (boolean): Enable or disable automatic termination. Defaults to `false`.
* **`max_idle_seconds`** (integer): The number of seconds of inactivity before the workspace is automatically terminated. The idle time is measured in seconds, and you can set this duration based on your needs. For example:
  * `1800` seconds = 30 minutes
  * `3600` seconds = 1 hour
  * `7200` seconds = 2 hours

**Example:**

```yaml theme={"system"}
termination_policy:
  enable_auto_termination: true
  max_idle_seconds: 1800
```

## Lifecycle management policies (optional)

Mage Pro workspaces support various lifecycle management policies that control how your workspace behaves throughout its runtime. These policies include automated termination settings, pre-initialization scripts, and post-initialization procedures, giving you complete control over your workspace's behavior from startup to shutdown.

### Termination Policy

The termination policy controls how your workspace shuts down when inactive. You can enable auto-termination, which will automatically shut down your workspace after a specified period of inactivity. The idle time is measured in seconds, and you can set this duration based on your needs. For example, you might set it to 1800 seconds (30 minutes) or 3600 seconds (1 hour).

### Pre-Start Scripts

Pre-start scripts run before your workspace container is initialized. These scripts must include a `get_custom_configs` method that returns a dictionary of configurations. This is where you can set up environment variables, modify container configurations, and prepare any necessary resources. The script executes before any workspace services start, making it ideal for initial setup and configuration tasks.

**Configuration:**

* **`pre_start_script_path`** (string): Path to the pre-start script. Can be:
  * Absolute path: `/absolute/path/to/script.sh`
  * Relative path: `scripts/pre_start.sh` (relative to the root project directory)

**Example:**

```yaml theme={"system"}
pre_start_script_path: /home/src/scripts/setup_workspace.sh
```

### Post-Start Scripts

Post-start scripts execute after requirements installation and before the workspace server starts. These scripts are executed via the `mage run-post-start-hooks` command and run with graceful error handling—if a post-start hook fails, it logs a warning but does not prevent the container from starting. These scripts are typically used for starting services, performing health checks, or completing any additional setup tasks that depend on installed requirements but must run before the workspace server is fully started.

**Configuration:**

* **`post_start.command`** (string or array of strings): Command to execute. Can be specified as:
  * **JSON array of strings**: `["echo", "Workspace started"]` or `["python", "script.py", "--arg", "value"]`
  * **Shell command string**: `echo "Hello" && echo "World"` or `pip install playwright && playwright install --with-deps`
* **`post_start.hook_path`** (string): Path to a script file. Can be:
  * Absolute path: `/absolute/path/to/hook.sh`
  * Relative path: `scripts/post_start.sh` (relative to the root project directory)

**Note**: You can specify either `command` or `hook_path`, or both. If both are specified, the command will execute first, followed by the hook script. Post-start hooks execute after requirements installation but before the Mage server starts, ensuring they run early in the container startup process.

**Example 1: Post-start command as JSON array**

```yaml theme={"system"}
post_start:
  command: ["echo", "Workspace started"]
```

**Example 2: Post-start command as shell string**

```yaml theme={"system"}
post_start:
  command: echo "Hello" && echo "World"
```

**Example 3: Post-start command with multiple operations (shell string)**

```yaml theme={"system"}
post_start:
  command: pip install playwright && playwright install --with-deps
```

**Example 4: Post-start hook script**

```yaml theme={"system"}
post_start:
  hook_path: /home/src/scripts/post_start.sh
```

Where `post_start.sh` contains multiple commands:

```bash theme={"system"}
#!/bin/bash
echo "Starting service 1..."
service1 start

echo "Starting service 2..."
service2 start
```

**Example 5: Command and hook script (command runs first, then hook)**

```yaml theme={"system"}
post_start:
  command: ["echo", "Workspace started"]
  hook_path: /home/src/scripts/start_services.sh
```

**Important Notes:**

* Post-start hooks execute after requirements installation but before the Mage server starts
* If a post-start hook fails, it logs a warning but does not prevent the container from starting
* Shell command strings support operators like `&&`, `||`, etc.
* Hook scripts must be executable and located in a path accessible to the container

## Setting workspaces for self-hosted Mage Pro cluster

> [!NOTE]
>
> This section applies to **self-hosted Mage Pro clusters** only. These configurations are not applicable to Managed Cloud deployments.

### Workspace Configuration Environment Variables

These environment variables are set at the **cluster level** to configure default workspace behavior. They control how workspaces are created, what storage they use, and how they're exposed via ingress.

#### Ingress Configuration

* **`WORKSPACE_INGRESS_NAME`** (optional): Name of the Kubernetes ingress resource to use for workspaces. Defaults to the cluster UUID (`MAGE_CLUSTER_UUID`). If not set, workspaces will be accessible through the service directly rather than via ingress.

* **`WORKSPACE_INGRESS_TYPE`** (optional): Type of ingress controller to use. Supported values:

  * `nginx` (default): Use NGINX ingress controller
  * `alb`: Use AWS Application Load Balancer ingress
  * `istio`: Use Istio ingress gateway

  Default: `nginx`

* **`WORKSPACE_INGRESS_PATH_TYPE`** (optional): Path type for ingress paths. Supported values:

  * `ImplementationSpecific` (default): Path matching is implementation-specific
  * `Prefix`: Path must match as a prefix

  Default: `ImplementationSpecific`

#### Storage Configuration

You can configure workspace storage in one of two ways:

##### Option 1: Individual PVCs per Workspace (using WORKSPACE\_STORAGE\_CLASS)

* **`WORKSPACE_STORAGE_CLASS`** (optional): Kubernetes storage class name to use for workspace persistent volumes. When configured, each workspace will create its own PersistentVolumeClaim (PVC) using this storage class.

  Default: `efs-sc-us-west-2`

  **Use case**: When you want each workspace to have its own isolated storage volume.

##### Option 2: Shared PVC across Workspaces (using WORKSPACE\_PVC\_NAME + WORKSPACE\_PVC\_SUBPATH\_PREFIX)

* **`WORKSPACE_PVC_NAME`** (optional): Name of an existing PersistentVolumeClaim (PVC) to share across all workspaces. When set, all workspaces will use this shared PVC instead of creating individual PVCs.

* **`WORKSPACE_PVC_SUBPATH_PREFIX`** (optional): Prefix for subpaths when using a shared PVC. Each workspace will use a separate subfolder under this prefix to isolate its data. **Required** when `WORKSPACE_PVC_NAME` is set.

  **Example**: If `WORKSPACE_PVC_SUBPATH_PREFIX=workspaces`, a workspace named `my-workspace` will use the subpath `workspaces/my-workspace`.

  **Use case**: When you want to share storage across multiple workspaces and manage storage more efficiently.

**Important**: Configure either `WORKSPACE_STORAGE_CLASS` (for individual PVCs) or `WORKSPACE_PVC_NAME` + `WORKSPACE_PVC_SUBPATH_PREFIX` (for shared PVC), but not both. If both are configured, the shared PVC option takes precedence.

#### Pod Configuration

* **`INHERIT_POD_ANNOTATIONS`** (optional, boolean): If `true`, workspace pods will inherit annotations from the main cluster pod. This is useful for passing through annotations like Prometheus scraping labels, service mesh configuration, etc.

  Default: `false`

#### Default Resource Configuration

These environment variables set default resource requests and limits for Kubernetes workspace containers. They are applied when a workspace's `container_config` does not specify resources (or when specific fields like `requests.cpu` or `limits.memory` are omitted). User-specified values in `container_config` always take precedence.

| Variable | Purpose | Example Value |
| - | - | - |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU` | Default CPU request for workspace containers | `500m` |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY` | Default memory request for workspace containers | `2Gi` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU` | Default CPU limit for workspace containers | `4` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY` | Default memory limit for workspace containers | `8Gi` |

**Example:**

```bash theme={"system"}
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU="500m"
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY="2Gi"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU="4"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY="8Gi"
```

#### Default Workspace Config File

* **`WORKSPACE_DEFAULT_CONFIG_PATH`** (optional): Path to a YAML file that defines default workspace configuration applied when creating or updating workspaces. The file is merged with user-defined config based on the `enforce` flag. The file is read on the **control-plane** only; it is not mounted into workspace pods.

  **Helm chart (recommended):** When deploying **Mage Pro** with the [Mage Helm chart](https://github.com/mage-ai/helm-charts), configure the default via **`workspaceConfig`** in `values.yaml`. The chart creates a ConfigMap from your config, mounts it on the control-plane, and sets `WORKSPACE_DEFAULT_CONFIG_PATH` automatically. Do not set the env var manually when using the chart. See [Using Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) for details.

  **Supported fields** (top-level in the YAML file):

  | Field | Type | Description |
  | - | - | - |
  | `container_config` | YAML string or object | Default container configuration (image, resources, env, volumeMounts, etc.). |
  | `service_account_name` | string | Default Kubernetes service account for workspace pods. |
  | `lifecycle_config` | object | Default lifecycle configuration (termination policy, pre-start scripts, post-start hooks). |

  **Policy fields:**

  | Field | Description |
  | - | - |
  | `enabled` | If `false`, the policy is not applied. |
  | `enforce` | Controls who wins when both the default file and the user set the same key. See **Priority when merging** below. |
  | *(other keys)* | `container_config`, `service_account_name`, `lifecycle_config` (see table above). |

  **Priority when merging default file with user config**

  The default file is merged with the workspace config supplied by the user (UI or API). The **effective priority** (who wins for a given key) depends on `enforce`:

  | `enforce` | Effective priority | Result |
  | - | - | - |
  | **`true`** | **Default file is highest.** Default file → user config. | Values from the file override user-defined values. Use this to enforce cluster-wide workspace defaults (e.g. resources, service account) that users cannot override. |
  | **`false`** | **User is highest.** User config → default file (fills gaps only). | User config overrides the default file for the same keys. The file only fills in keys the user did not set. Use this to provide optional defaults while allowing per-workspace overrides. |

  **Top-level format**:

  ```yaml theme={"system"}
  enabled: true   # Set to false to disable this policy entirely
  enforce: false  # User wins on conflicts; file only fills in missing keys

  container_config:
    resources:
      limits:
        cpu: "2"
        memory: "4Gi"
      requests:
        cpu: "1"
        memory: "2Gi"
    nodeSelector:
      node.kubernetes.io/instance-type: "m5.large"
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "mage"
        effect: "NoSchedule"

  lifecycle_config:
    post_start:
      command: echo "Hello World"

  service_account_name: workspace-sa
  ```

  The config file must exist at the specified path on the **control-plane** pod. Use a ConfigMap or volume mount to make the file available.

#### Example Configuration

**Example 1: Individual PVCs per workspace**

```bash theme={"system"}
# Ingress configuration
export WORKSPACE_INGRESS_NAME=my-cluster-ingress
export WORKSPACE_INGRESS_TYPE=nginx
export WORKSPACE_INGRESS_PATH_TYPE=Prefix

# Storage configuration - each workspace gets its own PVC
export WORKSPACE_STORAGE_CLASS=fast-ssd

# Pod configuration
export INHERIT_POD_ANNOTATIONS=true
```

**Example 2: Shared PVC across workspaces**

```bash theme={"system"}
# Ingress configuration
export WORKSPACE_INGRESS_NAME=my-cluster-ingress
export WORKSPACE_INGRESS_TYPE=nginx
export WORKSPACE_INGRESS_PATH_TYPE=Prefix

# Storage configuration - all workspaces share one PVC with separate subfolders
export WORKSPACE_PVC_NAME=shared-workspace-storage
export WORKSPACE_PVC_SUBPATH_PREFIX=workspaces

# Pod configuration
export INHERIT_POD_ANNOTATIONS=true
```

**Note**: These environment variables are set on the **main cluster container**, not on individual workspaces. They control the default behavior when creating new workspaces. Individual workspaces can override some of these settings (like storage class) in their own configuration.

### Docker Workspace Configuration

Docker workspaces support additional configuration options specific to Docker containers:

#### Docker Workspace Environment Variables

These environment variables are set at the **cluster level** to configure default Docker workspace behavior:

* **`DOCKER_HOST`** (optional): Docker daemon host. Defaults to `unix://var/run/docker.sock`.
* **`DOCKER_TLS_VERIFY`** (optional): Enable TLS verification for Docker connection. Set to `1` to enable, `0` to disable. Defaults to `0`.
* **`DOCKER_CERT_PATH`** (optional): Path to TLS certificates for Docker connection.
* **`DOCKER_SWARM_MODE`** (optional): Enable or disable Docker Swarm mode. Set to `1` to enable, `0` to disable. Defaults to `0`.

**Note**: These environment variables are set on the **main cluster container**, not on individual workspaces. They control the default behavior when creating new Docker workspaces. Individual workspaces can override these settings in their own configuration.

#### Docker Container Configuration (YAML)

Similar to Kubernetes, Docker workspaces support a `container_config` YAML string:

**Example Docker Workspace Configuration:**

```yaml theme={"system"}
image: mage-pro:version
volumes:
  /host/path:
    bind: /home/src
    mode: rw
environment:
  TEST_ENV_VAR: "1"
public_url: mage.example.com/test1
```

**Note**: For Docker workspaces, the `container_config` field is the primary way to configure image, volumes, and environment. The system will parse this YAML string and extract:

* `image`: Docker image to use
* `volumes`: Volume mounts configuration
* `environment`: Environment variables
* `public_url`: Public URL for the workspace

### Post-Start Scripts

Post-start scripts execute after your workspace is fully initialized and running. You can specify either a direct command or provide a path to a script for more complex procedures. These scripts are typically used for starting services, performing health checks, or completing any additional setup tasks that require the workspace to be fully operational.

## Global Roles

Users can be assigned global roles that apply across all workspaces within your Mage Pro environment. Global role assignment is managed by project administrators through Users tab in left navigation menu located in the main project interface.

## Workspace-Specific Roles

Individual workspaces can have unique role settings. Workspace owners can:

* Assign user roles specific to their workspace
* Modify existing role assignments
* Remove user access to their workspace

In the workspace management cluster, authenticated users without a global or main-project role can use `/manage` to find workspaces where they have a workspace-specific Viewer, Editor, Admin, or Owner role. This directory access requires `REQUIRE_USER_AUTHENTICATION=true` and `REQUIRE_USER_PERMISSIONS=false` or unset. Users with a global or main-project role retain their existing workspace visibility; fine-grained permission mode retains its existing authorization rules.

For users with only workspace-specific access, `/manage` lists only their authorized workspaces, hides **Create new workspace**, and shows only **Workspaces** in the management sidebar. **Open** is available when the workspace is running. The **Manage** link does not grant additional permissions: workspace details, configuration, and lifecycle actions continue to require the existing management permissions.

If the user has no qualifying workspace assignments, the directory displays “No accessible workspaces.” Searching without a match displays “No records match the search.” A failed workspace request displays “Unable to load workspaces. Refresh to try again.” rather than an empty directory.

![Workspace roles UI](https://mage-ai.github.io/assets/guides/workspaces/workspaces-roles-ui.png)

Understanding and properly configuring your Mage Pro workspaces is essential for efficient data pipeline development and management. With the right combination of resource allocation, lifecycle policies, and user permissions, you can create a stable and secure environment tailored to your project's needs.


# Configure Git
Source: https://docs.mage.ai/guides/git/configure

This page will walk you through the process of authenticating with SSH or HTTPS and configuring Git for use with Mage Git Sync.

If you're wondering which Git resource to use in Mage, see [this](docs/getting-started/setting-up-git) page for a high-level overview of Git functionality. If you'd like to use Mage's GitHub integration (recommended for GitHub users), there's a different method of configuration. See [this](/production/data-sync/github) page for more information.

If you've decided on a method that uses HTTPS or SSH authentication (Git Sync or Git Integration), this page will walk you through configuring one of those methods.

![](https://user-images.githubusercontent.com/78053898/198753409-c7c41a8d-c137-4273-bb3b-76ecc81f8ee2.png)

## Configure Git settings

> [!NOTE]
>
> In Mage Pro, the Git settings page is the legacy configuration workflow. For current Git configuration and deployments, use the Deploy app at `/apps/deploy`.

> [!NOTE]
>
> By using the Git integration through Mage, Mage will automatically set your local repo as a safe directory (i.e. `git config --global --add safe.directory <local repo path>`)

To use the Mage Git integration, you will need to first configure Git settings. The following instructions are tailored to Github, but the Git integration can work with other Git providers as well, such as GitLab, BitBucket, or Azure DevOps:

> **SSH**
>
> > [!NOTE]
> >
> > **SSH keys with passphrase are not supported**
> >
> > The Git integration currently does not support repositories that require an SSH key with a passphrase.
> >
> > You can, however, use the Mage terminal and the git CLI to perform your git actions.
> > You'll still need to generate your SSH keys as normal. You can start a ssh-agent in the terminal so that
> > you don't have to enter your SSH key passphrase every time you perform an action.
> >
> > `eval $(ssh-agent -s) && ssh-add ~/.ssh/id_rsa`
>
> **1. Check .gitignore settings**
>
> Ensure that `.preferences.yaml` is added to your repo's `.gitignore` file so that your Git settings aren't pushed to Github.
>
> **2. Navigate to Git (legacy)**
>
> Open the account menu, select **Settings**, then select **Workspace** → **Git (legacy)** in the settings navigation.
>
> **3. Select SSH as the authentication type.**
>
> Change the authentication type to SSH. The interface will change for you to input a remote repo url.
>
> ![](https://mage-ai.github.io/assets/git-settings.png)
>
> **4. Set up your SSH key**
>
> If you haven't already, configure your SSH key— this will allow your instance to access GitHub. Our [instructions](/development/git/configure#generate-ssh-token#generate-ssh-token) can help if you're unfamiliar with the process.
>
> **5. Add Git details**
>
> * `Remote repo url`: Your Github repository's remote url. Use the SSH link instead of HTTPS link.
> * `Local directory path`: The directory in your machine that will be synced with the Github repository. If this field is omitted, it will default to the path determined by Python's `os.getcwd()`. You can check what your current working directory is in a Scratchpad block.
>
> **6. \[Optional] Enable One-way Git Sync**
>
> <Snippet />
>
> **7. Fill out the remaining fields.**
>
> Filling out these fields will allow Mage to recreate your git and SSH settings if they are deleted (i.e. if your container restarts):
>
> > [!NOTE]
> >
> > If you are using a AWS CodeCommit repository, you will need to set the username to the SSH Key ID of the SSH public key added to your AWS IAM user.
>
> * `Username`: Your git username. If you push to your remote repo, this username will show up next to your commits.
> * `Email`: Your git email. Used for the same purpose as your username.
> * `SSH public key in base64`, `SSH public key in base64`: Enter your SSH public and private key encoded in base64. You can follow the instructions in the field description if you want to Mage to save your SSH keys.
>
> **8. Save your Git settings**
>
> Click "Save repository settings" to complete the setup.

> **HTTPS/Token**
>
> **1. Check .gitignore settings**
>
> Ensure that `.preferences.yaml` is added to your repo's `.gitignore` file so that your Git settings aren't pushed to Github.
>
> **2. Navigate to Git (legacy)**
>
> Open the account menu, select **Settings**, then select **Workspace** → **Git (legacy)** in the settings navigation.
>
> **3. Create a Git access token and save it for later**
>
> * [GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
> * [GitLab](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token)
>
> **4. Add Git details**
>
> * `Remote repo url`: Your Github repository's remote url. Use the SSH link instead of HTTPS link.
> * `Local directory path`: The directory in your machine that will be synced with the Github repository. If this field is omitted, it will default to the path determined by Python's `os.getcwd()`. You can check what your current working directory is in a Scratchpad block.
>
> **5. \[Optional] Enable One-way Git Sync**
>
> <Snippet />
>
> **6. Fill out the remaining fields.**
>
> * `Username`: Your git username. If you push to your remote repo, this username will show up next to your commits.
> * `Email`: Your git email. Used for the same purpose as your username.
> * `Access token`: Enter the Git access token that you created in step 4.
>
> **7. Save your Git settings**
>
> Click "Save repository settings" to complete the setup.

### Generate SSH Token

If you're using a method that requires an SSH token in Mage, this section will walk you through generating one. If you're using a method that requires an HTTPS token, you can skip this section.

> **SSH**
>
> **1. Check for existing SSH key**
>
> Follow GitHub's guide [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/checking-for-existing-ssh-keys) to check for an existing SSH key.
>
> **2. Generate a new SSH key**
>
> If no key exists, generate a new key using this [guide](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
>
> You may skip this step if you already have a key.
>
> **3. Add SSH key to your Git provider**
>
> This step allows your Git provider to recognize your SSH key and communicate.
>
> * [GitHub Docs](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
> * [GitLab Docs](https://docs.gitlab.com/ee/user/ssh.html#add-an-ssh-key-to-your-gitlab-account)
>
> **4. Test your SSH connection**
>
> * [GitHub Docs](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/testing-your-ssh-connection)
> * [GitLab Docs](https://docs.gitlab.com/ee/user/ssh.html#verify-that-you-can-connect)
>
> **5. Add SSH to known hosts**
>
> The first time you connect using your Git provider, you'll get a message about verifying authenticity and adding the provider to your known hosts.
> For example:
>
> ```txt theme={"system"}
> The authenticity of host 'gitlab.example.com (35.231.145.151)' can't be established.
> ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added 'gitlab.example.com' (ECDSA) to the list of known hosts.
> ```
>
> This is normal and expected. Type `yes` and press enter to continue.


# ETL pipeline tutorial
Source: https://docs.mage.ai/guides/load-api-data

Build a data pipeline that loads restaurant data, transforms it, then exports it to a DuckDB database. 🦆

## Overview

In this tutorial, we'll create a data pipeline with the following steps:

**1. Load data from an online endpoint**

We'll use a Python block to load data from an online endpoint— a CSV file containing restaurant user transactions. We'll also run a test to make sure the data is clean.

**2. Clean column names and add a new column**

We'll use a Python block to transform the data by cleaning the columns and creating a new column, `number of meals`, that counts the number of meals for each user.

**3. Write the transformed data to a local DuckDB database**

Finally, we'll write the transformed data to a local DuckDB table.

If you haven't created a Mage project before, follow the [setup guide](/getting-started/setup) before starting this tutorial.

## Quickstart

Want to dive in? Simply run the following command to clone a pre-built repo:

```bash Mac/Linux theme={"system"}
git clone https://github.com/mage-ai/etl-demo mage-etl-demo \
&& cd mage-etl-demo \
&& cp dev.env .env && rm dev.env \
&& docker compose up
```

```bash Windows theme={"system"}
git clone https://github.com/mage-ai/etl-demo mage-etl-demo
cd mage-etl-demo
cp dev.env .env
rm dev.env
docker compose up
```

Then navigate to `https://localhost:6789` in your browser to see the pipeline in action!

## Tutorial

> **1️⃣ Create a pipeline**
>
> Open the pipelines list page (`/pipelines`). This is the default page when navigating to Mage in your web browser. If you have Mage running, just click [here](https://localhost:6789/pipelines).
>
> In the top left corner, select the button labeled *+ New*, then select the option labeled *Standard (batch)* to create a new pipeline.
>
> In the left vertical navigation, click the last link labeled *Pipeline settings*. Change the pipeline's name to `ETL demo`. Click the button labeled *Save pipeline settings*.
>
> ![Create new pipeline](https://mage-ai.github.io/assets/tutorials/etl/create-new-pipeline.gif)

> **2️⃣ Load data from an API**
>
> In the left vertical navigation, click the 1st link labeled *Edit pipeline*. Click the button labeled *+ Data loader* then hover over *Python*, and click the option labeled *API*.
>
> The *Block name* dialog will appear. Change the name to `load data`, then click *Save*. Paste the following code in the data loader block:
>
> ```python theme={"system"}
> import io
> import pandas as pd
> import requests
>
>
> @data_loader
> def load_data_from_api(*args, **kwargs):
>     url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
>     response = requests.get(url)
>     return pd.read_csv(io.StringIO(response.text), sep=',')
>
>
> @test
> def test_row_count(df, *args) -> None:
>     assert len(df.index) >= 1000, 'The data does not have enough rows.'
> ```
>
> Run the block by clicking the play icon button in the top right corner of the data loader block or one of the following keyboard shortcuts:
>
> * `⌘ + Enter`
> * `Control + Enter`
> * `Shift + Enter`
>
> After you run the block (⌘ + Enter), you'll see a sample of the data that was loaded.
>
> ![Load data](https://mage-ai.github.io/assets/tutorials/etl/load-data.gif)

> **3️⃣ Transform data**
>
> Let's transform the data in 2 ways:
>
> * Add a column that counts the number of meals for each user.
> * Clean the column names to properly store in a PostgreSQL database.
>
> Follow these steps:
>
> Click the button labeled *+ Transformer*, then hover over *Python*, and click the option labeled *Generic (no template)*. Paste the following code in the transformer block:
>
> ```python theme={"system"}
> def number_of_rows_per_key(df, key, column_name):
>     data = df.groupby(key)[key].agg(['count'])
>     data.columns = [column_name]
>     return data
>
>
> def clean_column(column_name):
>     return column_name.lower().replace(' ', '_')
>
>
> @transformer
> def transform(df, *args, **kwargs):
>     # Add number of meals for each user
>     df_new_column = number_of_rows_per_key(df, 'user ID', 'number of meals')
>     df = df.join(df_new_column, on='user ID')
>
>     # Clean column names
>     df.columns = [clean_column(col) for col in df.columns]
>
>     return df.iloc[:100]
>
>
> @test
> def test_number_of_columns(df, *args) -> None:
>     assert len(df.columns) >= 11, 'There needs to be at least 11 columns.'
> ```
>
> Run the block by clicking the play icon button in the top right corner of the data loader block or press 1 of the following keyboard shortcuts:
>
> * `⌘ + Enter`
> * `Control + Enter`
> * `Shift + Enter`
>
> After you run the block (⌘ + Enter), you'll see a sample of the data that was transformed.
>
> ![Transform data](https://mage-ai.github.io/assets/tutorials/etl/transform-data.gif)

> **4️⃣ Export data to DuckDB**
>
> Let's export the data to a local DuckDB database! Create a new `Data Exporter` SQL block and change the connection type to be `DuckDB`.
>
> Next, change the `Table name` to be `magic_duck.restaurant_user_transactions` and make sure the content of the block is:
>
> ```sql theme={"system"}
> SELECT * FROM {{ df_1 }}
> ```
>
> This will take our data from the previous block and export it to a local DuckDB table— `magic_duck.restaurant_user_transactions`! 🎉
>
> After you run the block (⌘ + Enter), you'll see a sample of the data that was exported.
>
> ![Export data](https://mage-ai.github.io/assets/tutorials/etl/export-data.gif)
>
> You should also see some logs indicating the export was completed, e.g.
>
> ```bash theme={"system"}
> DuckDB initialized
> └─ Opening connection to DuckDB...DONE
> Exporting data from upstream block floral_star to restaurant_user_transactions.
> Type: integer
> Type: integer
> Type: string
> Type: integer
> Type: string
> Type: integer
> Type: string
> Type: integer
> Type: integer
> Type: integer
> ```

**🎉 Congratulations!**

You've successfully built an end-to-end ETL pipeline that loaded data,
transformed it, and exported it to a database.

Now you're ready to raid the dungeons and find magical treasures with your new powers!

![Lightning mage](https://raw.githubusercontent.com/mage-ai/assets/af9fafa41d823fe8e710d54f86164bef4378f5c3/mascots/earth/casting.svg)

If you have more questions or ideas, get real-time help in our live support
[Slack channel](https://www.mage.ai/chat).


# Activity
Source: https://docs.mage.ai/guides/operations/activity

Review workspace audit logs in Mage Pro.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=activity)

The Activity page gives administrators a searchable audit log for the selected Mage Pro workspace. Use it to investigate pipeline changes, review access and security events, confirm integration activity, and export the current set of visible audit records.

## Access Activity

Open **Tools** in the left navigation and select **Activity** while you are inside a workspace. Activity is available to administrators only. Users without administrator access cannot view the page, and sharing an Activity URL does not grant access to anyone who is not already authorized.

Activity is scoped to the selected workspace. Switch workspaces to review audit events for a different environment.

## Choose an activity category

Select a tab to focus the audit log on a category of workspace activity:

| Category | What it contains |
| - | - |
| **Pipeline Activity** | Pipeline, block, run, trigger, schedule, tag, variable, template, and execution activity. |
| **Terminal** | Terminal command activity. |
| **AI/LLM** | AI Sidekick prompts and responses, and LLM tool calls and outcomes. |
| **Access & Security** | User, role, permission, secret, OAuth, token, and session activity. |
| **Integrations & API** | Data provider, integration source, integration destination, sync, webhook, event, and API-related activity. |
| **Infrastructure** | Workspace, project, deployment, Git, compute, kernel, job, and Spark activity. |

Changing tabs resets the current search, filters, and page so the list starts from the beginning of the selected category.

## Search and filter events

Use the search field to find events by action, target, source, event details, or actor. Actor search matches the displayed full name, username, and email address, so searching for a user's first and last name returns events shown under that actor.

Combine search with filters to narrow the audit log:

| Filter | Use it to |
| - | - |
| **Time** | Limit the list to recent activity, such as the last hour, day, or three days. |
| **Result** | Show only successful, failed, warning, or error events. |
| **Source** | Separate events created from the UI, API, Terminal, AI Sidekick, LLM, system processes, webhooks, or other sources. |
| **Actor** | Review events created by a specific user. |

Filters are combined together. For example, search for a pipeline name, choose **API** as the source, and choose **Failure** as the result to review failed API changes related to that pipeline.

## Review event details

The table shows each event's time, source, action, actor, target, and result. Select a row to open the detail panel and inspect the full details available for that record. AI/LLM events can include model, prompt, response, response time, tool, tool input and output, token usage, pipeline, routine run, session, turn, approval, outcome, and error fields when those values are available. JSON-shaped detail values are shown as formatted JSON so nested tool inputs, outputs, and errors are easier to review. Expand long JSON or chart-shaped values to inspect them in a scrollable modal.

Use **Download** in the detail panel to export the selected event details to CSV. The detail export includes the common event metadata and any category-specific fields, including AI/LLM details for AI Sidekick and LLM audit events.

If you change tabs, search, filter, or move to a page where the selected event is no longer visible, Mage clears the detail panel so it does not show a stale event.

## Page through results

Activity shows 30 events per page. The toolbar shows the current range and total number of matching events, such as **1-30 of 228 events**. Use the pagination controls below the table to move between pages.

Searches and filters apply across all matching audit events in the selected category, not only the currently visible page. When you change a tab, search, or filter, Activity returns to the first page of the new result set.

## Save or share a view

The selected category, search term, filters, and page are stored in the browser URL. You can refresh the page, use browser back and forward, or share the URL with another authorized administrator to reopen the same view.

## Export visible events

Use **Download page** to export the currently displayed page of events to CSV. The export reflects the active category, search, filters, and page, but it does not export every matching page at once.

## Scope and limitations

* Activity shows audit events for the selected workspace only.
* Activity requires administrator access.
* URL-backed views preserve the selected view, but they do not bypass permissions.
* **Download page** exports only the visible page of results.
* AI/LLM audit metadata is sanitized before it is written. Sensitive keys and common token or secret patterns are redacted, and oversized values are generally truncated. AI response text is the exception: after redaction, the full response is retained for audit export even when the UI shows only a preview, so prompt, response, tool-call, and approval exports should still be treated as sensitive operational data.
* If no events match the selected category, search, and filters, Activity shows an empty result state.


# Mage Guides
Source: https://docs.mage.ai/guides/overview

Mage guides for getting started, building pipelines, and more.

![](https://github.com/mage-ai/mage-ai/assets/59450879/7ef1cc84-0b15-4e6c-aa5c-74fdba7c2d5f)

This section of our documentation contains helpful guides and tutorials to getting started with Mage.

We welcome contributions from our community. If you would like to contribute your own guides or content, we encourage that as well! It's written entirely in Markdown and *very* user friendly.

At the bottom of every doc, there's a link to edit the page on GitHub. Click that link and you'll be taken to the Markdown file for that page. You can edit the file directly in your browser and submit a pull request.


# Importing pipelines
Source: https://docs.mage.ai/guides/pipelines/importing-pipelines

Import a pipeline zip file into your project.

> [!NOTE]
>
> Requires version `0.9.63` or greater. Mage loads and schedules pipelines from one project
> repository per workspace. If you previously registered multiple projects through the
> project platform, migrate each project to a separate [Mage Pro
> workspace](/guides/developer-ux/workspaces) before upgrading.

## Overview

If you have a zip file with a pipeline's `metadata.yaml` configuration file and block files,
you can import that pipeline into your project. This could be useful if you want to import
a pipeline from a different project.

The pipeline importer supports a single pipeline ZIP file. In Mage Pro, the Pipelines dashboard
also supports bulk import from one ZIP archive that contains multiple pipelines, including
project-style archives with `pipelines/[pipeline_uuid]/metadata.yaml` or bundled exported
pipeline folders.

> [!NOTE]
>
> Use pipeline import for pipeline metadata and pipeline block source files. In Mage Pro, if your
> ZIP contains a full Mage project repository, upload it from the same dashboard importer or
> click **Set up with Git** to configure Git in the current workspace. Mage Pro loads one Mage
> project repository per workspace.

## Supported block file types

1. Custom
2. Callback
3. Chart
4. Conditional
5. Data exporter
6. Data loader
7. Extension
8. Markdown
9. Scratchpad
10. Sensor
11. Transformer

## Unsupported block file types

Including these types of block files in your pipeline zip file may cause errors when trying
to import it.

1. dbt

## Downloading a pipeline

You can download a pipeline from your project as a zip file by going to the Pipelines
Dashboard (`/pipelines`), right-clicking one of the pipeline rows, and selecting
"Download (keep folder structure)" or "Download (without folder structure)". You can use
this pipeline zip file to import the pipeline into another Mage project.

When you choose "Download (without folder structure)", Mage flattens file paths unless
multiple included files have the same basename. In that case, Mage rejects the flattened
download so files cannot overwrite each other; choose "Download (keep folder structure)"
to preserve each source file in the zip file.

In Mage Pro, the same right-click menu includes **Copy URL**, which copies the pipeline's direct
URL to your clipboard.

Downloaded pipeline zip files include the pipeline `metadata.yaml` and source files for
regular, callback, conditional, chart, and extension blocks when those files are present
in the project.

## Required files and folder structure of imported pipeline zip file

At a minimum, the pipeline zip file (which has a `.zip` file extension) should contain a
root folder and a pipeline configuration file (i.e. `metadata.yaml`). There could be zero block
files if the pipeline has no blocks.

### Single folder structure

* Root folder
  * Pipeline config file (`metadata.yaml`)
  * Block file 1 (e.g. `block1.py`)
  * Block file 2 (e.g. `block2.py`)
  * Additional block files...

![Pipeline single folder structure](https://mage-ai.github.io/assets/pipelines/importing/pipeline-single-folder-structure.png)

### Subfolder structure

* Root folder
  * Pipelines folder (i.e. `pipelines`)
    * Pipeline folder (this is the pipeline UUID)
      * Pipeline config file (`metadata.yaml`)
  * Block type 1 folder (e.g. `data_loaders`, `transformers`, `custom`, etc.)
    * Block file 1 for block type 1 (e.g. `data_loader_block_file1.py`)
    * Block file 2 for block type 1 (e.g. `data_loader_block_file2.py`)
    * Additional block files for block type 1...
  * Block type 2 folder (e.g. `data_exporters`, `sensors`, `scratchpads`, etc.)
    * Block file 1 for block type 2 (e.g. `transformer_block_file1.py`)
    * Block file 2 for block type 2 (e.g. `transformer_block_file2.py`)
    * Additional block files for block type 2...
  * Additional block folders and files...

![Pipeline subfolder structure](https://mage-ai.github.io/assets/pipelines/importing/pipeline-subfolder-structure.png)

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=bulk-pipeline-import)

## Mage Pro bulk pipeline imports

Mage Pro users can import one ZIP archive that contains multiple pipeline definitions.

### Multi-pipeline archive structure

A single ZIP can include multiple pipeline folders in either a shared project-style layout or a
bundle of exported pipeline folders.

For a shared project-style layout, each pipeline has its own `metadata.yaml` file under
`pipelines/[pipeline_uuid]/`, while block source folders are shared at the archive root.

* Root folder
  * Pipelines folder (i.e. `pipelines`)
    * First pipeline folder
      * Pipeline config file (`metadata.yaml`)
    * Second pipeline folder
      * Pipeline config file (`metadata.yaml`)
  * Block type 1 folder (e.g. `data_loaders`, `transformers`, `custom`, etc.)
    * Block file 1 for block type 1 (e.g. `data_loader_block_file1.py`)
    * Block file 2 for block type 1 (e.g. `data_loader_block_file2.py`)
    * Additional block files for block type 1...
  * Block type 2 folder (e.g. `data_exporters`, `sensors`, `scratchpads`, etc.)
    * Block file 1 for block type 2 (e.g. `transformer_block_file1.py`)
    * Block file 2 for block type 2 (e.g. `transformer_block_file2.py`)
    * Additional block files for block type 2...
  * Additional block folders and files...

For a bundle of exported pipeline folders, keep each downloaded folder intact and zip those folders
together. Mage resolves each pipeline's block files from the folder that contains the pipeline
metadata, or from that folder's exported project root when the metadata is under
`pipelines/[pipeline_uuid]/`.

* Root folder
  * First exported pipeline folder
    * Pipelines folder (optional when the exported folder already uses project structure)
      * Pipeline folder
        * Pipeline config file (`metadata.yaml`)
    * Block type 1 folder (e.g. `data_loaders`, `transformers`, `custom`, etc.)
      * Block file 1 for block type 1 (e.g. `data_loader_block_file1.py`)
    * Additional block folders and files...
  * Second exported pipeline folder
    * Pipelines folder (optional)
      * Pipeline folder
        * Pipeline config file (`metadata.yaml`)
    * Block type 1 folder
      * Block file 1 for block type 1
    * Additional block folders and files...

You can also place folders that each contain one pipeline `metadata.yaml` and its local block
folders inside a top-level `pipelines/` folder. Do not add a root project `metadata.yaml` unless
you are uploading a full Mage project archive.

Multiple pipelines in the same import can reference the same block file path when the block source
is shared. Mage reports import conflicts at the pipeline level, such as a pipeline that already
exists in the workspace or a duplicate pipeline UUID in the selected archive. Shared block files
do not create separate block-level conflicts. If an incoming block file path already exists with
different contents, **Rename** keeps the existing block file and writes the incoming block to a
renamed path, while **Overwrite** intentionally replaces matching block files.

Upload exactly one ZIP at a time. A single ZIP can contain multiple pipelines or one full Mage
project.

### How to import pipeline archives in bulk

1. Go to the Pipelines Dashboard (`/pipelines`).
2. Click the **New pipeline** menu.
3. Select **Upload**.
4. Add one pipeline ZIP file by clicking the file drop area or by dragging and dropping it into it.
   The selected ZIP file must be 5 MB or less. Mage also rejects imports when the selected ZIP
   archive expands to more than 500 MB uncompressed or contains more than 10,000 ZIP entries in
   total.
5. Do not select multiple ZIP files. If you have multiple exported pipeline ZIPs, zip their
   exported folders together into one archive before uploading.
6. If you upload one valid pipeline ZIP with no conflicts or failures, Mage shows a create form.
   Optionally update the pipeline name, description, and comma-separated tags, then click
   **Create**.
7. Otherwise, review the validation results before importing. Mage lists the pipelines found,
   conflicts, failures, skipped pipelines, full-project archives, and validation errors such as
   invalid `metadata.yaml` YAML.
8. If Mage finds conflicts, choose a conflict strategy:
   * **Rename** keeps existing pipelines and imports conflicting pipelines with suffixed UUIDs.
   * **Overwrite** replaces matching pipeline and block files.
   * **Skip** leaves conflicting pipelines unchanged.
9. Mage rolls back a pipeline if writing any of its files fails. Other pipelines imported
   successfully from the same archive remain available.
10. Click **Import pipelines** to process every pipeline in the selected archive.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=project-archive-import)

## Full Mage project archives

A full Mage project archive can include the root project `metadata.yaml`, the `pipelines`
folder, `io_config.yaml`, `requirements.txt`, `dbt`, `plugins`, `utils`, and other repository
assets. The project files can be at the ZIP root or inside a single top-level project folder. The
Mage Pro dashboard importer detects this archive type and keeps it in the upload modal.

For a full project migration:

1. Create or use the Mage Pro workspace that should own the project repository.
2. Go to the Pipelines Dashboard (`/pipelines`), click the **New pipeline** menu, and select
   **Upload**.
3. Add the full project ZIP. Mage shows **Full Mage project archive detected** and summarizes the
   project assets found in the ZIP.
4. Review the project file summary. Mage classifies project files as new files, conflicting files
   with different contents, identical files that can be reused, or protected files that are not
   uploaded. Protected files include `.env`, `.env.*` variants such as `.env.staging` and
   `.env.test`, `.envrc`, workspace variable folders, Git metadata, and generated caches.
5. If Mage finds conflicting project files, choose a project conflict strategy:
   * **Overwrite conflicting files** replaces existing project files that have different contents.
   * **Skip conflicting files** uploads only new files and leaves existing paths unchanged.
   * **Cancel and review** stops without writing files so you can inspect the archive first.
6. Click **Upload project** to upload project code and configuration into the current workspace.
   Project ZIP uploads are applied atomically: if any project file write fails, Mage restores every
   file changed by that upload.
7. To use a Git repository instead of a ZIP upload, click **Set up with Git** to open
   Deployments in the current workspace.
8. Configure credentials, environment variables, secrets, OAuth connections, workspace-level
   compute settings, and deployment-specific settings separately.

Use a separate [Mage Pro workspace](/guides/developer-ux/workspaces) for each separate Mage
project repository.


# ActiveMQ
Source: https://docs.mage.ai/guides/streaming/destinations/activemq



## Basic config

```yaml theme={"system"}
connector_type: activemq
connection_host: 'localhost'
connection_port: 61613
queue_name: 'queue_name'
username: 'admin'
password: 'admin'
```


# Amazon S3
Source: https://docs.mage.ai/guides/streaming/destinations/amazon-s3



## Basic config

```yaml theme={"system"}
connector_type: amazon_s3
bucket: bucket
prefix: prefix
file_type: parquet
buffer_size_mb: 5
buffer_timeout_seconds: 300
```

* `bucket`: The S3 bucket you want to use to store the data.
* `prefix`: The S3 path prefix. The whole S3 path will be `s3://{bucket}/{prefix}`.
* `file_type`: `parquet`, `csv`, or `delta` (for Delta Lake tables)
* `file_type`: `parquet` or `csv`
* `buffer_size_mb` and `buffer_timeout_seconds`: Mage puts messages in a buffer before uploading to S3.
  You can configure the size and timeout of the buffer to control the file size and the delay.

You may see a `region` parameter in the default config template, it can be ignored. If you specify any valid AWS region names or null, you can still access to S3 buckets in other regions. However, if you do include the parameter value for `region`, it must be a valid region name for AWS or null so it won't error out.

## Configure time-based partition

```yaml theme={"system"}
date_partition_format: "%Y%m%dT%H"

```

If you want to store the S3 data in time-based partition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.

## Delta Lake configuration

**Note:** Delta Lake format (`file_type: delta`) is a Mage Pro only feature.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=s3-deltalake)

When using `file_type: delta`, you can write data to Delta Lake tables in Amazon S3.

```yaml theme={"system"}
connector_type: amazon_s3
bucket: my-bucket
prefix: my-prefix
file_type: delta
table_uri: s3://my-bucket/path/to/delta-table  # Optional: if not provided, constructed from bucket/prefix
mode: append  # 'append' or 'overwrite'
buffer_size_mb: 5
buffer_timeout_seconds: 300
aws_access_key_id: YOUR_ACCESS_KEY  # Optional: falls back to environment variables or IAM roles
aws_secret_access_key: YOUR_SECRET_KEY  # Optional: falls back to environment variables or IAM roles
aws_region: us-west-2  # Optional: defaults to us-west-2
```

* `table_uri`: (Optional) Full URI for the Delta Lake table (e.g., `s3://bucket/path/to/table`). If not provided, the URI will be automatically constructed from `bucket` and `prefix`.
* `mode`: Write mode for Delta Lake tables. Valid values:
  * `append`: Adds new rows to the existing table (default)
  * `overwrite`: Replaces all existing rows in the table
* `aws_access_key_id`: (Optional) AWS access key ID. If not provided, will fall back to the `AWS_ACCESS_KEY_ID` environment variable or the IAM role attached to the instance. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
* `aws_secret_access_key`: (Optional) AWS secret access key. If not provided, will fall back to the `AWS_SECRET_ACCESS_KEY` environment variable or the IAM role attached to the instance. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
* `aws_region`: (Optional) AWS region. If not provided, will fall back to the `AWS_REGION` environment variable or default to `us-west-2`.

**Note:** When using Delta Lake format, time-based partitioning (`date_partition_format`) is not applied as Delta Lake manages its own internal file structure.

If you want to store the S3 data in time-based paratition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.

## Authentication

Here are the options to authenticate with the AWS S3 bucket.

1. Add the following keys and values to your environment variables

* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`

2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate.
   You can grant the ECS task permissions to access other AWS services by attaching IAM policies
   to this ECS task execution role.

**Security Note:** Storing credentials in configuration files can be insecure. Prefer using environment variables or IAM roles when possible, especially for production deployments. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.


# Azure Data Lake
Source: https://docs.mage.ai/guides/streaming/destinations/azure_data_lake



## Basic config

```yaml theme={"system"}
connector_type: azure_data_lake
table_uri: abfs://container@storagename.dfs.core.windows.net/
account_name: account_name
access_key: access_key
mode: append
file_type: delta
```

* `table_uri`: Azure Data Lake container location, or local path, where the Delta Table will be written.
* `account_name`: Azure Storage Account name
* `access_key`: Azure Storage Access key
* `mode`: write mode. 'append' or 'overwrite' are accepted. default is 'append'
* `file_type`: File type to insert in Azure Data Lake. Only 'delta' is supported.


# BigQuery
Source: https://docs.mage.ai/guides/streaming/destinations/bigquery

Stream data to Google BigQuery using the Storage Write API, with optional dead-letter queue and metadata interpolation for CDC.

The BigQuery streaming destination uses the **Generic IO** sink. Writes go through the **BigQuery Storage Write API** (default stream, Arrow format) for better throughput and lower cost. The target table must already exist.

***

## Basic config

```yaml theme={"system"}
connector_type: bigquery
profile: default    # profile in io_config.yaml
config:
  table_id: project_id.dataset_id.table_id   # or dataset_id.table_id (uses profile project)
  if_exists: "append"
# dead_letter is optional and top-level (outside config)
# dead_letter:
#   destination_type: file   # "file" or "table"
#   enabled: true
#   table_suffix: "_dlq"
#   table_name: null
#   file_path: null
```

* **`profile`** selects which credentials and connection settings to use. It is the name of a **profile** (section) in your project’s `io_config.yaml`. The sink loads that profile via the same [IO config](/development/io_config_setup) used elsewhere in Mage (e.g. `ConfigFileLoader(repo_path/io_config.yaml, profile)`). Use the `default` profile unless you have multiple BigQuery connections (e.g. dev vs prod); then define separate profiles (e.g. `default`, `prod`) in `io_config.yaml` and set `profile: prod` in the sink.
* **`config`** is a key-value map passed to the BigQuery **export** call. It controls *where* and *how* data is written (e.g. `table_id`, `if_exists`, `database`, `overwrite_types`). It is merged with defaults (e.g. `if_exists: append`) and supports [metadata interpolation](#metadata-interpolation) so you can use values from each message (e.g. `table_id: "{schema}.{table}"`). **`config` is separate from credentials**: credentials come from the chosen **profile**; **config** only holds export options like table and column types.

## How it works

* **Storage Write API**: Streaming inserts use the BigQuery Storage Write API (not the older streaming insert API). Requires `google-cloud-bigquery-storage` and PyArrow. If the table does not exist, the pipeline will not create it; create the table beforehand.
* **Schema**: Only DataFrame columns that exist in the BigQuery table schema are written. Extra columns are dropped. Column types are cast to match the table (STRING, INT64, TIMESTAMP, RECORD, etc.).
* **CDC columns**: The sink automatically maps `_mage_created_at`, `_mage_updated_at`, and `_mage_deleted_at` to BigQuery `TIMESTAMP` type when present in the data.
* **Required (NOT NULL) fields**: If the table has REQUIRED columns and any row has nulls in those columns (including nested RECORD fields), the write fails and failed rows can be sent to a dead-letter queue (see below).

## Optional config

```yaml theme={"system"}
config:
  table_id: project_id.dataset_id.table_id
  if_exists: "append"
  database: project_id   # override project when table_id is dataset.table

  # Override column types (e.g. for _mage_* or custom columns)
  overwrite_types:
    _mage_created_at: TIMESTAMP
    my_column: STRING

# Dead-letter queue (top-level, outside config). Schema: DeadLetterConfig
dead_letter:
  destination_type: file     # "file" (default) or "table"
  enabled: true
  table_suffix: "_dlq"       # for destination_type=table: append to main table_id
  table_name: null           # optional: full table id (overrides table_suffix)
  file_path: null            # for destination_type=file: path for JSONL file
```

## Config parameters

| Parameter | Type | Description | Default |
| - | - | - | - |
| `table_id` | string | BigQuery table identifier. Use `project_id.dataset_id.table_id` or `dataset_id.table_id` (project from profile or `database`). Supports [metadata interpolation](#metadata-interpolation), e.g. `"{schema}.{table}"` for CDC. | *(required)* |
| `database` | string | Default project when `table_id` is two-part (`dataset_id.table_id`). Overrides the profile project if set. | profile project |
| `if_exists` | string | Behavior when the table already exists: `append` (add rows), `replace` (truncate/overwrite existing rows; preserves table schema), or `fail` (throw an error). | `append` |
| `overwrite_types` | object | Map of column name → BigQuery type (e.g. `STRING`, `INT64`, `TIMESTAMP`, `RECORD`). Used to cast columns before write. Useful for `_mage_*` or custom columns. | — |
| `unique_constraints` | list (or string) | List of column names for upsert/merge. When set with `unique_conflict_method`, exports as MERGE instead of append. Supports interpolation, e.g. `"{key_columns}"` from [PostgreSQL CDC](/guides/streaming/sources/postgres-cdc). | — |
| `unique_conflict_method` | string | For upsert: e.g. `UPDATE` to update existing rows on conflict. Used only when `unique_constraints` is set. | — |

The sink automatically sets `streaming_inserts: true` so writes use the BigQuery Storage Write API. The target table must already exist (the sink does not create it).

## Dead-letter queue

When a batch fails (e.g. required field has null, or Storage Write API returns row errors), failed rows can be written to a dead-letter destination. **Dead-letter is top-level** (sibling to `config`), not inside `config`. Schema: **DeadLetterConfig**:

| Field | Description |
| - | - |
| `destination_type` | Where to write failed rows: `file` (default) = JSONL file; `table` = BigQuery table (same project/dataset as main table). |
| `enabled` | If `true`, write failed rows to the DLQ (per `destination_type`). If `false`, do not write to DLQ. Default: `true`. |
| `table_suffix` | When `destination_type` is `table`: suffix applied to the main `table_id` to form the DLQ table (e.g. `_dlq` → `project.dataset.table_dlq`). Ignored if `table_name` is set. Default: `_dlq`. |
| `table_name` | When `destination_type` is `table`: full table identifier for the DLQ table (e.g. `project_id.dataset_id.dlq_table_id`). Overrides `table_suffix` when set. |
| `file_path` | When `destination_type` is `file`: path for the JSONL file. Default: under the pipeline variables dir (e.g. `<variables_dir>/streaming_dlq.jsonl`). |

When `dead_letter` is omitted entirely, failed rows are written to the default file path (same as `destination_type: file` with default `file_path`).

**DLQ record format** (each failed row becomes one record):

* `original_record`: the failed row as a JSON object (file) or JSON string column (table).
* `failure_reason`: error message.
* `failure_timestamp`: ISO timestamp (UTC).
* `connector_type`: e.g. `bigquery`.
* `row_errors`: optional list of per-row error details (index, code, message) when the API provides them.

When writing to a **table**, the DLQ table must have columns compatible with these fields (e.g. `original_record` STRING/JSON, `failure_reason` STRING, `failure_timestamp` STRING/TIMESTAMP, `connector_type` STRING, `row_errors` STRING/JSON). When writing to a **file**, each line is one JSON object with the fields above.

If failed rows are successfully written to the dead-letter destination, the pipeline continues. If the DLQ write fails or is disabled, the pipeline raises the error so you can alert or retry.

## Metadata interpolation

You can use message metadata in the config (e.g. schema/table per source). Keys in `config` are interpolated with `{metadata_key}`. Example: `table_id: "{schema}.{table}"` using `metadata.schema` and `metadata.table` from each message.

This pattern works particularly well with **CDC** sources such as [PostgreSQL CDC](/guides/streaming/sources/postgres-cdc), where each event carries `schema`, `table`, and primary key metadata. For example, you can route changes from many source tables into matching BigQuery tables by setting:

```yaml theme={"system"}
config:
  table_id: "{schema}.{table}"
```

profile: default    # profile in the io\_config.yaml
config:
table\_id: table\_id
if\_exists: "append"

````

## Optional fields

```yaml
database: database  # Override the database
````


# ClickHouse
Source: https://docs.mage.ai/guides/streaming/destinations/clickhouse



## Basic config

```yaml theme={"system"}
connector_type: clickhouse
profile: default    # profile in the io_config.yaml
config:
  table_name: table_name
```

Enter the DB connection credentials in [`io_config.yaml`](/development/io_config) file and specify the profile in the config.

## Optional fields

```yaml theme={"system"}
database: database                  # Override the database
create_table_statement: true/false  # Whether to run create table statement
```


# DuckDB
Source: https://docs.mage.ai/guides/streaming/destinations/duckdb



## Basic config

```yaml theme={"system"}
connector_type: duckdb
profile: default    # profile in the io_config.yaml
config:
  table_name: table_name
```

## Optional fields

```yaml theme={"system"}
schema_name: schema_name    # Override the schema name. Default schema is "main"
```


# Dummy
Source: https://docs.mage.ai/guides/streaming/destinations/dummy



Dummy sink will print the message optionally and discard the message. This dummy sink will be useful when users want to trigger other pipelines or 3rd party services using the ingested data in transformer.

## Config

```yaml theme={"system"}
connector_type: dummy
print_msg: true
```


# Elasticsearch
Source: https://docs.mage.ai/guides/streaming/destinations/elasticsearch



## Basic config

```yaml theme={"system"}
connector_type: elasticsearch
host: http://localhost
index_name: elastic_index
# When using Data streams this must be set to create as they are Append-only indexes
# _op_type: None / index, create or update
# _source: None (Default '_doc') / Specify the source field
# Whether to have update for the documents, need to mention the unique key field in the document as _id
# _id: _key
# # Whether to verify SSL certificates to authenticate.
# verify_certs: true
# ca_cert: "certificate.cert"
```

## Authentication

```yaml theme={"system"}
verify_certs: true
api_key: api_key_for_elastic_search
ca_cert: "certificate.cert"
```

* **verify\_certs**: Whether to verify SSL certificates to authenticate.


# Google Cloud PubSub
Source: https://docs.mage.ai/guides/streaming/destinations/google-cloud-pubsub



## Basic config

```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
pubsub_emulator_host: null  # "host.docker.internal:8085"
path_to_credentials_json_file: "./google_credentials.json"
```

The `pubsub_emulator_host` parameter is optional, which could be used when you need to test
the Google Cloud PubSub service locally through an emulator (see more details at the bottom).

## Data format

The Google Cloud PubSub data exporter scalar values

```
message = <Any>
```

or dictionaries with a specific format as messages:

```
message = {
    'data': <Any>,
    'metadata': <Optional[Dict]>
}
```

If the data contains no dictionary it will be written to the pubsub as value.
Otherwise, every dictionary message contains at least some data in a `data` field.
The following messages are valid:

```python theme={"system"}
message = {'data': {'bees': 23, 'ants': 30}}
```

The following message shows the configuration of elements data:

```python theme={"system"}
message = {
    'data': {
        'bees': 23,
        'ants': 30,
    }
}
```

If not all elements are configured, default values are assumed.

* The project\_id and topic\_id has to be configured in the yaml file.

```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
```

## Set up Google Cloud Pubsub emulator

To test out Google Cloud PubSub source locally, follow the instructions in
[Testing apps locally with the emulator](https://cloud.google.com/pubsub/docs/emulator)
to set up the local Google Cloud PubSub emulator.

## Create a `project_id` for testing

Use the command listed under the `Starting the emulator` section to create a `project_id`, e.g.,

```
gcloud beta emulators pubsub start --project=test-project-id

$(gcloud beta emulators pubsub env-init)
```

## Create a topic and a subscription

Use the `gcloud pubsub topics create` command to create a topic:

```
gcloud pubsub topics create test-topic-id
```

After you create a topic, you can subscribe or publish to it. Use the
`gcloud pubsub subscriptions create` command to create a subscription.

```
gcloud pubsub subscriptions create test-subscription-id-0 --topic test-topic-id
```

Only messages published to the topic after the subscription is created
are available to subscriber applications.


# Google Cloud Storage
Source: https://docs.mage.ai/guides/streaming/destinations/google-cloud-storage



## Basic config

```yaml theme={"system"}
connector_type: google_cloud_storage
bucket: bucket
prefix: prefix
file_type: parquet
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
```

* `bucket`: The Google Cloud Storage bucket you want to use to store the data.
* `prefix`: The Google Cloud Storage path prefix. The whole GCS path will be `gs://{bucket}/{prefix}`.
* `file_type`: `parquet`, `csv`, or `delta` (for Delta Lake tables)
* `buffer_size_mb` and `buffer_timeout_seconds`: Mage puts messages in a buffer before uploading to Google Cloud Storage.
  You can configure the size and timeout of the buffer to control the file size and the delay.
* `path_to_credentials_json_file`: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
* `prefix`: The Google Cloud Storage path prefix. The whole S3 path will be `s3://{bucket}/{prefix}`.
* `file_type`: `parquet` or `csv`
* `buffer_size_mb` and `buffer_timeout_seconds`: Mage puts messages in a buffer before uploading to Google Cloud Storage.
  You can configure the size and timeout of the buffer to control the file size and the delay.
* `path_to_credentials_json_file`: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate.

## Configure time-based partition

```yaml theme={"system"}
date_partition_format: "%Y%m%dT%H"
```

If you want to store the Google Cloud Storage data in time-based partition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.

## Delta Lake configuration

**Note:** Delta Lake format (`file_type: delta`) is a Mage Pro only feature.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=gcs-deltalake)

When using `file_type: delta`, you can write data to Delta Lake tables in Google Cloud Storage.

```yaml theme={"system"}
connector_type: google_cloud_storage
bucket: my-bucket
prefix: my-prefix
file_type: delta
table_uri: gs://my-bucket/path/to/delta-table  # Optional: if not provided, constructed from bucket/prefix
mode: append  # 'append' or 'overwrite'
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
```

* `table_uri`: (Optional) Full URI for the Delta Lake table (e.g., `gs://bucket/path/to/table`). If not provided, the URI will be automatically constructed from `bucket` and `prefix`.
* `mode`: Write mode for Delta Lake tables. Valid values:
  * `append`: Adds new rows to the existing table (default)
  * `overwrite`: Replaces all existing rows in the table

**Note:** When using Delta Lake format, time-based partitioning (`date_partition_format`) is not applied as Delta Lake manages its own internal file structure.

**Security Note:** Storing credentials in configuration files can be insecure. Prefer using environment variables (e.g., `GOOGLE_APPLICATION_CREDENTIALS`) or secret variables when possible, especially for production deployments. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
If you want to store the Google Cloud Storage data in time-based paratition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.


# InfluxDB
Source: https://docs.mage.ai/guides/streaming/destinations/influxdb



## InfluxDB instance

InfluxDB can be [run locally](https://docs.influxdata.com/influxdb/v2.7/install/) or in the cloud.

## Basic config

Basic configuration includes the server connection details:

```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
```

## Data format

The InfluxDB data exporter accepts only dictionaries as messages following a specific format:

```
message = {
    'data': <Any>
    'metadata': <Optional[Dict]>
}
```

Every message contains at least some data in a `data` field. The following messages are valid:

```python theme={"system"}
message_1 = {'data': {'bees': 23, 'ants': 30}}
message_2 = {'data': 23}  # implicit for {'_value': 23}
message_3 = {'data': [23, 30]}  # implicit for {'_value': 23} and {'_value': 30}
```

InfluxDB supports [data elements](https://docs.influxdata.com/influxdb/v2.7/reference/key-concepts/data-elements/) for structuring your data.
The influxdb data exporter can create these elements if the messages received contain a metadata dictionary:

```python theme={"system"}
message['metadata'] = {
    'measurement': str
    'time': int
    'tags': Dict
}
```

The following message shows the configuration of all possible elements data and metadata:

```python theme={"system"}
message = {
    "data": {'bees': 23},
    "metadata": {
        'measurement': 'census',
        'time': 1693472675783,  # timestamp with ms precision
        'tags': {'location': 'klamath', 'scientist': 'anderson'}
    }
}
```

If not all elements are configured, default values are assumed.

* The default measurement name is `default` and can be configured in the yaml file.

```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
measurement: "default"
```

* The default time timestamp is the current time of execution.
* The default tags are none.

The `data` part of the message is taken for the fields elements.
If the data is a scalar value or a list type it will be written to the database with the field name `_value`.

## Advanced config

The influxdb data exporter writes data to the InfluxDB server in batches, if either the batch size or the timeout are reached. Both values can be changed if necessary in the config file:

```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
batch_size: 100
timeout_ms: 500
```


# Kafka
Source: https://docs.mage.ai/guides/streaming/destinations/kafka



## Basic config

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
api_version: "0.10.2"
batch_size: 100
timeout_ms: 500
```

## SSL authentication

```yaml theme={"system"}
security_protocol: "SSL"
ssl_config:
  cafile: "CARoot.pem"
  certfile: "certificate.pem"
  keyfile: "key.pem"
  password: password
  check_hostname: true
```

## SASL authentication

SASL\_SSL with PLAIN or SCRAM:

```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
  mechanism: "PLAIN"  # or "SCRAM-SHA-256", "SCRAM-SHA-512"
  username: username
  password: password
ssl_config:
  cafile: "pemfilename.pem"  # Optional but recommended
```

SASL\_SSL with OAUTHBEARER (OAuth 2.0):

```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
  mechanism: "OAUTHBEARER"
  oauth_token_url: "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token"
  oauth_client_id: "kafka-client"
  oauth_client_secret: "your-client-secret"
  oauth_scope: "kafka"  # Optional
ssl_config:
  cafile: "pemfilename.pem"
  check_hostname: true
```

> [!NOTE]
>
> **OAUTHBEARER Authentication:**
>
> * The `oauth_token_url`, `oauth_client_id`, and `oauth_client_secret` are **required** when using `OAUTHBEARER` mechanism.
> * The `oauth_scope` parameter is optional.
> * OAuth tokens are automatically cached and refreshed before expiry (60 seconds before expiration by default).
> * It's recommended to use `SASL_SSL` (not `SASL_PLAINTEXT`) with OAUTHBEARER for secure token transmission.
> * Compatible with OAuth 2.0 providers like Keycloak, Okta, and other identity servers that support client credentials flow.

```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
  mechanism: "PLAIN"
  username: username
  password: password
```

## Data format

The Kafka data exporter scalar values

```
message = <Any>
```

or dictionaries with a specific format as messages:

```
message = {
    'data': <Any>,
    'metadata': <Optional[Dict]>
}
```

If the data contains no dictionary it will be written to the database as value with the current time value and no key.
Otherwise, every dictionary message contains at least some data in a `data` field.
The following messages are valid:

```python theme={"system"}
message = {'data': {'bees': 23, 'ants': 30}}
```

Kafka supports [structuring and partitioning](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html#kafka.KafkaProducer.send) your data.
The Kafka data exporter can create these elements if the messages received contain a metadata dictionary:

```python theme={"system"}
message['metadata'] = {
    'dest_topic': str
    'key': str
    'time': int
}
```

The following message shows the configuration of all possible elements data and metadata:

```python theme={"system"}
message = {
    'data': {
        'bees': 23,
        'ants': 30,
    },
    'metadata': {
        'dest_topic': 'census',
        'key': 'key',
        'time': 1693396630163,  # timestamp with ms precision
    }
}
```

If not all elements are configured, default values are assumed.

* The default topic has to be configured in the yaml file.

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
```

* The default time timestamp is the current time of execution.

* In case you are using newer versions of Kafka brokers, you should consider using corresponding Kafka api\_version.


# Kinesis
Source: https://docs.mage.ai/guides/streaming/destinations/kinesis



## Config

```yaml theme={"system"}
connector_type: kinesis
stream_name: stream_name
partition_key: key
```

* **partition\_key**: A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to.

## Authentication

Here are the options to authenticate with the AWS Kinesis.

1. Add the following keys and values to your environment variables

* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`

2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate.
   You can grant the ECS task permissions to access other AWS services by attaching IAM policies
   to this ECS task execution role.

If you get the `botocore.exceptions.NoRegionError` error, try setting the `AWS_DEFAULT_REGION`
environment variable.


# MongoDB
Source: https://docs.mage.ai/guides/streaming/destinations/mongodb



## Basic config

```yaml theme={"system"}
connection_string: "mongodb://mongodb_container_name:27017/"
database_name: database_name
collection_name: collection_name
```

* `connection_string`: URI defines connection between application and MongoDB instance.
* `database_name`: The database where you want to store the data.
* `collection_name`: The collection where you want to group the documents.

## Optional fields

```yaml theme={"system"}
unique_constraints: ['col1', 'col2']    # Upsert documents based on unique constraints
```


# Microsoft SQL Server
Source: https://docs.mage.ai/guides/streaming/destinations/mssql



## Basic config

```yaml theme={"system"}
connector_type: mssql
profile: default    # profile in the io_config.yaml
config:
  table_name: table_name
```

## Optional fields

```yaml theme={"system"}
schema_name: schema_name    # Override the schema name. Default schema is "dbo"
```


# MySQL
Source: https://docs.mage.ai/guides/streaming/destinations/mysql



## Basic config

```yaml theme={"system"}
connector_type: mysql
profile: default    # profile in the io_config.yaml
config:
  table_name: table_name
```


# Opensearch
Source: https://docs.mage.ai/guides/streaming/destinations/opensearch



## Basic config

```yaml theme={"system"}
connector_type: opensearch
host: https://[cluster_name].[region].es.amazonaws.com
index_name: test_index
```

## Authentication

```yaml theme={"system"}
verify_certs: true
http_auth: "@awsauth"
```

* **verify\_certs**: Whether to verify SSL certificates to authenticate.
* **http\_auth**: Authentication setting
  1. "@awsauth": Authenticate with AWS Signature Version 4. Need to provide
     AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY in environment variables.
  2. "username:password": Authenticate with username and password.


# Oracle Database
Source: https://docs.mage.ai/guides/streaming/destinations/oracledb



## Basic config

```yaml theme={"system"}
connector_type: oracledb
user: your_user_name
password: your_user_password
host: host_name
port: port_number
table: exported_table_name
service_name: exported_service_name
```

* `user`: The user account in Oracle Database to export data with.
* `password`: password of the user account.
* `host`: Oracle Database host name.
* `port`: Oracle Database port number.
* `table`: Table name to export.
* `service_name`: Oracle Database service name.

## Optional fields

```yaml theme={"system"}
mode: mode_to_export
```

* `mode`: Export mode. Available values: `thin`, `thick`.


# Postgres
Source: https://docs.mage.ai/guides/streaming/destinations/postgres



## Basic config

```yaml theme={"system"}
connector_type: postgres
database: database
host: host
password: password
port: 5432
schema: schema
username: username
table: table
```

## Upsert record

To upsert records to Postgres, you can define `unique_conflict_method` and `unique_constraints` params
in the config.

```yaml theme={"system"}
unique_conflict_method: UPDATE
unique_constraints:
- col1
- col2
allow_reserved_words: False
auto_clean_name: True
case_sensitive: False
```


# RabbitMQ
Source: https://docs.mage.ai/guides/streaming/destinations/rabbitmq



## Basic config

```yaml theme={"system"}
connector_type: rabbitmq
connection_host: 'localhost'
connection_port: 5672
queue_name: 'queue_name'
username: 'guest'
password: 'guest'
amqp_url_virtual_host: '%2f'
url_protocol: 'amqp'
```


# Redshift
Source: https://docs.mage.ai/guides/streaming/destinations/redshift



## Basic config

```yaml theme={"system"}
connector_type: redshift
profile: default            # profile in the io_config.yaml
config:
  table_name: table_name
  schema_name: schema_name  # Override the schema name. Default schema is "public"
```

## Alternative

Another option to stream data into Redshift is to stream data to Kinesis.
You can configure streaming ingestion for your Amazon Redshift cluster and create a materialized view using SQL statements.

You can follow [the guide from AWS](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion-getting-started.html) to set it up.

Or you can follow the steps below to set up the materialized view.

### 1. Create and attach IAM role to Redshift cluster

Create an IAM Role which has the policies `AmazonKinesisReadOnlyAccess` and `AmazonRedshiftFullAccess`.
Go to the Redshift cluster's `Properties` tab. In the `Cluster permissions` section, attach the IAM role to the Redshift cluster.

### 2. Create an external schema from Kinesis

Go to Redshift cluster's query editor, execute the command to create the schema:

```sql theme={"system"}
CREATE EXTERNAL SCHEMA redshift_stream_test
FROM KINESIS
IAM_ROLE <kinesis-redshift-iam-role-arn>;
```

### 3. Create a materialized view which loads data from Kinesis stream

Execute the command in query editor to create the materialized view from the kinesis stream:

```sql theme={"system"}
CREATE MATERIALIZED VIEW mage_kinesis_redshift_stream
AUTO REFRESH YES
AS
SELECT
    *,
    json_parse(kinesis_data) as payload
FROM redshift_stream_test."kinesis-stream-name"
```

You can change the materialized view name to the name you want to use.

### 4. Query data in the materialized view.

Execute the command to query the data:

```sql theme={"system"}
SELECT * FROM mage_kinesis_redshift_stream;
```


# Snowflake
Source: https://docs.mage.ai/guides/streaming/destinations/snowflake



## Basic config

```yaml theme={"system"}
connector_type: snowflake
profile: default    # profile in the io_config.yaml
config:
  table_name: table_name
```

## Optional fields

```yaml theme={"system"}
database: database  # Override the database
schema: schema       # Override the schema
```


# Trino
Source: https://docs.mage.ai/guides/streaming/destinations/trino



## Basic config

```yaml theme={"system"}
connector_type: trino
profile: default    # profile in the io_config.yaml
config:
  table_name: table_name
```

## Optional fields

```yaml theme={"system"}
schema_name: schema_name    # Override the schema name.
```


# Overview
Source: https://docs.mage.ai/guides/streaming/introduction



Streaming pipelines in Mage are designed to process data in real-time as it arrives, rather than in batches at scheduled intervals. This feature is particularly beneficial for use cases where immediate data processing is crucial, such as real-time analytics, monitoring, and decision-making systems.

One of the key benefits of streaming pipelines in Mage is the reduction in latency from data ingestion to actionable insights. For example, a technology solutions company was able to reduce the latency of transforming stream data and writing to MongoDB from 1.5 seconds to 200ms, even for larger size messages, by replacing Spark Structured Streaming with Mage's streaming pipelines. This represents a significant improvement in performance, enabling faster response times and more timely data-driven actions.

Another benefit is the simplification of the data pipeline architecture. Streaming pipelines in Mage allow for a more straightforward setup compared to other tools that may require complex configurations and tuning for performance. This ease of use can lead to increased developer productivity and a lower barrier to entry for teams to implement real-time data processing solutions

## Case study

The use case where streaming pipelines in Mage outperform Spark Structured Streaming involves processing streaming data with high efficiency and low latency. Specifically, the example provided involves a company that needed to transform stream data and write it to MongoDB.

Using Spark Structured Streaming on Databricks for this task, the company experienced an average latency of 1.2 to 1.9 seconds, and for larger size messages, the latency could go up to 3 to 4 seconds. This level of latency was not optimal for their requirements, which necessitated a more efficient solution.

Upon switching to Mage's streaming pipelines, they were able to significantly reduce the latency of their data processing and writing operations to MongoDB. They achieved a latency of less than 200 milliseconds, even for larger size messages. This represents a substantial improvement over their previous setup with Spark Structured Streaming, making Mage's streaming pipelines a better choice for their specific use case.

## Performance difference

Spark Streaming processes data in micro-batches, which inherently introduces a delay as data is collected over a period before being processed. This micro-batching approach, while effective for many use cases, is not truly real-time.

Mage’s performance advantage over Spark Streaming in real-time data processing scenarios include focus on real-time streaming (as opposed to micro-batching), simpler configuration and setup process, and reduced need for performance tuning. It’s designed with streaming pipelines that can handle data in real-time, reducing latency significantly for applications that require immediate data processing and insights.

## Benefits

The key benefits of using Mage's streaming pipelines in this context include:

1. **Reduced latency**
   1. Mage’s streaming pipelines provided a drastic reduction in latency, from seconds to milliseconds, which is crucial for real-time data processing and analytics applications.
2. **Simplicity and efficiency**
   1. Mage’s approach to streaming pipelines allows for a more straightforward and efficient data processing workflow, reducing the complexity and overhead associated with managing Spark Structured Streaming jobs.
3. **Scalability and flexibility**
   1. Mage’s streaming pipelines are designed to be scalable and flexible, accommodating varying data volumes and processing requirements without significant adjustments to the infrastructure.

This use case demonstrates the effectiveness of Mage’s streaming pipelines in scenarios where low latency and high efficiency are critical, making it a superior alternative to Spark Structured Streaming for specific real-time data processing tasks.

***

## Introduction

Each streaming pipeline has three components:

* Source: The data stream source to consume message from.
* Transformer: Write python code to transform the message before writing to destination.
* Sink (destination): The data stream destination to write message to.

Check out this [tutorial](/guides/streaming-pipeline#create-a-new-pipeline) to set up an example streaming pipeline.

> [!NOTE]
>
> In version `0.8.80` or greater.

You can build a streaming pipeline with one source, multiple transformers, and multiple sinks. Here is an example streaming
pipeline:
![Complex streaming pipeline](https://media.graphassets.com/4ZWHyLpgTESu0rB1sXnP)

## Supported sources

* [Amazon SQS](/streaming/sources/amazon-sqs)
* [Azure Event Hub](/streaming/sources/azure-event-hub)
* [Google Cloud PubSub](/streaming/sources/google-cloud-pubsub)
* [Kafka](/streaming/sources/kafka)
* [Kinesis](/streaming/sources/kinesis)
* [RabbitMQ](/guides/streaming-pipeline-rabbitmq)

## Supported sinks (destinations)

* [Amazon S3](/streaming/destinations/amazon-s3)
* [BigQuery](/streaming/destinations/bigquery)
* [ClickHouse](/streaming/destinations/clickhouse)
* [DuckDB](/streaming/destinations/duckdb)
* [Dummy](/streaming/destinations/dummy)
* [Elasticsearch](/streaming/destinations/elasticsearch)
* [Kafka](/streaming/destinations/kafka)
* [Kinesis](/streaming/destinations/kinesis)
* [MongoDB](/streaming/destinations/mongodb)
* [Microsoft SQL Server](/streaming/destinations/mssql)
* [MySQL](/streaming/destinations/mysql)
* [Opensearch](/streaming/destinations/opensearch)
* [Postgres](/streaming/destinations/postgres)
* [Redshift](/streaming/destinations/redshift)
* [Snowflake](/streaming/destinations/snowflake)
* [Trino](/streaming/destinations/trino)

## Test pipeline execution

After finishing configuring the streaming pipeline, you can click the button `Execution pipeline` to test streaming pipeline execution.

## Run pipeline in production

Create the trigger in [triggers page](/design/data-pipeline-management#create-trigger) to run streaming pipelines in production.

## Monitor streaming health

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=streaming-health)

After a streaming trigger starts creating pipeline runs in Mage Pro, open the
pipeline monitoring page to view **Streaming health** for the selected pipeline
and trigger.

Streaming health helps answer whether the stream is healthy now:

* **Rows consumed** counts rows or messages accepted from the streaming source.
* **Destination writes** counts rows or messages accepted by Mage's sink write
  path for each destination.
* **Failed writes** counts destination write attempts that raised errors.
* **Average latency** shows streaming record processing latency when samples are
  available.

Destination writes are not always the same as destination-warehouse commits. A
connector may provide stronger destination-confirmed counts, but the default
Streaming health count means Mage accepted the rows in the sink write path.

The monitoring page can group runs by trigger. This matters for streaming
pipelines because a trigger can automatically create a new pipeline run after an
exception so streaming can continue. Use trigger-level health to review the
overall stream across those restart runs and compare destination-level metric
breakdowns.

### Executor count

If you want to run multiple executors at the same time to scale the streaming pipeline execution, you can set the `executor_count` variable
in the pipeline's metadata.yaml file. Here is an example:

```yaml theme={"system"}
blocks:
- ...
- ...
executor_count: 10
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline
```

### Executor type

When running Mage on Kubernetes cluster, you can also configure streaming pipeline to be run on separate k8s pods by setting `executor_type`
field in the pipeline's metadata.yaml to `k8s`.

Example config:

```yaml theme={"system"}
blocks:
- ...
- ...
executor_type: k8s
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline
```

## Contributing guide

Follow this [doc](contributing) to add a new source or destination (sink) to Mage streaming pipeline.


# ActiveMQ
Source: https://docs.mage.ai/guides/streaming/sources/activemq

This guide will walk you through the process of setting up and using ActiveMQ with Mage for streaming data pipelines.

### Step 1: Create a new streaming pipeline

1. Open Mage in your browser.
2. Click on "+ New Pipeline" and select "Streaming".
3. Give your pipeline a name and click "Create".

### Step 2: Add an ActiveMQ data loader

1. In your new pipeline, click "+ Add Block" and select "Data loader".
2. Choose "ActiveMQ" as the data source.

### Step 3: Configure your ActiveMQ connection

In the data loader block, use the following YAML configuration as a template:

```yaml theme={"system"}
connector_type: 'activemq'
connection_host: 'localhost'
connection_port: 61613
queue_name: 'default'
username: 'admin'
password: 'admin'
```

Adjust these values according to your ActiveMQ setup:

* `connection_host`: Your ActiveMQ broker's hostname or IP address.
* `connection_port`: The port your ActiveMQ broker is listening on (default STOMP port is 61613).
* `queue_name`: The name of the queue you want to consume messages from.
* `username` and `password`: Your ActiveMQ authentication credentials.

> [!NOTE]
>
> Ensure that the stomp port on ActiveMQ is open, which is 61613 by default.

### Step 4: Additional configuration (optional)

Depending on your specific requirements, you may need to add additional configuration options:

1. For SSL/TLS connections:

```YAML theme={"system"}
use_ssl: true
ssl_cert_path: '/path/to/cert.pem'
ssl_key_path: '/path/to/key.pem'
```

2. To set a connection timeout:

```YAML theme={"system"}
connection_timeout: 30 # in seconds
```

3. To control message prefetching:

```YAML theme={"system"}
prefetch_size: 100
```

### Step 5: Test your connection

1. Save your data loader block.
2. Click the "Test" button to verify your connection to ActiveMQ.
3. If successful, you should see a sample of messages from your queue.


# Amazon SQS
Source: https://docs.mage.ai/guides/streaming/sources/amazon-sqs



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=workspaces)

## Config

```yaml theme={"system"}
connector_type: amazon_sqs
queue_names:
  - producer-events
  - controller-events
# Optional. Defaults to false so startup fails if any queue can't be initialized.
skip_failed_queues: false
# Optional. Prefer IAM roles or environment variables for production.
# aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
# aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
# aws_session_token: "{{ env_var('AWS_SESSION_TOKEN') }}"  # Optional for temporary credentials
# aws_region: "{{ env_var('AWS_DEFAULT_REGION') }}"
batch_size: 10
wait_time_seconds: 1
message_deletion_method: AFTER_RECEIVED   # AFTER_RECEIVED or MANUAL

serde_config:
  serialization_method: JSON
```

Use `queue_names` when one streaming source block should consume from multiple SQS queues.
Mage polls every configured queue and sends received messages through the same downstream
streaming pipeline. Existing single-queue configs can continue to use `queue_name`:

```yaml theme={"system"}
connector_type: amazon_sqs
queue_name: producer-events
batch_size: 10
wait_time_seconds: 1
```

By default, startup fails if any configured queue can't be initialized, including when a queue
is missing, configured for the wrong AWS region, or denied by IAM. This avoids silently ignoring
events from one producer. If you want best-effort consumption from the queues that can be
initialized, opt in with:

```yaml theme={"system"}
skip_failed_queues: true
```

When `skip_failed_queues` is enabled, Mage logs the failed queue name and continues polling the
remaining queues. If every configured queue fails to initialize, startup still fails.

## Authentication

Here are the options to authenticate with AWS SQS.

1. Grant your Mage runtime an AWS IAM role with access to the configured queues. This is the
   recommended option for Mage deployments running on AWS infrastructure.
2. Add the following keys and values to your runtime environment variables:
   * `AWS_ACCESS_KEY_ID`
   * `AWS_SECRET_ACCESS_KEY`
   * `AWS_SESSION_TOKEN` if you're using temporary AWS credentials
   * `AWS_DEFAULT_REGION`
3. Configure credentials in the source config. Use environment-variable references instead of
   pasting raw secrets into the block YAML:
   ```yaml theme={"system"}
   aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
   aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
   aws_session_token: "{{ env_var('AWS_SESSION_TOKEN') }}"
   aws_region: us-east-1
   ```
4. If you deploy Mage on an AWS ECS cluster, you can use the ECS task execution role to
   authenticate. Grant the ECS task permissions to access SQS by attaching IAM policies to this
   role.

If you get the `botocore.exceptions.NoRegionError` error, try setting the `AWS_DEFAULT_REGION`
environment variable.

## Pass raw message to transformer

```yaml theme={"system"}
serde_config:
  serialization_method: RAW_VALUE
```

## Message deletion method

We support two ways to delete messages:

1. Delete the message in the data loader automatically after deserializing the message body.
   * Set `message_deletion_method: AFTER_RECEIVED` in the source config.
   * The input of the transformer is the list of deserialized message body.
2. Manually delete the message in transformer after processing the message.
   * Set `message_deletion_method: MANUAL` in the source config.
   * The input of the transformer is the list of dictionary with the structure

     ```python theme={"system"}
     {
       'parsed_msg_body': {'k1': 'v1', ...},
       'queue_name': 'producer-events',
       'queue_url': 'https://sqs...',
       'raw_message': sqs.Message(...)
     }
     ```

     You can use the following example code to process the message and delete it after.

     ```python theme={"system"}
     @transformer
     def transform(messages: List[Dict], *args, **kwargs):
         processed_msgs = []
         for msg in messages:
             processed_msg = msg['parsed_msg_body']
             # Add your own logic to process the message
             processed_msgs.append(processed_msg)
             msg['raw_message'].delete()
         return processed_msgs
     ```


# Azure Event Hub
Source: https://docs.mage.ai/guides/streaming/sources/azure-event-hub

This guide provides step-by-step instructions to set up and use Azure Event Hub with Mage. By following these instructions,  you can create a new streaming pipeline in Mage, authenticate with Azure Event Hub, and configure your data loader, transformer,  and exporter blocks to process and manage your data efficiently.

**Start Mage**

1. Open Mage in your browser and then navigate to the pipelines page.
2. Create a new pipeline by clicking the “+ New” button and then select “Streaming.”
3. Enter a name for the pipeline and click “Create.”

**Authentication**
To authenticate with Azure Event Hub, complete the following actions:

1. Configure your io\_config.yml file with the account variables below. If you need further information about how to store secrets in
   Mage check out the [documentation](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure).

```YAML theme={"system"}
  AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
  AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
  AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
  AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
```

2. Add a data loader block to your pipeline and then select "Azure Event Hub" as the source.
3. Complete the data loader block with the information below.

```YAML theme={"system"}
connector_type: azure_event_hub
connection_str: 'Endpoint=sb://[namespace].servicebus.windows.net/;...'
eventhub_name: event_hub_name
consumer_group: '$Default'
```

> [!NOTE]
>
> Connection string: Use the `connection_str` parameter in the config as shown above. You can obtain this from your Azure portal.

4. Add the template transformer block or create a custom block to transform data from Azure Event Hub
   * The template will populate with the code below. Make adjustments to your specific requirements.

```Python theme={"system"}
from typing import Dict, List

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer

@transformer
def transform(messages: List[Dict], *args, **kwargs):
    """
    Template code for a transformer block.

    Args:
        messages: List of messages from the Event Hub.

    Returns:
        Transformed messages
    """
# Specify your transformation logic here
    print(f'Transformed messages: {messages}')
    return messages
```

5. Add a data exporter block and select “Dummy”
6. Use the default setting as seen below.

```YAML theme={"system"}
connector_type: dummy
print_msg: true
```

**Additional Configuration Options**

These additional configuration options are specifically for the data loader block that uses the Azure Event Hub connector. They should be added to the YAML configuration of your data loader block.

1. prefetch: This determines the maximum number of events to retrieve in a single batch when reading from Event Hub.
2. max\_wait\_time: This sets the maximum time to wait for a batch to be filled before returning.
3. starting\_position: This optional parameter specifies where to start reading events from in the Event Hub.
   * Common starting positions include:
     * -1 or @latest: Start from the end of the stream
     * -2 or @earliest: Start from the end of the stream

```YAML theme={"system"}
prefetch: <batch size> # optional, usually between 100 - 1000
max_wait_time: <Wait time> # optional
starting_position: <starting position> #optional, example '@latest'
```

By following this guide, you can successfully set up a streaming pipeline in Mage using Azure Event Hub. This allows you to authenticate,
load, transform, and export data efficiently. With these steps, you can leverage Mage and Azure Event Hub to create powerful and scalable
streaming data pipelines.


# Google Cloud PubSub
Source: https://docs.mage.ai/guides/streaming/sources/google-cloud-pubsub

Ingest data from Google Cloud PubSub event streaming sources.

![Google Cloud PubSub Source](https://lh3.googleusercontent.com/1-WPjO3QIWh0PKPXhHNcKUNcTCeKX218WsC6nIv1xGjGg3VQci-nG9uTqUyBlStJ8YR5LVaC8QFXN6Y7Y9lz=w160-h160)

## Config

```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
subscription_id: test-subscription-id-0
timeout: 5
batch_size: 100
pubsub_emulator_host: null  # "host.docker.internal:8085"
path_to_credentials_json_file: "./google_credentials.json"
```

The `pubsub_emulator_host` parameter is optional, which could be used when you need to test
the Google Cloud PubSub service locally through an emulator (see more details at the bottom).

## Start Mage

### Create a new pipeline

1. Open Mage in your browser.

2. Click **`+ New pipeline`**, then select `Streaming`.

3. Add a data loader block, select `Google Cloud PubSub`, and change the settings to those of your PubSub project:

   ```yaml theme={"system"}
   connector_type: google_cloud_pubsub
   project_id: test-project-id
   topic_id: test-topic-id
   subscription_id: test-subscription-id-0
   timeout: 5
   batch_size: 100
   pubsub_emulator_host: null
   path_to_credentials_json_file: "./google_credentials.json"
   ```

4. Add a transformer block and paste the following:

   ```python theme={"system"}
   from typing import Dict, List

   if 'transformer' not in globals():
       from mage_ai.data_preparation.decorators import transformer


   @transformer
   def transform(messages: List[Dict], *args, **kwargs):
       """
       Template code for a transformer block.

       Args:
           messages: List of messages in the stream.

       Returns:
           Transformed messages
       """
       # Specify your transformation logic here
       print(f'Transform: {messages}')
       return messages
   ```

5. Add a data exporter block, select `Dummy` and keep the default settings:

   ```yaml theme={"system"}
   connector_type: dummy
   print_msg: true
   ```

## ![Image of the testing pipeline for "Google Cloud PubSub Source"](https://mintcdn.com/mage/KCVtYQD46PIylqWZ/guides/streaming/sources/google_cloud_pubsub_source.png?fit=max\&auto=format\&n=KCVtYQD46PIylqWZ\&q=85\&s=b7faef1a2baaeb12601ba99a66295bdf)

## Test pipeline

Open the streaming pipeline you just created, and in the right side panel near
the bottom, click the button **Execute pipeline** to test the pipeline.

### Publish some testing messages

Please use the publisher example given in the [Testing apps locally with the emulator](https://cloud.google.com/pubsub/docs/emulator)

```
python publisher.py test-project-id publish test-topic-id
```

You should see an output similar to the following:

```
[silent_mountain] [GoogleCloudPubSubSource] Subscription already exists: projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub
[silent_mountain] [GoogleCloudPubSubSource] Start consuming batch messages.
[silent_mountain] [GoogleCloudPubSubSource] Number of received messages: 4
[late_wood] Transform: [{'data': 'Google Cloud PubSub Message number 1'}, {'data': 'Google Cloud PubSub Message number 8'}, {'data': 'Google Cloud PubSub Message number 2'}, {'data': 'Google Cloud PubSub Message number 3'}]
[silent_mountain] [DummySink] Batch ingest 4 records, time=1682968557.6466439. Sample: {'data': 'Google Cloud PubSub Message number 1'}
[silent_mountain] [GoogleCloudPubSubSource] Received and acknowledged 4 messages from projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub.
[silent_mountain] [GoogleCloudPubSubSource] Number of received messages: 4
[late_wood] Transform: [{'data': 'Google Cloud PubSub Message number 7'}, {'data': 'Google Cloud PubSub Message number 9'}, {'data': 'Google Cloud PubSub Message number 4'}, {'data': 'Google Cloud PubSub Message number 5'}]
[silent_mountain] [DummySink] Batch ingest 4 records, time=1682968559.027885. Sample: {'data': 'Google Cloud PubSub Message number 7'}
[silent_mountain] [GoogleCloudPubSubSource] Received and acknowledged 4 messages from projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub.
[silent_mountain] [GoogleCloudPubSubSource] Number of received messages: 1
[late_wood] Transform: [{'data': 'Google Cloud PubSub Message number 6'}]
[silent_mountain] [DummySink] Batch ingest 1 records, time=1682968560.894928. Sample: {'data': 'Google Cloud PubSub Message number 6'}
[silent_mountain] [GoogleCloudPubSubSource] Received and acknowledged 1 messages from projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub.
```

## Set up Google Cloud Pubsub emulator

To test out Google Cloud PubSub source locally, follow the instructions in
[Testing apps locally with the emulator](https://cloud.google.com/pubsub/docs/emulator)
to set up the local Google Cloud PubSub emulator.

## Create a `project_id` for testing

Use the command listed under the `Starting the emulator` section to create a `project_id`, e.g.,

```
gcloud beta emulators pubsub start --project=test-project-id

$(gcloud beta emulators pubsub env-init)
```

## Create a topic and a subscription

Use the `gcloud pubsub topics create` command to create a topic:

```
gcloud pubsub topics create test-topic-id
```

After you create a topic, you can subscribe or publish to it. Use the
`gcloud pubsub subscriptions create` command to create a subscription.

```
gcloud pubsub subscriptions create test-subscription-id-0 --topic test-topic-id
```

Only messages published to the topic after the subscription is created
are available to subscriber applications.


# InfluxDB
Source: https://docs.mage.ai/guides/streaming/sources/influxdb



## InfluxDB instance

InfluxDB can be [run locally](https://docs.influxdata.com/influxdb/v2.7/install/) or in the cloud.

## Basic config

Basic configuration includes the server connection details:

```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
```

## Data format

InfluxDB supports [data elements](https://docs.influxdata.com/influxdb/v2.7/reference/key-concepts/data-elements/) for structuring your data.
The influxdb data exporter can create these elements if the messages received contain a metadata dictionary:

```python theme={"system"}
message['metadata'] = {
    'measurement': str
    'time': int
    'tags': Dict
}
```

The following message shows the configuration of all possible elements data and metadata:

```python theme={"system"}
message = {
    "data": {'bees': 23},
    "metadata": {
        'measurement': 'census',
        'time': 1693472675783,  # timestamp with ms precision
        'tags': {'location': 'klamath', 'scientist': 'anderson'}
    }
}
```

## Advanced config

The influxdb data reader reads data from InfluxDB server in timed intervals.
Every `timeout_ms` data with time stamps in the interval `(t-time_delay, t-time_delay+timeout_ms)` will be returned, where `t` is the current time.
Note that InfluxDB is not a streaming database, so trying to set `time_delay` too small may result in data loss.
This will occur if data is written to the database after the corresponding interval has been read.
The time delay is given as [InfluxDB duration](https://docs.influxdata.com/flux/v0.x/data-types/basic/duration/).

The `batch_size` limits the size of batches returned by the data reader.
If `print_intervals` is `true` the intervals are printed on every time step.
The `filter_fn` can be used to further specify the data, e.g. `r._measurement == "cpu" and r._field == "usage_system"` (note the usage of single quotation marks outside and double quotation marks inside the filter\_fn string).
Defaults to `true` for no further filtering.
For documentation, see [InfluxDB filter() function](https://docs.influxdata.com/flux/v0.x/stdlib/universe/filter/)

```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
time_delay: "5m"
timeout_ms: 1000
batch_size: 100
print_intervals: false
filter_fn: 'true'
```


# Kafka
Source: https://docs.mage.ai/guides/streaming/sources/kafka



## Basic config

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
topics: ['topic_name1', 'topic_name2']  # Optional. Used for specifying multiple topics.
consumer_group: unique_consumer_group
include_metadata: false
api_version: "0.10.2"
batch_size: 100
timeout_ms: 500
auto_offset_reset: latest # Optional
max_partition_fetch_bytes: 5242880 # Optional, default: 5MB (Pro), 1MB (OSS)
max_partition_fetch_bytes: 1048576 # Optional
offset:  # Optional
  value: null # Optional
  type: "int"
  partitions: # Optional
  - {"topic": "test", "partition": 1}
```

### Configuration Parameters

#### Required Parameters

* **`bootstrap_server`**: Kafka broker address (e.g., `"localhost:9092"` or `"broker1:9092,broker2:9092"`)
* **`consumer_group`**: Unique consumer group identifier for this pipeline
* **`topic`** or **`topics`**: Single topic name (string) or list of topic names to consume from

#### Performance Parameters

* **`batch_size`** (default: `100`): Maximum number of messages to process in a single batch. The consumer will accumulate messages across multiple polls to reach this target when possible.

  **Recommendations:**

  * Small messages (\< 1 KB): 500-2000
  * Medium messages (1-10 KB): 200-1000
  * Large messages (> 10 KB): 50-500

* **`max_partition_fetch_bytes`** (default: `5242880` = 5MB for Pro, `1048576` = 1MB for OSS): Maximum amount of data to fetch per partition in a single poll. This is a **per-partition limit** that can constrain your batch size.

  **Important:** This value must be:

  * Greater than your largest message size
  * Aligned with broker settings (`message.max.bytes`, `replica.fetch.max.bytes`)

  **Recommendations by message size:**

  * Small messages (\< 1 KB): 1-5 MB
  * Medium messages (1-10 KB): 5-10 MB
  * Large messages (10-100 KB): 10-50 MB
  * Very large messages (> 100 KB): 50-100 MB

  **Memory consideration:** Total in-flight data = `max_partition_fetch_bytes × number of partitions`. Ensure your consumer has sufficient heap memory.

* **`timeout_ms`** (default: `500`): Maximum time to wait for messages when polling (in milliseconds). Lower values reduce latency but may increase CPU usage.

#### Other Parameters

* **`auto_offset_reset`** (default: `latest`): What to do when there is no initial offset or the offset is out of range. Options: `earliest`, `latest`, `none`
* **`include_metadata`**: If `true`, includes Kafka metadata (key, partition, offset, timestamp, topic) with each message
* **`api_version`** (default: `"0.10.2"`): Kafka API version. Use a version matching your broker version for best compatibility

### Offset Configuration

> [!NOTE]
>
> Available in versions `>= 0.9.61`

The `offset` config allows you to reset offset when starting the streaming pipeline. If using the `offset` config, the `partitions` config is required.

`offset` config has 4 optional values:

1. `beginning` - Start from the earliest available message
2. `end` - Start from the latest message (skip all existing messages)
3. `int` - Start from a specific numeric offset
4. `timestamp` - Start from messages at or after a specific timestamp

**Examples:**

Start from beginning:

```yaml theme={"system"}
offset:
  type: "beginning"
  partitions:
    - {"topic": "my_topic", "partition": 0}
    - {"topic": "my_topic", "partition": 1}
```

Start from specific offset:

```yaml theme={"system"}
offset:
  type: "int"
  value: 1000
  partitions:
    - {"topic": "my_topic", "partition": 0}
```

Start from specific timestamp (milliseconds since epoch):

```yaml theme={"system"}
offset:
  type: "timestamp"
  value: 1693396630163  # Wed Aug 30 2023 11:57:10 UTC
  partitions:
    - {"topic": "my_topic", "partition": 0}
```

**Note:** Both `beginning` and `end` do not require `value` field.

1. beginning
2. end
3. int
4. timestamp

Both `beginning` and `end` set the consumer to consume data from the beginning and end of the queue, respectively.
Also, they do not require `offset_value`

`int` config set the consumer to consume data from the given offset value inside the queue.
This value correspond to the numeric position inside each partition.

`timestamp` config set the consumer to consume data from the given offset timestamp value inside the queue.
This value correspond to the timestamp of the message (Unit should be milliseconds since beginning of the epoch)

## SSL authentication

```yaml theme={"system"}
security_protocol: "SSL"
ssl_config:
  cafile: "CARoot.pem"
  certfile: "certificate.pem"
  keyfile: "key.pem"
  password: password
  check_hostname: true
```

## SASL authentication

SASL\_PLAINTEXT config:

```yaml theme={"system"}
security_protocol: "SASL_PLAINTEXT"
sasl_config:
  mechanism: "PLAIN"
  username: username
  password: password
```

SASL\_SSL config:

```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
  mechanism: "SCRAM-SHA-512"
  username: username
  password: password
ssl_config:
  cafile: "pemfilename.pem"
```

SASL\_SSL with OAUTHBEARER (OAuth 2.0):

```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
  mechanism: "OAUTHBEARER"
  oauth_token_url: "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token"
  oauth_client_id: "kafka-client"
  oauth_client_secret: "your-client-secret"
  oauth_scope: "kafka"  # Optional
ssl_config:
  cafile: "pemfilename.pem"
  check_hostname: true
```

> [!NOTE]
>
> **OAUTHBEARER Authentication:**
>
> * The `oauth_token_url`, `oauth_client_id`, and `oauth_client_secret` are **required** when using `OAUTHBEARER` mechanism.
> * The `oauth_scope` parameter is optional.
> * OAuth tokens are automatically cached and refreshed before expiry (60 seconds before expiration by default).
> * It's recommended to use `SASL_SSL` (not `SASL_PLAINTEXT`) with OAUTHBEARER for secure token transmission.
> * Compatible with OAuth 2.0 providers like Keycloak, Okta, and other identity servers that support client credentials flow.

## Data format

By default, if `include_metadata` is false, the kafka data loader returns data from value field, e.g.

```python theme={"system"}
{"location": "klamath", "scientist": "anderson", "count": 23}
```

Kafka supports [structuring and partitioning](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html#kafka.KafkaProducer.send) your data.
If `include_metadata` is set to true, the kafka data loader returns these elements as messages with `data = {key: value}` and `metadata` with (topic, partition, offset and time), e.g.

```python theme={"system"}
message = {
    "data": {"location": "klamath", "scientist": "anderson", "count": 23},
    "metadata": {
      "key": "bees",
      "partition": 0,
      "offset": 6,
      "time": 1693396630163,  # timestamp with ms precision (Wed Aug 30 2023 11:57:10 GMT+0)
      "topic": "census",
    }}
```

## Deserialize message with protobuf schema

```yaml theme={"system"}
serde_config:
  serialization_method: PROTOBUF
  schema_classpath: "path.to.schema.SchemaClass"
```

* Specify the `serialization_method` to `PROTOBUF`.
* Set the `schema_classpath` to the path to the Python schema class. Test whether you have access the the schema with the code
  in a scratchpad.
  ```python theme={"system"}
  from path.to.schema import SchemaClass
  ```

## Pass raw message to transformer

```yaml theme={"system"}
serde_config:
  serialization_method: RAW_VALUE
```

## Deserialize message with Avro schema in Confluent schema registry

```yaml theme={"system"}
serde_config:
  serialization_method: AVRO
  schema_registry_url: https://schema_registry_url
  schema_registry_username: username
  schema_registry_password: password
```

## API version

In case you are using newer versions of Kafka brokers, you should consider using corresponding Kafka api\_version.

## Performance Tuning

### Understanding Batch Size vs. Fetch Bytes

The `batch_size` and `max_partition_fetch_bytes` work together:

* **`batch_size`**: Target number of messages to process together
* **`max_partition_fetch_bytes`**: Maximum data (bytes) fetched per partition per poll

**Example:** If you set `batch_size: 500` but `max_partition_fetch_bytes: 1048576` (1MB) and your messages are 5KB each:

* Each partition can fetch \~200 messages (1MB ÷ 5KB)
* With 1 partition, you'll get \~200 messages per batch
* With multiple partitions, you may get closer to 500, but each partition is still limited to 1MB

**Solution:** Increase `max_partition_fetch_bytes` to allow more messages per partition:

```yaml theme={"system"}
batch_size: 500
max_partition_fetch_bytes: 5242880  # 5MB allows ~1000 messages per partition
```

### High-Throughput Configuration

For high-volume scenarios with consumer lag:

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "broker1:9092,broker2:9092"
topic: high_volume_topic
consumer_group: high_throughput_consumer
batch_size: 1000
max_partition_fetch_bytes: 10485760  # 10MB
timeout_ms: 1000
auto_offset_reset: earliest
```

### Low-Latency Configuration

For real-time processing with minimal delay:

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: realtime_topic
consumer_group: low_latency_consumer
batch_size: 10
max_partition_fetch_bytes: 1048576  # 1MB
timeout_ms: 100
auto_offset_reset: latest
```

### Memory Planning

Calculate memory requirements:

```
Estimated Memory = max_partition_fetch_bytes × number_of_partitions × 1.5
```

The 1.5x multiplier accounts for overhead. Ensure your consumer process has at least this much memory available.

## Troubleshooting

### Issue: Not Receiving Expected Batch Size

**Symptom:** Set `batch_size: 500` but only getting \~200-250 messages per batch.

**Cause:** `max_partition_fetch_bytes` is too low for your message size.

**Solution:**

1. Calculate: `max_partition_fetch_bytes ÷ average_message_size = max_messages_per_partition`
2. Increase `max_partition_fetch_bytes`:
   ```yaml theme={"system"}
   max_partition_fetch_bytes: 5242880  # 5MB (or higher)
   ```
3. Verify broker settings allow this value:
   * `message.max.bytes >= max_partition_fetch_bytes`
   * `replica.fetch.max.bytes >= max_partition_fetch_bytes`

### Issue: Consumer Lag Growing

**Symptom:** Consumer falling behind producer, lag increasing over time.

**Solutions:**

1. Increase `batch_size` and `max_partition_fetch_bytes`
2. Increase `timeout_ms` to allow more time for message accumulation
3. Scale horizontally: Set higher value for [executor\_count](/guides/streaming/introduction#executor-count) to run multiple consumer instances in the same consumer group so Kafka can distribute partitions among them.
4. Optimize downstream processing speed

### Issue: Out of Memory Errors

**Symptom:** Consumer crashes with memory errors.

**Cause:** `max_partition_fetch_bytes × number_of_partitions` exceeds available memory.

**Solution:**

1. Reduce `max_partition_fetch_bytes`
2. Reduce number of partitions consumed (use topic filtering)
3. Increase consumer heap memory
4. Reduce `batch_size` to process smaller batches

### Issue: Configuration Changes Not Taking Effect

**Symptom:** Changed `batch_size` or `max_partition_fetch_bytes` but behavior unchanged.

**Cause:** Kafka consumer is initialized at pipeline start. Configuration is read during initialization.

**Solution:** Restart the streaming pipeline after changing configuration parameters.

### Issue: Messages Not Being Consumed

**Symptom:** No messages received despite messages in topic.

**Checklist:**

1. Verify `consumer_group` is unique and not being used by another consumer
2. Check `auto_offset_reset` setting (use `earliest` to read from beginning)
3. Verify topic name spelling matches exactly
4. Check network connectivity to `bootstrap_server`
5. Verify consumer has read permissions for the topic
6. Check if using `offset` config - it may be seeking to a specific position

## Complete Example Configurations

### Example 1: Basic Setup with Small Messages

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: user_events
consumer_group: user_events_processor
batch_size: 500
max_partition_fetch_bytes: 2097152  # 2MB
timeout_ms: 500
auto_offset_reset: latest
include_metadata: false
```

### Example 2: High-Volume Processing with Large Messages

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "kafka-broker1:9092,kafka-broker2:9092"
topic: large_payload_topic
consumer_group: large_payload_consumer
batch_size: 100
max_partition_fetch_bytes: 52428800  # 50MB for large messages
timeout_ms: 2000
auto_offset_reset: earliest
include_metadata: true
```

### Example 3: Multiple Topics with Authentication

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "secure-kafka:9093"
topics: ['topic1', 'topic2', 'topic3']
consumer_group: multi_topic_consumer
batch_size: 200
max_partition_fetch_bytes: 10485760  # 10MB
timeout_ms: 1000
security_protocol: SASL_SSL
sasl_config:
  mechanism: "SCRAM-SHA-512"
  username: kafka_user
  password: kafka_password
ssl_config:
  cafile: "/path/to/ca-cert.pem"
```

### Example 4: Backfill from Specific Timestamp

```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: historical_data
consumer_group: backfill_consumer
batch_size: 1000
max_partition_fetch_bytes: 10485760  # 10MB
timeout_ms: 1000
offset:
  type: "timestamp"
  value: 1609459200000  # 2021-01-01 00:00:00 UTC
  partitions:
    - {"topic": "historical_data", "partition": 0}
    - {"topic": "historical_data", "partition": 1}
    - {"topic": "historical_data", "partition": 2}
```

## Quick Reference

### Configuration Quick Reference Table

| Parameter | Default | Recommended Range | Notes |
| - | - | - | - |
| `batch_size` | 100 | 50-2000 | Higher for small messages, lower for large |
| `max_partition_fetch_bytes` | 5MB (Pro), 1MB (OSS) | 1MB-100MB | Must exceed max message size |
| `timeout_ms` | 500 | 100-5000 | Lower = lower latency, higher CPU |
| `auto_offset_reset` | latest | earliest/latest | earliest for backfill, latest for real-time |

### Message Size to Configuration Mapping

| Message Size | Recommended `batch_size` | Recommended `max_partition_fetch_bytes` |
| - | - | - |
| \< 1 KB | 500-2000 | 1-5 MB |
| 1-10 KB | 200-1000 | 5-10 MB |
| 10-100 KB | 50-500 | 10-50 MB |
| > 100 KB | 10-100 | 50-100 MB |

## Best Practices

1. **Consumer Groups:** Use unique consumer group names per pipeline/environment (e.g., `pipeline_prod`, `pipeline_staging`)
2. **Batch Sizing:** Start with defaults and tune based on message size and throughput needs
3. **Monitoring:** Monitor consumer lag, batch sizes, and memory usage regularly
4. **Broker Alignment:** Ensure `max_partition_fetch_bytes` aligns with broker configuration (`message.max.bytes`, `replica.fetch.max.bytes`)
5. **Error Handling:** Implement proper error handling in your transformation logic to handle malformed messages
6. **Offset Management:**
   * Use `auto_offset_reset: earliest` for initial processing or backfills
   * Use `auto_offset_reset: latest` for real-time streaming
   * Use explicit `offset` config for precise control
7. **Multiple Topics:** When consuming from multiple topics, ensure all topics exist and are accessible
8. **Restart After Config Changes:** Always restart the pipeline after changing `batch_size` or `max_partition_fetch_bytes`
9. **Memory Planning:** Calculate memory needs: `max_partition_fetch_bytes × partitions × 1.5`
10. **Testing:** Test configuration changes in a staging environment before production


# Kinesis
Source: https://docs.mage.ai/guides/streaming/sources/kinesis



## Config

```yaml theme={"system"}
connector_type: kinesis
stream_name: stream_name
batch_size: 100
```

## Authentication

Here are the options to authenticate with the AWS Kinesis.

1. Add the following keys and values to your environment variables

* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`

2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate.
   You can grant the ECS task permissions to access other AWS services by attaching IAM policies
   to this ECS task execution role.

If you get the `botocore.exceptions.NoRegionError` error, try setting the `AWS_DEFAULT_REGION`
environment variable.


# MongoDB
Source: https://docs.mage.ai/guides/streaming/sources/mongodb



## Basic Config

```yaml theme={"system"}
connector_type: mongodb
connection_str: "mongodb://localhost:27017/" #Must be replica set and MongoVersion is higher than Mongo5.1
database: "test"
collection: "test"
```

The following additional configuration options can be included: batch\_size, pipeline, starting\_after, full\_document. Instructions on how to use these options can be found in the [pymongo documentation](https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.watch).


# Microsoft SQL Server CDC
Source: https://docs.mage.ai/guides/streaming/sources/mssql-cdc

Real-time Change Data Capture from SQL Server CDC change tables with automatic schema tracking

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mssql-cdc)

## Overview

The Microsoft SQL Server CDC streaming source captures real-time changes from SQL Server Change Data Capture (CDC) change tables and processes them into structured records. It uses SQL Server's built-in CDC feature to poll for changes and provides comprehensive CDC capabilities with automatic schema tracking and primary key detection.

> **Real-time CDC**
>
> Captures changes as they happen in SQL Server

> **Native CDC**
>
> Uses SQL Server's built-in Change Data Capture feature

> **Primary Key Detection**
>
> Automatically identifies and extracts primary key columns

> **LSN Tracking**
>
> Tracks Log Sequence Numbers for reliable resume capabilities

## Configuration

### Required Parameters

| Parameter | Type | Description |
| - | - | - |
| `host` | string | SQL Server hostname or IP address |
| `port` | number | SQL Server port (default: 1433) |
| `database` | string | Database name |
| `user` | string | SQL Server username with CDC access |
| `password` | string | SQL Server password |

### Optional Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `connect_timeout` | number | 10 | Connection timeout in seconds |
| `driver` | string | "ODBC Driver 18 for SQL Server" | ODBC driver name |
| `authentication` | string | - | Authentication method (e.g., "ActiveDirectoryPassword") |
| `schema` | string | dbo | Default schema name |
| `schemas` | array | - | Include only these schemas |
| `tables` | array | - | Include only these tables |
| `ignore_schemas` | array | - | Exclude these schemas (supports wildcards like "sys\*") |
| `ignore_tables` | array | - | Exclude these tables (supports wildcards like "temp\_\*") |
| `start_lsn` | string | - | Start LSN to begin reading from (hex string, e.g., '0x00000000000000000000') |
| `capture_instance` | string | - | Capture instance name (auto-detected if not provided) |
| `max_batch_size` | number | 100 | Maximum events per batch |
| `flush_interval_seconds` | number | 1.0 | Maximum time between batch flushes |
| `poll_interval_seconds` | number | 1.0 | How often to poll for new changes |
| `return_db_records_only` | boolean | true | Return only DB records with Mage timestamp columns |

## Record Formats

### DB Records Only Format

When `return_db_records_only: true` (default):

```json theme={"system"}
{
  "data": {
    "id": 1,
    "name": "John",
    "email": "john@example.com",
    "_mage_created_at": "2021-12-31T23:00:00+00:00",
    "_mage_updated_at": null,
    "_mage_deleted_at": null
  },
  "metadata": {
    "schema": "dbo",
    "table": "users",
    "key_columns": ["id"],
    "operation": "INSERT",
    "lsn": "0x00000000000000000000"
  }
}
```

**Note**: The `_mage_*` timestamp columns are returned as datetime objects (ISO 8601 format strings when serialized) in UTC timezone, not Unix timestamps. This ensures proper type handling when writing to databases.

## Event Types

> **Row Events (Core CDC)**
>
> * **Insert** (`operation: "INSERT"`): INSERT operations (CDC operation code 2)
> * **Update** (`operation: "UPDATE"`): UPDATE operations (CDC operation code 4, after image only)
> * **Delete** (`operation: "DELETE"`): DELETE operations (CDC operation code 1)
>
> These events contain the actual data changes. UPDATE operations only include the after image (the new values), not the before image.

## Primary Key Detection

The source automatically detects and extracts primary key columns:

1. **Schema Discovery**: Queries `INFORMATION_SCHEMA` for table structure
2. **Primary Key Detection**: Identifies actual primary key columns from database constraints
3. **Caching**: Stores schema and primary key information for performance
4. **Key Extraction**: Extracts primary key column names from row events

```json Example Output theme={"system"}
{
  "metadata": {
    "key_columns": ["user_id", "tenant_id"]
  }
}
```

**Note**: `key_columns` is a list of column names (not values) that can be used for deduplication or upsert operations in downstream sinks.

## Examples

```yaml Basic Setup theme={"system"}
connector_type: mssql
host: localhost
port: 1433
database: mydb
user: cdc_user
password: password
schemas: ["dbo"]
return_db_records_only: true
```

```yaml Production Setup theme={"system"}
connector_type: mssql
host: sqlserver-prod.company.com
port: 1433
database: production_db
user: cdc_user
password: secure_password
driver: "ODBC Driver 18 for SQL Server"
schemas: ["dbo", "sales"]
ignore_tables: ["temp_*", "log_*"]
max_batch_size: 500
flush_interval_seconds: 2.0
poll_interval_seconds: 0.5
return_db_records_only: true
```

```yaml Resume from LSN theme={"system"}
connector_type: mssql
host: localhost
port: 1433
database: mydb
user: cdc_user
password: password
start_lsn: "0x00000000000000000000"
schemas: ["dbo"]
return_db_records_only: true
```

```yaml With Azure AD Authentication theme={"system"}
connector_type: mssql
host: sqlserver.database.windows.net
port: 1433
database: mydb
user: user@domain.com
password: password
authentication: "ActiveDirectoryPassword"
driver: "ODBC Driver 18 for SQL Server"
schemas: ["dbo"]
return_db_records_only: true
```

```yaml Filter Specific Tables theme={"system"}
connector_type: mssql
host: localhost
port: 1433
database: mydb
user: cdc_user
password: password
tables: ["users", "orders", "products"]
return_db_records_only: true
```

## Prerequisites

### Enable CDC on Database

CDC must be enabled at the database level before it can be used:

```sql theme={"system"}
USE your_database_name;
GO
EXEC sys.sp_cdc_enable_db;
GO
```

Verify CDC is enabled:

```sql theme={"system"}
SELECT is_cdc_enabled FROM sys.databases WHERE name = 'your_database_name';
-- Should return: 1
```

### Enable CDC on Tables

After enabling CDC on the database, enable it for each table you want to monitor:

```sql theme={"system"}
USE your_database_name;
GO
EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name = N'YourTableName',
    @role_name = NULL,
    @supports_net_changes = 0;
GO
```

**Parameters:**

* `@source_schema`: Schema name (e.g., 'dbo')
* `@source_name`: Table name
* `@role_name`: Role name for CDC access control (NULL = no role restriction)
* `@supports_net_changes`: 0 = all changes, 1 = net changes only

### Verify CDC on Tables

Check which tables have CDC enabled:

```sql theme={"system"}
SELECT
    s.name AS schema_name,
    t.name AS table_name,
    c.capture_instance
FROM cdc.change_tables c
INNER JOIN sys.tables t ON c.source_object_id = t.object_id
INNER JOIN sys.schemas s ON t.schema_id = s.schema_id
ORDER BY s.name, t.name;
```

### SQL Server Agent

CDC requires SQL Server Agent to be running. Verify it's running:

```sql theme={"system"}
EXEC xp_servicecontrol 'QueryState', N'SQLServerAGENT';
```

If not running, start the SQL Server Agent service.

### User Permissions

The user needs the following permissions:

```sql theme={"system"}
-- Grant database access
USE your_database_name;
GO

-- Grant SELECT on tables
GRANT SELECT ON SCHEMA::dbo TO cdc_user;

-- Grant SELECT on CDC tables
GRANT SELECT ON SCHEMA::cdc TO cdc_user;

-- Grant SELECT on system tables for metadata queries
GRANT SELECT ON sys.tables TO cdc_user;
GRANT SELECT ON sys.schemas TO cdc_user;
GRANT SELECT ON sys.databases TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.TABLES TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.COLUMNS TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.TABLE_CONSTRAINTS TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.KEY_COLUMN_USAGE TO cdc_user;
```

## Troubleshooting

> **Common Issues**
>
> 1. **CDC Not Enabled on Database**: Run `EXEC sys.sp_cdc_enable_db;` on the database
> 2. **CDC Not Enabled on Table**: Run `sys.sp_cdc_enable_table` for each table
> 3. **SQL Server Agent Not Running**: Start SQL Server Agent service
> 4. **Permission Denied**: Ensure user has SELECT permissions on tables and CDC schema
> 5. **Connection Timeout**: Increase `connect_timeout` value
> 6. **LSN Format Error**: Ensure LSN is in hex format (e.g., '0x00000000000000000000')
> 7. **ODBC Driver Not Found**: Install appropriate ODBC driver (e.g., "ODBC Driver 18 for SQL Server")
> 8. **Max LSN Returns NULL**: This usually means CDC is not enabled on the database

> **Debugging**
>
> Check CDC status on database:
>
> ```sql theme={"system"}
> SELECT is_cdc_enabled FROM sys.databases WHERE name = 'your_database_name';
> ```
>
> Check CDC enabled tables:
>
> ```sql theme={"system"}
> SELECT
>     s.name AS schema_name,
>     t.name AS table_name,
>     c.capture_instance
> FROM cdc.change_tables c
> INNER JOIN sys.tables t ON c.source_object_id = t.object_id
> INNER JOIN sys.schemas s ON t.schema_id = s.schema_id;
> ```
>
> Check current max LSN:
>
> ```sql theme={"system"}
> SELECT sys.fn_cdc_get_max_lsn() AS max_lsn;
> ```
>
> Check min LSN for a capture instance:
>
> ```sql theme={"system"}
> SELECT sys.fn_cdc_get_min_lsn('dbo_YourTableName_CT') AS min_lsn;
> ```
>
> View recent CDC changes:
>
> ```sql theme={"system"}
> DECLARE @from_lsn binary(10) = sys.fn_cdc_get_min_lsn('dbo_YourTableName_CT');
> DECLARE @to_lsn binary(10) = sys.fn_cdc_get_max_lsn();
>
> SELECT * FROM cdc.fn_cdc_get_all_changes_dbo_YourTableName_CT(
>     @from_lsn, @to_lsn, N'all'
> );
> ```

> **Monitoring**
>
> Monitor key metrics:
>
> * **Batch Size**: Average events per batch
> * **Flush Rate**: How often batches are flushed
> * **Error Rate**: Failed events or connections
> * **Lag**: Time between event occurrence and processing
> * **LSN Progress**: Track Log Sequence Number progress
>
> Check CDC capture instance status:
>
> ```sql theme={"system"}
> SELECT
>     capture_instance,
>     source_schema,
>     source_table,
>     capture_job_id,
>     supports_net_changes
> FROM cdc.change_tables;
> ```
>
> Monitor CDC job status:
>
> ```sql theme={"system"}
> SELECT
>     job_id,
>     name,
>     enabled,
>     date_modified
> FROM msdb.dbo.sysjobs
> WHERE name LIKE 'cdc.%';
> ```

## Integration with Generic IO Sink

The SQL Server CDC source works seamlessly with the Generic IO sink, which provides:

* **Automatic Column Type Mapping**: Automatically maps `_mage_*` timestamp columns to appropriate database types (TIMESTAMP, DATETIME2, DateTime64, etc.) based on the target database
* **Metadata Interpolation**: Use metadata values from SQL Server CDC events in sink configurations for dynamic routing and table naming

### Supported Databases

Generic IO Sink supports the following databases:

* **BigQuery**
* **ClickHouse**
* **DuckDB**
* **MySQL**
* **MSSQL**
* **Postgres**

### Metadata Interpolation

You can use metadata values from SQL Server CDC events in your sink configuration using Python string formatting syntax:

* **`{schema}`**: Schema name from the event
* **`{table}`**: Table name from the event
* **`{key_columns}`**: List of primary key column names (e.g., `["id"]` or `["user_id", "tenant_id"]`)

When using `{key_columns}` in `unique_constraints`, it will be automatically converted from a string representation to a list. The format supports both Python-style (`"['id']"`) and JSON-style (`'["id"]'`) array strings.

### Example Configurations

```yaml BigQuery Example theme={"system"}
connector_type: bigquery
profile: default
config:
  table_id: "{schema}.{table}"  # Dynamic table names from metadata
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"  # Uses primary keys from metadata
```

```yaml Postgres Example theme={"system"}
connector_type: postgres
profile: default
config:
  schema_name: "{schema}"  # Dynamic schema from metadata
  table_name: "{table}"    # Dynamic table name from metadata
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"  # Uses primary keys from metadata
  if_exists: append
```

```yaml MSSQL Example theme={"system"}
connector_type: mssql
profile: default
config:
  schema_name: "{schema}"
  table_name: "{table}"
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"
  if_exists: append
```

```yaml Static Configuration theme={"system"}
connector_type: postgres
profile: default
config:
  schema_name: public
  table_name: cdc_events  # Static table name
  unique_constraints: ['id', 'event_id']  # Static list (string format also works: "['id', 'event_id']")
  unique_conflict_method: UPDATE
  if_exists: append
```

### How Metadata Interpolation Works

1. **Message Grouping**: Messages are automatically grouped by their interpolated config values. For example, if you use `table_name: "{schema}_{table}"`, messages from `dbo.users` will be grouped together and written to `dbo_users` table.

2. **Key Columns Interpolation**: When using `{key_columns}` in `unique_constraints`, the sink automatically:
   * Converts the list to a string representation during interpolation
   * Parses it back to a list (supports both `"['id']"` and `'["id"]'` formats)
   * Uses it for upsert operations based on `unique_conflict_method`

**Example**: If you have a table `users` with primary key `id`, and you configure `unique_constraints: "{key_columns}"`, it will automatically use `["id"]` for upsert operations.

## Best Practices

1. **Enable CDC on Database First**: Always enable CDC at the database level before enabling on tables
2. **Monitor SQL Server Agent**: Ensure SQL Server Agent is running for CDC to work
3. **Filter Events**: Use schema/table filters to reduce processing overhead
4. **Monitor Resources**: Watch memory and CPU usage, especially with high-volume tables
5. **Test Resume**: Verify checkpoint functionality with LSN tracking
6. **Secure Connections**: Use encrypted connections in production
7. **Regular Backups**: Backup checkpoint files
8. **Schema Validation**: Test with schema changes
9. **Use Generic IO Sink**: Leverage automatic type mapping and metadata interpolation for flexible data routing
10. **Metadata Interpolation**: Use `{schema}`, `{table}`, and `{key_columns}` for dynamic table routing and upsert configuration
11. **Poll Interval**: Adjust `poll_interval_seconds` based on your latency requirements (lower = more frequent polling, higher CPU usage)
12. **Batch Size**: Adjust `max_batch_size` based on your throughput needs (larger = fewer handler calls, more memory usage)

## Timestamp Handling

The SQL Server CDC source automatically adds Mage timestamp columns:

* **`_mage_created_at`**: Set to event timestamp (datetime) for INSERT operations
* **`_mage_updated_at`**: Set to event timestamp (datetime) for UPDATE operations
* **`_mage_deleted_at`**: Set to event timestamp (datetime) for DELETE operations

All timestamps are in UTC timezone and returned as datetime objects, which ensures:

* Proper type handling in downstream databases
* Automatic type conversion in Generic IO sink
* Consistent timezone handling across systems

## Limitations

* **SQL Server 2008+**: Requires SQL Server 2008 or later (Enterprise, Developer, or Standard edition)
* **CDC Feature**: Requires CDC to be enabled on database and tables
* **SQL Server Agent**: Requires SQL Server Agent to be running
* **Network Dependency**: Requires stable network connection
* **Memory Usage**: Schema caching uses memory
* **CDC Retention**: Depends on SQL Server CDC retention settings (default: 3 days)
* **Timezone**: All timestamps are in UTC timezone
* **Polling**: Uses polling mechanism (not push-based like logical replication)
* **UPDATE Events**: Only includes after image, not before image
* **ODBC Driver**: Requires appropriate ODBC driver to be installed

## LSN Format

SQL Server CDC uses Log Sequence Numbers (LSN) in binary format, which are converted to hex strings for storage and checkpointing:

* **Format**: `0x` followed by hexadecimal digits (e.g., `0x00000000000000000000`)
* **Length**: Typically 10 bytes (20 hex characters)
* **Comparison**: LSNs can be compared as integers for range queries
* **Checkpoint**: Last processed LSN is saved in checkpoint file for resume capability

## Capture Instance Names

When CDC is enabled on a table, SQL Server automatically creates a capture instance. The naming convention is:

* **Format**: `{schema}_{table}_CT`
* **Example**: `dbo_users_CT` for table `users` in schema `dbo`

The source automatically discovers capture instances, but you can also specify `capture_instance` in the configuration if needed.


# MySQL CDC
Source: https://docs.mage.ai/guides/streaming/sources/mysql-cdc

Real-time Change Data Capture from MySQL binary logs with schema evolution tracking

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mysql-cdc)

## Overview

The MySQL CDC streaming source captures real-time changes from MySQL binary logs and processes them into structured records. It provides comprehensive CDC capabilities with automatic schema evolution tracking and primary key detection.

> **Real-time CDC**
>
> Captures changes as they happen in MySQL

> **Schema Evolution**
>
> Tracks table structure changes automatically

> **Primary Key Detection**
>
> Automatically identifies and extracts primary key columns

> **Error Recovery**
>
> Handles server ID conflicts and binlog file issues

## Configuration

### Required Parameters

| Parameter | Type | Description |
| - | - | - |
| `host` | string | MySQL server hostname or IP address |
| `port` | number | MySQL server port (default: 3306) |
| `user` | string | MySQL username with replication privileges |
| `password` | string | MySQL password |

### Optional Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `connect_timeout` | number | 10 | Connection timeout in seconds |
| `charset` | string | utf8mb4 | Character set for connection |
| `databases` | array | - | Include only these databases |
| `tables` | array | - | Include only these tables |
| `ignore_databases` | array | - | Exclude these databases |
| `ignore_tables` | array | - | Exclude these tables |
| `server_id` | number | 54001 | Unique server ID for replication |
| `auto_generate_server_id` | boolean | true | Auto-generate unique server\_id if conflict |
| `resume_from_gtids` | string | - | Resume from specific GTID set |
| `start_log_file` | string | - | Resume from specific binlog file |
| `start_log_pos` | number | - | Resume from specific position in binlog file |
| `start_timestamp` | number | - | Resume from specific timestamp (Unix timestamp) |
| `include_ddl` | boolean | false | Include DDL events (CREATE, ALTER, DROP, etc.) |
| `include_heartbeat` | boolean | true | Include heartbeat events |
| `include_transaction_events` | boolean | true | Include transaction commit events |
| `include_gtid_events` | boolean | true | Include GTID events |
| `include_rotate_events` | boolean | false | Include binlog rotation events |
| `include_intvar_events` | boolean | true | Include integer variable events |
| `include_load_query_events` | boolean | true | Include LOAD DATA INFILE events |
| `include_table_map_events` | boolean | true | Include table map events (required for column mapping) |
| `heartbeat_seconds` | number | 1.0 | Heartbeat interval in seconds |
| `blocking` | boolean | true | Use blocking mode for event reading |
| `max_batch_size` | number | 100 | Maximum events per batch |
| `flush_interval_seconds` | number | 1.0 | Maximum time between batch flushes |
| `return_db_records_only` | boolean | true | Return only DB records with Mage timestamp columns |

### SSL Configuration

| Parameter | Type | Description |
| - | - | - |
| `ssl.ca` | string | Certificate Authority file path |
| `ssl.cert` | string | Client certificate file path |
| `ssl.key` | string | Client private key file path |

## Record Formats

### Standard Event Format

When `return_db_records_only: false`:

```json theme={"system"}
{
  "op": "c",
  "schema": "mydb",
  "table": "users",
  "ts": 1640995200,
  "tx": {
    "gtid": "uuid:1-100",
    "xid": 12345
  },
  "before": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "after": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "offset": {
    "file": "mysql-bin.000001",
    "pos": 1234,
    "gtid": "uuid:1-100"
  }
}
```

### DB Records Only Format

When `return_db_records_only: true`:

```json theme={"system"}
{
  "data": {
    "id": 1,
    "name": "John",
    "email": "john@example.com",
    "_mage_created_at": "2021-12-31T23:00:00+00:00",
    "_mage_updated_at": null,
    "_mage_deleted_at": null
  },
  "metadata": {
    "schema": "mydb",
    "table": "users",
    "key_columns": ["id"],
    "operation": "INSERT"
  }
}
```

**Note**: The `_mage_*` timestamp columns are returned as datetime objects (ISO 8601 format strings when serialized) in UTC timezone, not Unix timestamps. This ensures proper type handling when writing to databases.

## Event Types

> **Row Events (Core CDC)**
>
> * **WriteRowsEvent** (`op: "c"`): INSERT operations
> * **UpdateRowsEvent** (`op: "u"`): UPDATE operations with before/after data
> * **DeleteRowsEvent** (`op: "d"`): DELETE operations

> **DDL Events**
>
> * **QueryEvent** (`op: "ddl"`): DDL statements (CREATE, ALTER, DROP)
> * Automatically clears table schema cache when DDL occurs
> * Extracts table name from SQL for targeted cache clearing

> **Transaction Events**
>
> * **XidEvent** (`op: "transaction"`): Transaction commit events
> * Useful for maintaining transaction boundaries

> **Replication Events**
>
> * **GtidEvent** (`op: "gtid"`): Global Transaction Identifier events
> * **RotateEvent** (`op: "rotate"`): Binlog file rotation events
> * **HeartbeatLogEvent** (`op: "heartbeat"`): Replication heartbeat events

> **Metadata Events**
>
> * **TableMapEvent** (`op: "table_map"`): Table structure metadata
> * **IntvarEvent** (`op: "intvar"`): Integer variable changes
> * **LoadQueryEvent** (`op: "begin_load_query"`/`op: "execute_load_query"`): LOAD DATA INFILE events

## Primary Key Detection

The source automatically detects and extracts primary key columns:

1. **Schema Discovery**: Queries `INFORMATION_SCHEMA` for table structure
2. **Primary Key Detection**: Identifies actual primary key columns from database constraints
3. **Caching**: Stores schema and primary key information for performance
4. **Key Extraction**: Extracts primary key column names from row events

```json Example Output theme={"system"}
{
  "metadata": {
    "key_columns": ["user_id", "tenant_id"]
  }
}
```

**Note**: `key_columns` is a list of column names (not values) that can be used for deduplication or upsert operations in downstream sinks.

## Examples

```yaml Basic Setup theme={"system"}
connector_type: mysql
host: localhost
port: 3306
user: repl_user
password: password
databases: ["mydb"]
return_db_records_only: true
```

```yaml Production Setup theme={"system"}
connector_type: mysql
host: mysql-prod.company.com
port: 3306
user: cdc_user
password: secure_password
ssl:
  ca: "/etc/ssl/mysql/ca.pem"
  cert: "/etc/ssl/mysql/client-cert.pem"
  key: "/etc/ssl/mysql/client-key.pem"
databases: ["production_db"]
ignore_tables: ["temp_*", "log_*"]
server_id: 1001
auto_generate_server_id: false
max_batch_size: 500
flush_interval_seconds: 2.0
include_ddl: true
return_db_records_only: true
```

```yaml Resume from Position theme={"system"}
connector_type: mysql
host: localhost
port: 3306
user: repl_user
password: password
start_log_file: "mysql-bin.000123"
start_log_pos: 456789
databases: ["mydb"]
return_db_records_only: true
```

## Prerequisites

### MySQL Server Configuration

```sql theme={"system"}
-- Enable binary logging
SET GLOBAL log_bin = ON;

-- Set binlog format to ROW
SET GLOBAL binlog_format = 'ROW';

-- Enable GTID (recommended)
SET GLOBAL gtid_mode = ON;
SET GLOBAL enforce_gtid_consistency = ON;
```

### User Permissions

```sql theme={"system"}
-- Create replication user
CREATE USER 'repl_user'@'%' IDENTIFIED BY 'password';

-- Grant replication privileges
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl_user'@'%';

-- Grant SELECT privileges for schema queries
GRANT SELECT ON information_schema.* TO 'repl_user'@'%';
GRANT SELECT ON mydb.* TO 'repl_user'@'%';
```

## Troubleshooting

> **Common Issues**
>
> 1. **Server ID Conflict**: Enable `auto_generate_server_id: true`
> 2. **Binlog File Not Found**: Check if binlog file exists and is accessible
> 3. **Permission Denied**: Ensure user has replication privileges
> 4. **Connection Timeout**: Increase `connect_timeout` value
> 5. **Schema Not Found**: Ensure `include_table_map_events: true`

> **Debugging**
>
> Enable debug logging to see detailed event processing:
>
> ```python theme={"system"}
> import logging
> logging.basicConfig(level=logging.DEBUG)
> ```

> **Monitoring**
>
> Monitor key metrics:
>
> * **Batch Size**: Average events per batch
> * **Flush Rate**: How often batches are flushed
> * **Error Rate**: Failed events or connections
> * **Lag**: Time between event occurrence and processing

## Integration with Generic IO Sink

The MySQL CDC source works seamlessly with the Generic IO sink, which provides:

* **Automatic Column Type Mapping**: Automatically maps `_mage_*` timestamp columns to appropriate database types (TIMESTAMP, DATETIME2, DateTime64, etc.) based on the target database
* **Metadata Interpolation**: Use metadata values from MySQL CDC events in sink configurations for dynamic routing and table naming

### Supported Databases

Generic IO Sink supports the following databases:

* **BigQuery**
* **ClickHouse**
* **DuckDB**
* **MySQL**
* **MSSQL**
* **Postgres**

### Metadata Interpolation

You can use metadata values from MySQL CDC events in your sink configuration using Python string formatting syntax:

* **`{schema}`**: Database/schema name from the event
* **`{table}`**: Table name from the event
* **`{key_columns}`**: List of primary key column names (e.g., `["id"]` or `["user_id", "tenant_id"]`)

When using `{key_columns}` in `unique_constraints`, it will be automatically converted from a string representation to a list. The format supports both Python-style (`"['id']"`) and JSON-style (`'["id"]'`) array strings.

### Example Configurations

```yaml BigQuery Example theme={"system"}
connector_type: bigquery
profile: default
config:
  table_id: "{schema}.{table}"  # Dynamic table names from metadata
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"  # Uses primary keys from metadata
```

```yaml Postgres Example theme={"system"}
connector_type: postgres
profile: default
config:
  schema_name: "{schema}"  # Dynamic schema from metadata
  table_name: "{table}"    # Dynamic table name from metadata
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"  # Uses primary keys from metadata
  if_exists: append
```

```yaml MySQL Example theme={"system"}
connector_type: mysql
profile: default
config:
  schema_name: "{schema}"
  table_name: "{table}"
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"
  if_exists: append
```

```yaml Static Configuration theme={"system"}
connector_type: postgres
profile: default
config:
  schema_name: public
  table_name: cdc_events  # Static table name
  unique_constraints: ['id', 'event_id']  # Static list (string format also works: "['id', 'event_id']")
  unique_conflict_method: UPDATE
  if_exists: append
```

### How Metadata Interpolation Works

1. **Message Grouping**: Messages are automatically grouped by their interpolated config values. For example, if you use `table_name: "{schema}_{table}"`, messages from `mydb.users` will be grouped together and written to `mydb_users` table.

2. **Key Columns Interpolation**: When using `{key_columns}` in `unique_constraints`, the sink automatically:
   * Converts the list to a string representation during interpolation
   * Parses it back to a list (supports both `"['id']"` and `'["id"]'` formats)
   * Uses it for upsert operations based on `unique_conflict_method`

**Example**: If you have a table `users` with primary key `id`, and you configure `unique_constraints: "{key_columns}"`, it will automatically use `["id"]` for upsert operations.

## Best Practices

1. **Use GTID**: Enable GTID for reliable resume capabilities
2. **Filter Events**: Disable unnecessary event types for performance
3. **Monitor Resources**: Watch memory and CPU usage
4. **Test Resume**: Verify checkpoint functionality
5. **Secure Connections**: Use SSL in production
6. **Regular Backups**: Backup checkpoint files
7. **Schema Validation**: Test with schema changes
8. **Use Generic IO Sink**: Leverage automatic type mapping and metadata interpolation for flexible data routing
9. **Metadata Interpolation**: Use `{schema}`, `{table}`, and `{key_columns}` for dynamic table routing and upsert configuration

## Timestamp Handling

The MySQL CDC source automatically converts Unix timestamps to datetime objects:

* **`_mage_created_at`**: Set to event timestamp (datetime) for INSERT operations
* **`_mage_updated_at`**: Set to event timestamp (datetime) for UPDATE operations
* **`_mage_deleted_at`**: Set to event timestamp (datetime) for DELETE operations

All timestamps are in UTC timezone and returned as datetime objects, which ensures:

* Proper type handling in downstream databases
* Automatic type conversion in Generic IO sink
* Consistent timezone handling across systems

## Limitations

* **MySQL 5.7+**: Requires MySQL 5.7 or later
* **ROW Format**: Requires `binlog_format = ROW`
* **Network Dependency**: Requires stable network connection
* **Memory Usage**: Schema caching uses memory
* **Binlog Retention**: Depends on MySQL binlog retention settings
* **Timezone**: All timestamps are in UTC timezone


# NATS.io - JetStream
Source: https://docs.mage.ai/guides/streaming/sources/nats



## Basic config

```yaml theme={"system"}
connector_type: 'nats'
server_url: 'nats://localhost:4222'
subject: 'test_subject'
```


# OracleDB
Source: https://docs.mage.ai/guides/streaming/sources/oracledb

Stream changes from OracleDB with OracleDB CDC.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=oracledb)

Mage Pro streaming pipeline supports OracleDB CDC as the source. This document explains how to set up OracleDB CDC and integrate it into Mage's streaming pipeline using LogMiner.

## Basic Config

```yaml theme={"system"}
connector_type: oracledb
host: "oracledb.example.com"
port: 1521
service: "xepdb1"
password: "password"
user: "user"
mode: "thin"        # Value can be `thin` or `thick`
table_names: []     # Filter table names
```

## Setting Up OracleDB for CDC

### **Step 1: Enable Supplemental Logging**

Run the following SQL commands to enable CDC:

```sql theme={"system"}
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
```

### **Step 2: Create a User for CDC**

```sql theme={"system"}
CREATE USER cdc_user IDENTIFIED BY YourPassword;
GRANT CONNECT, RESOURCE TO cdc_user;
GRANT EXECUTE ON DBMS_LOGMNR TO cdc_user;
GRANT EXECUTE ON DBMS_LOGMNR_D TO cdc_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO cdc_user;
GRANT SELECT ON V_$DATABASE TO cdc_user;
GRANT SELECT ON V_$ARCHIVED_LOG TO cdc_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO cdc_user;
GRANT SELECT ON V_$LOGMNR_PARAMETERS TO cdc_user;
GRANT SELECT ANY TABLE TO cdc_user;
```

### **Step 3: Start LogMiner Session**

```sql theme={"system"}
BEGIN
  DBMS_LOGMNR.START_LOGMNR(
    OPTIONS => DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG + DBMS_LOGMNR.NEW
  );
END;
/
```

***

## Test using LogMiner for CDC

LogMiner reads redo logs to track **INSERT, UPDATE, DELETE** operations.

### **Mage Data Pipeline for LogMiner-Based CDC**

Create a data loader in Mage to pull real-time CDC data:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.oracledb import OracleDB
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_oracledb(*args, **kwargs):
    query = """
    SELECT SCN, OPERATION, TIMESTAMP, SQL_REDO 
    FROM V$LOGMNR_CONTENTS 
    WHERE SEG_NAME = 'YOUR_TABLE' 
      AND OPERATION IN ('INSERT', 'UPDATE', 'DELETE')
    """
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with OracleDB.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```


# PostgreSQL CDC
Source: https://docs.mage.ai/guides/streaming/sources/postgres-cdc

Real-time Change Data Capture from PostgreSQL WAL with logical replication

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=postgres-cdc)

## Overview

The PostgreSQL CDC streaming source captures real-time changes from PostgreSQL Write-Ahead Log (WAL) using logical replication (pgoutput) and processes them into structured records. It provides comprehensive CDC capabilities with automatic schema evolution tracking and primary key detection.

> **Real-time CDC**
>
> Captures changes as they happen in PostgreSQL

> **Logical Replication**
>
> Uses PostgreSQL's native logical replication protocol

> **Primary Key Detection**
>
> Automatically identifies and extracts primary key columns

> **LSN Tracking**
>
> Tracks Log Sequence Numbers for reliable resume capabilities

## Configuration

### Required Parameters

| Parameter | Type | Description |
| - | - | - |
| `host` | string | PostgreSQL server hostname or IP address |
| `port` | number | PostgreSQL server port (default: 5432) |
| `database` | string | Database name (default: postgres) |
| `user` | string | PostgreSQL username with replication privileges |
| `password` | string | PostgreSQL password |

### Optional Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `connect_timeout` | number | 10 | Connection timeout in seconds |
| `schemas` | array | - | Include only these schemas |
| `tables` | array | - | Include only these tables |
| `ignore_schemas` | array | - | Exclude these schemas |
| `ignore_tables` | array | - | Exclude these tables |
| `replication_slot` | string | mage\_slot | Logical replication slot name |
| `publication_name` | string | mage\_pub | Publication name for logical replication |
| `start_lsn` | string | - | Log Sequence Number to start from (e.g., '0/1234567') |
| `max_batch_size` | number | 100 | Maximum events per batch |
| `flush_interval_seconds` | number | 1.0 | Maximum time between batch flushes |
| `keep_alive_time` | number | 10.0 | Time between keep-alive messages in seconds |
| `heartbeat_table_schema` | string | - | Schema name for heartbeat table (e.g., 'public') |
| `heartbeat_table_name` | string | - | Table name for heartbeat (e.g., 'cdc\_status') |
| `heartbeat_interval_seconds` | number | 60.0 | How often to write heartbeat in seconds |
| `heartbeat_update_query` | string | - | Custom SQL query to update heartbeat (overrides default). **WARNING**: Query is executed directly without sanitization. Only use queries from trusted sources to prevent SQL injection vulnerabilities. |
| `return_db_records_only` | boolean | true | Return only DB records with Mage timestamp columns |

### SSL Configuration

| Parameter | Type | Description |
| - | - | - |
| `ssl.sslmode` | string | SSL mode (e.g., 'require', 'prefer', 'verify-ca', 'verify-full') |
| `ssl.sslcert` | string | Client certificate file path |
| `ssl.sslkey` | string | Client private key file path |
| `ssl.sslrootcert` | string | Root certificate file path |

## Record Formats

### DB Records Only Format

When `return_db_records_only: true` (default):

```json theme={"system"}
{
  "data": {
    "id": 1,
    "name": "John",
    "email": "john@example.com",
    "_mage_created_at": "2021-12-31T23:00:00+00:00",
    "_mage_updated_at": null,
    "_mage_deleted_at": null
  },
  "metadata": {
    "schema": "public",
    "table": "users",
    "key_columns": ["id"],
    "operation": "INSERT",
    "lsn": "0/1234567"
  }
}
```

**Note**: The `_mage_*` timestamp columns are returned as datetime objects (ISO 8601 format strings when serialized) in UTC timezone, not Unix timestamps. This ensures proper type handling when writing to databases.

## Event Types

> **Row Events (Core CDC)**
>
> * **Insert** (`operation: "INSERT"`): INSERT operations
> * **Update** (`operation: "UPDATE"`): UPDATE operations with before/after data
> * **Delete** (`operation: "DELETE"`): DELETE operations
>   These events contain the actual data changes.

> **Schema Events**
>
> * **Relation** (`operation: "RELATION"`): Table structure metadata
> * **Truncate** (`operation: "TRUNCATE"`): TRUNCATE TABLE operations
>   Automatically tracks table schema changes for column name mapping.

> **Transaction Events**
>
> * **Begin** (`operation: "BEGIN"`): Transaction begin events
> * **Commit** (`operation: "COMMIT"`): Transaction commit events
>   Useful for maintaining transaction boundaries and ensuring data consistency.

## Primary Key Detection

The source automatically detects and extracts primary key columns:

1. **Schema Discovery**: Queries `INFORMATION_SCHEMA` for table structure
2. **Primary Key Detection**: Identifies actual primary key columns from database constraints
3. **Caching**: Stores schema and primary key information for performance
4. **Key Extraction**: Extracts primary key column names from row events

```json Example Output theme={"system"}
{
  "metadata": {
    "key_columns": ["user_id", "tenant_id"]
  }
}
```

**Note**: `key_columns` is a list of column names (not values) that can be used for deduplication or upsert operations in downstream sinks.

## Examples

```yaml Basic Setup theme={"system"}
connector_type: postgres
host: localhost
port: 5432
database: mydb
user: repl_user
password: password
schemas: ["public"]
return_db_records_only: true
```

```yaml Production Setup theme={"system"}
connector_type: postgres
host: postgres-prod.company.com
port: 5432
database: production_db
user: cdc_user
password: secure_password
ssl:
  sslmode: require
  sslcert: "/etc/ssl/postgres/client-cert.pem"
  sslkey: "/etc/ssl/postgres/client-key.pem"
  sslrootcert: "/etc/ssl/postgres/ca.pem"
schemas: ["public", "analytics"]
ignore_tables: ["temp_*", "log_*"]
replication_slot: prod_cdc_slot
publication_name: prod_cdc_pub
max_batch_size: 500
flush_interval_seconds: 2.0
return_db_records_only: true
```

```yaml Resume from LSN theme={"system"}
connector_type: postgres
host: localhost
port: 5432
database: mydb
user: repl_user
password: password
start_lsn: "0/1234567"
schemas: ["public"]
return_db_records_only: true
```

```yaml With Heartbeat Table theme={"system"}
connector_type: postgres
host: aiven-postgres.company.com
port: 5432
database: mydb
user: cdc_user
password: password
heartbeat_table_schema: public
heartbeat_table_name: cdc_status
heartbeat_interval_seconds: 60.0
schemas: ["public"]
return_db_records_only: true
```

```yaml With Custom Heartbeat Query theme={"system"}
connector_type: postgres
host: aiven-postgres.company.com
port: 5432
database: mydb
user: cdc_user
password: password
heartbeat_table_schema: public
heartbeat_table_name: cdc_status
heartbeat_interval_seconds: 60.0
heartbeat_update_query: "UPDATE public.cdc_status SET timestamp = NOW(), last_updated_by = 'mage_cdc' WHERE id = 1"
schemas: ["public"]
return_db_records_only: true
```

## Heartbeat Table

The heartbeat table is used to prevent WAL logs from filling up in managed PostgreSQL services (like Aiven). When configured, the source periodically writes to this table to keep the replication slot active.

### Schema

The heartbeat table is automatically created if it doesn't exist. It has the following schema:

```sql theme={"system"}
CREATE TABLE {schema}.{table} (
    id INTEGER PRIMARY KEY DEFAULT 1,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

**Columns:**

* `id` (INTEGER PRIMARY KEY): Primary key column with default value of 1
* `timestamp` (TIMESTAMP): Timestamp column that gets updated with the current timestamp on each heartbeat write

### How It Works

1. **Automatic Creation**: If the table doesn't exist, it's automatically created when the source initializes
2. **Periodic Updates**: The source updates the `timestamp` column every `heartbeat_interval_seconds` (default: 60 seconds)
3. **Default UPSERT Pattern**: Uses `INSERT ... ON CONFLICT DO UPDATE` to update the single row with `id=1`, or falls back to `UPDATE`/`INSERT` if the database doesn't support `ON CONFLICT`
4. **Custom Query**: You can override the default heartbeat update query by providing `heartbeat_update_query` in the configuration

### Security Warning for Custom Heartbeat Query

> [!WARNING]
>
> **Important Security Notice**: The `heartbeat_update_query` parameter executes SQL queries directly without sanitization or parameterization. This could introduce SQL injection vulnerabilities if the query is sourced from untrusted input.
>
> **Best Practices:**
>
> * Only use queries from trusted, validated sources
> * Never construct queries dynamically from user input or external APIs
> * Validate that the query is not empty or whitespace-only before use
> * Consider using the default heartbeat mechanism unless you have specific requirements
> * Review and test custom queries thoroughly before deploying to production
>
> The source performs basic validation to ensure the query is not empty or whitespace-only, but does not perform SQL injection protection. Always ensure your configuration management system properly secures and validates this parameter.

### Manual Creation

If you prefer to create the table manually, you can use:

```sql theme={"system"}
CREATE TABLE public.cdc_status (
    id INTEGER PRIMARY KEY DEFAULT 1,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Insert initial row
INSERT INTO public.cdc_status (id, timestamp)
VALUES (1, CURRENT_TIMESTAMP);
```

**Note**: The table must have a primary key or unique constraint on the `id` column for the UPSERT pattern to work correctly.

## Prerequisites

### PostgreSQL Server Configuration

```sql theme={"system"}
-- Enable logical replication
ALTER SYSTEM SET wal_level = logical;
-- Restart PostgreSQL for changes to take effect

-- Verify WAL level
SHOW wal_level;
-- Should return: logical
```

### Create Replication Slot

```sql theme={"system"}
-- Create logical replication slot
SELECT pg_create_logical_replication_slot('mage_slot', 'pgoutput');
```

### Create Publication

```sql theme={"system"}
-- Create publication for all tables in a schema
CREATE PUBLICATION mage_pub FOR ALL TABLES;

-- Or create publication for specific tables
CREATE PUBLICATION mage_pub FOR TABLE users, orders, products;

-- Or create publication for specific schemas
CREATE PUBLICATION mage_pub FOR TABLES IN SCHEMA public, analytics;
```

### User Permissions

```sql theme={"system"}
-- Create replication user
CREATE USER repl_user WITH PASSWORD 'password';

-- Grant replication privileges
ALTER USER repl_user WITH REPLICATION;

-- Grant connect privileges
GRANT CONNECT ON DATABASE mydb TO repl_user;

-- Grant usage on schemas
GRANT USAGE ON SCHEMA public TO repl_user;
GRANT USAGE ON SCHEMA analytics TO repl_user;

-- Grant SELECT privileges for schema queries
GRANT SELECT ON ALL TABLES IN SCHEMA public TO repl_user;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO repl_user;

-- Grant SELECT on information_schema for metadata queries
GRANT SELECT ON ALL TABLES IN SCHEMA information_schema TO repl_user;
```

## Troubleshooting

> **Common Issues**
>
> 1. **Replication Slot Not Found**: Create the replication slot using `pg_create_logical_replication_slot()`
> 2. **Publication Not Found**: Create the publication using `CREATE PUBLICATION`
> 3. **Permission Denied**: Ensure user has `REPLICATION` privilege and `SELECT` on tables
> 4. **WAL Level Not Logical**: Set `wal_level = logical` and restart PostgreSQL
> 5. **Connection Timeout**: Increase `connect_timeout` value
> 6. **LSN Format Error**: Ensure LSN is in format 'X/Y' (e.g., '0/1234567')

> **Debugging**
>
> Check replication slot status:
>
> ```sql theme={"system"}
> SELECT * FROM pg_replication_slots WHERE slot_name = 'mage_slot';
> ```
>
> Check publication status:
>
> ```sql theme={"system"}
> SELECT * FROM pg_publication WHERE pubname = 'mage_pub';
> SELECT * FROM pg_publication_tables WHERE pubname = 'mage_pub';
> ```

> **Monitoring**
>
> Monitor key metrics:
>
> * **Batch Size**: Average events per batch
> * **Flush Rate**: How often batches are flushed
> * **Error Rate**: Failed events or connections
> * **Lag**: Time between event occurrence and processing
> * **LSN Progress**: Track Log Sequence Number progress
>
> Check replication lag:
>
> ```sql theme={"system"}
> SELECT 
>     slot_name,
>     pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS replication_lag
> FROM pg_replication_slots
> WHERE slot_name = 'mage_slot';
> ```

## Integration with Generic IO Sink

The PostgreSQL CDC source works seamlessly with the Generic IO sink, which provides:

* **Automatic Column Type Mapping**: Automatically maps `_mage_*` timestamp columns to appropriate database types (TIMESTAMP, DATETIME2, DateTime64, etc.) based on the target database
* **Metadata Interpolation**: Use metadata values from PostgreSQL CDC events in sink configurations for dynamic routing and table naming

### Supported Databases

Generic IO Sink supports the following databases:

* **BigQuery**
* **ClickHouse**
* **DuckDB**
* **MySQL**
* **MSSQL**
* **Postgres**

### Metadata Interpolation

You can use metadata values from PostgreSQL CDC events in your sink configuration using Python string formatting syntax:

* **`{schema}`**: Schema name from the event
* **`{table}`**: Table name from the event
* **`{key_columns}`**: List of primary key column names (e.g., `["id"]` or `["user_id", "tenant_id"]`)

When using `{key_columns}` in `unique_constraints`, it will be automatically converted from a string representation to a list. The format supports both Python-style (`"['id']"`) and JSON-style (`'["id"]'`) array strings.

### Example Configurations

```yaml BigQuery Example theme={"system"}
connector_type: bigquery
profile: default
config:
  table_id: "{schema}.{table}"  # Dynamic table names from metadata
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"  # Uses primary keys from metadata
```

```yaml Postgres Example theme={"system"}
connector_type: postgres
profile: default
config:
  schema_name: "{schema}"  # Dynamic schema from metadata
  table_name: "{table}"    # Dynamic table name from metadata
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"  # Uses primary keys from metadata
  if_exists: append
```

```yaml MySQL Example theme={"system"}
connector_type: mysql
profile: default
config:
  schema_name: "{schema}"
  table_name: "{table}"
  unique_conflict_method: UPDATE
  unique_constraints: "{key_columns}"
  if_exists: append
```

```yaml Static Configuration theme={"system"}
connector_type: postgres
profile: default
config:
  schema_name: public
  table_name: cdc_events  # Static table name
  unique_constraints: ['id', 'event_id']  # Static list (string format also works: "['id', 'event_id']")
  unique_conflict_method: UPDATE
  if_exists: append
```

### How Metadata Interpolation Works

1. **Message Grouping**: Messages are automatically grouped by their interpolated config values. For example, if you use `table_name: "{schema}_{table}"`, messages from `public.users` will be grouped together and written to `public_users` table.

2. **Key Columns Interpolation**: When using `{key_columns}` in `unique_constraints`, the sink automatically:
   * Converts the list to a string representation during interpolation
   * Parses it back to a list (supports both `"['id']"` and `'["id"]'` formats)
   * Uses it for upsert operations based on `unique_conflict_method`

**Example**: If you have a table `users` with primary key `id`, and you configure `unique_constraints: "{key_columns}"`, it will automatically use `["id"]` for upsert operations.

## Best Practices

1. **Use Replication Slots**: Always use replication slots for reliable CDC tracking
2. **Monitor WAL Growth**: Monitor WAL size and ensure proper cleanup
3. **Filter Events**: Use schema/table filters to reduce processing overhead
4. **Monitor Resources**: Watch memory and CPU usage
5. **Test Resume**: Verify checkpoint functionality with LSN tracking
6. **Secure Connections**: Use SSL in production
7. **Regular Backups**: Backup checkpoint files
8. **Schema Validation**: Test with schema changes
9. **Use Generic IO Sink**: Leverage automatic type mapping and metadata interpolation for flexible data routing
10. **Metadata Interpolation**: Use `{schema}`, `{table}`, and `{key_columns}` for dynamic table routing and upsert configuration
11. **Heartbeat Tables**: Use heartbeat tables for managed services (like Aiven) to prevent WAL log filling

## Timestamp Handling

The PostgreSQL CDC source automatically adds Mage timestamp columns:

* **`_mage_created_at`**: Set to event timestamp (datetime) for INSERT operations
* **`_mage_updated_at`**: Set to event timestamp (datetime) for UPDATE operations
* **`_mage_deleted_at`**: Set to event timestamp (datetime) for DELETE operations

All timestamps are in UTC timezone and returned as datetime objects, which ensures:

* Proper type handling in downstream databases
* Automatic type conversion in Generic IO sink
* Consistent timezone handling across systems

## Limitations

* **PostgreSQL 10+**: Requires PostgreSQL 10 or later for logical replication
* **WAL Level**: Requires `wal_level = logical`
* **Network Dependency**: Requires stable network connection
* **Memory Usage**: Schema caching uses memory
* **WAL Retention**: Depends on PostgreSQL WAL retention settings
* **Timezone**: All timestamps are in UTC timezone
* **Replication Slot**: Requires a dedicated replication slot per consumer
* **Publication**: Tables must be added to a publication to be replicated


# RabbitMQ
Source: https://docs.mage.ai/guides/streaming/sources/rabbitmq



## Basic config

```yaml theme={"system"}

connector_type: 'rabbitmq'
connection_host: 'localhost'
connection_port: 5672
queue_name: 'default'
username: 'guest'
password: 'guest'
amqp_url_virtual_host: '%2f'
url_protocol: 'amqp'
```

You can even control some of the RabbitMQ consumer configurations using the following fields.

```yaml theme={"system"}
configure_consume: True

consume_config:
  auto_ack: False
  exclusive: False
  inactivity_timeout: 10.0
```


# 🎏 Local stream to dbt transform
Source: https://docs.mage.ai/guides/streaming/tutorials/magic-devcontainer

This project uses sample data from the Google NYC Taxi Pubsub to create a streaming pipeline in Mage that reads and transforms a sample stream, ultimately writing output to a SCD Type-2 table in Postgres.

## 👋 Intro

This project uses sample data from [the Google NYC Taxi Pubsub](https://github.com/googlecodelabs/cloud-dataflow-nyc-taxi-tycoon#public-pubsub-data-stream) to create a streaming pipeline in Mage that:

* Reads and transforms the stream
* Writes the stream as parquet to S3

Then, it creates a batch pipeline to:

* Read parquet files from S3.
* Execute a dbt model to store the raw data in a local Postgres instance.
* Execute a dbt model to transform the raw data to an SCD Type-2 table, logging the status updates for each ride.

Read more or make updates to the repo [here](https://github.com/mage-ai/magic-devcontainer/tree/main).

![](https://github.com/mage-ai/magic-devcontainer/blob/main/assets/img/taxi-gif.gif?raw=True)

## 🛠️ Configuration

**Prerequisites**

* Docker
* VSCode
  * No VSCode? That's ok: you can use `docker compose` too
* An AWS account (or other cloud storage provider) to store the stream

Let's get started 🎉

1. `git clone https://github.com/mage-ai/pubsub-devcontainer-demo`
2. Open the cloned file in VSCode.
3. Create a `.env` file with the requisite variables. See the `.env.example` file for guidance.
4. Select the prompt to `reopen in Container` to start the devcontainer *or* open the command prompt and select `Devcontainers: Rebuild and Reopen in Container`. This will build a devcontainer (a Docker container) and install all dependencies to your project. This may take a few minutes.
   1. If you don't have VSCode installed, simply run `docker compose up` from the command line.
5. Navigate to `localhost:6789` when the container is ready. You should see the Mage UI!

## 🙋‍♂️ Wait, what's happening?

![](https://github.com/mage-ai/magic-devcontainer/blob/main/assets/magic-devcontainer.png?raw=True)

By performing the above, VSCode is creating an isolated environment, installing a few extensions + building and running Docker as defined in `docker-compose.yaml`.

We're using [Devcontainers](https://containers.dev/) to create a consistent development environment for our project. This means that we can be sure that our code will run in the same environment as our teammates and that we can easily share our code with others.

As for the stream itself, you might notice `kafka` and `zookeeper` in the `docker-compose.yaml` file. These are the tools we use to manage the stream.

We're using `kafka` to manage the stream and `zookeeper` to manage `kafka`. You can read more about these tools [here](https://kafka.apache.org/).

The *actual* stream is managed by the `stream` container, which spins up a python container and executes `./stream/bin/send_stream.py`— this is the script that reads from the pubsub topic data (`./stream/data/taxi-stream.json`).

Data will begin streaming 30 seconds after the container starts.

## 🤓 Stream data from a dummy source to S3

1. Navigate to the "Pipelines" tab in the Mage UI.
2. Two pipelines exist— `kafka_demo` and `dbt_demo`— click `kafka_demo` then the "code" icon in the top left to open the pipeline code.
3. Double check the config— these should pull from your `.env` file.
4. Click `Execute pipeline` in the bottom right to run your stream. You should see data flowing!

![](https://github.com/mage-ai/magic-devcontainer/blob/main/assets/img/mage-gif.gif?raw=True)

Nice! We've got a working stream. You can click `Cancel pipeline` after some sample data has been loaded or let the stream run.

Behind the scenes, we're kicking off a Kafka + Zookeeper instance, creating a topic, and writing data to that topic. Mage is then pulling in this "simulated" stream, transforming it, and writing to S3!

## 🧱 Run dbt transformations

This container builds a postgres database alongside our Mage app for us to run transformations in dbt. Navigate to the `pipelines` page and select `dbt demo`. Select the code icon.

* The first cell reads in the data from our stream (stored in s3 or your cloud provider of choice). Click the play icon or hit CMD + Enter to run.
* You might notice there are some tests in this cell— Mage incorporates *runtime tests* to check the quality of your data. What tests are we running here?
* The next cell writes our data to a postgres table using `dbt`. Note how we can read from our mage data *directly*, without the need for an intermediate table.
* Finally, we apply a transform to build an [SCD Type-2](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) table from our dataset. This is also known as a "change-log" format.
* The final cell pulls a sample ride! Do the pickup, enroute, & dropoff times lineup?


# Kafka streaming pipeline
Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-pipeline

Build pipelines that ingest data from event streaming sources like Kafka.

![Streaming pipeline](https://c.tenor.com/sezVCSw-n7sAAAAC/waterfall-stream.gif)

## Set up Kafka

If you don’t have Kafka already setup, here is a quick guide on how to run and
use Kafka locally:

### Using Kafka locally

1. In your terminal, clone this repository:
   `git clone https://github.com/wurstmeister/kafka-docker.git`.
2. Change directory into that repository: `cd kafka-docker`.
3. Edit the `docker-compose.yml` file to match this:
   ```yaml theme={"system"}
   version: "2"
   services:
     zookeeper:
       image: wurstmeister/zookeeper
       ports:
         - "2181:2181"
     kafka:
       build: .
       container_name: docker_kafka
       ports:
         - "9092:9092"
       expose:
         - "9093"
       environment:
         KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
         KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
         KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
         KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
         KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock
   ```
4. Start Docker:
   ```bash theme={"system"}
   docker-compose up
   ```
   If you encounter the error `command not found: docker-compose`,
   try running the following command instead:
   ```bash theme={"system"}
   docker compose up
   ```
5. Start a terminal session in the running container:
   ```bash theme={"system"}
   docker exec -i -t -u root $(docker ps | grep docker_kafka | cut -d' ' -f1) /bin/bash
   ```
6. Create a topic:
   ```bash theme={"system"}
   $KAFKA_HOME/bin/kafka-topics.sh --create --partitions 4 --bootstrap-server kafka:9092 --topic test
   ```
7. List all available topics in Kafka instance:
   ```bash theme={"system"}
   $KAFKA_HOME/bin/kafka-topics.sh --bootstrap-server kafka:9092 --list
   ```
8. Start a producer on topic named `test`:
   ```bash theme={"system"}
   $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic=test
   ```
9. Send messages to the topic named `test` by typing the following JSON strings in the
   terminal (Note that Kafka messages in Mage are assumed to be in JSON format):
   ```bash theme={"system"}
   >{ "hello": 1 }
   >{ "this is a test": 1 }
   >{ "test": 1 }
   >{ "test": 2 }
   >{ "test": 3 }
   ```
10. Open another terminal and start a consumer on the topic named `test`:

```bash theme={"system"}
$KAFKA_HOME/bin/kafka-console-consumer.sh --from-beginning --bootstrap-server kafka:9092 --topic=test
```

11. The output should look something like this:

```bash theme={"system"}
{ "hello": 1 }
{ "this is a test": 1 }
{ "test": 1 }
{ "test": 2 }
{ "test": 3 }
```

<sub>
  Original
  [source](https://towardsdatascience.com/introduction-to-kafka-stream-processing-in-python-e30d34bf3a12)
  of instructions.
</sub>

***

## Build streaming pipeline

### Start Mage

#### Using Kafka locally in a Docker container

Start Mage using Docker. Run the following command to run Docker in network
mode:

```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
  --env AWS_ACCESS_KEY_ID=your_access_key_id \
  --env AWS_SECRET_ACCESS_KEY=your_secret_access_key \
  --env AWS_REGION=your_region \
  --network kafka-docker_default \
  mageai/mageai /app/run_app.sh  mage start default_repo
```

> [!NOTE]
>
> Change the environment variables argument depending on your cloud provider.

If the network named `kafka-docker_default` doesn’t exist, create a new network:

```bash theme={"system"}
docker network create -d bridge kafka-docker_default
```

Check that it exists:

```bash theme={"system"}
docker network ls
```

***

If you can’t connect to Kafka locally in a Docker container using Mage in a
Docker container, do the following:

1. Clone Mage: `git clone https://github.com/mage-ai/mage-ai.git`.
2. Change directory into Mage: `cd mage-ai`.
3. Edit the `docker-compose.yml` file to match this:
   ```yaml theme={"system"}
   version: '3'
   services:
     server:
       ... (original config)
       networks:
         - kafka
     app:
       ... (original config)
   networks:
     kafka:
       name: kafka-docker_default
       external: true
   ```
4. Run the following script in your terminal: `./scripts/dev.sh`.

This will run Mage in development mode; which runs it in a Docker container
using `docker compose` instead of `docker run`.

#### Using Kafka without a Docker container

Start Mage using Docker. If you haven’t done this before, refer to the
[setup guide](/getting-started/setup).

### Create a new pipeline

1. Open Mage in your browser.

2. Click **`+ New pipeline`**, then select `Streaming`.

3. Add a data loader block, select `Kafka`, and paste the following:
   ```yaml theme={"system"}
   connector_type: kafka
   bootstrap_server: "localhost:9092"
   topic: test
   consumer_group: unique_consumer_group
   include_metadata: false
   api_version: 0.10.2
   batch_size: 100
   ```
   1. By default, the `bootstrap_server` is set to `localhost:9092`. If you’re
      running Mage in a docker container, the `bootstrap_server` should be
      `kafka:9093`.
   2. Messages are consumed from source in micro batch mode for better efficiency.
      The default batch size is 100. You can adjust the batch size in the source config.

4. Add a transformer block and paste the following:

   ```python theme={"system"}
   from typing import Dict, List

   if 'transformer' not in globals():
       from mage_ai.data_preparation.decorators import transformer


   @transformer
   def transform(messages: List[Dict], *args, **kwargs):
       for msg in messages:
           print(msg)

       return messages
   ```

5. Add a data exporter block, select `OpenSearch` and paste the following:
   ```yaml theme={"system"}
   connector_type: opensearch
   host: https://search-something-abcdefg123456.us-west-1.es.amazonaws.com/
   index_name: python-test-index
   ```
   1. Change the `host` to match your OpenSearch domain’s endpoint.
   2. Change the `index_name` to match the index you want to export data into.

***

## Test pipeline

Open the streaming pipeline you just created, and in the right side panel near
the bottom, click the button **`Execute pipeline`** to test the pipeline.

You should see an output like this:

```
[streaming_pipeline_test] [KafkaSource] Start initializing consumer.
[streaming_pipeline_test] [KafkaSource] Finish initializing consumer.
[streaming_pipeline_test] [KafkaSource] Test connection successfully.
[streaming_pipeline_test] [KafkaSource] Start consuming messages in batches.
```

### Publish messages using Python

1. Open a terminal on your local workstation.
2. Install `kafka-python`:
   ```bash theme={"system"}
   pip install kafka-python
   ```
3. Open a Python shell and write the following code to publish messages:

   ```python theme={"system"}
   from kafka import KafkaProducer
   from random import random
   import json


   topic = 'test'
   producer = KafkaProducer(
       bootstrap_servers='kafka:9093',
   )


   def publish_messages(limit):
       for i in range(limit):
           data = {
               'title': 'test_title',
               'director': 'Bennett Miller',
               'year': '2011',
               'rating': random(),
           }
           producer.send(topic, json.dumps(data).encode('utf-8'))

   publish_messages(5)
   ```

Once you run the code snippet above, go back to your streaming pipeline in Mage
and the output should look like this:

```
[streaming_pipeline_test] Start initializing kafka consumer.
[streaming_pipeline_test] Finish initializing kafka consumer.
[streaming_pipeline_test] Start consuming messages from kafka.
[streaming_pipeline_test] [Kafka] Receive message 2:16: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.7010424523477785}', time=1665618592.226788
[streaming_pipeline_test] [Kafka] Receive message 0:16: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.7886308380991354}', time=1665618592.2268753
[streaming_pipeline_test] [Kafka] Receive message 0:17: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.0673276352704153}', time=1665618592.2268832
[streaming_pipeline_test] [Kafka] Receive message 3:10: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.37935417366095525}', time=1665618592.2268872
[streaming_pipeline_test] [Kafka] Receive message 3:11: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.21110511524126563}', time=1665618592.2268918
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7010424523477785}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7886308380991354}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.0673276352704153}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.37935417366095525}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.21110511524126563}
[streaming_pipeline_test] [Opensearch] Batch ingest data [{'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7010424523477785}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7886308380991354}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.0673276352704153}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.37935417366095525}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.21110511524126563}], time=1665618592.2294626
```

### Consume messages using Python

If you want to programmatically consume messages from a Kafka topic, here is a
code snippet:

```python theme={"system"}
from kafka import KafkaConsumer
import time


topic = 'test'
consumer = KafkaConsumer(
    topic,
    group_id='test',
    bootstrap_servers='kafka:9093',
)

for message in consumer:
    print(f"{message.partition}:{message.offset}: v={message.value}, time={time.time()}")
```

***

## Run in production

1. [Create a trigger](/design/data-pipeline-management#create-trigger).
2. Once trigger is created, click the **`Start trigger`** button at the top
   of the page to make the streaming pipeline active.


# RabbitMQ streaming pipeline
Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-pipeline-rabbitmq

Build pipelines that ingest data from event streaming sources like RabbitMQ.

![Streaming pipeline](https://i.gifer.com/4Ze9.gif)

## Set up RabbitMQ

If you don't have RabbitMQ already setup, here is a quick guide on how to run and
use RabbitMQ locally

### Using RabbitMQ locally

1. In your terminal, execute this command:

   `docker run --rm -it -p 15672:15672 -p 5672:5672 rabbitmq:3-management`.

   (This should start the RabbitMQ server on port `5672` and a management UI on port `15672`)

### Create a Queue (via RabbitMQ management)

1. Log into your RabbitMQ management UI ( if you just created one, default user/password is guest/guest)

2. Go to "Queues" tab

3. Click on "Add a new queue", define your settings and click on "Add Queue"

4. Your queue should be ready to be used!

### Create a Queue (via python Pika)

1. Install pika in your favorite dependency manager:
   `pipenv install pika`

2. Execute this snippet:

   ```Python theme={"system"}
   import pika

   connection = pika.BlockingConnection(
       pika.ConnectionParameters(host=localhost,port=5672))

   # in case of failure connection, uncomment this code
   # username = guest
   # password = guest
   # connection_host = localhost
   # connection_port = 5672
   # vt_host = r'%2f'
   # generated_url = f"amqp://{username}:{password}@" \
   #                 f"{connection_host}:{connection_port}/{vt_host}"
   # create_connection = pika.BlockingConnection(
   #                 pika.URLParameters(
   #                     generated_url
   #                 )
   #             )

   channel = connection.channel()

   channel.queue_declare(queue='magic_queue', durable=True) # durable is optional

   ```

3. Your queue should be ready to be used!

## Start Mage

### Create a new pipeline

1. Open Mage in your browser.

2. Click **`+ New pipeline`**, then select `Streaming`.

3. Add a data loader block, select `RabbitMQ`, and paste the following:

   ```yaml theme={"system"}
   connector_type: 'rabbitmq'
   connection_host: localhost
   connection_port: 5672
   queue_name: default
   configure_consume: False
   username: 'guest'
   password: 'guest'
   amqp_url_virtual_host: '%2f'
   ```

4. Add a transformer block and paste the following:

   ```python theme={"system"}
   from typing import Dict, List
   import logging
   import json

   if 'transformer' not in globals():
       from mage_ai.data_preparation.decorators import transformer


   @transformer
   def transform(messages: List[Dict], *args, **kwargs):
       kwargs['channel'].basic_ack(messages[0].delivery_tag)
       print(messages.body.decode())
       return json.dumps({'message':messages.body.decode()})
   ```

5. Add a data exporter block, select `OpenSearch` and paste the following:

   ```yaml theme={"system"}
   connector_type: opensearch
   host: https://search-something-abcdefg123456.us-west-1.es.amazonaws.com/
   index_name: python-test-index
   ```

   1. Change the `host` to match your OpenSearch domain’s endpoint.
   2. Change the `index_name` to match the index you want to export data into.

***

## Test pipeline

Open the streaming pipeline you just created, and in the right side panel near
the bottom, click the button **Execute pipeline** to test the pipeline.

You should see an output like this:

```
[RabbitMQSource] Trying to connect on {your url}
[RabbitMQSource] Connected on broker
[RabbitMQSource] Start consuming messages.
```

### Publish messages using RabbitMQ Management UI

1. Log into your RabbitMQ management UI ( if you just created one, default user/password is guest/guest)

2. Go to "Queues" tab, and click on you desired Queue

3. Open "Publish Message" dropdown and write "Hello World!" on the payload block

4. Click on the black button called "Publish Message". This message should be seen on both mage-ai and your opensearch index!

### Publish messages using Python Pika

1. Execute this snippet:

   ```python theme={"system"}
   import json

   import pika

   connection = pika.BlockingConnection(
       pika.ConnectionParameters(host='insert_host',port=5672))

   # in case of failure connection, uncomment this code
   # username = guest
   # password = guest
   # connection_host = localhost
   # connection_port = 5672
   # vt_host = r'%2f'
   # generated_url = f"amqp://{username}:{password}@" \
   #                 f"{connection_host}:{connection_port}/{vt_host}"
   # create_connection = pika.BlockingConnection(
   #                 pika.URLParameters(
   #                     generated_url
   #                 )
   #             )

   channel = connection.channel()

   message = "Hello World!"

   channel.basic_publish(
     exchange='',
     routing_key='magic_queue',
     body=message,
     properties=pika.BasicProperties(
         delivery_mode=pika.spec.PERSISTENT_DELIVERY_MODE
     )
   )

   print(" [x] Sent ")
   ```

2. This message should be seen on both mage-ai and your OpenSearch index!


# Stateful Streaming Pipeline
Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-stateful-store

Create and use stateful store in a streaming pipeline.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=workspaces)

Stateful store now is supported in the streaming pipeline. You can access it through the
`state_store` object in the `kwargs` parameter. Redis is the only supported state store.

## Configuration

State store configuration is required to set up in metadata.yml in order to enable stateful store in a streaming pipeline.

```yaml theme={"system"}
state_store_config:
  type: 'REDIS'
  redis:
    redis_url: 'redis_url'
```

This configuration ensures that the pipeline can access and utilize Redis as its state store.

## Usage

Code example to access state store in your streaming pipeline:

```python theme={"system"}
    state_store = kwargs['state_store']
    state_store.set_state('string_key', 'Transformer Hello, Redis!')
    state_store.set_state('int_key', 123)
    state_store.set_state('list_key', [1, 2, 3,4, 5])
    state_store.set_state('dict_key', {'key1': 'value1', 'key2': 'value2'})
```

```python theme={"system"}
    state_store = kwargs['state_store']
    state_store.get_state('string_key')
    state_store.get_state('list_key')
    state_store.get_state('int_key')
    state_store.get_state('dict_key')
```

## Window Aggregation

The Window Aggregation Block is now supported in streaming pipeline, allowing for the aggregation of input data over a specified window of time and send it to down stream blocks.

### Usage

When you create a transformer and select the Window Aggregation type, it creates a window aggregation block with the following configuration:

```yaml theme={"system"}
transformer_type: window_aggregation
method: time
type: tumbling
interval_window: 60 # seconds
advance_window: 10
```

You can update `interval_window` with the desired interval.
In the meantime, you can add the following blocks after the window aggregation block.
These blocks will be triggered periodically based on the configuration.
For example, you can add a transformer block to report those data into monitoring system.
It can be used as alerts to your pipeline.

### How It Works

The window block is registered during the streaming pipeline execution with the specified configuration. An independent thread is created to manage each window block and trigger it periodically based on the configuration. When triggered, the downstream blocks are also executed accordingly.

***

🚀 **With the stateful store and window aggregation support, your streaming pipeline can now efficiently manage and process stateful data.**


# Mage tips & tricks
Source: https://docs.mage.ai/guides/tips-and-tricks

Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.

> **Version Control**
>
> Mage has native version control for your files, so you'll never lose work again!
> Check it out in our docs [here](/getting-started/file-versions) or see the video below for a tutorial.
>
> [YouTube video player](https://www.youtube.com/embed/bz8xLeoKzFo)

> **DuckDB integration**
>
> Mage integrates with DuckDB! Watch the video below to learn more or read on [here](/integrations/databases/DuckDB).
>
> [YouTube video player](https://www.youtube.com/embed/FmATfXoWudM)

> **Edit Pipelines with AI**
>
> Mage has an integration with the OpenAI API. Watch the video to learn about how you can generate pipelines using the power of AI.
>
> See our full docs [here](/development/ai/overview) for more info.
>
> [YouTube video player](https://www.youtube.com/embed/KNBkVhtflzU)

> **Stream data to any destination**
>
> Mage supports streaming data to any of our destinations. Watch the video below to learn about configuring a streaming pipeline in under a minute!
>
> See our full docs [here](/streaming/overview) for more info.
>
> [YouTube video player](https://www.youtube.com/embed/9STvcYIBXdc)

> **Use Conditional Blocks**
>
> Watch this video to learn how you can use Conditional Blocks to add conditional execution and branching logic to your next data pipeline!
>
> See our full docs [here](/design/blocks/conditionals) for more info.
>
> [YouTube video player](https://www.youtube.com/embed/D05C7atfCRQ)

> **Replicate Blocks**
>
> Watch this video to learn how you can use Replica Blocks to add conditional execution and branching logic to your next data pipeline!
>
> See our full docs [here](/guides/blocks/replicate-blocks) for more info.
>
> [YouTube video player](https://www.youtube.com/embed/M_bBCSTNDw4)

> **Use Split Pane View**
>
> The Mage pipeline editor can be used in a single, notebook-style interface *or* in a split-pane view, with block output displayed on the right and blocks themselves on the left.
>
> Split pane view can be enabled by selecting the "split" icon in the pipeline editor, as shown below:
>
> ![](https://mage-ai.github.io/assets/split-pane-view.png)
>
> Can't wait to try it out? Check out the video below for a demo!
>
> [YouTube video player](https://www.youtube.com/embed/x4VLyIG6fCc)

> **Group & Filter Pipelines**
>
> This week's Mage tip is all about organization!
>
> Did you know you can tag, group, and filter your pipelines from the overview page? Watch the video below to learn more 😀
>
> Read more in our docs [here](/pipelines/pipeline-tagging)!
>
> [YouTube video player](https://www.youtube.com/embed/QG88AWdB-Mo)

> **New Tree View**
>
> We've received a ton of feedback from our users, and we've been hard at work making updates to the tree view based on your suggestions. So, what's new? Let me give you the lowdown. 👇
>
> 🔹 Enhanced Functionality: The tree view now comes with a whole right-click menu that allows you to easily run, rename, insert, or delete blocks. You have more control than ever before!
>
> 🔹 Seamless Connections: You can now remove and add connections with a simple click. Plus, you can insert new blocks between existing ones effortlessly.
>
> 🔹 Enhanced Visualizations: With the new tree view, you get grouping and better lineage for certain block types 🔄
>
> Not only does the new tree view offer improved functionality, but it also makes your pipelines look absolutely stunning! After all, our goal is to make data beautiful. 💫
>
> [YouTube video player](https://www.youtube.com/embed/G0OjD38MnXE)

> **Dynamic Blocks**
>
> What are dynamic blocks? Think of them as the magical way to create tasks dynamically in our tool. If you're familiar with Airflow and the concept of Dynamic Task Mapping, this is pretty similar.
>
> If you have a bunch of small tasks that you want to execute in parallel, Dynamic Blocks are here to save the day!
>
> Dynamic Blocks in Mage give you the power to parallelize your workflow and reduce output effortlessly. It's a pattern that has been around in data engineering, but we've made it incredibly easy and efficient.
>
> Learn more [here](/design/blocks/dynamic-blocks).
>
> [YouTube video player](https://www.youtube.com/embed/jQZb3lrAb8w)

> **API Triggers**
>
> Did you know you can trigger pipelines directly with API calls in Mage? 🧙‍♂️💻
>
> With API Triggers, you can kick-off your ETL runs in just about any way possible— a Python request, a cURL command, or a well-timed carrier pigeon (hopefully not). 🐦‍⬛
>
> API triggers carry the added benefit of supporting runtime parameters, meaning you can trigger your pipelines and supply external variables to customize functionality/output.
>
> You can read more in our docs [here](/orchestration/triggers/trigger-pipeline-api) & get started today.
>
> [YouTube video player](https://www.youtube.com/embed/cHNTyGW51j8)

> **Backfills**
>
> Managing pipelines can be tricky, especially when it comes to backfilling missing data or re-running pipelines. But with Mage, backfills are a breeze! No need for custom scripts or complex setups. 🙌
>
> You can easily define a backfill for a specific date range. Whether you're missing data or want to add new columns, Mage has got you covered. Simply specify the start and end dates, and let Mage do the rest.
>
> Backfills simulate multiple runs with different execution dates, fetching the data you need and transforming it as required. 📅 🔃
>
> To learn more about backfills using Mage, check out our documentation [here](/orchestration/backfills/overview).
>
> [YouTube video player](https://www.youtube.com/embed/V-wUccaafEo)

## Run asynchronous code in blocks

```python theme={"system"}
import asyncio
from typing import Any, Dict

import aiohttp


@data_loader
def load_data_sync(*args, **kwargs) -> Dict[str, Any]:
    """
    Synchronously load data from a remote API endpoint using aiohttp and asyncio.
    Uses a configurable URL with a fallback to a working test endpoint.
    Includes error handling to gracefully manage connection issues.

    Returns:
        Dict[str, Any]: The JSON response from the API as a dictionary, or error info.
    """
    url: str = kwargs.get('url', 'https://jsonplaceholder.typicode.com/todos/1')

    async def fetch_data(url: str) -> Dict[str, Any]:
        # Create an aiohttp session and fetch the data
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                response.raise_for_status()
                data: Dict[str, Any] = await response.json()
                return data

    # Check if there's already a running event loop
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None

    if loop and loop.is_running():
        # If running in an existing event loop (e.g., Jupyter), use create_task and await
        result: Dict[str, Any] = loop.run_until_complete(fetch_data(url))
    else:
        # If no running event loop, use asyncio.run
        result: Dict[str, Any] = asyncio.run(fetch_data(url))
    return result
```


# Use a completed pipeline
Source: https://docs.mage.ai/guides/train/complete-project



## 1. Clone project

In your terminal, clone this [repository](https://github.com/mage-ai/demo_project)
by running the following command:

```bash theme={"system"}
git clone https://github.com/mage-ai/demo_project.git
```

## 2. Start the tool

**Docker**

```bash theme={"system"}
./scripts/start.sh demo_project
```

**pip**

```bash theme={"system"}
mage start demo_project
```

Open [http://localhost:6789](http://localhost:6789) in your browser.

On the left side in the file browser underneath the folder `demo_project/pipelines`,
click the pipeline named `titanic_survivors`.

## 3. Follow the quick start tutorial

Go back to the [quick start tutorial](/guides/train-model),
start from step **3. Play around with scratchpad**,
and use the existing code to follow along.


# File versioning and history
Source: https://docs.mage.ai/guides/version-control/file-versions

Local file edit tracking and version history for restoring changes made in the past.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=file-versioning)

Mage Pro includes built-in file versioning, eliminating the need to configure Git for rolling back changes or recovering previous file versions.

### How It Works

Every update to a file's contents automatically creates a new version that can be restored at any time. This applies to all files, including pipeline configurations.

### Restoring a Previous File Version

1. **Open the file** – In the pipeline editor, select the file you want to recover from the left hand file browser
2. **Access version history** – Click the "Version history" button in the top-left portion of the file browser

![Version history button](https://raw.githubusercontent.com/mage-ai/assets/main/pro/features/click-version-history.png)

3. **Compare versions** – Version history selects the most recent saved version by default. Choose another version from the newest-first list when needed. Use **Split** to show the current saved file in the **Current** pane on the left and the selected historical version in the **Version** pane on the right, or use **Inline** to review the changes in one editor. On narrow screens, the comparison automatically uses the inline layout.
4. **Restore** – Review the highlighted changes, then click the **Restore version** button to apply the selected version

To leave version history without restoring a version, click **Back to current file**.

Once restored, the selected version becomes the current active version of your file.

### Opening Version History from Activity

When an Activity audit event targets a file, Mage records the file path in the event details and shows readable file actions such as **Created file** and **Updated file**. URL-derived file paths are decoded for display, while literal percent sequences provided in request payload paths are preserved. You can search Activity with the full path or a filename segment to find the event, then click **Open versions** in the detail panel to open that file in Coder with version history visible.

You can also link directly to the same Coder view by adding both `paths[]` and `file_versions_path` to the URL:

```text theme={"system"}
/apps/coder?paths[]=/home/src/demo/pipelines/example.py&file_versions_path=/home/src/demo/pipelines/example.py
```

When Coder opens or reloads with `file_versions_path`, it loads the file, opens version history, and selects the most recent saved version automatically. The history panel preserves the normal newest-first version list; select another saved version to compare it with the current saved file or restore it.

Version history gives you the confidence to experiment and iterate freely without fear of losing work. If you make a mistake or need to reference previous logic, you can always roll back to any saved version.


# Airbyte in Mage
Source: https://docs.mage.ai/integrations/airbyte

Trigger a connection sync in Airbyte.

# Airbyte cloud

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=airbyte)

## Configuration

Mage Pro supports triggering **Airbyte Cloud** connection syncs directly from your data pipelines. To connect to Airbyte Cloud, you need to configure the following keyword arguments.

| Keyword argument | Description | Default value |
| - | - | - |
| `use_cloud` | Whether to use Airbyte Cloud | `False` |
| `client_id` | Airbyte cloud application client\_id | `None` |
| `client_secret` | Airbyte cloud application client\_secret | `None` |

Refer to the [Airbyte documentation](https://docs.airbyte.com/using-airbyte/configuring-api-access#step-1-create-an-application) to retrieve your `client_id` and `client_secret`.

## Example data loader

You can either explicitly hard code the `connection_id` in the data loader block
or you can add the value of the `connection_id`
as a [runtime variable](/getting-started/runtime-variable#creating-runtime-variables-in-editor).

from mage\_ai.services.airbyte import Airbyte

```python theme={"system"}
from mage_ai.services.airbyte import Airbyte


@data_loader
def load_data(*args, **kwargs):
    connection_id = kwargs['connection_id']
    client = Airbyte(
        use_cloud=True,
        client_id='client_id',
        client_secret='client_secret',
    )
    job = client.run_sync(connection_id, poll_interval=2)

    return job
```

# Open source

## Configuration

Here are the following keyword arguments that can be used to configure `Airbyte`:

| Keyword argument | Description | Default value |
| - | - | - |
| `api_version` | API version | `v1` |
| `host` | Airbyte server host | `host.docker.internal` |
| `password` | Password to log into Airbyte | `password` |
| `port` | Airbyte server port | `8000` |
| `use_ssl` | If `True`, then service will use HTTPS | `False` |
| `username` | Username to log into Airbyte | `airbyte` |

## Example code

```python theme={"system"}
from mage_ai.services.airbyte import Airbyte


client = Airbyte(
    api_version='v1',
    host='host.docker.internal',
    password='password',
    port=8000,
    use_ssl=False,
    username='airbyte',
)
job = client.run_sync('7a749f2f-74b4-492e-9d13-30a3f390d111', poll_interval=2)
```

Sample result:

```json theme={"system"}
{
  "connection_id": "7a749f2f-74b4-492e-9d13-30a3f390d111",
  "connection_status": "active",
  "job": {
    "id": 9,
    "configType": "sync",
    "configId": "7a749f2f-74b4-492e-9d13-30a3f390d111",
    "createdAt": 1671909838,
    "updatedAt": 1671909843,
    "status": "succeeded"
  }
}
```

## Example data loader

You can either explicitly hard code the `connection_id` in the data loader block
or you can add the value of the `connection_id`
as a [runtime variable](/getting-started/runtime-variable#creating-runtime-variables-in-editor).

```python theme={"system"}
from mage_ai.services.airbyte import Airbyte


@data_loader
def load_data(*args, **kwargs):
    connection_id = kwargs['connection_id']
    client = Airbyte(
        host='host.docker.internal',
        password='password',
        username='airbyte',
    )
    job = client.run_sync(connection_id, poll_interval=2)

    return job
```


# Run Mage pipelines in Airflow
Source: https://docs.mage.ai/integrations/airflow/index



Need help integrating Mage into an existing Airflow project?

Check out this [tutorial](/guides/integrate-mage-airflow) or
get instant help from us in [Slack](https://www.mage.ai/chat).

***

We support running the pipeline in Airflow DAGs.

* You need to firstly install `mage_ai` library by adding `mage_ai` to your requirements.txt file.
* Then you need to download the mage pipeline code into your Airflow directory. You can achieve it by using a git submodule in your Airflow directory.
* In your Mage project's metadata.yaml file, please specify a variables\_dir that you want to store the output of each block execution (You need to have write permission to the variables\_dir). Example: `variables_dir: /tmp`.

We provide multiple ways to run mage pipelines in Airflow.

* [Create DAGs for all the pipelines in Mage project](#create-dags-for-all-the-pipelines-in-mage-project)
* [Run pipeline in a BashOperator](#run-pipeline-in-a-bashoperator)
* [Run pipeline in a PythonOperator](#run-pipeline-in-a-pythonoperator)
* [Run pipeline as an Airflow DAG](#run-pipeline-as-an-airflow-dag)

## Create DAGs for all the pipelines in Mage project

```python theme={"system"}
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from mage_ai.orchestration.airflow import create_dags
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
create_dags(
    project_path,
    DAG,
    PythonOperator,
    blacklist_pipelines=[],     # Blacklisted pipeline uuids
    dag_settings=dict(
        start_date=datetime(2022, 8, 2),
    ),
    globals_dict=globals(),
)
```

## Run pipeline in a BashOperator

Example code:

```python theme={"system"}
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
dag = DAG(
    'test_mage_pipeline',
    start_date=datetime(2022, 7, 14),
    description='Test running mage pipeline.',
    schedule_interval='@once',
    catchup=False,
)
task = BashOperator(
    dag=dag,
    task_id='run_pipeline',
    bash_command=f'mage run {project_path} {pipeline_name}',
)
```

## Run pipeline in a PythonOperator

Example code:

```python theme={"system"}
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
import mage_ai
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
dag = DAG(
    'test_mage_pipeline',
    start_date=datetime(2022, 7, 14),
    description='Test running mage pipeline.',
    schedule_interval='@once',
    catchup=False,
)
def build_execute_pipeline(project_path, pipeline_name):
    def _callable(ds, **kwargs):
        mage_ai.run(pipeline_name, project_path)
    return _callable
task = PythonOperator(
    dag=dag,
    task_id='run_pipeline',
    python_callable=build_execute_pipeline(project_path, pipeline_name),
)
```

## Run pipeline as an Airflow DAG

Example code:

```python theme={"system"}
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
from mage_ai.data_preparation.models.pipeline import Pipeline
from mage_ai.shared.hash import ignore_keys
import mage_ai
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
dag = DAG(
    'test_mage_pipeline',
    start_date=datetime(2022, 7, 14),
    description='Test running mage pipeline.',
    schedule_interval='@once',
    catchup=False,
)
pipeline = Pipeline(pipeline_name, repo_path=project_path)
tasks = []
def build_execute_block(block):
    def _callable(ds, **kwargs):
        mage_ai.run(
            pipeline_name,
            project_path=project_path,
            block_uuid=block.uuid,
            analyze_outputs=False,
            update_status=False,
        )
    return _callable
for uuid, b in pipeline.blocks_by_uuid.items():
    if b.type == 'scratchpad':
        continue
    tasks.append(dict(
        task_id=uuid,
        upstream_task_ids=b.upstream_block_uuids,
        python_callable=build_execute_block(b),
    ))
def initialize_tasks(dag, tasks):
    operators = {}
    for task_dict in tasks:
        task_operator = PythonOperator(dag=dag, **ignore_keys(task_dict, [
            'upstream_task_ids',
        ]))
        operators[task_operator.task_id] = task_operator
    for task_dict in tasks:
        for task_id in task_dict.get('upstream_task_ids', []):
            dag.set_dependency(task_id, task_dict['task_id'])
    return operators
initialize_tasks(dag, tasks)
```


# Integrate Mage into an existing Airflow project
Source: https://docs.mage.ai/integrations/airflow/integrate-mage-airflow

In this tutorial, we’ll create a DAG in Airflow for scheduling and running a data pipeline; all from the Mage UI.

![wind](https://www.penccil.com/files/U_95_240356858903_wt5.gif)

> [!NOTE]
>
> This tutorial requires that you already have Airflow setup and running
> locally.

## 1. Add `mage-ai` as a dependency in Airflow

Open the `requirements.txt` file in the root directory of your Airflow project,
and add the `mage-ai` library:

```
mage-ai
```

## 2.  Install Mage

You can install and run Mage using Docker or using pip.

### Using Docker

```bash theme={"system"}
docker pull mageai/mageai:latest
```

### Using pip

```bash theme={"system"}
pip install mage-ai
```

## 3. Initialize Mage project

Change directory into your Airflow’s DAGs folder. This is typically in the
folder `dags/`.

```bash theme={"system"}
cd dags
```

Then, initialize a new Mage project in the `dags/` folder.

If you’re using Docker, run the following command in the `dags/` folder:

```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai /app/run_app.sh mage init demo_project

```

If you used pip to install Mage, run the following command in the `dags/`
folder:

```bash theme={"system"}
mage init demo_project
```

Once finished, you should have a folder named `demo_project` inside your `dags/`
folder.

Your current folder structure should look like this:

```
airflow_root_directory/
| -- dags/
| -- | -- demo_project/
```

## 4. Create one-time DAG for pipelines

In the `dags/` folder, create a new file named `create_mage_pipelines.py`.

Then, add the following code:

```python theme={"system"}
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from mage_ai.orchestration.airflow import create_dags
import os


ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'demo_project')

create_dags(
    project_path,
    DAG,
    PythonOperator,
    dag_settings=dict(
        start_date=datetime(2022, 8, 5),  # Change this to any start date you want
    ),
    globals_dict=globals(),
)
```

## 5. Create pipeline

### Start Mage

In the `dags/` folder, start the Mage tool.

If you’re using Docker, run the following command in the `dags/` folder:

```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai /app/run_app.sh mage start demo_project

```

If you used pip to install Mage, run the following command in the `dags/`
folder:

```bash theme={"system"}
mage start demo_project
```

Open [http://localhost:6789](http://localhost:6789) in your browser.

## 6. Add a block

Follow steps 1, 2, and 4 in this [tutorial](/guides/load-api-data) to create
a new pipeline, add 1 data loader block, and add 1 transformer block.

## 7. Run DAG in Airflow for pipeline

1. Open the Airflow webserver UI at
   [http://localhost:8080](http://localhost:8080) in your browser.
2. If you named your pipeline `etl demo` based on the tutorial from the previous
   step, then find a DAG named `mage_pipeline_etl_demo`. If you named it
   something else, find a DAG with the prefix `mage_pipeline_`.
3. Click on the DAG to view the detail page. The URL could typically be this:
   [http://localhost:8080/admin/airflow/tree?dag\_id=mage\_pipeline\_etl\_demo](http://localhost:8080/admin/airflow/tree?dag_id=mage_pipeline_etl_demo).
4. Turn that DAG on if its currently off.
5. Trigger a new DAG run.
6. Watch the DAG as it runs each task according to the pipeline you created in
   Mage.

![Mage in Airflow](https://mage-ai.github.io/assets/mage-airflow.jpg)


# Databricks
Source: https://docs.mage.ai/integrations/compute/databricks

This is a guide for using Databricks Spark cluster with Mage.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=databricks)

Besides supporting running Spark pipelines in AWS EMR cluster and standalone Spark cluster, Mage also supports
running Spark pipelines in Databricks cluster.

## Set up

Here is an overview of the steps required to use Mage with Databricks Cluster:

1. [Set up Databricks cluster](#1-set-up-databricks-cluster)
2. [Use Mage databricks docker image](#2-use-mage-databricks-docker-image)
3. [Configure environment variables](#3-configure-environment-variables)
4. [Sample pipeline with PySpark code](#4-sample-pipeline-with-pyspark-code)
5. [Verify everything worked](#5-verify-everything-worked)

If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat)

### 1. Set up Databricks cluster

Set up a Databricks workspace and cluster following the docs:

* [Create a workspace using the account console](https://docs.databricks.com/administration-guide/account-settings-e2/workspaces.html)
* [Create a cluster](https://docs.databricks.com/clusters/create-cluster.html)

### 2. Use Mage databricks docker image

Contact Mage team to update your Mage Pro cluster to use Mage databricks docker image.

### 3. Configure environment variables

Set the following environment variables in your Mage Pro cluster to enable connectivity with your Databricks workspace:

* **`DATABRICKS_HOST`**\
  The base URL of your Databricks workspace.\
  Example: `https://<your-databricks-instance>.cloud.databricks.com`

* **`DATABRICKS_TOKEN`**\
  A personal access token (PAT) used for authenticating with Databricks.\
  You can generate this token in the Databricks UI by navigating to:\
  **Settings** > **Developer** > **Access Tokens**.

* **`DATABRICKS_CLUSTER_ID`**\
  The unique identifier for the Databricks cluster where queries will be executed.\
  You can find this in your Databricks workspace under: **Compute** > **Clusters**.\
  Refer to [Databricks documentation](https://docs.databricks.com/aws/en/workspace/workspace-details#cluster-url-and-id) for detailed steps on retrieving the cluster ID.

### 4. Sample pipeline with PySpark code

1. Create a new pipeline by clicking `New pipeline` in the `/pipelines` page.
2. Open the pipeline's metadata.yaml file and set the config.
   ```
   cache_block_output_in_memory: true
   run_pipeline_in_one_process: true
   ```
3. Click `+ Data loader`, then `Base template (generic)` to add a new data loader
   block.
4. Paste the following sample code in the new data loader block:
5. Click "Run code" button to run the block.

```python theme={"system"}
from databricks.connect import DatabricksSession

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data(*args, **kwargs):
    spark = DatabricksSession.builder.remote().getOrCreate()

    data = [("John", 28), ("Anna", 23), ("Mike", 35)]
    columns = ["Name", "Age"]

    df = spark.createDataFrame(data, columns)
    return df


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

1. Click `+ Data exporter`, then `Base template (generic)` to add a new data
   exporter block.
2. Paste the following sample code in the new data exporter block:
3. Click "Run code" button to run the block.

```python theme={"system"}
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(df, *args, **kwargs):
    df.write.format("delta").mode("append").saveAsTable("user_table")

    return df
```

### 5. Verify everything worked

Check the table in your Unity Catalog to verify whether the data is written to it correctly.


# Compute management for Spark
Source: https://docs.mage.ai/integrations/compute/management

Manage your Spark compute resources and track Spark pipeline execution metrics.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=compute-management)

**1. Setup**

![](https://mage-ai.github.io/assets/pro/features/compute/management/setup.png)

**2. Resources**

![](https://mage-ai.github.io/assets/pro/features/compute/management/resources.png)

**3. Clusters**

![](https://mage-ai.github.io/assets/pro/features/compute/management/clusters.png)

**4. Manage**

![](https://mage-ai.github.io/assets/pro/features/compute/management/clusters-detail.png)

**5. Monitoring**

![](https://mage-ai.github.io/assets/pro/features/compute/management/monitoring-jobs-stages-detail.png)

**6. Monitoring jobs**

![](https://mage-ai.github.io/assets/pro/features/compute/management/monitoring-jobs.png)

**7. Monitor job stages**

![](https://mage-ai.github.io/assets/pro/features/compute/management/monitoring-stages.png)

**8. Monitoring job stage tasks**

![](https://mage-ai.github.io/assets/pro/features/compute/management/monitoring-jobs-detail.png)

**9. Monitor SQL execution**

![](https://mage-ai.github.io/assets/pro/features/compute/management/monitoring-sql.png)

**10. System**

![](https://mage-ai.github.io/assets/pro/features/compute/management/monitoring-system.png)

**11. Authenticate**

![](https://mage-ai.github.io/assets/pro/features/compute/management/ssh-tunnel.png)

**12. Connect to remote cluster**

![](https://mage-ai.github.io/assets/pro/features/compute/management/pipeline-subheader.png)

**13. Monitor block execution**

![](https://mage-ai.github.io/assets/pro/features/compute/management/block-tabs.png)

**14. Block execution jobs**

![](https://mage-ai.github.io/assets/pro/features/compute/management/block-jobs.png)

**15. Block execution stages and tasks**

![](https://mage-ai.github.io/assets/pro/features/compute/management/block-stages.png)

**16. Block execution SQL**

![](https://mage-ai.github.io/assets/pro/features/compute/management/block-sql.png)

***

> **[Get started for free](https://cloud.mage.ai/sign-up?source=cloud-infrastructure)**
>
> ![](https://mage-ai.github.io/assets/pro/banner-dark.png)
>
> A fully managed service, where we maintain the infrastructure, guarantee uptime,
> automatically scale your workloads to handle any volume of pipeline runs,
> automatically upgrade new versions of Mage Pro only features,
> monitor your production pipelines, and provide enterprise level support.


# PySpark in Mage Pro
Source: https://docs.mage.ai/integrations/compute/spark-pro

This guide explains how to use PySpark in Mage Pro, including how to run PySpark blocks in batch pipelines, connect to Spark on Kubernetes, manage dependencies like Apache Iceberg, and access cloud storage securely. blocks in batch pipelines, manage dependencies like Apache Iceberg, and access cloud storage securely.


Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=spark)

Mage Pro provides built-in support for running PySpark code inside your pipelines. You can run distributed Spark jobs directly within a Mage Pro batch pipeline, with automatic resource management, cloud integration, and observability.

Use PySpark in Mage to transform large datasets, connect to cloud object storage like **GCS or S3**, and integrate with modern data lake formats like **Apache Iceberg**.

## How to Use PySpark in Mage Pro

Follow these steps to run PySpark code in Mage Pro:

1. Create a **batch pipeline** in the Mage UI.
2. Add a **block** of type: `Data Loader`, `Transformer`, `Data Exporter`, or `Custom`.
3. In your block, write PySpark code using the provided `SparkSession`.
4. Install or mount any required Spark JARs, such as those for Iceberg or cloud storage access.

## Example Pipeline

Create a standard **batch pipeline** and configure the following settings in the pipeline's `metadata.yaml` file to ensure PySpark works properly:

```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```

These settings ensure data is passed in-memory between PySpark blocks and the pipeline runs in a single JVM process for Spark compatibility.

### Data Loader Block (PySpark)

```python theme={"system"}
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test
from pyspark.sql import Row, SparkSession

@data_loader
def load_data(*args, **kwargs):
    spark = SparkSession.builder.getOrCreate()
    data = [
        Row(id=1, name="Alice", age=29, salary=50000),
        Row(id=2, name="Bob", age=35, salary=60000),
        Row(id=3, name="Charlie", age=40, salary=70000),
        Row(id=4, name="David", age=45, salary=None)  # Null value example
    ]

    df = spark.createDataFrame(data)
    df.show()
    return df
```

### Data Exporter Block

```python theme={"system"}
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter

@data_exporter
def export_data(df, *args, **kwargs):
    df.write.csv("spark_output", header=True, mode="overwrite")
```

This example pipeline loads structured data into a Spark DataFrame and writes it to disk in CSV format.

## Connecting to Spark on Kubernetes

Mage Pro can connect to an external Spark cluster running on Kubernetes. This allows you to leverage existing Spark infrastructure or run Spark jobs on a dedicated Kubernetes cluster.

### Option 1: Configure via metadata.yaml (Recommended)

Configure Spark connection settings in your project's `metadata.yaml` file:

```yaml theme={"system"}
spark_config:
  # Application name
  app_name: 'my spark app'
  # Master URL to connect to Spark on Kubernetes
  # For Kubernetes native mode:
  spark_master: 'k8s://https://kubernetes.default.svc:443'
  # Or for standalone Spark cluster in Kubernetes:
  # spark_master: 'spark://spark-master-service:7077'
  
  # Executor environment variables
  executor_env: {}
  
  # Jar files to be uploaded to the cluster and added to the classpath
  spark_jars: []
  
  # Path where Spark is installed on worker nodes (if applicable)
  spark_home: null
  
  # Additional Spark configuration options
  others:
    spark.executor.memory: '4g'
    spark.executor.cores: '2'
    spark.kubernetes.namespace: 'default'
    spark.kubernetes.container.image: 'your-spark-image:tag'
```

### Option 2: Configure via Environment Variable

Alternatively, you can set the `SPARK_MASTER_HOST` environment variable in your Mage Pro workspace configuration:

* For Kubernetes native mode: `SPARK_MASTER_HOST=k8s://https://kubernetes.default.svc:443`
* For standalone Spark cluster: `SPARK_MASTER_HOST=spark://spark-master-service:7077`

### Setting Up Spark on Kubernetes

If you need to deploy Spark on Kubernetes first, you can use Helm:

```bash theme={"system"}
# Add the Bitnami Helm repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install Spark cluster
helm install my-spark bitnami/spark

# Get the Spark master service URL
kubectl get svc my-spark-spark-master-svc
```

Then use the service URL in your `spark_master` configuration. For example:

* Service name: `my-spark-spark-master-svc`
* Port: `7077`
* Configuration: `spark_master: 'spark://my-spark-spark-master-svc:7077'`

### Using Spark in Your Code

Once configured, you can use Spark in your pipeline blocks:

```python theme={"system"}
from pyspark.sql import SparkSession

# SparkSession will automatically use the configuration from metadata.yaml
spark = SparkSession.builder.getOrCreate()

# Your Spark operations here
df = spark.read.csv("your-data.csv")
```

The Spark session will automatically connect to your configured Kubernetes Spark cluster.

## Benefits of Running PySpark in Mage Pro

Mage Pro handles all the infrastructure so you can focus on your PySpark code:

* ⚙️ Distributed execution with automatic pod scheduling and resource allocation
* ☁️ Seamless cloud integration with GCS, S3, and service account/IAM-based authentication
* 🧩 Support for Spark JARs and connectors like Apache Iceberg, GCS connectors, Delta Lake, and more
* 📈 Built-in observability, with access to logs, resource usage, and block-level monitoring in the Mage UI

## Notes

* You can customize the `SparkSession` in any block using `.builder.config(...)` to tune performance or integrate external tools.
* Cloud storage credentials (e.g., a GCP service account key or AWS credentials) must be mounted and accessible inside the Mage Pro cluster.
* For advanced use cases (e.g., Apache Iceberg), see the [Iceberg + PySpark guide](/integrations/databases/Iceberg#using-iceberg-with-pyspark).


# Spark and PySpark
Source: https://docs.mage.ai/integrations/compute/spark-pyspark

This is a guide for using Spark (PySpark) with Mage in different cloud providers or Kubernetes cluster.

Want to become a
[Sparkmage](https://c1.scryfall.com/file/scryfall-cards/large/front/0/3/030f4058-54e5-4333-bd6c-2789c334bf12.jpg)?

![](https://user-images.githubusercontent.com/78053898/199308674-afd8860e-09de-4003-a7ed-9ab2db221b0c.png)

# Mage Pro: Effortless PySpark Integration

Mage Pro makes it easy to run PySpark pipelines with a streamlined, configuration-free setup. Get started immediately without worrying about complex configurations—just write your PySpark code and execute it seamlessly within your Mage Pro cluster.

## Example: Simple PySpark Pipeline in Mage Pro

### Create the pipeline

Create a standard batch pipeline and configure the below settings in pipeline `metadata.yaml` file:

```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```

### Data loader block code:

```python theme={"system"}
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test
from pyspark.sql import Row, SparkSession


@data_loader
def load_data(*args, **kwargs):
    spark = SparkSession.builder.getOrCreate()
    data = [
        Row(id=1, name="Alice", age=29, salary=50000),
        Row(id=2, name="Bob", age=35, salary=60000),
        Row(id=3, name="Charlie", age=40, salary=70000),
        Row(id=4, name="David", age=45, salary=None)  # Null value example
    ]

    df = spark.createDataFrame(data)
    df.show()
    return df
```

### Data exporter block code:

```python theme={"system"}
if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(df, *args, **kwargs):
    df.write.csv("spark_output", header=True, mode="overwrite")
```

***

# Open source

## Kubernetes

When you run Mage and Spark in the same Kubernetes cluster, you can set the environment variable `SPARK_MASTER_HOST`
to the url of the master node of the Spark cluster in Mage container. Then you'll be able to connect Mage to your Spark
cluster and execute PySpark code in Mage.

Here is an overview of the steps required to use Mage with Spark in Kubernetes cluster.

### 1. Run Spark cluster in Kubernetes cluster

You can use Helm command to run Spark cluster in the Kubernetes cluster:

```bash theme={"system"}
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-release bitnami/spark
```

Then follow the instructions from the output to get the Spark master WebUI URL:

```bash theme={"system"}
kubectl port-forward --namespace default svc/my-release-spark-master-svc 80:80
echo "Visit http://127.0.0.1:80 to use your application"
```

Find the Spark Master URL at the top of the web page.

### 2. Build Mage docker image with Spark environment

* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Change the current directory to the directory that contains the `Dockerfile`. Build the docker image with the command `docker build -t mage_spark .`

### 3. Run Mage in Kubernetes cluster with the `SPARK_MASTER_HOST` environment variable

* Use the docker image built from step 2 as the container image in your Kubernetes yaml file.
* Add `SPARK_MASTER_HOST` environment variable with the Spark Master URL from step 1 to your the container spec
  in your Kubernetes yaml file.
* Then deploy Mage to Kubernetes with the updated yaml file.

Here is the example Pod config:

```yaml theme={"system"}
apiVersion: v1
kind: Pod
metadata:
  name: mage-server
spec:
  containers:
  - name: mage-server
    image: mage_spark
    imagePullPolicy: Never
    ports:
    - containerPort: 6789
    volumeMounts:
    - name: mage-fs
      mountPath: /home/src
    env:
      - name: KUBE_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
      - name: SPARK_MASTER_HOST
        value: "spark://spark_master_url:7077"
  volumes:
  - name: mage-fs
    hostPath:
      path: /path/to/mage_project
  serviceAccountName: mage-user
```

Updates

* In Mage version `0.8.83` and above, you don't need to specify the environment variable anymore.
* You can specify the spark config in the project's `metadata.yaml` file as stated in [Custom Spark Session](#custom-spark-session-at-the-project-level) section.

### 4. Run PySpark code in Mage

When Mage pod is running, you can create a "Standard (batch)" pipeline (`python` kernel) and then write PySpark code in any blocks.

In `Scratchpad` block, you'll need to manually create the Spark session with the code:

```python theme={"system"}
import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.master(os.getenv('SPARK_MASTER_HOST', 'local')).getOrCreate()
```

In other blocks, you can access the Spark session from the `kwargs` via `kwargs['spark']`.

***

## Standalone Spark cluster

### 1. Build Mage docker image with Spark environment

* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Change the current directory to the directory that contains the `Dockerfile`. Build the docker image with the command `docker build -t mage_spark .`

### 2. Start Mage with `SPARK_MASTER_HOST` environment variable

Type this command in your terminal to start Mage using docker

```bash theme={"system"}
docker run -it --name mage_spark -e SPARK_MASTER_HOST='local' -p 6789:6789 -v $(pwd):/home/src mage_spark \
  /app/run_app.sh mage start demo_project
```

Notes

* `demo_project` is the name of your project, you can change it to anything you want
* Set the Spark cluster's master node url as the value for environment variable `SPARK_MASTER_HOST`. If you
  use local Spark, you can set the value of `SPARK_MASTER_HOST` to `local` or not set the environment variable.

Updates

* In Mage version `0.8.83` and above, you don't need to specify the environment variable anymore.
* You can specify the spark config in the project's `metadata.yaml` file as stated in [Custom Spark Session](#custom-spark-session-at-the-project-level) section.

### 3. Run PySpark code in Mage

When Mage is running, you can create a "Standard (batch)" pipeline (`python` kernel) and then write PySpark code in any blocks.

In `Scratchpad` block, you'll need to manually create the Spark session with the code:

```python theme={"system"}
import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.master(os.getenv('SPARK_MASTER_HOST', 'local')).getOrCreate()
```

In other blocks, you can access the Spark session from the `kwargs` via `kwargs['spark']`.

***

## Custom Spark Session at the Project Level

### 1. Build Mage docker image with Spark environment

* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Build the docker image with the command `docker build -t mage_spark .`

### 2. Start Mage using docker with the following command in your terminal

```bash theme={"system"}
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
  /app/run_app.sh mage start demo_project
```

Notes

* `demo_project` is the name of your project, you can change it to anything you want

### 3. Review the `spark_config` section in `metadata.yaml` under the project folder, and make necessary adjustments

```yaml theme={"system"}
spark_config:
  # Application name
  app_name: 'my spark app'
  # Master URL to connect to
  # e.g., spark_master: 'spark://host:port', or spark_master: 'yarn'
  spark_master: 'local'
  # Executor environment variables
  # e.g., executor_env: {'PYTHONPATH': '/home/path'}
  executor_env: {}
  # Jar files to be uploaded to the cluster and added to the classpath
  # e.g., spark_jars: ['/home/path/example1.jar']
  spark_jars: []
  # Path where Spark is installed on worker nodes,
  # e.g. spark_home: '/usr/lib/spark'
  spark_home: null
  # List of key-value pairs to be set in SparkConf
  # e.g., others: {'spark.executor.memory': '4g', 'spark.executor.cores': '2'}
  others: {}
```

### 4. Run PySpark code in Mage

When Mage is running, you can create a "Standard (batch)" pipeline (`python` kernel), and add a block,
then write PySpark code using the Spark session via `kwargs['spark']`, e.g.,

```python theme={"system"}
spark = kwargs.get('spark')
print(spark.sql('select 1'))
```

In `Scratchpad` block, you'll need to manually create the Spark session with the code:

```python theme={"system"}
from mage_ai.data_preparation.repo_manager import RepoConfig, get_repo_path
from mage_ai.services.spark.config import SparkConfig
from mage_ai.services.spark.spark import get_spark_session

repo_config = RepoConfig(repo_path=get_repo_path())
spark_config = SparkConfig.load(
    config=repo_config.spark_config)

spark = get_spark_session(spark_config)
print(spark.sql('select 1'))
```

## Custom Spark Session at the Pipeline Level

It is also possible to use different Spark Sessions in different pipelines under the same project.

### 1. Build Mage docker image with Spark environment

* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Build the docker image with the command `docker build -t mage_spark .`

### 2. Start Mage using docker with the following command in your terminal

```bash theme={"system"}
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
  /app/run_app.sh mage start demo_project
```

Notes

* `demo_project` is the name of your project, you can change it to anything you want

### 3. Create a "Standard (batch)" pipeline (`python` kernel), and update the `spark_config` section in `metadata.yaml` under the pipeline folder:

```yaml theme={"system"}
spark_config:
  # Application name
  app_name: 'my spark app at pipeline level'
  # Master URL to connect to
  # e.g., spark_master: 'spark://host:port', or spark_master: 'yarn'
  spark_master: 'local'
  # Executor environment variables
  # e.g., executor_env: {'PYTHONPATH': '/home/path'}
  executor_env: {}
  # Jar files to be uploaded to the cluster and added to the classpath
  # e.g., spark_jars: ['/home/path/example1.jar']
  spark_jars: []
  # Path where Spark is installed on worker nodes,
  # e.g. spark_home: '/usr/lib/spark'
  spark_home: null
  # List of key-value pairs to be set in SparkConf
  # e.g., others: {'spark.executor.memory': '4g', 'spark.executor.cores': '2'}
  others: {}
```

### 4. Run PySpark code in Mage

Add a block to the pipeline, then write PySpark code using the Spark session via `kwargs['spark']`, e.g.,

```python theme={"system"}
spark = kwargs.get('spark')
print(spark.sql('select 1'))
```

## Custom Spark session via code

It is also possible to create custom Spark ssesion using Python code. Here are the instructions:

1. Set the `run_pipeline_in_one_process: true` in your pipeline's metadata.yaml
2. In the `spark_config` of your project's metadata.yaml or pipeline's metadata.yaml, set `use_custom_session` to `true`. Example config:
   ```yaml theme={"system"}
   spark_config:
     use_custom_session: true
     custom_session_var_name: 'spark'
   ```
3. In the first block in your pipeline, write Python code to create the Spark session and set it in the `kwargs['context']['spark']`. Example code:
   ```python theme={"system"}
   from pyspark.conf import SparkConf
   from pyspark.sql import SparkSession

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader

   @data_loader
   def load_data(*args, **kwargs):
       spark = (
           SparkSession
           .builder
           .appName('Test spark')
           .getOrCreate()
       )
       kwargs['context']['spark'] = spark
       ...
   ```
4. Then, you can access the Spark session in the subsequent blocks via `kwargs['spark']`.

***

## Hadoop and Yarn cluster for Spark

### 1. Build Mage docker image with a Hadoop Yarn environment

* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/hadoop/Dockerfile) for Mage with Hadoop environment.
* Build the docker image with the command `docker build -t mage_hadoop .`

### 2. Start Mage with the following command in your terminal using docker

```bash theme={"system"}
docker run -it --name mage_hadoop -p 6789:6789 -v $(pwd):/home/src mage_hadoop \
  /app/run_app_with_hadoop.sh mage start demo_project
```

Notes

* `demo_project` is the name of your project, you can change it to anything you want

### 3. Run PySpark code in Mage using Hadoop Yarn

* Change the `metadata.yaml` file in the main pipeline folder to include the following Spark settings:

```yaml theme={"system"}
spark_config:
  # Application name
  app_name: 'my spark yarn app'
  # Master URL to connect to
  # e.g., spark_master: 'spark://host:port', or spark_master: 'yarn'
  spark_master: 'yarn'
  # Executor environment variables
  # e.g., executor_env: {'PYTHONPATH': '/home/path'}
  executor_env: {}
  # Jar files to be uploaded to the cluster and added to the classpath
  # e.g., spark_jars: ['/home/path/example1.jar']
  spark_jars: []
  # Path where Spark is installed on worker nodes,
  # e.g. spark_home: '/usr/lib/spark'
  spark_home: null
  # List of key-value pairs to be set in SparkConf
  # e.g., others: {'spark.executor.memory': '4g', 'spark.executor.cores': '2'}
  others: {}
```

* Create a new `Standalone (batch)` pipeline and add a `Data loader`, then run it with the following code:

```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
    """
    Template code for loading data from any source.

    Returns:
        Anything (e.g. data frame, dictionary, array, int, str, etc.)
    """
    # Specify your data loading logic here
    spark = kwargs.get('spark')
    spark_conf = spark.sparkContext.getConf()
    print('#' * 40)
    print(spark_conf.toDebugString())
    print('#' * 40)
    print(spark.sql('select 1'))
    return {}
```

* Verify that the added Spark code is running through a Yarn master.

***

## AWS

Here is an overview of the steps required to use Mage locally with Spark in AWS:

1. [Create an S3 bucket for Spark](#1-create-an-s3-bucket-for-spark)
2. [Start Mage](#2-start-mage)
3. [Configure project’s metadata settings](#3-configure-projects-metadata-settings)
4. [Launch EMR cluster](#4-launch-emr-cluster)
5. [Allow EMR connection permissions](#5-allow-emr-connection-permissions)
6. [Sample pipeline with PySpark code](#6-sample-pipeline-with-pyspark-code)
7. [Debugging](#7-debugging)
8. [Clean up](#8-clean-up)

If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat)

### 1. Create an S3 bucket for Spark

Using Spark on AWS EMR requires an AWS S3 bucket to store logs, scripts, etc.

Follow
[AWS’s guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)
to create an S3 bucket. You don’t need to add any special permissions or
policies to this bucket.

Once you created an S3 bucket, note the name of the bucket (we’ll need it
later).

### 2. Start Mage

Using Mage with Spark is much easier if you use Docker.

Type this command in your terminal to start Mage using docker (Note:
`demo_project` is the name of your project, you can change it to anything you
want):

```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage start demo_project
```

### 3. Configure project’s metadata settings

Open your project’s `metadata.yaml` file located at the root of your project’s
directory: `demo_project/metadata.yaml` (presuming your project is named
`demo_project`).

Change the values for the keys mentioned in the following steps.

* Set `remote_variables_dir`.

  Change the value for key `remote_variables_dir` to equal the S3 bucket you
  created in an earlier step.
  For example, if your S3 bucket is named `my-awesome-bucket`, then the value for
  the key `remote_variables_dir` should be `s3://my-awesome-bucket`.

* Remove optional settings

  You can remove the following 2 keys:

  * `master_security_group`
  * `slave_security_group`

Your final `metadata.yaml` file could look like this:

```yaml theme={"system"}
variables_dir: ./
remote_variables_dir: s3://my-awesome-bucket

emr_config:
  master_instance_type: "r5.4xlarge"
  slave_instance_type: "r5.4xlarge"
```

Additionally, you can customize the instance count and Spark properties in your EMR cluster. Here is
an example config:

```yaml theme={"system"}
emr_config:
  bootstrap_script_path: /path/to/emr_bootstrap.sh  # Use a custom bootstrap script for EMR cluster
  master_instance_type: 'r5.4xlarge'
  master_spark_properties:
    spark.driver.memory: 400M
    spark.executor.cores: '8'
  slave_instance_count: 2
  slave_instance_type: 'r5.4xlarge'
  slave_spark_properties:
    spark.driver.memory: 400M
    spark.executor.cores: '8'
  scaling_policy:             # Set up auto scaling of the EMR cluster
    unit_type: 'Instances'    # 'InstanceFleetUnits'|'Instances'|'VCPU'
    minimum_capacity_units: 1
    maximum_capacity_units: 4
    maximum_on_demand_capacity_units: 4
    maximum_core_capacity_units: 3
  spark_jars:                 # External Jar libraries
  - s3://bucket/mysql-connector-java-8.0.28.jar
```

> [!NOTE]
>
> You may need to request an increase in quota limits for using those instance
> types.
>
> For more information on how to view your quota limits and request an increase,
> check out this
> [AWS document](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html).

### 4. Launch EMR cluster

You’ll need an AWS Access Key ID and an AWS Secret Access Key. This is provided
from AWS’s IAM Management console.

Once you’ve acquired those credentials, you can switch your kernel in your Mage notebook
to `pyspark` kernel. Mage will automatically creates the EMR cluster when you switch to `pyspark`
kernel. The cluster is usually created and initialized within 8\~10 minutes. Then you can select
your EMR cluster in the dropdown of your `pyspark` kernel selector.

### 5. Allow EMR connection permissions

You need to whitelist your IP in the security group of your EMR cluster to access it.

1. Open the EC2 dashboard in AWS
2. Click on "Security Groups" on the left side panel under the section "Network
   & Security"
3. Find the security group named "ElasticMapReduce-master"
4. Add a new inbound rule with the following values:

| Type | Protocol | Port range | Source |
| - | - | - | - |
| Custom TCP | TCP | 8998 | My IP |

This will allow your locally running Mage to remotely access AWS EMR.

### 6. Sample pipeline with PySpark code

If you aren’t using Docker and you installed Mage using `pip`, you must run the
following commands in your terminal to use the `pyspark` kernel:

```bash theme={"system"}
pip install sparkmagic
mkdir ~/.sparkmagic
wget https://raw.githubusercontent.com/jupyter-incubator/sparkmagic/master/sparkmagic/example_config.json
mv example_config.json ~/.sparkmagic/config.json
sed -i 's/localhost:8998/host.docker.internal:9999/g' ~/.sparkmagic/config.json
```

***

1. Create a new pipeline by going to `File` in the top left corner of the page
   and then clicking `New pipeline`.
2. Change the pipeline’s kernel from `python` to `pyspark`. Click the button
   with the green dot and the word `python` next to it. This is located at the
   top of the page on the right side of your header.
3. Click `+ Data loader`, then `Generic (no template)` to add a new data loader
   block.
4. Paste the following sample code in the new data loader block:

```python theme={"system"}
from pandas import DataFrame
import io
import pandas as pd
import requests


if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


def data_from_internet():
    url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'

    response = requests.get(url)
    return pd.read_csv(io.StringIO(response.text), sep=',')


@data_loader
def load_data(**kwargs) -> DataFrame:
    df_spark = kwargs['spark'].createDataFrame(data_from_internet())

    return df_spark
```

1. Click `+ Data exporter`, then `Generic (no template)` to add a new data
   exporter block.
2. Paste the following sample code in the new data exporter block (change the
   `s3://bucket-name` to the bucket you created from a previous step):

```python theme={"system"}
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data(df: DataFrame, **kwargs) -> None:
    (
        df.write
        .option('delimiter', ',')
        .option('header', 'True')
        .mode('overwrite')
        .csv('s3://mage-spark-cluster/demo_project/demo_pipeline/')
    )
```

#### Verify everything worked

Let’s load the data from S3 that we just created using Spark:

1. Click `+ Data loader`, then `Generic (no template)` to add a new data loader
   block.
2. Paste the following sample code in the new data loader block (change the
   `s3://bucket-name` to the bucket you created from a previous step):

```python theme={"system"}
from pandas import DataFrame


if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data(**kwargs) -> DataFrame:
    df = (
        kwargs['spark'].read
        .format('csv')
        .option('header', 'true')
        .option('inferSchema', 'true')
        .option('delimiter', ',')
        .load('s3://mage-spark-cluster/demo_project/demo_pipeline/*')
    )

    return df
```

### 7. Debugging

If you run into any problems, 1st thing to try is restarting the kernel: `Run` >
`Restart kernel`.

If the block is hanging when running the block, it could be due to network connection issue.
Make sure the EMR cluster is accessible from the Mage server. You can verify whether Mage server's
security group or IP is whitelisted in EMR cluster's security group following section [Allow EMR connection permissions](#5-allow-emr-connection-permissions)

If that doesn’t work, restart the app by stopping the docker container and
starting it again.

### 8. Clean up

Please make sure to terminate your EMR cluster when you’re done using it so you
can save money.

***


# Algolia Integration in Mage
Source: https://docs.mage.ai/integrations/databases/Algolia



## Credentials Configuration

Open the file named `io_config.yaml` at the root of your Mage project and enter required fields:

```yaml theme={"system"}
version: 0.1.1
default:
  ALGOLIA_APP_ID: app_id
  ALGOLIA_API_KEY: api_key
  ALGOLIA_INDEX_NAME: index_name
```

## Required Algolia Libraries

Add `algoliasearch==3.0.0` into `requirements.txt` to install the required weaviate library.

## Integration using Python Blocks

Follow these steps to integrate Algolia into your workflow using Python blocks:

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter with Template. Under "Databases" category you can
   find the "Algolia" template.

* Data loader arguments:
  Args:
  query\_texts (str): Texts to query.
  index\_name (str): Name of the index. Defaults to the name defined in `io_config.yaml`.
  column\_names (List): columns to fetch.
  Returns:
  DataFrame: Data frame loaded.

* Data exporter arguments:
  Write  data into Algolia.

  Args:
  df (DataFrame): dataframes to write.
  index\_name (str): Name of the index. Defaults to the name defined in io\_config.yaml.

3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.


# Athena
Source: https://docs.mage.ai/integrations/databases/Athena



![](https://user-images.githubusercontent.com/78053898/198754439-87922b6d-e8ed-4b6c-852c-96a1d3545612.png)

## Prerequisites

Before using Athena in Mage, make sure you have the following:

* ✅ An **S3 bucket** to store Athena query results (this is required by Athena)
* ✅ A **DynamoDB table** if using Athena’s table versioning (optional but recommended for Glue-backed catalogs)
* ✅ A **database and table** defined in Athena (via AWS Glue, or manually in the Athena console)
* ✅ An **IAM user or role** with the following permissions:
  * `athena:StartQueryExecution`
  * `athena:GetQueryExecution`
  * `athena:GetQueryResults`
  * `s3:PutObject`, `s3:GetObject`, `s3:ListBucket` (for the staging S3 bucket)
* ✅ The [`pyathena`](https://pypi.org/project/pyathena/) Python package installed

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data loader).
3. Select `Generic (no template)`.
4. Enter the following code snippet using `pyathena`:

   ```python theme={"system"}
   from pyathena import connect
   import pandas as pd

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader

   @data_loader
   def load_from_athena(**kwargs) -> pd.DataFrame:
       conn = connect(
           aws_access_key_id='your-access-key',
           aws_secret_access_key='your-secret-key',
           region_name='us-west-2',
           s3_staging_dir='s3://your-athena-query-results/'
       )

       query = 'SELECT * FROM your_database.your_table LIMIT 100'
       df = pd.read_sql(query, conn)
       return df
   ```
5. Run the block.

## Install dependencies

Before running the code, install the required Python package:

```bash theme={"system"}
pip install pyathena
```

## Common errors

### ❌ Missing or invalid staging directory

Athena requires a staging location in S3 to store temporary query results.

Make sure `s3_staging_dir` is:

* A valid S3 URI (e.g. `s3://your-bucket/path/`)
* Located in the **same region** as your Athena workgroup
* Writable by the credentials you're using

### ❌ Access Denied

If you see access-related errors:

* Verify your AWS access key and secret are correct
* Confirm that your IAM user or role has permissions for both Athena and S3
* Check the S3 bucket policy to ensure it allows access from your IAM identity

### ❌ Region mismatch

Make sure all resources are in the same region:

* Athena workgroup
* S3 staging bucket
* AWS region specified in your connection config (e.g. `region_name='us-west-2'`)

Using mismatched regions can result in silent failures or confusing errors.


# Azure Blob Storage
Source: https://docs.mage.ai/integrations/databases/AzureBlobStorage



## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

**Required:** storage account name. For authentication you can use either Azure AD (service principal) or default credentials (e.g. Azure CLI, managed identity).

```yaml theme={"system"}
version: 0.1.1
default:
  AZURE_STORAGE_ACCOUNT_NAME: your_storage_account_name

  # Optional: Service principal (if not using DefaultAzureCredential)
  AZURE_CLIENT_ID: ...
  AZURE_CLIENT_SECRET: ...
  AZURE_TENANT_ID: ...
```

If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are omitted, the client uses `DefaultAzureCredential` (Azure CLI, environment variables, or managed identity when running in Azure).

## Using Python block

1. Create a new pipeline or open an existing pipeline.

2. Add a data loader or transformer block (the code snippet below is for a data
   loader).

3. Select `Generic (no template)`.

4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

   ```python theme={"system"}
   from mage_ai.settings.repo import get_repo_path
   from mage_ai.io.config import ConfigFileLoader
   from mage_ai.io.azure_blob_storage import AzureBlobStorage
   from os import path
   from pandas import DataFrame

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader


   @data_loader
   def load_from_azure_blob(**kwargs) -> DataFrame:
       config_path = path.join(get_repo_path(), 'io_config.yaml')
       config_profile = 'default'

       container_name = '...'  # Your container name
       blob_path = '...'       # Blob path (e.g. folder/data.parquet)

       return AzureBlobStorage.with_config(
           ConfigFileLoader(config_path, config_profile)
       ).load(container_name, blob_path)
   ```

5. Run the block.

### Export data to Azure Blob Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_azure_blob(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    container_name = '...'
    blob_path = '...'

    AzureBlobStorage.with_config(
        ConfigFileLoader(config_path, config_profile)
    ).export(df, container_name, blob_path)
```

## Supported formats

Azure Blob Storage supports loading and exporting:

* `.csv`
* `.json`
* `.parquet`

## Permissions

Ensure your Azure AD app or managed identity has the appropriate role on the storage account, for example:

* **Storage Blob Data Contributor** – read and write blobs
* **Storage Blob Data Reader** – read-only


# Azure Data Lake Storage Gen2
Source: https://docs.mage.ai/integrations/databases/AzureDataLakeStorage



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=azure-data-lake-storage)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

**Required:** storage account name. For authentication you can use either Azure AD (service principal) or default credentials.

```yaml theme={"system"}
version: 0.1.1
default:
  AZURE_STORAGE_ACCOUNT_NAME: your_storage_account_name

  # Optional: Service principal (if not using DefaultAzureCredential)
  AZURE_CLIENT_ID: ...
  AZURE_CLIENT_SECRET: ...
  AZURE_TENANT_ID: ...
```

If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are omitted, the client uses `DefaultAzureCredential` (Azure CLI, environment variables, or managed identity when running in Azure).

## Using Python block

1. Create a new pipeline or open an existing pipeline.

2. Add a data loader or transformer block (the code snippet below is for a data
   loader).

3. Select `Generic (no template)`.

4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

   ```python theme={"system"}
   from mage_ai.settings.repo import get_repo_path
   from mage_ai.io.config import ConfigFileLoader
   from mage_ai.io.azure_data_lake_storage import AzureDataLakeStorage
   from os import path
   from pandas import DataFrame

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader


   @data_loader
   def load_from_azure_data_lake(**kwargs) -> DataFrame:
       config_path = path.join(get_repo_path(), 'io_config.yaml')
       config_profile = 'default'

       container_name = '...'  # File system / container name
       file_path = '...'       # Path to file (e.g. folder/data.parquet)

       return AzureDataLakeStorage.with_config(
           ConfigFileLoader(config_path, config_profile)
       ).load(container_name, file_path)
   ```

5. Run the block.

### Export data to Azure Data Lake Storage Gen2

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_data_lake_storage import AzureDataLakeStorage
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_azure_data_lake(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    container_name = '...'
    file_path = '...'

    AzureDataLakeStorage.with_config(
        ConfigFileLoader(config_path, config_profile)
    ).export(df, container_name, file_path)
```

## Supported formats

Azure Data Lake Storage Gen2 supports loading and exporting:

* `.csv`
* `.json`
* `.parquet`

## Permissions

Ensure your Azure AD app or managed identity has the appropriate role on the storage account with Data Lake Storage Gen2 (e.g. hierarchical namespace enabled), for example:

* **Storage Blob Data Contributor** – read and write
* **Storage Blob Data Reader** – read-only


# Backblaze B2
Source: https://docs.mage.ai/integrations/databases/BackblazeB2



## Prerequisites

Backblaze B2 uses Mage's S3-compatible client, which depends on `boto3`. Docker
and full dev installs already include it. If you installed Mage with a plain
`pip install mage-ai`, install the `s3` extra so the dependency is present:

```bash theme={"system"}
pip install "mage-ai[s3]"
```

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

   ```yaml theme={"system"}
   version: 0.1.1
   default:
     B2_APPLICATION_KEY_ID: ...
     B2_APPLICATION_KEY: ...
     # Optional. Defaults to us-west-004. If your bucket is in another region,
     # uncomment and set this to that region's endpoint (shown on the B2 bucket
     # page), e.g. us-west-001, us-east-005, eu-central-003:
     # B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com
   ```

   For backwards compatibility, the loader also falls back to
   `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_ENDPOINT` (for the
   endpoint URL) when the corresponding `B2_*` keys are not set.

## Using Python block

1. Create a new pipeline or open an existing pipeline.

2. Add a data loader or transformer block (the code snippet below is for a data
   loader).

3. Select `Generic (no template)`.

4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

   ```python theme={"system"}
   from mage_ai.settings.repo import get_repo_path
   from mage_ai.io.config import ConfigFileLoader
   from mage_ai.io.backblaze_b2 import BackblazeB2
   from os import path
   from pandas import DataFrame

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader


   @data_loader
   def load_from_backblaze_b2(**kwargs) -> DataFrame:
       config_path = path.join(get_repo_path(), 'io_config.yaml')
       config_profile = 'default'

       bucket_name = '...'  # Change to your B2 bucket name
       object_key = '...'   # Change to your object key

       return BackblazeB2.with_config(ConfigFileLoader(config_path, config_profile)).load(
           bucket_name,
           object_key,
       )
   ```

5. Run the block.

### Endpoint override

The default endpoint is `https://s3.us-west-004.backblazeb2.com`. If your
bucket lives in another B2 region, add:

```yaml theme={"system"}
default:
  B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com
```

under your profile in the `io_config.yaml` file. `AWS_ENDPOINT` is honored as a fallback.

### Errors

**B2 connection endpoint URL error**

Open the `io_config.yaml` file at the root of your project (e.g.
`default_repo/io_config.yaml`) and confirm that `B2_ENDPOINT_URL` (if set)
matches the region of your B2 bucket.

## Permissions

Ensure the [application key](https://www.backblaze.com/docs/cloud-storage-create-and-manage-app-keys)
you create in the Backblaze B2 console grants the following capabilities on
your bucket:

* `readFiles`
* `writeFiles`
* `listFiles`

These permissions are required to:

* Read data from B2 (e.g. `.csv`, `.parquet`, `.json`)
* Write query results or transformed data to B2
* List contents of a bucket when needed

> [!NOTE]
>
> Two common Backblaze B2 gotchas:
>
> * The **master application key** is not S3-compatible. Create a standard
>   (non-master) application key for use with Mage.
> * App keys restricted to a single bucket may also need the
>   `listAllBucketNames` capability for S3 SDK/integration compatibility.


# BigQuery
Source: https://docs.mage.ai/integrations/databases/BigQuery



![](https://www.vectorlogo.zone/logos/google_bigquery/google_bigquery-ar21.svg)

## Add credentials

<Snippet />

## Required permissions

```
bigquery.datasets.create
bigquery.datasets.get
bigquery.jobs.create
bigquery.readsessions.create
bigquery.readsessions.getData
bigquery.tables.create
```

If you're running queries in the existing BigQuery dataset, make sure your account also have "BigQuery Data Editor"
role for the BigQuery dataset.

## Using SQL blocks

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `BigQuery`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Next to the `Database` label, enter the database name you want this block to
   save data to.
7. Next to the `Save to schema` label, enter the schema name you want this block
   to save data to.
8. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks)
   for more information on write policies).
9. Enter in this test query: `SELECT 1`.
10. Run the block.

## Using Python blocks

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_big_query(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    return BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).load(query)
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to BigQuery:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_big_query(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a BigQuery warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#bigquery
    """
    table_id = 'your-project.your_dataset.your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        table_id,
        if_exists='replace',  # Specify resolution policy if table name already exists
        overwrite_types=None, # Specify the column types to overwrite in a dictionary
    )
```

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config.
You can find the supported types in this [doc](https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema.FIELDS.type). Here is the example code:

```python theme={"system"}
    BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        table_id,
        if_exists='replace',  # Specify resolution policy if table name already exists
        overwrite_types={'col_name': 'STRING'}, # Specify the column types to overwrite
    )
```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types. See [BigQuery data types](https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema.FIELDS.type) | `{'column1': 'STRING', 'column2': 'INTEGER'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |

**Example: Using UPSERT (UPDATE or INSERT)**

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_big_query(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to BigQuery with UPSERT support.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#bigquery
    """
    table_id = 'your-project.your_dataset.your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        table_id,
        if_exists='append',  # Use 'append' with UPSERT
        unique_constraints=['user_id', 'email'],  # Columns that form unique constraint
        unique_conflict_method='UPDATE',  # Update existing rows, insert new ones
    )
```


# Chroma
Source: https://docs.mage.ai/integrations/databases/Chroma



## Credentials

Open the file named `io_config.yaml` at the root of your Mage project and enter chroma required fields:

```yaml theme={"system"}
version: 0.1.1
default:
  CHROMA_COLLECTION: collection_name
  CHROMA_PATH: path of the chroma persisitant storage
```

## Dependencies

The dependency libraries are not installed in the docker image by default. You'll need to add the libraries to
project `requirements.txt` file manually and install them.

```
chromadb>=0.4.17
```

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter with Template. Under "Databases" category you can
   find the "Chroma" template.

* Chroma data loader arguments:
  * n\_results: Number of results to match.
  * collection: collection to use. Otherwise the collection defined in io\_config.yml will be used.
  * query\_texts: the texts or documents used to query.
  * query\_embeddings: the embeddings used to query.
    Either `query_texts` or `query_embeddings` should be available.

* Chroma data exporter arguments:
  * df: Dataframe contains the actual data.
  * collection: collection to use. Otherwise the collection defined in io\_config.yml will be used.
  * document\_column: specify the column in df dataframe that contains documents to be stored in Chroma.
  * id\_column: specify the column in df dataframe that contains the id of the document
  * metadata\_column: specify the column in df dataframe that contains the metadata of the document (metadata should be dictionary format)

3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.


# ClickHouse
Source: https://docs.mage.ai/integrations/databases/ClickHouse



![ClickHouse — open source distributed column-oriented DBMS](https://clickhouse.com/images/ch_gh_logo_rounded.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  # ClickHouse
  CLICKHOUSE_DATABASE: ...
  CLICKHOUSE_HOST: ...
  CLICKHOUSE_INTERFACE: http
  CLICKHOUSE_PASSWORD: ...
  CLICKHOUSE_PORT: 8123
  CLICKHOUSE_USERNAME: ...
```

***

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `ClickHouse`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Enter the optional table name of the table to write to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks) for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.

***

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.clickhouse import ClickHouse
from mage_ai.io.config import ConfigFileLoader
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_clickhouse(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    return ClickHouse.with_config(ConfigFileLoader(config_path, config_profile)).load(query)
```

5. Run the block.

***

## Destination table in Data Exporter

If the destination table does not exist and the `Write policy` is set
to `Replace`, `data exporter` will automatically create a table in `ClickHouse`
with `Engine = Memory` and a default schema inferred from the data.
However, this may not be optimal for various use cases. Since [table creation](https://clickhouse.com/docs/en/sql-reference/statements/create/table)
in `ClickHouse` can involve numerous details, it is strongly advised to
create the destination table manually before loading data to ensure it
meets specific requirements.

If generating the destination table via Mage export method, one can overwrite specific column types
by using the `overwrite_types` dict setting.

Example:

```python theme={"system"}
overwrite_types = {'column_name': 'VARCHAR(255)'}
```

Here is the ClickHouse data exporter code snippet that you can use:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.clickhouse import ClickHouse
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_clickhouse(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Clickhouse database.
    Specify your configuration settings in 'io_config.yaml'.

    """
    database = 'database_name'  # Specify the name of the schema to export data to
    table_name = 'table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    ClickHouse.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        table_name=table_name,
        database=database,
        index=False,  # Specifies whether to include index in exported table
        if_exists='replace',  # Specify resolution policy if table name already exists
        overwrite_types={'column_name': 'VARCHAR(255)'},  # Overwrite column types
    )
```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'VARCHAR(255)', 'column2': 'INTEGER'}` |


# Databricks
Source: https://docs.mage.ai/integrations/databases/Databricks



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=databricks)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  DATABRICKS_ACCESS_TOKEN: access_token
  DATABRICKS_DATABASE: database  # Databricks Catalog
  DATABRICKS_HOST: dbc-123456-abcd.cloud.databricks.com
  DATABRICKS_HTTP_PATH: "/sql/1.0/warehouses/abcd1234"
  DATABRICKS_SCHEMA: schema
```

## Using SQL Block with Databricks

Follow these steps to use a **SQL block** connected to **Databricks** in your pipeline.

1. **Create or open a pipeline**\
   Create a new pipeline or open an existing one.

2. **Add a block**\
   Add a **Data loader**, **Transformer**, **Data exporter**, or **Custom** block.

3. **Select block type**\
   Set the block type to `SQL`.

4. **Configure the data provider**\
   In the `Data provider` dropdown, select `Databricks`.

5. **Choose a profile**\
   In the `Profile` dropdown, select `default` (or the profile you configured your Databricks credentials under).

6. **Configure output saving (optional)**\
   By default, the SQL block saves query results to a table in your Databricks database.

   * **Schema** (optional): Enter the schema where the block’s output should be saved, default to the schema in `io_config.yaml`.
   * **Table** (optional): Enter the table where the block’s output should be saved, default table is generated based on block uuid.
   * **Write policy**: Choose either `Replace` or `Append`. See the [SQL blocks guide](/guides/blocks/sql-blocks) for more details on write policies.
   * If you **don’t want to save results** to an intermediate table, enable the **Use raw SQL** option. See the [Raw SQL documentation](/guides/blocks/sql-blocks#using-raw-sql) for more information.

7. **Enter a test query**\
   Add a simple test query to confirm the connection works:
   ```
   SELECT 1
   ```

8. **Run the block**\
   Click **Run** to execute the query and verify the block works as expected.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select the `Data lakes -> Databricks SQL` template or `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.databricks_sql import DatabricksSQL
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader

@data_loader
def load_data_from_databricks(*args, **kwargs):
    """
    Template for loading data from Databricks with SQL.
    Specify your configuration settings in 'io_config.yaml'.
    """
    query = 'SELECT 1'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with DatabricksSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to Databricks:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.databricks_sql import DatabricksSQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_databricks(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to Databricks.
    Specify your configuration settings in 'io_config.yaml'.
    """
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with DatabricksSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name=schema_name,
            table_name=table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
        )
```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |


# Druid
Source: https://docs.mage.ai/integrations/databases/Druid



![](https://druid.apache.org/img/druid_nav.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  DRUID_HOST: ...
  DRUID_PASSWORD: ...
  DRUID_PATH: /druid/v2/sql/
  DRUID_PORT: 8082
  DRUID_SCHEME: http
  DRUID_USER: ...
```

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Druid`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Select `Use raw SQL`.
7. Enter in this test query: `SELECT 1`.
8. Run the block.


# DuckDB
Source: https://docs.mage.ai/integrations/databases/DuckDB

Execute SQL commands in DuckDB.

![](https://miro.medium.com/v2/resize:fit:1400/format:webp/0*Qcgc4cmJk5lFGBwi.png)

## Credentials

Open the file named `io_config.yaml` at the root of your Mage project and enter DuckDB database name:

```yaml theme={"system"}
version: 0.1.1
default:
  DUCKDB_DATABASE: database
  DUCKDB_SCHEMA: main
```

## How to use

1. Create a new pipeline or edit an existing pipeline.

2. Add a data loader or transformer SQL block.

3. In the block’s header, click the dropdown menu labeled **Connection** and select the option
   labeled **DuckDB**.

4. In the block’s header, click the dropdown menu labeled **Profile** and select the option
   labeled **default**.

   > [!NOTE]
   >
   > If the name of your profile in the `io_config.yaml` file is different,
   > then select the profile name that matches your settings.

5. You can optionally check the box labeled **Use raw SQL**.
   If you do, read more about [how to use SQL blocks with raw SQL](/guides/blocks/sql-blocks#using-raw-sql).

6. Write your SQL statements in the SQL block.

7. Once you’re finished writing your SQL statements,
   click the play button in the top right corner of the block to execute the SQL block and preview
   the results.


# Google Cloud Storage
Source: https://docs.mage.ai/integrations/databases/GoogleCloudStorage



## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

Use **either** a path to your service account key file **or** the key contents as a mapping (e.g. from a secret):

```yaml theme={"system"}
version: 0.1.1
default:
  # Option 1: Path to service account JSON file
  GOOGLE_SERVICE_ACC_KEY_FILEPATH: /path/to/service_account_credentials.json

  # Option 2: Inline credentials (e.g. from env or secrets)
  # GOOGLE_SERVICE_ACC_KEY: "{{ json_value(env_var('GCP_CREDENTIALS_JSON'), 'private_key') }}"
  # Or use mage_secret_var / aws_secret_var etc. for the full credentials object
```

If the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is set (e.g. to the path of your service account JSON), you can also use the IO class without adding keys to `io_config.yaml`; otherwise use `with_config` as shown below.

## Using Python block

1. Create a new pipeline or open an existing pipeline.

2. Add a data loader or transformer block (the code snippet below is for a data
   loader).

3. Select `Generic (no template)`.

4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

   ```python theme={"system"}
   from mage_ai.settings.repo import get_repo_path
   from mage_ai.io.config import ConfigFileLoader
   from mage_ai.io.google_cloud_storage import GoogleCloudStorage
   from os import path
   from pandas import DataFrame

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader


   @data_loader
   def load_from_gcs(**kwargs) -> DataFrame:
       config_path = path.join(get_repo_path(), 'io_config.yaml')
       config_profile = 'default'

       bucket_name = '...'  # Change to your bucket name
       object_key = '...'   # Change to your object key (e.g. path/to/file.parquet)

       return GoogleCloudStorage.with_config(
           ConfigFileLoader(config_path, config_profile)
       ).load(bucket_name, object_key)
   ```

5. Run the block.

### Export data to Google Cloud Storage

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_gcs(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = '...'
    object_key = '...'

    GoogleCloudStorage.with_config(
        ConfigFileLoader(config_path, config_profile)
    ).export(df, bucket_name, object_key)
```

## Supported formats

Google Cloud Storage supports loading and exporting:

* `.csv`
* `.json`
* `.parquet`

## Permissions

Ensure your Google Cloud service account has the appropriate roles, for example:

* **Storage Object Viewer** – to read objects
* **Storage Object Creator** – to write objects
* **Storage Object Admin** – for full read/write/delete


# Google Sheets
Source: https://docs.mage.ai/integrations/databases/GoogleSheets

Mage supports writing/reading from Google Sheets using the Google Sheets API via the gspread library. Read on to learn about reading/writing data to Sheets.

![](https://github.com/mage-ai/mage-ai/assets/59450879/0a10316e-1b0d-4727-858f-c743cc1e296c)

## Configuration

To configure Google Sheets with Mage, you'll need to enable several APIs in your GCP project and add credentials to Mage.

### Enable relevant APIs

Using Google Sheets in Mage requires the following to be enabled via GCP:

* Google Drive API
* Google Sheets API

Additionally, the Google Sheet must be shared with the service account used in Mage.

You can follow [this guide](https://docs.gspread.org/en/latest/oauth2.html) to enable relevant services.

### Add credentials

Adding credentials for Sheets is the same as adding credentials for other Google services, i.e. BigQuery, GCP.

<Snippet />

## Usage

We have two templates for using Google Sheets with Mage. They can be found in Python "Data Loader" and "Data Exporter" templates.

You can see the source code for the templates [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/templates/data_exporters/google_sheets.py) and [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/templates/data_loaders/google_sheets.py).


# Iceberg
Source: https://docs.mage.ai/integrations/databases/Iceberg

Connect to Apache Iceberg tables stored in Amazon S3 using various catalog types

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=iceberg)

## Add credentials

To access **Apache Iceberg tables stored in Amazon S3**, you'll need to configure your AWS credentials.

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (or the profile you are using):

   ```yaml theme={"system"}
   version: 0.1.1
   default:
     AWS_ACCESS_KEY_ID: ...
     AWS_SECRET_ACCESS_KEY: ...
     AWS_REGION: us-west-2  # or the region where your S3 bucket is located
   ```

   > These credentials must have read/write access to the S3 bucket that contains your Iceberg tables.

## Using Python block

You can use Mage to load data from Iceberg tables stored in S3 or export data to Iceberg tables using a configurable Python block.

### Steps

1. **Create or open a pipeline** in your Mage Pro cluster.
2. **Add a block** of type **Data Loader** or **Data Exporter**.
3. From the block template list, choose:\
   **Data lakes → Apache Iceberg**
4. In the generated code block, update the following configuration parameters:
   * `base_uri`: Base S3 URI for the Iceberg warehouse (e.g., `s3://your-bucket-name/warehouse/`) - required for SQL catalog
   * `namespace`: Namespace for the Iceberg catalog (default: `'default'`)
   * `catalog_type`: Type of catalog to use - `'sql'` (default) or `'glue'`
   * `table_name`: Name of the Iceberg table
   * `bucket_name`: Name of your S3 bucket (for exports)
   * `mode`: Write mode for exports - `'append'` (default) or `'overwrite'`
   * `metadata_file`: Optional - used to directly access S3 metadata file when table is not in catalog
5. If you're using a non-default profile, update the `config_profile` field accordingly.
6. **Run the block** to load or export data from your Iceberg table stored on S3.

### Configuration Options

#### Catalog Types

Mage supports multiple catalog types for Iceberg, including:

* **SQL Catalog (default)**: Uses a Postgres-backed catalog to store table metadata
  * Requires `base_uri` to specify the warehouse location
  * Tables can be registered in the catalog or accessed directly via metadata files

* **AWS Glue Catalog**: Uses AWS Glue as the catalog
  * No `base_uri` required
  * Tables must be registered in AWS Glue

Additional catalog types are also supported. Set `catalog_type` to the appropriate value for your catalog implementation.

#### Loading Data

When loading data, you can:

* Load from catalog: If the table is registered in the catalog, just provide `table_name`
* Load from metadata file: If the table is not in the catalog, provide `metadata_file` to access the S3 metadata file directly
  * The metadata file path will be: `{base_uri}{table_name}/metadata/{metadata_file}`

Additional kwargs (e.g., `row_filter`, `selected_fields`, `case_sensitive`, `snapshot_id`) can be passed into the `load()` method to customize the data retrieval. These parameters are passed through to pyiceberg's scan method.

#### Scan Parameters

The following parameters can be passed to the `load()` method to customize data retrieval. These parameters are passed through to pyiceberg's `scan()` method. For the complete method signature, see the [pyiceberg source code](https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L1133-L1135).

| Parameter | Type | Default | Description | Example |
| - | - | - | - | - |
| `row_filter` | `str` or `BooleanExpression` | `AlwaysTrue()` | A string or BooleanExpression that describes the desired rows | `'id > 100'` or `'status == "active"'` |
| `selected_fields` | `tuple[str]` | `("*",)` | A tuple of strings representing the column names to return in the output dataframe | `('id', 'name', 'created_at')` |
| `case_sensitive` | `bool` | `True` | If `True`, column matching is case sensitive | `True` |
| `snapshot_id` | `int` or `None` | `None` | Optional Snapshot ID to time travel to. If `None`, scans the table as of the current snapshot ID | `12345` |
| `limit` | `int` or `None` | `None` | An integer representing the number of rows to return in the scan result. If `None`, fetches all matching rows | `1000` |

#### Exporting Data

When exporting data, you can:

* **Append mode** (default): Adds new data to the existing table
* **Overwrite mode**: Replaces all data in the table with the new data

### Example: Loading Data

```python SQL Catalog theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader

# Using SQL catalog
iceberg = Iceberg.with_config(
    ConfigFileLoader(config_path, config_profile),
    base_uri='s3://my-bucket/warehouse/',
    namespace='analytics',
    catalog_type='sql',
)

# Load from catalog
df = iceberg.load('my_table')

# Or load from metadata file (if not in catalog)
df = iceberg.load('my_table', metadata_file='00000-abc123.metadata.json')
```

```python AWS Glue Catalog theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader

# Using AWS Glue catalog
iceberg = Iceberg.with_config(
    ConfigFileLoader(config_path, config_profile),
    namespace='analytics',
    catalog_type='glue',
)

df = iceberg.load('my_table')
```

```python With Scan Parameters theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader

iceberg = Iceberg.with_config(
    ConfigFileLoader(config_path, config_profile),
    base_uri='s3://my-bucket/warehouse/',
    namespace='analytics',
    catalog_type='sql',
)

# Loading with additional scan parameters
df = iceberg.load(
    'my_table',
    row_filter='id > 100',  # Filter rows
    selected_fields=['id', 'name', 'created_at'],  # Select specific fields
    case_sensitive=True,  # Case-sensitive field matching
    snapshot_id=12345,  # Read from specific snapshot
    limit=1000,  # Limit number of rows
)
```

### Example: Exporting Data

```python Append Mode theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader

iceberg = Iceberg.with_config(
    ConfigFileLoader(config_path, config_profile),
    namespace='analytics',
    catalog_type='sql',
)

# Append data to table
iceberg.export(
    df,
    bucket_name='my-bucket',
    table_name='my_table',
    mode='append',
)
```

```python Overwrite Mode theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader

iceberg = Iceberg.with_config(
    ConfigFileLoader(config_path, config_profile),
    namespace='analytics',
    catalog_type='sql',
)

# Overwrite table with new data
iceberg.export(
    df,
    bucket_name='my-bucket',
    table_name='my_table',
    mode='overwrite',
)
```

### Additional Methods

The Iceberg integration also provides methods for managing tables and namespaces:

* `list_namespaces()`: List all namespaces in the catalog
* `list_tables(namespace)`: List all tables in a namespace
* `get_table_schema(table_name, namespace)`: Get the schema of a table
* `drop_table(table_name, namespace)`: Drop (delete) a table
* `drop_namespace(namespace)`: Drop (delete) a namespace

### Notes

* The Iceberg integration supports direct access to tables through metadata stored in S3 when tables are not registered in the catalog.
* For SQL catalog, `base_uri` should point to your warehouse location and include a trailing slash (e.g., `s3://bucket/warehouse/`).
* For AWS Glue catalog, tables must be registered in AWS Glue.
* When using `metadata_file`, the path is constructed as: `{base_uri}{table_name}/metadata/{metadata_file}`.
* Additional scan parameters (e.g., `row_filter`, `selected_fields`, `case_sensitive`, `snapshot_id`) can be passed to the `load()` method.

## Using Iceberg with PySpark

Mage Pro supports reading from and writing to [Apache Iceberg](https://iceberg.apache.org/) tables using PySpark, enabling scalable and efficient data lake operations.

### Example code

Using Iceberg with Google Cloud Storage (GCS)

```python theme={"system"}
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("IcebergWithGCS") \
    .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.1") \
    .config("spark.jars", "/opt/spark/jars/gcs-connector-hadoop3-2.2.5-shaded.jar") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.iceberg.type", "hadoop") \
    .config("spark.sql.catalog.iceberg.warehouse", "gs://mage_icerberg_test/test") \
    .config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") \
    .config("spark.hadoop.google.cloud.auth.service.account.enable", "true") \
    .config("spark.hadoop.google.cloud.auth.service.account.json.keyfile", "/home/src/default_repo/your_google_service_account_key.json") \
    .getOrCreate()
```

Create and Write an Iceberg Table

```python theme={"system"}
# Sample DataFrame
df = spark.createDataFrame([
    (1, "apple"),
    (2, "banana"),
    (3, "banana"),
], ["id", "fruit"])

# Create the database if it doesn't exist
spark.sql("CREATE DATABASE IF NOT EXISTS iceberg.db_name")

# Write data to an Iceberg table
df.writeTo("iceberg.db_name.iceberg_table") \
  .using("iceberg") \
  .createOrReplace()
```

### Notes

* `iceberg.db_name.iceberg_table` uses the Hadoop catalog type and stores metadata in the specified GCS path.
* You can modify the catalog configs to use **Hive**, **Glue**, or **Nessie** depending on your architecture.
* For AWS S3, update the warehouse path and authentication configurations accordingly.
* The Google service account key file path must be accessible inside the Mage Pro cluster.
* You can run this code inside a block in Mage Pro batch pipeline.


# Microsoft Fabric Warehouse
Source: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=workspaces)

![](https://learn.microsoft.com/en-us/fabric/media/fabric-icon.png)

## Overview

Microsoft Fabric Warehouse is a fully managed, cloud-native data warehouse built on Azure Synapse SQL. It provides enterprise-grade performance, scalability, and security for your data analytics workloads.

This integration allows you to:

* Connect to Microsoft Fabric Data Warehouse using Azure AD Service Principal authentication (with optional token-based auth via `AZURE_TENANT_ID`)
* Run SQL in pipelines via **SQL blocks** (transformer) or Python blocks
* Export data using batched row-based inserts (standard export)
  This integration allows you to connect to Microsoft Fabric Data Warehouse using Azure AD Service Principal authentication and perform SQL operations directly from Mage pipelines.

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  AZURE_CLIENT_ID: '12345678-1234-1234-1234-123456789012'
  AZURE_CLIENT_SECRET: 'your-azure-ad-service-principal-secret'
  MICROSOFT_FABRIC_WAREHOUSE_NAME: 'your_database_name'
  MICROSOFT_FABRIC_WAREHOUSE_ENDPOINT: 'your-endpoint-here'
  MICROSOFT_FABRIC_WAREHOUSE_SCHEMA: 'dbo'

  # Optional: use token-based auth (recommended for Fabric). Must be a valid GUID.
  # AZURE_TENANT_ID: '87654321-4321-4321-4321-210987654321'
```

### Configuration Parameters

| Parameter | Description | Required | Example |
| - | - | - | - |
| `AZURE_CLIENT_ID` | Azure AD Service Principal Application ID (GUID format) | ✅ | `12345678-1234-1234-1234-123456789012` |
| `AZURE_CLIENT_SECRET` | Azure AD Service Principal Secret | ✅ | `your-secret-here` |
| `MICROSOFT_FABRIC_WAREHOUSE_NAME` | Name of your Fabric Data Warehouse database | ✅ | `my_database` |
| `MICROSOFT_FABRIC_WAREHOUSE_ENDPOINT` | Fabric workspace endpoint URL | ✅ | `abcde12345.datawarehouse.fabric.microsoft.com` |
| `MICROSOFT_FABRIC_WAREHOUSE_SCHEMA` | Default schema for operations | ❌ | `dbo` (default) |
| `AZURE_TENANT_ID` | Azure AD tenant ID (GUID format). When set, connection uses **token-based auth** (OAuth2 token injected via ODBC); recommended for Fabric. When omitted, connection uses UID/PWD with ActiveDirectoryServicePrincipal. | ❌ | `87654321-4321-4321-4321-210987654321` |

## Service Principal Setup

To use service principals with Microsoft Fabric Data Warehouse, you need to:

1. **Create a service principal** in Microsoft Entra ID
2. **Enable service principal access** in Microsoft Fabric
3. **Assign appropriate permissions** to the service principal

For detailed setup instructions, see:

* [Register a Microsoft Entra app and create a service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal)
* [Service principal in Fabric Data Warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/service-principals#prerequisites)

***

## Using SQL blocks

Run SQL directly against Microsoft Fabric Warehouse from a **SQL block** (transformer):

1. Add a **Transformer** block and set its language to **SQL**.
2. In the block’s connection settings, set **Connection** to **Microsoft Fabric** and **Profile** to your `io_config.yaml` profile (e.g. `default`).
3. Optionally set **Schema** (default `dbo`) and **Use raw SQL** to run statements as-is.
4. Write your SQL. Reference upstream block output with **`{{ df_1 }}`**, **`{{ df_2 }}`**, etc. The block uses the profile’s credentials and schema.

A SQL block writes data to the warehouse in two ways:

1. **Materialized output (no “Use raw SQL”)**
   When “Use raw SQL” is **off**, the block runs your query and **exports the result** to a table (creates or appends) by executing the query string in the warehouse. The warehouse runs the SQL directly.

2. **Upstream data referenced in SQL (`{{ df_1 }}`, `{{ df_2 }}`, …)**
   When your SQL references **`{{ df_1 }}`** (or `{{ df_2 }}`, etc.), the upstream block’s output must be available in the warehouse. If not already there, Mage **uploads** that DataFrame using batched inserts (see [Export limits](#export-limits)).

***

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.microsoft_fabric_warehouse import MicrosoftFabricWarehouse
from os import path

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_fabric_warehouse(*args, **kwargs):
    """
    Template for loading data from Microsoft Fabric Warehouse.
    Specify your configuration settings in 'io_config.yaml'.
    Set the following in your io_config:

    Docs: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
    """
    query = 'SELECT * FROM your_table LIMIT 1000'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MicrosoftFabricWarehouse.with_config(ConfigFileLoader(config_path, config_profile)) as warehouse:
        return warehouse.load(query)
```

5. Run the block.

### Export a dataframe

Use a data exporter block and call `warehouse.export()`. Data is inserted in batches (see [Export limits](#export-limits)).
Here is an example code snippet to export a dataframe to Microsoft Fabric Warehouse:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.microsoft_fabric_warehouse import MicrosoftFabricWarehouse
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_fabric_warehouse(df: DataFrame, **kwargs) -> None:
    """
    Export to Microsoft Fabric Warehouse.
    Set credentials (and optional AZURE_TENANT_ID for token auth) in io_config.yaml.
    """
    schema_name = 'dbo'
    table_name = 'your_table_name'
    Template for exporting data to Microsoft Fabric Warehouse.
    Specify your configuration settings in 'io_config.yaml'.
    Set the following in your io_config:

    Docs: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
    """
    schema_name = 'dbo'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MicrosoftFabricWarehouse.with_config(ConfigFileLoader(config_path, config_profile)) as warehouse:
        warehouse.export(
            df,
            schema_name,
            table_name,
            if_exists='replace',
            index=False,
            if_exists='replace',  # Specify resolution policy if table name already exists
            index=False,  # Specifies whether to include index in exported table
        )
```

### Standard export

Export uses batched `INSERT` statements. Use the same `io_config.yaml` profile as for loading. Optionally set `AZURE_TENANT_ID` (GUID) for token-based auth (recommended for Fabric).

**Configuration:** See [Add credentials](#add-credentials). Optionally add `AZURE_TENANT_ID` for token auth.

**Code:** Call `warehouse.export(df, schema_name, table_name, ...)` as in the example above. See [Export limits](#export-limits) for the column limit.

***

### Export limits

* **Column count:** SQL Server limits the number of parameters per statement (2,000). If your DataFrame has **more than 2,000 columns**, export raises a clear error. Reduce the number of columns or use a different upload method (e.g. export a subset of columns, or stage data elsewhere and load via another path).

***

### Custom types

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.microsoft_fabric_warehouse import MicrosoftFabricWarehouse
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_fabric_warehouse(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to Microsoft Fabric Warehouse.
    Specify your configuration settings in 'io_config.yaml'.
    Set the following in your io_config:

    Docs: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
    Docs: /integrations/databases/MicrosoftFabricWarehouse
    """
    schema_name = 'dbo'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with MicrosoftFabricWarehouse.with_config(ConfigFileLoader(config_path, config_profile)) as warehouse:
        warehouse.export(
            df,
            schema_name,
            table_name,
            if_exists='replace',  # Specify resolution policy if table name already exists
            overwrite_types=overwrite_types,
            index=False,  # Specifies whether to include index in exported table
        )
```

## Troubleshooting errors

### Connection Timeout Errors

**Error**: `Login timeout expired` or `Connection timeout`

> [!TIP]
>
> *"Verify your Service Principal credentials and network connectivity"*

**Solutions**:

* Ensure `AZURE_CLIENT_ID` is a valid GUID format
* Verify `AZURE_CLIENT_SECRET` is correct and not expired
* Check that the Service Principal has appropriate permissions on your Fabric workspace
* Ensure the Service Principal is not disabled in Azure AD
* Verify network connectivity to the endpoint
* Check if corporate firewall or proxy is blocking the connection
* Try increasing connection timeout settings

### TCP Provider Errors

**Error**: `TCP Provider: Error code 0x2746 (10054)` or `Connection reset by peer`

> [!TIP]
>
> *"Verify your endpoint URL and network connectivity"*

**Solutions**:

* Ensure `MICROSOFT_FABRIC_WAREHOUSE_ENDPOINT` is correct and accessible
* Check network connectivity to the endpoint (ping, telnet, etc.)
* Verify the endpoint is accessible from your current location
* Ensure the endpoint is not blocked by firewall rules
* Check if Azure region is experiencing connectivity issues
* Verify the Microsoft Fabric service is operational
* Try connecting from a different network to isolate the issue

### Database Object Errors

**Error**: `Invalid object name 'table_name'` or `Database 'database_name' does not exist`

> [!TIP]
>
> *"Verify your database name, schema, and table existence"*

**Solutions**:

* Ensure `MICROSOFT_FABRIC_WAREHOUSE_NAME` matches exactly (case-sensitive)
* Check that the Service Principal has access to the database
* Verify the database exists in your Fabric workspace
* Ensure the database is not paused or offline
* Verify the table exists in the specified schema
* Check if you're using the correct schema name (default is 'dbo')
* Ensure table names are properly quoted if they contain special characters

### Authentication Errors

**Error**: `Failed to authenticate with Azure AD` or `Login failed for user` or `Could not login because the authentication failed.`
**Error**: `Failed to authenticate with Azure AD` or `Login failed for user`

> [!TIP]
>
> *"Verify your Service Principal credentials and permissions"*

**Solutions**:

* Ensure `AZURE_CLIENT_ID` is a valid GUID format
* Verify `AZURE_CLIENT_SECRET` is correct and not expired
* Check that the Service Principal has appropriate permissions on your Fabric workspace
* Ensure the Service Principal is not disabled in Azure AD
* Verify the Service Principal has the correct role assignments
* Check if the Service Principal's permissions have been revoked

### Export column limit

**Error**: `DataFrame has X columns, which exceeds SQL Server parameter limit (2000)`

> [!TIP]
>
> *"Reduce the number of columns or use a different upload method"*

**Solutions**:

* Export a subset of columns (e.g. drop or select only the columns you need)
* Split the DataFrame into multiple tables
* Use a different load path if you have very wide tables (e.g. stage to storage and load via another tool)

### Schema Errors

**Error**: `Schema 'schema_name' does not exist`

> [!TIP]
>
> *"Verify your schema name and permissions"*

**Solutions**:

* Ensure `MICROSOFT_FABRIC_WAREHOUSE_SCHEMA` is correct
* Check that the Service Principal has access to the schema
* Verify the schema exists in your database
* Use 'dbo' as the default schema if unsure
* Check if the schema name is case-sensitive
* Verify the Service Principal has CREATE/ALTER permissions on the schema

## Best Practices

1. **Service Principal Management**:
   * Use dedicated Service Principals for different environments
   * Rotate secrets regularly
   * Grant minimum required permissions

2. **Connection Management**:
   * Always use context managers (`with` statements) when possible
   * Close connections explicitly when not using context managers
   * Test connections before running large operations

3. **Performance Optimization**:
   * Export uses batched inserts; keep tables under the [export column limit](#export-limits) (2,000 columns). For very wide DataFrames, reduce columns or use another load path.
   * Use appropriate `if_exists` policies ('replace', 'append', 'fail')
   * Consider using `overwrite_types` for optimal column types
   * Use LIMIT clauses in queries when testing

4. **Security**:
   * Never hardcode credentials in your code
   * Use `io_config.yaml` for all configuration
   * Ensure `io_config.yaml` is not committed to version control
   * Use environment variables for sensitive values in production

## Related Documentation

* [Microsoft Fabric Documentation](https://learn.microsoft.com/en-us/fabric/)
* [Microsoft Fabric Warehouse Documentation](https://learn.microsoft.com/en-us/fabric/data-warehouse/)
* [Azure AD Service Principal Setup](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
* [ODBC Driver Installation](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server)


# Microsoft SQL Server
Source: https://docs.mage.ai/integrations/databases/MicrosoftSQLServer



![](https://www.commvault.com/wp-content/uploads/2019/08/sql-server_logo.jpg)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  MSSQL_DATABASE: database
  MSSQL_SCHEMA: schema
  MSSQL_DRIVER: "ODBC Driver 18 for SQL Server"
  MSSQL_HOST: host
  MSSQL_PASSWORD: password
  MSSQL_PORT: 1433
  MSSQL_USER: SA

  # (Optional) SSH tunnel config
  MSSQL_CONNECTION_METHOD: direct
  MSSQL_SSH_HOST:
  MSSQL_SSH_PORT: 22
  MSSQL_SSH_USERNAME:
  MSSQL_SSH_PASSWORD:
  MSSQL_SSH_PKEY:
```

### SSH tunneling

> [!NOTE]
>
> SSH tunneling is a Mage Pro only feature.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mssql)

If you want to connect to SQL Server with SSH tunnel, update the value of `MSSQL_CONNECTION_METHOD` to `ssh_tunnel`, and enter the values for keys with prefix `MSSQL_SSH`.

| Key | Description | Sample value |
| - | - | - |
| `MSSQL_SSH_HOST` | The host of the intermediate bastion server. | `123.45.67.89` |
| `MSSQL_SSH_PORT` | The port of the intermediate bastion server. Default value: 22 | `22` |
| `MSSQL_SSH_USERNAME` | The username used to connect to the bastion server. | `username` |
| `MSSQL_SSH_PASSWORD` | (Optional) The password used to connect to the bastion server. It should be set if you authenticate with the bastion server with password. | `password` |
| `MSSQL_SSH_PKEY` | (Optional) The path to the private key used to connect to the bastion server. It should be set if you authenticate with the bastion server with private key. | `/path/to/private/key` |

> [!NOTE]
>
> When using SSH tunnel, the `fast_execute` option will automatically be disabled to ensure reliable connections through the tunnel.

## Dependencies

To connect to the Microsoft SQL Server, you'll need to make sure the driver is installed.
By default, ODBC Driver 18 is installed in the docker image.
If you want to use other ODBC Driver versions, you'll need to build a custom docker image (use Mage image as the base image) and install the drivers.
Here is the doc for installing ODBC drivers for SQL Server: [https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server)

***

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider`/`Connection` dropdown, select `Microsoft SQL Server`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Enter the schema and optional table name of the table to write to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks) for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mssql import MSSQL
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_mssql(*args, **kwargs):
    """
    Template for loading data from a MSSQL database.
    Specify your configuration settings in 'io_config.yaml'.
    Set the following in your io_config:

    Docs: /integrations/databases/MicrosoftSQLServer
    """
    query = 'Your MSSQL query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MSSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to MSSQL:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mssql import MSSQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mssql(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a MSSQL database.
    Specify your configuration settings in 'io_config.yaml'.
    Set the following in your io_config:

    Docs: /integrations/databases/MicrosoftSQLServer
    """
    schema_name = 'dbo'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MSSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            fast_execute=True,  # Use fast_executemany option to speed up bulk inserting rows
            unique_conflict_method='UPDATE',  # Specify method to resolve row conflicts
            unique_constraints=['col'],
        )
```

6. Custom types

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config

Here is an example code snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mssql import MSSQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mssql(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a MSSQL database.
    Specify your configuration settings in 'io_config.yaml'.
    Set the following in your io_config:

    Docs: /integrations/databases/MicrosoftSQLServer
    """
    schema_name = 'dbo'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with MSSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            overwrite_types=overwrite_types,
        )
```

## Troubleshooting errors

### `error: ODBC SQL type -155 is not yet supported.`

> [!TIP]
>
> *“I changed the datetime with timezone data type to a datetime and it starting working”*

* [Windy Martin](https://mageai.slack.com/archives/C068SHY6YG6/p1705874712639659?thread_ts=1705668895.141169\&cid=C068SHY6YG6)


# MongoDB
Source: https://docs.mage.ai/integrations/databases/MongoDB



![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/93/MongoDB_Logo.svg/2560px-MongoDB_Logo.svg.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  # Specify either the connection string or the (host, password, user, port) to connect to MongoDB.
  MONGODB_CONNECTION_STRING: "mongodb://{username}:{password}@{host}:{port}/"
  MONGODB_HOST: host
  MONGODB_PORT: 27017
  MONGODB_USER: user
  MONGODB_PASSWORD: password
  MONGODB_DATABASE: database
  MONGODB_COLLECTION: collection
```

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mongodb import MongoDB
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_from_mongodb(*args, **kwargs):
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    query = {}

    return MongoDB.with_config(ConfigFileLoader(config_path, config_profile)).load(
        query=query,
        collection='collection_name',
    )
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to MongoDB:

```python theme={"system"}
from os import path

from pandas import DataFrame

from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mongodb import MongoDB

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mongodb(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    MongoDB.with_config(ConfigFileLoader(config_path, config_profile)).export(
        df,
        collection='collection_name',
    )
```


# MotherDuck
Source: https://docs.mage.ai/integrations/databases/MotherDuck

Execute SQL commands against a MotherDuck database.

![](https://github.com/mage-ai/mage-ai/assets/59450879/0e410c12-73a0-43d8-97fe-396772d2767b)

> [!NOTE]
>
> The MotherDuck integration is supported in Mage versions `>=0.9.63`

## Credentials

Open the file named `io_config.yaml` at the root of your Mage project and enter DuckDB database name & schema.
To connect to a MotherDuck database, you'll need to add a `MOTHERDUCK_TOKEN` and a `md:` to the database name.

```yaml theme={"system"}
version: 0.1.1
default:
  DUCKDB_DATABASE: md:database
  DUCKDB_SCHEMA: main
  MOTHERDUCK_TOKEN: token
```

## How to use

1. Create a new pipeline or edit an existing pipeline.

2. Add a data loader or transformer SQL block.

3. In the block’s header, click the dropdown menu labeled **Connection** and select the option
   labeled **DuckDB**.

4. In the block’s header, click the dropdown menu labeled **Profile** and select the option
   labeled **default**.

   > [!NOTE]
   >
   > If the name of your profile in the `io_config.yaml` file is different,
   > then select the profile name that matches your settings.

5. You can optionally check the box labeled **Use raw SQL**.
   If you do, read more about [how to use SQL blocks with raw SQL](/guides/blocks/sql-blocks#using-raw-sql).

6. Write your SQL statements in the SQL block.

7. Once you’re finished writing your SQL statements,
   click the play button in the top right corner of the block to execute the SQL block and preview
   the results.


# MySQL
Source: https://docs.mage.ai/integrations/databases/MySQL



![](https://user-images.githubusercontent.com/78053898/198753513-4a149790-853a-4dcd-8c93-388f84ef6aeb.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

### Standard connection

```yaml theme={"system"}
version: 0.1.1
default:
  MYSQL_DATABASE: ...
  MYSQL_HOST: ...
  MYSQL_PASSWORD: ...
  MYSQL_PORT: 3306
  MYSQL_USER: root
```

### SSH tunnel connection

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mysql)

If your MySQL database is in a private network and you need to connect via an SSH bastion host, Mage Pro supports configuring an SSH tunnel directly in io\_config.yaml:

```yaml theme={"system"}
version: 0.1.1
default:
  MYSQL_DATABASE: ...
  MYSQL_HOST: ...
  MYSQL_PASSWORD: ...
  MYSQL_PORT: 3306
  MYSQL_USER: root

  # Pro only: SSH tunnel settings
  MYSQL_SSH_HOST: bastion.example.com
  MYSQL_SSH_PORT: 22
  MYSQL_SSH_USERNAME: ec2-user
  # Use one of the following for authentication:
  MYSQL_SSH_PKEY: /path/to/private/key.pem   # path to SSH private key
  MYSQL_SSH_PASSWORD: your_ssh_password      # only if using password auth
```

* `MYSQL_SSH_HOST`: Hostname or IP of the SSH bastion.
* `MYSQL_SSH_PORT`: SSH port (default 22).
* `MYSQL_SSH_USERNAME`: SSH user to log in as.
* `MYSQL_SSH_PKEY`: Path to private key file (preferred).
* `MYSQL_SSH_PASSWORD`: Password for SSH user (alternative to key).

⚠️ You only need either `MYSQL_SSH_PKEY` or `MYSQL_SSH_PASSWORD`, not both.

***

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider`/`Connection` dropdown, select `Mysql`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Enter the optional table name of the table to write to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks) for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.

***

## Using Python block

### Using MySQL connector

```python theme={"system"}
import mysql.connector

conn = mysql.connector.connect(
    host='...',
    password='...',
    port=3306,
    user='root',
    database='...',
)

sql = """
SELECT
  id
  , email
FROM auth_user
"""


cursor = conn.cursor()
cursor.execute(sql)

rows = cursor.fetchall()

import pandas as pd

df = pd.DataFrame(rows, columns=['id', 'email'])
df.to_csv('default_repo/users.csv', index=False)
```

***

### Using Mage's MySQL python blocks

1. Create a Data Loader block from the MySQL Template

Example snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path

@data_loader
def load_data_from_mysql(*args, **kwargs):
    """
    Template for loading data from a MySQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#mysql
    """
    query = 'Your MySQL query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)

```

2. Create a Data Exporter block from the MySQL Template

Example snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a MySQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#mysql
    """
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            None,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
        )

```

3. Custom types

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config

Here is an example code snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a MySQL database.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#mysql
    """
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            None,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            overwrite_types=overwrite_types,
        )

```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'VARCHAR(255)'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |

**Example: Using UPSERT (UPDATE or INSERT)**

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a MySQL database with UPSERT support.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#mysql
    """
    table_name = 'your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            None,
            table_name,
            index=False,
            if_exists='append',  # Use 'append' with UPSERT
            unique_constraints=['user_id', 'email'],  # Columns that form unique constraint
            unique_conflict_method='UPDATE',  # Update existing rows, insert new ones
        )
```


# OracleDB
Source: https://docs.mage.ai/integrations/databases/OracleDB



## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  ORACLEDB_USER: ...
  ORACLEDB_PASSWORD: ...
  ORACLEDB_HOST: ...
  ORACLEDB_PORT: 1521
  ORACLEDB_SERVICE: ...   # Service name (e.g. ORCL, XEPDB1)
  ORACLEDB_MODE: thin      # Optional: 'thin' (default) or 'thick'
```

Use `ORACLEDB_MODE: thick` only if you need the Oracle thick mode (e.g. for advanced features); otherwise `thin` is recommended.

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Oracle`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Enter your query (e.g. `SELECT 1 FROM DUAL`) and run the block.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.oracledb import OracleDB
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_oracledb(**kwargs) -> DataFrame:
    query = 'SELECT 1 FROM DUAL'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with OracleDB.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

### Export a dataframe to OracleDB

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.oracledb import OracleDB
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_oracledb(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    table_name = 'your_table_name'

    with OracleDB.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(df, table_name=table_name, if_exists='replace')
```

## Connection modes

* **thin** (default): Pure Python driver; no Oracle Client installation required.
* **thick**: Uses Oracle Client libraries; required for some advanced scenarios. Ensure Oracle Instant Client or full client is installed and configured when using `ORACLEDB_MODE: thick`.


# Pinot
Source: https://docs.mage.ai/integrations/databases/Pinot



![](https://pinot.apache.org/img/pinot-navbar-logo-722f37.svg)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  PINOT_HOST: ...
  PINOT_PASSWORD: ...
  PINOT_PATH: /query/sql
  PINOT_PORT: 8000
  PINOT_SCHEME: http
  PINOT_USER: ...
```

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader.
3. Select `Pinot`.
4. You will get the below code snippet:

```python theme={"system"}
from mage_ai.data_preparation.repo_manager import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.pinot import Pinot
from os import path
if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test


@data_loader
def load_data_from_pinot(*args, **kwargs):
    """
    Template for loading data from a Pinot warehouse.
    Specify your configuration settings in 'io_config.yaml'.
    Docs: https://docs.mage.ai/design/data-loading#pinot
    """
    query = 'your Pinot query'  # Specify your SQL query here
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Pinot.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)


@test
def test_output(output, *args) -> None:
    """
    Template code for testing the output of the block.
    """
    assert output is not None, 'The output is undefined'
```

5. Enter the SQL query in place of 'your Pinot query'.
6. Run the block.


# PostgreSQL
Source: https://docs.mage.ai/integrations/databases/PostgreSQL



![](https://user-images.githubusercontent.com/78053898/198754399-3b594af0-cf84-41d3-8faf-e1f6e4c5182d.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_DBNAME: ...
  POSTGRES_HOST: ...
  POSTGRES_PASSWORD: ...
  POSTGRES_PORT: ...
  POSTGRES_USER: ...
  POSTGRES_CONNECTION_METHOD: direct
  POSTGRES_SSH_HOST:
  POSTGRES_SSH_PORT: 22
  POSTGRES_SSH_USERNAME:
  POSTGRES_SSH_PASSWORD:
  POSTGRES_SSH_PKEY:
```

### Pro-only fields

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=postgresql)

```yaml theme={"system"}
  POSTGRES_SSL_MODE:
  POSTGRES_SSL_ROOTCERT:
  POSTGRES_SSL_CERT:
  POSTGRES_SSL_KEY:
```

**Notes:**

* `POSTGRES_SSL_MODE` corresponds to the `sslmode` parameter in psycopg2 (e.g., `require`, `verify-full`).
* `POSTGRES_SSL_ROOTCERT`, `POSTGRES_SSL_CERT`, and `POSTGRES_SSL_KEY` should be absolute paths to your SSL files on disk.

### SSH tunneling

If you want to connect to Postgres cluster with ssh tunnel, update the value of `POSTGRES_CONNECTION_METHOD` to `ssh_tunnel`, and enter the values for keys with prefix `POSTGRES_SSH`.

| Key | Description | Sample value |
| - | - | - |
| `POSTGRES_SSH_HOST` | The host of the intermediate bastion server. | `123.45.67.89` |
| `POSTGRES_SSH_PORT` | The port of the intermediate bastion server. Default value: 22 | `22` |
| `POSTGRES_SSH_USERNAME` | The username used to connect to the bastion server. | `username` |
| `POSTGRES_SSH_PASSWORD` | (Optional) The password used to connect to the bastion server. It should be set if you authenticate with the bastion server with password. | `password` |
| `POSTGRES_SSH_PKEY` | (Optional) The path to the private key used to connect to the bastion server. It should be set if you authenticate with the bastion server with private key. | `/path/to/private/key` |

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `PostgreSQL`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Next to the `Save to schema` label, enter the schema name you want this block
   to save data to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks) for more
   information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_postgres(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to Postgres:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            drop_table_on_replace=False,   # Whether to drop the table when "if_exists" param is set to "replace"
            allow_reserved_words=True,
            unique_conflict_method='UPDATE',
            unique_constraints=['col'],
        )

```

6. Custom types

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config

Here is an example code snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            allow_reserved_words=True,
            unique_conflict_method='UPDATE',
            unique_constraints=['col'],
            overwrite_types=overwrite_types,
        )

```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| cascade\_on\_drop | Whether to cascade the table drop. | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'VARCHAR'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. | 'UPDATE' (default: None) |
| unique\_constraints | The unique constraints of the table. | \['col1', 'col2'] (default: None) |

#### Pro-only fields

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=postgresql)

| Field name | Description | Example values |
| - | - | - |
| batch\_write | Use `COPY` command to write to PostgreSQL to improve performance. | True/False (default: False) |


# Power BI
Source: https://docs.mage.ai/integrations/databases/PowerBI



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=powerbi)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

**Required:** Azure subscription ID and Power BI workspace ID. For authentication you can use either Azure AD (service principal) or default credentials.

```yaml theme={"system"}
version: 0.1.1
default:
  AZURE_SUBSCRIPTION_ID: your_azure_subscription_id
  POWER_BI_WORKSPACE_ID: your_power_bi_workspace_id

  # Optional: Service principal (if not using DefaultAzureCredential)
  AZURE_CLIENT_ID: ...
  AZURE_CLIENT_SECRET: ...
  AZURE_TENANT_ID: ...
```

If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are omitted, the client uses `DefaultAzureCredential` (e.g. Azure CLI login or managed identity). The service principal or user must have access to the Power BI workspace and the **Power BI Service** API.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

### Trigger a dataset refresh

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.power_bi import PowerBI
from os import path

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def trigger_power_bi_refresh(**kwargs):
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    dataset_id = '...'  # Your Power BI dataset ID

    client = PowerBI.with_config(ConfigFileLoader(config_path, config_profile))
    client.load(dataset_id)
```

### Export data to a Power BI dataset

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.power_bi import PowerBI
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_power_bi(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    dataset_id = '...'

    client = PowerBI.with_config(ConfigFileLoader(config_path, config_profile))
    client.export(df, dataset_id=dataset_id)
```

### Check if a dataset exists

```python theme={"system"}
client = PowerBI.with_config(ConfigFileLoader(config_path, config_profile))
exists = client.exists('your-dataset-id')
```

## Permissions

* Your Azure AD app or user needs the **Power BI Service** scope (e.g. `https://analysis.windows.net/powerbi/api/.default`).
* The app or user must have access to the specified Power BI workspace and dataset (e.g. Admin, Member, or Contributor on the workspace).


# Qdrant
Source: https://docs.mage.ai/integrations/databases/Qdrant



## Credentials

Open the file named `io_config.yaml` at the root of your Mage project and enter qdrant required fields:

```yaml theme={"system"}
version: 0.1.1
default:
  QDRANT_COLLECTION: collection_name
  QDRANT_PATH: path of the qdrant persisitant storage
```

## Dependencies

The dependency libraries are not installed in the docker image by default. You'll need to add the libraries to
project `requirements.txt` file manually and install them.

```
qdrant-client==1.6.9
sentence-transformers==2.2.2
```

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter using the Qdrant template under the "Databases" category.
   Both the data loader and exporter use SentenceTransformer 'all-MiniLM-L6-v2' as the default embedding function.
3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.

## Available functions

* Qdrant data loader arguments:
  * limit\_results (int): Number of results to return.
  * query\_vector (List): vector lit used for query.
  * collection\_name (str): name of the collection. Default to use the name defined in io\_config.yaml.

* Qdrant data exporter arguments:
  * df (DataFrame): Data to export.
  * document\_column (str): Column name containing documents to export.
  * id\_column (str): Column name of the id. Default will use index in df.
  * vector\_column (str): Column name of the vector. Will use default encoder to auto generate query vector.
  * collection\_name (str): name of the collection. Default to use the name defined in io\_config.yaml.
  * vector\_size (int): dimension size of vector.
  * distance (models.Distance): distance metric to use.

At the same time there is `create_collection` function can be used in your block to create new collection.


# Redshift
Source: https://docs.mage.ai/integrations/databases/Redshift



![](https://user-images.githubusercontent.com/78053898/198754422-f3800160-b36a-4e3c-af78-cf322e55a7d3.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  REDSHIFT_DBNAME: ...
  REDSHIFT_HOST: ...
  REDSHIFT_PORT: ...
  REDSHIFT_TEMP_CRED_USER: ...
  REDSHIFT_TEMP_CRED_PASSWORD: ...
```

When connecting to **Redshift Serverless**, you can use `workgroup-name.account-number.aws-region.redshift-serverless.amazonaws.com`
as the `REDSHIFT_HOST` value.

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Redshift`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Next to the `Save to schema` label, enter the schema name you want this block
   to save data to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks)
   for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_redshift(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

6) Custom types.

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config

Here is an example code snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_redshift(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Redshift cluster.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#redshift
    """
    table_name = 'your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            table_name,
            if_exists='replace',  # Specify resolution policy if table already exists
            overwrite_types=overwrite_types,
        )

```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'VARCHAR(255)'}` |


# S3
Source: https://docs.mage.ai/integrations/databases/S3



![](https://user-images.githubusercontent.com/78053898/198754439-87922b6d-e8ed-4b6c-852c-96a1d3545612.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)
   ```yaml theme={"system"}
   version: 0.1.1
   default:
     AWS_ACCESS_KEY_ID: ...
     AWS_SECRET_ACCESS_KEY: ...
   ```

## Using Python block

1. Create a new pipeline or open an existing pipeline.

2. Add a data loader or transformer block (the code snippet below is for a data
   loader).

3. Select `Generic (no template)`.

4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

   ```python theme={"system"}
   from mage_ai.settings.repo import get_repo_path
   from mage_ai.io.config import ConfigFileLoader
   from mage_ai.io.s3 import S3
   from os import path
   from pandas import DataFrame

   if 'data_loader' not in globals():
       from mage_ai.data_preparation.decorators import data_loader


   @data_loader
   def load_from_s3_bucket(**kwargs) -> DataFrame:
       config_path = path.join(get_repo_path(), 'io_config.yaml')
       config_profile = 'default'

       bucket_name = '...'  # Change to your bucket name
       object_key = '...'   # Change to your object key

       return S3.with_config(ConfigFileLoader(config_path, config_profile)).load(
           bucket_name,
           object_key,
       )
   ```

5. Run the block.

### MinIO Support

To execute requests on MinIO databases, simply add:

```yaml theme={"system"}
AWS_ENDPOINT: your_endpoint:port #example: "https://play.min.io:9000"
```

to the `io_config.yaml` file

### Errors

**S3 connection endpoint URL error**

Open the `io_config.yaml` file at the root of your project (e.g.
`default_repo/io_config.yaml`).

Change the value for the `AWS_REGION` key to the region of your S3 bucket (e.g.
`us-west-2`).

## Permissions

Ensure your AWS IAM user or role has the following permissions:

* `s3:GetObject`
* `s3:PutObject`
* `s3:ListBucket`

These permissions are required to:

* Read data from S3 (e.g. `.csv`, `.parquet`, `.json`)
* Write query results or transformed data to S3
* List contents of a bucket when needed


# SQLite
Source: https://docs.mage.ai/integrations/databases/SQLite



![](https://user-images.githubusercontent.com/78053898/198754399-3b594af0-cf84-41d3-8faf-e1f6e4c5182d.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  sqlite:
    database: /path/to/your/sqlite.db
```

**Notes:**

* `database` should be the absolute path to your SQLite database file
* The database file will be created automatically if it doesn't exist
* Make sure the directory containing the database file has proper write permissions

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `SQLite`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. Next to the `Save to schema` label, enter the schema name you want this block
   to save data to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks) for more
   information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.

### Using raw SQL

You can also use raw SQL with SQLite by enabling the "Use raw SQL" option in the SQL block configuration. This allows you to write complete SQL statements including `CREATE TABLE`, `INSERT`, `UPDATE`, and `DELETE` statements.

**Example raw SQL:**

```sql theme={"system"}
CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    email TEXT UNIQUE
);

INSERT INTO users (name, email) VALUES 
    ('John Doe', 'john@example.com'),
    ('Jane Smith', 'jane@example.com');

SELECT * FROM users;
```

**Note:** When using raw SQL, you're responsible for writing the complete SQL statements. Mage won't automatically create tables or handle data insertion.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_sqlite(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to SQLite:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_sqlite(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            drop_table_on_replace=False,   # Whether to drop the table when "if_exists" param is set to "replace"
        )

```

### Custom types

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config

Here is an example code snippet:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_sqlite(df: DataFrame, **kwargs) -> None:
    schema_name = 'your_schema_name'  # Specify the name of the schema to export data to
    table_name = 'your_table_name'  # Specify the name of the table to export data to
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'TEXT'}

    with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,  # Specifies whether to include index in exported table
            if_exists='replace',  # Specify resolution policy if table name already exists
            overwrite_types=overwrite_types,
        )

```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'TEXT'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |

**Example: Using UPSERT (UPDATE or INSERT)**

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_sqlite(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to SQLite with UPSERT support.
    Specify your configuration settings in 'io_config.yaml'.
    """
    schema_name = 'your_schema_name'
    table_name = 'your_table_name'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            schema_name,
            table_name,
            index=False,
            if_exists='append',  # Use 'append' with UPSERT
            unique_constraints=['user_id', 'email'],  # Columns that form unique constraint
            unique_conflict_method='UPDATE',  # Update existing rows, insert new ones
        )
```

## SQLite-specific considerations

### Database file location

* SQLite databases are stored as single files on disk
* Ensure the file path is accessible and has proper read/write permissions
* Use absolute paths to avoid issues with working directory changes

### Schema support

* SQLite doesn't have true schema support like PostgreSQL
* The "schema" parameter in Mage is used for table naming conventions
* Tables are created in the main database without schema prefixes

### Data types

* SQLite uses dynamic typing (type affinity)
* Common types: `INTEGER`, `REAL`, `TEXT`, `BLOB`, `NULL`
* Mage will map pandas data types to appropriate SQLite types

### Performance tips

* SQLite performs best with smaller datasets (\< 1GB)
* Consider using `PRAGMA` statements for performance tuning
* Use transactions for bulk operations


# Snowflake
Source: https://docs.mage.ai/integrations/databases/Snowflake



![](https://user-images.githubusercontent.com/78053898/198754469-d3c289d7-65c7-44d0-818c-a7a15b393d4c.png)

## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

```yaml theme={"system"}
version: 0.1.1
default:
  SNOWFLAKE_USER: ...
  SNOWFLAKE_PASSWORD: ...
  SNOWFLAKE_ACCOUNT: ...
  SNOWFLAKE_DEFAULT_WH: ...
  SNOWFLAKE_DEFAULT_DB: ...
  SNOWFLAKE_DEFAULT_SCHEMA: ...
  SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: ...
  SNOWFLAKE_PRIVATE_KEY_PATH: ...
  SNOWFLAKE_ROLE: ...
  SNOWFLAKE_TIMEOUT: ...
```

## Using SQL block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Snowflake`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
   credentials underneath).
6. In the `Database` input in the block header, enter the database name you want this block to
   save data to.
7. In the `Schema` input in the block header, enter the schema name you want this block
   to save data to.
8. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
   [SQL blocks guide](/guides/blocks/sql-blocks)
   for more information on write policies).
9. Enter in this test query: `SELECT 1`.
10. Run the block.

## Methods for configuring database and schema

You only need to include the database and schema config in one of these 3 places, in
order of priority (so if all 3 are included, #1 takes priority, then #2, then #3):

1. Include the db and schema directly in the query (e.g.
   `select * from [database_name].[schema_name].[table_name];`). This is supported
   when NOT using the "raw sql" query option.
2. Include the db and schema in the SQL code block header inputs, as mentioned in Steps
   6 and 7 of the "Using SQL block" section above.
3. Include the default db and schema in the `io_config.yaml` file using these fields:

```
SNOWFLAKE_DEFAULT_DB: YOUR_DB_NAME
SNOWFLAKE_DEFAULT_SCHEMA: YOUR_SCHEMA_NAME
```

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_snowflake(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

5. Run the block.

### Export a dataframe

Here is an example code snippet to export a dataframe to Snowflake:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Snowflake warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#snowflake
    """
    table_name = 'your_table_name'
    database = 'your_database'
    schema = 'your_schema'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            table_name,
            database=database,
            schema=schema,
            if_exists='append',  # Specify resolution policy if table name already exists
        )
```

### Custom types

To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config:

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to a Snowflake warehouse.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#snowflake
    """
    table_name = 'your_table_name'
    database = 'your_database'
    schema = 'your_schema'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    overwrite_types = {'column_name': 'VARCHAR(255)'}

    with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            table_name,
            database=database,
            schema=schema,
            if_exists='replace',
            overwrite_types=overwrite_types,
        )
```

### Method arguments

| Field name | Description | Example values |
| - | - | - |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "append") |
| overwrite\_types | Overwrite the column types | `{'column1': 'VARCHAR', 'column2': 'NUMBER'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |

**Example: Using UPSERT (UPDATE or INSERT)**

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
    """
    Template for exporting data to Snowflake with UPSERT support.
    Specify your configuration settings in 'io_config.yaml'.

    Docs: https://docs.mage.ai/design/data-loading#snowflake
    """
    table_name = 'your_table_name'
    database = 'your_database'
    schema = 'your_schema'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        loader.export(
            df,
            table_name,
            database=database,
            schema=schema,
            if_exists='append',  # Use 'append' with UPSERT
            unique_constraints=['user_id', 'email'],  # Columns that form unique constraint
            unique_conflict_method='UPDATE',  # Update existing rows, insert new ones
        )
```


# Spark
Source: https://docs.mage.ai/integrations/databases/Spark



## Add credentials

1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
   have multiple profiles, add it under whichever is relevant to you)

For a **local Spark session**:

```yaml theme={"system"}
version: 0.1.1
default:
  SPARK_HOST: local
  SPARK_METHOD: session
  SPARK_SCHEMA: default
```

For a **remote Spark cluster**, set `SPARK_HOST` to the Spark master URL (for example, `spark://host:7077`, `local[*]`, or another valid `SparkSession.builder.master(...)` value). You can also set `SPARK_METHOD` (e.g. `session`) and `SPARK_SCHEMA` to control how the session is created and which default database/schema is used.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
   below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
   you have a different profile):

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.spark import Spark
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_spark(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    loader = Spark.with_config(ConfigFileLoader(config_path, config_profile))
    return loader.load(query)
```

5. Run the block.

## Data Preview for Spark paths

In Mage Pro, Data Preview can read file-backed Spark datasets without requiring
a persistent Hive metastore table. Use a configured Spark profile and select a
project-relative path with one of the supported Spark file formats: `parquet`,
`csv`, `json`, or `orc`.

For example, a Spark Data Preview source can target a Parquet dataset stored
under the Mage project:

```yaml theme={"system"}
provider: spark
profile: default
format: parquet
path: datasets/events
```

The path must resolve inside the Mage project directory. If you need catalog
table discovery across separate Spark sessions or worker processes, configure a
durable Spark catalog or metastore and use the table name instead.

### Export a dataframe to Spark

```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.spark import Spark
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_spark(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    table_name = 'your_table_name'
    database = 'default'

    loader = Spark.with_config(ConfigFileLoader(config_path, config_profile))
    loader.export(df, table_name=table_name, database=database, if_exists='replace')
```

## Notes

* Spark runs in-process; ensure PySpark and any required cluster dependencies are installed in your Mage environment.
* For local development, `SPARK_HOST: local` typically creates a session with `SparkSession.builder.master('local').getOrCreate()`.
* Use `SPARK_SCHEMA` to set the default database/schema for queries and exports.


# Weaviate
Source: https://docs.mage.ai/integrations/databases/Weaviate



## Credentials

Open the file named `io_config.yaml` at the root of your Mage project and enter weaviate required fields:

```yaml theme={"system"}
version: 0.1.1
default:
  WEAVIATE_ENDPOINT: https://some-endpoint.weaviate.network
  WEAVIATE_INSTANCE_API_KEY: YOUR-WEAVIATE-API-KEY
  WEAVIATE_INFERENCE_API_KEY: YOUR-OPENAI-API-KEY
  WEAVIATE_COLLECTION: collectionn_name
```

## Required Weaviate Libraries

Add `weaviate-client==3.25.3` into `requirements.txt` to install the required weaviate library.

## Using Python block

1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter with Template. Under "Databases" category you can
   find the "Weaviate" template. Then use corresponding load and export functions.

* data loader arguments:
  * properties (List): list of columns to load.
  * collection (str): name of the collection. Defaults to the name defined in io\_config.yaml.
  * with\_limit (int): limit number of results returned.
  * with\_text (str): text to query.

* data exporter arguments:
  * df (DataFrame): data frame to write.
  * collection (str): name of the collection.

3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.


# Trino SQL blocks
Source: https://docs.mage.ai/integrations/databases/trino

Execute SQL commands in Trino.

![Trino](https://dbdb.io/media/logos/trino.png)

## Credentials

Open the file named `io_config.yaml` at the root of your Mage project and enter the following:

```yaml theme={"system"}
version: 0.1.1
default:
  trino:
    catalog: postgresql                       # Change this to the catalog of your choice
    host: 127.0.0.1
    http_headers:
      X-Something: 'mage=power'
    http_scheme: http
    password: mage1337                        # Optional
    port: 8080
    schema: core_data
    session_properties:                       # Optional
      acc01.optimize_locality_enabled: false
      optimize_hash_generation: true
    source: trino-cli                         # Optional
    user: admin
    verify: /path/to/your/ca.crt              # Optional
    data_type_properties:
      timestamp_precision: 3                              # Optional. Set TIMESTAMP(n) precision
    overwrite_types: path_to_types_json # Optional
```

> [!NOTE]
>
> Change the values according to your Trino settings.

## How to use

1. Trino must be running on a remote server or on your local workstation.
   Read these [instructions](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/trino/README.md#run-trino-locally)
   to run Trino locally.

2. Create a new pipeline or edit an existing pipeline.

3. Add a data loader or transformer SQL block.

4. In the block’s header, click the dropdown menu labeled **Connection** and select the option
   labeled **Trino**.

5. In the block’s header, click the dropdown menu labeled **Profile** and select the option
   labeled **default**.

   > [!NOTE]
   >
   > If the name of your profile in the `io_config.yaml` file is different,
   > then select the profile name that matches your settings.

6. You can optionally check the box labeled **Use raw SQL**.
   If you do, read more about [how to use SQL blocks with raw SQL](/guides/blocks/sql-blocks#using-raw-sql).

7. You can optionally overwrite your table columns types by adding a JSON to `overwrite_types` config, the format needs to be `{"column_name":"VARCHAR"}`

8. Write your SQL statements in the SQL block.

9. Once you’re finished writing your SQL statements,
   click the play button in the top right corner of the block to execute the SQL block and preview
   the results.


# dbt Cloud in Mage
Source: https://docs.mage.ai/integrations/dbt-cloud

Trigger model runs in dbt Cloud.

## Configuration

Here are the following keyword arguments that can be used for configuration:

| Keyword argument | Description | Sample value |
| - | - | - |
| `account_id` | dbt Cloud account ID | `12345` |
| `api_token` | [Service account tokens](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) | `abc123` |

***

## Example code

Trigger job run with `job_id`.

```python theme={"system"}
from mage_ai.services.dbt.dbt import DbtCloudClient


client = DbtCloudClient(dict(account_id=12345, api_token='abcdefg'))

# Set poll_status=True to wait for job completion.
# Set poll_status=False to not wait for job completion.
client.trigger_job_run(12345, poll_status=True)
```

Sample output:

```
Polling dbt Cloud run 12345. Current status: Starting.
Polling dbt Cloud run 12345. Current status: Running.
Polling dbt Cloud run 12345. Current status: Success.
Job run status for run 12345: Success. Polling complete
```

***

## Methods

Common [APIs in dbt Cloud](https://docs.getdbt.com/dbt-cloud/api-v2) implemented in the
[`DbtCloudClient`](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/services/dbt/dbt.py).

1. `get_run`
2. `list_jobs`
3. `list_runs`
4. `trigger_job_run`


# Hightouch in Mage
Source: https://docs.mage.ai/integrations/hightouch

Trigger syncs in Hightouch.

## Configuration

Here are the following keyword arguments that can be used for configuration:

| Keyword argument | Description | Sample value |
| - | - | - |
| `api_key` | [Hightouch API key](https://hightouch.com/docs/developer-tools/api-guide) | `abc123` |

***

## Example code

Trigger sync with `sync_id`.

```python theme={"system"}
from mage_ai.services.hightouch.hightouch import HightouchClient


client = HightouchClient(config=dict(api_key='api_key'))
syncs = client.list_syncs()      # Get all the syncs
sync_id = syncs['data'][0]['id'] # Get sync id
client.sync_and_poll(sync_id)    # Trigger sync and poll status
```

Sample output:

```
Polling Hightouch Sync [id]. Current status: querying. 0% completed.
Polling Hightouch Sync [id]. Current status: success. 100% completed.
Sync request status: success. Polling complete
```


# Run Mage pipelines in Prefect
Source: https://docs.mage.ai/integrations/prefect

We support running the pipeline in Prefect flows.

You need to firstly install `mage_ai` library by adding `mage_ai` to your
requirements.txt file. Then you need to download the mage pipeline code into
your Prefect directory. You can achieve it by using a git submodule in your
Prefect directory.

We provide two ways to run mage pipelines in Prefect.

1. Run pipeline in as a task or ShellTask.
2. Run pipeline as a Prefect Flow.

## Run pipeline as a Prefect task

```python Example code theme={"system"}
from prefect import task, Flow
import mage_ai
import os


ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'

@task
def run_pipeline():
    mage_ai.run(pipeline_name, project_path)

with Flow('My Mage Flow') as flow:
    run_pipeline()

flow.run()
```

## Run pipeline in a ShellTask

```python Example code theme={"system"}
from prefect import task, Flow
from prefect.tasks.shell import ShellTask
from prefect.triggers import all_finished
import mage_ai
import os
import prefect

ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'

mage_task = ShellTask(return_all=True)

@task(trigger=all_finished)
def output_print(output):
    logger = prefect.context.get("logger")
    for o in output:
        logger.info(o)

with Flow('My Mage Flow') as flow:
    run_pipeline = mage_task(command=f'mage run {project_path} {pipeline_name}')
    output = output_print(run_pipeline)

flow.run()
```

## Run pipeline as a Prefect flow

```python Example code theme={"system"}
from prefect import Task, Flow
from mage_ai.data_preparation.models.pipeline import Pipeline
import asyncio
import os


ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'prefect_repo')
pipeline_name = 'default_pipeline'

pipeline = Pipeline(pipeline_name, repo_path=project_path)

tasks = {}

class ExecuteBlock(Task):
    def run(self, block):
        asyncio.run(
            block.execute(
                analyze_outputs=False,
                update_status=False,
            )
        )

flow = Flow('My Mage Flow')
blocks_by_uuid = pipeline.blocks_by_uuid

for uuid, b in blocks_by_uuid.items():
    if b.type == 'scratchpad':
        continue
    task = ExecuteBlock()
    tasks[uuid] = task

for uuid, task in tasks.items():
    flow.set_dependencies(
        task=task,
        upstream_tasks=[
            tasks[block_uuid] for block_uuid in blocks_by_uuid[uuid].upstream_block_uuids
        ],
        keyword_tasks=dict(block=blocks_by_uuid[uuid])
    )

flow.run()
```


# Stitch in Mage
Source: https://docs.mage.ai/integrations/stitch

Trigger syncs in Stitch.

## Configuration

Here are the following keyword arguments that can be used for configuration:

| Keyword argument | Description | Sample value |
| - | - | - |
| `access_token` | [Stitch API access token](https://www.stitchdata.com/docs/developers/stitch-connect/api#obtain-access-token) | `abc123` |

***

## Example code

Trigger sync with `source_id`.

```python theme={"system"}
from mage_ai.services.stitch.stitch import StitchClient


client = StitchClient(config=dict(access_token='access_token'))

sources = client.list_sources()
source_id = sources[0]['id'] # Get source id

client.start_replication_job(source_id) # Start replication job
```

Sample output:

```
Start replication job for source 11111. Job name: 111111.222222.sync.abcd1234-a96e-4b48-a10c-9de147a72d64.
Polling Stitch extraction status for source 11111. Current status: running.
Polling Stitch extraction status for source 11111. Current status: running.
Polling Stitch extraction status for source 11111. Current status: running.
Extraction for source 11111 completed.
Finish loading data for all streams: ['stream_1', 'stream_2'].
```


# StrongDM
Source: https://docs.mage.ai/integrations/strongdm



![StrongDM](https://www.strongdm.com/docs/images/logos/strongdm.svg)

## Setup Docker environment

To effectively connect on a StrongDM Resource inside Mage's docker, you may need to add:

If using docker run command:

`--add-host=host.docker.internal:host-gateway`

If using docker-compose setup:

```yaml theme={"system"}
services:
    server:
        .....
        extra_hosts:
            - "host.docker.internal:host-gateway"
```

## Install StrongDM CLI tool

To install StrongDM CLI tool, first access Mage's docker bash and execute the installation script:

```bash theme={"system"}
./scripts/strongdm/install.sh
```

(In case you receive a "Permission denied" error, executing `chmod u+x scripts/strongdm/install.sh` should fix this issue)

Now, login in your StrongDM account using:

```bash theme={"system"}
sdm login
```

And connect to the desired Resources using:

```bash theme={"system"}
sdm connect RESOURCE-NAME
```

## Accessing StrongDM Resources

Now, to connect to a Resource, simply add the required host parameter as "localhost" and the respective port number

MongoDB Source example:

```yaml theme={"system"}
database: admin
host: localhost
password: example
port: 10006 # Specified in StrongDM Resource Generation
user: root
```


# Using an external IDE
Source: https://docs.mage.ai/integrations/text-editor

We love our UI, but here's how to develop in your favorite IDE, too.

## Create a new pipeline

Each pipeline is represented by a YAML file in a folder named `pipelines/` under the Mage
project directory.

For example, if your project is named `demo_project` and your pipeline is named `etl_demo` then
you’ll have a folder structure that looks like this:

```
demo_project/
|   pipelines/
|   |   etl_demo/
|   |   |   __init__.py
|   |   |   metadata.yaml
```

Create a new folder in the `demo_project/pipelines/` directory. Name this new folder after the name
of your pipeline.

Add 2 files in this new folder:

1. `__init__.py`
2. `metadata.yaml`

In the `metadata.yaml` file, add the following content:

```yaml theme={"system"}
blocks: []
name: etl_demo
type: python
uuid: etl_demo
```

> [!NOTE]
>
> Change `etl_demo` to whatever name you’re using for your new pipeline.

## Sample pipeline metadata content

This sample pipeline `metadata.yaml` will produce the following block dependencies:

![Sample pipeline](https://mage-ai.github.io/assets/data-pipeline-overview.png)

```yaml theme={"system"}
blocks:
- downstream_blocks:
    - select_columns
  executor_type: local_python
  language: python
  name: load_data_from_file
  type: data_loader
  upstream_blocks: []
  uuid: load_data_from_file
- downstream_blocks:
    - export_to_file
  executor_type: local_python
  language: python
  name: select_columns
  type: transformer
  upstream_blocks:
    - load_data_from_file
  uuid: select_columns
- downstream_blocks: []
  executor_type: local_python
  language: python
  name: export_to_file
  type: data_exporter
  upstream_blocks:
    - select_columns
  uuid: export_to_file
name: etl_demo
type: python
uuid: etl_demo
```

## `metadata.yaml` sections

### Pipeline attributes

**blocks** `array of objects`

An array of blocks that are in the pipeline.

**name** `string`

Unique name of the pipeline.

**type** `string enum`

The type of pipeline. Currently available options are:

* `databricks`
* `integration`
* `pyspark`
* `python` (most common)
* `streaming`

**uuid** `string`

Unique identifier of the pipeline. This UUID must be unique across all pipelines.

**description** `string`

Optional description of what the pipeline does.

**executor\_type** `string enum`

Pipeline level executor type. Supported values:

* `ecs`
* `gcp_cloud_run`
* `azure_container_instance`
* `k8s`
* `local_python` (most common)
* `pyspark`

**executor\_count** `integer`

Number of concurrent executors to run the pipeline. Used in streaming pipeline.

**executor\_config** `object`

Optional configuration specific to the selected executor type.
Refer to the following documentation for executor-specific options:

* [`ecs`](/production/executors/aws)
* [`gcp_cloud_run`](/production/executors/gcp)
* [`azure_container_instance`](/production/executors/azure)
* [`k8s`](/production/executors/k8)

**spark\_config** `object`

Spark-specific configuration for PySpark pipelines. Mirrors the keys you
would normally pass to `SparkConf` (e.g., `spark_master`, `executor_env`,
`spark_jars`).

**retry\_config** `object`

Retry configuration at the pipeline level.
See [documentation](/orchestration/pipeline-runs/retrying-block-runs#3-pipeline-level-retry-config) for details.

**notification\_config** `object`

Configuration for pipeline notification messages (e.g., on failure or success).
See [documentation](/observability/alerting/alerting-slack) for details.

**concurrency\_config** `object`

Concurrency settings for block execution within the pipeline.
See [documentation](/design/data-pipeline-management#pipeline-level-concurrency) for details.

* `block_run_limit`: Maximum number of blocks that can run in parallel.
* `pipeline_run_limit`
* `pipeline_run_limit_all_triggers`
* `pipeline_run_priority` (Mage Pro only): Higher values start first when pipeline run limits are reached.
* `on_pipeline_run_limit_reached`

**cache\_block\_output\_in\_memory** `boolean`

Whether to cache block output in memory during execution.

**run\_pipeline\_in\_one\_process** `boolean`

If true, runs all blocks in a single process or k8s pod.

**callbacks** `array of objects`

Blocks that run after your main graph finishes (e.g., notifications, cleanup).
Same shape as items in `blocks`.

**conditionals** `array of objects`

Conditional blocks that can short-circuit or branch execution. Same shape as
items in `blocks`.

**extensions** `array of objects`

Extension blocks (e.g., Great Expectations). Same shape as items in `blocks`.

**widgets** `array of objects`

Chart or reporting blocks shown in the pipeline UI. Same shape as items in
`blocks`.

**settings** `object`

Pipeline-level settings; currently supports `triggers.save_in_code_automatically`
to control whether new/updated triggers are persisted to YAML.

**variables** `object`

Key/value pairs available to the pipeline for templating. Useful for
environment-specific values that are not secrets.

**state\_store\_config** `object`

Configure an external state store for pipeline runs.

**tags** `array of strings`

Free-form labels to organize and search for pipelines. Each tag name can contain up to 70
characters.

**created\_by** `string`

Identifier for the user or process that created the pipeline.

**overrides** `object`

Environment-specific overrides for any top-level pipeline field. Mage Pro only.
See [environment overrides](/extensibility/env-config/pipeline) for guidance.

### Block attributes

**downstream\_blocks** `array of strings`

An array of block UUIDs that depend on this current block.
These downstream blocks will have access to this current block’s data output.

**executor\_type** `string enum`

The method for running this block of code. Currently available options are:

* `ecs`
* `gcp_cloud_run`
* `azure_container_instance`
* `k8s`
* `local_python` (most common)
* `pyspark`

**executor\_config** `object`

Optional configuration specific to the selected executor type.
Refer to the following documentation for executor-specific options:

* [`ecs`](/production/executors/aws)
* [`gcp_cloud_run`](/production/executors/gcp)
* [`azure_container_instance`](/production/executors/azure)
* [`k8s`](/production/executors/k8)

**language** `string enum`

Programming language used by the block. Supported values:

* `python` (most common)
* `r`
* `sql`
* `yaml`

**name** `string`

Unique name of the block.

**type** `string enum`

The type of block. Currently available options are:

* `chart`
* `custom` (most common)
* `data_exporter`
* `data_loader`
* `dbt`
* `scratchpad`
* `sensor`
* `transformer`

The type of block will determine which folder it needs to be in. For example, if the block `type`
is `data_loader`, then the file must be in the `[project_name]/data_loaders/` folder. It can be
nested in any number of subfolders.

**upstream\_blocks** `array of strings`

An array of block UUIDs that this current block depends on.
These upstream blocks will pass its data output to this current block.

**uuid** `string`

Unique identifier of the block. This UUID must be unique within the current pipeline. The UUID
corresponds to the name of the file for this block.

For example, if the UUID is `load_data` and the `language` is `python`, then the file name
will be `load_data.py`.

**color** `string`

Optional HEX color used to visually organize blocks in the UI.

**configuration** `object`

Block-specific configuration. For integrations or dbt blocks this mirrors the
configuration visible in the block settings panel.

**documentation** `string`

Markdown content rendered in the block’s documentation tab.

**priority** `integer`

Optional priority for scheduling blocks when multiple are runnable.

**retry\_config** `object`

Retry configuration at the block level.
See [documentation](/orchestration/pipeline-runs/retrying-block-runs#2-block-level-retry-config) for details.

**tags** `array of strings`

Labels applied to the block for organization and filtering.

**timeout** `integer`

Maximum execution time for the block in seconds.


# It’s magic.
Source: https://docs.mage.ai/introduction/about-mage-pro

Give your data team magical powers.

Mage Pro is a modern data pipeline tool designed to address data engineering challenges at scale. It’s a unified platform for data extraction, transformation, loading, and orchestration. It bridges the gap between raw data sources and analytics ready systems. Mage Pro is more performant, reliable, and cost effective than Mage OSS.

Mage Pro can be deployed in 4 ways:

* **Mage Pro SaaS** - Is fully managed and your Mage code executes in our environment.
* **Mage Pro hybrid cloud** - Where Mage Pro infrastructure maintains the control plane, but the data plane is deployed in your cloud.
* **Mage Pro private cloud** - Both control and data plane are deployed in your cloud.
* **Mage Pro on-prem** - Where Mage Pro infrastructure is deployed on your on-prem servers.

Additionally, Mage Pro adds some core capabilities that are not found in the Mage OSS environment. True to Mage’s philosophy on developer implementation and collaboration, many of the additional features make Mage Pro even more developer friendly. Some of those core capabilities are:

* **Cluster health insights**, check how your clusters are performing in Mage Pro’s cluster manager insights application. Get pricing estimates and real time CPU and Memory usage.
* **Workspaces** - Segment one or more of your Mage Pro clusters into multiple development environments
* **CICD Deployments** - A powerful tool to easily deploy new versions of your code at the click of a button.
* **Pipeline dependency views** - Have pipelines that orchestrate or depend on other pipelines? Now you can have a view of this in this Mage Pro only application.

Mage’s SaaS offering can be deployed from any cloud region in the world. Currently Mage Pro is deployed in the following regions US West, US East, Canada, Europe, Asia, and Australia.

Let’s get you started on Mage Pro now!


# It’s magic.
Source: https://docs.mage.ai/introduction/overview

Give your data team magical powers.

[![Mage AI GitHub repo stars](https://img.shields.io/github/stars/mage-ai/mage-ai?style=for-the-badge\&logo=github\&labelColor=000000\&logoColor=FFFFFF\&label=stars\&color=0500ff)](https://www.mage.ai/oss)

[![Mage AI Docker downloads](https://img.shields.io/docker/pulls/mageai/mageai?style=for-the-badge\&logo=docker\&labelColor=000000\&logoColor=FFFFFF\&label=pulls\&color=956fff)](https://hub.docker.com/r/mageai/mageai)

[![Mage AI license](https://img.shields.io/github/license/mage-ai/mage-ai?style=for-the-badge\&logo=codeigniter\&labelColor=000000\&logoColor=FFFFFF\&label=license\&color=FFCC19)](https://github.com/mage-ai/mage-ai/blob/master/LICENSE)

[![Join the Mage AI community](https://img.shields.io/badge/Join%20the%20community-black.svg?style=for-the-badge\&logo=lightning\&labelColor=000000\&logoColor=FFFFFF\&label=\&color=DD55FF\&logoWidth=20)](https://www.mage.ai/chat)

<picture>
  <source />

  ![Mage AI hero](https://mage-ai.github.io/assets/oss/hero.png)
</picture>

[Open](https://cloud.mage.ai/sign-up?source=introduction)

Mage is a hybrid framework for transforming and integrating data. It combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.

* Extract and synchronize data from 3rd party sources.
* Transform data with real-time and batch pipelines using Python, SQL, and R.
* Load data into your data warehouse or data lake using our pre-built connectors.
* Run, monitor, and orchestrate thousands of pipelines without losing sleep.

Plus hundreds of enterprise-class features, infrastructure innovations, and magical surprises.

***

**Available in two spellbinding versions**

> ![](https://mage-ai.github.io/assets/oss/card-pro.png)
>
> **For teams.** Fully managed platform
> for integrating and transforming data.
>
> [Open](https://cloud.mage.ai/sign-up?source=introduction)

> ![](https://mage-ai.github.io/assets/oss/card-oss.png)
>
> **Self-hosted.** System to build, run, and
> manage data pipelines.
>
> [Open](https://www.mage.ai/oss?source=introduction)

[Get started with Mage Pro for free](https://cloud.mage.ai/sign-up?source=introduction)

***

## Getting started

Get quickly started on building and deploying magical data pipelines
by watching a short product video and trying out the interactive demo playground.

> **[Product tour](https://youtu.be/GswOdShLGmg)**
>
> Get a 5 minute overview

> **[Live demo](https://demo.mage.ai)**
>
> Open our interactive demo

## 🦄 Make data magical

We put the *fun* in fun-ctional programming. Mage is a hybrid framework that combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.

* **Extract** and synchronize data from 3rd party sources.
* **Transform** data with real-time and batch pipelines using Python, SQL, and R.
* **Load** data into your data warehouse or data lake using our pre-built connectors.
* Run, monitor, and **orchestrate** thousands of pipelines without losing sleep.

## 1. 🔨 Build

Have you met anyone who loves developing in Airflow? That's why we designed an easy developer experience that you'll enjoy.

* **Simple developer experience**: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
* **Language of choice**: Write code in Python, SQL, or R in the same pipeline for *ultimate* flexibility.
* **Engineering best practices built-in**: Every step in your pipeline is a standalone file with modular code that's reusable and testable. No more DAGs with spaghetti code 🍝.
* **Bring your favorite tools:** Write [dbt](/guides/dbt/overview) models, use your favorite [IDE](/guides/developer-ux/code-editor), [track changes with Git](/guides/data-sync/version-control), and much much more.

![](https://mage-ai.github.io/assets/overview/mage-build.gif)

## 2. 🔍 Preview

Don't waste time waiting for your DAGs to finish testing. Get instant feedback from your code *every* time you run it.

* **Interactive code**: Immediately see results from your code's output with an interactive notebook UI.
* **Data is a first-class citizen**: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
* **Collaborate on cloud**: Develop collaboratively on [cloud resources](/production/deploying-to-cloud/architecture), version control with [Git](/development/git/setting-up-git), and [test](/observability/testing/data-validation) pipelines without waiting for an available shared staging environment.

![](https://mage-ai.github.io/assets/overview/mage-preview.gif)

## 3. 🚀 Launch

Don't have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.

* **Fast deploy**: Deploy Mage to [AWS](/production/deploying-to-cloud/aws/setup), [GCP](/production/deploying-to-cloud/gcp/setup), or [Azure](/production/deploying-to-cloud/azure/setup) with only 2 commands using maintained Terraform templates.
* **Scaling made simple**: Transform very large datasets directly in your data warehouse or through a native integration with [Spark](/integrations/spark-pyspark).
* **Observability**: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.

![](https://mage-ai.github.io/assets/overview/observability.gif)

## 🔮 [Features](/about/features)

| | | |
| - | - | - |
| 🎶 | [Orchestration](/design/data-pipeline-management) | Schedule and manage data pipelines with observability. |
| 📓 | [Notebook editor](/about/features#notebook-for-building-data-pipelines) | Interactive Python, SQL, & R editor for coding data pipelines. |
| 🏗️ | [Data integration](/data-integrations) | Synchronize data from 3rd party sources to your internal destinations. |
| 🚰 | [Streaming](/guides/streaming/introduction) | Ingest and transform real-time data. |
| 🧱 | [dbt](/guides/dbt) | Build, run, and manage your dbt models with Mage. |

## 🏔️ [Core design principles](/design/core-design-principles)

Every user experience and technical design decision adheres to these principles.

| | | |
| - | - | - |
| 💻 | [Easy developer experience](/design/core-design-principles#easy-developer-experience) | Open-source engine that comes with a custom notebook UI for building data pipelines. |
| 🚢 | [Engineering best practices](/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
| 💳 | [Data as a first-class citizen](/design/core-design-principles#data-is-a-first-class-citizen) | Designed from the ground up specifically for running data-intensive workflows. |
| 🪐 | [Scaling made simple](/design/core-design-principles#scaling-is-made-simple) | Analyze and process large data quickly for rapid iteration. |

## 🛸 [Core abstractions](/design/core-abstractions)

These are the fundamental concepts that Mage uses to operate.

| | | |
| - | - | - |
| 🏢 | [Project](/design/core-abstractions#project) | Like a repository on GitHub; this is where you write all your code. |
| 🪈 | [Pipeline](/design/core-abstractions#pipeline) | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
| 🧱 | [Block](/design/core-abstractions#block) | A file with code that can be executed independently or within a pipeline. |
| 🤓 | [Data product](/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage. |
| ⏰ | [Trigger](/design/core-abstractions#trigger) | A set of instructions that determine when or how a pipeline should run. |
| 🏃‍♂️ | [Run](/design/core-abstractions#run) | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |

## 👨‍👩‍👧‍👦 Community

Individually, we're a mage.

> 🧙 Mage
>
> Magic is indistinguishable from advanced technology. A mage is someone who
> uses magic (aka advanced technology).

Together, we're Magers!

> 🧙‍♂️🧙 Magers (`/ˈmājər/`)
>
> A group of mages who help each other realize their full potential! Join us on [Slack](https://www.mage.ai/chat).

✨ This documentation & project are brought to you by the following *magical* individuals (learn more about contributing [here](/contributing/overview)):

[![](https://contrib.rocks/image?repo=mage-ai/mage-ai\&columns=10)](https://github.com/mage-ai/mage-ai/graphs/contributors)

***

![](https://mage-ai.github.io/assets/mascots/mascots-shorter.jpeg)


# Migrate from Airbyte to Mage Pro
Source: https://docs.mage.ai/migrations/airbyte-to-mage-pro



Many teams start with Airbyte for data integration—but soon encounter challenges with flexibility, orchestration, and observability. Mage Pro offers a unified alternative: a single platform for data integration, transformation, and orchestration with **AI-assisted pipeline development** and **enterprise-ready scalability**.

## Why migrate from Airbyte to Mage Pro?

Airbyte is popular for ELT pipelines, but it wasn't designed for:

* Native orchestration of transformations
* Visual, modular pipeline building
* AI-powered debugging and code generation
* Git-native CI/CD and multi-environment deployments
* Streaming pipelines and event triggers

Mage Pro replaces Airbyte’s connector-based model with **end-to-end data integration pipelines** that seamlessly combine **extract, load, and transform steps**—all while giving teams full code flexibility and better observability.

***

## ✨ Mage Pro vs Airbyte: Benefits Overview

Mage Pro combines [**data integration**](/data-integrations/overview) and [**data orchestration**](/orchestration/triggers/triggering-pipelines) in a single, developer-friendly platform.

| Capability | Airbyte | Mage Pro |
| - | - | - |
| **Data integration** | ✅ Pre-built connectors | ✅ Native connectors for APIs, databases, files |
| **Orchestration** | ⚠️ Basic scheduling only | ✅ Full DAG-level orchestration |
| **Visual pipeline UI** | ⚠️ Connector-based only | ✅ Drag-and-drop pipeline builder |
| **Custom transformations** | ⚠️ Limited (requires dbt or custom code) | ✅ Python, SQL, or AI-generated transformations |
| **AI assistance** | ❌ | ✅ Generate, debug, and optimize pipelines with AI Sidekick |
| **Incremental loads** | ✅ Supported via config | ✅ Supported natively in pipeline settings |
| **Schema discovery & testing** | ⚠️ Limited | ✅ Data previews, schema validation, tests |
| **Streaming & CDC** | ⚠️ Limited | ✅ Native support for Kafka, CDC, and real-time ingestion |
| **Git integration** | ⚠️ Manual export/import only | ✅ Git-backed pipelines and CI/CD |
| **Secrets management** | ⚠️ Env-only | ✅ Built-in secrets & workspace isolation |
| **Multi-environment support** | ❌ Manual, not native | ✅ Dev, staging, and production workspaces |
| **Observability** | ⚠️ Basic logs | ✅ Detailed logs, lineage, and monitoring |
| **Scalability** | ⚠️ Self-managed in OSS, limited control in Cloud | ✅ Auto-scaled executors on Kubernetes or ECS |

***

## 🛠️ Step-by-Step Migration Instructions

1. **Inventory your Airbyte connectors**
   * List all sources and destinations in your Airbyte setup
   * Note incremental or full-load settings
   * Export transformation logic (e.g., dbt, SQL scripts)

2. **Create a Mage Pro workspace**
   * Sign up at [Mage Pro](https://cloud.mage.ai)
   * (Optional) Connect your Git repository for version control

3. **Recreate connectors as Mage data integration blocks**
   * Each Airbyte source → **Extract block** in Mage
   * Each Airbyte destination → **Load block** in Mage
   * Any dbt or SQL logic → **SQL block** (dbt-like features built-in)
   * Use **Python blocks** for custom transformations

4. **Configure pipelines**
   * Combine extract → transform → load in a single pipeline
   * Use the visual editor or YAML pipeline config

5. **Set up scheduling and triggers**
   * Replace Airbyte sync schedules with Mage cron or event-based triggers
   * Optionally add webhooks or file-based triggers

6. **Run and validate pipelines**
   * Preview data outputs at every step
   * Compare final tables and schema to ensure parity
   * Validate incremental load and deduplication logic

7. **Monitor and optimize**
   * Use Mage’s built-in logs and data lineage
   * Configure alerts and retry rules
   * Leverage auto-scaling for parallel extracts and loads

***

## 🤖 Convert Airbyte Configs with AI Sidekick

Mage Pro’s **AI Sidekick** can automatically help convert Airbyte connector configurations and dbt logic into Mage pipelines.

### 🔧 How to Use It

1. Open your Mage Pro workspace and click **“Ask AI”**
2. Paste your Airbyte connector settings or dbt SQL models
3. Ask: **“Convert this Airbyte source-destination flow into a Mage pipeline.”**
4. Sidekick will:
   * Generate Extract/Load blocks for your sources and targets
   * Translate transformation steps into SQL or Python blocks
   * Create a complete pipeline with triggers and dependencies
5. Insert the generated pipeline directly into your workspace.

***

## 🧠 Tips for Migrating Complex Airbyte Flows

* Merge multiple Airbyte connectors into a **single end-to-end pipeline**
* Use **pipeline variables** for shared config (e.g., table names, batch sizes)
* Replace dbt transformations with **SQL blocks** (supports `ref()`, tests, and incremental models)
* For large tables, enable **batching and streaming modes**

***

## ✅ After Migration: What You Gain

* Unified **integration + transformation + orchestration**
* Visual pipelines and instant debugging
* AI-powered pipeline creation and troubleshooting
* Git-based CI/CD and environment isolation
* Real-time streaming support (Kafka, CDC, webhooks)
* Enterprise-grade RBAC, audit logs, and observability

> No more juggling Airbyte, Airflow, and dbt. With Mage Pro, you get a single platform for all data workflows.

👉 [Start Migrating to Mage Pro](https://cloud.mage.ai)


# Migrate from Airflow to Mage Pro
Source: https://docs.mage.ai/migrations/airflow-to-mage-pro



Many teams start with Apache Airflow for orchestration—but soon encounter challenges with scalability, debugging, and developer productivity. Mage Pro offers a modern alternative: visual pipelines, built-in observability, and AI-assisted development, with zero DevOps overhead.

## Why migrate from Airflow to Mage Pro?

Airflow is a powerful scheduler, but it wasn't designed for:

* Debugging pipelines visually
* Mixing SQL and Python naturally
* Streaming or event-driven workflows
* Managing secrets, Git, or multi-tenant workspaces
* AI-powered pipeline development

Mage Pro is built to solve these challenges out-of-the-box, with a modern developer experience and enterprise-grade scalability.

***

## ✨ Mage Pro vs Airflow: Benefits Overview

Mage Pro goes far beyond orchestration. It’s a unified platform for **data integration**, **SQL modeling (dbt-like blocks)**, **AI-powered transformation**, and **streaming pipelines** — all within a collaborative, Git-native workspace.

| Capability | Airflow | Mage Pro |
| - | - | - |
| **Visual pipeline UI** | ❌ Code-only (Jinja, Python) | ✅ Hybrid UI: drag-and-drop DAG builder + notebook-style coding experience |
| **Visual pipeline UI** | ❌ Code-only (Jinja, Python) | ✅ Drag-and-drop editor |
| **Built-in lineage** | ⚠️ Plugin-based (e.g., OpenLineage) | ✅ Native, auto-generated |
| **AI assistance** | ❌ | ✅ Generate, fix, and explain code with AI Sidekick |
| **Multi-language support** | ⚠️ Python only by default | ✅ SQL, Python, streaming, APIs |
| **Data integration pipelines** | ⚠️ Custom Python or 3rd-party plugins | ✅ Native connectors for databases, files, APIs |
| **SQL block support** | ⚠️ SQLOperator runs raw strings only | ✅ dbt-style SQL blocks with `ref()`, preview, and test support |
| **Incremental modeling** | ❌ (requires custom logic) | ✅ Native in SQL block config |
| **Environment isolation** | ⚠️ Requires Airflow deployments per team | ✅ Per-workspace configs, secrets, variables |
| **Git integration** | ⚠️ Manual DAG sync | ✅ Git-backed version control and CI/CD |
| **Scheduling & triggers** | ✅ Cron + sensors | ✅ Cron, events, file triggers, webhooks |
| **Secrets management** | ⚠️ Requires Vault or plugins | ✅ Built-in across workspaces and pipelines |
| **RBAC & SSO** | ⚠️ Manual plugins / external auth | ✅ Built-in, enterprise-ready |
| **Observability & logging** | ⚠️ Requires Prometheus, Grafana, etc. | ✅ Native UI for logs, metrics, traces |
| **Scalability** | ⚠️ Manual tuning of workers and queues | ✅ Auto-scaled executors on K8s or ECS |
| **Streaming support** | ❌ | ✅ Kafka, CDC, real-time ingestion |
| **Data previews & interactive runs** | ❌ | ✅ Preview data at every step |
| **Team collaboration** | ⚠️ Airflow not multi-tenant | ✅ Workspaces, permissions, activity logs |

***

## 🛠️ Step-by-Step Migration Instructions

1. **Inventory your Airflow DAGs**
   * List your active DAGs and their task types:
     * `PythonOperator`, `BashOperator`, `SQLOperator`, etc.
   * Identify dependencies and scheduling logic

2. **Create a cluster in Mage Pro**
   * Visit [Mage Pro](https://cloud.mage.ai) and sign in
   * (Optional) Connect your Git repo for version control

3. **Recreate your DAGs as Mage pipelines**
   * Each Airflow DAG → 1 Mage pipeline
   * Each Airflow task → 1 Mage block:
     * `PythonOperator` → Python block
     * `SQLOperator` → SQL block
     * `BranchOperator` → Dynamic block
     * `Sensor` → Trigger block
   * Define dependencies visually using the UI

4. **Configure scheduling and triggers**
   * Use cron, interval, or event-based triggers
   * Configure schedules via UI or YAML

5. **Run and validate pipelines**
   * Use Mage Pro’s UI to test individual blocks
   * Preview dataframes, logs, and outputs
   * Compare results to your Airflow jobs

6. **Monitor, scale, and automate**
   * Monitor pipeline runs and resource usage
   * Set alerts and retry rules
   * Mage Pro autoscaling handles executor resources automatically

***

## 🤖 Convert Airflow DAGs with AI Sidekick

Skip the manual rewrites — Mage Pro’s **AI Sidekick** can automatically convert your Airflow DAG code into a Mage pipeline with just one prompt.

### 🔧 How to Use It

1. Click the **“Ask AI”** field in the Mage Pro header.
2. Paste your Airflow DAG code — including `@dag`, `@task`, or `PythonOperator`-based workflows.
3. Ask: **“Convert this Airflow DAG to a Mage pipeline.”**
4. Sidekick will:
   * Parse the DAG structure and task relationships
   * Generate the corresponding Mage blocks (Python, SQL, dynamic, trigger)
   * Define block dependencies and scheduling logic
5. Review and insert the generated pipeline directly into your project.

### 💡 Why Use Sidekick?

* **Faster migration**: turn entire DAGs into Mage pipelines in seconds
* **Less error-prone**: Sidekick understands scheduling, dependencies, and operator types
* **Context-aware**: uses your project setup and prior block structure
* **Fully editable**: review, tweak, and insert blocks before saving

👉 Learn more in the [AI Sidekick docs](/ai/sidekick).

***

## 🧠 Tips for Migrating Complex DAGs

* Split large DAGs into smaller, modular pipelines
* Use **global variables** or **shared outputs** to pass data between blocks
* Replace Airflow SubDAGs with **block groups**
* Use **dynamic blocks** to support branching and conditional logic

***

## ✅ After Migration: What You Get with Mage Pro

* AI-powered block generation and debugging (via AI Sidekick)
* Visual pipeline builder and real-time logs
* Git-backed version control and CI/CD
* Built-in access controls, audit logs, and workspace isolation
* Support for SQL, Python, streaming, dbt, Delta Lake, Iceberg, and more

> Your pipelines, your logic—augmented by AI, automated for scale.

👉 [Migrate to Mage Pro Today](https://cloud.mage.ai)


# Migrate from Dagster to Mage Pro
Source: https://docs.mage.ai/migrations/dagster-to-mage-pro



Many teams start with Dagster for data orchestration—but soon encounter challenges with data engineering workflows, visual debugging, and developer productivity. Mage Pro offers a modern alternative: visual pipelines, built-in data integration, AI-assisted development, and UI-based deployment management—all within a unified platform for your data workflows.

## Why migrate from Dagster to Mage Pro?

Dagster is a powerful orchestration tool focused on data assets, but it wasn't designed for:

* Visual pipeline development and debugging
* Native data integration (ETL/ELT workflows)
* Mixing SQL and Python naturally in the same pipeline
* Streaming or event-driven data processing
* Built-in data previews and interactive development
* AI-powered pipeline development

Mage Pro is built to solve these challenges out-of-the-box, combining orchestration with data engineering capabilities in a modern, developer-friendly platform.

***

## Mage Pro vs Dagster: Benefits Overview

Mage Pro goes far beyond orchestration. It's a unified platform for **data integration**, **SQL modeling (native dbt blocks and Mage SQL blocks)**, **AI-powered transformation**, and **streaming pipelines** — all within a collaborative, Git-native workspace with UI-based deployment management.

| Capability | Dagster | Mage Pro |
| - | - | - |
| **Visual pipeline UI** | ⚠️ Code-only (Python decorators) | ✅ Hybrid UI: drag-and-drop DAG builder + notebook-style coding experience |
| **Built-in lineage** | ✅ Asset lineage | ✅ Native, auto-generated data lineage |
| **AI assistance** | ❌ | ✅ Generate, optimize, fix, and explain code with AI Sidekick |
| **Multi-language support** | ⚠️ Python only | ✅ SQL, Python, R, streaming, APIs |
| **Data integration pipelines** | ⚠️ Custom Python or 3rd-party tools | ✅ 200+ native connectors for databases, files, APIs |
| **SQL block support** | ⚠️ Requires custom ops | ✅ Native dbt blocks, or Mage SQL block, preview, and test support |
| **Incremental modeling** | ⚠️ Requires custom logic | ✅ Native in SQL block config and Data integration pipeline |
| **Data previews & interactive runs** | ⚠️ Limited | ✅ Preview data at every step, run blocks independently |
| **Environment isolation** | ⚠️ Requires separate deployments | ✅ Per-workspace configs, secrets, variables |
| **Git integration** | ⚠️ Manual sync | ✅ Git-backed version control and CI/CD, UI based Deployment App |
| **Scheduling & triggers** | ✅ Cron + sensors | ✅ Cron, events, file triggers, webhooks, API |
| **Secrets management** | ⚠️ Requires Dagster Cloud or external config | ✅ Support both built-in secret manager, or external secret manager |
| **RBAC & SSO** | ⚠️ Dagster Cloud only | ✅ Built-in, enterprise-ready |
| **Observability & logging** | ✅ Dagster UI | ✅ Native UI for logs, metrics, traces, lineage |
| **Scalability** | ⚠️ Manual worker configuration | ✅ Auto-scaled executors on K8s or Docker |
| **Streaming support** | ❌ | ✅ Kafka, CDC, real-time ingestion |
| **Team collaboration** | ⚠️ Limited multi-tenancy | ✅ Workspaces, permissions, activity logs |

***

## Step-by-Step Migration Instructions

1. **Inventory your Dagster assets and jobs**
   * List your active assets, ops, and jobs:
     * `@asset` and `@op` decorators
     * Dagster resources (storage, infrastructure)
     * Dependencies and scheduling logic
     * Partitions and materializations
   * Identify data sources, transformations, and destinations

2. **Create a workspace in Mage Pro**
   * Visit [Mage Pro](https://cloud.mage.ai) and sign in
   * (Optional) Connect your Git repo for version control
   * Set up your deployment configuration using the **Deployment App** in the UI

3. **Recreate your Dagster assets as Mage pipelines**
   * Each Dagster **job** → 1 Mage pipeline
   * Each Dagster **asset/op** → 1 Mage block:
     * `@asset` or `@op` with data loading → **Data loader block**
     * `@asset` or `@op` with transformations → **Transformer block** (Python or SQL)
     * `@asset` or `@op` with data writing → **Data exporter block**
     * Dagster resources → **Data loader/exporter blocks** or pipeline configuration
   * Define dependencies visually using the UI (no decorators needed)

4. **Configure scheduling and triggers**
   * Dagster schedules → Mage **triggers** (cron, interval, or event-based)
   * Dagster sensors → Mage **event triggers** or **file triggers**
   * Configure schedules via UI or YAML

5. **Run and validate pipelines**
   * Use Mage Pro's UI to test individual blocks
   * Preview dataframes, logs, and outputs at each step
   * Compare results to your Dagster jobs

6. **Monitor, scale, and automate**
   * Monitor pipeline runs and resource usage
   * Set alerts and retry rules
   * Mage Pro autoscaling handles executor resources automatically on Kubernetes or Docker
   * Use the **Deployment App** to manage deployments and rollbacks

***

## Mapping Dagster Concepts to Mage Pro

### Assets & Jobs → Pipelines

**Dagster:**

```python theme={"system"}
from dagster import asset, job, op

@asset
def raw_data():
    return pd.read_csv("data.csv")

@asset
def transformed_data(raw_data):
    return raw_data.dropna()

@job
def my_etl_job():
    transformed_data(raw_data())
```

**Mage Pro:**

* Create a new **pipeline** in the UI
* Each asset/op becomes a **block** (data loader, transformer, or data exporter)
* Dependencies are defined by connecting blocks in the UI
* No need for explicit job definitions—pipeline structure defines execution

### Assets & Ops → Blocks

**Dagster:**

```python theme={"system"}
@asset
def extract_data():
    return pd.read_csv("data.csv")

@op
def transform_data(context, df):
    return df.dropna()

@asset
def load_data(transformed_data):
    transformed_data.to_sql("table", engine)
```

**Mage Pro:**

* `extract_data` → **Data loader block** (Python or connector)
* `transform_data` → **Transformer block** (Python, SQL, or dbt block)
* `load_data` → **Data exporter block** (Python or connector)

### Dagster Resources → Mage Blocks

| Dagster Resource Type | Mage Pro Equivalent |
| - | - |
| `S3Resource`, `GCSResource` | **Data loader/exporter blocks** with native S3/GCS connectors |
| `PostgresResource`, `SnowflakeResource` | **Data loader/exporter blocks** with database connectors |
| `DockerResource`, `K8sResource` | **Executor configuration** in pipeline settings |
| Config resources | **Workspace variables** or **pipeline variables** |

### Partitions & Materializations

**Dagster:**

```python theme={"system"}
from dagster import DailyPartitionsDefinition, asset

daily_partitions = DailyPartitionsDefinition(start_date="2024-01-01")

@asset(partitions_def=daily_partitions)
def partitioned_asset(context):
    partition_date = context.partition_key
    # Process data for specific partition
    ...
```

**Mage Pro:**

* Use **pipeline variables** or **dynamic blocks** for partition logic
* Configure **incremental processing** in SQL blocks or Python blocks
* Use **backfill triggers** for historical data processing
* Schedule pipelines with cron expressions for partition-based runs

### Scheduling & Sensors

**Dagster:**

```python theme={"system"}
from dagster import schedule, sensor, RunRequest

@schedule(cron_schedule="0 0 * * *", job=my_etl_job)
def daily_schedule(context):
    return RunRequest()

@sensor(job=my_etl_job)
def file_sensor(context):
    if file_exists("new_data.csv"):
        return RunRequest()
```

**Mage Pro:**

* Create a **schedule trigger** in the pipeline UI
* Configure cron expression, interval, or event-based triggers
* Use **file triggers** for file-based sensors
* Use **event triggers** for webhook-based sensors
* Use the **Deployment App** to manage deployments across environments (dev, staging, prod)

### Secrets & Configuration

**Dagster:**

```python theme={"system"}
from dagster import ConfigurableResource

class MyResource(ConfigurableResource):
    api_key: str

@asset
def use_resource(my_resource: MyResource):
    key = my_resource.api_key
```

**Mage Pro:**

* Use **workspace variables** or **pipeline variables**
* Access via `variables` dictionary in blocks
* **Built-in secret manager** with encryption, or integrate with **external secret managers** (AWS Secrets Manager, HashiCorp Vault, etc.)
* Secrets are isolated per workspace and pipeline

***

## Convert Dagster Code with AI Sidekick

Skip the manual rewrites — Mage Pro's **AI Sidekick** can automatically convert your Dagster asset and job code into a Mage pipeline with just one prompt.

### How to Use It

1. Click the **"Ask AI"** field in the Mage Pro header.
2. Paste your Dagster code — including `@asset`, `@op`, `@job`, or asset-based workflows.
3. Ask: **"Convert this Dagster job to a Mage pipeline."**
4. Sidekick will:
   * Parse the asset/op structure and dependencies
   * Generate the corresponding Mage blocks (data loader, transformer, data exporter)
   * Define block dependencies and scheduling logic
   * Convert Dagster resources to Mage connectors where applicable
   * **Optimize** the generated code for performance and best practices
5. Review and insert the generated pipeline directly into your project.

### Why Use Sidekick?

* **Faster migration**: turn entire jobs into Mage pipelines in seconds
* **Less error-prone**: Sidekick understands asset dependencies, partitions, and op types
* **Code optimization**: automatically optimizes generated code for performance
* **Context-aware**: uses your project setup and prior block structure
* **Fully editable**: review, tweak, and insert blocks before saving

👉 Learn more in the [AI Sidekick docs](/ai/sidekick).

***

## Tips for Migrating Complex Jobs

### Handling Dagster Asset Dependencies

**Dagster:**

```python theme={"system"}
@asset
def parent_asset():
    ...

@asset
def child_asset(parent_asset):
    ...
```

**Mage Pro:**

* Dependencies are automatically inferred from block connections in the UI
* Use **upstream blocks** to define data flow
* Blocks with no dependencies run in parallel automatically

### Dagster Partitions

**Dagster:**

```python theme={"system"}
from dagster import DailyPartitionsDefinition

daily_partitions = DailyPartitionsDefinition(start_date="2024-01-01")

@asset(partitions_def=daily_partitions)
def partitioned_asset(context):
    partition_date = context.partition_key
    ...
```

**Mage Pro:**

* Use **pipeline variables** to pass partition information
* Configure **incremental processing** in SQL blocks
* Use **backfill triggers** to process historical partitions
* Schedule pipelines with appropriate cron expressions

### Dagster Resources & Configuration

**Dagster:**

```python theme={"system"}
from dagster import ConfigurableResource, asset

class DatabaseResource(ConfigurableResource):
    connection_string: str

@asset
def use_database(database: DatabaseResource):
    conn = database.connection_string
    ...
```

**Mage Pro:**

* Use **workspace variables** or **pipeline variables** for configuration
* Access via `variables` dictionary in blocks
* Use **data loader/exporter blocks** with native connectors for databases
* Secrets are managed through built-in or external secret managers

### Dagster Retries & Error Handling

**Dagster:**

```python theme={"system"}
@asset(retry_policy=RetryPolicy(max_retries=3, delay=60))
def risky_asset():
    ...
```

**Mage Pro:**

* Configure **retry settings** in block or pipeline configuration
* Set retry count, delay, and backoff strategy
* Built-in error handling and alerting

### Dagster Caching & Materializations

**Dagster:**

```python theme={"system"}
@asset(
    compute_kind="pandas",
    metadata={"cache": True}
)
def cached_asset():
    ...
```

**Mage Pro:**

* Use **block output caching** in pipeline settings
* Configure cache expiration and invalidation rules
* Preview cached outputs in the UI

***

## After Migration: What You Get with Mage Pro

* **Unified data platform**: orchestration + data integration + transformation + streaming in one tool
* **Visual pipeline builder** with real-time data previews
* **AI-powered block generation, optimization, and debugging** (via AI Sidekick)
* **Git-backed version control** and CI/CD with **UI-based Deployment App**
* **Built-in access controls**, audit logs, and workspace isolation
* **Support for SQL, Python, R, streaming, dbt blocks, Delta Lake, Iceberg**, and more
* **200+ native connectors** for databases, APIs, and cloud storage
* **Streaming pipelines** for real-time data processing
* **Flexible secrets management**: built-in secret manager or integrate with external systems (AWS Secrets Manager, Vault, etc.)
* **Auto-scaled executors** on Kubernetes or Docker
* **Transparent pricing**: usage-based pricing (SaaS) or cluster/workspace license-based pricing (self-hosted)

> Dagster excels at asset-based orchestration. But when you need **data engineering, visual development, AI assistance, and native data integration** — **Mage Pro is built for modern data teams**.

👉 [Migrate to Mage Pro Today](https://cloud.mage.ai)


# Migrate from dbt to Mage Pro
Source: https://docs.mage.ai/migrations/dbt-to-mage-pro



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=dbt-to-mage-pro)

## **Overview**

Migrating your dbt Cloud project to Mage Pro enables your team to unify analytics and data engineering workflows, combining dbt’s data transformation power with Mage Pro’s orchestration, monitoring, and modular pipeline capabilities. This guide provides a step-by-step process for a seamless migration, allowing you to run, schedule, and monitor dbt models directly inside Mage Pro—alongside SQL, Python, and R blocks.

[https://www.youtube.com/embed/-XRcSzMJKZI](https://www.youtube.com/embed/-XRcSzMJKZI)

**✨ Mage Pro vs dbt Cloud: Benefits Overview**

Mage Pro goes far beyond SQL modeling. It's a unified platform for **data integration**, **transformation (including native dbt support)**, **AI-powered development**, and **streaming pipelines** — all within a collaborative, Git-native workspace.

| **Capability** | **dbt Cloud** | **Mage Pro** |
| - | - | - |
| **Visual pipeline UI** | ⚠️ Basic lineage view only | ✅ Full drag-and-drop pipeline editor |
| **dbt model execution** | ✅ Native | ✅ Native with enhanced orchestration |
| **Multi-language support** | ❌ SQL only | ✅ SQL, Python, R, and dbt blocks |
| **Data ingestion** | ❌ Transformation only | ✅ 200+ built-in connectors |
| **End-to-end pipelines** | ❌ Requires external orchestration | ✅ Complete ETL/ELT workflows |
| **Real-time streaming** | ❌ Batch only | ✅ Kafka, CDC, event-driven processing |
| **AI assistance** | ⚠️ Limited dbt Assist (private beta) | ✅ AI Sidekick for all pipeline development |
| **Pricing model** | ⚠️ Per-seat + model builds (unpredictable) | ✅ Transparent per-seat pricing |
| **Cost predictability** | ❌ Consumption charges can spike unexpectedly | ✅ Fixed monthly costs with no usage surprises |
| **IDE experience** | ✅ Browser-based SQL IDE | ✅ Advanced notebook-style blocks + VS Code integration |
| **Git integration** | ⚠️ Basic Git functionality, complex workflows limited | ✅ Full Git workflows with CI/CD |
| **Orchestration** | ⚠️ Requires Airflow/external tools | ✅ Native scheduling and event triggers |
| **Environment isolation** | ⚠️ Limited workspace management | ✅ Per-workspace configs and secrets |
| **Data previews** | ⚠️ Limited preview capabilities | ✅ Interactive data preview at every step |
| **Scalability** | ⚠️ Limited concurrency (1-5 jobs) | ✅ Auto-scaled execution on K8s/ECS |
| **Security concerns** | ⚠️ Data passes through dbt infrastructure | ✅ Direct warehouse connections |
| **Team collaboration** | ⚠️ Limited to SQL transformations | ✅ Full-stack data team collaboration |

## **Migration Process Outline:**

Developers should create a Git repo and push the base Mage Pro project into their chosen vendor. This could be Github, GitLab, GitBucket, etc. For the sake of this migration tutorial we will be using Github.

## Step 1: Connect Mage Pro to Gituhub

Before migrating your dbt project, you must connect Mage Pro to your GitHub repository to enable version control and deployment capabilities.

### Quick Setup

1. Open **Tools** in the left navigation, then select **Deploy**
2. **Click "Setup connection"** to begin GitHub integration
3. **Authenticate with GitHub** using OAuth tokens
4. **Select your organization and repository** from the dropdown menus
5. **Choose your target branch** (typically `main` or `master`)
6. **Complete the connection** by clicking "Next step to deployment stage"

*For detailed authentication steps, see the* [Version Control Guide](https://docs.mage.ai/guides/data-sync/version-control-guide)*.*

Once authenticated, the Git Terminal automatically uses your credentials for all Git operations without requiring additional setup.

## Step 2: Push base Mage Pro project to Github

After connecting to GitHub, you'll need to push your initial Mage Pro project structure to your repository to establish the foundation for your dbt migration.

### **Quick Setup**

1. **Access Git Terminal** by opening **Tools** in the left navigation and selecting **Terminal**

2. **Stage your project files** using standard Git commands:

   ```bash theme={"system"}
   git add .
   git commit -m "Initial Mage Pro project setup"
   git push origin main
   ```

3. **Verify the push** by checking your GitHub repository for the Mage Pro project structure

NOTE - you may need to force push the initial commit to Github

### **What Gets Pushed**

Your initial commit will include:

* Mage Pro project structure (pipelines, data\_loaders, transformers, etc.)
* Configuration files (`io_config.yaml`, `metadata.yaml`)
* Environment setup files

### **Authentication**

The Git Terminal uses the authentication you configured in Step 1, so no additional setup is required. All standard Git operations work seamlessly within the Mage Pro interface.

*For detailed Git Terminal usage and advanced features, see the [Git Terminal Guide](https://docs.mage.ai/guides/data-sync/git-terminal).*

Once your base project is pushed to GitHub, you can proceed with importing and integrating your existing dbt models into the Mage Pro environment.

## Step 3: Clone your dbt project

After pushing your base Mage Pro project, you'll need to clone your existing dbt project into the Mage environment to begin the migration process.

### Quick Setup

1. **Access Git Terminal** by opening **Tools** in the left navigation and selecting **Terminal**

2. **Navigate to the dbt directory** that Mage provides:

   `cd dbt`

3. **Clone your dbt project** into the dbt folder:

   `git clone https://github.com/your-org/your-dbt-project.git
    cd your-dbt-project`

4. **Verify the clone** by checking that your dbt models, tests, and configurations are present

### Integration Notes

* The dbt folder is specifically designed for dbt project integration
* Your cloned project will work seamlessly with Mage Pro's native dbt support
* Authentication uses the same credentials from your GitHub connection

*For detailed Git Terminal usage and advanced features, see the [Git Terminal Guide](https://docs.mage.ai/guides/data-sync/git-terminal).*

Once your dbt project is cloned into Mage Pro, you can begin creating pipelines that incorporate your existing dbt models alongside new data processing workflows.

## Step 4: Add Mage profiles.yaml file

You will need to custom configure your profiles.yaml file in

After cloning your dbt project, you need to configure the `profiles.yml` file to connect your dbt models to your data warehouse within the Mage Pro environment.

### Quick Setup

1. **Navigate to your dbt project folder** in the Git Terminal:

   `cd dbt/your-dbt-project`

2. **Create or edit the profiles.yml file**:

   `touch profiles.yml`

3. **Add your warehouse configuration** using the appropriate connection details

NOTE - for this step you should use the Mage terminal not the Git terminal

### BigQuery Configuration Example

```yaml theme={"system"}
mage_dbt_demos:
  outputs:
    dev:
      dataset: BigQuery dataset
      job_execution_timeout_seconds: 300
      job_retries: 1
      keyfile: /home/src/<project-name>/<file-name.json>
      location: US
      method: service-account
      priority: interactive
      project: CGP_PROJECT_ID
      threads: 4
      type: bigquery
  target: dev
```

### Key Configuration Notes

* **Profile name** should match your `dbt_project.yml` profile setting
* **Keyfile path** must point to your service account credentials within Mage Pro
* **Dataset and project** should match your BigQuery setup
* **Target environment** determines which output configuration dbt uses

### Environment-Specific Configurations

You can add multiple environments (dev, staging, prod) within the same profile to support different deployment targets as your project grows.

Once configured, your dbt models will be able to connect to your data warehouse and execute transformations within the Mage Pro environment.

## **Step 5: Testing the dbt connection**

After migrating your dbt project to Mage Pro and configuring your `profiles.yml`, you’ll want to verify that Mage Pro can successfully connect to your data warehouse and run dbt commands. This ensures that your credentials, connection details, and environment setup are correct before integrating further workflows.

### **How to Test Your dbt Connection in Mage Pro**

1. **Open Mage Pro and locate your dbt block** within the relevant pipeline.

2. **Click into the dbt block** to access its configuration and terminal interface.

3. **Run a basic dbt command** to test connectivity. For example:

   `dbt debug`

   This command will check the connection parameters defined in your `profiles.yml` and confirm dbt is able to reach your data warehouse.

4. **Verify the output** in the terminal panel:
   * If the last line is `All checks passed!`, your connection is successful.
   * If you see errors, review the credentials in `profiles.yml`, check access permissions, and validate that your warehouse is network accessible from Mage Pro.

## **Step 6: Loading dbt Dependencies After Migrating From GitHub**

When bringing a dbt project from GitHub, you may have dependencies declared in your `packages.yml` file (e.g. dbt-utils). Mage Pro supports full dependency resolution for dbt projects cloned or migrated from GitHub.

### **How to Load dbt Dependencies in Mage Pro**

1. **Open Mage Pro and navigate to your cloned dbt project directory** using the built-in terminal or via the dbt block settings.

2. **Locate your `packages.yml`** file (usually in the root of your dbt project).

3. **Install dependencies** by running:

   `dbt deps`

   This command will download and install all packages specified in your `packages.yml`.

4. **Verify successful installation**:
   * Check the console output for confirmation that packages were installed (look for messages like “Dependencies installed!”).
   * Confirm that new folders such as `dbt_packages` are present in your dbt project directory.

5. **After installing dependencies, you can now compile and run your dbt models** as usual in Mage Pro.

## Step 7: Connect dbt models to Mage blocks

When you connect a dbt block in Mage Pro to an upstream block (Python, SQL, or R), Mage automatically generates and manages a special `mage_sources.yml` file in your dbt project's model directory. This file describes upstream Mage blocks as dbt sources, making their outputs available to your dbt models.

Below is a sample `mage_sources.yml` that gets created when you connect a data loader or transformation block (e.g., Python, SQL, or R) to a downstream dbt model block:

```yaml theme={"system"}
version: 2
sources:
  - description: Dataframes Mage upstream blocks
    loader: mage
    name: mage_dbt_tutorial
    schema: public
    tables:
      - description: Dataframe for block `get_data` of the `dbt_tutorial` mage pipeline.
        identifier: mage_dbt_tutorial_get_data
        meta:
          block_uuid: get_data
          pipeline_uuid: dbt_tutorial
        name: dbt_tutorial_get_data
```

### **Explanation**

* **version**: dbt sources file version (always `2` for dbt v1+).
* **sources**: List of source definitions.
  * **description**: Human-readable note (optional, here noting that sources come from Mage blocks).
  * **loader**: Indicates Mage created the source.
  * **name**: The dbt source name, usually `mage_<pipeline_name>` or similar.
  * **schema**: Target schema for the seed tables (commonly `public`).
  * **tables**: List of upstream Mage blocks exposed as tables to dbt.
    * **description**: Helps you identify the block in Mage.
    * **identifier**: Table identifier in your warehouse, typically `mage_<pipeline_uuid>_<block_uuid>`.
    * **meta**: Traceability information, recording Mage pipeline and block UUIDs.
    * **name**: dbt-level source table name (used as the second argument for `source()` in dbt SQL).

## Step 8: Make changes to your sources in dbt blocks

Mage Pro manages the mapping between upstream pipeline outputs and your dbt models. Any time you update upstream logic—such as renaming a block, changing a pipeline, or adding new data sources—Mage Pro ensures your `mage_sources.yml` file reflects the latest available sources.

To use an upstream Mage block as a source in your dbt model:

1. In Mage Pro, open your dbt model block from the pipeline.

2. Check that your most recent changes to Python, SQL, or R blocks upstream are reflected in the generated `mage_sources.yml` file. For example, if you add a new transformation block called `get_data`, the following entry should appear:

   ```yaml theme={"system"}
   sources:
     - name: mage_dbt_tutorial
       tables:
         - name: dbt_tutorial_get_data
           identifier: mage_dbt_tutorial_get_data
   ```

3. Reference the Mage source in your dbt SQL model using dbt's `source()` macro:

   `select *
    from {{ source('mage_dbt_tutorial', 'dbt_tutorial_get_data') }}`

4. Run and test your dbt model as usual. Model compilation and execution automatically reference the latest outputs from your upstream Mage pipeline logic.

## Conclusion

After completing these steps, you've successfully migrated your dbt project to Mage Pro, creating a unified platform that combines dbt transformations with Mage Pro's orchestration and multi-language support. Your dbt models now run seamlessly alongside Python, SQL, and R blocks, while maintaining GitHub version control. This migration enables more comprehensive data workflows and enhanced collaboration while preserving your existing dbt investments.


# Migrate from Estuary to Mage Pro
Source: https://docs.mage.ai/migrations/estuary-to-mage-pro



Estuary is great for real-time data capture—but teams often hit limits around transformation, orchestration, and control. **Mage Pro** offers an end-to-end solution for batch and streaming pipelines with **Python & SQL transformations**, full **orchestration**, and **AI-powered pipeline generation**.

## Why migrate from Estuary to Mage Pro?

Estuary specializes in real-time syncs with pre-built connectors. But it lacks flexibility for:

* Transforming data beyond the destination database
* Managing complex orchestration and dependencies
* Debugging and testing pipelines visually
* Git-native workflows and team collaboration
* Full control over schema evolution and failure handling

Mage Pro fills these gaps with **streaming + batch pipelines**, modular blocks for ETL, and full visibility into each step of your data flow.

***

## ✨ Mage Pro vs Estuary: Benefits Overview

| Capability | Estuary | Mage Pro |
| - | - | - |
| **Streaming support** | ✅ Native | ✅ Native (Kafka, CDC, webhooks) |
| **Batch processing** | ⚠️ Limited | ✅ Full batch support |
| **Transformations** | ⚠️ Destination-only | ✅ Python, SQL, AI-generated |
| **Orchestration** | ❌ None | ✅ Full DAG-level orchestration |
| **Visual pipeline UI** | ⚠️ Schema-level only | ✅ Drag-and-drop builder |
| **AI assistance** | ❌ | ✅ Sidekick for code & pipeline generation |
| **Custom sources/targets** | ⚠️ Plugin SDK only | ✅ Python-based SDK for APIs, files, DBs |
| **Incremental loads** | ✅ Supported | ✅ Native with customizable logic |
| **Git integration** | ❌ | ✅ Git-backed projects & CI/CD |
| **Environments** | ⚠️ Single workspace | ✅ Dev, staging, and production |
| **Observability** | ⚠️ Basic logs | ✅ Detailed logs, retries, lineage |
| **Hosting** | ✅ Cloud only | ✅ Cloud or self-hosted |
| **Cost** | ⚠️ Event-volume pricing | ✅ Transparent usage-based plans |

***

## 🛠️ Step-by-Step Migration Instructions

1. **Audit your Estuary flows**
   * Export list of **flows** and their source → destination pairs
   * Note any field mappings or transformations
   * Identify real-time vs batch sync requirements

2. **Set up a Mage Pro workspace**
   * Sign up at [Mage Pro](https://cloud.mage.ai)
   * Connect a Git repo if needed for versioning and CI/CD

3. **Recreate flows as Mage pipelines**
   * Estuary source → **Source block** in Mage Streaming Pipeline
   * Estuary destination → **Destination block** in Mage Streaming Pipeline
   * Custom transforms → **SQL** or **Python blocks**
   * Choose between **streaming** (CDC, Kafka, webhook) or **batch**

4. **Configure scheduling or triggers**
   * Use streaming triggers (Kafka, webhooks) or batch schedulers (cron)
   * Add event-based triggers for more control

5. **Validate pipeline output**
   * Run pipelines in staging
   * Preview outputs, compare with Estuary’s destination tables
   * Verify schema integrity, incremental updates, and deduplication

6. **Go live and monitor**
   * Enable alerts, retry logic, and audit logging
   * Use observability tools to track pipeline health and performance
   * Scale pipeline workers automatically via Kubernetes or ECS

***

## 🤖 Use AI Sidekick to Migrate Faster

Estuary flows are often schema-mapped and repetitive—perfect for AI Sidekick to help convert into Mage pipelines.

### 🔧 How to Use It

1. Open a Mage Pro pipeline and click **"Ask AI"**
2. Paste in an Estuary flow JSON or describe your flow
3. Ask: **"Convert this Estuary flow into a Mage pipeline."**
4. The AI will:
   * Create Extract/Load blocks
   * Generate SQL or Python for transforms
   * Add scheduling and triggers
5. Insert and run the pipeline directly in your workspace

***

## 🧠 Tips for Migrating Streaming Workflows

* Use **Kafka Extract blocks** or **webhook listeners** for streaming sources
* Combine **streaming + batch** in hybrid pipelines
* Use **block-level caching** to test transforms on historical data
* Handle schema evolution with **data validation and pipeline tests**

***

## ✅ After Migration: What You Gain

* One platform for **streaming + batch pipelines**
* Visual editor with modular, testable blocks
* AI-powered pipeline creation and debugging
* Git-native development & multi-environment deployments
* Better cost control and full observability
* Built-in orchestration and transformation logic

> Estuary handles streams. Mage Pro handles **everything**—with visibility, scalability, and flexibility built-in.

👉 [Start Migrating to Mage Pro](https://cloud.mage.ai)


# Migrate from Fivetran to Mage Pro
Source: https://docs.mage.ai/migrations/fivetran-to-mage-pro



Many teams start with Fivetran for ELT pipelines—but face limitations in flexibility, cost, and transparency. Mage Pro offers a powerful alternative: a unified platform for **data integration, transformation, and orchestration** with full code control, **AI-assisted development**, and **cloud-native scalability**.

## Why migrate from Fivetran to Mage Pro?

Fivetran is convenient for basic replication—but it wasn’t built for:

* Custom transformations or logic
* Managing orchestration and dependencies
* Running in your own cloud for cost control and security
* Debugging and optimizing individual steps
* Streaming data or event-driven triggers

Mage Pro replaces Fivetran’s “black-box” connector model with **modular, open pipelines** that give you full control over data extraction, transformation, and loading—plus visibility into every step.

***

## ✨ Mage Pro vs Fivetran: Benefits Overview

Mage Pro offers more flexibility, transparency, and scalability than Fivetran—at a fraction of the cost.

| Capability | Fivetran | Mage Pro |
| - | - | - |
| **Data integration** | ✅ Managed connectors | ✅ Native connectors for APIs, DBs, files |
| **Transformation support** | ⚠️ External only (via dbt or dest SQL) | ✅ Built-in SQL & Python support |
| **Orchestration** | ❌ None | ✅ Full DAG-level orchestration |
| **Visual pipeline UI** | ❌ | ✅ Drag-and-drop builder |
| **AI assistance** | ❌ | ✅ AI Sidekick for code & pipeline generation |
| **Incremental loads** | ✅ Supported | ✅ Native with customizable logic |
| **Custom logic** | ⚠️ Limited (no in-flow transforms) | ✅ Full Python & SQL control |
| **Streaming & event triggers** | ❌ | ✅ Real-time CDC, Kafka, webhooks |
| **Data preview & validation** | ⚠️ Destination-only | ✅ Block-level previews & schema validation |
| **Git integration & CI/CD** | ❌ | ✅ Git-native pipelines & deployments |
| **Secrets management** | ⚠️ Env-only | ✅ Built-in secrets & workspace isolation |
| **Multi-environment support** | ⚠️ Manual setup | ✅ Dev, staging, and production workspaces |
| **Observability & debugging** | ⚠️ Basic logs, no pipeline introspection | ✅ Logs, retries, lineage, and alerting |
| **Scalability** | ⚠️ Controlled by Fivetran, usage-priced | ✅ Auto-scaled executors on your infra |
| **Deployment options** | ✅ Cloud, Hybrid, Self-Hosted (HVR) | ✅ Cloud, Hybrid, or Fully On-Prem |
| **Data stays in your VPC** | ✅ With Hybrid deployment | ✅ Always (runs in your infra) |
| **Control plane ownership** | ⚠️ Fivetran-managed (except self-hosted) | ✅ Fully under your control |

***

## 🛠️ Step-by-Step Migration Instructions

1. **Review your Fivetran connectors**
   * List all source → destination syncs
   * Identify transformation steps (dbt, SQL, external pipelines)
   * Export any sync configuration (e.g., schedule, sync mode)

2. **Create your Mage Pro workspace**
   * Sign up at [Mage Pro](https://cloud.mage.ai)
   * (Optional) Connect GitHub or GitLab for version control

3. \*\*Rebuild Fivetran sync using Mage pipeline \*\*
   * Each Fivetran **source** → Mage Data Integration **Source block**
   * Each Fivetran **destination** → Mage Data Integration **Destination block**
   * dbt or SQL logic → Mage **SQL block**
   * Add **Python blocks** for advanced transforms or conditionals

4. **Design pipelines**
   * Combine extract → transform → load in one visual pipeline
   * Use UI builder or YAML configs for setup

5. **Configure schedules and triggers**
   * Replace Fivetran sync schedules with Mage **cron, event, or file triggers**
   * Add webhooks or file watchers if needed

6. **Validate pipelines**
   * Preview outputs at each block
   * Compare target tables and schema to Fivetran outputs
   * Test incremental sync and backfill behavior

7. **Monitor and scale**
   * View logs, lineage, and metrics in Mage
   * Configure **alerts, retries, and concurrency**
   * Enable **auto-scaling** for high-volume loads

***

## 🤖 Use AI Sidekick to Convert Fivetran Configs

Mage Pro’s **AI Sidekick** helps migrate Fivetran configurations, dbt models, and logic into Mage pipelines.

### 🔧 How to Use It

1. Open your Mage Pro workspace
2. Click the **“Ask AI”** field in the Mage Pro header.
3. Paste in your Fivetran connector settings, sync SQL, or dbt model
4. Ask: **“Convert this Fivetran sync into a Mage pipeline.”**
5. The AI will:
   * Generate Extract and Load blocks
   * Create SQL/Python transforms
   * Add schedule and retry logic
6. Insert the generated pipeline into your workspace

***

## 🧠 Tips for Migrating Complex Pipelines

* Use **multiple pipelines** for large source groups (e.g., one per schema)
* Parameterize **table names and filters** with pipeline variables
* Replace dbt logic with **modular SQL blocks** using `ref()` and tests
* Use **batching or streaming** modes for large datasets

***

## ✅ After Migration: What You Gain

* Complete **ELT pipelines** with built-in orchestration
* Visual and modular pipeline builder
* AI-powered development & debugging
* Git-native deployments across environments
* Real-time ingestion with event-based triggers
* Transparent cost, better performance, full control

> Fivetran is great for getting started. But when you're ready for control, observability, and flexibility—**Mage Pro is built for scale**.

👉 [Start Migrating to Mage Pro](https://cloud.mage.ai)


# Migrate from Mage OSS to Mage Pro
Source: https://docs.mage.ai/migrations/mage-oss-to-mage-pro



Many teams start with Mage OSS for building data pipelines—but as they scale, they need enterprise-grade features like multi-environment workspaces, CI/CD deployments, cluster management, and enhanced observability. **Mage Pro** offers a seamless upgrade path with zero code changes, plus exclusive features that make data engineering more productive and reliable.

## Why migrate from Mage OSS to Mage Pro?

Mage OSS is excellent for getting started, but teams often need:

* **Multi-environment isolation** (dev, staging, production)
* **CI/CD workflows** for safe deployments
* **Cluster health monitoring** and resource optimization
* **Pipeline dependency visualization** across projects
* **AI-powered development** assistance
* **Enhanced enterprise security** (granular RBAC, SSO with group mapping, audit logs)
* **Managed infrastructure** with autoscaling
* **Dedicated support** and SLAs

Mage Pro provides all of this while maintaining **100% code compatibility** with Mage OSS—your existing pipelines work without modification.

***

## Common Features: OSS vs Pro

Both Mage OSS and Mage Pro share a strong foundation of core features, ensuring a consistent development experience regardless of which version you choose. Most importantly, both provide a **unified platform** that consolidates all data engineering capabilities in one place.

### Unified Platform Capabilities

Both OSS and Pro offer a single, unified platform that combines:

* **Batch pipelines**: Traditional ETL/ELT workflows for scheduled data processing
* **No-code data integration pipelines**: Pre-built connectors for seamless data ingestion and loading
* **Streaming pipelines**: Real-time data processing for event-driven architectures (Pro includes stateful streaming with checkpointing)
* **Unified ingestion, transformation, and orchestration**: All pipeline types share the same interface, configuration, and execution framework
* **Integrated observability**: Built-in monitoring, logging, and data previews across all pipeline types

This unification means you can build end-to-end data workflows—from ingestion to transformation to orchestration—all within a single platform, without needing separate tools for different pipeline types.

### Features Available in Both OSS and Pro

* **Visual pipeline builder**: Drag-and-drop interface for building data pipelines
* **Multi-language support**: Python, SQL, and R blocks for data transformation
* **Data connectors**: Access to core data sources and destinations (Pro includes 200+ connectors with exclusive options)
* **Built-in secret manager**: Secure storage and management of credentials
* **External vault integration**: Support for AWS Secrets Manager, GCP Secret Manager, Azure Key Vault
* **SSO authentication**: SAML/OAuth/Okta integration for single sign-on
* **TLS encryption**: Secure data in transit
* **Read-only mode**: Protect production notebooks and assets from accidental edits
* **Git integration**: Version control for pipeline code
* **Pipeline scheduling**: Cron-based and event-driven triggers
* **Data previews**: Preview data at each pipeline step
* **Block library**: Reusable code blocks across projects
* **Testing framework**: Built-in testing capabilities for data quality
* **dbt integration**: Run dbt models within pipelines

### When to Use Mage OSS

Mage OSS is ideal for:

* **Small teams or individual developers** getting started with data pipelines
* **Proof of concept projects** or experimental pipelines
* **Organizations with strong DevOps capabilities** who prefer self-managed infrastructure
* **Budget-conscious teams** who can manage their own infrastructure
* **Simple use cases** that don't require multi-environment isolation
* **Teams comfortable with manual Git workflows** and self-service upgrades
* **Organizations with strict data governance** requiring full control over infrastructure

**Best for**: Teams that want the core Mage functionality with full infrastructure control and are comfortable managing deployments, upgrades, and scaling manually.

### When to Use Mage Pro

Mage Pro is recommended for:

* **Growing teams** that need multi-environment workspaces (dev, staging, production)
* **Enterprise organizations** requiring granular RBAC, audit logs, and compliance certifications (SOC 2, ISO 27001)
* **Teams needing CI/CD workflows** with automated deployments and rollback capabilities
* **Organizations requiring managed infrastructure** with autoscaling and high availability
* **Teams building stateful streaming pipelines** or using database CDC (MySQL, PostgreSQL, MSSQL, Oracle)
* **Organizations needing dedicated support** with SLAs and proactive monitoring
* **Teams requiring advanced observability** with cluster health insights and pipeline dependency views
* **Organizations with compliance requirements** that need per-workspace secret scoping and audit trails
* **Teams wanting AI-powered development** assistance with AI Sidekick
* **Organizations needing multi-region deployment** options

**Best for**: Teams that need enterprise-grade features, managed infrastructure, advanced orchestration capabilities, and dedicated support to scale their data engineering operations efficiently.

### Migration Path

Since Mage Pro maintains **100% code compatibility** with Mage OSS, you can start with OSS and seamlessly migrate to Pro when you need additional features. Your pipelines, blocks, and configurations will work without any code changes.

***

## Mage Pro Deployment Options

Mage Pro supports **4 flexible deployment options** to meet your infrastructure and compliance needs:

1. **Mage Pro SaaS** - Fully managed service where Mage maintains the infrastructure and your code executes in our environment. Zero DevOps overhead with automatic scaling and updates.

2. **Mage Pro Hybrid Cloud** - Mage Pro infrastructure maintains the control plane, while the data plane is deployed in your cloud. Best of both worlds: reduced overhead with full data privacy.

3. **Mage Pro Private Cloud** - Both control and data plane are deployed in your cloud infrastructure. Ideal for teams with data residency or compliance requirements who want full control.

4. **Mage Pro On-Prem** - Mage Pro infrastructure is deployed on your on-premises servers. Perfect for organizations with strict data governance or air-gapped environments.

All deployment options provide the same enterprise features, including workspaces, CI/CD deployments, cluster management, and AI Sidekick. Choose the option that best fits your security, compliance, and infrastructure requirements.

***

## Mage Pro vs Mage OSS: Benefits Overview

Mage Pro is more performant, reliable, and cost-effective than Mage OSS, with enterprise-grade features built-in.

| Capability | Mage OSS | Mage Pro |
| - | - | - |
| **Code compatibility** | ✅ | ✅ 100% compatible (no code changes needed) |
| **Workspaces & Multi-tenancy** | ❌ Single project environment | ✅ Workspace-based tenant isolation with hard separation |
| **CI/CD Deployments** | ❌ Manual Git sync | ✅ Git-based CI/CD with environment promotion, approvals, and rollback |
| **Managed upgrades & rollbacks** | ❌ Manual (via images/Helm) | ✅ Coordinated releases and managed rollbacks |
| **Cluster management** | ❌ Self-managed infrastructure | ✅ Cluster health insights, monitoring, and managed infrastructure |
| **Pipeline dependency views** | ❌ Manual tracking | ✅ Visual dependency graph across pipelines |
| **AI assistance** | ❌ | ✅ AI Sidekick for code generation, debugging, and fixes |
| **Enhanced code editor** | ⚠️ Basic editor | ✅ Smart autocomplete, live error checking, documentation hints |
| **Stateful streaming** | ❌ Not available | ✅ Native stateful streaming with checkpointing and operators |
| **Database CDC** | ❌ Not available | ✅ Native Change Data Capture for MySQL, PostgreSQL, MSSQL, and Oracle |
| **Unified IO Config** | ⚠️ Separate configs | ✅ Single config framework across all pipeline types |
| **Data connectors** | ✅ Core connectors | ✅ 200+ connectors + exclusive Pro connectors (e.g., YouTube Analytics) |
| **Spark/Databricks** | ⚠️ Manual setup | ✅ Simplified integration with less configuration |
| **Iceberg support** | ❌ | ✅ Native Iceberg table format support |
| **SSO (SAML/OAuth/Okta)** | ✅ Supported | ✅ Enterprise SSO integration |
| **RBAC** | ⚠️ Basic auth | ✅ Granular roles and permissions at workspace/project levels |
| **Audit logs** | ❌ | ✅ User activity tracking and audit logs for compliance |
| **SCIM provisioning** | ❌ | ✅ Automated user/group lifecycle from your Identity Provider (IdP) |
| **Per-workspace secret scoping** | ❌ | ✅ Workspace-level secret isolation |
| **Static CP egress IPs** | ❌ | ✅ IP allowlisting for Hybrid/Cloud deployments |
| **Autoscaling pipeline orchestration** | ❌ Manual scaling | ✅ Auto-scales pipelines vertically/horizontally |
| **Per-tenant quotas/limits** | ❌ | ✅ Compute and concurrency limits per workspace/tenant |
| **Observability** | ⚠️ Basic logs | ✅ Full dashboards (CPU, memory, errors, alerts) with org-level views |
| **Infrastructure management** | ⚠️ Self-hosted | ✅ 4 deployment options: SaaS, Hybrid Cloud, Private Cloud, On-Prem |
| **SOC 2 / ISO 27001** | ❌ | ✅ Vendor certifications for Pro service/software |
| **Support** | ⚠️ Community | ✅ 24/7 expert support with SLA-backed incident response |
| **Proactive monitoring** | ❌ | ✅ Health checks and post-mortems under support plans |
| **Migration assistance** | ❌ | ✅ Implementation/migration credits and onboarding assistance |

***

## Step-by-Step Migration Instructions

1. **Set up Mage Pro**
   * **For Mage Pro SaaS**:
     * Sign up at [https://cloud.mage.ai](https://cloud.mage.ai) and create an account
     * Create your cluster directly in the cloud UI
     * Proceed to step 2 to create workspaces
   * **For other deployment options** (Hybrid Cloud, Private Cloud, On-Prem):
     * Purchase a license from the Mage team by contacting [pro@mage.ai](mailto:pro@mage.ai)
     * The Mage team will provide detailed deployment instructions for your chosen option
     * Follow the provided instructions to deploy Mage Pro in your cloud infrastructure or on-premises servers
     * Once deployed, proceed to step 2

2. **Enable workspaces (optional)**
   * **For Mage Pro SaaS**: Contact the Mage team at [pro@mage.ai](mailto:pro@mage.ai) to enable the workspaces feature
   * **For other deployment options**: Follow the instructions provided by the Mage team to enable workspaces
   * Once workspaces are enabled, you can create workspaces for different environments:
     * Development workspace for active development
     * Staging workspace (optional) for pre-production validation
     * Production workspace for live pipelines
   * Configure resource limits and Kubernetes settings per workspace

3. **Import your Mage OSS project and set up CI/CD**
   * **Connect your Git repository**:
     * In Mage Pro, navigate to the Deployment settings
     * Connect your Git repository (GitHub, GitLab, Bitbucket, or Azure DevOps)
     * Authenticate using OAuth or SSH keys
   * **Import your project**:
     * Select the branch containing your Mage OSS pipelines
     * Mage Pro will automatically detect and import:
       * All pipelines and blocks
       * Configuration files (including IO configs)
       * Variables and secrets (if stored in Git)
       * Project structure
       * Trigger files (triggers will be created automatically)
     * No code changes required—Mage Pro is 100% compatible with Mage OSS code
   * **Migrate triggers**:
     * **Before migration**: In Mage OSS, save your triggers in code by following the [configure triggers in code guide](/orchestration/triggers/configure-triggers-in-code#configure-triggers-in-code)
     * **After pulling code from GitHub**: When you pull code from GitHub in Mage Pro, the trigger files will be automatically imported and triggers will be created in Mage Pro
     * All your scheduled triggers, event triggers, and API triggers will be automatically set up
   * **Deploy commits**:
     * The deployments page will show a list of commits from your GitHub branch
     * Click the "Deploy" button next to any commit to deploy that version
     * Your pipelines will be updated with the code from the selected commit
   * **Update USER\_CODE\_PATH environment variable**:
     * After pulling code from GitHub, update the `USER_CODE_PATH` environment variable (default value is `/home/src/default_repo`) to point to your Mage project path
     * **For SaaS or Hybrid Cloud**: Configure in the [cloud.mage.ai](https://cloud.mage.ai) portal for the cluster, or in the workspace UI
     * **For Private Cloud or On-Prem**: Configure via Helm chart values for the cluster, or in the workspace UI

4. **Configure environment variables and credentials**
   * **Set up environment variables**:
     * **For SaaS or Hybrid Cloud**:
       * Configure environment variables in the [cloud.mage.ai](https://cloud.mage.ai) portal for each cluster, OR
       * Configure workspace environment variables in the workspace UI (see [workspace environment configuration guide](/guides/developer-ux/workspaces#environment-configuration))
     * **For Private Cloud or On-Prem**:
       * Configure environment variables for the cluster via Helm chart values, OR
       * Configure workspace environment variables in the workspace UI (see [workspace environment configuration guide](/guides/developer-ux/workspaces#environment-configuration))
     * Add environment-specific variables (e.g., `ENV`, `API_BASE_URL`)
     * Use variable interpolation syntax for dynamic values
   * **Upload necessary credentials**:
     * Store API keys, database passwords, and service account credentials in Mage Pro's secrets vault
     * Upload certificate files, SSH keys, or other authentication files as needed
     * Configure secrets at workspace level for environment isolation
   * **Enable database connectivity**:
     * **For Mage Pro SaaS or Hybrid Cloud**: If connecting to databases in private networks:
       * **Static CP egress IPs / IP allowlisting**: Mage Pro provides static outbound IP addresses for the control plane. Get these IP addresses from the cluster settings and whitelist them in your database firewall/security groups
       * **PrivateLink / VPC Peering**: Set up PrivateLink or VPC peering between Mage Pro's network and your private database network for secure, direct connectivity without exposing databases to the internet
       * Contact Mage Pro support to obtain the required IP ranges and VPC peering/PrivateLink configuration details
     * **For Private/Hybrid Cloud**: Configure network connectivity:
       * Ensure proper network routing between Mage Pro workspaces and your databases
       * Set up security groups, firewall rules, or network policies as needed
       * Configure VPC peering or VPN connections if databases are in separate networks
     * **For On-Prem**: Configure network access:
       * Ensure Mage Pro can reach your databases through your internal network
       * Configure firewall rules to allow connections from Mage Pro to database servers
   * **Test database connections**:
     * Use Mage Pro's connection testing feature to verify connectivity
     * Test from each workspace (dev, staging, prod) to ensure proper isolation

5. **Verify data sources and test connections**
   * IO configs are automatically imported from your GitHub repository
   * Update connection strings to use environment variables and secrets (if needed)
   * Verify all data connectors are properly configured
   * Test connections to external systems (databases, APIs, cloud storage) using the connection testing feature

6. **Set up authentication and access control**
   * Configure RBAC (Role-Based Access Control) for team members
   * Set up SSO (Single Sign-On) if needed for enterprise authentication
   * Define workspace-level permissions
   * Configure resource limits and autoscaling policies per workspace

7. **Test and validate pipelines**
   * Run your pipelines in the development workspace
   * Use Mage Pro's data preview feature to validate outputs
   * Check logs and metrics in the observability dashboard
   * Compare results with your Mage OSS environment
   * Verify all database connections and external integrations work correctly

8. **Deploy to production**
   * Navigate to the deployments page in Mage Pro
   * The page will display a list of commits from your GitHub branch
   * Click the "Deploy" button next to the commit you want to deploy to production
   * Monitor pipeline runs and resource usage in real-time
   * Set up alerts for failures or performance issues
   * Leverage pipeline dependency views to understand cross-pipeline relationships

***

## Alternative Migration Method: Manual Export/Import

If you prefer not to use Git-based import, you can manually migrate your project:

1. **Export from Mage OSS**
   * Copy your entire project directory
   * Ensure all pipelines, blocks, and configs are included

2. **Upload to Mage Pro**
   * Use the Git terminal in Mage Pro to push your code
   * Or use the file browser to upload project files directly

3. **Verify and configure**
   * Check that all pipelines are recognized
   * Update any environment-specific configurations
   * Set up workspace-specific variables and secrets
   * Upload credentials and configure database connectivity (see step 5 in migration instructions)

***

## Tips for a Smooth Migration

### Code Compatibility

* **No code changes needed**: All Mage OSS pipelines work in Mage Pro without modification
* **Block types**: All block types (data loaders, transformers, data exporters, sensors) are fully supported
* **Dependencies**: Python dependencies defined in `requirements.txt` are automatically installed
* **IO Configs**: Existing IO configs are automatically imported from your GitHub repository and work as-is, with optional Pro enhancements available
* **Triggers**: Save triggers in code in Mage OSS (see [configure triggers in code guide](/orchestration/triggers/configure-triggers-in-code#configure-triggers-in-code)). When you pull code from GitHub in Mage Pro, trigger files are automatically imported and triggers are created automatically

### Workspace Organization

* **Start with one workspace**: Begin with a single workspace, then create additional ones as needed
* **Use naming conventions**: Name workspaces clearly (e.g., `dev`, `staging`, `prod`)
* **Isolate resources**: Configure different resource limits per workspace based on workload
* **Environment variables**: Use workspace-specific variables for environment differences

### Environment Variables and Credentials

* **Centralize secrets**: Store all credentials in Mage Pro's secrets vault, not in code
* **Workspace isolation**: Use workspace-specific secrets to ensure environment isolation
* **Variable interpolation**: Leverage Mage Pro's enhanced variable interpolation for dynamic configuration
* **Credential migration**: Export credentials from Mage OSS and import them into Mage Pro's secrets vault

### Database Connectivity

* **SaaS deployments**: Contact Mage Pro support early to get IP ranges for whitelisting and VPC peering options
* **Network planning**: Plan network connectivity before migration to avoid delays
* **Test connections**: Always test database connections from each workspace before running pipelines
* **Security groups**: Update security groups and firewall rules to allow Mage Pro access
* **Connection pooling**: Consider connection pooling settings for high-volume pipelines

### CI/CD Best Practices

* **Branch strategy**: Use clear branch naming (main, staging, development)
* **Rollback capability**: Test rollback procedures before going live
* **Approval workflows**: Use manual approvals for production deployments

### Monitoring and Observability

* **Set up alerts**: Configure alerts for pipeline failures and performance issues
* **Monitor resources**: Use cluster health insights to optimize resource allocation
* **Track dependencies**: Use pipeline dependency views to understand relationships
* **Review logs**: Leverage enhanced logging and observability dashboards

***

## After Migration: What You Get with Mage Pro

### Enhanced Developer Experience

* **AI Sidekick**: Get AI-powered code suggestions, explanations, and debugging
* **Enhanced code editor**: Smart autocomplete, live error checking, and documentation hints
* **VSCode & Cursor integration**: Edit and sync directly with your local IDE
* **Personalized UI themes**: Customize your workspace appearance

### Enterprise Features

* **Multi-environment workspaces**: Hard-separated tenant isolation with workspace-based environments
* **CI/CD deployments**: Built-in Git-based deployment with environment promotion, approvals, and rollback
* **Managed upgrades & rollbacks**: Coordinated releases and managed rollback capabilities
* **Cluster health insights**: Real-time CPU and memory monitoring with cost estimates
* **Pipeline dependency views**: Visualize how pipelines orchestrate and depend on each other
* **Enterprise SSO**: Enhanced SAML/OAuth/Okta integration with group mapping per tenant
* **Granular RBAC**: Fine-grained roles and permissions at workspace and project levels
* **SCIM provisioning**: Automated user and group lifecycle management from your Identity Provider (IdP) such as Okta, Azure AD, or Google Workspace
* **Audit logs**: Comprehensive user activity tracking and audit logs for compliance
* **Per-workspace secret scoping**: Workspace-level secret isolation for enhanced security
* **Per-tenant quotas/limits**: Compute and concurrency limits per workspace/tenant
* **Database CDC**: Native Change Data Capture support for MySQL, PostgreSQL, MSSQL, and Oracle

### Advanced Pipeline Capabilities

* **Stateful streaming pipelines**: Run real-time pipelines with built-in state management
* **Database CDC**: Native Change Data Capture for MySQL, PostgreSQL, MSSQL, and Oracle databases
* **Unified IO Config Framework**: Single config structure across all pipeline types
* **Exclusive connectors**: Access to Pro-only data sources and destinations
* **Enhanced integrations**: Optimized Spark, Databricks, and Iceberg support
* **More variable interpolation**: Support for env vars, secrets, and config vars

### Production-Grade Infrastructure

* **Autoscaling pipeline orchestration**: Automatically scale pipelines vertically and horizontally to handle workload spikes
* **High availability**: Built-in redundancy and failover capabilities
* **Multi-region deployment**: Deploy in US, Canada, Europe, Asia, or Australia
* **Static CP egress IPs**: IP allowlisting support for Hybrid and Cloud deployments
* **PrivateLink / VPC peering**: Direct private connectivity to Mage-hosted control plane (Hybrid/Cloud)
* **SOC 2 / ISO 27001**: Vendor certifications for Pro service and software
* **24/7 expert support**: SLA-backed incident response and expert guidance
* **Proactive monitoring**: Health checks and post-mortems under support plans
* **Migration assistance**: Implementation and migration credits with onboarding assistance
* **Managed infrastructure**: Zero DevOps overhead with fully managed deployments and coordinated upgrades

***

## Getting Help

* **Documentation**: Explore the [Mage Pro documentation](/production/mage/pro)
* **Support**: Contact Mage Pro support for migration assistance
* **Community**: Join the [Mage community](https://www.mage.ai/chat) for tips and best practices

> Your pipelines, your logic—now with enterprise-grade infrastructure, AI assistance, and zero DevOps overhead.

👉 [Start Your Migration to Mage Pro Today](https://cloud.mage.ai)


# Migrate from Prefect to Mage Pro
Source: https://docs.mage.ai/migrations/prefect-to-mage-pro



Many teams start with Prefect for workflow orchestration—but soon encounter challenges with data engineering workflows, visual debugging, and developer productivity. Mage Pro offers a modern alternative: visual pipelines, built-in data integration, AI-assisted development, and UI-based deployment management—all within a unified platform for your data workflows.

## Why migrate from Prefect to Mage Pro?

Prefect is a powerful orchestration tool, but it wasn't designed for:

* Visual pipeline development and debugging
* Native data integration (ETL/ELT workflows)
* Mixing SQL and Python naturally in the same pipeline
* Streaming or event-driven data processing
* Built-in data previews and interactive development
* AI-powered pipeline development

Mage Pro is built to solve these challenges out-of-the-box, combining orchestration with data engineering capabilities in a modern, developer-friendly platform.

***

## Mage Pro vs Prefect: Benefits Overview

Mage Pro goes far beyond orchestration. It's a unified platform for **data integration**, **SQL modeling (native dbt blocks and Mage SQL blocks)**, **AI-powered transformation**, and **streaming pipelines** — all within a collaborative, Git-native workspace with UI-based deployment management.

| Capability | Prefect | Mage Pro |
| - | - | - |
| **Visual pipeline UI** | ⚠️ Code-only (Python decorators) | ✅ Hybrid UI: drag-and-drop DAG builder + notebook-style coding experience |
| **Built-in lineage** | ⚠️ Requires external tools | ✅ Native, auto-generated data lineage |
| **AI assistance** | ❌ | ✅ Generate, optimize, fix, and explain code with AI Sidekick |
| **Multi-language support** | ⚠️ Python only | ✅ SQL, Python, R, streaming, APIs |
| **Data integration pipelines** | ⚠️ Custom Python or 3rd-party tools | ✅ 200+ native connectors for databases, files, APIs |
| **SQL block support** | ⚠️ Requires custom tasks | ✅ Native dbt blocks, or Mage SQL block, preview, and test support |
| **Incremental modeling** | ❌ (requires custom logic) | ✅ Native in SQL block config and Data integration pipeline |
| **Data previews & interactive runs** | ❌ | ✅ Preview data at every step, run blocks independently |
| **Environment isolation** | ⚠️ Requires separate deployments | ✅ Per-workspace configs, secrets, variables |
| **Git integration** | ⚠️ Manual sync | ✅ Git-backed version control and CI/CD, UI based Deployment App |
| **Scheduling & triggers** | ✅ Cron + event triggers | ✅ Cron, events, file triggers, webhooks, API |
| **Secrets management** | ⚠️ Requires Prefect Cloud or external Vault | ✅ Support both built-in secret manager, or external secret manager |
| **RBAC & SSO** | ⚠️ Prefect Cloud only | ✅ Built-in, enterprise-ready |
| **Observability & logging** | ✅ Prefect Cloud UI | ✅ Native UI for logs, metrics, traces, lineage |
| **Scalability** | ⚠️ Manual worker configuration | ✅ Auto-scaled executors on K8s or Docker |
| **Streaming support** | ❌ | ✅ Kafka, CDC, real-time ingestion |
| **Team collaboration** | ⚠️ Limited multi-tenancy | ✅ Workspaces, permissions, activity logs |

***

## Step-by-Step Migration Instructions

1. **Inventory your Prefect flows**
   * List your active flows and their task types:
     * `@flow` and `@task` decorators
     * Prefect blocks (storage, infrastructure)
     * Dependencies and scheduling logic
   * Identify data sources, transformations, and destinations

2. **Create a workspace in Mage Pro**
   * Visit [Mage Pro](https://cloud.mage.ai) and sign in
   * (Optional) Connect your Git repo for version control
   * Set up your deployment configuration using the **Deployment App** in the UI

3. **Recreate your Prefect flows as Mage pipelines**
   * Each Prefect **flow** → 1 Mage pipeline
   * Each Prefect **task** → 1 Mage block:
     * `@task` with data loading → **Data loader block**
     * `@task` with transformations → **Transformer block** (Python or SQL)
     * `@task` with data writing → **Data exporter block**
     * Prefect blocks (storage) → **Data loader/exporter blocks**
   * Define dependencies visually using the UI (no decorators needed)

4. **Configure scheduling and triggers**
   * Prefect deployments → Mage **triggers** (cron, interval, or event-based)
   * Prefect schedules → Mage **schedule triggers**
   * Configure schedules via UI or YAML

5. **Run and validate pipelines**
   * Use Mage Pro's UI to test individual blocks
   * Preview dataframes, logs, and outputs at each step
   * Compare results to your Prefect flows

6. **Monitor, scale, and automate**
   * Monitor pipeline runs and resource usage
   * Set alerts and retry rules
   * Mage Pro autoscaling handles executor resources automatically on Kubernetes or Docker
   * Use the **Deployment App** to manage deployments and rollbacks

***

## Mapping Prefect Concepts to Mage Pro

### Flows → Pipelines

**Prefect:**

```python theme={"system"}
from prefect import flow, task

@flow(name="my-etl-flow")
def my_etl_flow():
    data = extract_data()
    transformed = transform_data(data)
    load_data(transformed)
```

**Mage Pro:**

* Create a new **pipeline** in the UI
* Add blocks visually or via code
* Dependencies are defined by connecting blocks in the UI

### Tasks → Blocks

**Prefect:**

```python theme={"system"}
@task
def extract_data():
    return pd.read_csv("data.csv")

@task
def transform_data(df):
    return df.dropna()

@task
def load_data(df):
    df.to_sql("table", engine)
```

**Mage Pro:**

* `extract_data` → **Data loader block** (Python or connector)
* `transform_data` → **Transformer block** (Python, SQL, or dbt block)
* `load_data` → **Data exporter block** (Python or connector)

### Prefect Blocks → Mage Blocks

| Prefect Block Type | Mage Pro Equivalent |
| - | - |
| `S3Block`, `GCSBlock` | **Data loader/exporter blocks** with native S3/GCS connectors |
| `PostgresBlock`, `SnowflakeBlock` | **Data loader/exporter blocks** with database connectors |
| `DockerBlock`, `KubernetesJob` | **Executor configuration** in pipeline settings |

### Scheduling & Deployments

**Prefect:**

```python theme={"system"}
from prefect import serve
from prefect.schedules import CronSchedule

@flow
def scheduled_flow():
    ...

if __name__ == "__main__":
    scheduled_flow.serve(
        cron=CronSchedule(cron="0 0 * * *")
    )
```

**Mage Pro:**

* Create a **schedule trigger** in the pipeline UI
* Configure cron expression, interval, or event-based triggers
* Use the **Deployment App** to manage deployments across environments (dev, staging, prod)
* No complex deployment setup needed—triggers and deployments are configured via UI

### Secrets & Configuration

**Prefect:**

```python theme={"system"}
from prefect.blocks.system import Secret

@task
def use_secret():
    api_key = Secret.load("my-api-key")
```

**Mage Pro:**

* Use **workspace variables** or **pipeline variables**
* Access via `variables` dictionary in blocks
* **Built-in secret manager** with encryption, or integrate with **external secret managers** (AWS Secrets Manager, HashiCorp Vault, etc.)
* Secrets are isolated per workspace and pipeline

***

## Convert Prefect Flows with AI Sidekick

Skip the manual rewrites — Mage Pro's **AI Sidekick** can automatically convert your Prefect flow code into a Mage pipeline with just one prompt.

### How to Use It

1. Click the **"Ask AI"** field in the Mage Pro header.
2. Paste your Prefect flow code — including `@flow`, `@task`, or flow-based workflows.
3. Ask: **"Convert this Prefect flow to a Mage pipeline."**
4. Sidekick will:
   * Parse the flow structure and task relationships
   * Generate the corresponding Mage blocks (data loader, transformer, data exporter)
   * Define block dependencies and scheduling logic
   * Convert Prefect blocks to Mage connectors where applicable
   * **Optimize** the generated code for performance and best practices
5. Review and insert the generated pipeline directly into your project.

### Why Use Sidekick?

* **Faster migration**: turn entire flows into Mage pipelines in seconds
* **Less error-prone**: Sidekick understands scheduling, dependencies, and task types
* **Code optimization**: automatically optimizes generated code for performance
* **Context-aware**: uses your project setup and prior block structure
* **Fully editable**: review, tweak, and insert blocks before saving

👉 Learn more in the [AI Sidekick docs](/ai/sidekick).

***

## Tips for Migrating Complex Flows

### Handling Prefect Subflows

**Prefect:**

```python theme={"system"}
@flow
def parent_flow():
    result = subflow()

@flow
def subflow():
    ...
```

**Mage Pro:**

* Split into **separate pipelines** and trigger them via API or use **dynamic blocks** for conditional logic
* Use **block groups** to organize related blocks within a pipeline

### Prefect Task Runners & Concurrency

**Prefect:**

```python theme={"system"}
from prefect.task_runners import ConcurrentTaskRunner

@flow(task_runner=ConcurrentTaskRunner())
def parallel_flow():
    ...
```

**Mage Pro:**

* Configure **concurrency settings** in pipeline configuration
* Blocks with no dependencies run in parallel automatically
* Use **executor configuration** for custom parallelism on Kubernetes or Docker

### Prefect Retries & Error Handling

**Prefect:**

```python theme={"system"}
@task(retries=3, retry_delay_seconds=60)
def risky_task():
    ...
```

**Mage Pro:**

* Configure **retry settings** in block or pipeline configuration
* Set retry count, delay, and backoff strategy
* Built-in error handling and alerting

### Prefect Caching

**Prefect:**

```python theme={"system"}
@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=1))
def cached_task():
    ...
```

**Mage Pro:**

* Use **block output caching** in pipeline settings
* Configure cache expiration and invalidation rules
* Preview cached outputs in the UI

***

## After Migration: What You Get with Mage Pro

* **Unified data platform**: orchestration + data integration + transformation + streaming in one tool
* **Visual pipeline builder** with real-time data previews
* **AI-powered block generation, optimization, and debugging** (via AI Sidekick)
* **Git-backed version control** and CI/CD with **UI-based Deployment App**
* **Built-in access controls**, audit logs, and workspace isolation
* **Support for SQL, Python, R, streaming, dbt blocks, Delta Lake, Iceberg**, and more
* **200+ native connectors** for databases, APIs, and cloud storage
* **Streaming pipelines** for real-time data processing
* **Flexible secrets management**: built-in secret manager or integrate with external systems (AWS Secrets Manager, Vault, etc.)
* **Auto-scaled executors** on Kubernetes or Docker
* **Transparent pricing**: usage-based pricing (SaaS) or cluster/workspace license-based pricing (self-hosted)

> Prefect excels at orchestration. But when you need **data engineering, visual development, and AI assistance** — **Mage Pro is built for modern data teams**.

👉 [Migrate to Mage Pro Today](https://cloud.mage.ai)


# Alerting status updates in Discord
Source: https://docs.mage.ai/observability/alerting/alerting-discord

Get status updates in your Discord channel.

![Discord](https://cdn-icons-png.flaticon.com/512/2111/2111370.png)

## Overview of steps

1. Set up webhooks for Discord
2. Update Mage project settings

***

## Configure alerts in the Mage Pro UI

After you create your Discord webhook, you can configure alerts in the Mage Pro UI
instead of editing `metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Discord, enter the webhook URL, and use
   **Send test notification** to send a test alert before saving.

You can still reference secrets with values such as
`{{ env_var('MAGE_DISCORD_WEBHOOK_URL') }}`.

***

## Set up webhooks for Discord

Follow these
[instructions on Discord](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks)
to setup incoming webhooks for your workspace.

***

## Update Mage project settings

Once you’ve set up webhooks for Discord, you should have a webhook URL that Discord
provides.

Here is an example webhook (yours may vary):

```
https://discordapp.com/api/webhooks/XXX3514XXXXX/MHhahaduwa183-ccgfxcgv88kladJkjBJKB78CHCfds
```

Follow these steps to add that webhook URL to your project settings:

1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).

2. Open a pipeline and start editing it (e.g.
   [http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).

3. In your left sidebar in the file browser, scroll all the way down and click
   on a file named `metadata.yaml`.

4. In the `metadata.yaml` file, add a section with the following
   configuration or update the existing `notification_config` section:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       - trigger_failure
       - trigger_passed_sla
   ```

   If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.

   Options:

   * `trigger_failure`: alert when a run of a trigger fails
   * `trigger_success`: alert when a run of a trigger succeeds
   * `trigger_passed_sla`: alert when a SLA is missed.
     * In this context, SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
     * If the pipeline runs *longer* than the SLA, an alert will be sent.

5. In the `metadata.yaml` file, add the following values:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       ...
     discord_config:
       webhook_url: ...
   ```

   Change the `webhook_url` value to be the webhook URL you created from Discord.

## Customize message templates

You can customize message templates in `notification_config`. Here is an example config:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  discord_config:
    ...
  message_templates:
    failure:
      title: 'Failed to execute pipeline {pipeline_uuid} at {execution_time}'
      summary: >
        Failed to execute pipeline {pipeline_run_url}.
        Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
        Test custom message.
    success:
      title: ...
      summary: ...
    passed_sla:
      title: ...
      summary: ...
```

You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify the sentence on `summary` and the `title`.

To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:

1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
   * available only for the `failure` message template
8. `stacktrace`
   * available only for the `failure` message template

> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.

### 🚀 Pro Only: Interpolate Mage Variables in Message Templates

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=notification)

If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Discord message templates using the familiar Jinja-like syntax:

```yaml theme={"system"}
notification_config:
  message_templates:
    failure:
      details: >
        Failed at {{ env_var('ENVIRONMENT') }} environment.
        Pipeline run: {pipeline_run_url}.
```

In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:

1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)

This makes your Discord notifications more dynamic and context-aware — ideal for multi-environment deployments and advanced workflows.

***

## What next?

Whenever a pipeline run is successfully completed or fails, a Discord Channel message will
appear in the channel you configured the webhook URL for.

Here is an example of what that message could look like:

| |
| - |
| ![Discord](https://mage-ai.github.io/assets/third-party/discord-message-run-alert.jpg) |


# Alerting status updates in Email
Source: https://docs.mage.ai/observability/alerting/alerting-email

Get status updates sent to your email inbox.

![Email](https://yourreclaimedlife.com/wp-content/uploads/2018/12/email-marketing-without-permission.jpg)

## Configure alerts in the Mage Pro UI

You can configure email alerts in the Mage Pro UI instead of editing
`metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Email, and select a delivery method:
   **Custom SMTP** or **Mage managed email**.
4. Enter recipients or use managed API default recipients when they are
   configured, then use **Send test notification** before saving. Custom SMTP
   sends through your SMTP server; Mage managed email sends through the managed
   notification service.

You can still reference secrets with values such as
`{{ env_var('MAGE_SMTP_PASSWORD') }}`.

***

## Create notification\_config

In the root of your project’s folder (e.g. `default_repo/`), open the file
`metadata.yaml`.

> Project folder name
>
> If you initialized Mage using a different project name, then your root folder
> will be named differently. `default_repo` is the default project name if you
> didn’t customize it.

In the `default_repo/metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:

```yaml theme={"system"}
notification_config:
  alert_on:
    - trigger_failure
    - trigger_passed_sla
```

If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.

Options:

* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a run of a trigger passes sla

## Configure email settings

In the `default_repo/metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  email_config:
    smtp_host: ...
    smtp_mail_from: ...
    smtp_user: ...
    smtp_password: ...
    to_emails:
      - someone_lucky@mage.ai
      - eng@mage.ai
```

Change the values for each key under the `email_config` section.

## Customize message templates

You can customize message templates in `notification_config`. Here is an example config:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  email_config:
    ...
  message_templates:
    failure:
      title: Failed to execute pipeline {pipeline_uuid} at {execution_time}
      details: >
        Failed to execute pipeline {pipeline_run_url}.
        Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
        Test custom message.
    success:
      title: ...
      details: ...
    passed_sla:
      title: ...
      details: ...
```

You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify `title` and either `summary` or `details`.

* `title` is used in email subject
* If you specify the `summary`, the email body will be your `summary` + the url of the pipeline run page
* If you specify the `details`, the `details` will be used as your email body

To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:

1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
   * available only for the `failure` message template
8. `stacktrace`
   * available only for the `failure` message template

> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.

### 🚀 Pro Only: Interpolate Mage Variables in Message Templates

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=notification)

If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Email message templates using the familiar Jinja-like syntax:

```yaml theme={"system"}
notification_config:
  message_templates:
    failure:
      details: >
        Failed at {{ env_var('ENVIRONMENT') }} environment.
        Pipeline run: {pipeline_run_url}.
```

In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:

1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)

This makes your Email notifications more dynamic and context-aware — ideal for multi-environment deployments and advanced workflows.

### Managed notification email delivery

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=managed-notifications)

Mage Pro can send pipeline status emails through the managed notification delivery API instead of requiring each
project to own SMTP credentials. The project or pipeline `metadata.yaml` owns who receives alerts, while the
companion Mage Pro API deployment owns the provider credentials, idempotency, rate limiting, and delivery.
Managed email uses the same notification title and body as SMTP email, including the pipeline run link when
available.

In the Mage UI, enable Email and choose **Mage managed email** as the delivery
method. In `metadata.yaml`, enable managed email delivery with
`notification_config.managed_config`:

```yaml theme={"system"}
notification_config:
  alert_on:
    - trigger_failure
    - trigger_passed_sla
  managed_config:
    enabled: true
    channels:
      - email
    email_config:
      to_emails:
        - data-oncall@example.com
    timeout: 10
```

Fields:

* `enabled`: opt in to managed delivery.
* `channels`: currently supports `email` only. Include `email` explicitly when you want managed email
  delivery.
* `email_config.to_emails`: optional direct customer email recipients. Mage trims, deduplicates, and forwards
  these addresses to the managed API.
* `timeout`: total seconds Mage waits for the managed API token exchange and delivery request. Mage caps this
  value at `30` seconds. Provider-specific request timeouts are configured in the Mage Pro API service.

Mage only forwards `email_config` when `to_emails` includes at least one address. Leave `to_emails`
empty to use managed API default recipients when the operator has configured them.

Managed notifications are email-only in this implementation. Do not configure `sms`, `sms_config`,
`to_phone_numbers`, or phone recipients.

Operators configure the companion Mage Pro API with deployment environment variables:

| Environment variable | Default | Description |
| - | - | - |
| `MANAGED_NOTIFICATION_DRY_RUN` | `true` | Keeps the API from contacting live providers while the API contract, recipient resolution, idempotency, and logging are validated. Set to `false` only after live recipients and provider credentials are approved. |
| `MANAGED_NOTIFICATION_EMAIL_PROVIDER` | `ses` | Email provider. AWS SES SMTP is the default; SES aliases include `ses`, `ses-smtp`, `ses_smtp`, `aws-ses`, `aws_ses`, `aws-ses-smtp`, and `aws_ses_smtp`. Use `resend` for the legacy Resend provider. |
| `MANAGED_NOTIFICATION_IDEMPOTENCY_TTL_SECONDS` | `86400` | How long the API replays completed idempotent responses and keeps duplicate-send protection for deterministic delivery keys. |
| `MANAGED_NOTIFICATION_RATE_LIMIT_PER_MINUTE` | `60` | Per-cluster managed notification rate limit. Values less than or equal to `0` disable the limit. |
| `MANAGED_NOTIFICATION_PROVIDER_TIMEOUT_SECONDS` | `10` | Provider request timeout used by SES SMTP or Resend delivery, and the short wait budget for in-progress idempotent responses. |

For AWS SES SMTP, configure:

* `AWS_SES_FROM_EMAIL`: sender address. `SES_FROM_EMAIL` is also accepted as a fallback.
* `AWS_SES_USERNAME` and `AWS_SES_PASSWORD`: SES SMTP credentials.
* `AWS_SES_REGION`, `AWS_REGION`, or `AWS_DEFAULT_REGION`: used to build `email-smtp.<region>.amazonaws.com`
  when `AWS_SES_SMTP_HOST` is not set.
* `AWS_SES_SMTP_HOST`: optional explicit SMTP host.
* `AWS_SES_SMTP_PORT`: SMTP port, default `587`.
* `AWS_SES_SMTP_STARTTLS`: whether to call STARTTLS, default `true`.

For Resend, set `MANAGED_NOTIFICATION_EMAIL_PROVIDER=resend`, `RESEND_API_KEY`, and `RESEND_FROM_EMAIL`.

***

## What next?

Whenever a pipeline run is successfully completed or fails, an email will be
delivered to all the inboxes listed under `to_emails`.

Here is an example of what an email could look like:

| |
| - |
| Successfully ran Pipeline `example_pipeline` with Trigger 79 `hourly_trigger` at execution time `2022-09-28 19:00:00`. Open [http://localhost:6789/pipelines/example\_pipeline/triggers/79](http://localhost:6789/pipelines/example_pipeline/triggers/79) to check pipeline run results and logs. |


# Alerting status updates in Opsgenie
Source: https://docs.mage.ai/observability/alerting/alerting-opsgenie

Get status updates in Opsgenie.

![Opsgenie](https://logotyp.us/files/opsgenie.svg)

## Overview of steps

1. Set up Api Integration for Opsgenie
2. Update Mage project settings

***

## Configure alerts in the Mage Pro UI

After you create your Opsgenie API integration, you can configure alerts in the
Mage Pro UI instead of editing `metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Opsgenie, enter the API URL and API key,
   and use **Send test notification** to send a test alert before saving.

You can still reference secrets with values such as
`{{ env_var('MAGE_OPSGENIE_API_KEY') }}`.

***

## Set up Api Integration for Opsgenie

Follow these
[instructions on Opsgenie](https://support.atlassian.com/opsgenie/docs/create-a-default-api-integration/)
to setup an API Integration in your account.

***

## Update Mage project settings

Once you’ve set up API Integration for Opsgenie, you should have an API Key that Opsgenie
provides.

Follow these steps to add that Opsgenie Configuration to your project settings:

1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).

2. Open a pipeline and start editing it (e.g.
   [http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).

3. In your left sidebar in the file browser, scroll all the way down and click
   on a file named `metadata.yaml`.

4. In the `metadata.yaml` file, add a section with the following
   configuration or update the existing `notification_config` section:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       - trigger_failure
       - trigger_passed_sla
   ```

   If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.

   Options:

   * `trigger_failure`: alert when a run of a trigger fails
   * `trigger_success`: alert when a run of a trigger succeeds
   * `trigger_passed_sla`: alert when a run of a trigger passes sla

5. In the `metadata.yaml` file, add the following values:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       ...
     opsgenie_config:
       url: "https://api.eu.opsgenie.com/v2/alerts"
       api_key: "<your_api_key>"
       priority: "P2" # Default P3
       tags: # Default empty
       - production
       details: # Default empty
         environment: production
       responders: # Default empty
       - id: "4513b7ea-3b91-438f-b7e4-e3e54af9147c"
         type: "team"
   ```

   Change the `your_api_key` value to be the API Key you created from Opsgenie.

***

## What next?

Whenever a pipeline run is successfully completed or fails, an Opsgenie alert be
created.

Here is an example of what the alert could look like:

| |
| - |
| ![Opsgenie](https://mage-ai.github.io/assets/opsgenie-alert-example.jpeg) |


# Alerting status updates in Slack
Source: https://docs.mage.ai/observability/alerting/alerting-slack

Get real-time status updates from Mage pipelines directly in your Slack channels.

![Slack](https://d34u8crftukxnk.cloudfront.net/slackpress/prod/sites/6/2019-01_BrandRefresh_Old-to-New-Final.gif)

## Overview of steps

1. Set up webhooks for Slack
2. Update Mage project settings
3. Customize message templates with dynamic variables (Pro only)

***

## Configure alerts in the Mage Pro UI

After you create your Slack webhook, you can configure alerts in the Mage Pro UI
instead of editing `metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Slack, enter the webhook URL, and use
   **Send test notification** to send a test alert before saving.

You can still reference secrets with values such as
`{{ env_var('MAGE_SLACK_WEBHOOK_URL') }}`.

***

## Set up webhooks for Slack

To send notifications to Slack, you need to configure an incoming webhook:

* Follow the official [Slack webhook setup guide](https://api.slack.com/messaging/webhooks).
* Create a webhook URL for your desired Slack channel.

***

## Update Mage project settings

Once you’ve set up webhooks for Slack, you should have a webhook URL that Slack
provides.

Here is an example webhook (yours may vary):

```
https://hooks.slack.com/services/T01XXXXRKMJ/B04XXXXGY67/1UgUkao8pXXXXPmsuXXXX2lk
```

Follow these steps to add that webhook URL to your project settings:

1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).

2. Open a pipeline and start editing it (e.g.
   [http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).

3. In your left sidebar in the file browser, scroll all the way down and click
   on a file named `metadata.yaml`.

4. In the `metadata.yaml` file, add a section with the following
   configuration or update the existing `notification_config` section:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       - trigger_failure
       - trigger_passed_sla
   ```

   If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.

   Options:

   * `trigger_failure`: alert when a run of a trigger fails
   * `trigger_success`: alert when a run of a trigger succeeds
   * `trigger_passed_sla`: alert when a SLA is missed.
     * In this context, SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
     * If the pipeline runs *longer* than the SLA, an alert will be sent.

5. In the `metadata.yaml` file, add the following values:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       ...
     slack_config:
       webhook_url: ...
   ```

   Change the `webhook_url` value to be the webhook URL you created from Slack.

## Customize message templates

You can customize message templates in `notification_config`. Here is an example config:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  slack_config:
    ...
  message_templates:
    failure:
      details: >
        Failed to execute pipeline {pipeline_run_url}.
        Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
        Test custom message.
    success:
      details: ...
    passed_sla:
      details: ...
```

You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify either `summary` or `details`.

* If you specify the `summary`, the slack message will be your `summary` + the url of the pipeline run page
* If you specify the `details`, the `details` will be used as your slack message directly

To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:

1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
   * available only for the `failure` message template
8. `stacktrace`
   * available only for the `failure` message template

> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.

### 🚀 Pro Only: Interpolate Mage Variables in Message Templates

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=notification)

If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Slack message templates using the familiar Jinja-like syntax:

```yaml theme={"system"}
notification_config:
  message_templates:
    failure:
      details: >
        Failed at {{ env_var('ENVIRONMENT') }} environment.
        Pipeline run: {pipeline_run_url}.
```

In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:

1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)

This makes your Slack notifications more dynamic and context-aware — ideal for multi-environment deployments and advanced workflows.

***

## What next?

Whenever a pipeline run is successfully completed or fails, a Slack message will
appear in the channel you configured the webhook URL for.

Here is an example of what that message could look like:

| |
| - |
| ![Slack](https://mage-ai.github.io/assets/third-party/slack-message-run-alert.jpg) |


# Alerting status updates in Teams
Source: https://docs.mage.ai/observability/alerting/alerting-teams

Get status updates in your Teams channel.

![Teams](https://img.icons8.com/color-glass/512/microsoft-teams.png)

## Overview of steps

1. Set up webhooks for Teams
2. Update Mage project settings

***

## Configure alerts in the Mage Pro UI

After you create your Teams webhook, you can configure alerts in the Mage Pro UI
instead of editing `metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Microsoft Teams, enter the webhook URL,
   and use **Send test notification** to send a test alert before saving.

You can still reference secrets with values such as
`{{ env_var('MAGE_TEAMS_WEBHOOK_URL') }}`.

***

## Set up webhooks for Teams

Follow these
[instructions on Teams](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook)
to setup incoming webhooks for your workspace.

***

## Update Mage project settings

Once you’ve set up webhooks for Teams, you should have a webhook URL that Teams
provides.

Here is an example webhook (yours may vary):

```
https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lk
```

Follow these steps to add that webhook URL to your project settings:

1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).

2. Open a pipeline and start editing it (e.g.
   [http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).

3. In your left sidebar in the file browser, scroll all the way down and click
   on a file named `metadata.yaml`.

4. In the `metadata.yaml` file, add a section with the following
   configuration or update the existing `notification_config` section:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       - trigger_failure
       - trigger_passed_sla
   ```

   If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.

   Options:

   * `trigger_failure`: alert when a run of a trigger fails
   * `trigger_success`: alert when a run of a trigger succeeds
   * `trigger_passed_sla`: alert when a run of a trigger passes sla

5. In the `metadata.yaml` file, add the following values:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       ...
     teams_config:
       webhook_url:
         - 'https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lk'
         - 'https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lx'
   ```

   Change the `webhook_url` values to be the webhook URLs you created from Teams.

   The `webhook_url` must be a string if you are using version 0.9.73 or older and it only accepts one endpoint.

   ```
   teams_config:
       webhook_url: 'https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lk'
   ```

## Customize message templates

You can customize message templates in `notification_config`. Here is an example config:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  teams_config:
    ...
  message_templates:
    failure:
      details: >
        Failed to execute pipeline {pipeline_run_url}.
        Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
        Test custom message.
    success:
      details: ...
    passed_sla:
      details: ...
```

You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify either `summary` or `details`.

* If you specify the `summary`, the Teams message will be your `summary` + the url of the pipeline run page
* If you specify the `details`, the `details` will be used as your Teams message directly

To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:

1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
   * available only for the `failure` message template
8. `stacktrace`
   * available only for the `failure` message template

> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.

### 🚀 Pro Only: Interpolate Mage Variables in Message Templates

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=notification)

If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Teams message templates using the familiar Jinja-like syntax:

```yaml theme={"system"}
notification_config:
  message_templates:
    failure:
      details: >
        Failed at {{ env_var('ENVIRONMENT') }} environment.
        Pipeline run: {pipeline_run_url}.
```

In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:

1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)

This makes your Teams notifications more dynamic and context-aware — ideal for multi-environment deployments and advanced workflows.

***

## What next?

Whenever a pipeline run is successfully completed or fails, a Teams message will
appear in the channel you configured the webhook URL for.

Here is an example of what that message could look like:

| |
| - |
| ![Teams](https://mage-ai.github.io/assets/third-party/teams-message-run-alert.jpg) |


# Alerting status updates in Telegram
Source: https://docs.mage.ai/observability/alerting/alerting-telegram

Get status updates in your Telegram Group.

![Telegram](https://cdn0.iconfinder.com/data/icons/social-media-2092/100/social-56-512.png)

## Overview of steps

1. Set Bot with **BotFather** for Telegram
2. Get the chat\_id of your Telegram Group
3. Update Mage project settings

***

## Configure alerts in the Mage Pro UI

After you create your Telegram bot webhook URL, you can configure alerts in the
Mage Pro UI instead of editing `metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Telegram, enter the webhook URL, and use
   **Send test notification** to send a test alert before saving.

You can still reference secrets with values such as
`{{ env_var('MAGE_TELEGRAM_WEBHOOK_URL') }}`.

***

## Set Bot with BotFather for Telegram

Follow these
[Tutorial](https://www.directual.com/lesson-library/how-to-create-a-telegram-bot)
to setup Bot with BotFather and **Don't Forget to add your Bot to the Group**.

***

## Get the chat\_id of your Telegram Group

1. Open [https://web.telegram.org/](https://web.telegram.org/) and Follow the instructions
2. Click your Telegram Group and your Group chat\_id is on your url like `https://web.telegram.org/a/#-7222xxxxxx`

## Update Mage project settings

Once you’ve set up the **Bot** and get the `chat_id` of your group, you need to set the webhook like example below:

Here is an example webhook (yours may vary):

```
https://api.telegram.org/bot{your_token:from_botfather}/sendMessage?chat_id=-7222xxxxxx
```

Follow these steps to add that webhook URL to your project settings:

1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).

2. Open a pipeline and start editing it (e.g.
   [http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).

3. In your left sidebar in the file browser, scroll all the way down and click
   on a file named `metadata.yaml`.

4. In the `metadata.yaml` file, add a section with the following
   configuration or update the existing `notification_config` section:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       - trigger_failure
       - trigger_passed_sla
   ```

   If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.

   Options:

   * `trigger_failure`: alert when a run of a trigger fails
   * `trigger_success`: alert when a run of a trigger succeeds
   * `trigger_passed_sla`: alert when a SLA is missed.
     * In this context, SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
     * If the pipeline runs *longer* than the SLA, an alert will be sent.

5. In the `metadata.yaml` file, add the following values:

   ```yaml theme={"system"}
   notification_config:
     alert_on:
       ...
     telegram_config:
       webhook_url: ...
   ```

   Change the `webhook_url` value to be the webhook URL like mentioned above.

## Customize message templates

You can customize message templates in `notification_config`. Here is an example config:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  telegram_config:
    ...
  message_templates:
    failure:
      title: 'Failed to execute pipeline {pipeline_uuid} at {execution_time}'
      summary: >
        Failed to execute pipeline {pipeline_run_url}.
        Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
        Test custom message.
    success:
      title: ...
      summary: ...
    passed_sla:
      title: ...
      summary: ...
```

You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify the sentence on `summary` and the `title`.

To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:

1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
   * available only for the `failure` message template
8. `stacktrace`
   * available only for the `failure` message template

> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.

### 🚀 Pro Only: Interpolate Mage Variables in Message Templates

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=notification)

If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Telegram message templates using the familiar Jinja-like syntax:

```yaml theme={"system"}
notification_config:
  message_templates:
    failure:
      title: 'Failed to execute pipeline {pipeline_uuid} at {{ env_var('ENVIRONMENT') }} environment'
      summary: >
        Failed to execute pipeline {pipeline_run_url}.
        Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
        Test custom message.
```

In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:

1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)

This makes your Telegram notifications more dynamic and context-aware — ideal for multi-environment deployments and advanced workflows.

***

## What next?

Whenever a pipeline run is successfully completed or fails, a Telegram Group message will
appear in the group you configured the webhook URL for.

Here is an example of what that message could look like:

| |
| - |
| ![Telegram](https://mage-ai.github.io/assets/third-party/telegram-message-run-alert.jpg) |


# Alerting status updates in Zendesk
Source: https://docs.mage.ai/observability/alerting/alerting-zendesk

Create Zendesk tickets from Mage pipeline events to track incidents and resolution.

![Zendesk](https://cdn.simpleicons.org/zendesk)

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=notification)

## Overview of steps

1. Create an API token in Zendesk
2. Update Mage project settings with Zendesk config
3. Customize ticket templates with dynamic variables (Pro only)

***

## Configure alerts in the Mage Pro UI

After you create your Zendesk API token, you can configure alerts in the Mage Pro
UI instead of editing `metadata.yaml` directly:

1. For project defaults, go to **Settings > Workspace > Alerts**.
2. For one pipeline, open the pipeline and go to **Alerts**. Pipeline-level
   settings override the project-level alert config for that pipeline.
3. Choose the alert outcomes, enable Zendesk, enter the subdomain, email, and
   API token, and use **Send test notification** to create a test ticket before
   saving.

You can still reference secrets with values such as
`{{ env_var('MAGE_ZENDESK_API_TOKEN') }}`.

***

## Create an API token in Zendesk

To allow Mage to create tickets in Zendesk, you need to set up an API token:

1. Log in to Zendesk as an **admin**.
2. Go to **Admin Center** → **Apps and integrations** → **Zendesk API**.
3. Enable **Token Access**.
4. Click **Add API token**.
5. Enter a description (e.g., "Mage Alerts") and copy the generated token.
   > **Note:** This token will only be shown once. Store it securely.
6. Identify your Zendesk **subdomain** (e.g., for `https://acme.zendesk.com` the subdomain is `acme`).
7. Ensure the account you use (email) has **agent** or **admin** permissions.

***

## Update Mage project settings

Once you have your subdomain, email, and API token, add them to your project settings.

Here’s an example `zendesk_config` (add under `notification_config` in `metadata.yaml`):

```yaml theme={"system"}
notification_config:
  alert_on:
    - trigger_failure
    - trigger_passed_sla
  zendesk_config:
    subdomain: your-domain
    email: bot@example.com
    api_token: YOUR_ZENDESK_API_TOKEN
    group_id: "1234567890"          # optional: numeric group ID
    assignee_id: "9876543210"       # optional: numeric agent ID
    priority: high                  # optional: low | normal | high | urgent
    status: new                     # optional: new | open | pending | hold
    tags: mage,alert,pipeline       # optional: comma-separated tags
```

If you omit the `alert_on` section, it will default to `trigger_failure` and `trigger_passed_sla`.

Options for `alert_on`:

* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a SLA is missed.
  * SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
  * If the pipeline runs *longer* than the SLA, an alert will be sent.

***

## Customize ticket templates

You can customize ticket templates in `notification_config` just like Slack message templates:

```yaml theme={"system"}
notification_config:
  alert_on:
    ...
  zendesk_config:
    ...
  message_templates:
    failure:
      details: >
        Pipeline {pipeline_uuid} failed.
        See logs: {pipeline_run_url}.
    success:
      details: ...
    passed_sla:
      details: ...
```

You can customize templates for `success`, `failure`, and `passed_sla` scenarios.\
For each template:

* If you specify `summary`, the ticket subject will be your summary, and `details` will be used as the body.
* If you specify only `details`, it will be used as the ticket body directly.

**Supported variables:**

1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error` (failure only)
8. `stacktrace` (failure only)

***

### 🚀 Pro Only: Interpolate Mage Variables in Ticket Templates

In **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) in your Zendesk ticket templates using Jinja-like syntax:

```yaml theme={"system"}
notification_config:
  message_templates:
    failure:
      details: >
        Failed at {{ env_var('ENVIRONMENT') }} environment.
        Run details: {pipeline_run_url}.
```

Additional variables available in Mage Pro:

1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (value of the `ENV` environment variable)

***

## What next?

Whenever a pipeline run completes or fails, Mage will create a Zendesk ticket based on your configuration.\
You can route these tickets to specific groups or agents, tag them, and track resolution in Zendesk.

Example ticket subject/body:

| |
| - |
| **Subject:** Mage Notification: Failed to run Mage pipeline example\_pipeline |
| **Body:** Pipeline `example_pipeline` failed at 2025-08-13. View logs at: https\://mage\_url/pipelines/example\_pipeline/runs/123 |


# Data Explorer
Source: https://docs.mage.ai/observability/data-explorer

Explore warehouse, database, source, destination, and pipeline output data from Mage Pro.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-explorer)

Data Explorer is the fastest way to see what is inside the systems connected to
Mage Pro. It brings your existing `io_config.yaml` profiles, saved data
integration source and destination configs, and compatible pipeline execution
outputs into one workspace for browsing schemas, previewing data, running
guarded SQL, and handing useful context to AI Sidekick.

Use Data Explorer when you want to answer questions before building or changing
a pipeline:

* Which tables, streams, schemas, and columns are available in this warehouse?
* Does this source stream contain the records I expect?
* What does the latest order, event, customer, metric, or operational table look like?
* Can I validate a destination table after a sync without writing a throwaway block?
* What did this upstream pipeline block actually produce?
* Which pipeline run or block run produced the data shape I am debugging?
* What query should become the first version of a loader, chart, dashboard, or investigation?

Data Explorer is intentionally lightweight. It does not create pipelines, run
pipeline schedules, or persist transformation code. It gives teams a governed,
bounded, read-oriented window into connected data so they can move from
discovery to implementation with confidence.

![Data Explorer showing AI-assisted exploration of Titanic passenger pipeline output rows](https://mintcdn.com/mage/xlfFBQh2VT61fQmA/media/data-explorer/data-explorer-hero.png?fit=max\&auto=format\&n=xlfFBQh2VT61fQmA\&q=85\&s=da2ad9f48db2eed127cbd6cde1885409)

## Explain Data

If AI Sidekick is enabled, Data Explorer shows **Explain Data** in the page toolbar.
Clicking it starts a new Sidekick thread from the Data Explorer page.

Use this when you want Sidekick to help with questions such as:

* "Find the best connection for recent customer activity."
* "Help me write a query for failed payments in the last week."
* "Inspect available tables before I build a loader."
* "Compare the destination table columns with the source stream shape."
* "Help me decide whether the issue is in the source table or a pipeline block output."
* "Find the block run output that explains why this pipeline run produced fewer rows."

The Sidekick handoff is useful when you want to turn exploration into a focused
debugging, validation, or implementation plan.

## Open Data Explorer

In Mage Pro, open Data Explorer from either location:

1. Open **Library** in the left navigation, then select **Data**.
2. Open **Apps**, then select **Data Explorer**.

The page is available at:

```text theme={"system"}
/apps/data-explorer
```

Because Data Explorer is a Mage Pro feature, the page is available only in Mage
Pro workspaces.

## What you can do

Data Explorer combines connection inventory, schema metadata, previews, SQL, and
AI-assisted investigation in one page.

| Capability | What it gives you |
| - | - |
| Connection inventory | See supported IO profiles, saved integration sources, saved integration destinations, and compatible pipeline execution outputs in one searchable list. |
| Connection filters | Switch between **All**, **IO configs**, **Sources**, and **Destinations** to focus on the kind of system you are investigating. |
| Connection management | Select **Add / Manage** in the page toolbar to open the workspace settings page in a new tab, where you can create or update IO config profiles. |
| Schema browser | Browse catalogs or databases, schemas, tables, and columns for supported IO config SQL providers. For supported saved destinations, browse tables directly within the configured catalog and schema. |
| Table, stream, and output preview | Load a bounded preview from a table, stream, destination table, pipeline block output, or compatible output from a pipeline run or block run. |
| Guarded SQL | Run one read-oriented `SELECT` or `WITH` query against supported SQL connections. |
| Result table | Sort and filter the returned preview window without changing the source system. |
| Run investigation | Inspect the sample rows and output shape from compatible pipeline and block runs so you can debug a real execution without adding temporary code. |
| Expanded results | Hide the explorer panels after a query succeeds and give the result table more room. |
| Metadata refresh | Refresh connection inventory or schema metadata when upstream objects change. |
| Error and warning display | See unavailable connection reasons, metadata load failures, query errors, and preview warnings without exposing credentials. |
| AI handoff | Start Sidekick from Data Explorer when you want help deciding what to inspect next. |

## Supported connections

Data Explorer lists existing connections from:

* `io_config.yaml` SQL provider profiles.
* Saved data integration source configs.
* Saved data integration destination configs.
* Compatible pipeline execution outputs, including block outputs associated with
  pipeline runs and block runs.

SQL-capable connections can run queries. Connections with preview support can
preview a table, stream, destination table, or block output. Connections with
neither preview nor query support still appear in the inventory when Mage can
identify them, but Data Explorer marks them as unavailable and shows the reason.
This makes setup issues visible instead of silently hiding incomplete or
placeholder configs.

Supported `io_config.yaml` SQL providers include:

* BigQuery
* ClickHouse
* Databricks
* Druid
* DuckDB
* Microsoft Fabric
* MySQL
* OracleDB
* PostgreSQL
* Redshift
* Snowflake
* Spark
* SQLite
* Microsoft SQL Server
* Trino

SQLite `io_config.yaml` profiles are preview-only in Data Explorer V1. Local
DuckDB profiles can preview or run guarded read-only SQL when their database
file exists. MotherDuck `md:` profiles remain preview-only. Other supported
SQL-capable IO config profiles can preview or query data when their connection
settings are complete.

Saved source and destination configs are discovered from data integration
pipelines. When a source or destination has stream or table options in its
catalog, Data Explorer shows those options in a **Table or stream** selector and
uses the selected value to prepare preview and starter SQL state.

For saved BigQuery, MySQL, PostgreSQL, Redshift, Snowflake, and Microsoft SQL
Server destinations with a complete provider-specific scope, Data Explorer uses
that scope automatically. It loads the tables within the configured scope, so
you select a table instead of browsing or re-entering it. For BigQuery, the
configured project and dataset define the scope. For MySQL, the configured
database defines it.

Saved DuckDB destinations support table previews and read-only SQL exploration
when the destination config points to an existing local DuckDB database file.
Queries must pass Data Explorer's single `SELECT` or `WITH` statement validation.
Configs that point to a missing local file or a MotherDuck `md:` database remain
unavailable in Data Explorer.

For Delta Lake destinations on Amazon S3, Data Explorer can preview a bounded
window from the saved `delta_lake_s3` config when the config points to a readable
Delta table with either a full `s3://` table URI or a bucket, object key path,
and table name.

Pipeline execution outputs are available for compatible executable blocks and
run outputs. Data Explorer uses Mage's preview system to load a bounded output
window so you can inspect intermediate results from a pipeline run or block run
without adding temporary print statements, export blocks, or one-off validation
code.

For pipeline and block outputs, Data Explorer reads the real saved output for
the selected run. JSON-backed outputs use the saved `data.json` output. Data
Explorer does not use `sample_data`, `sample=True` output reads, or
`sample_data.json` as backing data for pipeline output previews.

## Permissions

Viewers can open Data Explorer and review the connection inventory. Editor access
is required to browse schemas, preview a table, or run SQL.

Saved source and destination configs follow the permissions of the pipeline that
owns the config:

* Users only see integration configs from pipelines they can view.
* Users must be able to edit the owning pipeline before running a preview or SQL
  query through that config.
* Pipeline and block run outputs follow the permissions of the pipeline that
  produced the output.

Connection metadata and error messages are redacted so credentials, tokens,
passwords, and URI credentials are not exposed in page errors.

## Browse schemas and columns

For supported `io_config.yaml` SQL profiles, Data Explorer can browse metadata
without loading a full warehouse inventory in the initial connection list. It
loads each level as you select it:

1. Catalogs or databases
2. Schemas
3. Tables
4. Columns

When metadata loads, Data Explorer selects the first available catalog or
database, schema, and table by default. Selecting a table fills the preview
target and updates the starter SQL query to `SELECT * FROM <selected_table>`.
Selecting a table also loads its columns and types.

The schema browser includes a search box that filters visible catalogs, schemas,
tables, and columns. It also includes a refresh control for the selected
connection so you can reload metadata after upstream tables or schemas change.
During dependent metadata loads, Data Explorer keeps stable panels visible where
possible so the page does not blank out while the next level loads.

For a supported saved data integration destination, its provider-specific scope
replaces the catalog and schema selection levels. Data Explorer opens a
**Tables** browser scoped to that destination, selects the first available table
by default, fills the preview target, and updates the starter SQL. Search and
refresh operate only on tables in that configured scope. Selecting a table also
loads its columns and types.

Schema browsing is currently supported for these IO config providers:

* DuckDB
* Microsoft Fabric
* MySQL
* PostgreSQL
* Redshift
* Snowflake
* SQLite
* Microsoft SQL Server

Other SQL-capable IO config profiles can still preview or query data when their
connection settings are complete, but you must type the table name or SQL query
manually.

## Preview a table, stream, or run output

1. Select a connection from **All**, **IO configs**, **Sources**, or
   **Destinations**.
2. For `io_config.yaml` connections with metadata support, use the schema
   browser to confirm or change the selected catalog or database, schema, and
   table.
3. For supported saved SQL destinations with a complete provider-specific scope,
   choose a table from the destination-scoped **Tables** browser. For other saved
   source or destination configs, choose a table or stream when options are
   available.
4. For compatible pipeline or block run outputs, choose the output you want to
   inspect.
5. If the connection does not provide table options, or the schema browser does
   not load a matching table, type the table or stream name.
6. Set a row limit.
7. Click **Preview table**.

Table and stream previews run a bounded `SELECT * FROM <table_or_stream>` query.
Pipeline and block run output previews load a bounded output window through
Mage's data preview system. The default row limit is `100`, and Mage clamps row
limits to protect the workspace from accidentally large previews.

Delta Lake S3 previews read the table through Mage's Delta Lake connector and
return only the requested preview window. They are useful when you want to
validate a Delta destination table before using it in Sidekick exploration,
Analytics chart creation, or dashboard refresh.

Preview is useful for quick checks:

```text theme={"system"}
analytics.orders
```

```text theme={"system"}
public.customer_events
```

```text theme={"system"}
stripe_balance_transactions
```

```text theme={"system"}
Pipeline run output / daily_orders / enrich_orders / run 12841
```

Use preview when you need a quick first look. Use SQL when you need joins,
filters, ordering, selected columns, grouping, or source-side predicates.

## Run a SQL query

1. Select a SQL-capable connection.
2. Enter one `SELECT` or `WITH` query.
3. Set a row limit.
4. Click **Run query**.

Data Explorer accepts one `SELECT` or `WITH` statement at a time. Mutating
statements such as `INSERT`, `UPDATE`, `DELETE`, and `DROP` are rejected.
Locking clauses such as `FOR UPDATE` or `FOR SHARE` are rejected. Multi-statement
SQL is rejected. Data Explorer also rejects `SELECT` or `WITH` queries that call
known unsafe functions with side effects, such as functions that update
sequences, terminate sessions, sleep, read server-local files, or execute
provider-specific system commands.

These safeguards are not the same as database-enforced read-only execution for
every provider. PostgreSQL and Redshift preview and query execution start a
database read-only transaction before running the query and roll it back
afterward. Metadata browsing uses server-generated `SELECT` queries with
provider timeout settings. Metadata discovery, preview, and query execution are
also bounded by server-side deadlines. For providers with native timeout
controls, Data Explorer passes those settings to the driver or query job. If a
provider call exceeds the relevant deadline, Data Explorer attempts
provider-native cancellation where available, closes or stops the bounded
execution operation, and returns a timeout error. Other supported providers rely
on statement validation, restricted-function checks, provider timeout or
lock-timeout settings where available, bounded server execution deadlines, and
row-limit enforcement.

For those providers, use credentials that already have the least privileges
needed for exploration.

```sql theme={"system"}
SELECT
  order_id,
  customer_id,
  status,
  total_amount,
  created_at
FROM analytics.orders
WHERE created_at >= CURRENT_DATE - INTERVAL '7 days'
ORDER BY created_at DESC
LIMIT 100;
```

Another common workflow is using Data Explorer to shape a loader query before
turning it into pipeline code:

```sql theme={"system"}
WITH recent_customers AS (
  SELECT
    customer_id,
    email,
    updated_at
  FROM warehouse.customers
  WHERE updated_at >= CURRENT_DATE - INTERVAL '30 days'
)
SELECT *
FROM recent_customers
ORDER BY updated_at DESC
LIMIT 250;
```

## Review results

After a preview or query succeeds, Data Explorer shows the returned rows in a
table. Use the column headers to sort the returned rows, or open a column filter
to narrow the rows shown in the current result.

These sort and filter controls apply only to the returned sample. To filter the
full source table, add the filter to the SQL query. Returned row counts describe
the loaded preview window, not the total number of rows in the source table. To
answer full-table cardinality questions, run an exact `COUNT(*)` query against
the selected table or filtered query.

When a tabular result is available, click **Expand results** to hide the
connection, schema, and query browser and give the results table more space.
Click **Back** to restore the explorer panels. Data Explorer remembers this
layout preference for future result tables.

If a preview returns warnings from the data preview service, Data Explorer shows
those warnings with the result so the loaded rows are not separated from
important context.

## Practical workflows

### Validate a new connection

1. Add or update an `io_config.yaml` profile.
2. Open Data Explorer.
3. Search for the provider or profile name.
4. Confirm the connection is available.
5. Browse schemas and tables, or type a known table name.
6. Preview a small row window.
7. Run a targeted `SELECT` query to confirm credentials and schema access.

### Inspect a data integration source

1. Open Data Explorer.
2. Select **Sources**.
3. Choose the saved source config from the connection list.
4. Select a stream when the source catalog provides stream options.
5. Preview the stream to confirm field names and representative records.
6. Use the table output to decide what should become a transformation or
   downstream validation.

### Check a destination after a sync

1. Open Data Explorer.
2. Select **Destinations**.
3. Choose the destination config owned by the integration pipeline.
4. Select or type the destination table.
5. Preview rows or run a narrow SQL query that checks the records affected by
   the latest sync.

For a saved DuckDB destination, use **Preview table** or run a read-only SQL query.
The destination config must point to an existing local DuckDB database file.

### Inspect a pipeline run or block run output

1. Open Data Explorer.
2. Search for the pipeline, block, or run identifier.
3. Choose the compatible pipeline output, block output, pipeline run output, or
   block run output from the available results.
4. Preview a bounded output window.
5. Compare the output shape and sample rows with the source, destination, or next
   block in the pipeline.
6. Use the result to decide whether the issue came from the upstream data,
   transformation code, runtime variables, or a downstream destination.

### Turn discovery into implementation

1. Use the schema browser to find the right table and columns.
2. Run a guarded SQL query with filters, ordering, and selected columns.
3. Copy the query into a data loader, chart workflow, or dashboard workflow.
4. Keep Data Explorer open as a quick verification tool while you iterate.

## Safety model

Data Explorer is built for exploration, not uncontrolled execution.

* It accepts only one `SELECT` or `WITH` statement per query request.
* It rejects mutating statements, locking clauses, multi-statement SQL, and
  known unsafe functions.
* It clamps preview and query limits on the server.
* It bounds metadata requests separately from preview and query requests.
* It loads compatible pipeline and block run outputs as bounded preview windows.
* It applies provider timeout settings where available.
* It attempts provider-native cancellation when supported.
* It redacts sensitive config values and error details before returning them.
* PostgreSQL and Redshift query execution use read-only transactions that roll
  back after the query.

Use least-privilege database credentials for exploration, especially with
providers where the database driver does not support the same read-only
transaction enforcement as PostgreSQL and Redshift.

## Troubleshooting

### No connections appear

Confirm that the project has a supported `io_config.yaml` SQL profile, saved
data integration source or destination config, or compatible outputs from
pipelines, pipeline runs, or block runs the user can view.

### A connection appears as unavailable

Open the connection and read the unavailable reason. Common causes include
placeholder values, incomplete `io_config.yaml` provider settings, missing
credentials, a saved source or destination provider that cannot be mapped to a
SQL-capable IO provider, a pipeline run whose real output is no longer
available, or a block type whose output cannot be previewed.

### Preview or query buttons are disabled

Confirm that the selected connection supports the action, a table or query is
entered, metadata is not still loading, and the user has Editor access. Viewers
can inspect the inventory, but they cannot browse schemas, preview data, or run
SQL.

### Schema browser does not appear

The schema browser appears for supported `io_config.yaml` connections with
metadata support. For other SQL-capable connections, type the table name or SQL
query manually.

### Results are sorted or filtered differently than the source table

Result table sort and filter controls operate on the returned preview window
only. Add `WHERE`, `ORDER BY`, grouping, or joins to the SQL query when the
filter must run against the full source table.


# Writing metrics to Datadog
Source: https://docs.mage.ai/observability/external/datadog



## Provide your Datadog credentials

You will need to add the following environment variables in order to use
Datadog in mage:

* DD\_API\_KEY: your datadog api key
* DD\_APP\_KEY: your datadog app key

## Using Datadog in Mage

You can now call Datadog helper functions in your Mage blocks to create or update metrics.
You can see the existing helper functions [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/services/datadog/__init__.py).

Example:

```python theme={"system"}
from mage_ai.services import datadog as dd

dd.increment('<your metric here>', tags=dict(pipeline='<pipeline_uuid>'))
```


# Metaplane in Mage
Source: https://docs.mage.ai/observability/external/metaplane

Run monitors in Metaplane.

![Metaplane](https://assets.website-files.com/6010c8c1dcb1d8def7311218/62fba80fddc9a0e808d2cff0_metaplane_full_filled_dark_purple.svg)

## Configuration

Here are the following keyword arguments that can be used for configuration:

| Keyword argument | Description | Sample value |
| - | - | - |
| `api_token` | Metaplane API token. | `abc123...` |

***

## Get all connections

```python theme={"system"}
from mage_ai.services.metaplane.metaplane import Metaplane


client = Metaplane(config=dict(api_token='abc123...'))
client.connections()
```

## Example code

Trigger sync with `source_id`.

```python theme={"system"}
from mage_ai.services.metaplane.metaplane import Metaplane


client = Metaplane(config=dict(api_token='abc123...'))
connection_id = client.connections()[0]['id']

# Run all monitors for a connection, and keep polling until they are all complete.
client.process(
    connection_id=connection_id,
    poll_interval=60,            # optional: check the status of a monitor every N seconds
    raise_on_failure=False,      # optional: if a monitor test fails, raise an error
)
```

Sample output:

```
Metaplane process started.
Running 1 monitor(s).
Running monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 ran with status 200.
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: False; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:35:52.125462+00:00).
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: False; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:35:52.125462+00:00).
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: False; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:35:52.125462+00:00).
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: True; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:39:50.791930+00:00).
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 passed: True.
Metaplane process completed.
```


# Monitoring with New Relic
Source: https://docs.mage.ai/observability/external/newrelic



## Enable New Relic

You will need to add `ENABLE_NEW_RELIC` environment variable as `true` in order to use
New Relic in Mage.

## New Relic Configuration File

Following [New Relic Python integration guide](https://docs.newrelic.com/install/python/?python-non-web=non-web-yes),
run command: `newrelic-admin generate-config YOUR_LICENSE_KEY newrelic.ini`
to generate the `newrelic.ini` configuration file and edit your `newrelic.ini` and insert values for the following:
...
license\_key = INSERT\_YOUR\_LICENSE\_KEY
...
app\_name = INSERT\_YOUR\_APP\_NAME
...

An example `example_newrelic.ini` file is attacehd in Mage repo under folder `integrations/newrelic`.

Add path of your newrelic configuration into environment variable `NEW_RELIC_CONFIG_PATH`.

## Using New Relic in Mage

Monitoring with New Relic will be enabled the next time you start the Mage server.

The following logs and errors will be reported to New Relic:

* Trigger runs.
* Mage scheduler.
* `mage run` CLI command.


# Monitoring with OpenTelemetry
Source: https://docs.mage.ai/observability/external/opentelemetry



## Overview

This guide details how to monitor your `mage_ai` application using OpenTelemetry. OpenTelemetry provides a unified approach to collecting telemetry data, such as traces and metrics, which are essential for understanding your application's performance and behavior.

## Prerequisites

Ensure the following prerequisites are in place before proceeding:

* An OpenTelemetry Collector is set up and configured to receive telemetry data.
* The `mage_ai` application is ready for instrumentation with OpenTelemetry.

## Configuring OpenTelemetry in `mage_ai`

### Setting Environment Variables

Configure the OpenTelemetry Exporters in your application by setting the following environment variables:

1. **OTLP HTTP Exporter**:
   Use `OTEL_EXPORTER_OTLP_HTTP_ENDPOINT` to specify the HTTP endpoint of your OpenTelemetry Collector.

   Example:

   ```bash theme={"system"}
   export OTEL_EXPORTER_OTLP_HTTP_ENDPOINT="http://192.168.1.56:3418/v1/traces"
   ```

2. **OTLP Exporter**:

The OTEL\_EXPORTER\_OTLP\_ENDPOINT environment variable sets a general collector endpoint, which can be used for both gRPC and HTTP connections.

Example:

```bash theme={"system"}
export OTEL_EXPORTER_OTLP_ENDPOINT="192.168.1.56:3417"
```

### Instrumentation in the Application

1. **SQLAlchemy Instrumentation**:
   The application uses the SQLAlchemyInstrumentor from OpenTelemetry to instrument database operations. This generates telemetry data like traces for database interactions.

2. **Tornado Instrumentation**:
   To monitor HTTP server operations, the application integrates OpenTelemetry's Tornado instrumentation. This collects valuable data related to HTTP requests and server performance.

## Telemetry Data Collection and Analysis

With the environment variables set and the application running, OpenTelemetry will start collecting telemetry data based on the defined instrumentation.

1. **Data Collection**:
   The application will send the collected telemetry data, including traces and metrics, to the specified OpenTelemetry Collector endpoint.

2. **Analyzing Telemetry Data**:
   For effective visualization and analysis, you can connect your OpenTelemetry Collector to backend tools that support OpenTelemetry data (like Grafana or Jaeger).


# Monitoring with Prometheus
Source: https://docs.mage.ai/observability/external/prometheus



## Enable Prometheus

You will need to add `ENABLE_PROMETHEUS` environment variable as `true` in order to
enable Prometheus style metrics on the \<BASE\_PATH>/metrics route.

## Prometheus metrics

When enabled, Mage will publish Metrics for the HTTP Server (Tornado) as well as for
the Python Runtime. The /metrics route will be updated on-demand to respond to scrape requests.

## Kubernetes resource usage in Mage Pro

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=kubernetes-resource-usage)

In Mage Pro, Kubernetes executor resource usage summaries can optionally read
historical CPU and memory samples from Prometheus. Set
`K8S_RESOURCE_USAGE_PROMETHEUS_URL` to your Prometheus base URL to make
Prometheus the preferred source for K8s resource usage metrics. If Prometheus is
not configured or does not return samples for the run pod, Mage falls back to the
Kubernetes Metrics API when it is available.

Mage queries Prometheus for the completed run pod by namespace and pod name. CPU
usage is read from `container_cpu_usage_seconds_total`, and memory usage is read
from `container_memory_working_set_bytes`. The stored samples power the
**Resources** tab on pipeline run pages and the K8s resource usage panel in
pipeline monitoring.

### Set up Prometheus for resource usage

For self-hosted or private-cloud Mage Pro clusters, install Prometheus in the
same Kubernetes cluster as the Mage deployment and expose it to the Mage server
pod through an internal Kubernetes service. Keep Prometheus as a `ClusterIP`
service unless your platform already provides an authenticated internal gateway.

Install Prometheus separately from the Mage Pro Helm chart:

```bash theme={"system"}
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace
```

After installation, confirm the Prometheus service name in your cluster:

```bash theme={"system"}
kubectl get svc -n monitoring | grep prometheus
```

The default service name from `kube-prometheus-stack` is commonly
`prometheus-kube-prometheus-prometheus`, but it can differ based on the Helm
release name and chart values.

### Verify required metrics

Mage Pro needs cAdvisor container CPU and memory metrics. Port-forward
Prometheus, then check that the metrics exist for pods in the namespace where
K8s executor jobs run:

```bash theme={"system"}
kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
```

```bash theme={"system"}
curl -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(container_cpu_usage_seconds_total{namespace="mage",pod!=""})'

curl -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(container_memory_working_set_bytes{namespace="mage",pod!=""})'
```

Replace `mage` with the namespace used by your K8s executor pods. If either
query returns `0`, confirm that Prometheus is scraping kubelet/cAdvisor metrics
for that namespace before enabling Mage Pro resource usage collection.

### Configure Mage Pro

Set the Prometheus URL on the Mage Pro web/server deployment. Use the in-cluster
service DNS name so traffic stays inside the cluster:

```yaml theme={"system"}
K8S_RESOURCE_USAGE_PROMETHEUS_URL: http://prometheus-kube-prometheus-prometheus.monitoring.svc:9090
K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS: "15"
K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS: "15"
K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS: "5"
K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS: "true"
```

If Prometheus is behind an authenticated proxy, configure either bearer token
auth:

```yaml theme={"system"}
K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN: "<token>"
```

or basic auth:

```yaml theme={"system"}
K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME: "<username>"
K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD: "<password>"
```

Bearer token auth takes precedence over basic auth when both are set.

### Managed cloud deployments

For managed cloud deployments, do not configure customer Mage Pro clusters to
query a shared Prometheus service directly. Route resource usage requests
through a tenant-aware gateway, such as `mage-pro-api`, so authentication,
authorization, and tenant label scoping happen before Prometheus is queried.

### Kubernetes Metrics API fallback

When `K8S_RESOURCE_USAGE_PROMETHEUS_URL` is unset, Mage Pro can use the
Kubernetes Metrics API as a best-effort fallback. The fallback samples active
K8s executor pods while the job is still running, then stores average and peak
CPU/RAM from the collected snapshots. It does not provide historical pod
metrics after Kubernetes removes the pod, so Prometheus is still recommended for
completed runs, fast runs, and long-term pipeline monitoring history.

Install Metrics Server if your cluster does not already expose
`metrics.k8s.io`:

```bash theme={"system"}
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm repo update

helm upgrade --install metrics-server metrics-server/metrics-server \
  --namespace kube-system
```

Verify the APIService is available:

```bash theme={"system"}
kubectl get apiservice v1beta1.metrics.k8s.io
```

Verify Metrics Server returns pod metrics in the namespace where K8s executor
jobs run:

```bash theme={"system"}
kubectl top pods -n <job-namespace>
```

The Mage service account that runs the web/server process must be able to read
pod metrics:

```bash theme={"system"}
kubectl auth can-i get pods.metrics.k8s.io \
  --as system:serviceaccount:<mage-namespace>:<mage-service-account>
```

If the command returns `no`, grant read access to `pods.metrics.k8s.io` in the
namespace where job pods run:

```yaml theme={"system"}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: mage-k8s-metrics-reader
  namespace: <job-namespace>
rules:
  - apiGroups:
      - metrics.k8s.io
    resources:
      - pods
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mage-k8s-metrics-reader
  namespace: <job-namespace>
subjects:
  - kind: ServiceAccount
    name: <mage-service-account>
    namespace: <mage-namespace>
roleRef:
  kind: Role
  name: mage-k8s-metrics-reader
  apiGroup: rbac.authorization.k8s.io
```

If the UI shows the Metrics API fallback as unavailable, check that the run used
the Kubernetes executor, that Mage identified the job pod, that the pod stayed
running long enough to be sampled, and that `kubectl top pods` works for active
pods in the job namespace.

### Configuration reference

| Variable | Description | Default |
| - | - | - |
| `K8S_RESOURCE_USAGE_PROMETHEUS_URL` | Prometheus base URL used for Mage Pro K8s resource usage summaries. Leave unset to use only the Kubernetes Metrics API fallback. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN` | Bearer token for Prometheus requests. Takes precedence over basic auth when set. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME` | Username for Prometheus basic authentication. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD` | Password for Prometheus basic authentication. | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS` | Request timeout for each Prometheus query. | `5` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` | Query range step used when sampling each completed run pod. | `15` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS` | Delay before retrying Prometheus once for completed runs that initially return too few samples. | `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS` | Set to `0`, `false`, `no`, or `off` to skip TLS certificate verification for Prometheus requests. | `true` |


# Monitoring with Sentry
Source: https://docs.mage.ai/observability/external/sentry



## Provide your Sentry credentials

You will need to add the following environment variables in order to use
Sentry in Mage:

* `SENTRY_DSN`: sentry dsn url
* `SENTRY_TRACES_SAMPLE_RATE`: (optional) value between 0 and 1
* `SENTRY_SERVER_NAME`: (optional) Name of the Mage server

## Using Sentry in Mage

Once you set the `SENTRY_DSN` environment variable, the Sentry integration will be
enabled the next time you start the Mage server.

The following errors will be reported to Sentry:

* Exceptions from trigger runs.
* Exceptions from the Mage scheduler.
* Exceptions from runs started by the `mage run` CLI command.


# OpenSearch log search
Source: https://docs.mage.ai/observability/log-search-opensearch



Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=log-search-opensearch)

Use OpenSearch-backed log search when you want Mage to search pipeline, block,
trigger, and run logs across log files instead of only reading the latest files
from disk.

Search includes the standard log message and block identifier. For grouped
Python and dbt exception entries, Mage Pro also searches the redacted structured
exception summary, exception type and message, stack trace text, dbt command,
and dbt model or node context when those fields are present. The Logs page keeps
those grouped exception entries together when you filter, open details, or
download matching logs.

For Mage Pro SaaS and Hybrid Cloud deployments, OpenSearch log search is enabled
automatically. No Helm chart changes or Kubernetes objects are required.

For other deployment options, including private cloud and on-prem deployments,
use this guide to update Helm chart values and create any required Kubernetes
Secrets for secured OpenSearch. In Kubernetes deployments, the Mage Helm chart
can wire the Mage containers, Fluent Bit sidecar, OpenSearch index setup job,
and shared log volume from the `logSearch.*` values.

## Quick start

For a private cloud or on-prem quick start, enable the bundled OpenSearch
subchart and log search in the same Helm upgrade. This is the shortest path
because the chart can create the OpenSearch resources, Fluent Bit config, and
index setup job for you. Leave `logSearch.opensearch.host` empty; the chart
derives the in-cluster OpenSearch service name.

### Requirements

* A Mage deployment managed by the
  [Mage Helm chart](https://github.com/mage-ai/helm-charts/tree/master/charts/mageai).
* A Kubernetes Secret with a strong `OPENSEARCH_INITIAL_ADMIN_PASSWORD` value
  for bundled OpenSearch.
* A `ReadWriteOnce` storage class for the bundled OpenSearch data volume when
  your cluster does not have a suitable default StorageClass. For example, an
  EKS cluster that uses EFS for shared Mage project files may still need an
  EBS-backed class such as `gp2` or `gp3` for OpenSearch.
* A Kubernetes PVC that contains the Mage project files and log files. The
  quick start reuses this PVC through `logSearch.persistence.existingClaim`.

If the OpenSearch PVC stays pending, the setup job cannot connect to OpenSearch
and Helm may wait on the hook.

### Minimum values

Create a Secret for the bundled OpenSearch admin bootstrap password.

```bash theme={"system"}
kubectl -n mage create secret generic opensearch-initial-admin \
  --from-literal=OPENSEARCH_INITIAL_ADMIN_PASSWORD=<strong-password>
```

Then reference the Secret from the OpenSearch subchart values.

```yaml theme={"system"}
opensearch:
  enabled: true
  extraEnvs:
    - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
      valueFrom:
        secretKeyRef:
          name: opensearch-initial-admin
          key: OPENSEARCH_INITIAL_ADMIN_PASSWORD

logSearch:
  enabled: true
  persistence:
    existingClaim: <mage-project-pvc>
  setupJob:
    enabled: true
```

`logSearch.opensearch.host`, `logSearch.opensearch.port`, and
`logSearch.opensearch.index` can be omitted in bundled mode. The chart derives
the bundled OpenSearch host and defaults to port `9200` and index `mage_logs`.
Add `logSearch.persistence.subPath` only when your project/log PVC requires it.
`logSearch.setupJob.enabled` defaults to `false`, so keep it set to `true` when
you want Helm to create the `mage_logs` index during the upgrade.
Do not commit the real `OPENSEARCH_INITIAL_ADMIN_PASSWORD` value into
`values.yaml`.

If your cluster does not have a suitable default StorageClass for OpenSearch,
add one explicitly:

```yaml theme={"system"}
opensearch:
  persistence:
    storageClass: <rwo-storage-class>
```

### Render and upgrade

Render the exact chart source and values you plan to deploy.

```bash theme={"system"}
helm template mageai <path-to-helm-charts>/charts/mageai \
  --namespace mage \
  --values values-log-search.yaml
```

Confirm the rendered manifest includes OpenSearch resources,
`USE_OPENSEARCH_FOR_LOGS=true`, the Fluent Bit sidecar, the chart-rendered
Fluent Bit ConfigMap, and the chart-rendered index setup Job. Then upgrade the
release.

```bash theme={"system"}
helm dependency build <path-to-helm-charts>/charts/mageai

helm upgrade --install mageai <path-to-helm-charts>/charts/mageai \
  --namespace mage --create-namespace \
  --values values-log-search.yaml \
  --reuse-values
```

The setup job is idempotent. After it creates the index successfully, you can
set `logSearch.setupJob.enabled=false` in later upgrades if you do not want Helm
to run it again.

## Customize the quick start

Use the quick start as the base, then customize only the pieces your deployment
needs.

### External OpenSearch

To use an external or managed OpenSearch cluster, leave `opensearch.enabled`
unset or `false`, and set `logSearch.opensearch.host`.

```yaml theme={"system"}
logSearch:
  enabled: true
  opensearch:
    host: opensearch-cluster-master.mage-search.svc.cluster.local
    port: "9200"
    index: mage_logs
  persistence:
    existingClaim: <mage-project-pvc>
    subPath: <optional-project-subpath>
  setupJob:
    enabled: true
```

When `opensearch.enabled=false`, `logSearch.opensearch.host` is required. The
chart fails rendering if log search is enabled for an external OpenSearch
cluster without a host.

### Auth and TLS

For OpenSearch auth, create a Secret with `OPENSEARCH_USERNAME` and
`OPENSEARCH_PASSWORD`, then reference it from `logSearch.opensearch.auth`.

```bash theme={"system"}
kubectl -n mage create secret generic opensearch-auth \
  --from-literal=OPENSEARCH_USERNAME=<username> \
  --from-literal=OPENSEARCH_PASSWORD=<password>
```

```yaml theme={"system"}
logSearch:
  opensearch:
    auth:
      enabled: true
      existingSecret: opensearch-auth
```

For TLS, create a Secret with `ca.crt`, then reference it from
`logSearch.opensearch.tls`.

```yaml theme={"system"}
logSearch:
  opensearch:
    tls:
      enabled: true
      existingSecret: opensearch-tls-ca
      verify: true
```

When TLS is enabled, the chart sets `OPENSEARCH_HTTP_SCHEME=https` for Mage,
uses HTTPS in the index setup hook Job, and mounts the CA certificate at
`/etc/ssl/opensearch/ca.crt`. Fluent Bit receives `OPENSEARCH_TLS=On`.

Securing a bundled OpenSearch cluster also requires OpenSearch security
configuration, not only Mage-side auth variables. Make sure OpenSearch has valid
node/admin certificates, `internal_users.yml`, and role mappings, then run the
OpenSearch security bootstrap for the cluster if your OpenSearch distribution
requires it. Avoid mounting a custom `roles.yml` unless you intentionally need
to replace roles; overriding built-in static roles can prevent security
initialization. If Helm keeps reusing an earlier failed nested security value,
rerender with your intended values and consider an upgrade with `--reset-values`
using a complete values file.

### Project mounts and generated PVCs

If an existing Mage release already has important `extraVolumes` or
`extraVolumeMounts`, check the rendered Deployment carefully. The chart
preserves existing mounts and avoids injecting a duplicate `/home/src` project
mount when `extraVolumeMounts` already contains the log-search mount path. If
your release already mounts the project path, set
`logSearch.persistence.mountInMageContainer=false` and let Fluent Bit reuse that
mount read-only.

For generated log-search PVCs, use `logSearch.persistence.accessModes` and
`storageClassName` that match your topology. When `standaloneScheduler=true` or
`replicaCount > 1`, generated log-search PVCs must include `ReadWriteMany`; use
an existing RWX claim or set `logSearch.persistence.accessModes` accordingly.

### Custom Fluent Bit config

The default path uses a chart-rendered Fluent Bit ConfigMap. Use
`logSearch.fluentBit.existingConfigMap` or
`logSearch.fluentBit.existingParsersConfigMap` only when you need to provide
custom Fluent Bit ConfigMaps yourself.

If you override `logSearch.fluentBit.config`, keep the OpenSearch output
environment variables wired to the chart values:

```text theme={"system"}
Host               ${OPENSEARCH_HOST}
Port               ${OPENSEARCH_PORT}
Index              ${OPENSEARCH_LOG_INDEX}
tls                ${OPENSEARCH_TLS}
tls.verify         ${OPENSEARCH_VERIFY_CERTS}
```

## Verify the deployment

Check that the Mage pod has both the Mage container and the Fluent Bit sidecar.

```bash theme={"system"}
kubectl -n mage get pods -l app.kubernetes.io/name=mageai
```

Confirm Fluent Bit is sending records.

```bash theme={"system"}
kubectl -n mage logs -l app.kubernetes.io/name=mageai -c fluent-bit --tail=50
```

Confirm the OpenSearch index exists and has documents.

```bash theme={"system"}
kubectl -n mage-search exec -it <opensearch-pod-name> -- \
  curl -s localhost:9200/_cat/indices/mage_logs?v
```

If OpenSearch uses auth or TLS, include the matching `curl` flags for your
cluster, for example `-u '<username>:<password>'`, `https://localhost:9200`, and
`-k` or `--cacert`.

Finally, open a pipeline's **Logs** page in Mage and enter text that appears in a
recent pipeline or block run. Typing filters the displayed logs; click **Search**
or press **Enter** to search all matching logs across the full time range.

For a **Custom UTC range**, choose the start and end dates and times, then click
**Apply Range**. The button appears for unapplied changes and disappears once
the selected range is applied. Invalid or incomplete ranges cannot be applied.

If logs have already loaded and a later refresh fails, Mage keeps the most
recent loaded results visible and shows **Retry refresh**. A failure before any
results load instead shows a blocking **Retry** action.

When AI Sidekick is enabled, an issue card can appear below the log count.
Click the affected block name to open its editor. Users with edit access can
click **Resolve ✦** to open AI Sidekick with the relevant log context; viewers
see the issue context without the resolve action.

## Filter log sources

On a pipeline's **Logs** page in Mage Pro, use the **Log sources** filters to
narrow results by where each log entry came from.

* Available source rows are **User code**, **Mage scheduler**, **Executor**,
  **Connector/library**, **Kubernetes**, and **System**.
* All source rows are checked by default; unchecking a row hides that source
  without changing the original INFO, WARNING, or ERROR level.
* **Infrastructure logs** controls Mage scheduler, executor, Kubernetes, and
  system together, and shows a partial state when only some child rows are
  checked.
* **Reset filters** is disabled when no filters are applied. It removes active
  source query state and returns to the complete authorized log view with all
  selectable source rows checked.
* Downloads include active filter context so filtered exports are not mistaken
  for complete unfiltered log files.

Kubernetes executor runtime and job-manager status entries are tagged as
**Kubernetes**. Legacy OpenSearch documents are classified from conservative
structured fields when possible; entries that cannot be classified safely remain
unknown-source, stay visible in unfiltered results, and do not appear as a
separate filter-panel row. If Mage falls back to loaded disk rows while source
filters are active, the API response metadata and export header disclose that
source filters are limited to those fallback rows.

## Troubleshooting

If searches return no results, check these items first:

* `USE_OPENSEARCH_FOR_LOGS` is set to `true` in the Mage web and scheduler pods.
* `OPENSEARCH_HOST`, `OPENSEARCH_PORT`, and `OPENSEARCH_LOG_INDEX` match the
  OpenSearch service and index.
* The setup job completed, or the `mage_logs` index already exists.
* The Fluent Bit sidecar can read the mounted project/log path.
* The Fluent Bit sidecar logs show successful flushes instead of connection,
  parser, or path errors.
* Auth and TLS settings are consistent across Mage, Fluent Bit, the setup job,
  and the OpenSearch cluster.
* The `logSearch.persistence.existingClaim` and `subPath` point at the same
  project files that Mage writes logs into.
* Any `source_category` or `exclude_source_category` values in the URL use the
  supported source identifiers. Unsupported values return no OpenSearch-backed
  rows until the invalid filter is removed.
* If `opensearch.enabled=false`, `logSearch.opensearch.host` is set. The chart
  fails rendering when log search is enabled for an external OpenSearch cluster
  without a host.

When troubleshooting Helm rendering, compare your values against the examples in
the Helm chart repository under `examples/aws-eks/values-log-search.yaml` and
`examples/aws-eks/log-search-deployment.md`.

## Reference

### What the chart creates

The Helm chart owns the non-secret log-search resources. In the default path,
you do not need to create Fluent Bit ConfigMaps or an `opensearch_setup.py`
ConfigMap before running `helm upgrade`.

When `logSearch.enabled=true`, the chart can:

* set `USE_OPENSEARCH_FOR_LOGS=true` in the Mage web and scheduler containers;
* set `OPENSEARCH_HOST`, `OPENSEARCH_PORT`, and `OPENSEARCH_LOG_INDEX`;
* optionally set OpenSearch auth and TLS environment variables from Kubernetes
  Secrets;
* mount the project/log PVC into the Mage container and the Fluent Bit sidecar;
* render the `mageai-fluent-bit` ConfigMap from `logSearch.fluentBit.config`
  and `logSearch.fluentBit.parsers`, unless you provide existing ConfigMaps;
* run Fluent Bit to tail Mage log files and ship them to OpenSearch;
* render the `mageai-log-search-index` ConfigMap from
  `logSearch.setupJob.mapping`;
* create a log-search PVC when `logSearch.persistence.enabled=true` and no
  `existingClaim` is set;
* create a Helm hook Job that applies `logSearch.setupJob.mapping` to create
  the OpenSearch log index.

The default index name is `mage_logs`.

If you use a packaged chart from a Helm repository, render that packaged chart
too. If the rendered manifest does not include the `logSearch` environment
variables, OpenSearch subchart resources, Fluent Bit sidecar, chart-rendered
Fluent Bit ConfigMap, and chart-rendered index setup Job, use a newer chart
package or deploy from a checked-out chart source that includes the `logSearch`
templates.

### Helm values

| Helm value | Runtime effect |
| - | - |
| `logSearch.enabled` | Enables OpenSearch-backed log search and sets `USE_OPENSEARCH_FOR_LOGS=true`. |
| `logSearch.opensearch.host` | Sets `OPENSEARCH_HOST`. Leave empty only when `opensearch.enabled=true`. |
| `logSearch.opensearch.port` | Sets `OPENSEARCH_PORT`. |
| `logSearch.opensearch.index` | Sets `OPENSEARCH_LOG_INDEX`; defaults to `mage_logs`. |
| `logSearch.opensearch.auth.existingSecret` | Reads `OPENSEARCH_USERNAME` and `OPENSEARCH_PASSWORD` from a Kubernetes Secret. |
| `logSearch.opensearch.tls.enabled` | Uses HTTPS for Mage and the setup job, and TLS for Fluent Bit. |
| `logSearch.opensearch.tls.existingSecret` | Mounts `ca.crt` at `/etc/ssl/opensearch/ca.crt`. |
| `logSearch.persistence.enabled` | Creates a log-search PVC when `existingClaim` is empty. |
| `logSearch.persistence.existingClaim` | Reuses a shared Mage project/log PVC for Mage and Fluent Bit. |
| `logSearch.persistence.mountInMageContainer` | Controls whether the chart mounts the log-search PVC into the Mage container. |
| `logSearch.fluentBit.enabled` | Adds the Fluent Bit sidecar that ships log files to OpenSearch. |
| `logSearch.fluentBit.config` | Renders the chart-managed Fluent Bit config. |
| `logSearch.fluentBit.existingConfigMap` | Uses an existing Fluent Bit config ConfigMap instead of rendering one. |
| `logSearch.setupJob.enabled` | Runs the index creation job after Helm install or upgrade. |
| `logSearch.setupJob.mapping` | Renders the index mapping used by the setup job. |


# Logging
Source: https://docs.mage.ai/observability/logging



## Server Logging

The server logger can be configured to adjust the amount of detail that is included in the server logs. This can be helpful for debugging or troubleshooting server issues. You can change the verbosity level of the logger by setting the `SERVER_VERBOSITY` environment variable.

The `SERVER_VERBOSITY` variable accepts the following values: `DEBUG`, `INFO`, `WARNING`, `ERROR` and `CRITICAL`.

## Server logging format

You can customize the format by setting the `SERVER_LOGGING_FORMAT` environment variable. By default it is set as
`SERVER_LOGGING_FORMAT=plaintext`. If it is set to unknown format it will use default value instead.

### Plaintext format

The plaintext logging format will be used by default. Or it can be set explicitly by setting the
`SERVER_LOGGING_FORMAT=plaintext` environment variable.

By default, the server will log messages in the following format:

```
%(levelname)s:%(name)s:%(message)s
```

It can be customized by setting the `SERVER_LOGGING_TEMPLATE` environment variable.
Example:

```
SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'
```

> [!WARNING]
>
> **Important**
>
> Format string with special characters like `\t`, `\n`, etc. can't be passed through `.env` file as they are getting
> escaped.
>
> To pass such string you can
>
> 1. Use `export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'` syntax for local
>    python deployment or docker compose with exported env variables
> 2. Use combination of `.env` file and `-e` flag for plain docker deployment
>    `docker run -d --env-file .env -e SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t\t[%(name)25.25s]\t%(levelname)5s: %(message)s' mageai/mageai:latest`

### JSON format

To use JSON logging format, set `SERVER_LOGGING_FORMAT=json`.

JSON formatted logs will have the following fields:

```json theme={"system"}
{
  "timestamp": "...",
  "level": "...",
  "message": "...",
  "logger": "...",
  "module": "...",
  "function": "...",
  "line_number": "..."
}
```

More logging formats/customization to come...

## Pipelines logging

By default, logs for each pipeline run will be stored in the
`<path_to_project>/pipelines/<pipeline_name>/.logs` folder.

Learn more about logs [here](/design/core-abstractions#log).

### Set logging level

The default logging level is `INFO`. To customize the logging level for logs of block runs and pipeline runs,
you can set the `level` of `logging_config` in your project's `metadata.yaml` file.

Example logging config to only log errors:

```yaml theme={"system"}
logging_config:
  level: ERROR
```

### Set log retention period

To delete old logs in your persistent volume, you can specify the `retention_period` in the `logging_config`. Valid period units
are 'h', 'd', and 'w'.

Example logging config:

```yaml theme={"system"}
logging_config:
  retention_period: '15d'
```

Then you can run command `mage clean-old-logs [project_path]` to clean up old logs

### Block logging

A logger is available in the block's context. You can use it to log messages to the block's log file. The logger is passed into
each block as a keyword argument. You can retrieve the logger by calling `kwargs.get('logger')` within the block.

In Mage Pro, Python and dbt block exceptions are grouped into one structured log
entry per exception. dbt exception entries preserve the multiline traceback,
exception type and message, run and block identity, dbt command and model or
node context when available, redaction status, and truncation metadata when a
trace is too large to store in full. Secrets in exception text are redacted
before the log entry is written, so the grouped entry is safe to view, search,
and download from the Logs page.

Example:

```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
    kwarg_logger = kwargs.get('logger')

    kwarg_logger.info('Test logger info')
    kwarg_logger.warning('Test logger warning')
    kwarg_logger.error('Test logger error')

    ...
```

## Logging to external destination

### S3

To store logs in S3, you need to set the `logging` config in your project's `metadata.yaml` file.

Example S3 logging config:

```yaml theme={"system"}
logging_config:
  type: s3
  level: INFO
  destination_config:
    bucket: <bucket name>
    prefix: <prefix path>
    aws_access_key_id: <(optional) AWS access key ID>
    aws_secret_access_key: <(optional) AWS secret access key>
    endpoint_url: <(optional) custom endpoint url>
```

To authenticate with S3, the credentials need to be configured in one of the following ways:

* Configure credentials in the `logging_config`.
* Configure credentials in environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
* Authenticate with IAM role. Grant S3 access to the IAM role of the cloud instance.

The `endpoint_url` can be configured to connect to S3 compatible services (e.g. MinIO).

### Google Cloud Storage

To store logs in GCS, you need to set the `logging` config in your project's `metadata.yaml` file.

Example GCS logging config:

```yaml theme={"system"}
logging_config:
  type: gcs
  level: INFO
  destination_config:
    path_to_credentials: <(optional) path to gcp credentials json file>
    bucket: <bucket name>
    prefix: <prefix path>
```

*More destinations coming...*

***

## Edit pipeline logging

When you’re editing a pipeline (e.g. `/pipelines/[uuid]/edit`), you can execute the code for an
individual block and see the output. Any `print` statements in the block of code is displayed
in the block’s output.

However, you can redirect those `print` statements to output to logs.

To toggle this feature, go to the pipeline settings page (e.g. `/pipelines/[uuid]/settings`)
and check the box labeled
**When running a block while editing a pipeline, output the block messages to the logs**.


# Monitoring
Source: https://docs.mage.ai/observability/monitoring

You can monitor many metrics for each of your pipelines and blocks.

In Mage OSS, use the **Overview** page to monitor aggregate pipeline run health
across your project. In Mage Pro, select **Monitoring** in the left navigation. The
monitoring panels show recent run status, failing runs, active runs, and trigger
activity for the selected time range.

In Mage Pro, the global Monitoring page supports rolling monitoring ranges for
**Past 24 hours**, **Past 7 days**, and **Past 30 days**, plus a **Custom** UTC
range. The selected range drives the overview metric tiles, run-health chart,
failed runs, active runs, longest active runs, and AI Sidekick prompts. The global
Monitoring page does not include an average-duration tile or completed-run duration
trends; those runtime metrics remain available on individual pipeline monitoring
pages. Custom ranges require the end time to be after the start time before
they can be applied, and the selected range is preserved through refreshes and
relevant navigation.

For a custom range in Mage Pro, edit the UTC date and time segments directly in
each `YYYY-MM-DD HH:mm` input, or choose them in the calendar and time picker.
The input keeps separators in place and shows placeholders for missing digits.
Complete valid values and picker selections stay in sync; incomplete entries
or an invalid range prevent the range from being applied.
Monitoring data availability can further limit the dates you can select.

In Mage Pro, the **Longest active pipeline runs** chart at the bottom of the
Monitoring page ranks non-streaming pipeline runs by elapsed runtime, with the
longest run first. It shows five runs by default; use the **Show** selector to
display 5, 10, 25, or 50 runs. The ranked list scrolls within the panel when
needed. Each row shows the pipeline name, run ID, trigger name when available,
and elapsed duration. Streaming pipelines are excluded. Durations refresh
automatically, and selecting a pipeline name opens that run's details. If the
initial request fails, use **Retry** in the panel. If a later refresh fails, the
panel keeps the last loaded runs visible and shows a warning with **Retry**.

## Metrics

Here are the following metrics that are monitored for each pipeline and block:

* Pipeline runs completed successfully
* Pipeline runs failed
* Block runs completed successfully
* Block runs failed
* How long each block ran for
* CPU usage for Kubernetes block and pipeline runs in Mage Pro
* Memory usage for Kubernetes block and pipeline runs in Mage Pro
* *Data validation success for each block (coming soon)*
* *Data validation failure for each block (coming soon)*

In Mage Pro, the pipeline monitoring page includes runtime trends for performance
debugging. **Pipeline runtime over time** shows the average duration of successfully
completed pipeline runs, while **Block runtime by block** lets you select a block
and chart the average duration of its completed samples. The existing **Runtime
hot spots** list remains useful for finding blocks with the highest average and
maximum runtimes before selecting one for its time-series breakdown.

Runtime trends use the dashboard's selected UTC time range and trigger filter.
The **Past 24 hours** view groups samples by hour; longer ranges group them by
date. A chart needs completed samples in at least two time buckets, so a block or
pipeline with fewer populated buckets shows an empty state instead of a trend
line. If runtime data fails to load, use **Retry** in the affected chart.

For pipelines that run with the Kubernetes executor in Mage Pro, the pipeline
monitoring dashboard can show historical peak CPU and memory usage, sample
counts, and copied resource request and limit recommendations based on stored
run metrics. See [Monitoring with Prometheus](/observability/external/prometheus#kubernetes-resource-usage-in-mage-pro)
to set up Prometheus as the preferred resource usage source.

## Monitoring actions

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=monitoring-actions)

The **Monitoring** page in Mage Pro includes actions for the monitoring panels.
**Fix runs** and **Analyze runs** are available when AI Sidekick is enabled.

* **Fix runs** opens AI Sidekick with a prompt to investigate failing runs for
  the selected time range.
* **Analyze runs** opens AI Sidekick with a prompt to inspect active runs for
  the selected time range.
* **View triggers** opens the trigger list, preserving the selected compute
  cluster filter when one is active.

## Streaming health

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=streaming-health)

Streaming pipelines in Mage Pro include a **Streaming health** view on the
pipeline monitoring page. Use this view to check whether a running or recent
stream is still accepting source records and whether Mage is writing those
records to each destination.
When no time range is specified in the URL, Streaming health opens to
**Last day**.

The Streaming health view shows:

* Rows consumed from the streaming source.
* Destination writes accepted by Mage, broken down by destination.
* Failed writes for destinations that reported write errors.
* Average processing latency when latency samples are available.
* The last metric update time and the time bucket size.

The throughput chart compares source consumption with destination writes over
time. Source consumption is shown separately from destination writes because
destination totals can be higher than consumed rows when a stream writes to more
than one destination.

For longer selected time ranges, Mage Pro rolls chart points into coarser time
buckets before returning Streaming health metadata so the chart stays responsive
while preserving the selected range totals.

Destination rows include each destination block, its exported count, failed
write count, and last successful write time.

On the pipeline monitoring page, streaming runs are grouped by trigger. This is
useful for long-running streaming triggers because Mage can create a replacement
pipeline run for the same trigger after an exception. Trigger-level health keeps
those automatic restarts visible as one operational stream.


# Data validation
Source: https://docs.mage.ai/observability/testing/data-validation

Every data loader and transformer block has data validation capabilities built-in.

You can define 1 or more test functions in a single block. Each test function
accepts a data object as an argument.

Within the body of the function, you can write any type of test you want to
validate the input data.

After the block’s main code is executed, the output data is passed into each
test function for validation. If any tests fail, then the block run will also
fail.

***

### Example

Here is an example of a transformer block with 2 tests:

```python theme={"system"}
from pandas import DataFrame

if 'transformer' not in globals():
    from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
    from mage_ai.data_preparation.decorators import test

COLUMNS_TO_USE = ['name']


@transformer
def transform_df(df: DataFrame, *args, **kwargs) -> DataFrame:
    return df.iloc[:1][COLUMNS_TO_USE]


@test
def test_number_of_rows(df) -> None:
    assert len(df.index) >= 2, 'The output has more than 1 row.'


@test
def test_columns(df) -> None:
    assert df.columns[0] != COLUMNS_TO_USE[0], 'The output columns don’t match.'
```

> [!NOTE]
>
> You can combine all your data validations into 1 test function or you can
> split them up into multiple test functions. The benefit of splitting them up
> is that they can run in parallel, speeding up the data validation.

***

### Log output

Each test run is recorded and can be viewed in the logs. Here is an example:

```
Start executing block.
--------------------------------------------------------------
2/2 tests passed.
Finish executing block.
```

***

# Data Quality with Great Expectations

![Great Expectations](https://images.ctfassets.net/ycwst8v1r2x5/22OiZltRmxCktJkYGUR2Su/c6da39d7884a9ec2d8b2d451cd81db6c/GX_Dark_BG_2x.png)

## Setup

1. Before adding expectations to your pipeline, please make sure you have at least 1
   data loader, transformer, or data exporter block.
   They must be Python blocks (SQL block support coming soon).

   If you don’t have any blocks, add a data loader block and paste the following code:

   ```python theme={"system"}
   import io
   import pandas as pd
   import requests


   @data_loader
   def load_data_from_api(*args, **kwargs):
       url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
       response = requests.get(url)
       return pd.read_csv(io.StringIO(response.text), sep=',')
   ```

   For more expectations, read Great Expectation’s [documentation](https://greatexpectations.io/expectations/).

2. [Add Great Expectations power up to your pipeline](/design/blocks/extension#how-to-add-extensions-to-your-pipeline).

3. In the list of available power ups, click the option for **Great Expectations**.
   ![Power ups](https://github.com/mage-ai/assets/blob/main/notebook/extensions/power-ups.jpg?raw=true)

***

## Adding expectations to your pipeline

1. Once you’re on the Great Expectations power up detail page, you can add extension blocks to
   the current pipeline by clicking the button labeled **\[+ Extension block]**.

2. In dropdown menu, click the template option labeled **Empty template**.

3. The **Block name** dialog may appear for the new extension block.
   If it appears, enter `test number of rows` and click **Save**.

4. Paste the following code in the extension block named `test number of rows`:

   ```python theme={"system"}
   @extension('great_expectations')
   def validate(validator, *args, **kwargs):
       validator.expect_table_row_count_to_be_between(
           min_value=1000,
           max_value=10000,
       )
   ```

   > [!NOTE]
   >
   > You can add expectations using code or from a JSON object.
   > See section [Defining expectations](#defining-expectations) below for more details.

5. In the extension block near the bottom, click the input field that says
   “Select blocks to run expectations on”.

6. Once you click that input field, a list of blocks from your pipeline will appear.

7. Check the checkbox on the right side of the dropdown to associate that block to this extension block.

8. Click the button labeled **Save selected blocks**.

9. After you save, a button that is labeled after the name of the block you just selected will appear.
   For example, if your block is named `load_api_data_demo`,
   then a button labeled **load\_api\_data\_demo** will appear.
   ![Great Expectations extension block](https://github.com/mage-ai/assets/blob/main/notebook/extensions/great-expectation-extension-block.jpeg?raw=true)

10. Click that button to run your extension block for the block `load_api_data_demo`.

11. The output should look something like this:
    ```
    Calculating Metrics:   0%|          | 0/1 [00:00<?, ?it/s]
    Calculating Metrics:   0%|          | 0/1 [00:00<?, ?it/s]
    Expectations from extension test_number_of_rows for block load_api_data_demo succeeded.
    {
      "results": [
        {
          "expectation_config": {
            "expectation_type": "expect_table_row_count_to_be_between",
            "kwargs": {
              "min_value": 1000,
              "max_value": 10000,
              "batch_id": "9946450cf9609e633658e9c4ee38efa5"
            },
            "meta": {}
          },
          "success": true,
          "meta": {},
          "exception_info": {
            "raised_exception": false,
            "exception_traceback": null,
            "exception_message": null
          },
          "result": {
            "observed_value": 10000
          }
        }
      ],
      "success": true,
      "evaluation_parameters": {},
      "meta": {
        "great_expectations_version": "0.15.50",
        "expectation_suite_name": "expectation_suite_for_block_load_api_data_demo",
        "run_id": {
          "run_name": null,
          "run_time": "2023-03-14T01:41:09.117251+00:00"
        },
        "batch_spec": {
          "data_asset_name": "data_asset_load_api_data_demo",
          "batch_data": "PandasDataFrame"
        },
        "batch_markers": {
          "ge_load_time": "20230314T014109.103956Z",
          "pandas_data_fingerprint": "76d98f6ff10d31abaae6e9a7c47cc7c1"
        },
        "active_batch_definition": {
          "datasource_name": "datasource_name_load_api_data_demo",
          "data_connector_name": "data_connector_name_load_api_data_demo",
          "data_asset_name": "data_asset_load_api_data_demo",
          "batch_identifiers": {
            "default_identifier_name": "default_identifier"
          }
        },
        "validation_time": "20230314T014109.117222Z",
        "checkpoint_name": null
      },
      "statistics": {
        "evaluated_expectations": 1,
        "successful_expectations": 1,
        "unsuccessful_expectations": 0,
        "success_percent": 100.0
      }
    }
    ```

### Other ways to run expectations

Whenever you run a block while editing your pipeline,
any associated Great Expectations extension blocks will also be ran.

If any expectations fail, the block output will display an error message with the failure results.

***

## Defining expectations

### Code

```python theme={"system"}
@extension('great_expectations')
def validate(validator, *args, **kwargs):
    validator.expect_table_row_count_to_be_between(
        min_value=1000,
        max_value=10000,
    )
```

### JSON object

```python theme={"system"}
expectations_json = [
  {
    "expectation_type": "expect_table_row_count_to_be_between",
    "kwargs": {
      "min_value": 1000,
      "max_value": 10000
    }
  },
  {
    "expectation_type": "expect_column_values_to_not_be_null",
    "kwargs": {
      "column": "user ID",
      "result_format": "BASIC"
    }
  }
]


@extension('great_expectations', expectations=expectations_json)
def validate(validator, *args, **kwargs):
    pass
```

### Code and JSON object

```python theme={"system"}
expectations_json = [
  {
    "expectation_type": "expect_column_values_to_not_be_null",
    "kwargs": {
      "column": "user ID",
      "result_format": "BASIC"
    }
  }
]


@extension('great_expectations', expectations=expectations_json)
def validate(validator, *args, **kwargs):
    validator.expect_table_row_count_to_be_between(
        min_value=1000,
        max_value=10000,
    )
```

***

## Running expectations end-to-end

When your pipeline runs, the expectations you defined in your extension blocks will be ran for
every associated data loader, transformer, or data exporter block in your pipeline.

### Success

If all expectations for a block pass, the success message in the pipeline run logs will look
something like this:

```
Expectations from extension test_number_of_rows for block load_api_data_demo succeeded.
```

### Failure

If any expectation fails, the block it was associated with will also fail.

You can check the pipeline logs for the block failure. There will be an entry containing the error
message with the failure results. It could look something like this:

```
Traceback (most recent call last):
  File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 648, in execute_sync
    output = self.execute_block(
  File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 885, in execute_block
    outputs = self._execute_block(
  File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 949, in _execute_block
    outputs = self.execute_block_function(
  File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 975, in execute_block_function
    output = block_function(*input_vars, **global_vars)
  File "/home/src/mage_ai/data_preparation/models/block/extension/block.py", line 46, in func
    raise Exception(
Exception: Expectations from extension test_number_of_rows for block load_api_data_demo failed:

{
  "exception_info": {
    "raised_exception": false,
    "exception_traceback": null,
    "exception_message": null
  },
  "meta": {},
  "success": false,
  "expectation_config": {
    "expectation_type": "expect_table_row_count_to_be_between",
    "kwargs": {
      "min_value": 1000,
      "max_value": 10000,
      "batch_id": "9946450cf9609e633658e9c4ee38efa5"
    },
    "meta": {}
  },
  "result": {
    "observed_value": 10
  }
}
```


# Unit tests
Source: https://docs.mage.ai/observability/testing/unit-tests

Write unit tests to test pipeline code.

## Create unit tests and run tests in CI/CD pipeline

You can create a `tests` folder in your project folder and put unit tests in this folder.

In your CI/CD pipeline, you can `cd` into the project's folder and run the following command
to run unit tests.

```bash theme={"system"}
python3 -m unittest discover -s tests --failfast
python3 -m pytest tests -x
```

## Example unit tests

End-to-end test for a pipeline:

```python theme={"system"}
import unittest

from mage_ai.data_preparation.models.pipeline import Pipeline


class ExamplePipelineTest(unittest.TestCase):
    def test_pipeline_execution(self):
        pipeline = Pipeline.get('example_pipeline')
        pipeline.execute_sync()
```

## Test environment

When running unit tests, the `env` variable's value in the method's `kwargs` is `test`.
You can leverage this variable to change the code logic or config used in the unit tests.

## Mocking IO libraries

When writing unit tests, you may need to mock the IO libraries to verify that the pipeline
is being executed properly without actually calling the database or warehouse directly.

If you are using the provided Mage IO libraries, you can mock the libraries in the following way:

```python theme={"system"}
import pandas as pd
import unittest
from unittest.mock import patch, MagicMock, Mock

from mage_ai.data_preparation.models.pipeline import Pipeline


class ExampleIOTest(unittest.TestCase):
    def setUp(self):
        self.pipeline = Pipeline.get('load_data_to_postgres')

    def tearDown(self):
        """
        Optionally clean up the output of the pipeline/block execution.
        Make sure to only do this if you are ok losing all variable data for
        the project.
        """
        # import shutil
        # from mage_ai.settings.repo import get_variables_dir
        # shutil.rmtree(get_variables_dir())

    def test_pipeline_execution(self):
        bq_mock = MagicMock()
        bq_mock.__enter__ = Mock(return_value=bq_mock)
        bq_mock.load.return_value = pd.DataFrame(
            data=dict(
                a=[1, 2, 3],
                b=[4, 5, 6],
                c=[7, 8, 9],
            )
        )

        postgres_mock = MagicMock()
        postgres_mock.__enter__ = Mock(return_value=postgres_mock)

        with patch(
            'mage_ai.io.postgres.Postgres.with_config', return_value=postgres_mock
        ), patch('mage_ai.io.bigquery.BigQuery.with_config', return_value=bq_mock):
            self.pipeline.execute_sync(global_vars=dict(num_rows=10))

            bq_mock.load.assert_called_once()
            bq_mock.load.assert_called_with('SELECT * FROM xyz')

            postgres_mock.export.assert_called_once()
            args, kwargs = postgres_mock.export.call_args
            df = args[0]
            schema_name = args[1]
            table_name = args[2]

            self.assertEqual(df.shape, (3, 3))
            self.assertEqual(schema_name, 'public')
            self.assertEqual(table_name, 'test_table_name')
            self.assertEqual(kwargs['index'], False)
            self.assertEqual(kwargs['if_exists'], 'replace')

    def test_block_execution(self):
        block1 = self.pipeline.get_block('primal_dawn')
        block2 = self.pipeline.get_block('little_glitter')

        bq_mock = MagicMock()
        bq_mock.load.return_value = pd.DataFrame(
            data=dict(
                a=[1, 2, 3],
                b=[4, 5, 6],
                c=[7, 8, 9],
            )
        )

        with patch('mage_ai.io.bigquery.BigQuery.with_config', return_value=bq_mock):
            block1.execute_sync()

            bq_mock.load.assert_called_once()
            bq_mock.load.assert_called_with('SELECT * FROM xyz')

        postgres_mock = MagicMock()
        postgres_mock.__enter__ = Mock(return_value=postgres_mock)

        with patch(
            'mage_ai.io.postgres.Postgres.with_config', return_value=postgres_mock
        ):
            block2.execute_sync()

            postgres_mock.export.assert_called_once()
            args, kwargs = postgres_mock.export.call_args
            df = args[0]
            schema_name = args[1]
            table_name = args[2]

            self.assertEqual(df.shape, (3, 3))
            self.assertEqual(schema_name, 'public')
            self.assertEqual(table_name, 'test_table_name')
            self.assertEqual(kwargs['index'], False)
            self.assertEqual(kwargs['if_exists'], 'replace')
```

In the above example, we are mocking the `BigQuery` IO class in the data loader, and the
`Postgres` IO class in the data exporter.


# Analytics
Source: https://docs.mage.ai/observability/visualizations/analytics

Create AI-powered dashboards from the data already flowing through Mage Pro.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=analytics)

Analytics turns the data already flowing through Mage into interactive dashboards for business
reporting, operational insight, and team decision-making. Use it to explain revenue trends,
customer growth, product adoption, campaign performance, data quality, model behavior, warehouse
costs, and any other metric your pipelines produce.

Instead of exporting data to another tool before anyone can understand it, teams can create and
refresh dashboards directly where the data is prepared. Analysts, data engineers, operators, and
business stakeholders can all work from the same trusted source.

Analytics is different from monitoring. Monitoring tells you whether your pipelines, schedules,
logs, alerts, and infrastructure are healthy. Analytics tells you what the data means and what your
team should do next.

![Mage Pro Analytics dashboard with revenue, engagement, channel, regional sales, customer segment, acquisition funnel, and product performance charts.](https://mintcdn.com/mage/xlfFBQh2VT61fQmA/media/analytics-dashboard.png?fit=max\&auto=format\&n=xlfFBQh2VT61fQmA\&q=85\&s=601a2cc037965e6274ffc3f86f9c46f4)

## Access Analytics

Select **Analytics** in the left navigation, or go directly to:

```text theme={"system"}
/apps/analytics/dashboards
```

This is the Analytics workspace. From there, you can create dashboards, open existing dashboards,
add charts, edit charts with AI, refresh data, and arrange the dashboard layout.

## What Analytics helps you do

Analytics is built for fast, practical answers:

| Goal | Example |
| - | - |
| Track business performance | Revenue, churn, expansion, renewals, account health, conversion, and retention. |
| Understand product usage | Weekly active users, feature adoption, activation, engagement, and funnel drop-off. |
| Operate data workflows | Failed records, validation errors, source freshness, SLA checks, and quality trends. |
| Review customer operations | Support volume, ticket aging, onboarding progress, and customer success risk. |
| Monitor models and experiments | Accuracy, drift, feature importance, confusion matrices, and experiment results. |
| Analyze warehouse or infrastructure spend | Cost by source, team, warehouse, workflow, or time period. |

Dashboards can include time series, comparisons, distributions, rankings, tables, heatmaps, funnels,
scorecards, and other visual summaries. Start with the business question you want answered, and
Mage helps turn the available data into a useful view.

## Create a dashboard

1. Open `/apps/analytics/dashboards`.
2. Click **New dashboard**.
3. Name the dashboard for the audience or decision it supports, such as `Executive revenue`,
   `Customer health`, `Product adoption`, or `Data quality`.
4. Open the dashboard.
5. Click **Add chart** to create the first visualization.

Keep dashboards focused. A good dashboard usually answers one type of question for one audience:
executive reporting, growth analysis, customer operations, data quality, model review, or cost
tracking.

## Add charts with AI

Click **Add chart**, then describe the outcome you want. AI Sidekick uses the dashboard and the
data you are working with to create the chart and save it to the dashboard.

Useful prompts describe the metric, segment, time range, and decision:

```text theme={"system"}
Show monthly recurring revenue by plan type and highlight net revenue retention.
```

```text theme={"system"}
Compare weekly active users by customer tier for the last 12 weeks.
```

```text theme={"system"}
Show failed records by data source and list the most common validation errors.
```

```text theme={"system"}
Create a model quality dashboard with accuracy, recall, precision, and drift by week.
```

You can also create a chart while reviewing pipeline results. Click the chart action from a
supported result or preview, describe what you want to see, and Sidekick keeps the chart connected
to the right data.

For an existing chart, use **Edit with AI** to ask for changes:

1. Change the metric, grouping, or time period.
2. Add a comparison line or segment split.
3. Rename the chart or clarify labels.
4. Turn a trend into a ranking, scorecard, table, or heatmap.
5. Simplify a noisy chart for an executive audience.

## Choose the right data

Analytics can visualize the data your Mage project already works with:

| Data source | Customer outcome |
| - | - |
| IO data sources | Build dashboards from connected databases, warehouses, object storage, APIs, and other sources your project can already read from. |
| Pipeline block outputs | Turn the results produced by loaders, transformers, exporters, and custom blocks into dashboards without moving data into another BI tool first. |
| Pipeline runs | Chart run volume, success rate, duration, freshness, or business metrics captured each time a pipeline runs. |
| Block runs | Compare block-level outcomes, failures, processing time, row counts, validation results, or model metrics across runs. |
| Data previews | Explore a dataset, choose the columns and shape that answer the question, then save a chart from the preview lineage instead of only the visible row window. |
| Warehouse-backed data | Summarize large tables into focused metrics before they appear on the dashboard. |
| Transformed datasets | Apply business logic, cleanup, or grouping before the chart is shown. |
| Structured workflow outputs | Visualize tables, records, metrics, matrices, rankings, and other useful shapes produced by your workflows. |

That means Analytics can cover both business data and workflow activity. For example, one
dashboard can show revenue by customer segment, failed records by source, pipeline run duration,
block run failures, and model quality by training run.

When you build charts from pipeline or block outputs across multiple runs, Sidekick checks the
output grain before choosing the chart. Run-summary outputs are useful for operational trends such
as run volume, checked time, row counts, failures, and block duration. Business-date charts, such as
deals by close date or invoices by paid date, need rows that represent the business records and a
matching business date field. If the available output only contains run summaries or nested saved
state, Sidekick should state that coverage before saving the chart.

For pipeline and block output datasets, Analytics chart creation uses the real saved output data
for the selected run or partitions. JSON-backed outputs use readable `data.json` backing data. Chart-backed
datasets do not use `sample_data`, `sample=True` output reads, or `sample_data.json` preview
artifacts to create or refresh charts.

For large datasets, ask for the summary you need rather than every raw row. For example, ask for
"weekly active users by plan for the last 12 weeks" or "top 20 accounts by expansion revenue" so
the dashboard stays fast and readable. When a chart starts from a data preview, Analytics uses the
preview source and query plan to build a chart-owned dataset snapshot; it does not use only the
rows currently visible in the preview table.

Analytics stores a bounded chart-ready snapshot for each saved chart. If Sidekick warns that a
time-series or ordered chart may have reached its dataset preview limit, narrow the request, group
the data into coarser buckets, or ask Sidekick to state the covered date or category range before
using the chart for decisions.

## Organize dashboards for the team

Each chart appears as a card on the dashboard. You can:

1. Rename charts so viewers immediately understand the metric.
2. Edit charts with AI as the question changes.
3. Duplicate a chart to test another view without losing the original.
4. Drag charts into a new order to create a clear story from top to bottom.
5. Resize charts so the most important metrics get more space.
6. Group related charts together so trends, drivers, and details are easy to scan.
7. Remove charts that are no longer useful.
8. Refresh charts when the underlying data has changed.

Use the top of the dashboard for the metrics that matter most. Put detailed breakdowns, tables,
and diagnostics after the summary so viewers can move from signal to detail.

Dashboard layouts are flexible. Drag a chart to rearrange the page, make headline metrics larger,
give detailed tables more room, or place related charts side by side. This makes it easy to turn a
collection of charts into a polished dashboard for a meeting, executive review, customer health
check, or recurring operational workflow.

## Refresh data

Analytics dashboards are meant to stay useful after the first version. When the source data
changes, refresh the chart or dashboard so viewers see the latest available results.

Charts with loaded data stay visible and dim while refreshing, so you can view the previous
results until the new data is ready. A moving indicator marks the refresh; with reduced motion
enabled, charts dim without animation.

Refresh is useful for recurring reporting:

1. Daily revenue and usage snapshots.
2. Weekly customer health reviews.
3. Data quality checks after each pipeline run.
4. Model evaluation after new training or scoring jobs.
5. Warehouse cost reviews at the end of a billing period.

If a refresh cannot complete, Mage keeps the previous dashboard view available so your team does
not lose the last known good result.

### Choose pipeline output partitions

For a chart backed by pipeline block outputs, open the chart's **Data source** action to choose
which partitions it reads when you refresh:

| Selection | Refresh behavior |
| - | - |
| Fixed partition | Re-read the saved run or execution partition. New pipeline runs do not change the selected source. |
| Latest partitions | Re-select the newest N distinct completed execution partitions. Set Partition count to `1` to follow the latest completed partition. |
| All partitions | Re-select all completed execution partitions within the safety limits below. |

Latest partitions and All partitions select partitions again on each explicit chart or dashboard
**Refresh**, while keeping the chart's saved query and transformations. Reloading the page alone
keeps the saved chart snapshot. Partitions are ordered by run completion time, newest first.
Retries for the same execution partition
count as one partition. These settings apply only to charts backed by pipeline block outputs;
other data sources keep their existing refresh behavior. Existing charts also keep their saved
source behavior until you change this setting.

Choose Latest partitions when each partition contains a new batch of records. If each run produces
a full snapshot, combining partitions can repeat records; use a count of `1` to chart the latest
snapshot.

Selection is limited to 1,000 partitions and 10,000 scanned runs. If selection exceeds a safety
limit, or a selected output is missing or unreadable, refresh fails explicitly instead of silently
showing an incomplete selection. Narrow the partition selection or restore the missing output,
then refresh again. All partitions does not remove the chart dataset's separate row limits.

You can also ask Sidekick to change a chart's partition selection:

```text theme={"system"}
Make this chart use the latest completed partition whenever I refresh it.
```

```text theme={"system"}
Use the last 7 completed partitions from this block for this chart.
```

The saved source field is `source_ref.partition_selection`. Its `mode` is `fixed`, `last_n`, or
`all`; `last_n` also requires a positive integer `count`, for example:

```json theme={"system"}
{
  "partition_selection": {
    "mode": "last_n",
    "count": 1
  }
}
```

## Recommended dashboard patterns

### Executive dashboard

Use this for a high-level business review. Start with scorecards and simple trends, then add a few
breakdowns that explain why the numbers changed.

Example charts:

1. Monthly recurring revenue.
2. New, expansion, contraction, and churned revenue.
3. Net revenue retention by segment.
4. Top growing accounts.
5. Accounts at risk.

### Product dashboard

Use this for adoption, activation, and engagement analysis.

Example charts:

1. Weekly active users by plan.
2. Activation funnel by signup cohort.
3. Feature adoption by account tier.
4. Retention by cohort.
5. Most common user paths or events.

### Data quality dashboard

Use this for operational trust in your pipelines and datasets.

Example charts:

1. Failed records by source.
2. Validation errors by type.
3. Freshness by table or pipeline.
4. Duplicate records over time.
5. Missing values by field.

### Model quality dashboard

Use this for ML and AI workflows where stakeholders need to see model behavior over time.

Example charts:

1. Accuracy, recall, and precision by evaluation date.
2. Prediction volume by class.
3. Drift by feature.
4. Confusion matrix summary.
5. Experiment comparison by metric.

## Best practices

1. Start with the decision the dashboard should support.
2. Use plain-language chart titles, such as `Revenue by plan` or `Failed records by source`.
3. Group charts by story: summary, drivers, segments, and diagnostics.
4. Keep raw-detail tables secondary unless the dashboard is for debugging or operations.
5. Use AI edits to iterate quickly, then keep the clearest version.
6. Create separate dashboards for separate audiences.
7. Refresh dashboards before recurring business reviews.

## Troubleshooting

| Symptom | What to check |
| - | - |
| The dashboard is empty | Create a dashboard, then add at least one chart from data available in Mage. |
| Sidekick cannot create a chart | Run the relevant pipeline or block so there is data to inspect, then try **Add chart** again. |
| A chart shows no data | Check whether the selected data is empty, filtered too narrowly, or missing the metric you asked for. |
| Refresh fails | Confirm the source data is still available and retry. The previous dashboard view remains available. |
| The dashboard feels too busy | Split it into separate dashboards by audience or decision, then keep only the most important charts on each one. |
| The layout feels crowded on a small screen | Charts stack on compact screens. Use a desktop browser for precise dashboard layout work. |


# Customizable dashboards
Source: https://docs.mage.ai/observability/visualizations/dashboards

There are 2 dashboards: a dashboard for all your pipelines and a dashboard for each pipeline. You can add charts of various types with different sources of data. Use these dashboards for observability or for analytics.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=monitoring-dashboards)

![Customizable dashboards](https://mage-ai.github.io/assets/visualizations/demo.gif)

## Pages with customizable dashboards

1. Monitoring page (`/overview?tab=Dashboard`)
2. Pipeline dashboard page (`/pipelines/[:id]/dashboard`)

***

## Adding charts

1. In the top right corner of the dashboard, click the button labeled
   **+ Add content**.
2. Click the dropdown option labeled **Create new chart**.

***

## Configuring chart

### Chart name

Human readable name for your chart.

### Chart type

Choose from a variety of visualizations.

#### Bar chart

![Bar chart](https://mage-ai.github.io/assets/visualizations/bar-chart.png)

#### Histogram

![Histogram](https://mage-ai.github.io/assets/visualizations/histogram.png)

#### Line chart

![chart](https://mage-ai.github.io/assets/visualizations/line-chart.png)

#### Pie chart

![chart](https://mage-ai.github.io/assets/visualizations/pie-chart.png)

#### Table

![Table](https://mage-ai.github.io/assets/visualizations/table.png)

#### Time series bar chart

![chart](https://mage-ai.github.io/assets/visualizations/time-series-bar-chart.png)

#### Time series line chart

![chart](https://mage-ai.github.io/assets/visualizations/time-series-line-chart.png)

### Data source

A chart has at least 1 data source.
In the near future, a chart can have more than 1 data source.

#### Block data output (type)

Get the output data from a block in a specific pipeline.

| Field | Description | Required | Example |
| - | - | - | - |
| Pipeline UUID | The pipeline UUID the block belongs to. | ✅ | `example_pipeline` |
| Block UUID | The block UUID you want data from. | ✅ | `load_api_data` |
| Partitions | Enter a positive or a negative number. If positive, then data from the block will be the most recent N partitions. If negative, then data from the block will be the oldest N partitions. | | `3` |

#### Block runs (type)

Get data from all block runs or
all block runs from a specific pipeline.

#### Custom code (type)

Define a function using the `@data_source` decorator that returns the data
for your chart.

```python theme={"system"}
@data_source
def d(*args, **kwargs):
    return [[dict(id=1), dict(id=2)]]
```

#### Pipelines (type)

Get data from all pipelines.

#### Pipeline runs (type)

Get data from all pipeline runs or
all pipeline runs from a specific pipeline or
all pipeline runs from a specific trigger.

#### Triggers (type)

Get data from all triggers or
all triggers from a specific pipeline.

#### Refresh interval

How frequent do you want this chart to automatically fetch new data from its data source?
Enter a number in milliseconds (e.g. 1000ms is 1 second).

### Chart display settings

#### Group by columns

Select the column to group your data by.

#### Metrics

| Field | Description | Required | Example |
| - | - | - | - |
| Aggregation | After grouping the data by 1 or more columns, perform the following aggregation function on a column. | ✅ | Average |
| Column | Select the column to perform the aggregation function on. | ✅ | `country` |

#### Chart style

Some charts have multiple styles of the same visualization.

#### Sort direction

Some charts can sort the visualizations in ascending or descending order.

#### Number of buckets or slices

You can limit the number of buckets or slices that are displayed for some charts.

#### Time column

For the time series charts, you must choose which column contains the date or time
value to group the data by.

#### Time interval

For the time series charts, you can choose how granular the interval is between each
date or time values in the time column. For example, you can display your time series
data on a daily basis or annual basis.

***

## Custom code for chart

The following decorators can optionally be used to further customize your chart.

### `@data_source`

If the data source type is `Custom code`, then this function’s return value is the
data that the chart will display.

However, if the data source type is anything else besides `Custom code`,
then this function can be used for post processing of the data from the selected
data source.

For example, let’s say the data source type is `Block data output` and
the number of partitions is `2`. The chart will retrieve the 2 most recent partitions of data from
that block.

In the `@data_source` decorated function, you can write code to operate on 2 of
those block output data:

```python theme={"system"}
from typing import List


@data_source
def filter_data(block_data_output: List, **kwargs) -> List:
    df_partition_1, df_partition_2 = block_data_output

    df_1 = df_partition_1[df_partition_1['Age'] >= 21]
    df_2 = df_partition_2[df_partition_2['Age'] >= 21]

    return [df_1, df_2]
```

### `@x`

Write custom code to return the `x` values for your chart.
X values typically contain the column that you group your data on.

For example, group pipelines by their type.

```python theme={"system"}
from typing import List


@x
def some_columns(data_from_data_source, **kwargs) -> List:
    df_partition_1 = block_data_output[0]

    return df_partition_1.columns[:3]
```

### `@y`

Write custom code to return the `y` values for your chart.
Y values typically contain the value for a group of rows.

For example, the number of pipelines that are data integration pipelines.

```python theme={"system"}
from typing import List


@y
def number_of_data_integration_pipelines(data_from_data_source, **kwargs) -> List:
    df_partition_1 = block_data_output[0]

    count = len(df_partition_1[df_partition_1['type'] == 'data_integration'].index)

    return [count]
```

### `@xy`

This function combines the purpose of `@x` and `@y` into 1 function.
Use this function if you want to custom both the `x` and `y` values and you want to share the
same variables defined within the function.

The difference is that you must return a tuple where
the `x` values are 1st and the `y` values are 2nd.

```python theme={"system"}
from typing import List, Tuple


@xy
def columns_and_numbers(data_from_data_source, **kwargs) -> Tuple[List]:
    df_partition_1 = block_data_output[0]

    count = len(df_partition_1[df_partition_1['type'] == 'data_integration'].index)

    return df_partition_1.columns[:1], [count]
```

### `@configuration`

Each chart has a configuration dictionary that stores the chart settings.
These chart settings are stored in a YAML file.

However, if you want to encode the configurations into the chart file and reuse the configurations
in other dashboards, you can write a function that returns a dictionary that will be used
to configure the chart.

```python theme={"system"}
from typing import Dict


@configuration
def chart_settings(*args, **kwargs) -> Dict:
    return dict(
        chart_type='time series line chart',
        group_by=['execution_date'],
        metrics=[dict(
            aggregation='count_distinct',
            column='id',
        )],
        time_interval='day',
    )
```

### `@columns`

In the chart settings, when choosing the group by columns, number columns, time column, metrics,
etc., the chart must know about what available columns are selectable.

Typically, the chart will attempt to infer the columns from the data source.
However, you can customize the logic of how the columns are inferred or
which columns are available to select.

```python theme={"system"}
from typing import List


@columns
def some_columns(data_from_data_source, **kwargs) -> List:
    df_partition_1 = block_data_output[0]

    return df_partition_1.columns[:3]
```

### `@render`

Define a function that returns a Base64 string representation of a JPEG/JPG image,
PNG image, or a complete HTML string (with html and body tags).

#### Render JPEG or JPG

```python theme={"system"}
import base64
import io
import matplotlib.pyplot as plt


# render_type can be 'jpeg' or 'jpg'
@render(render_type='jpeg')
def r(*args, **kwargs):
    # creating the dataset
    data = {'C':20, 'C++':15, 'Java':30,
            'Python':35}
    courses = list(data.keys())
    values = list(data.values())

    fig = plt.figure(figsize = (10, 5))

    # creating the bar plot
    plt.bar(courses, values, color ='maroon',
            width = 0.4)

    plt.xlabel("Courses offered")
    plt.ylabel("No. of students enrolled")
    plt.title("Students enrolled in different courses")
    plt.show()

    my_stringIObytes = io.BytesIO()
    plt.savefig(my_stringIObytes, format='jpg')
    my_stringIObytes.seek(0)
    my_base64_jpgData = base64.b64encode(my_stringIObytes.read()).decode()

    plt.close()

    return my_base64_jpgData
```

#### Render PNG

```python theme={"system"}
import base64
import io
import matplotlib.pyplot as plt
import numpy as np


@render(render_type='png')
def r(*args, **kwargs):
    t = np.arange(0.0, 2.0, 0.01)
    s = 1 + np.sin(2*np.pi*t)
    plt.plot(t, s)

    plt.xlabel('time (s)')
    plt.ylabel('voltage (mV)')
    plt.title('About as simple as it gets, folks')
    plt.grid(True)
    plt.show()

    my_stringIObytes = io.BytesIO()
    plt.savefig(my_stringIObytes, format='png')
    my_stringIObytes.seek(0)
    my_base64_jpgData = base64.b64encode(my_stringIObytes.read()).decode()

    plt.close()

    return my_base64_jpgData
```

#### Render HTML

```python theme={"system"}
import plotly.graph_objects as go


@render(render_type='html')
def r(*args, **kwargs):
    fig = go.Figure(
        data=[go.Bar(y=[2, 1, 3])],
        layout_title_text="A Figure Displaying Itself"
    )

    return fig.to_html(full_html=False)
```

***

## Recommended charts

Depending on which page you’re on, adding recommended charts will automatically configure
and add a few charts to your dashboard with little effort.

The added charts will show visualizations with different chart types and various data sources.

***

## Resize chart

When you resize a chart on the dashboard, you can change the width and height.

### Width

Set the width to a number.
This number is divided by the sum of the width numbers from all the charts in the same row.
The resulting number is the percentage of the row the chart will expand horizontally to.

For example, if you have a row with 5 charts and the 1st chart has a width of 6 and the other
4 charts have a width of 1, then the 1st chart will take up 60% of the entire row’s width.

### Max width percentage

Even if a chart is to take up a large percentage of the row’s entire width,
you can set a limit as to how wide the chart should be.

For example, in the above example the 1st chart should be 60% of the entire row’s width.
However, you can set that chart’s max width percentage to 50 and the 1st chart will only
expand to 50% of the row’s entire width.

### Height

Set this value to the number of pixels you want your chart to have in height.

***

## Move chart

Click and hold down on a chart to begin dragging it. Then, drag the chart to move it and
then release the click while on top of another chart to insert the initial chart after the
chart that you dropped it on.

***

## Remove chart

This will remove the chart from the current dashboard.


# Backfill guide
Source: https://docs.mage.ai/orchestration/backfills/guides

How to create a backfill.

1. From the pipelines list page, click a pipeline to go to the pipeline detail page.
2. In the vertical navigation bar on the left side of the screen, click the link for **Backfills**.
3. Click the button labeled **+ Create new backfill**.
4. Under the section labeled **Backfill type**, select **Date and time window** card.
5. Enter a name for your backfill.
6. Select a start date and time.
7. Select an end date and time.
8. Select an interval type; this value will help determine how many pipeline runs to create.
9. Enter an interval unit; this value is multiplied by the interval type to determine how many pipeline runs to create.
10. Click the button labeled **Save changes**.
11. Once the backfill is successful saved, click the button labeled **Start backfill**.

Check out this [blog article](https://www.mage.ai/blog/backfilling-historical-data-using-mage) for more details!


# Backfilling pipelines
Source: https://docs.mage.ai/orchestration/backfills/overview

Run a pipeline multiple times.

A backfill creates 1 or more pipeline runs for a pipeline. There are 2 types of backfills:

1. Date and time window
2. Custom code

## Date and time window

Create 1 or more pipeline runs between 2 datetime values.

The datetime of each instance is used as the `execution_date` for the pipeline run.

For example, if the backfill has the following attributes:

| Attribute | Value |
| - | - |
| Start datetime | `2023-01-01T03:00:00` |
| End datetime | `2023-01-05T03:00:00` |
| Interval type | `day` |
| Interval units | `2` |

Then the following pipeline runs will be created:

| `id` | `execution_date` | `ds` | `hr` |
| - | - | - | - |
| 1 | `2023-01-01T03:00:00` | `2023-01-01` | `03` |
| 2 | `2023-01-03T03:00:00` | `2023-01-03` | `03` |

***

## Custom code

The output of a backfill code will be used to generate the pipeline runs.

For example, if the backfill code has the following content:

```python theme={"system"}
backfill_data = []

for index, _ in range(3):
    backfill_data.append(dict(
        partition=index,
        power=5,
    ))

backfill_data
```

Then the following pipeline runs will be created:

| `id` | `execution_date` | `ds` | `hr` | `partition` | `power` |
| - | - | - | - | - | - |
| 1 | `2023-01-01T00:00:00` | `2023-01-01` | `00` | `0` | `5` |
| 2 | `2023-01-01T00:00:00` | `2023-01-01` | `00` | `1` | `5` |
| 3 | `2023-01-01T00:00:00` | `2023-01-01` | `00` | `2` | `5` |

***

## Extra runtime variables from pipeline run

If your pipeline run belongs to a trigger that is scheduled, then the following extra
variables are available in your Python block’s keyword arguments (e.g. `kwargs`).

| Key | Description | Example |
| - | - | - |
| `interval_start_datetime` | The `datetime` when the pipeline run is scheduled for. | `datetime.datetime(2023, 7, 23, 7, 0, 0, 0)` |
| `interval_end_datetime` | The `datetime` when the next pipeline run is scheduled for. | `datetime.datetime(2023, 7, 24, 7, 0, 0, 0)` |
| `interval_seconds` | The number of seconds between the current pipeline run and the next pipeline run. | `86400` |
| `interval_start_datetime_previous` | The `datetime` when the previous pipeline run was scheduled for. | `datetime.datetime(2023, 7, 22, 7, 0, 0, 0)` |

### SQL block

If you’re using a SQL block, here is an example of how you can access these variables:

```sql theme={"system"}
SELECT
    '{{ interval_start_datetime }}' AS interval_start_datetime
    , '{{ interval_end_datetime }}' AS interval_end_datetime
    , '{{ interval_seconds }}' AS interval_seconds
    , '{{ interval_start_datetime_previous }}' AS interval_start_datetime_previous
```


# Query pipeline run metadata from database
Source: https://docs.mage.ai/orchestration/pipeline-runs/database-query



Pipeline runs and Block runs data are stored in `pipeline_run` and `block_run` tables in the database.
If you want to query the `PipelineRun` and `BlockRun` metadata, you can either directly query the data
from database or use Python code to fetch `PipelineRun` and `BlockRun` models.

## Query from DB

Here is the query from Postgres. You can adapt it to the syntax of other databases when needed.

```sql theme={"system"}
SELECT
    *
FROM pipeline_run
WHERE pipeline_uuid = 'example_pipeline' -- Filter by pipeline_uuid
AND status = 'COMPLETED' -- Filter by status
```

## Query DB models via Python

```python theme={"system"}
from mage_ai.orchestration.db import db_connection
from mage_ai.orchestration.db.models.schedules import PipelineRun, BlockRun

db_connection.start_session()

# Fetch running pipeline runs of a pipeline
pipeline_runs = PipelineRun.query.filter(
    # Filter by pipeline uuid
    PipelineRun.pipeline_uuid == 'example_pipeline',
    # Filter by status
    PipelineRun.status == PipelineRun.PipelineRunStatus.RUNNING,
    # Filter by execution_date
    PipelineRun.execution_date >= '2024-01-01',
    PipelineRun.execution_date < '2024-02-01',
)

# Get the block runs of a pipeline run
block_runs = pipeline_runs.all()[0].block_runs
print(f'Block runs: {[b.to_dict() for b in block_runs]}')

# Get pipeline run count
print(f'Number of pipeline runs {pipeline_runs.count()}')

# Batch update pipeline run status
PipelineRun.batch_update_status([p.id for p in pipeline_runs], PipelineRun.PipelineRunStatus.CANCELLED)

```


# Pipeline Run
Source: https://docs.mage.ai/orchestration/pipeline-runs/overview

Learn how to monitor, manage, and troubleshoot pipeline runs in Mage for optimal data pipeline performance.

## Overview

Pipeline runs represent **individual executions of your data pipelines** in Mage. Each time a pipeline is triggered (manually or automatically), a pipeline run record is created that tracks the entire execution process, including all block runs, status updates, timing information, and any errors that occur.

### Key Benefits of Pipeline Run Management:

* **Execution tracking** - Monitor the status and progress of every pipeline execution
* **Performance insights** - Analyze execution times, resource usage, and bottlenecks
* **Error debugging** - Detailed logs and error information for troubleshooting
* **Historical analysis** - Review past executions to identify patterns and issues
* **Retry capabilities** - Restart failed or incomplete pipeline runs
* **Audit trail** - Complete record of when and how pipelines were executed

> [!NOTE]
>
> **Pipeline Run** and **Pipeline Execution** are the same thing in Mage. You may see either term used throughout the product.

## Pipeline Run Lifecycle

### Run Statuses

Pipeline runs progress through several statuses during their lifecycle:

* **`initial`** - Pipeline run has been created but not yet started
* **`running`** - Pipeline is currently executing
* **`completed`** - Pipeline finished successfully
* **`failed`** - Pipeline encountered an error and stopped
* **`cancelled`** - Pipeline run was manually cancelled or timed out

### Block Run Statuses

Within each pipeline run, individual blocks have their own statuses:

* **`initial`** - Block run created but not started
* **`queued`** - Block run is in the queue and waiting to be executed
* **`running`** - Block run is currently executing
* **`completed`** - Block run finished successfully
* **`failed`** - Block run encountered an error
* **`cancelled`** - Block run was cancelled
* **`upstream_failed`** - Block cannot run because upstream blocks failed
* **`condition_failed`** - The block was skipped because its own or an upstream block's condition was not satisfied.

## Viewing Pipeline Runs

### Project-Wide Pipeline Runs Dashboard

View all pipeline runs across your entire project:

1. **Navigate to the homepage** and select **Pipeline runs** in Mage OSS or **Runs** in Mage Pro
2. **Or go directly to** `/pipeline-runs` URL
3. **Filter** by pipeline, status, or date range, and **search** by pipeline or trigger
4. **Monitor key metrics** like success rate, average duration, and recent activity

#### View Types

**Current and Past Runs:**

* View all completed, failed, cancelled, and currently running pipeline runs
* Available in both Mage OSS and Mage Pro
* Shows historical execution data and current status

**Upcoming Runs (Mage Pro only):**

* View scheduled pipeline runs that are yet to execute
* See future run dates and trigger information
* Monitor upcoming pipeline executions and scheduling

#### Pipeline Run List Filters

The pipeline run list page provides comprehensive filtering options:

**Available Filters:**

* **Pipeline UUID** - Filter by specific pipeline identifiers
* **Status** - Filter by run status (Cancelled, Done, Failed, Last run failed, Ready, Running)
* **Pipeline Tag** - Filter by pipeline tags
* **Date range** - Filter current and past runs by bounded UTC preset ranges (Last hour, Last day, Last week, Last 30 days, Last year), all time, or a custom UTC start and end date and time
* **Search** - Text search by pipeline or trigger (Mage Pro only)

**Filter Actions:**

* **Apply filters** - Apply selected filter criteria
* **Reset filters** - Clear all filters and return to defaults

In Mage Pro, select **Custom UTC range** to enter exact start and end values
directly in the `YYYY-MM-DD HH:mm` inputs, or choose them with the calendar and
time picker. The inputs keep separators and placeholders in place while you edit,
and **Apply** remains disabled until both values form a complete valid range.

Pipeline UUID, status, and pipeline tag filters are preserved when you navigate away and return
during the same browser session. Selecting **Clear** removes the saved filters. Free-text search is
cleared when you leave the page. Saved filters and search text are not restored in a new browser
session unless they are part of the URL you open.
Applied Pipeline UUID, status, and pipeline tag filters also appear in a band below the toolbar for
current and past runs. Select an individual chip to remove one value, or select **Clear** to remove
all applied filters while preserving unrelated list state such as search, date range, and page size.

> [!NOTE]
>
> **Mage OSS vs Pro**:
>
> * **Mage OSS**: Supports viewing current and past runs with basic filtering (Pipeline UUID, Status, Pipeline Tag)
> * **Mage Pro**: Supports both current/past runs AND upcoming runs, plus advanced filtering including text search and monitoring features
>
> The project-wide dashboard shows runs from all pipelines, making it easy to monitor your entire data infrastructure at a glance.

#### Project-Wide Batch Operations

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=project-wide-pipeline-run-actions)

The project-wide **Pipeline runs** page supports Mage Pro-only batch actions
across all pipelines. Selected-run actions operate on the selected runs, while
retry-all and cancel-all actions operate on the current filtered set.

**Project-wide actions:**

* **Retry selected** - Retry specific selected pipeline runs
* **Retry all failed** - Retry all failed pipeline runs in the current filtered set
* **Cancel selected running** - Cancel specific selected Ready or Running pipeline runs (Mage Pro only)
* **Cancel all running** - Cancel all currently running pipeline runs

**How to use project-wide batch operations:**

1. **Select pipeline runs** by checking the boxes next to the runs you want to manage
2. **Click the "Actions" dropdown** to access batch operation options
3. **Choose your desired action** from the dropdown menu
4. **Confirm the operation** when prompted

**Cancel selected running** can include runs from different pipelines. It does
not cancel other active runs in the project.

### Pipeline-Specific Run History

View runs for a specific pipeline:

1. **Click on a pipeline** from the main Pipelines Dashboard
2. **Navigate to the "Runs" section** or click on a trigger from the "Triggers" section
3. **View detailed run information** including block runs, logs, and metrics
4. **Access individual run details** by clicking on specific runs

**Direct URL**: `/pipelines/[pipeline_uuid]/runs`

In Mage Pro, right-click a run on either the project-wide **Pipeline runs** page or a
pipeline's **Runs** page to open the run actions menu. From this menu, you can open the run,
edit its pipeline, view its logs, select **Copy URL** to copy its direct URL, select **Retry
pipeline run** to restart the entire run, or select **Retry incomplete blocks** on a failed run
to resume from its incomplete blocks.
You can also select a failed run's status chip and choose **Retry incomplete blocks** from
the status menu; the option remains visible but disabled for runs that cannot be retried.
**Cancel** is enabled only for running runs, and **Fix with AI** is enabled only for failed
runs when AI Sidekick is enabled; unavailable actions remain visible but disabled.

#### Pipeline-Specific Batch Operations

The pipeline-specific **Runs** page supports batch operations for the current
pipeline.

**Pipeline-specific actions:**

* **Retry selected** - Retry specific selected pipeline runs
* **Retry incomplete block runs of selected** - Retry incomplete block runs for selected pipeline runs
* **Retry all incomplete block runs** - Retry all incomplete block runs across that pipeline
* **Cancel selected running** - Cancel specific selected Ready or Running pipeline runs
* **Cancel all running** - Cancel all currently running pipeline runs
* **Delete selected** - Permanently delete selected pipeline runs after confirmation

**How to Use Batch Operations:**

1. **Select pipeline runs** by checking the boxes next to the runs you want to manage
2. **Click the "Actions" dropdown** to access batch operation options
3. **Choose your desired action** from the dropdown menu
4. **Confirm the operation** when prompted

> [!NOTE]
>
> Batch operations are particularly useful for managing multiple failed runs or stopping multiple running pipelines at once, improving operational efficiency.

### Individual Pipeline Run Details

For detailed analysis of a specific run:

1. **Click on a pipeline run** from the runs list
2. **View the individual run page** at `/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`
3. **Examine block run details** including execution order, timing, and status
4. **Access logs and error information** for each block
5. **Review runtime variables** and configuration used
6. **Review observed Kubernetes resource usage** in the **Resources** tab when stored average and peak CPU/RAM metrics are available for Mage Pro runs that use the Kubernetes executor. Use pipeline **Monitoring** for pipeline-level Kubernetes executor recommendations.

In Mage Pro, right-click a block run to open its block in the pipeline editor, view its logs,
or retry it after the pipeline run finishes. When CSV output download is available for a
completed block run, the menu also includes **Download output**.

## Pipeline Run Retry

### Retrying Failed Runs

When a pipeline run fails, you can retry it in several ways:

> [!NOTE]
>
> For detailed information about retry configuration and management, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).

#### Retry Incomplete Blocks

Retry from the failed block instead of restarting the entire pipeline. For detailed instructions, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).

#### Retry from Selected Block

Retry starting from any selected block regardless of its current status. For detailed instructions, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).

#### Retry a Specific Integration Stream

Select an incomplete stream row in a data integration pipeline run to retry only that stream. The broader retry incomplete action remains available for retrying every incomplete stream. For detailed instructions, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).

### Automatic Retry Configuration

Configure automatic retries to handle transient failures. For detailed retry configuration examples and advanced settings, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).

## Pipeline Run Data and Monitoring

### Key Metrics and Information

Each pipeline run contains valuable information:

* **Execution timing** - Start time, completion time, and duration
* **Status tracking** - Current status and any status changes
* **Block run details** - Individual block execution information
* **Runtime variables** - Variables used during execution
* **Error information** - Detailed error messages and stack traces
* **Resource usage** - CPU, memory, and other resource metrics
* **SLA compliance** - Whether the run met configured SLAs

### Database Storage

Pipeline run data is stored in database tables:

* **`pipeline_run`** - Main pipeline run records
* **`block_run`** - Individual block run records within each pipeline run

> [!NOTE]
>
> You can query pipeline run data directly from the database or use Mage's Python API to access run information programmatically. For detailed database querying examples, see [Query pipeline run metadata from database](/orchestration/pipeline-runs/database-query). For API reference documentation, see [Pipeline runs API reference](/extensibility/api-reference/pipeline-runs/overview).

### Querying Pipeline Run Data

Query pipeline run data directly from the database or use Mage's Python API. For comprehensive database querying examples and Python API usage, see [Query pipeline run metadata from database](/orchestration/pipeline-runs/database-query).

## Advanced Pipeline Run Features

### Runtime Variables

Override pipeline variables for specific runs:

* **Trigger-level variables** - Set variables when creating or editing triggers (see [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines))
* **API-triggered variables** - Pass variables via API requests (see [Trigger pipeline via API](/orchestration/triggers/trigger-pipeline-api))
* **Manual run variables** - Override variables when manually triggering runs

> [!NOTE]
>
> Runtime variables allow you to customize pipeline behavior for different environments, data sources, or processing requirements without modifying the pipeline code. Learn more about [using runtime variables](/getting-started/runtime-variable).

### SLA Monitoring

Configure and monitor Service Level Agreements. For detailed SLA configuration, see [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines#sla-configuration).

* **Expected completion time** - Set target completion times for pipeline runs
* **Time units** - Configure in minutes, hours, or days
* **SLA tracking** - Monitor whether runs meet their SLA targets
* **Alerting** - Get notified when SLAs are missed

### Timeout Configuration

Control pipeline run execution time. For detailed timeout configuration, see [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines#timeout-configuration).

* **Timeout duration** - Set maximum execution time in seconds
* **Timeout status** - Choose how to handle runs that exceed timeout (failed, cancelled, warning)
* **Resource protection** - Prevent runaway pipelines from consuming excessive resources

## Debugging Pipeline Run Failures

When pipeline runs fail, use these debugging techniques to identify and resolve issues:

### Accessing Run Information

**Individual Run Details:**

1. **Navigate to the failed run** in the pipeline runs list
2. **Click on the run** to view detailed information
3. **Examine the execution timeline** to see where the failure occurred
4. **Review block run statuses** to identify which blocks failed

**Block-Level Debugging:**

1. **Click on individual block runs** to see detailed information
2. **Access block logs** for error messages and stack traces
3. **Open the log source** from the selected error log's source-location link, or click the block name in the Block column, to open the failing block in the pipeline editor. When line metadata is available, Mage focuses that line in the editor.
4. **Review block output** to understand data state at failure
5. **Check upstream dependencies** to ensure data flow is correct

### Common Failure Patterns

**Upstream Block Failures:**

* **Issue**: Downstream blocks show `upstream_failed` status
* **Solution**: Fix the upstream block that failed first, then retry

**Data Validation Errors:**

* **Issue**: Blocks fail due to unexpected data format or missing columns
* **Solution**: Add data validation and error handling in your blocks

**Resource Exhaustion:**

* **Issue**: Pipeline runs timeout or consume excessive resources
* **Solution**: Optimize block code, increase timeout settings, or break into smaller pipelines

**External Dependency Failures:**

* **Issue**: API calls, database connections, or file access failures
* **Solution**: Implement retry logic and proper error handling

**Pipeline Runs Stuck in Initial/Queued Status:**

* **Issue**: Pipeline runs remain in `initial` or `queued` status with no visible errors in logs
* **Solution**: Follow these troubleshooting steps:
  1. **Check concurrency limits** - Verify that block run concurrency or pipeline run concurrency limit is not set to 0
  2. **Check resource usage** - Monitor CPU and RAM usage to ensure sufficient resources are allocated
  3. **Check server error logs** - Review server-side logs for underlying issues
  4. **Restart the server** - Try restarting the Mage server to clear any stuck processes
  5. **Contact Mage team** - If the issue persists, reach out to the Mage team for debugging assistance at [mage.ai/chat](https://mage.ai/chat)

### Debugging Tools

* **Detailed logs** - Access comprehensive execution logs for each block
* **Error tracking** - View error messages, stack traces, and debugging information
* **Execution timeline** - See the order and timing of block executions
* **Variable inspection** - Review runtime variables and their values
* **Kubernetes resource summary** - In Mage Pro, review observed average and peak CPU/RAM for Kubernetes executor runs; use pipeline Monitoring for pipeline-level Kubernetes executor recommendations
* **Block output export** - Save block output as CSV for analysis (see [Saving block output as CSV](/orchestration/pipeline-runs/saving-block-output-as-csv))

## Best Practices for Pipeline Run Management

### Monitoring and Alerting

* **Set up monitoring** for key pipeline runs and their success rates
* **Configure alerts** for failed runs, SLA violations, and performance issues
* **Regular review** of run history to identify patterns and improvements

### Error Handling

* **Implement retry logic** for transient failures
* **Use proper error handling** in your block code
* **Validate data** at each step to catch issues early
* **Test thoroughly** before deploying to production

### Performance Optimization

* **Monitor execution times** and optimize slow blocks
* **Use appropriate resources** for your workload
* **Implement caching** where appropriate
* **Consider parallel processing** for independent operations
* **Configure concurrency limits** to control resource usage (see [Concurrency](/design/data-pipeline-management#concurrency))

### Documentation and Maintenance

* **Document pipeline purposes** and expected behavior
* **Maintain run history** for audit and compliance purposes
* **Regular cleanup** of old run data to manage storage
* **Version control** your pipeline code and configurations

## Frequently Asked Questions

> **How can I retrieve the output of a block after triggering a pipeline via API?**
>
> You can retrieve the output of a block run by calling the block run output API. For example:
>
> ```
> {MAGE_HOST}/api/block_runs/<block_run_id>/outputs?sample_count=100
> ```
>
> **Steps to get block run ID:**
>
> 1. Trigger your pipeline via API or UI
> 2. Navigate to the pipeline run details page
> 3. Find the specific block run ID from the block runs list
> 4. Use the API endpoint above with the block run ID
>
> You can find the relevant API calls in your browser console when using the Mage UI. More details are available in the [API reference guide](/extensibility/api-reference/pipeline-runs/overview).

> **Can I monitor the status of a pipeline run programmatically?**
>
> Yes, you can monitor pipeline run status programmatically using multiple methods:
>
> **API Methods:**
>
> * Use the Mage API to query pipeline run status and outputs
> * Access individual pipeline run details via `/api/pipeline_runs/<pipeline_run_id>`
> * Monitor block run statuses within a pipeline run
>
> **Database Query Methods:**
>
> * Query the `pipeline_run` and `block_run` tables directly
> * Use Python scripts to filter by status, execution date, or pipeline UUID
> * Batch update pipeline run statuses programmatically
>
> See [Query pipeline run metadata from database](/orchestration/pipeline-runs/database-query) for detailed examples and [API reference guide](/extensibility/api-reference/pipeline-runs/overview) for API documentation.

> **How do I cancel all pipelines that are in Ready/Running status?**
>
> You can cancel all running pipeline runs using either method:
>
> **UI Method:**
>
> * Navigate to `/pipelines/[pipeline_uuid]/runs` and click **"Cancel all running"** button
> * This cancels all running pipeline runs for a specific pipeline
>
> **Programmatic Method:**
>
> * Write a Python script to cancel all ready or running pipeline runs
> * Example code and guidance are available in the [database query documentation](/orchestration/pipeline-runs/database-query)

> **How are runtime variables managed in pipeline runs?**
>
> Runtime variables are managed at multiple levels in pipeline runs:
>
> **Variable Storage:**
>
> * Run records store information about runtime variables used during execution
> * Variables are typically specified at the time of pipeline trigger
> * Variables can be overridden at the trigger level for different environments
>
> **Variable Sources:**
>
> * **Trigger-level variables** - Set when creating or editing triggers
> * **API-triggered variables** - Passed via API requests in the payload
> * **Manual run variables** - Overridden when manually triggering runs
>
> **Variable Usage:**
>
> * Access variables in blocks using `kwargs.get("variable_name", "default_value")`
> * Variables allow customization for different environments without code changes
> * Runtime variables are stored in the `variables` field of pipeline run records
>
> Learn more about [using runtime variables](/getting-started/runtime-variable) and [trigger configuration](/orchestration/triggers/triggering-pipelines).

> **Why do some pipeline runs get stuck or blocks remain in the initial status?**
>
> Pipeline runs can get stuck in `initial` or `queued` status due to several common issues:
>
> **Common Causes:**
>
> * **Concurrency limits** - Block run concurrency or pipeline run concurrency limit set to 0
> * **Resource constraints** - Insufficient CPU or RAM allocation
> * **Executor issues** - Problems with k8s\_executor, volume mounting, or resource constraints
> * **Server issues** - Mage server experiencing problems or stuck processes
>
> **Troubleshooting Steps:**
>
> 1. **Check concurrency limits** - Verify concurrency settings are not set to 0
> 2. **Monitor resource usage** - Ensure sufficient CPU and RAM are allocated
> 3. **Check server error logs** - Review server-side logs for underlying issues
> 4. **Restart the server** - Try restarting Mage to clear stuck processes
> 5. **Switch executors** - Try local\_python executor for testing to isolate issues
>
> **Additional Debugging:**
>
> * If logs are not visible in the UI, check log files directly in the log path
> * Use the debugging tools available in the pipeline run details page
> * Contact the Mage team at [mage.ai/chat](https://mage.ai/chat) for persistent issues

> **How do I set up automated or scheduled pipeline runs?**
>
> You can set up automated pipeline runs using scheduled triggers in Mage:
>
> **Creating Scheduled Triggers:**
>
> 1. **Navigate to pipeline** - Go to your pipeline's trigger page
> 2. **Create new trigger** - Click "New trigger" and select "Schedule"
> 3. **Configure frequency** - Choose from hourly, daily, weekly, monthly, or custom cron expressions
> 4. **Set additional options** - Configure SLA, timeout, runtime variables, and other settings
> 5. **Save trigger** - Enable the trigger to start automated runs
>
> **Available Frequencies:**
>
> * **Run exactly once** - Execute once and disable
> * **Hourly/Daily/Weekly/Monthly** - Standard recurring schedules
> * **Always on** - Immediately create new runs when previous completes
> * **Custom cron** - Advanced scheduling with cron expressions
>
> **Requirements:**
>
> * **Docker deployments** - Container must remain running for scheduled triggers to work
> * **SaaS version** - Mage offers a managed service that handles scheduling automatically
>
> **Advanced Configuration:**
>
> * Set runtime variables for different environments
> * Configure SLA monitoring and timeout settings
> * Override global variables at trigger level
> * Use "Run once" button for testing triggers
>
> See [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines) for detailed configuration and [Trigger pipeline via API](/orchestration/triggers/trigger-pipeline-api) for API-based triggering.

> [!NOTE]
>
> **Need more help?** If your question isn't answered here, join the Mage community on [Slack](https://mage.ai/chat) to get help from other users and the Mage team.


# Retrying block runs from a pipeline run
Source: https://docs.mage.ai/orchestration/pipeline-runs/retrying-block-runs



> [!NOTE]
>
> Requires version `0.8.76` or greater.

## Automatic retry

Mage supports configuring automatic retry for block runs in the following ways.
Automatic retry applies to standard block runs and data integration pipeline stream block runs.
When multiple retry configs are present, Mage merges them in this order:
project `metadata.yaml` `retry_config`, pipeline `metadata.yaml` `retry_config`, then block `retry_config`.
Later configs override earlier values for matching keys only when the later value is not blank or `null`.

> [!NOTE]
>
> A resolved retry config retries only when `retries` is greater than `0`. If `retries` is blank,
> `null`, or not set at the pipeline or block level, Mage inherits the lower-precedence value. To turn
> off inherited retries, set `retries: 0` on the pipeline or block. If `delay` or `max_delay` is blank
> or `null`, Mage also inherits the lower-precedence value; if the resolved config still does not set
> them, Mage uses `5` and `60` seconds.

### 1. Global retry config

The global `retry_config` is saved in the project's `metadata.yaml` and applies
to all block runs by default. Pipeline-level and block-level retry settings can
override the global values.

When retry settings are configured at multiple levels, Mage merges them from
least specific to most specific: global project defaults, then pipeline-level
settings, then block-level settings. For the same retry field, the most specific
configured value wins. Blank or `null` retry fields are ignored during this
merge, so a blank pipeline or block field inherits the next available value from
the less-specific config instead of clearing it.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=automatic-retry)

In Mage Pro, go to **Settings > Workspace > Automatic retry** or open
`/settings/workspace/automatic-retry` to configure the global retry settings for
the project. Users with viewer access can open the page, but the retry controls
are read-only. Set the number of retries, the initial delay in seconds, the
maximum accumulated retry delay in seconds before scheduling the next attempt,
and whether to use exponential backoff. The maximum delay cap applies whether or
not exponential backoff is enabled. Leave a field blank to omit that value from
the saved project metadata, or set retries to `0` to explicitly disable global
automatic retries. Retry count, delay, and max delay must be non-negative
integers; retries can be up to `100`, and delay/max delay can be up to `86400`
seconds. `0` is accepted for numeric retry fields. If `exponential_backoff` is
omitted, Mage uses the default enabled backoff behavior. Select **Save Settings**
below the retry controls to persist your changes.

You can also edit the project's `metadata.yaml` directly:

```yaml theme={"system"}
retry_config:
  # Number of retry times
  retries: 3
  # Initial delay (in seconds) before retry. If exponential_backoff is true,
  # the delay time is multiplied by 2 for the next retry
  delay: 5
  # Maximum accumulated retry delay before scheduling the next attempt
  max_delay: 60
  # Whether to use exponential backoff retry
  exponential_backoff: true
```

### 2. Block level retry config

Add `retry_config` to the block config in pipeline's `metadata.yaml`. The block level `retry_config` will
override the global and pipeline-level `retry_config` for fields that have
configured values. Leave a block retry field blank or set it to `null` to inherit
that field from the pipeline or global config. Here is an example

```yaml theme={"system"}
blocks:
- uuid: example_data_loader
  type: data_loader
  upstream_blocks: []
  downstream_blocks: []
  retry_config:
    retries: 2
    delay: 5
    max_delay: 10
    exponential_backoff: false
  ...
```

### 3. Pipeline level `retry_config`

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=automatic-retry)

Add `retry_config` to the pipeline config in pipeline's `metadata.yaml`. The pipeline level `retry_config` will
override the global `retry_config` for fields that have configured values unless
a block sets its own retry value. Leave a pipeline retry field blank or set it to
`null` to inherit that field from the global config. Here is an example

```yaml theme={"system"}
blocks:
- ...
- ...
- ...
retry_config:
  retries: 2
  delay: 5
  max_delay: 10
  exponential_backoff: false
  ...
```

The retry logs show in the same block run log page.

## Retry incomplete blocks

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=retry-incomplete-blocks)

* Supported pipeline types: Standard, Streaming, and Integration pipelines
* If a pipeline run fails, you can retry from the failed block
  run instead of restarting the entire pipeline run. In Mage Pro, open the failed
  run's actions menu from the pipeline's **Runs** table, or select its failed status chip,
  and choose **Retry incomplete blocks**. The status-menu option is disabled for runs that
  cannot be retried. The separate **Retry pipeline run** action restarts the entire pipeline run.
  For Streaming pipelines, **Retry incomplete blocks** reuses the existing pipeline
  run, resets its incomplete block runs, and resumes it instead of creating a new run.
* You can also retry incomplete blocks from the individual pipeline run page
  (`/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`).
* At the top of the individual pipeline runs page, there should be a red
  `Retry incomplete blocks` button. Click on it, and the block runs should
  retry from the failed block.
* For Integration pipelines, this broad retry keeps the existing behavior and
  retries every incomplete stream in the run.

## Retry a specific integration stream

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=data-integration)

* Supported pipeline types: Integration pipelines
* If a data integration pipeline run has a failed stream, go to the individual
  pipeline run page (`/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`).
* Select any block run row for the stream you want to retry. The selected row
  can be completed if another block run for that stream failed.
* At the top of the page, click the `Retry` button with the stream name shown
  next to it. Mage retries only the stream shown on the button and leaves other
  streams unchanged.

## Retry from selected block

* Supported pipeline types: Standard and Integration pipelines
* If you want to retry block runs for a pipeline run starting from a specific
  block (regardless of the block run's status), go to the same page for the individual
  pipeline run.
* Select a block run row, and a blue `Retry from selected block` button should appear.
* Note that the `Retry from selected block` button will only appear if the pipeline
  run is not currently running.
* In Mage Pro, for Standard pipelines, selecting a generated dynamic child block run retries
  that child and downstream block runs that depend on it. Unaffected generated siblings
  and unrelated branches stay unchanged. If downstream blocks consume a reduced collection
  or a shared ordinary block output affected by the retry, all rows that depend on that
  collection or output rerun.
* For Integration pipelines, selecting a block run in a completed pipeline run keeps
  the existing selected-block retry behavior. Selecting a stream in an incomplete
  run uses the specific stream retry described above.
* If there is a failed block run and you try to retry from a selected block that is
  downstream from that failed block, you may get `upstream_failed` block run errors.

### Accessing the individual pipeline runs page

1. Click on a pipeline from the main Pipelines Dashboard.
2. Click on the `Runs` section or on an individual trigger from the `Triggers` section.
3. Underneath the `Block runs` column of the pipeline runs table, there is a link that
   says something like `See block runs (x)`. Click on that link to redirect to the
   individual pipeline runs page, which lists the blocks runs specific to a pipeline run.


# Saving block output as CSV file
Source: https://docs.mage.ai/orchestration/pipeline-runs/saving-block-output-as-csv



> [!NOTE]
>
> Requires version `0.8.92` or greater.

For standard pipelines (not currently supported in integration or streaming
pipelines), you can save the output of a block that has been successfully
run and saved to disk as a CSV file.

Note that there may be limitations for saving very
large block outputs (e.g. greater than 200 mb). Downloading block output
saved to the cloud (e.g. S3) is not currently supported.

There are two places to save the block output as a CSV file:

1. In the Pipeline Editor page after successfully running a block

* After a block has been successfully run, click the `Save` icon
  in the block output footer.

2. In the Block Runs table after a block run has successfully completed

* The Block Runs table can be found by going to a pipeline's `Runs`
  section (found in left navigation sidebar) of the Pipeline Detail page
  and clicking the `Block runs` tab or clicking the `X / X`
  link under the `Block runs` column of the Pipeline Runs table.
* Go to the far right of the Block Runs table and click the `Save` button
  under the `Output` column.
* In Mage Pro, when CSV output download is available, you can instead right-click the
  completed block run and select `Download output`.


# Configure triggers in code
Source: https://docs.mage.ai/orchestration/triggers/configure-triggers-in-code

Configure triggers in triggers.yaml under pipeline folder.

In addition to configuring triggers via UI, you can also configure triggers in code.

You can either manually define the triggers in code or you can go to the triggers detail page in the
UI and click the button labeled **Save trigger in code** on the left side panel to store the
current trigger’s settings to code.

## How triggers in code are synced to the database and UI

* **Source of truth:** The `triggers.yaml` file is the source of truth for trigger definitions.
* **Sync flow:** A background scheduler process periodically syncs triggers from `triggers.yaml` into the database. Only triggers that pass the **envs filter** (see [Envs filter](#envs-filter)) are created or updated on that instance. After a trigger is synced to the database, it appears in the Triggers UI.
* **New/updates flow both ways:** New or updated triggers in the YAML are pushed to the database when the scheduler runs. Edits you make in the UI to a trigger that's defined in code are written back to `triggers.yaml` so UI and code stay in sync.
* **Deletions are not synced:** Removing a trigger from `triggers.yaml` does **not** remove it from the database. You must delete the trigger in the UI (or via API) to remove it from the database. See [Delete triggers](#delete-triggers).

## Create and configure triggers

Here are the steps to create and configure triggers via code:

1. Create a `triggers.yaml` file under your pipeline folder. The file path should be `pipelines/[pipeline_uuid]/triggers.yaml`.

2. Enter your trigger configs into the `triggers.yaml` file.
   1. Content structure
      ```yaml theme={"system"}
      triggers:
      - name: test_trigger
        schedule_type: time
        start_time: 2023-01-01
        status: active
      - name: xxx
        ...
      - name: xxx
        ...
      ```
   2. **Required trigger fields**
      * `name`: Unique identifier of the trigger (per pipeline).
      * `schedule_type`: `time` or `api` for new triggers. `event` is deprecated and should only be used to maintain existing legacy event triggers.
      * For **time (schedule)** triggers: `schedule_interval` and `start_time` are required.
        * `schedule_interval`: `@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`, `@always_on`, or a Cron expression (e.g. `* * * * *`).
        * `start_time`: Persisted schedule start timestamp. For `@once`, this is the single run time (e.g. `2023-01-01` or `2023-01-01 00:00:00`).
      * For **api** triggers: `schedule_interval` and `start_time` are typically not used for execution but may still be present in the config.
   3. **Optional trigger fields**
      * `status`: `active` or `inactive`. Defaults to `inactive` if omitted.
      * `variables`: A dictionary of [runtime variables](/getting-started/runtime-variable) for this trigger.
      * `sla`: SLA in seconds for this trigger.
      * `settings`: A dictionary of [advanced settings](/orchestration/triggers/triggering-pipelines#additional-trigger-settings):
        * `skip_if_previous_running`: `true`/`false` — do not start a new run if the previous run is still in progress.
        * `allow_blocks_to_fail`: `true`/`false` — allow the pipeline to continue running non-dependent blocks if some blocks fail.
        * `create_initial_pipeline_run`: `true`/`false` — create an initial run when the start time is in the past (schedule triggers).
      * `envs`: List of environments where this trigger should be synced and run. See [Envs filter](#envs-filter) below.

3. Save the `triggers.yaml` file. On the next scheduler run, trigger configs are synced to the database (subject to the envs filter) and then show in the Triggers UI.
   1. Required trigger fields
      1. `name`: Unique identifier of the trigger
      2. `schedule_type`: `time` or `api` for new triggers. `event` is deprecated for legacy event triggers only.
      3. `schedule_interval`: `@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`, `@always_on`, or Cron syntax(`* * * * *`)
      4. `start_time`: Persisted schedule start timestamp. For `@once`, this is the single run time (e.g. `2023-01-01`)
   2. Optional trigger fields
      1. `status`: `active` or `inactive`
      2. `variables`: A dictionary of variables that's associated with the trigger
      3. `sla`: SLA in seconds
      4. `settings`: a dictionary of [advanced settings](/orchestration/triggers/triggering-pipelines#additional-trigger-settings)
         1. `skip_if_previous_running`: `true`/`false`
         2. `allow_blocks_to_fail`: `true`/`false`
         3. `create_initial_pipeline_run`: `true`/`false`
      5. `envs`: The environments that the trigger runs in.
         * If `envs` is not set or empty, the trigger will run in all environments.
         * The environment of the Mage app is set via the `ENV` [environment variable](/development/variables/environment-variables).
         * **OSS:** Only supports `dev`, `staging`, `prod`, `test`.
         * **Mage Pro:** Supports any value, which must match the `ENV` environment variable.

4. Save the `triggers.yaml` file. The trigger configs will be synced to the triggers UI shortly.

### Example triggers config:

```yaml theme={"system"}
triggers:
- name: test_example_trigger
  schedule_type: time
  schedule_interval: "@daily"
  start_time: 2023-01-01
  status: active
- name: test_example_trigger_2
  schedule_type: time
  schedule_interval: "@hourly"
  start_time: 2023-03-01
  status: inactive
- name: test_example_trigger_with_extra_settings
  schedule_type: time
  schedule_interval: "@hourly"
  start_time: 2023-03-01
  status: inactive
  settings:
    skip_if_previous_running: true
    allow_blocks_to_fail: true
  envs:
  - prod
  - dev
```

## Envs filter

The optional `envs` field limits which instances or workspaces sync and run the trigger. Each instance/workspace has an environment set by the `ENV` [environment variable](/development/variables/environment-variables).

* **When `envs` is not set or empty:** The trigger is synced to **all** instances/workspaces. Every environment will get the trigger in the database and can run it.
* **When `envs` is set:** The trigger is synced **only** to instances/workspaces whose `ENV` value is in the `envs` list. Other instances will not create or update this trigger in their database, so they will not run it.
  * **OSS:** Allowed values are `dev`, `staging`, `prod`, and `test`.
  * **Mage Pro:** Any value is allowed; it must match the `ENV` set for that instance or workspace.

**If you add an `envs` filter to a trigger that already exists everywhere:** Triggers that were previously synced (e.g. when `envs` was empty) remain in the database on instances that no longer match the new `envs` list. The sync process only creates or updates triggers; it does not delete them. To stop such a trigger from existing (and running) on those instances, you must **delete the trigger manually** in the UI on each affected instance or workspace.

## Modify triggers

If an existing trigger is defined in the YAML file, changes flow both ways: updates in the UI are written back to `triggers.yaml`, and updates in the YAML are synced to the database (and reflected in the UI when you reload). Only triggers that pass the [envs filter](#envs-filter) are synced to the database on each instance.

## Delete triggers

The `triggers.yaml` file is synced into the database automatically, but **deletions work differently**:

* **Delete one trigger:** Removing a trigger from the YAML file does not delete it from the database.
  Delete it in the UI on each instance where it exists:
  * In the Pipeline triggers table (`/pipelines/[pipeline_uuid]/triggers`), click the trash can icon.
  * In **Mage Pro**, right-click the trigger row and click **Delete**.

* **Delete selected triggers in Mage Pro:** On the global Triggers page, select one or more rows on
  the current page, open **Actions**, and choose **Delete selected (N)**. Selection applies only
  to the visible page and does not carry across pagination.

* **Delete matching triggers in Mage Pro:** Apply at least one search term or filter on the global
  Triggers page, then open **Actions** and choose **Delete matching triggers**. This action is
  unavailable without a search term or filter. Mage previews the matching count across all pages
  and requires you to type `DELETE` exactly before deletion. If the set of matching triggers
  changes after the preview, Mage stops the deletion so you can review the filters again. Triggers
  outside the current search and filters are not deleted.

The Mage Pro bulk deletion actions require Editor access. Each request can delete up to 100 triggers
and cancel up to 100 associated active runs; refine the selection or filters, or wait for active runs
to finish, if either limit is exceeded. It removes each trigger's database record and its definition
from the pipeline's `triggers.yaml` file. Mage marks observed `initial` and `running` pipeline-run and
block-run records as cancelled and attempts to stop their executors. Executor cancellation is best
effort, so a worker or a run created concurrently by another scheduler process may continue after the
trigger is deleted. Run and block-run history is preserved, and completed, failed, and
already-cancelled runs are unchanged. After Mage commits the trigger's inactive state, a cancellation
or final deletion failure leaves the trigger and its YAML definition inactive so you can retry. If an
[envs filter](#envs-filter) previously created the trigger on other instances, delete it manually on
those instances as well; sync does not remove triggers from their databases.

As a reversible alternative, set the trigger's `status` to `inactive` in `triggers.yaml`. Status
changes are automatically synced to the database and UI.


# Query trigger metadata from database
Source: https://docs.mage.ai/orchestration/triggers/database-query



Pipeline triggres are stored in `pipeline_schedule` table in the database. If you want to query
the trigger metadata, you can either directly query the data from database or use Python code
to fetch `PipelineSchedule` models.

## Query from DB

Here is the query from Postgres. You can adapt it to the syntax of other databases when needed.

```sql theme={"system"}
SELECT
    *
FROM pipeline_schedule
WHERE pipeline_uuid = 'example_pipeline' -- Filter by pipeline_uuid
AND status = 'ACTIVE' -- Filter by status
AND repo_path = '/home/src/default_repo' -- Filter by repo path
```

## Query DB models via Python

```python theme={"system"}
from mage_ai.orchestration.db import db_connection
from mage_ai.orchestration.db.models.schedules import PipelineSchedule, PipelineRun
from mage_ai.data_preparation.models.triggers import ScheduleStatus
from mage_ai.orchestration.pipeline_scheduler import PipelineScheduler

db_connection.start_session()

pipeline_schedules = PipelineSchedule.query.filter(
    # Filter by pipeline uuid
    PipelineSchedule.pipeline_uuid == 'example_pipeline',
    # Filter by trigger name
    PipelineSchedule.name == 'trigger_name',
)

# Get pipeline schedule count
print(f'Number of triggers: {pipeline_schedules.count()}')

# Print the detailed info
print([p.to_dict() for p in pipeline_schedules.all()])

# Update trigger status to ACTIVE
for p in pipeline_schedules:
    p.update(status=ScheduleStatus.ACTIVE)

# Update trigger status to INACTIVE
for p in pipeline_schedules:
    p.update(status=ScheduleStatus.INACTIVE)

# Cancel pipeline runs associated with the triggers
pipeline_schedule_ids = [s.id for s in pipeline_schedules]
pipeline_runs_to_cancel = PipelineRun.query.filter(
    PipelineRun.pipeline_schedule_id.in_(pipeline_schedule_ids)
).filter(
    PipelineRun.status.in_([
        PipelineRun.PipelineRunStatus.INITIAL,
        PipelineRun.PipelineRunStatus.RUNNING,
    ])
)
for pipeline_run in pipeline_runs_to_cancel:
    PipelineScheduler(pipeline_run).stop()

# Delete triggers
for p in pipeline_schedules:
    p.delete()
```


# Schedules
Source: https://docs.mage.ai/orchestration/triggers/schedule-pipelines

Schedule pipelines to run periodically

![Schedule pipelines to run periodically](https://media.tenor.com/lyU9j2EckEUAAAAd/running-out-of-time-no-time.gif)

You can schedule your pipeline to run on these intervals:

* Once:  The pipeline will run only once
* Hourly: The pipeline will run at the beginning of every hour (i.e 0100, 0200, 0300, etc)
* Daily: The pipeline will run at 0000 UTC / GMT every day
* Weekly: The pipeline will run at 0000 UTC / GMT the first day (Monday) of every week
* Monthly: The pipeline will run at 0000 UTC / GMT on the first day of every month
* Always on: The pipeline will be run as soon as the previous pipeline run is finished
* Custom schedule using
  [CRON Expression](https://en.wikipedia.org/wiki/Cron#CRON_expression)
* Every minute using CRON expression: `* * * * *`
* *Streaming pipeline*: coming soon
  Learn more about how to
  [schedule pipelines](/design/data-pipeline-management#create-trigger).

## Extra runtime variables from pipeline run

Default runtime variables from pipeline run:

| Key | Description | Example |
| - | - | - |
| `env` | The value is always `prod` in pipeline runs. The value is `dev` when running blocks in notebook. | `prod` |
| `event` | If the pipeline is triggered by event, the event variable contains the event payload. | `{"key1": "value1"}` |
| `execution_date` | A datetime object that the pipeline is executed at. | `86400` |
| `execution_partition` | An automatically formatted partition of the pipeline run using the execution date. | `10/20231225T122520` |
| `pipeline_run_id` | The id of the pipeline run. | `10` |
| `trigger_name` | The trigger name of the pipeline run. | `test run 1` |

If your pipeline run belongs to a trigger that is scheduled, then the following extra
variables are available in your Python block’s keyword arguments (e.g. `kwargs`).

| Key | Description | Example |
| - | - | - |
| `interval_start_datetime` | The `datetime` when the pipeline run is scheduled for. | `datetime.datetime(2023, 7, 23, 7, 0, 0, 0)` |
| `interval_end_datetime` | The `datetime` when the next pipeline run is scheduled for. | `datetime.datetime(2023, 7, 24, 7, 0, 0, 0)` |
| `interval_seconds` | The number of seconds between the current pipeline run and the next pipeline run. | `86400` |
| `interval_start_datetime_previous` | The `datetime` when the previous pipeline run was scheduled for. | `datetime.datetime(2023, 7, 22, 7, 0, 0, 0)` |

### SQL block

If you’re using a SQL block, here is an example of how you can access these variables:

```sql theme={"system"}
SELECT
    '{{ interval_start_datetime }}' AS interval_start_datetime
    , '{{ interval_end_datetime }}' AS interval_end_datetime
    , '{{ interval_seconds }}' AS interval_seconds
    , '{{ interval_start_datetime_previous }}' AS interval_start_datetime_previous
```


# Trigger pipeline from a block
Source: https://docs.mage.ai/orchestration/triggers/trigger-pipeline

You can trigger another pipeline from a block within a different pipeline.

## How to trigger a pipeline

There are 2 ways you can trigger a pipeline from code:

### Block template

Add a [data loader](/development/blocks/data_loaders/templates#trigger-pipeline)
or [data exporter](/development/blocks/data_exporters/templates#trigger-pipeline)
block template named `Orchestration → Trigger pipeline`.

### Custom code

Add a block and use the following code example within the block’s function:

```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline


trigger_pipeline(
    'pipeline_uuid',
    variables={},
    check_status=False,
    error_on_failure=False,
    poll_interval=60,
    poll_timeout=None,
    schedule_name=None,  # Enter a unique name to create a new trigger each time
    verbose=True,
)
```

## Parameters

| Parameter name | Description | Default | Sample | Required |
| - | - | - | - | - |
| `pipeline_uuid` | The UUID of the pipeline to trigger. | | `example_pipeline` | ✅ |
| `variables` | Runtime variables for the pipeline. | `{}` | `{ 'env': 'dev' }` | |
| `check_status` | Poll and check the status of the triggered pipeline. | `False` | `False` or `True` | |
| `error_on_failure` | Raise an exception if the triggered pipeline fails. | `False` | `False` or `True` | |
| `poll_interval` | Check the status of triggered pipeline every N seconds. | `60` | `300` | |
| `poll_timeout` | After N seconds have elapsed, raise an exception if the triggered pipeline hasn’t finished running successful. | `None` | `3600` | |
| `schedule_name` | Specify the trigger name. If the trigger doesn't exist, it'll get created with the trigger name | `None` | `trigger_name` | |
| `verbose` | Print status of triggered pipeline run. | `True` | `False` or `True` | |


# Triggering pipeline via API request
Source: https://docs.mage.ai/orchestration/triggers/trigger-pipeline-api

You can trigger a pipeline by making an API request.

## Prerequisites

1. [Create a pipeline](/about/features#data-pipeline-management)
2. [Create a trigger](/design/data-pipeline-management#create-trigger)

***

## Configuring trigger type

When you create a new trigger or edit an existing one, choose the option `API`
under trigger type.

***

## API endpoint

The endpoint to use when making an API request will depend on the domain you’re
running Mage and the trigger’s unique ID.

For example, if you’re running Mage on localhost with port 6789 and the trigger
you just created has an ID of 3, then the API endpoint you’ll use is:

```
POST http://localhost:6789/api/pipeline_schedules/3/pipeline_runs/abc123
```

***

## Request payload

You can optionally include runtime variables in your request payload. These
runtime variables are accessible from within each pipeline block.

### Example

```json theme={"system"}
{
  "pipeline_run": {
    "variables": {
      "env": "staging",
      "schema": "public"
    }
  }
}
```

Or

```json theme={"system"}
{
  "env": "staging",
  "schema": "public"
}
```

When the pipeline is triggered by this API request, each block in the pipeline
can access all the values defined in the variables object (e.g. `env`, `schema`)
within the request payload (in addition to the global variables defined within
the pipeline itself).

To use those runtime variables, use the variable name as a key when accessing
the keyword arguments dictionary (`kwargs`) in the decorated function of your
block. For example:

```python theme={"system"}
@data_loader
def load_data(**kwargs) -> DataFrame:
    env = kwargs['env']
    schema = kwargs['schema']

    # ...other code
```

You can include as many key value pairs within the variables object in the
request payload as you want.

***

## Sample cURL command

Here is a sample cURL command using the examples listed above:

```bash theme={"system"}
curl -X POST http://localhost:6789/api/pipeline_schedules/3/pipeline_runs/abc123 \
  --header 'Content-Type: application/json' \
  --data '
{
  "pipeline_run": {
    "variables": {
      "env": "staging",
      "schema": "public"
    }
  }
}'
```


# Pipeline Trigger Overview
Source: https://docs.mage.ai/orchestration/triggers/triggering-pipelines

Learn how to configure and manage pipeline triggers in Mage

## Overview

Pipeline triggers are automated scheduling mechanisms that determine **when and how data pipelines should run** in Mage. A single pipeline can have multiple triggers, each with different configurations for various use cases like production runs, testing, or data backfills.

### Key Benefits of Pipeline Triggers:

* **Automated execution** - Run pipelines without manual intervention
* **Flexible scheduling** - Support for cron expressions and API calls
* **Environment management** - Different trigger configurations for dev, staging, and production
* **Resource optimization** - Control when pipelines consume compute resources
* **Monitoring and SLA** - Built-in performance tracking and alerting

> [!NOTE]
>
> **Pipeline Schedule** and **Trigger** are the same thing in Mage. You may see either term used throughout the product.

## Trigger Types

> [!NOTE]
>
> New trigger creation supports Schedule and API triggers. Event triggers are deprecated and may
> only appear for existing legacy configurations.

### Schedule triggers

Schedule triggers are the most common type, designed for **recurring data pipeline execution** at predetermined intervals. They're perfect for ETL processes, data updates, and regular analytics jobs.

Available scheduling frequencies include:

* **Run exactly once** - Run one time when the schedule starts, then disable the trigger
* **Hourly** - Run at minute 0 of each hour in UTC
* **Daily** - Run once per day at 00:00 UTC
* **Weekly** - Run every Monday at 00:00 UTC
* **Monthly** - Run on the first day of each month at 00:00 UTC
* **Always on** - Keep streaming pipelines running continuously by starting a new run when the previous run completes
* **Custom (cron syntax)** - Use cron expressions for exact clock times and advanced scheduling patterns

> [!NOTE]
>
> **Custom frequency** allows you to use cron syntax for advanced scheduling patterns. This gives you complete control over when your pipeline runs.

#### Common cron examples

Here are some common cron patterns you can use with custom frequency:

```bash theme={"system"}
# Every 15 minutes
*/15 * * * *

# Every weekday at 9 AM
0 9 * * 1-5

# Every day at 2:30 PM
30 14 * * *

# Every Sunday at midnight
0 0 * * 0

# Every 1st of the month at 6 AM
0 6 1 * *

# Every 5 minutes during business hours (9 AM - 5 PM, weekdays)
*/5 9-17 * * 1-5
```

**Cron expression format:** `[minute] [hour] [day(month)] [month] [day(week)]`

> [!WARNING]
>
> **Timezone behavior:** If you have the `display_local_timezone` setting enabled, the local cron expression will match your local timezone only if the minute and hour values are single values without any special characters (comma, hyphen, or slash). You can still use cron expressions with special characters for the minute/hour values, but they will be based in UTC time.

Use online cron expression generators or cron documentation to create more complex scheduling patterns that fit your specific needs.

### API triggers

API triggers enable **on-demand pipeline execution** through HTTP requests. They're ideal for webhook integrations, external system triggers, and manual pipeline execution from external applications. For more details, refer to the [API Trigger documentation](/orchestration/triggers/trigger-pipeline-api).

**Use cases for API triggers:**

* Webhook integrations with external services
* Manual pipeline execution from external applications
* Event-driven data processing
* Integration with CI/CD pipelines

You can make a POST request to an endpoint provided in the UI when creating or
editing a trigger. The endpoint depends on the Mage domain and the trigger's unique ID.

**API endpoint:**

```bash theme={"system"}
POST {MAGE_HOST}/api/pipeline_schedules/3/pipeline_runs/abc123
```

**Request payload:**
Runtime variables can optionally be included in the request payload and are accessible within each pipeline block.

```json theme={"system"}
{
  "pipeline_run": {
    "variables": {
      "env": "staging",
      "schema": "public"
    }
  }
}
```

### Deprecated Event triggers

> [!WARNING]
>
> Event triggers are deprecated for new trigger creation. Use API triggers for new event-driven
> and webhook-style workflows. Existing legacy event triggers may still appear in older projects.

## How to Create and Manage Pipeline Triggers

### Creating a New Trigger

To create a new trigger for your pipeline:

1. Navigate to your pipeline in the Mage UI
2. Click the **Triggers** tab in the pipeline view
   * Direct URL: `/pipelines/[pipeline_uuid]/triggers`
3. Click **+ New trigger** button
4. Select the trigger type (Schedule or API)
5. Configure the trigger settings based on the type you selected
6. Click **Create trigger** to create the trigger

The new trigger form opens as a draft. Mage does not create the trigger until you click
**Create trigger**, and it warns you before leaving the page if the draft has unsaved changes.

> [!NOTE]
>
> **Trigger storage**: Triggers created via the UI are stored in the database. For environment synchronization and version control, you can [save triggers in code](/orchestration/triggers/configure-triggers-in-code) to sync them across different environments.

#### Trigger configuration fields

When creating or editing a trigger, you'll see the following configuration fields:

**Basic settings:**

* **Name** - A descriptive name for your trigger. Names are trimmed before validation, must be unique within the pipeline, must include at least one letter or number, and cannot exceed 255 characters.
* **Description** - Optional description to explain the trigger's purpose
* **Tags** - Add or search tags to organize and categorize your triggers. Each tag name can contain up to 70 characters.

**Schedule-specific fields:**

* **Frequency** - Choose from preset options (Hourly, Daily, Weekly, Monthly, Once, Always on) or select Custom for CRON syntax when you need more precise timing
* **Schedule start** - Choose **Start at next Frequency time** to let the trigger run at the next scheduled **Frequency** time, or **Do not start before date/time** to set the earliest allowed start. That date/time is not the run time; **Frequency** still controls when runs happen. **Schedule start** only controls when that trigger plan is allowed to begin.
* **Schedule load guidance** - For `@once` triggers with **Do not start before date/time**, Mage may show load or quiet-hour guidance for the selected start time. This is planning guidance only; use it to decide whether the selected start time is acceptable. If overlap is high, use suggested lower-load times, or keep your selected time if overlap is acceptable.
* **Enable landing time** - Instead of scheduling starts at a specific time, schedule runs so the pipeline can complete by a specified time

**Additional options:**

* **Runtime variables** - Override pipeline variables for this specific trigger
* **Run settings** - Configure timeouts, SLA, and failure handling
* **Advanced settings** - Additional trigger-specific configurations

### Viewing and Monitoring Triggers

#### Pipeline-Specific Trigger Management

To view all triggers for a specific pipeline:

1. Navigate to your pipeline in the Mage UI
2. Click the **Triggers** tab
   * Direct URL: `/pipelines/[pipeline_uuid]/triggers`
3. You'll see a comprehensive table listing all configured triggers with the following information:

Frequency, status, tag, and type filters are preserved when you navigate away and return to this
pipeline's Triggers page during the same browser session. Saved filters are scoped to the pipeline.
They are removed when you select **Clear** and are not restored in a new browser session unless they
are part of the URL you open.
Applied filters appear in a band below the toolbar. Select an individual chip to remove one value,
or select **Clear** to remove all applied filters while preserving unrelated list state.

#### Project-Wide Trigger Dashboard

To view all triggers across your entire project:

1. Go to the Mage homepage
2. Click **Triggers** in the left navigation menu
   * Direct URL: `/triggers`
3. You'll see all triggers from all pipelines in your project

UUID, type, and status filters are preserved when you navigate away and return during the same
browser session. Selecting **Clear** removes the saved filters. Free-text search is cleared when you
leave the page. Saved filters and search text are not restored in a new browser session unless they
are part of the URL you open.
On the List view, applied filters appear in a band below the toolbar. Select an individual chip to
remove one value, or select **Clear** to remove all applied filters while preserving unrelated list
state.

**Trigger list columns:**

* **Active** - Toggle switch showing if the trigger is enabled/disabled
* **Logs** - Icon to access trigger execution logs
* **Name** - The trigger name (clickable link with arrow icon for code-defined triggers)
* **Type** - Trigger type (Schedule, API, or deprecated legacy Event)
* **Frequency** - How often the trigger runs (e.g., @once, @daily)
* **Status** - Current status of the trigger
* **Next** - Scheduled time for the next execution (with info icon for details)
* **Runs** - Total number of pipeline runs initiated by this trigger
* **Tags** - Associated tags for organization
* **Created** - Timestamp when the trigger was created (with info icon for details)

You can click on the logs icon to view detailed execution history and logs for each trigger.

#### Day calendar

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=trigger-calendar)

Open **Triggers → Calendar** to review a day's planned starts and inspect their execution
status. Calendar is the default tab; **List** and **Load** open the existing trigger and load
views.

##### Pick a day and filter starts

1. Select a date, type `YYYY-MM-DD`, or choose **Today**.
2. Search by pipeline or trigger name. Use **All statuses** to filter by execution status,
   or select the succeeded or failed summary count.
3. Open **Calendar options** (the three-dot menu) to search for a display timezone or filter
   by an existing trigger tag. Choose **All trigger tags** to remove the tag filter.
4. Select **Refresh calendar** to update the selected day. Today's view also refreshes
   periodically.

Rows represent hours and horizontal positions represent minutes. Hover over the grid for
an exact time. Entries mark scheduled starts; their width does not show run duration.
Overlapping entries form groups: **+N** counts additional starts. Tag colors identify
categories, while separate status badges keep failures visible inside groups. A branch
icon and count indicate verified downstream runs when available.

Changing the display timezone changes the displayed day and times, not the trigger's UTC
schedule. The default follows your local-time display preference; an explicit calendar
selection takes precedence. Daylight-saving transitions can add or remove hour rows.

##### Inspect a scheduled start

Select an entry to open its details. For a group, choose an occurrence from the list;
**Back** returns to that group. Different triggers remain separate occurrences even when
they belong to the same pipeline.

Open **Timing details** to see the interval or cron, UTC schedule timezone, display timezone,
trigger ID, and available execution start and end times. Select the pipeline name, trigger
name, or **run** button to open its page in a new tab. The panel toggle beside **Refresh
calendar** hides or reopens details while retaining your selection.

**Unconfirmed** means a start is due but no execution is confidently matched; it does not
mean failure. **Unknown** means execution data cannot be verified. **All statuses** also
includes waiting, unconfirmed, and unknown states that have no individual filter option.
A downstream failure does not change the selected execution's own status.

##### Trace connected runs

For a matched execution, review the downstream summary, then select **lineage ↗** to open
its graph. Select a failed or blocked count to focus an affected execution directly.

Pan, zoom, or fit the graph; search for a pipeline and choose a matching run, or select a
node to inspect it. **View run** opens the execution in a new tab. Close the modal to return
to the same calendar selection. The modal keeps its opening snapshot; close and reopen it
to see refreshed relationships.

Lineage shows recorded execution links, including serial and parallel branches, with each
run counted once. It does not predict the next pipeline after success or dependencies for
unstarted schedules. Downstream runs are not added as calendar starts unless independently
scheduled. Results are bounded and permission-filtered: **Some connections unavailable**
means verified counts are partial. **Execution lineage unavailable** offers a retry action;
neither state establishes that no dependencies exist.

> [!NOTE]
>
> The calendar plots enabled, recurring time triggers using current settings. Past dates do
> not preserve historical schedule configurations. Disabled, run-once, always-on, landing-time,
> and streaming schedules are omitted. Known manual, API, backfill, and dependency-triggered
> runs are excluded from scheduled-run matching; older runs may lack origin information.
> Unsupported schedules, missing metadata, unavailable data, or result limits can make the
> calendar incomplete. An alert appears even when some starts remain visible; an incomplete
> empty day is identified separately. Summary counts describe the records shown.

**Enabling/Disabling triggers:**

* **From trigger list**: Use the toggle switch in the "Active" column to enable or disable a trigger
* **From trigger detail page**: Navigate to `/pipelines/[pipeline_uuid]/triggers/[trigger_id]` and use the "Disable trigger" button
* **Disable all triggers in Mage Pro**: On the global Triggers page, the current search and filters
  define the scope; only matching active triggers are disabled
* **Disabled triggers**: Will not run automatically but remain configured for future use
* **Enabled triggers**: Will run according to their configured schedule, API endpoint, or legacy event conditions

**Testing triggers:**

* **Run once button**: Click the "Run\@once" button in the trigger list or detail page to manually trigger a pipeline run
* **Testing purpose**: Use this to test your trigger configuration before enabling it for automatic execution
* **Immediate execution**: The pipeline will start running immediately when you click the button

**Backfill triggers:**

* **Historical data processing**: Create [backfill triggers](/orchestration/backfills/overview) to process historical data for a specific date range
* **One-time execution**: Backfill triggers run once and then disable themselves after completion
* **Data gap filling**: Use backfills to fill in missing data or reprocess data with updated logic

**Deleting triggers:**

* **Mage OSS**: Click the delete (trash) icon in the Actions column to delete a trigger
* **Mage Pro**: Right-click a trigger row to open its context menu. Select **Copy URL** to copy the
  trigger's direct URL, or select **Delete** and confirm **Yes** to delete the trigger.

## Advanced Trigger Configuration

### Run Settings and Performance Optimization

#### Timeout configuration

* **Set a timeout for each run of this trigger (optional)**
  * Configure a timeout in seconds for pipeline runs initiated by this trigger
  * If a run exceeds the timeout, it will be marked with the specified timeout status
* **Status for runs that exceed the timeout (default: failed)**
  * Choose how to handle runs that exceed the configured timeout
  * Options typically include: failed, cancelled, or warning

#### SLA configuration

* **Configure trigger SLA**
  * Set up Service Level Agreement monitoring for your trigger
  * Define expected completion time with configurable time units (minutes, hours, days)
  * Helps monitor trigger performance against business requirements

### Additional trigger settings

* **Override runtime variables**
  * You can override the [runtime variables](/getting-started/runtime-variable) defined in the pipeline to use different values for different triggers.
  * This option only appears if you have runtime variables defined in the pipeline.
  * Use **Form** mode to add, edit, or delete runtime variables one at a time.
  * Switch to **JSON** mode to edit the full runtime variables object in a code editor. Click **Format JSON** to reformat valid JSON before saving.
  * Runtime variable JSON must be an object. Variable names cannot contain whitespace, and reserved Mage variables cannot be included.
  * **Benefits**: Allows you to run the same pipeline with different configurations for different use cases (e.g., different data sources, environments, or processing parameters) without creating separate pipelines.
* **Keep running pipeline even if blocks fail**
  * When enabled, the pipeline continues running even if individual blocks fail
  * The pipeline will continue running blocks that are not dependent on the failed blocks until completion
* **(Schedule triggers only) Skip run if previous run still in progress**
  * When enabled, the scheduler will not start a new run if a previous run for this trigger is still in progress.
* **(Schedule triggers only)** \[!NOTE] Available starting in v0.9.62 **Create initial pipeline run if start date is before current execution period**
  * When enabled, this creates an initial pipeline run if the schedule's execution date is after its start datetime but before the current time.
  * **Example**: If current time is `1/4/24 12:00 UTC` and you create a daily trigger with start date `1/3/24 12:00 UTC`, a pipeline run with execution date `2024-01-04 00:00` is immediately created.

## Frequently Asked Questions

### What is a trigger in Mage?

A trigger is a set of instructions that determine when or how a pipeline should run. New trigger creation supports Schedule and API triggers. Event triggers are deprecated and may only appear for existing legacy configurations.

* **Schedule**: Runs pipelines on a set interval (e.g., hourly, daily, weekly, monthly, or custom cron)
* **API**: Runs pipelines when a specific API endpoint is called, optionally with runtime variables

### How do I create a trigger for my pipeline?

You can create triggers via the Mage UI or by defining them in a `triggers.yaml` file within your pipeline folder. In the UI, go to the pipeline editor, select "Triggers," and click "New trigger." In code, add your trigger configuration to `pipelines/[pipeline_uuid]/triggers.yaml`.

### Can I configure triggers in code and in the UI? How are changes synced?

Yes, you can configure triggers in both the UI and code. Changes made in the UI are synced to the YAML file, and vice versa. However, deleting a trigger from the YAML file does not remove it from the database; you must delete it in the UI.

### Why does my daily trigger run at 2AM instead of my specified time?

By default, @daily triggers run at midnight UTC. If you want your pipeline to run at a specific time, use the "custom" frequency option and enter a cron expression that matches your desired schedule.

### What happens if I delete a pipeline—are its triggers deleted too?

No, deleting a pipeline does not automatically delete its associated triggers. You need to manually remove triggers in the UI to avoid orphaned triggers.

### How do I prevent a pipeline from running if a previous run is still in progress?

There is a trigger setting called "Skip run if previous run still in progress." Enabling this will prevent the scheduler from starting a new run if the previous one is still active.

### Can I allow a pipeline to continue even if some blocks fail?

Yes, you can enable the "Allow blocks to fail" setting in your trigger configuration. This allows the pipeline to continue running blocks that are not dependent on failed blocks.

### How do I trigger a pipeline via API?

You can set up an API-type trigger, which allows you to start a pipeline by making a POST request to a specific endpoint. You can also pass runtime variables in your request payload.


# Pipeline filtering and tagging
Source: https://docs.mage.ai/pipelines/pipeline-tagging

How to organize, filter, group, and tag pipelines in Mage.

![Mario pipe](https://github.com/mage-ai/mage-ai/assets/59450879/750168bc-9a42-4083-badd-adc120b5f97b)

A [pipeline](/design/core-abstractions#pipeline) contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.

In the Mage GUI, pipelines can be filtered and grouped to make it easier to find the pipeline you want to run. You can also tag pipelines to make it easier to find pipelines that are related to each other.

![Filtering pipelines](https://user-images.githubusercontent.com/1066980/255380712-3d86d4ac-255b-4894-8a5f-45c10b7c68c8.gif)

In Mage Pro, you can apply or remove tags from the pipeline list by right clicking the desired pipeline and selecting "Update tags." In the modal, type a tag name and press Enter, select an existing matching tag suggestion, or leave a valid typed tag in the field and click "Save." You can add multiple tags this way and remove selected tags as needed.

In Mage Pro, you can also manage tags from a pipeline's settings page. In the **Tags** field, type a tag name and press Enter. You can add multiple tags this way or select existing matching tag suggestions, then click **Save pipeline settings**. Tag names are trimmed, must be non-empty, and can contain up to 70 characters.

Mage will populate a list of existing tags for consistency. Once tags have been applied, you can group/filter your pipelines by tag on the "Pipeline" page.

Status, tag, and type filters appear in a band below the toolbar on the Pipelines page. Select an
individual chip to remove one value, or select **Clear** to remove all applied filters while
preserving unrelated list state such as search, grouping, sorting, and page size.

Status, tag, and type filters are preserved when you navigate away and return during the same browser
session. Free-text search is cleared when you leave the Pipelines page. Saved filters and search text
are not restored in a new browser session unless they are part of the URL you open. Selecting
**Clear** also removes the saved filters for the current session.


# Deployment architecture overview
Source: https://docs.mage.ai/pro-architecture/architecture-overview

Deploy Mage Pro in any environment

This document provides technical details on the **four deployment architectures** available for Mage Pro. These options help technical teams identify which deployment best suits their use case, data governance policies, and operational constraints.

***

### Deployment Architecture Overview

Mage Pro’s platform architecture consists of two primary components:

1. **Control Plane**
   * Handles deployment of Mage Pro clusters
   * Manages cluster lifecycle: version upgrades, security patches, monitoring of cluster health
   * Provides centralized governance and operational visibility

2. **Data Plane (Mage Pro Cluster)**
   * The **actual Mage Pro cluster** running inside Mage cloud or your environment
   * Handles pipeline management and orchestration
   * Executes pipelines and manages compute resources
   * Provides secret management and metadata services
   * Hosts the Mage Pro UI and APIs for day-to-day operations

***

### Deployment Options

#### 1. **SaaS (Cloud-Hosted by Mage)**

* **Control Plane & Data Plane**: Fully managed by Mage in a secure multi-tenant cloud environment.
* **Best for**: Teams who want the fastest time-to-value and minimal operational overhead.
* **Key benefits**: Automatic upgrades, lifecycle management, managed infrastructure, secure isolation, and 24/7 monitoring.
* **Considerations**: Data leaves your environment (not suitable for strict data residency requirements).

***

#### 2. **Hybrid Cloud**

* **Control Plane**: Hosted by Mage (SaaS).
* **Data Plane**: Runs inside your private cloud or on-prem infrastructure.
* **Best for**: Organizations that want Mage to manage cluster lifecycle while ensuring pipelines, secrets, and workloads stay in their own environment.
* **Key benefits**: Compliance with data residency/security policies, reduced operational burden.
* **Considerations**: Requires secure network connectivity (e.g., VPC peering, VPN, or private link).

***

#### 3. **Private Cloud (Customer-Managed)**

* **Control Plane & Data Plane**: Fully deployed in your own cloud account (AWS, Azure, GCP, etc.).
* **Best for**: Enterprises requiring strict compliance, tenant isolation, and cloud sovereignty.
* **Key benefits**: Data never leaves your environment, full Mage Pro cluster (orchestration, secrets, pipelines, UI) runs inside your VPC, integrates with your cloud security controls (IAM, Key Vaults, etc.).
* **Considerations**: Your team is responsible for lifecycle management, upgrades, and monitoring.

***

#### 4. **On-Premises**

* **Control Plane & Data Plane**: Installed in your on-prem infrastructure (e.g., Kubernetes, Docker, bare-metal).
* **Best for**: Highly regulated industries or air-gapped environments where no external connectivity is allowed.
* **Key benefits**: Maximum control, full Mage Pro cluster deployed in your data center, compliance with strictest regulations, supports OpenShift and other enterprise Kubernetes distributions.
* **Considerations**: Highest operational overhead (infrastructure, lifecycle management, monitoring, scaling, DR handled by your team).

***

### Choosing the Right Architecture

* **Speed of deployment** → SaaS or Hybrid Cloud
* **Strict data residency/compliance** → Private Cloud or On-Prem
* **Operational simplicity** → SaaS
* **Air-gapped or high-security environments** → On-Prem


# Hybrid cloud deployment
Source: https://docs.mage.ai/pro-architecture/hybrid-cloud

Deploy the data plane in your cloud environment

Mage Pro’s hybrid deployment architecture splits the control plane and the data place components across environments. The control plane is maintained in Mage’s cloud environment while the data processed in the data plane never leaves your cloud infrastructure.

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/deployments/Hybrid%20Deployment%20\(1\).png)

### Technical Specifications

* **Control Plane**: Managed by Mage in secure cloud infrastructure
* **Data Plane**: Deployed in your AWS, GCP, or Azure environment
* **Connectivity**: Secure agent-based communication between planes
* **Resource Management**: Configurable compute profiles with auto-scaling
* **Pipeline Execution**: All data processing occurs within your network boundary

### Integration Requirements

* VPC/VNET connectivity configuration between control and data planes
* IAM roles or service accounts for resource provisioning
* Network security groups configured to allow necessary communication
* Container registry access for execution images


# On-prem deployment
Source: https://docs.mage.ai/pro-architecture/on-prem

Deploy securely from your own on-prem servers

Deploy the control plane and the data plane on your own servers for the most secure environment.

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/deployments/On-prem%20deployment%20\(1\).png)

## Deployment options

Mage Pro supports two modes of on-prem deployment. The Mage team will provide all necessary deliverables and installation support.

### 1. Kubernetes cluster

Run Mage Pro in your own on-prem Kubernetes cluster for production-grade orchestration and scalability. This includes Kubernetes distributions such as **Red Hat OpenShift**, Rancher, or other enterprise Kubernetes platforms.

**Deliverables from Mage**:

* Helm chart(s) pre-configured for Mage Pro (control plane, scheduler, and executor pods)
* Docker image (if your environment does not have internet access) **or** secure registry credentials to pull the image (if your cluster has network access)
* Configuration templates for networking, secrets, monitoring, and scaling
* Documentation and best practices for integrating with your Kubernetes environment

**Highlights**:

* Pod-level isolation for workspaces and jobs
* Leverage Kubernetes-native scaling, security, and monitoring
* Recommended for enterprise-grade, multi-team deployments

***

### 2. Docker

For environments without Kubernetes, Mage Pro can be deployed directly on Docker. This simplifies setup while still providing workspace/job isolation using containers.

**Deliverables from Mage**:

* Docker images for Mage Pro services
  * Air-gapped environments (no internet): Mage team provides image files directly
  * Connected environments: Mage team provides secure registry credentials to pull images
* Docker Compose file with all required configuration

**Highlights**:

* Quick and easy installation with minimal infrastructure requirements
* Run the Mage Pro control plane in a long-running Docker container
* Option to run pipeline jobs in separate, short-lived containers (for isolation) or locally within the same container process (Pro-only feature)
* Workspaces can also run in their own dedicated containers (Pro-only feature)
* Ideal for smaller teams or environments without Kubernetes

***

## License key

A valid **Mage Pro license key** is required for all on-prem deployments.

* The license key is provided when you purchase Mage Pro.
* It must be included in your deployment configuration (Kubernetes or Docker) before the server will start.
* License keys can be rotated at any time without downtime.

***

## Benefits

* **Maximum security**: All services and data stay within your on-prem environment.
* **Flexibility**: Choose between Kubernetes (e.g., OpenShift) or Docker depending on your infra maturity.
* **Consistency**: Same Mage Pro features and workspace management UI across deployment modes.


# Private cloud deployment
Source: https://docs.mage.ai/pro-architecture/private-cloud

Deploy securely in your private cloud environment

Private cloud deployments allow developers to have complete architectural control over the control plane and the data plane. The entire Mage Pro platform will run on your cloud environment.

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/deployments/Copy%20of%20Private%20Cloud%20\(1\).png)

### Technical Specifications

* **Deployment Platform**: Kubernetes (AKS, EKS, GKE) or OpenShift
* **Installation Method**: Helm charts or operator-based deployment
* **Resource Requirements**:
  * Minimum 2 vCPUs and 8GB RAM for control plane services
  * Elastic resources for data plane based on workload
  * Persistent storage for metadata and logs
* **Networking**: Internal service mesh for component communication
* **Monitoring**: Prometheus-compatible metrics endpoints
* **Backup**: Automated database backup with configurable retention

### Integration Requirements

* Kubernetes cluster with admin access for service deployment
* Persistent volume provisioning capability
* Load balancer or ingress controller for web UI access
* Container registry access for system images


# Managed cloud
Source: https://docs.mage.ai/pro-architecture/saas

Fully managed Mage Pro

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=managed-cloud)

Mage Pro SaaS is fully managed and deployed in the Mage cloud infrastructure. In this configuration, both the control and data planes are hosted and managed within Mage's cloud infrastructure:

![SaaS Deployment](https://raw.githubusercontent.com/mage-ai/assets/main/pro/deployments/SaaS%20Deployment%20Architecture%20\(1\).png)

### Technical Specifications

* **Deployment Method**: Zero touch deployment with automatic provisioning
* **Infrastructure**: Containerized environment with auto scaling in Kubernetes
* **Networking**: Secure TLS connections to data sources with IP allowlisting
* **Authentication**: SAML/OAuth2 support for SSO integration
* **Updates**: Automated version management with zero-downtime updates
* **Scalability**: Dynamic resource allocation based on workload patterns
* **Monitoring**: Internal monitoring tools, capable of integrating with external monitoring tools

For Google Identity Platform, Microsoft Active Directory, and [Okta](/production/authentication/okta) setup guidance, open **Settings > Access & security > Authentication**. To compare the setup guides for Mage's built-in and external secret managers, open **Settings > Access & security > Secrets management**. To create and manage built-in project secrets, open **Library** in the left navigation, select **Secrets**, and see the [Secrets guide](/development/variables/secrets).

### Allowlist IP addresses

Enhance your network security by configuring firewall rules or security groups to allow traffic only from Mage Pro's static IP addresses.
This prevents unauthorized access by restricting connections to trusted sources only.

In Mage, open **Settings > Access & security > Network access** to view and copy individual addresses or the complete regional list. The former `/onboarding/security` and `/settings/workspace/security` URLs redirect to this settings page.

Allowlist every address below when your Mage Pro workspace region is listed. Contact Mage before configuring your firewall if your region is not listed.

**United States**

```
100.21.110.88
34.213.159.59
35.155.191.84
35.80.236.114
44.231.28.12
52.34.99.83
52.37.240.164
52.38.195.57
52.42.154.61
54.185.105.221
54.190.141.196
```

**Europe**

```
13.43.35.64
3.11.158.8
3.8.69.248
51.24.28.212
```

**Australia**

```
13.236.104.140
3.26.253.87
54.153.177.143
54.66.123.121
```

If your region is not listed above or your firewall policy requires a single static IP address, contact Mage to discuss the available network configuration for your workspace.

### Integration Requirements

* Ensure network connectivity between your data sources and Mage's cloud environment
* Configure firewall rules to allow connection from Mage's IP ranges
* Set up authentication credentials for data source access
* Establish monitoring integrations if desired

### Regional deployment

Mage Pro’s SaaS is currently deployed in the following regions U.S. West, U.S. East, Canada, Europe, Asia, and Australia.

Please reach out to Mage if you would like to deploy Mage Pro from a specific region other than what’s listed above.


# Sign in with Google
Source: https://docs.mage.ai/production/authentication/google

Enable signing in with a Google account in Mage.

### Pre-requisites

First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).

### Enable Google sign in

> [!NOTE]
>
> Mage will request access for the following scopes:
>
> ```
> https://www.googleapis.com/auth/userinfo.email
> https://www.googleapis.com/auth/userinfo.profile
> ```

By default, users will not see the "Sign in with Google" option in their sign in page.

**1. Add Oauth application**

You will need to add Mage as an OAuth client in your Google Cloud Platform project.
For more instructions on how to do this, please see the [Google Cloud Platform documentation](https://developers.google.com/identity/protocols/oauth2/openid-connect#appsetup).

**2. Retrieve credentials**

Once you have created the credentials, you will need to get the application's client ID and client secret.
You will also need to add `http(s)://<your-mage-url>/oauth` as an authorized redirect URI.

**3. Set environment variables**

Set the following environment variables in your Mage setup.

| Name | Value |
| - | - |
| `GOOGLE_CLIENT_ID` | `<client id>` |
| `GOOGLE_CLIENT_SECRET` | `<client secret>` |

You should now see the "Sign in with Google" button when you start Mage and attempt to sign in.

![Sign in with Google](https://mage-ai.github.io/assets/production/authentication/sign-in-with-google.png)


# Lightweight Directory Access Protocol
Source: https://docs.mage.ai/production/authentication/ldap

Sign in and authenticate users using LDAP

If you want to enable LDAP authentication mode, Set the following environment variable name and value:

| Variable name | Value |
| - | - |
| `AUTHENTICATION_MODE` | `ldap` |

In this case, you will also need to configure the following variables:

| Variable name | Description | Example value |
| - | - | - |
| `LDAP_ADMIN_USERNAME` | The username of the owner user created on first run | `admin` |
| `LDAP_SERVER` | The host of the LDAP server | `ldap.example.com` |
| `LDAP_BIND_DN` | The `bind_dn` for the LDAP account that will be used to perform search operations | `cn=admin,dc=example,dc=com` |
| `LDAP_BIND_PASSWORD` | The password for the LDAP account that will be used to perform search operations | `password` |
| `LDAP_BASE_DN` | The base DN of the server | `dc=example,dc=com` |
| `LDAP_AUTHENTICATION_FILTER` | The authentication filter to use in template format using the variable `username` | `(&(\|(objectClass=inetOrgPerson)(objectClass=groupOfNames))(cn={username}))` |
| `LDAP_AUTHORIZATION_FILTER` | The authorization filter to use in template format using the variable `user_dn` | `(&(objectClass=groupOfNames)(cn=group)(member={user_dn}))` |
| `LDAP_GROUP_FIELD` | Name of user attribute to use to determine a user's LDAP group. | `memberOf` |
| `LDAP_ROLES_MAPPING` | Mapping of user LDAP group to Mage roles. Must be in JSON string format. | `'{"Admin": ["Admin"]}'` |
| `LDAP_DEFAULT_ACCESS` | Default access for a new user authenticated through LDAP (default user will have no role) | `Viewer`, `Editor`, `Admin` |
| `UPDATE_ROLES_ON_LOGIN` | Update roles on every login. Otherwise, the roles will only be set when the user logs in for the first time. | `1` |

Make sure to adjust the values of the variables to match your LDAP configuration.

When LDAP authentication is enabled, users will need to provide their LDAP credentials to log in to the system. Once authenticated, Mage will use the authorization filter to determine the user's permissions based on their LDAP group membership.


# Multi-factor authentication
Source: https://docs.mage.ai/production/authentication/mfa

Configure multi-factor authentication for Mage user accounts.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=mfa)

## Overview

Multi-factor authentication (MFA) adds a second verification step when a user signs in.
After MFA is enabled for an account, Mage requires the user's password or SSO sign in
and then a 6-digit code from an authenticator app before issuing a session.

Mage supports time-based one-time passwords (TOTP), which work with authenticator apps
such as Google Authenticator, Microsoft Authenticator, Authy, and 1Password.

> [!NOTE]
>
> MFA is configured per user account. Each user must enable MFA from their own account settings.

## Pre-requisites

Before configuring MFA, make sure:

* User authentication is enabled. See [User authentication](/production/authentication/overview).
* Your Mage deployment has run the latest database migrations.
* The Mage server has the MFA dependencies installed from `requirements.txt`, including `pyotp`
  and `qrcode`.
* The user has access to an authenticator app that can scan QR codes.

## Enable MFA

**1. Open MFA settings**

Sign in to Mage and go to **Settings** → **Access & security** → **MFA**.

**2. Start setup**

Click **Set up MFA**.

**3. Install or open an authenticator app**

Open an authenticator app that supports TOTP codes. Mage supports apps such as:

* Google Authenticator
* Microsoft Authenticator
* Authy
* 1Password

**4. Scan the QR code**

Click **Next**, then scan the QR code with your authenticator app.

The QR code adds a new Mage entry to the authenticator app. The app will start showing
rotating 6-digit verification codes for that Mage account.

**5. Verify the code**

Enter the current 6-digit code from the authenticator app and click **Next**.

After the code is accepted, Mage shows a confirmation step.

**6. Complete setup**

Click **Complete Setup** to close the setup flow and return to the MFA settings page.

The status changes to **Active** after MFA is enabled for your account.

## Sign in with MFA

After MFA is enabled, signing in has two steps:

1. Enter your email or username and password, or complete your configured SSO sign in.
2. Enter the 6-digit code from your authenticator app on the MFA verification page.

If the MFA verification session expires, return to the sign in page and start again.

> [!WARNING]
>
> After too many invalid MFA codes, Mage temporarily locks MFA verification for that user.
> Wait 5 minutes, then sign in again and enter a new code from the authenticator app.

## Disable MFA

To disable MFA for your account:

**1. Open MFA settings**

Sign in to Mage and go to **Settings** → **Access & security** → **MFA**.

**2. Choose disable**

Click **Disable MFA**.

**3. Confirm with a current code**

Enter the current 6-digit code from your authenticator app and confirm.

Mage deactivates the active MFA devices for your account after the code is verified.

## Troubleshooting

| Issue | What to check |
| - | - |
| The MFA page does not appear | Confirm that your Mage version includes MFA and that user authentication is enabled. |
| QR code generation fails | Confirm that `pyotp` and `qrcode` are installed in the Mage server environment and restart the server after installing dependencies. |
| The verification code is rejected | Enter the newest 6-digit code from the authenticator app. If codes continue to fail, check that the device running the authenticator app has the correct time. |
| The verification page says the session expired | Sign in again to create a new MFA challenge session. |
| Disable MFA fails | Enter the current 6-digit authenticator code for the account that is signed in. |


# Sign in with Microsoft Entra ID aka Active Directory
Source: https://docs.mage.ai/production/authentication/microsoft

Enable signing in with a Microsoft account in Mage.

### Pre-requisites

First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).

### Enable Microsoft sign in

There are two ways to enable Microsoft sign in. You can either register an application in your organization's
own Microsoft Entra ID, or use the default Mage application. We recommend registering your own application unless
there is a specific reason you cannot do so.

> **Register and use your own Mage application**
>
> You will need to register an application in your organization's Microsoft Entra ID.
>
> **1. Log into your Azure Portal**
>
> **2. Click into the Microsoft Entra ID service**
>
> ![Azure Active Directory](https://mage-ai.github.io/assets/production/authentication/ad.png)
>
> **3. Navigate to the tenant you want users to sign into**
>
> **4. Click App registrations on the left sidebar**
>
> ![App registrations](https://mage-ai.github.io/assets/production/authentication/ad-app-registrations.png)
>
> **5. Register a new application for Mage**
>
> Click the "New registration" button at the top of the page, and fill in the details for your application.
>
> ![New registration](https://mage-ai.github.io/assets/production/authentication/ad-new-registration.png)
>
> Make sure you add the following web redirect URI: `http(s)://<your-mage-url>/oauth` (for Mage open-source)
> or  `https://cluster.mage.ai/<your-cluster-base-path>/oauth?provider=active_directory` (for Mage Pro)
> or `https://<mage-host-url>/oauth?provider=active_directory` (for Mage self-hosted).
>
> **6. Create client secret for the application**
>
> Click into the application you created, and click "Certificates & secrets" in the left sidebar.
>
> ![Certificates & secrets](https://mage-ai.github.io/assets/production/authentication/ad-certificates.png)
>
> Copy the "Application (client) ID" and the "Directory (tenant) ID" values.
>
> ![Copy values](https://mage-ai.github.io/assets/production/authentication/ad-application-fields.png)
>
> Create a new client secret and copy the value.
>
> **7. Set environment variable for Mage**
>
> Once you have registered your application, you will need to set the following environment variables
> in your Mage setup.
>
> | Name | Value |
> | - | - |
> | `ACTIVE_DIRECTORY_DIRECTORY_ID` | `<tenant_id>` |
> | `ACTIVE_DIRECTORY_CLIENT_ID` | `<application_client_id>` |
> | `ACTIVE_DIRECTORY_CLIENT_SECRET` | `<application_client_secret>` |
>
> You should now see the "Sign in with Microsoft" button when you start Mage and attempt to sign in.
> You may need to restart your Mage server for the changes to take effect.
>
> ![Sign in with Microsoft](https://mage-ai.github.io/assets/production/authentication/sign-in-with-microsoft.png)

> **Use the default Mage application**
>
> You will need to get your organization's Tenant ID. You can find this in the
> Azure Portal:
>
> **1. Log into your Azure Portal**
>
> **2. Click into the Microsoft Entra ID service**
>
> ![Azure Active Directory](https://mage-ai.github.io/assets/production/authentication/ad.png)
>
> **3. Navigate to the directory you want users to sign into**
>
> **4. Click 'Properties' in the left sidebar**
>
> ![Active Directory Properties](https://mage-ai.github.io/assets/production/authentication/ad-properties.png)
>
> **5. Copy the value under Tenant ID**
>
> ![Active Directory Tenant ID](https://mage-ai.github.io/assets/production/authentication/ad-tenant-id.png)
>
> **6. Set environment variable for Mage**
>
> Once you have found your tenant ID, you will need to set the following environment variable
> in your Mage setup.
>
> | | |
> | - | - |
> | Name | `ACTIVE_DIRECTORY_DIRECTORY_ID` |
> | Value | `<tenant_id>` |
>
> You should now see the "Sign in with Microsoft" button when you start Mage and attempt to sign in.
> You may need to restart your Mage server for the changes to take effect.
>
> ![Sign in with Microsoft](https://mage-ai.github.io/assets/production/authentication/sign-in-with-microsoft.png)

### Configurable environment variables

To configure the environment for Microsoft Azure Cloud and Microsoft Azure Gov Cloud sign-in,
you need to set the following environment variables:

1. You can set the `AZURE_HOST` environment variable to one of the following based on your environment:

```
Public cloud host url: "https://login.microsoftonline.com"
China host url: "https://login.chinacloudapi.cn"
Germany host url: "https://login.microsoftonline.de"
Government host url: "https://login.microsoftonline.us"
```

2. You can set the `MICROSOFT_GRAPH_API_HOST` environment variable to one of the following basd on your environment:

```
Microsoft Graph global service: "https://graph.microsoft.com"
Microsoft Graph for US Government L4: "https://graph.microsoft.us"
Microsoft Graph for US Government L5 (DOD): "https://dod-graph.microsoft.us"
Microsoft Graph China operated by 21Vianet: "https://microsoftgraph.chinacloudapi.cn"
```

### Map AD role to Mage role

> [!NOTE]
>
> You will need to have created your own application and have app roles in order for this mapping to work.

You can map Active Directory roles to Mage roles by setting the `ACTIVE_DIRECTORY_ROLES_MAPPING` environment variable.
The value should be a JSON object with the AD role as the key and the name of the Mage role as the value. For example:

`export ACTIVE_DIRECTORY_ROLES_MAPPING='{"Mage.Edit": "Editor", "Mage.Admin": "Admin"}'`

You can find the AD role value in the App roles section of your application in the Azure Portal.

![App roles](https://mage-ai.github.io/assets/production/authentication/ad-app-role-value.png)

When a user signs in for the first time with their Microsoft account, Mage will use this mapping to automatically add
the appropriate roles to the user.


# Sign in with OIDC
Source: https://docs.mage.ai/production/authentication/oidc

Enable signing in with OIDC in Mage.

### Pre-requisites

First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).

### Enable OIDC sign in

By default, users will not see the "Sign in with OIDC" option in their sign in page.

**1. Add Oauth application**

You will need to add Mage as a client for your OIDC provider. The way to do this will
be different depending on your OIDC provider. The redirect URL should be `https://<your-mage-url>/oauth`.

**2. Retrieve credentials**

Once you have registered Mage, you will need to retrieve the client ID, client secret,
and the OIDC discovery URL for your provider.

**3. Set environment variables**

Set the following environment variables in your Mage setup.

| Name | Value |
| - | - |
| `OIDC_DISCOVERY_URL` | `<oidc domain url>` |
| `OIDC_CLIENT_ID` | `<client id>` |
| `OIDC_CLIENT_SECRET` | `<client secret>` |

You should now see a "Sign in with OIDC" button when you start Mage and attempt to sign in.

Mage will fetch the following OIDC endpoints from the discovery URL:

* authorization\_endpoint
* token\_endpoint
* userinfo\_endpoint


# Sign in with Okta
Source: https://docs.mage.ai/production/authentication/okta

Enable signing in with Okta in Mage.

### Pre-requisites

First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).

### Enable Okta sign in

By default, users will not see the "Sign in with Okta" option in their sign in page.

**1. Add Oauth application**

You will need to add Mage as an OAuth client in your Okta organization admin dashboard.
For more instructions on how to do this, please see the [Okta developer documentation](https://developer.okta.com/docs/guides/implement-oauth-for-okta/main/#create-an-oauth-2-0-app-in-okta).
Make sure you select the "Authorization Code" grant flow, and add `http(s)://<your-mage-url>/oauth` as a Sign-in redirect URI.

**2. Retrieve credentials**

Once you have created the credentials, you will need to get the application's client ID and client secret.
Make sure you also have your Okta domain URL.

**3. Set environment variables**

Set the following environment variables in your Mage setup.

| Name | Value |
| - | - |
| `OKTA_DOMAIN_URL` | `<okta domain url>` |
| `OKTA_CLIENT_ID` | `<client id>` |
| `OKTA_CLIENT_SECRET` | `<client secret>` |

You should now see the "Sign in with Okta" button when you start Mage and attempt to sign in.

![Sign in with Okta](https://mage-ai.github.io/assets/production/authentication/sign-in-with-okta.png)


# User authentication
Source: https://docs.mage.ai/production/authentication/overview

Create users, manage users, and require sign in to authenticate and use Mage.

![Gandalf](https://media.tenor.com/bIJa2uRURiQAAAAC/lord-of-the-rings-you-shall-not-pass.gif)

***

## Overview

Mage supports user authentication to secure access to the UI and APIs.
Authentication is required before creating, managing, or running pipelines.

* **Mage OSS**:
  * Versions **0.9.78 and above** → user authentication is **enabled by default**.
  * Versions **0.8.4 – 0.9.77** → user authentication can be turned on optionally.

* **Mage Pro**:
  * User authentication is **always enabled by default**.

## Configure user authentication

For Mage OSS versions **0.8.4 – 0.9.77**, you can enable authentication manually by setting the following environment variable:

| | |
| - | - |
| Name | `REQUIRE_USER_AUTHENTICATION` |
| Value | `1` |

For Mage OSS **0.9.78+** and **Mage Pro**, this setting is no longer required because authentication is enabled by default.

### Set access token expiration time

You can set the `MAGE_ACCESS_TOKEN_EXPIRY_TIME` environment variable to customize the expiration time
of a Mage access token. The value should be the length of time in seconds.

Defaults to 2592000 which is 30 days.

| | |
| - | - |
| Name | `MAGE_ACCESS_TOKEN_EXPIRY_TIME` |
| Value | `2592000` |

### Running Mage in Docker

If you’re running Mage using Docker, you can run Mage and set the environment variable
in the `docker run` command. Follow these [instructions](/getting-started/setup#environment-variables)
to learn how.

Set the environment variable using `-e REQUIRE_USER_AUTHENTICATION=1`.

For example:

```bash theme={"system"}
docker run -it -p 6789:6789 \
  -v $(pwd):/home/src \
  -e REQUIRE_USER_AUTHENTICATION=1 \
  mageai/mageai /app/run_app.sh mage start demo_project
```

### Running Mage without Docker

If you installed Mage using `pip`, `conda`, `poetry`, etc. and are running it using the `mage start`
command, then you need to set the environment variable on your workstation.

#### macOS

Run the following command to set environment variables:

```bash theme={"system"}
export REQUIRE_USER_AUTHENTICATION=1
```

#### Windows

Read Microsoft’s [instructions](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1)
on how to do this.

## Default owner user

When Mage starts, if there is no existing owner user, then a new user will be created with owner
permissions (e.g. all permissions).

The owner user can create, edit, and delete other users.

Here are the default credentials for the owner user to sign in with:

| Field | Value |
| - | - |
| Email | `admin@admin.com` |
| Password | `admin` |

> [!NOTE]
>
> Available in versions >= `0.9.72`

To customize the email, password and username of the default owner user, you can configure these environment variables:

* `DEFAULT_OWNER_EMAIL`
* `DEFAULT_OWNER_PASSWORD`
* `DEFAULT_OWNER_USERNAME`

## Sign in page

Before using Mage, users will have to sign in at `/sign-in`. If you go to a page that requires
authentication and you’re not signed in, you’ll be redirected to the sign in page.

![Sign in page](https://raw.githubusercontent.com/mage-ai/assets/main/users/sign-in.jpg)

## User management page

If you’re signed in as the owner user, you’ll have access to the user management page at
`/settings/workspace/users`.

On this page, you can:

1. Add new users
2. Edit existing users
3. Reset user passwords
4. Delete users

![Sign in page](https://raw.githubusercontent.com/mage-ai/assets/main/users/users-list.jpg)

## Roles and permissions

| Role | Permissions |
| - | - |
| Viewer | Can read. |
| Editor | Can do everything a viewer can, create, update, and delete. |
| Admin | Can do everything a viewer can, an editor can, and manage users who are viewers/editors (update usernames, emails, passwords, and make viewers editors or vice versa. |
| Owner | In addition to what admins can do, add and delete users. Manage all users, including admins/owners. Make other users any role, including admins/owners. |

## Authenticated requests

If you require user authentication by turning it on via the environment variable mentioned above,
then each API request the front-end client makes to the backend server will include an API key
and an OAuth2 token.

The API key and OAuth2 token is used to authenticate the incoming request. If the requester
doesn’t have the proper permissions, the backend server will respond with a `4XX` error code.

### API errors

If a page loads and no data is visible, it could either mean there is no data or the user
doesn’t have the proper permissions.

If there is no visible error message on the page, you can view the browser’s network tab and
inspect the API request response to view the full error details
(we’re working on showing better errors and making it easier to display them).

> [!TIP]
>
> Learn how to view the network requests in Chrome by following these
> [instructions](https://developer.chrome.com/docs/devtools/network/).

| Error code | Description |
| - | - |
| `401` | Expired OAuth token. |
| `401` | Invalid OAuth token. |
| `402` | Record is invalid. |
| `403` | Invalid API key. |
| `403` | Unauthorized access. |
| `404` | Record not found. |
| `500` | API resource error. |


# Password reset
Source: https://docs.mage.ai/production/authentication/password-reset

Use self-service password reset for Mage Pro accounts.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=password-reset)

Self-service password reset lets a user choose a new password without asking an
administrator. It applies to Mage accounts that have an email address and use a
Mage password. If your organization uses an external identity provider, reset
your password with that provider instead.

## Before you begin

Password reset is available in Mage Pro releases that include the feature.
Hosted Mage Pro deployments include the Mage-managed email infrastructure;
customers do not need to add email credentials or password-reset environment
variables to individual clusters.

To make the feature available, a Mage Pro administrator should deploy a Mage Pro
release that includes password reset.

## Reset your password

**1. Request a reset link**

On the Mage sign-in page, select **Forgot password?**, enter the email address on
your Mage account, and select **Send reset email**.

Mage shows the same accepted response whether or not the address belongs to an
account. This prevents the page from revealing which email addresses are
registered.

**2. Open the email**

Open the message and follow the reset link. Each link expires after one hour and
can be used only once. If a link has expired or has already been used, request a
new one.

**3. Set a new password**

Enter and confirm a password that satisfies the requirements shown on the page,
then select **Update password**. After the password changes, Mage invalidates all
existing sessions and all other outstanding reset links for the account. Sign in
again with the new password.

## Email delivery and request limits

Password-reset email delivery is best effort. An accepted request means Mage has
accepted it for delivery; it does not guarantee that a message has reached the
inbox. Delivery can still be delayed or fail because of the email provider,
recipient server, spam filtering, or a service interruption.

Requests are limited per recipient email address to 3 per hour and 5 per day.
Attempts count toward these limits even when delivery does not complete. If a
message does not arrive, check the spam folder and wait before requesting another
link. If delivery continues to fail after the request limit resets,
[contact Mage Support](https://www.mage.ai/help).

## Troubleshooting

| Issue | What to do |
| - | - |
| **Forgot password?** is unavailable | Ask an administrator to confirm that a Mage Pro release with password reset is deployed. If it remains unavailable, [contact Mage Support](https://www.mage.ai/help). |
| The email does not arrive | Check the spam folder and confirm that the account uses the submitted email address. Because delivery is best effort, wait before requesting another link, then [contact Mage Support](https://www.mage.ai/help) if the issue continues. |
| The link is invalid or expired | Request a new link. Links expire after one hour, are single-use, and are invalidated when another link successfully changes the password. |
| Too many requests | Wait for the hourly or daily per-recipient limit to reset. |


# Role-based access controls (RBAC)
Source: https://docs.mage.ai/production/authentication/permissions/overview

Create user defined permissions for CRUD operations on any action or user experience. Add permissions to roles and add roles to users.

![User defined permissions](https://i.pinimg.com/originals/4d/96/3b/4d963bcbda38f5981a3c83b8f99933ae.gif)

## Overview

Define roles with 1 or more permissions.
Each permission can grant or deny read and write operations on
specific resources (e.g. API endpoints).
One or more roles can be assigned to 1 or multiple users.

### Why is this important?

Control which user can perform specific actions at a granular level.

## Features

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=permissions)

* Define roles.
* Define permissions.
* Add permissions to 1 or more roles.
* Add roles to 1 or more users.

***

## How to use

### Create roles

> [!NOTE]
>
> You must be the owner or have a role that grants permissions to read and write on the following
> entities:
>
> * `Permission`
> * `Role`
> * `RolePermission`
> * `UserRole`
> * `User`

1. Go the the Mage project settings and click on the navigation row labeled **Roles**.
2. On the roles list page, there is a button labeled **Create default roles and permissions**.
   If you click this button, 6 roles and a few hundred permissions will be created.

   The roles are:

   * Admin default permissions
   * Editor default permissions
   * Editor with notebook edit access
   * Editor with pipeline edit access
   * Owner default permissions
   * Viewer default permissions

   Each role will have a set of permissions that match the Mage API policies that are normally
   used to authenticate operations when the user authentication feature is turned on but
   the user permissions feature is turned off.
3. Alternatively, click the button **Add new role** to create a single new role.
4. Enter in a unique name for the role.
5. Click the button **Create new role** to save the role.
6. Once you create the role, you’ll be taken to the role detail page.
7. From here, you can add existing permissions to the role.
8. From here, you can add existing users to the role. This will grant permissions or
   deny operations for user.

![Roles](https://mage-ai.github.io/assets/authentication/permissions/roles-detail.png)

### Create permissions

1. Go the the Mage project settings and click on the navigation row labeled **Permissions**.
2. Click the button **Add new permission** to create a new permission.
3. Select an entity that these permissions will be applied to.
   An entity refers to a defined API endpoint that currently exists in the Mage application.

   For example, the entity `Pipeline` applies to the API endpoints `/pipelines`.
4. Optionally enter an entity UUID that these permissions are applied to.
   For example, you can grant or deny permissions to a specific pipeline by selecting the entity
   `Pipeline` and entering the entity `UUID` `example_pipeline`.
   This permission will only be applied to the pipeline with the UUID `example_pipeline`.
5. Under the section labeled **Access**, toggle 1 or more accesses to grant.
   For more information on what each access grants or denies,
   read the [permissions documentation](/authentication/permissions/permissions).
6. Click the button **Create new permission** to save the permission.
7. Once you create the role, you’ll be taken to the role detail page.
8. From here, you can edit the permission.
9. From here, you can add this permission to existing roles.

![Permissions](https://mage-ai.github.io/assets/authentication/permissions/permissions-detail.png)

### Assign roles to users

1. Go the the Mage project settings and click on the navigation row labeled **Users**.
2. Click on a row for an existing user.
3. From here, you can assign existing roles to the user.
4. Alternatively, you can add a user to a role by following these steps:
   1. Go to the [roles list page](http://localhost:6789/settings/workspace/roles).
   2. Click on a row for an existing role.
   3. Click the button labeled **+Add user**.
   4. Select 1 or more users to assign this role to.

![Users](https://mage-ai.github.io/assets/authentication/permissions/users-detail.png)


# Permissions
Source: https://docs.mage.ai/production/authentication/permissions/permissions

Create granular permissions for CRUD operations on any API endpoint.

![User defined permissions](https://media.tenor.com/NFeV9svhtREAAAAC/community-ken-jeong.gif)

## Overview

* A permission grants or denies read and write operations and access on a specific entity.
* An entity maps to an existing API endpoint in the Mage application.
* A permission can grant or deny access to a specific entity with a specific UUID.
* Each operation and attribute operation has an access level that denies access
  to a specific entity for that particular operation or attribute operation.

  * When denying an attribute operation,
    define the set of attributes that the permission forbids the user from
    querying, reading, or writing.

## Operations

> **List**
>
> This access level grants permission to perform a `GET` request
> to the collections API endpoint for a specific entity.

> **Create**
>
> This access level grants permission to perform a `POST` request
> to an API endpoint for a specific entity.

> **Detail**
>
> This access level grants permission to perform a `GET` request
> to the details API endpoint for a specific entity.

> **Update**
>
> This access level grants permission to perform a `PUT` request
> to an API endpoint for a specific entity.

> **Delete**
>
> This access level grants permission to perform a `DELETE` request
> to an API endpoint for a specific entity.

> **All operations**
>
> This access level grants allows the following operations to be
> performed for a specific entity:
>
> * List
> * Create
> * Detail
> * Update
> * Delete

## Attribute operations

> **Query**
>
> This access level grants permission to use a specific set of query parameters
> when making an API request for a given entity.

> **Read**
>
> This access level grants permission to read a specific set of attributes
> from an API response for a specific entity.

> **Write**
>
> This access level grants permission to write a specific set of attributes
> when submitting a payload in the API request body for a specific entity.

### Query attributes

When granting `Query` access, you must define the set of query parameters this permission allows.

### Read attributes

When granting `Read` access, you must define the set of attributes this permission
allows the user to read.

### Write attributes

When granting `Write` access, you must define the set of attributes this permission
allows the user to write.

***

## Groups

The following access levels contain logic that grants access to multiple operations
and attribute operations.

> [!NOTE]
>
> The attribute operation that the group access grants still requires you
> to define the specific set of attributes that the user is permitted to
> query, read, or write.

> **Viewer**
>
> This access level grants the following for a specific entity:
>
> * Operations
>   * List
>   * Detail
> * Attribute operations
>   * Read

> **Editor**
>
> This access level grants the following for a specific entity:
>
> * Operations
>   * *Everything from Viewer*
>   * Create
>   * Update
>   * Delete
> * Attribute operations
>   * *Everything from Viewer*
>   * Query
>   * Write

> **Admin**
>
> This access level grants the following for a specific entity:
>
> * Operations
>   * *Everything from Viewer*
>   * *Everything from Editor*
> * Attribute operations
>   * *Everything from Viewer*
>   * *Everything from Editor*

> **Owner**
>
> This access level grants the following for a specific entity:
>
> * Operations
>   * *Everything from Viewer*
>   * *Everything from Editor*
>   * *Everything from Admin*
> * Attribute operations
>   * *Everything from Viewer*
>   * *Everything from Editor*
>   * *Everything from Admin*

> **All**
>
> This access level grants every operation, attribute operation,
> all query attributes, all read attributes, and all write attributes
> for a specific entity.

***

## Special conditions

> **Disable unless user has notebook edit access**
>
> This access level will deny the user from performing an operation or attribute operation
> on a specific entity
> unless the user has notebook edit access.

> **Disable unless user has pipeline edit access**
>
> This access level will deny the user from performing an operation or attribute operation
> on a specific entity
> unless the user has pipeline edit access.

> **Disable unless user owns the current entity**
>
> This access level will deny the user from performing an operation or attribute operation
> on a specific entity
> unless the user owns the current entity they are attempting to perform an action on.
>
> > [!WARNING]
> >
> > The only entity this access level supports currently is the `User` entity.

***

## Entity names

> **List of available entities**
>
> 1. `ALL`
> 2. `ALL_EXCEPT_RESERVED`
> 3. `AutocompleteItem`
> 4. `Backfill`
> 5. `Block`
> 6. `BlockLayoutItem`
> 7. `BlockOutput`
> 8. `BlockRun`
> 9. `BlockTemplate`
> 10. `Chart`
> 11. `ClientPage`
> 12. `Cluster`
> 13. `CustomTemplate`
> 14. `DataProvider`
> 15. `Database`
> 16. `EventMatcher`
> 17. `EventRule`
> 18. `ExtensionOption`
> 19. `Feature`
> 20. `File`
> 21. `FileContent`
> 22. `FileVersion`
> 23. `Folder`
> 24. `GitBranch`
> 25. `GitCustomBranch`
> 26. `GitFile`
> 27. `GlobalDataProduct`
> 28. `IntegrationDestination`
> 29. `IntegrationSource`
> 30. `IntegrationSourceStream`
> 31. `Interaction`
> 32. `Kernel`
> 33. `Llm`
> 34. `Log`
> 35. `MonitorStat`
> 36. `Oauth`
> 37. `OauthAccessToken`
> 38. `OauthApplication`
> 39. `Output`
> 40. `PageBlockLayout`
> 41. `PageComponent`
> 42. `Permission`
> 43. `Pipeline`
> 44. `PipelineInteraction`
> 45. `PipelineRun`
> 46. `PipelineSchedule`
> 47. `PipelineTrigger`
> 48. `Project`
> 49. `PullRequest`
> 50. `Role`
> 51. `RolePermission`
> 52. `Scheduler`
> 53. `SearchResult`
> 54. `Secret`
> 55. `Session`
> 56. `SparkApplication`
> 57. `SparkEnvironment`
> 58. `SparkExecutor`
> 59. `SparkJob`
> 60. `SparkSql`
> 61. `SparkStage`
> 62. `SparkStageAttempt`
> 63. `SparkStageAttemptTask`
> 64. `SparkStageAttemptTaskSummary`
> 65. `SparkThread`
> 66. `Status`
> 67. `Sync`
> 68. `Tag`
> 69. `User`
> 70. `UserRole`
> 71. `Variable`
> 72. `Widget`
> 73. `Workspace`

### `ALL`

Using this entity for a permission will grant the operation or attribute operation
for every entity listed above.

### `ALL_EXCEPT_RESERVED`

Using this entity for a permission will grant the operation or attribute operation
for every entity listed except the following entities:

1. `Oauth`
2. `OauthAccessToken`
3. `OauthApplication`
4. `Workspace`


# Roles
Source: https://docs.mage.ai/production/authentication/permissions/roles

Add permissions to roles and add roles to users.

![Roles](https://media3.giphy.com/media/l0HUbtILos6CdAtxu/giphy.gif)

## Overview

* A role contains a set of 1 or more permissions.
* A user can have 1 or more roles.
* Each role must have a unique name across the entire project.


# Cloud cloud
Source: https://docs.mage.ai/production/ci-cd/cloud-cloud





# Buildkite deployments
Source: https://docs.mage.ai/production/ci-cd/local-cloud/buildkite

CI/CD with Buildkite.

## Mage project setup

Follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).

***

## Buildkite setup for deployment to AWS

Use these templates for Buildkite:

1. [`deploy-sc.sh`](https://github.com/mage-ai/mage-ai/tree/master/templates/cicd/buildkite/deploy-ecs.sh)
2. [`pipeline.yml`](https://github.com/mage-ai/mage-ai/tree/master/templates/cicd/buildkite/pipeline.yml)


# GitHub Actions
Source: https://docs.mage.ai/production/ci-cd/local-cloud/github-actions

Development (local) and production (cloud) using GitHub Actions.

Mage supports GitHub Actions for CI/CD workflows. We have templates for deploying to [AWS ECS, GCP Cloud Run, and Azure](https://github.com/mage-ai/mage-ai/tree/master/templates/github_actions).

Before you configure your action, follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).

***

## GitHub Actions setup

1. Create a new repository on GitHub.
2. Open your repository on GitHub, then click the tab labeled **Settings**.
3. Click the section labeled **Security** on the left hand side to expand it.
4. Click the link labeled **Actions**.
5. Click the button labeled **New repository secret** in the top right corner.
6. Follow the instructions below for your specific cloud provider:

***

## AWS

1. If you haven’t already, create a new AWS ECR repository.
2. You’ll need AWS credentials with the following policy permissions:
   ```json theme={"system"}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "ecr:BatchCheckLayerAvailability",
           "ecr:CompleteLayerUpload",
           "ecr:GetAuthorizationToken",
           "ecr:InitiateLayerUpload",
           "ecr:PutImage",
           "ecr:UploadLayerPart",
           "ecs:DeregisterTaskDefinition",
           "ecs:DescribeClusters",
           "ecs:DescribeServices",
           "ecs:DescribeTaskDefinition",
           "ecs:RegisterTaskDefinition",
           "ecs:UpdateService",
           "iam:PassRole"
         ],
         "Resource": "*"
       }
     ]
   }
   ```
3. In the field labeled **Name**, enter the value `AWS_ACCESS_KEY_ID`.
4. In the field labeled **Secret**, enter your AWS Access Key ID.
5. Click the button labeled **Add secret** to save.
6. Add a 2nd secret by clicking the button labeled **New repository secret** in
   the top right corner.
7. In the field labeled **Name**, enter the value `AWS_SECRET_ACCESS_KEY`.
8. In the field labeled **Secret**, enter your AWS Secret Access Key.
9. Click the button labeled **Add secret** to save.
10. Click on the tab labeled **Actions**.
11. On the left side, click the button labeled **New workflow**.
12. Find the link labeled **`set up a workflow yourself`** and click it.
13. Copy the contents from the GitHub Action YAML file for AWS at
    [templates/github\_actions/build\_and\_deploy\_to\_aws\_ecs.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_aws_ecs.yml),
    and paste it into the textarea.
14. Change the following values under the key labeled `env`:

```yaml theme={"system"}
env:
  AWS_REGION: ...
  ECR_REPOSITORY: ...
  ECS_CLUSTER: ...
  ECS_SERVICE: ...
  ECS_TASK_DEFINITION: ...
  CONTAINER_NAME: ...
```

| Key | Description | Sample value |
| - | - | - |
| `AWS_REGION` | Region of your AWS ECS cluster. | `us-west-2` |
| `CONTAINER_NAME` | Set this to the name of the container in the containerDefinitions section of your task definition. | `mage-data-production-container` |
| `ECR_REPOSITORY` | The name of the AWS ECR repository you created to store your Docker images. | `mage-data` |
| `ECS_CLUSTER` | The name of your AWS ECS cluster. | `mage-production-cluster` |
| `ECS_SERVICE` | The name of your AWS ECS service. | `mage-production-ecs-service` |
| `ECS_TASK_DEFINITION` | Go to your AWS ECS task definition for the above service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |

1. Click the button labeled **Start commit** in the top right corner.
2. Click the button labeled **Commit new file**.
3. Every time you merge a pull request into the master branch, this GitHub
   Action will run, building a Docker image using your GitHub code, then
   updating AWS ECS to use the new image with the updated code.

## GCP

1. You’ll need to add these roles to the service account that you’ll be using to
   deploy Mage from GitHub:
   * Artifact Registry Read
   * Artifact Registry Writer
   * Cloud Run Admin
   * Service Account Token Creator
   * Service Account User

2. In the field labeled **Name**, enter the value `GCP_CREDENTIALS`.

3. In the field labeled **Secret**, enter the JSON string containing your GCP
   service account credentials. It should look something like this:
   `json { "type": "service_account", "project_id": "mage-123456", "private_key_id": "...", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", "client_email": "mage@mage-123456.iam.gserviceaccount.com", "client_id": "...", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mage%40mage-123456.iam.gserviceaccount.com" } `

4. Click the button labeled **Add secret** to save.

5. Click on the tab labeled **Actions**.

6. On the left side, click the button labeled **New workflow**.

7. Find the link labeled **`set up a workflow yourself`** and click it.

8. Copy the contents from the GitHub Action YAML file for GCP at
   [templates/github\_actions/build\_and\_deploy\_to\_gcp\_cloud\_run.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_gcp_cloud_run.yml),
   and paste it into the textarea.

9. Change the following values under the key labeled `env`:

   ```yaml theme={"system"}
   env:
     GAR_LOCATION: ...
     GOOGLE_CLOUD_RUN_SERVICE_NAME: ...
     IMAGE: ...
     PROJECT_ID: ...
     REPOSITORY: ...
   ```

   | Key | Description | Sample value |
   | - | - | - |
   | `GAR_LOCATION` | Region that Mage is already deployed in. | `us-east1` |
   | `GOOGLE_CLOUD_RUN_SERVICE_NAME` | The name of your Google Cloud Run service. Go to the Google Cloud Run dashboard to find it. | `mage-data` |
   | `IMAGE` | The name of the Docker image you pushed to the above GCP Artifact Registry. | `mageai` |
   | `PROJECT_ID` | Project ID of where you launched Mage using Terraform. | `mage-123456` |
   | `REPOSITORY` | The name of your GCP Artifact Registry that is storing your Docker image. | `mageprod` |

10. Click the button labeled **Start commit** in the top right corner.

11. Click the button labeled **Commit new file**.

12. Every time you merge a pull request into the master branch, this GitHub
    Action will run, building a Docker image using your GitHub code, then
    updating Google Cloud Run to use the new image with the updated code.

## Azure

Mage can be deployed to Azure using GitHub Actions.

Our template for deploying to an Azure Container instance is located [here](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_azure_container_instance.yml) and based off the "GitHub Workflow" option in Azure's docs.

To configure this workflow, you will need to add the following secrets to your GitHub repo:

| Key | Description | | | |
| - | - | - | - | - |
| `AZURE_CREDENTIALS` | The entire JSON output from the service principal creation step | | | |
| `REGISTRY_LOGIN_SERVER` | The login server name of your registry (all lowercase). Example: myregistry.azurecr.io | | | |
| `REGISTRY_USERNAME` | The clientId from the JSON output from the service principal creation | | | |
| `REGISTRY_PASSWORD` | The clientSecret from the JSON output from the service principal creation | | | |
| `RESOURCE_GROUP` | The name of the resource group you used to scope the service principal | | | |

We recommend [Microsoft's Guide](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-github-action?tabs=userlevel#configure-github-workflow) for obtaining these credentials via the Azure CLI.

Microsoft also has templates that are similar to ours— we use the `Service ID Principal` method for authentication, but you can check Microsoft's docs for the [`OpenID Connect`](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-github-action?tabs=openid#save-credentials-to-github-repo) method.

Alternatively, you can use the `az container app up` command in the [Deploy to Azure](https://github.com/Azure/deploy-to-azure-cli-extension) extension in the Azure CLI. This command streamlines creation of the GitHub workflow and deployment steps.


# GitLab deployments
Source: https://docs.mage.ai/production/ci-cd/local-cloud/gitlab-ci-cd

Development (local) and production (cloud) using GitLab CI/CD.

## Mage project setup

Follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).

***

## GitLab CI/CD setup

1. Create a new repository on GitLab.
2. Open your repository on GitLab, then click the tab labeled **Settings**.
3. Click the section labeled **CI/CD**.
4. Under the section labeled **Variables**, click the button labeled **Expand**.
5. Click the button labeled **Add variable**.
6. Follow the instructions below for your specific cloud provider:

***

## AWS

1. If you haven’t already, create a new AWS ECR repository.

2. You’ll need AWS credentials with the following policy permissions:
   ```json theme={"system"}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "ecr:BatchCheckLayerAvailability",
           "ecr:CompleteLayerUpload",
           "ecr:GetAuthorizationToken",
           "ecr:InitiateLayerUpload",
           "ecr:PutImage",
           "ecr:UploadLayerPart",
           "ecs:DeregisterTaskDefinition",
           "ecs:DescribeClusters",
           "ecs:DescribeServices",
           "ecs:DescribeTaskDefinition",
           "ecs:RegisterTaskDefinition",
           "ecs:UpdateService",
           "iam:PassRole"
         ],
         "Resource": "*"
       }
     ]
   }
   ```

3. When adding a variable, in the field labeled **Key**, enter the value `AWS_ACCESS_KEY_ID`.

4. When adding a variable, in the field labeled **Value**, enter your AWS Access Key ID.

5. Check the box labeled **Mask variable**..

6. Click the button labeled **Add variable** to save.

7. Add a 2nd secret by clicking the button labeled **Add variable** in
   the top right corner.

8. In the field labeled **Key**, enter the value `AWS_SECRET_ACCESS_KEY`.

9. In the field labeled **Value**, enter your AWS Secret Access Key.

10. Check the box labeled **Mask variable**..

11. Click the button labeled **Add variable** to save.

12. In the field labeled **Key**, enter the value `AWS_DEFAULT_REGION`.

13. In the field labeled **Value**, enter your AWS default region.

14. Click the button labeled **Add variable** to save.

15. Create a new file in your repository and name it `.gitlab-ci.yml`.
    If you followed the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup),
    then your current folder structure could look like this:
    ```
    my_team/
    |   demo_project/
    |   .gitlab-ci.yml
    |   Dockerfile
    ```

16. Copy the contents of this [GitLab CI/CD template file](https://github.com/mage-ai/mage-ai/blob/master/templates/gitlab_cicd/aws.gitlab-ci.yml)
    and paste it into the file named `.gitlab-ci.yml`.

17. Change the following values in the file named `.gitlab-ci.yml` under the key labeled `variables`:

    ```yaml theme={"system"}
    variables:
      REPOSITORY_URL: ...
      CLUSTER_NAME: ...
      SERVICE_NAME: ...
      TASK_DEFINITION_NAME: ...
    ```

    | Key | Description | Sample value |
    | - | - | - |
    | `REPOSITORY_URL` | The URL of the AWS ECR repository you created to store your Docker images. | `679812356111.dkr.ecr.us-west-2.amazonaws.com/mage-data` |
    | `CLUSTER_NAME` | The name of your AWS ECS cluster. | `mage-production-cluster` |
    | `SERVICE_NAME` | The name of your AWS ECS service. | `mage-production-ecs-service` |
    | `TASK_DEFINITION_NAME` | Go to your AWS ECS task definition for the above service. Use the name of that task definition. | `mage-production-task` |

18. If you use a different branch than `main`, for example `master`,
    you must edit the value `only` under the `build` and `deploy` key in the file `.gitlab-ci.yml`.
    For example:
    ```yaml theme={"system"}
    build:
      # other settings
      only:
        - master

    deploy:
      # other settings
      only:
        - master
    ```

19. Commit your changes with the new file `.gitlab-ci.yml`.

20. Push your commits to GitLab.

21. Every time you merge a pull request into the main or master branch, this GitLab
    CI/CD pipeline will run, building a Docker image using your GitLab code, then
    updating AWS ECS to use the new image with the updated code.

***

## GCP

*Coming soon...*

***

## Azure

*Coming soon...*


# Repository setup
Source: https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup

Setup your Mage project repository.

1. Create a parent folder for your Mage project (e.g. `my_team`).

2. Change directory (e.g. `cd my_team`) into the parent folder and start Mage
   locally: For example:

   ```bash theme={"system"}
   docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
     /app/run_app.sh mage start demo_project
   ```

   For more examples, read the [setup guide](/getting-started/setup).

3. Once you’re done developing, copy the contents of the this
   [Dockerfile template](https://github.com/mage-ai/docker/blob/master/Dockerfile)
   and paste it into a new `Dockerfile` located in the parent folder of your
   Mage project (e.g. `my_team/Dockerfile`). Replace all instances of the
   string `[project_name]` with your project name. For example, if your project
   name is `demo_project`, then your Dockerfile will look like this: The
   contents of your `Dockerfile` are:

   ```dockerfile theme={"system"}
   FROM mageai/mageai:latest

   ARG PROJECT_NAME=[project_name]
   ARG MAGE_CODE_PATH=/home/mage_code
   ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}

   WORKDIR ${MAGE_CODE_PATH}

   # Replace [project_name] with the name of your project (e.g. demo_project)
   COPY ${PROJECT_NAME} ${PROJECT_NAME}

   # Set the USER_CODE_PATH variable to the path of user project.
   # The project path needs to contain project name.
   # Replace [project_name] with the name of your project (e.g. demo_project)
   ENV USER_CODE_PATH=${USER_CODE_PATH}

   # Install custom Python libraries
   RUN pip3 install -r ${USER_CODE_PATH}/requirements.txt
   # Install custom libraries within 3rd party libraries (e.g. dbt packages)
   RUN python3 /app/install_other_dependencies.py --path ${USER_CODE_PATH}

   ENV PYTHONPATH="${PYTHONPATH}:/home/mage_code"

   CMD ["/bin/sh", "-c", "/app/run_app.sh"]
   ```

4. Your current folder structure should look like this:
   ```
   my_team/
   |   demo_project/
   |   Dockerfile
   ```

5. Build a custom Docker image using `mageai/mageai:latest` as the base and
   using the newly created Dockerfile as the additional set of instructions:

   ```
   docker build --platform linux/amd64 --tag mageprod:latest .
   ```

   > [!NOTE]
   >
   > Change `mageprod` to any other name. You’ll need this tag name when
   > deploying to production in the cloud.

6. To test the new image works, run the following command:
   ```bash theme={"system"}
   docker run -it -p 6789:6789 mageprod:latest /app/run_app.sh mage start demo_project
   ```

7. Open your browser and go to [http://localhost:6789/](http://localhost:6789/)
   1. You should see all your pipelines there.
   2. Changing the contents of files won’t change the contents on your local
      file system because all your code was packaged together within the Docker
      image (this is intentional).

8. The next steps depends on your deployment method. If you are using
   [Terraform](/production/deploying-to-cloud/using-terraform), then you’ll need to use the previously
   tag name (e.g. `mageprod`) when pushing a Docker image to a remote Docker
   registry.


# Continuous deployment with Mage Pro
Source: https://docs.mage.ai/production/ci-cd/mage/pro

Manage, deploy, and rollback new code changes, from a remote repository to any environment, all from within the Mage platform.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=deployment)

**1. Connect remote repository**

![Mage Pro Connect remote repository](https://mage-ai.github.io/assets/pro/features/deployment-configurations.jpg)

**2. Deployment stages**

![Mage Pro Deployment stages](https://mage-ai.github.io/assets/pro/features/deployment-stages.jpg)

**3. Deploy new code changes or rollback**

![Mage Pro Deploy new code changes or rollback](https://mage-ai.github.io/assets/pro/features/deployment-deploys.jpg)

Automatically deploy new code changes and data pipelines to different environments. Mage
attempts to install packages from `requirements.txt`; if installation fails, the deployment
continues and records a warning in the deployment logs.

***

> **[Get started for free](https://cloud.mage.ai/sign-up?source=deployment)**
>
> ![](https://mage-ai.github.io/assets/pro/banner-dark.png)
>
> A fully managed service, where we maintain the infrastructure, guarantee uptime,
> automatically scale your workloads to handle any volume of pipeline runs,
> automatically upgrade new versions of Mage Pro only features,
> monitor your production pipelines, and provide enterprise level support.


# Overview
Source: https://docs.mage.ai/production/ci-cd/overview

There are 4 development and deployment workflows.

## Single environment

### Development and production (local)

Develop locally and run pipelines locally in a single environment.

| Resources | Development | Production |
| - | - | - |
| Local | `X` | `X` |
| Cloud | | |

### Development and production (cloud)

Develop on cloud and run pipelines on cloud in a single environment.

| Resources | Development | Production |
| - | - | - |
| Local | | |
| Cloud | `X` | `X` |

***

## Multiple environments

### Development (local) and production (cloud)

Develop locally (development environment) and
run pipelines on cloud (production environment) in a separate environments.

| Resources | Development | Production |
| - | - | - |
| Local | `X` | |
| Cloud | | `X` |

### Development (cloud) and production (cloud)

Develop on cloud (development environment) and
run pipelines on cloud (production environment) in a separate environments.

| Resources | Development | Production |
| - | - | - |
| Local | | |
| Cloud | `X` | |
| Cloud | | `X` |


# GitHub Actions
Source: https://docs.mage.ai/production/ci-cd/staging-production/github-actions

Development (local), staging (cloud), and production (cloud) using GitHub Actions

Mage supports GitHub Actions for CI/CD workflows. We have templates for deploying to [AWS ECS, GCP Cloud Run, and Azure](https://github.com/mage-ai/mage-ai/tree/master/templates/github_actions).

## GitHub Actions setup

1. Create a new repository on GitHub.
2. Open your repository on GitHub, then click the tab labeled **Settings**.
3. Click the section labeled **Secrets and variables** on the left hand side to expand it.
4. Create separate staging and production GitHub environments in the **Environments** section.
   1. You can also choose to require approval before running jobs in your production environment.
      More information [here](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers).
   2. You may need to change the `environment` variable in the jobs section of the GitHub Actions below
      based on the name of your GitHub environments.
5. Click the link labeled **Actions**.
6. Click the button labeled **New repository secret** in the top right corner.
7. Follow the instructions below for your specific cloud provider:

***

## AWS

1. If you haven’t already, create a new AWS ECR repository.
2. You’ll need AWS credentials with the following policy permissions:
   ```json theme={"system"}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "ecr:BatchCheckLayerAvailability",
           "ecr:CompleteLayerUpload",
           "ecr:GetAuthorizationToken",
           "ecr:InitiateLayerUpload",
           "ecr:PutImage",
           "ecr:UploadLayerPart",
           "ecs:DeregisterTaskDefinition",
           "ecs:DescribeClusters",
           "ecs:DescribeServices",
           "ecs:DescribeTaskDefinition",
           "ecs:RegisterTaskDefinition",
           "ecs:UpdateService",
           "iam:PassRole"
         ],
         "Resource": "*"
       }
     ]
   }
   ```
3. In the field labeled **Name**, enter the value `AWS_ACCESS_KEY_ID`.
4. In the field labeled **Secret**, enter your AWS Access Key ID.
5. Click the button labeled **Add secret** to save.
6. Add a 2nd secret by clicking the button labeled **New repository secret** in
   the top right corner.
7. In the field labeled **Name**, enter the value `AWS_SECRET_ACCESS_KEY`.
8. In the field labeled **Secret**, enter your AWS Secret Access Key.
9. Click the button labeled **Add secret** to save.
10. Click on the tab labeled **Actions**.
11. On the left side, click the button labeled **New workflow**.
12. Find the link labeled **`set up a workflow yourself`** and click it.
13. Copy the contents from the GitHub Action YAML file for AWS at
    [templates/github\_actions/build\_and\_deploy\_to\_aws\_ecs\_staging\_production.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml),
    and paste it into the textarea.
14. Change the following values under the key labeled `env`:

```yaml theme={"system"}
env:
  AWS_REGION: ...
  CONTAINER_NAME: ...
  ECR_REPOSITORY: ...
  ECS_CLUSTER: ...
  ECS_STAGING_SERVICE: ...
  ECS_PRODUCTION_SERVICE: ...
  ECS_STAGING_TASK_DEFINITION: ...
  ECS_PRODUCTION_TASK_DEFINITION: ...
```

| Key | Description | Sample value |
| - | - | - |
| `AWS_REGION` | Region of your AWS ECS cluster. | `us-west-2` |
| `CONTAINER_NAME` | Set this to the name of the container in the containerDefinitions section of your task definition. | `mage-data-production-container` |
| `ECR_REPOSITORY` | The name of the AWS ECR repository you created to store your Docker images. | `mage-data` |
| `ECS_CLUSTER` | The name of your AWS ECS cluster. | `mage-production-cluster` |
| `ECS_STAGING_SERVICE` | The name of your AWS ECS staging service. | `mage-production-cluster` |
| `ECS_PRODUCTION_SERVICE` | The name of your AWS ECS production service. | `mage-production-ecs-service` |
| `ECS_STAGING_TASK_DEFINITION` | Go to your AWS ECS task definition for the staging service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
| `ECS_PRODUCTION_TASK_DEFINITION` | Go to your AWS ECS task definition for the production service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |

1. Click the button labeled **Start commit** in the top right corner.
2. Click the button labeled **Commit new file**.
3. Every time you merge a pull request into the master branch, this GitHub
   Action will run, building a Docker image using your GitHub code, then
   updating AWS ECS to use the new image with the updated code.


# Compute resources
Source: https://docs.mage.ai/production/configuring-production-settings/compute-resource



Our fully managed solution for teams is now available!

Mage Pro will automatically scale your workloads to handle any volume of pipeline runs, while optimizing resource utilization and minimizing costs.

[Try Mage Pro for free](https://cloud.mage.ai/sign-up?source=compute-resources)

[Open](https://cloud.mage.ai/sign-up?source=compute-resources)

Follow the instructions in this [document](/production/deploying-to-cloud/using-terraform) to deploy
Mage in a production environment. When running Mage in production,
you can customize the compute resource for the following services:

1. Front-end application
2. Pipeline executor
3. Block executor

## Front-end application

Customize the compute resource of the Mage web service.

The Mage web service is responsible for running Mage web backend, scheduler service
and local block executions.

You can customize the CPU and memory of the Mage web
service by updating the Terraform variables and then running `terraform apply`.

### Amazon Web Services (AWS)

Update the `ecs_task_cpu` and `ecs_task_memory` variables in the
[`mage-ai-terraform-templates/aws/variables.tf`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf)
file.

### Google Cloud Platform (GCP)

Update the `container_cpu` and `container_memory` variables in the
[`mage-ai-terraform-templates/gcp/variables.tf`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
file.

***

## Pipeline executor

Set the pipeline’s executor type to customize the compute resources for pipeline runs.
Here are the available executor types:

* `local_python`
* `k8s`
* `ecs`

### Default for all blocks in a pipeline

You can set the executor type for all the blocks in a pipeline by specifying the `executor_type` at pipeline level.

***

## Block executor

Mage provides multiple executors to execute blocks.
Here are the available executor types:

* `local_python`
* `k8s`
* `ecs`
* `azure_container_instance`
* `gcp_cloud_run`

### Default for all blocks

Mage uses `local_python` executor type by default. If you want to specify another executor\_type as the default executor type for blocks,
you can set the environment variable `DEFAULT_EXECUTOR_TYPE` to one executor type mentioned above.


# Production Configuration
Source: https://docs.mage.ai/production/configuring-production-settings/overview



## Settings

### Read only access

You can configure Mage to not allow any edits to pipelines or blocks and/or not allow edit or execute
access for the notebook. Users will only be able to create triggers and view the existing pipelines.
They will have access to features such as syncing with GitHub and future remote repository integrations.

To enable this feature set the following environment variable:
`export DISABLE_NOTEBOOK_EDIT_ACCESS=<value>`

The available values are

* 0: this is the same as omitting the variable
* 1 (most restrictive): no edit/execute access to the notebook within Mage. Users will not be able to use the notebook to edit pipeline content, execute blocks,
  create secrets, or create variables.
* 2 (less restrictive): only disables edits for pipelines and blocks. Users will not be able to edit pipeline/block metadata or content.

### Databases

By default, Mage uses sqlite to store orchestration data (trigger, pipeline run, and block run). To use a different database, you can set the `MAGE_DATABASE_CONNECTION_URL` environment variable.

In production, you can set the environment variable in the corresponding Terraform script.

#### PostgreSQL

`export MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://user:password@host:port/dbname`

To use a specific `schema_name` in PostgreSQL:

`export MAGE_DATABASE_CONNECTION_URL="postgresql+psycopg2://user:password@host:port/dbname?options=-c%%20search_path%%3Dschema_name"`

#### MSSQL

`export MAGE_DATABASE_CONNECTION_URL="mssql+pyodbc://?odbc_connect=DRIVER={ODBC Driver 18 for SQL Server};SERVER=host;DATABASE=dbname;UID=user;PWD=password;ENCRYPT=yes;TrustServerCertificate=yes;"`

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=production-settings)

If Mage fails to start because an orchestration database migration failed, use `mage db diagnose` first. See [Troubleshoot database migration failures](/development/databases/default#troubleshoot-database-migration-failures) for migration history, table schema inspection, migration, rollback, and Alembic version stamping commands.

### ulimit

ulimit is a built-in Linux command that allows viewing or limiting system resource used in processes.

If you get `OSError: [Errno 24] Too many open files` error when running in Mage in production,
you can set the environment variable `ULIMIT_NO_FILE` to a larger number (e.g. 8192 or 16384) and restart
Mage server to increase the maximum number of open files allowed in Mage processes.

If you get `OSError: [Errno 24] Too many open files` error when running Mage locally on your laptop,
you can increase the limit with the command `ulimit -n 8192` (or a larger number). Add this command to
`~/.bash_profile` or `~/.zshrc` file so that this config can be applied automatically in the terminal.


# Trust internal certificate authorities
Source: https://docs.mage.ai/production/configuring-production-settings/trusted-ca-certificates

Configure Mage runtimes to trust internal certificate authorities for outbound TLS connections.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=trusted-ca-certificates)

Use `TRUSTED_CA_CERTS_PATH` to trust internal certificate authorities without disabling TLS certificate verification. Mage reads the certificates when its process starts, merges them with the default CA bundle, and configures common Python and command-line HTTPS clients to use the merged bundle.

## Prepare the certificate directory

`TRUSTED_CA_CERTS_PATH` must point to a directory containing the trusted root and any required intermediate CA certificates.

Certificate files must:

* Use a `.crt`, `.pem`, or `.cer` filename, or have no extension.
* Be PEM encoded and contain one or more `-----BEGIN CERTIFICATE-----` blocks. DER-encoded `.cer` files are not supported.

Mage ignores hidden files and certificate files that do not contain a complete PEM certificate block.

## Configure Docker

Mount the certificate directory in the Mage container and set `TRUSTED_CA_CERTS_PATH` to the container path. For Docker Compose, merge the following settings into the Mage service:

```yaml theme={"system"}
services:
  mage:
    environment:
      TRUSTED_CA_CERTS_PATH: /etc/mage/trusted-ca
    volumes:
      - ./trusted-ca:/etc/mage/trusted-ca:ro
```

Restart the Mage container after changing the mount or environment variable.

## Configure Kubernetes with Helm

Create a Kubernetes Secret containing the CA certificate files, then add the volume, mount, and environment variable to the Mage Helm `values.yaml`:

```yaml theme={"system"}
extraVolumes:
  - name: internal-ca
    secret:
      secretName: <ca-secret-name>

extraVolumeMounts:
  - name: internal-ca
    mountPath: /etc/mage/trusted-ca
    readOnly: true

config:
  TRUSTED_CA_CERTS_PATH: /etc/mage/trusted-ca
```

Apply the Helm upgrade and restart the affected Mage runtimes so they load the certificate directory during startup.

### Kubernetes workspaces

For Mage Pro workspaces, also configure the workspace manager to inherit the CA volume into workspace pods:

```yaml theme={"system"}
config:
  TRUSTED_CA_CERTS_PATH: /etc/mage/trusted-ca
  WORKSPACE_INHERIT_VOLUMES: internal-ca
```

`WORKSPACE_INHERIT_VOLUMES` is a comma-separated list of Kubernetes volume names to copy, with their matching mounts, from the workspace manager into workspace pod specifications. Use the volume name (`internal-ca` in this example), not the Kubernetes Secret name.

After upgrading the Helm release, restart the workspace manager and update or recreate affected workspaces so Mage regenerates their pod specifications. Restarting an existing workspace pod alone does not add a missing volume.

## Confirm the trust bundle loaded

When Mage starts successfully in Docker or Kubernetes, its logs include:

```text theme={"system"}
Trust store updated: added <number> custom CA(s) from /etc/mage/trusted-ca
```

Mage sets `REQUESTS_CA_BUNDLE`, `SSL_CERT_FILE`, and `CURL_CA_BUNDLE` inside its process to the generated merged bundle. A shell opened separately in the same container does not necessarily inherit those values, so validate the connection through the Mage runtime that executes the pipeline.

## Verify certificate trust

Test a read-only internal HTTPS endpoint from a Mage block:

```python theme={"system"}
import requests

response = requests.get(
    'https://<internal-host>/<read-only-endpoint>',
    timeout=30,
)
response.raise_for_status()
```

Keep certificate verification enabled. Do not use `verify=False` or explicitly set `verify=certifi.where()`, because either option disables or bypasses the merged Mage trust bundle.

If verification still fails, confirm that the certificate directory is mounted in the runtime executing the request, the Mage startup log reports the expected certificates, and the server presents its required intermediate certificate chain.


# Architecture
Source: https://docs.mage.ai/production/deploying-to-cloud/architecture



Our fully managed solution for teams is now available!

[Try Mage Pro for free](https://cloud.mage.ai/sign-up?source=architecture)

[Open](https://cloud.mage.ai/sign-up?source=architecture)

![Mage OSS rchitecture](https://mage-ai.github.io/assets/mage-architecture.png)

Mage architecture diagram

## Server

Mage server runs in an instance or container. Here are the responsibilities of the Mage server.

* Handle all the API requests.
* Interact with the storage to read and write data.
* Run the websocket server to handle websocket requests.
* Execute the pipeline code in the kernel.
* Handle user authentication.

Mage server creates the scheduler process once the server starts. The scheduler process is responsible for
scheduling pipeline runs for active pipeline triggers.

## Storage

In the storage layer, Mage stores the following data:

* **Code files**: Mage stores code files on the local disk. You can use [git](/production/data-sync/git-integration) to sync the code between your local disk and remote the git repository. If you deploy Mage in containers, you can mount an Network File System share in your container to persist the code and data on your disk.
* **Data and log files**: By default, Mage stores data and log files on the local disk. You can also configure it to store [logs](/development/observability/logging#logging-to-external-destination) in an external storage.
* **Pipeline Trigger, Pipeline Run, User data, etc**.: Orchestration related objects and authentication related objects are stored in a database. By default, Mage uses a local SQLite database. In a production environment, You can configure Mage to use a [Postgres database](/production/configuring-production-settings/overview#databases).
* **Secrets**: Mage supports storing [secrets](/development/variables/secrets) in the database. It's also recommended to use an external [Secret Manager](/production/deploying-to-cloud/secrets/AWS) to store secrets.

## Executors

Mage supports different types of executors to execute jobs:

* Local process executor
* ECS executor (AWS)
* GCP Cloud Run executor
* Kubernetes executor

Check out this [doc](/production/configuring-production-settings/compute-resource#2-customize-the-compute-resource-of-the-mage-executor) to learn about how to configure these executors.

## Scale web server and schedulers

By default, Mage runs the web server and scheduler in the same container. Mage supports horizontally scaling by separating the web server and scheduler,
and run multiple replicas of web servers and schedulers.

To separate the web server and scheduler, you can override the container command with the commands below:

* Run scheduler only (The scheduler doesn't need dedicated IP and port)
  ```bash theme={"system"}
  /app/run_app.sh mage start project_name --instance-type scheduler
  ```
* Run web server only
  ```bash theme={"system"}
  /app/run_app.sh mage start project_name --instance-type web_server
  ```

To run multiple schedulers simultaneously, you will need to specify the `REDIS_URL` environment variable in your container. The Redis storage is used to coordinate the job scheduling between multiple containers to resolve race conditions. Here is an example Redis URL format: `redis://redis:6379/0`

## Autoscaling

Do you want to run your pipelines at any scale without having to deal with infrastructure setup,
configurations, deployment, and maintenance?

1. Autoscaling orchestration scheduler for maximum pipeline trigger frequency
2. Vertical and horizontal autoscaling data pipelines and block run executions

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=autoscaling)


# Mage and Native Cloud Environments
Source: https://docs.mage.ai/production/deploying-to-cloud/aws-without-terraform

Just like a traditional notebook, Mage supports execution in native cloud environments. Below are guides on how to integrate with native cloud resources.

* [Amazon EC2](#amazon-ec2)
* [Amazon ECS](#amazon-ecs). For Amazon ECS, we recommend using Terraform. Here
  is the [guide](/production/deploying-to-cloud/using-terraform).

***

# Amazon EC2

Mage can be run in an Amazon EC2 instance, either by

* using SSH to port forward localhost requests to your EC2 instance
* opening port 6789 on your EC2 instance for access

***

## Prerequisites

Your EC2 instance must have `python3` installed. All Python versions between
3.7.0 (inclusive) and 3.10.0 (exclusive) are supported.

When launching an EC2 instance, we suggest using the following AMI:

| AMI ID | AMI name |
| - | - |
| `ami-0d70546e43a941d70` | `ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20220609` |

***

## Quick Connection via SSH

### Install Python

Once the EC2 instance is in a running state, SSH into the instance. You can SSH
using the UI in AWS or you can SSH from your terminal using a command like the
following:

```bash theme={"system"}
ssh -i [path_to_key_pair] [ec2_username]@[ec2_public_dns_name]
```

Note: the `ec2_username` is typically `ubuntu` if you used the AMI we
recommended above.

An example SSH command could look like this:

```bash theme={"system"}
ssh -i ~/.ssh/aws-ec2.pem ubuntu@ec2-55-186-46-136.us-west-2.compute.amazonaws.com
```

Once you SSH into the instance, run the following command to install Python and
pip:

```bash theme={"system"}
sudo apt update && \
  sudo apt install -y software-properties-common && \
  sudo add-apt-repository -y ppa:deadsnakes/ppa && \
  sudo apt install -y python3.7 \
  sudo apt install -y python3-pip
```

### Run script to start Mage

We provide the `scripts/run_ec2.sh` script to launch Mage in an EC2 instance.
Currently only connections via SSH are supported. To access [this script](https://github.com/mage-ai/mage-ai/blob/master/scripts/run_ec2.sh)
clone this [repository](https://github.com/mage-ai/mage-ai).

To run Mage in an EC2 instance, you need to provide

* Path to the key pair used with the EC2 instance
* Your EC2 instance username
* Your EC2 public DNS name

```bash theme={"system"}
./scripts/run_ec2.sh [path_to_key_pair] [ec2_user_name] [ec2_public_dns_name]
```

Note: the `ec2_username` is typically `ubuntu` if you used the AMI we
recommended above.

You can optionally add `--name custom_repo_name` to the end of the command above
to name your project something different than the default of `default_repo`.

An example command could look like this:

```bash theme={"system"}
./scripts/run_ec2.sh ~/.ssh/aws-ec2.pem ubuntu@ec2-55-186-46-136.us-west-2.compute.amazonaws.com --name demo_project
```

This script will

1. Connect to your EC2 instance
2. Install Mage if not already installed on instance
3. Create a default repository (default repository name is 'default\_repo', use
   `--name` to specify your own custom repository name)
4. Launch the Mage tool pointing at this repository

To access the Mage tool, open [localhost:6789](http://localhost:6789). All
actions made here will be forwarded to your EC2 instance for execution.

To quit out of Mage, stop execution in your current terminal window (Ctrl+C).
This will shutdown the Mage tool alongside closing the connection to your EC2
instance.

***

## Manual connection via Open Port

You can also connect to the Mage app running on your EC2 instance by editing
your security group to expose the port that Mage runs on.

1. When creating your EC2 instance, edit your security group rules to allow a
   Custom TCP connection to port 6789 (the port that Mage runs on).

2. Connect to your EC2 instance via SSH and install Mage if you haven't already:
   ```bash theme={"system"}
   ssh -i path/to/key/pair.pem ec2_username@ec2_public_dns_name
   pip install mage-ai
   ```

3. Start Mage on your EC2 instance:

   ```bash theme={"system"}
   mage init <your_repo_name>
   mage start <your_repo_name>
   ```

4. Then from your browser, access `<ec2-instance-public-ip>:6789` to access the
   Mage app, where `ec2-instance-public-ip` is the Public IPv4 address of your
   EC2 instance.

   ***

## Manual connection via SSH

You can also manually start a connection to the Mage tool running on an EC2
instance. This process involves opening two separate SSH connections:

1. Connect to the EC2 instance to run the Mage Tool. First, open an SSH
   connection with your EC2 Key Pair and your EC2 login information to connect
   the EC2 instance.

   ```bash theme={"system"}
   ssh -i path/to/key/pair.pem ec2_username@ec2_public_dns_name
   ```

   If the Mage tool is not installed, install Mage using `pip`. It is
   recommended to use a virtual environment to store the Python packages to
   isolate your Python dependencies per project.

   ```bash theme={"system"}
   python3 -m venv env
   source ./env/bin/activate
   pip3 install mage-ai
   ```

   Next, initialize your Mage repository using `mage init`. You can change your
   repository name as per your preference.

   ```bash theme={"system"}
   mage init default_repo
   ```

   Then start the Mage app. This will start the notebook at
   [localhost:6789](http://localhost:6789), but you can't directly access it yet
   as this notebook is started and is executed on the EC2 instance

   ```bash theme={"system"}
   mage start default_repo
   ```

2. Forward all requests sent to [localhost:6789](http://localhost:6789) on
   *your* computer back to the EC2 instance. This means that as you interact the
   user interface on your computer, the data will be forwarded to the EC2
   instance for execution. Open another terminal window and run
   ```bash theme={"system"}
   ssh -i path/to/key/pair.pem -N -L 6789:localhost:6789 ec2_username@ec2_public_dns_name
   ```
   which opens up a SSH connection to your EC2 instance which forwards requests
   to your localhost.
   * The `-N` option tells your SSH client to not send user commands to the EC2
     server through this connection since this connection is only for port
     forwarding

Now you can access the Mage tool at [localhost:6789](http://localhost:6789) on
your computer, and all executions will be processed on your EC2 instance!

***

# Amazon ECS

You can launch Mage in an Amazon Elastic Container Service (ECS) cluster,
allowing you to perform tasks like:

* [Using the Mage code editor in the ECS cluster](#running-the-mage-app-in-ecs)
* [Running a Mage ETL pipeline in production](#running-mage-pipeline-in-ecs)

## Running the Mage App in ECS

Follow the steps below to launch the Mage app on your ECS cluster:

1. Create a new Docker image for launching the Mage App— you can start with [our template](https://github.com/mage-ai/docker/blob/master/Dockerfile). Upload this Docker image to a repository like Amazon ECR so ECS tasks can pull and create Docker containers using this image.

2. Create a new ECS cluster. Both Fargate and EC2 based clusters work for this
   task.

3. Create a new task definition based off the template below. This task, when
   run:

   1. Creates a new Mage repository named "default\_repo".
   2. Starts the Mage app, serving requests sent to "localhost:6789"

   ```json theme={"system"}
   {
     "containerDefinitions": [
       {
         "name": "mage-data-prep-start",
         "image": "[aws-account-id].dkr.ecr.[region].amazonaws.com/[your-image-repo-name]:[tag]",
         "portMappings": [
           {
             "containerPort": 6789,
             "hostPort": 6789,
             "protocol": "tcp"
           }
         ],
         "essential": true,
         "entryPoint": ["sh", "-c"],
         "command": ["mage init default_repo && mage start default_repo"],
         "interactive": true,
         "pseudoTerminal": true
       }
     ],
     "family": "mage-data-prep",
     "networkMode": "awsvpc",
     "requiresCompatibilities": ["FARGATE", "EC2"],
     "cpu": "256",
     "memory": "512",
     "executionRoleArn": "arn:aws:iam::[aws-account-id]:role/[ecs-task-execution-role-name]"
   }
   ```

   You must specify:

   * `"image"` - Docker image URI. If using ECR, you can use the template above
     and fill in the following information:
     | Parameter | Description |
     | - | - |
     | `aws-account-id` | AWS Account ID |
     | `region` | Region in which ECR is being used |
     | `your-image-repo-name` | Name of the repository holding your repo |
     | `tag` | tag for the version of the image to use |
   * `"executionRoleArn"` - Name of the
     [ECS Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
     Assigning this role enables this task to make AWS API calls. This is needed
     to pull the Docker image from an ECR repository. Fill in the following
     information:

     | Parameter | Description |
     | - | - |
     | `aws-account-id` | AWS Account ID |
     | `ecs-task-execution-role-name` | Name of task execution IAM role |

     This parameter can be ignored if you are not using Amazon ECR

   In addition, you may want to edit the allocated CPU and Memory resources
   based on the type of data you plan to handle and intensity of the
   computations performed.

4. Launch the task in your new cluster. Make sure the following conditions are
   met:
   * Task is launched in a VPC with a public subnet and has a Public IP
     assigned. Allows you to connect to the ECS task and use the code editor.
   * Task security group allows an inbound TCP connection to port 6789 from your
     IP (or the IP with which the app will be accessed). Enables you to connect
     to port 6789 where the Mage app is running.
   * The task has network routes to any service that you plan to connect to,
     such as data warehouses or Amazon ECR. Simplest way to add these network
     routes are to add outbound connection rules to the security group.

5. Find the public IP of your task. Then to access the Mage app, go to
   `your-public-ip:6789`. If you followed the previous steps correctly, you
   should be able to access the Mage app running in your cluster.

## Running Mage Pipeline in ECS

If you already have a Mage pipeline developed, you can deploy the pipeline as an
ECS task that periodically executes.

Follow the steps below to setup running a Mage pipeline in ECS.

**Prerequisite:** You must have already created a Mage repository containing the
pipeline you wish to deploy to ECS. Make sure this repository includes any data
files and configuration setting files needed for the pipeline to run.

1. Create a new Docker image for running your pipeline— you can start with [our template](https://github.com/mage-ai/docker/blob/master/Dockerfile). Make sure to add any other environment variables defined locally to the image to be able to access those secrets in the cluster. Upload this Docker image to a repository like Amazon ECR so ECS tasks can pull and deploy containers using the image.

2. If not already created, create a new ECS cluster. This task can be run on both Fargate and EC2 based instances

3. Create a new task definition based off the template below. This task calls `mage run` on your pipeline when started.

   ```json theme={"system"}
   {
     "containerDefinitions": [
       {
         "name": "mage-data-prep-deploy",
         "image": "[aws-account-id].dkr.ecr.[region].amazonaws.com/[your-image-repo-name]:[tag]",
         "essential": true,
         "entryPoint": ["sh", "-c"],
         "command": ["mage run default_repo [your-pipeline-name]"],
         "interactive": true,
         "pseudoTerminal": true
       }
     ],
     "family": "mage-data-prep",
     "networkMode": "awsvpc",
     "requiresCompatibilities": ["FARGATE"],
     "cpu": "1024",
     "memory": "2048",
     "executionRoleArn": "arn:aws:iam::[aws-account-id]:role/[ecs-task-execution-role-name]"
   }
   ```

   You must specify:

   * `"image"` - Docker image URI. If using ECR, you can use the template above
     and fill in the following information:
     | Parameter | Description |
     | - | - |
     | `aws-account-id` | AWS Account ID |
     | `region` | Region in which ECR is being used |
     | `your-image-repo-name` | Name of the repository holding your repo |
     | `tag` | tag for the version of the image to use |

   * `"executionRoleArn"` - Name of the
     [ECS Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
     Assigning this role enables this task to make AWS API calls. This is needed
     to pull the Docker image from an ECR repository. Fill in the following
     information:

     | Parameter | Description |
     | - | - |
     | `aws-account-id` | AWS Account ID |
     | `ecs-task-execution-role-name` | Name of task execution IAM role |

     This parameter can be ignored if you are not using Amazon ECR

   * `your-pipeline-name` - name of the pipeline to run. This pipeline must be
     stored in the repository you added to your Docker image.

   In addition, you may want to edit the allocated CPU and Memory resources
   based on the type of data you plan to handle and intensity of the
   computations performed.

4. Launch the task in your new cluster. Make sure your task has network routes
   to any service that you plan to connect to, such as data warehouses or Amazon
   ECR. Simplest way to add these network routes are to add outbound connection
   rules to the security group.

5. Your pipeline is now running in the ECS task


# AWS ECS deployment architecture
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/aws-ecs-architecture



## Overview

The most common AWS deployment of Mage is with AWS ECS. Our Terraform scripts manage the process of creating an ECS cluster, a load balancer, and a database. [Here's a link](/production/deploying-to-cloud/aws/resources) to the default output of `terraform plan` on AWS ECS.

![Contributing](https://mage-ai.github.io/assets/architecture/aws-ecs.png)

An architecture diagram of a Mage AWS ECS deployment.

We use the following AWS services in our deployment:

## Elastic Container Service (ECS)

Amazon ECS is a service provided by AWS for managing and deploying containerized applications. The ECS service in Terraform is used to define and configure the ECS clusters and services.

ECS allows you to run containers at scale, and the Terraform configuration for ECS service includes specifying the task definition, desired task count, launch type, and other settings related to container orchestration.

The Mage application runs as multiple containers in an Amazon ECS service.

## Application Load Balancer (ALB)

An Application Load Balancer is a service provided by AWS that distributes incoming application traffic across multiple targets, such as EC2 instances, ECS containers, or Lambda functions.

In Terraform, the ALB resource is used to define and configure the load balancer, including listeners, target groups, and rules for routing incoming requests to the appropriate backend targets.

Our ALB is configured to route traffic to the ECS cluster and provides a public URL for communication with clients. Read more [here](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-ip-address-type.html).

## Elastic File System (EFS)

Amazon EFS provides a fully managed, scalable, and shared file storage service that can be used with AWS services and on-premises resources.

We use EFS to store a persistent volume where the `mage_data` for your project is stored. You can learn more about what is stored in `mage_data` [here](/design/abstractions/project-structure#project-data). Without EFS, your project data would be lost when your ECS service is terminated. EFS allows us to persist data between containers within ECS.

## Relational Database Service (RDS)

The RDS Instance resource is used to define and configure a specific database instance within the Amazon RDS service.

We use a Postgres RDS to hold the the backend database for your Mage instance. This database contains the metadata that powers Mage. Read more [here](/design/abstractions/project-structure#project-data).

## Virtual Private Cloud (VPC)

Amazon VPC allows you to create a virtual network within AWS, enabling you to isolate and control resources in a private, logically isolated section of the AWS cloud.

The VPC resource in Terraform is used to define and configure the virtual network, including its subnets, route tables, security groups, and other networking-related settings.

The Mage application, ALB, ECS, EFS, and RDS are all deployed within a VPC in our Terraform scripts.

## Read More

You can refer to the instructions found [here](/production/deploying-to-cloud/aws/setup) to learn more about deploying to ECS.


# Deploy to AWS ECS with AWS CodePipeline
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/code-pipeline



## Pre-requisites

### Terraform and AWS ECS

This document assumes you're using Terraform to modify and deploy your AWS instructure. If you're
not using Terraform yet, you can set it up by reading [this document](production/deploying-to-cloud/using-terraform).

This document also assumes that you have already set up your ECS cluster, services, and tasks.

* If you are not using ECS to run Mage, then this document will not apply for you.
  You can submit a feature request in our [Slack channel](https://www.mage.ai/chat) if you want Mage to
  support deploying to your infrastructure through AWS CodePipeline.
* If you have not set up your ECS yet, you can check out [this document](/production/deploying-to-cloud/aws/setup#deploy-to-ecs-cluster)
  to set it up with Terraform.

### AWS CodeCommit

The pipeline that is created through the Terraform scripts has a AWS CodeCommit repo as a source. You
will need to have a CodeCommit repo already set up that has your Mage repository code.

Your CodeCommit repository should have a Dockerfile in the top level of your CodeCommit.
[Here is a reference Dockerfile](https://github.com/mage-ai/docker/blob/master/Dockerfile)
that you can use for your own Dockerfile.

You will also need to add a `buildspec.yml` file to your repository.
The `buildspec.yml` file can follow the following format:

```
version: 0.2

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...          
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG      
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
```

### AWS ECR

One pipeline created by the script will push a Docker build to AWS ECR, and the other pipeline will read from the same
AWS ECR repo. Make sure you have already created your AWS ECR repository.

## Setting up AWS CodePipeline pipeline with Terraform

Use Terraform scripts in [aws-code-pipeline](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/aws-code-pipeline)
folder to provision resources for AWS CodePipeline.

***

### Terraform plan

You can run the following command to see all the resources that will be created
by Terraform:

```bash theme={"system"}
cd aws
terraform plan
```

***

### 1. Environment variables

If you don’t have the AWS CLI installed, you’ll need to create this file:
`~/.aws/credentials`.

In that file, add the following values:

```
[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX
```

Alternatively, you can install the AWS CLI by following
[Amazon’s instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

***

### 2. Customize Terraform variables

In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-code-pipeline/variables.tf),
you will need to update the default values for the following variables:

**AWS settings**

```
variable "AWS_ACCESS_KEY_ID" {
  type    = string
  default = "AWS_ACCESS_KEY_ID"
}

variable "AWS_SECRET_ACCESS_KEY" {
  type    = string
  default = "AWS_SECRET_ACCESS_KEY"
}

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "us-west-2"
}

variable "aws_account_id" {
  type        = string
  description = "AWS Account ID"
  default     = "12345678910"
}
```

**S3 bucket name**

```
variable "s3_bucket_name" {
  type        = string
  description = "S3 bucket to store pipeline artifacts"
  default     = "s3-bucket-name"
}
```

**Code pipeline name (optional)**

```
variable "code_pipeline_name" {
  type        = string
  description = "Name for AWS CodePipeline"
  default     = "mage-codepipeline"
}
```

**ECR repo values**

```
variable "ecr_repo_name" {
  type        = string
  description = "ECR repo name"
  default     = "ecr-repo-name"
}

variable "ecr_image_tag" {
  type        = string
  description = "ECR docker image tag"
  default     = "latest"
}
```

**ECS values**

```
variable "ecs_cluster_name" {
  type        = string
  description = "ECS cluster name"
  default     = "ecs-cluster-name"
}

variable "ecs_service_name" {
  type        = string
  description = "ECS service name"
  default     = "ecs-service-name"
}

variable "ecs_container_name" {
  type        = string
  description = "ECS container name"
  default     = "ecs-container-name"
}

variable "ecs_task_role_name" {
  type        = string
  description = "ECS execution task role name"
  default     = "ecs-task-role-name"
}
```

**AWS CodeCommit values**

```
variable "codecommit_repo_name" {
  type        = string
  description = "Codecommit repo name"
  default     = "codecommit_repo_name"
}

variable "codecommit_branch" {
  type        = string
  description = "Codecommit repo branch"
  default     = "main"
}
```

### 3. IAM Permissions

The IAM permissions are defined in the `aws-code-pipeline/iam.tf` file. Make adjustments to the
permissions as needed.

### 4. Apply Terraform

**Using CLI**

1. Change directory into scripts folder:
   ```bash theme={"system"}
   cd aws-code-pipeline
   ```
2. Initialize Terraform:
   ```bash theme={"system"}
   terraform init
   ```
   * If you run into errors like the following:
     ```
     │ Error: Failed to install provider
     │
     │ Error while installing hashicorp/template v2.2.0: the local package for registry.terraform.io/hashicorp/template 2.2.0 doesn't match any of the checksums previously recorded in the dependency lock file (this might be because the available checksums are for packages targeting different
     │ platforms); for more information: https://www.terraform.io/language/provider-checksum-verification
     ```
     then run the following commands to resolve this:
     ```
     brew install kreuzwerker/taps/m1-terraform-provider-helper
     m1-terraform-provider-helper activate
     m1-terraform-provider-helper install hashicorp/template -v v2.2.0
     rm .terraform.lock.hcl
     terraform init --upgrade
     ```
3. Deploy:
   ```bash theme={"system"}
   terraform apply
   ```

### Results

After running the Terraform script, you should see two AWS CodePipeline pipelines created, `mage-codepipeline`
and `mage-codepipeline-ecr` (unless you changed the `code_pipeline_name` variable). The `mage-codepipeline` pipeline
will be triggered every time a change is detected in the AWS CodeCommit repo. The `mage-codepipeline-ecr` pipeline
will be triggered every time a new image is pushed with into the AWS ECR repo.

![mage-codepipeline](https://mage-ai.github.io/assets/mage-codepipeline.png)

![mage-codepipeline-ecr](https://mage-ai.github.io/assets/mage-codepipeline-ecr.png)


# EMR policies
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/emr-policy



```json theme={"system"}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:AddJobFlowSteps",
        "elasticmapreduce:CancelSteps",
        "elasticmapreduce:DescribeCluster",
        "elasticmapreduce:DescribeStep",
        "elasticmapreduce:ListClusters",
        "elasticmapreduce:ListSteps",
        "elasticmapreduce:ModifyCluster",
        "elasticmapreduce:ModifyInstanceFleet",
        "elasticmapreduce:ModifyInstanceGroups",
        "elasticmapreduce:PutAutoScalingPolicy",
        "elasticmapreduce:PutAutoTerminationPolicy",
        "elasticmapreduce:PutManagedScalingPolicy",
        "elasticmapreduce:RemoveAutoScalingPolicy",
        "elasticmapreduce:RemoveAutoTerminationPolicy",
        "elasticmapreduce:RemoveManagedScalingPolicy",
        "elasticmapreduce:RunJobFlow",
        "elasticmapreduce:SetTerminationProtection",
        "elasticmapreduce:TerminateJobFlows"
      ],
      "Resource": "*"
    }
  ]
}
```


# Terraform AWS resources
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/resources



Here is the default output of `terraform plan`:

```bash theme={"system"}
data.http.myip: Reading...
data.http.myip: Read complete after 0s [id=http://ipv4.icanhazip.com]
data.aws_iam_policy_document.assume_role_policy: Reading...
data.aws_iam_policy_document.assume_role_policy: Read complete after 0s [id=320642683]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_ecs_task_definition.main will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_ecs_task_definition" "main" {
      + arn                  = (known after apply)
      + arn_without_revision = (known after apply)
      + execution_role_arn   = (known after apply)
      + family               = (known after apply)
      + id                   = (known after apply)
      + network_mode         = (known after apply)
      + revision             = (known after apply)
      + status               = (known after apply)
      + task_definition      = "mage-data-prep-task"
      + task_role_arn        = (known after apply)
    }

  # data.template_file.env_vars will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "env_vars" {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = jsonencode(
            [
              + {
                  + name  = "ENV"
                  + value = "production"
                },
              + {
                  + name  = "AWS_ACCESS_KEY_ID"
                  + value = "${aws_access_key_id}"
                },
              + {
                  + name  = "AWS_SECRET_ACCESS_KEY"
                  + value = "${aws_secret_access_key}"
                },
              + {
                  + name  = "AWS_REGION_NAME"
                  + value = "${aws_region_name}"
                },
              + {
                  + name  = "MAGE_DATABASE_CONNECTION_URL"
                  + value = "${database_connection_url}"
                },
              + {
                  + name  = "MAGE_EC2_SUBNET_ID"
                  + value = "${ec2_subnet_id}"
                },
            ]
        )
      + vars     = {
          + "aws_access_key_id"       = "AWS_ACCESS_KEY_ID"
          + "aws_region_name"         = "us-west-2"
          + "aws_secret_access_key"   = "AWS_SECRET_ACCESS_KEY"
          + "database_connection_url" = (sensitive value)
          + "ec2_subnet_id"           = (known after apply)
        }
    }

  # aws_alb.application_load_balancer will be created
  + resource "aws_alb" "application_load_balancer" {
      + arn                                         = (known after apply)
      + arn_suffix                                  = (known after apply)
      + desync_mitigation_mode                      = "defensive"
      + dns_name                                    = (known after apply)
      + drop_invalid_header_fields                  = false
      + enable_deletion_protection                  = false
      + enable_http2                                = true
      + enable_tls_version_and_cipher_suite_headers = false
      + enable_waf_fail_open                        = false
      + enable_xff_client_port                      = false
      + id                                          = (known after apply)
      + idle_timeout                                = 60
      + internal                                    = false
      + ip_address_type                             = (known after apply)
      + load_balancer_type                          = "application"
      + name                                        = "mage-data-prep-production-alb"
      + preserve_host_header                        = false
      + security_groups                             = (known after apply)
      + subnets                                     = (known after apply)
      + tags                                        = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-alb"
        }
      + tags_all                                    = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-alb"
        }
      + vpc_id                                      = (known after apply)
      + xff_header_processing_mode                  = "append"
      + zone_id                                     = (known after apply)
    }

  # aws_cloudwatch_log_group.log-group will be created
  + resource "aws_cloudwatch_log_group" "log-group" {
      + arn               = (known after apply)
      + id                = (known after apply)
      + name              = "mage-data-prep-production-logs"
      + name_prefix       = (known after apply)
      + retention_in_days = 0
      + skip_destroy      = false
      + tags              = {
          + "Application" = "mage-data-prep"
          + "Environment" = "production"
        }
      + tags_all          = {
          + "Application" = "mage-data-prep"
          + "Environment" = "production"
        }
    }

  # aws_db_instance.rds will be created
  + resource "aws_db_instance" "rds" {
      + address                               = (known after apply)
      + allocated_storage                     = 20
      + apply_immediately                     = false
      + arn                                   = (known after apply)
      + auto_minor_version_upgrade            = true
      + availability_zone                     = (known after apply)
      + backup_retention_period               = (known after apply)
      + backup_window                         = (known after apply)
      + ca_cert_identifier                    = (known after apply)
      + character_set_name                    = (known after apply)
      + copy_tags_to_snapshot                 = false
      + db_name                               = "mage"
      + db_subnet_group_name                  = (known after apply)
      + delete_automated_backups              = true
      + endpoint                              = (known after apply)
      + engine                                = "postgres"
      + engine_version                        = "13.7"
      + engine_version_actual                 = (known after apply)
      + hosted_zone_id                        = (known after apply)
      + id                                    = (known after apply)
      + identifier                            = "mage-data-prep-production-db"
      + identifier_prefix                     = (known after apply)
      + instance_class                        = "db.t3.micro"
      + iops                                  = (known after apply)
      + kms_key_id                            = (known after apply)
      + latest_restorable_time                = (known after apply)
      + license_model                         = (known after apply)
      + listener_endpoint                     = (known after apply)
      + maintenance_window                    = (known after apply)
      + master_user_secret                    = (known after apply)
      + master_user_secret_kms_key_id         = (known after apply)
      + monitoring_interval                   = 0
      + monitoring_role_arn                   = (known after apply)
      + multi_az                              = false
      + name                                  = (known after apply)
      + nchar_character_set_name              = (known after apply)
      + network_type                          = (known after apply)
      + option_group_name                     = (known after apply)
      + parameter_group_name                  = (known after apply)
      + password                              = (sensitive value)
      + performance_insights_enabled          = false
      + performance_insights_kms_key_id       = (known after apply)
      + performance_insights_retention_period = (known after apply)
      + port                                  = (known after apply)
      + publicly_accessible                   = true
      + replica_mode                          = (known after apply)
      + replicas                              = (known after apply)
      + resource_id                           = (known after apply)
      + skip_final_snapshot                   = true
      + snapshot_identifier                   = (known after apply)
      + status                                = (known after apply)
      + storage_throughput                    = (known after apply)
      + storage_type                          = (known after apply)
      + tags                                  = {
          + "Environment" = "production"
        }
      + tags_all                              = {
          + "Environment" = "production"
        }
      + timezone                              = (known after apply)
      + username                              = "mageuser"
      + vpc_security_group_ids                = (known after apply)
    }

  # aws_db_subnet_group.rds_subnet_group will be created
  + resource "aws_db_subnet_group" "rds_subnet_group" {
      + arn                     = (known after apply)
      + description             = "mage-data-prep RDS subnet group"
      + id                      = (known after apply)
      + name                    = "mage-data-prep-production-rds-subnet-group"
      + name_prefix             = (known after apply)
      + subnet_ids              = (known after apply)
      + supported_network_types = (known after apply)
      + tags                    = {
          + "Environment" = "production"
        }
      + tags_all                = {
          + "Environment" = "production"
        }
      + vpc_id                  = (known after apply)
    }

  # aws_ecs_cluster.aws-ecs-cluster will be created
  + resource "aws_ecs_cluster" "aws-ecs-cluster" {
      + arn                = (known after apply)
      + capacity_providers = (known after apply)
      + id                 = (known after apply)
      + name               = "mage-data-prep-production-cluster"
      + tags               = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-ecs"
        }
      + tags_all           = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-ecs"
        }

      + setting {
          + name  = "containerInsights"
          + value = "enabled"
        }
    }

  # aws_ecs_service.aws-ecs-service will be created
  + resource "aws_ecs_service" "aws-ecs-service" {
      + cluster                            = (known after apply)
      + deployment_maximum_percent         = 200
      + deployment_minimum_healthy_percent = 100
      + desired_count                      = 1
      + enable_ecs_managed_tags            = false
      + enable_execute_command             = false
      + force_new_deployment               = true
      + iam_role                           = (known after apply)
      + id                                 = (known after apply)
      + launch_type                        = "FARGATE"
      + name                               = "mage-data-prep-production-ecs-service"
      + platform_version                   = (known after apply)
      + scheduling_strategy                = "REPLICA"
      + tags_all                           = (known after apply)
      + task_definition                    = (known after apply)
      + triggers                           = (known after apply)
      + wait_for_steady_state              = false

      + load_balancer {
          + container_name   = "mage-data-prep-production-container"
          + container_port   = 6789
          + target_group_arn = (known after apply)
        }

      + network_configuration {
          + assign_public_ip = true
          + security_groups  = (known after apply)
          + subnets          = (known after apply)
        }
    }

  # aws_ecs_task_definition.aws-ecs-task will be created
  + resource "aws_ecs_task_definition" "aws-ecs-task" {
      + arn                      = (known after apply)
      + arn_without_revision     = (known after apply)
      + container_definitions    = (known after apply)
      + cpu                      = "512"
      + execution_role_arn       = (known after apply)
      + family                   = "mage-data-prep-task"
      + id                       = (known after apply)
      + memory                   = "1024"
      + network_mode             = "awsvpc"
      + requires_compatibilities = [
          + "FARGATE",
        ]
      + revision                 = (known after apply)
      + skip_destroy             = false
      + tags                     = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-ecs-td"
        }
      + tags_all                 = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-ecs-td"
        }
      + task_role_arn            = (known after apply)

      + volume {
          + name = "mage-data-prep-fs"

          + efs_volume_configuration {
              + file_system_id     = (known after apply)
              + root_directory     = "/"
              + transit_encryption = "ENABLED"
            }
        }
    }

  # aws_efs_file_system.file_system will be created
  + resource "aws_efs_file_system" "file_system" {
      + arn                     = (known after apply)
      + availability_zone_id    = (known after apply)
      + availability_zone_name  = (known after apply)
      + creation_token          = (known after apply)
      + dns_name                = (known after apply)
      + encrypted               = true
      + id                      = (known after apply)
      + kms_key_id              = (known after apply)
      + number_of_mount_targets = (known after apply)
      + owner_id                = (known after apply)
      + performance_mode        = "generalPurpose"
      + size_in_bytes           = (known after apply)
      + tags                    = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-efs"
        }
      + tags_all                = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-efs"
        }
      + throughput_mode         = "elastic"
    }

  # aws_efs_mount_target.mount_target[0] will be created
  + resource "aws_efs_mount_target" "mount_target" {
      + availability_zone_id   = (known after apply)
      + availability_zone_name = (known after apply)
      + dns_name               = (known after apply)
      + file_system_arn        = (known after apply)
      + file_system_id         = (known after apply)
      + id                     = (known after apply)
      + ip_address             = (known after apply)
      + mount_target_dns_name  = (known after apply)
      + network_interface_id   = (known after apply)
      + owner_id               = (known after apply)
      + security_groups        = (known after apply)
      + subnet_id              = (known after apply)
    }

  # aws_efs_mount_target.mount_target[1] will be created
  + resource "aws_efs_mount_target" "mount_target" {
      + availability_zone_id   = (known after apply)
      + availability_zone_name = (known after apply)
      + dns_name               = (known after apply)
      + file_system_arn        = (known after apply)
      + file_system_id         = (known after apply)
      + id                     = (known after apply)
      + ip_address             = (known after apply)
      + mount_target_dns_name  = (known after apply)
      + network_interface_id   = (known after apply)
      + owner_id               = (known after apply)
      + security_groups        = (known after apply)
      + subnet_id              = (known after apply)
    }

  # aws_iam_role.ecsTaskExecutionRole will be created
  + resource "aws_iam_role" "ecsTaskExecutionRole" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ecs-tasks.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "mage-data-prep-execution-task-role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + role_last_used        = (known after apply)
      + tags                  = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-iam-role"
        }
      + tags_all              = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-iam-role"
        }
      + unique_id             = (known after apply)
    }

  # aws_iam_role_policy_attachment.ecsTaskExecutionRole_policy will be created
  + resource "aws_iam_role_policy_attachment" "ecsTaskExecutionRole_policy" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
      + role       = "mage-data-prep-execution-task-role"
    }

  # aws_internet_gateway.aws-igw will be created
  + resource "aws_internet_gateway" "aws-igw" {
      + arn      = (known after apply)
      + id       = (known after apply)
      + owner_id = (known after apply)
      + tags     = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-igw"
        }
      + tags_all = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-igw"
        }
      + vpc_id   = (known after apply)
    }

  # aws_lb_listener.listener will be created
  + resource "aws_lb_listener" "listener" {
      + arn               = (known after apply)
      + id                = (known after apply)
      + load_balancer_arn = (known after apply)
      + port              = 80
      + protocol          = "HTTP"
      + ssl_policy        = (known after apply)
      + tags_all          = (known after apply)

      + default_action {
          + order            = (known after apply)
          + target_group_arn = (known after apply)
          + type             = "forward"
        }
    }

  # aws_lb_target_group.target_group will be created
  + resource "aws_lb_target_group" "target_group" {
      + arn                                = (known after apply)
      + arn_suffix                         = (known after apply)
      + connection_termination             = false
      + deregistration_delay               = "300"
      + id                                 = (known after apply)
      + ip_address_type                    = (known after apply)
      + lambda_multi_value_headers_enabled = false
      + load_balancing_algorithm_type      = (known after apply)
      + load_balancing_cross_zone_enabled  = (known after apply)
      + name                               = "mage-data-prep-production-tg"
      + port                               = 6789
      + preserve_client_ip                 = (known after apply)
      + protocol                           = "HTTP"
      + protocol_version                   = (known after apply)
      + proxy_protocol_v2                  = false
      + slow_start                         = 0
      + tags                               = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-lb-tg"
        }
      + tags_all                           = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-lb-tg"
        }
      + target_type                        = "ip"
      + vpc_id                             = (known after apply)

      + health_check {
          + enabled             = true
          + healthy_threshold   = 3
          + interval            = 30
          + matcher             = "200"
          + path                = "/api/kernels"
          + port                = "traffic-port"
          + protocol            = "HTTP"
          + timeout             = 5
          + unhealthy_threshold = 2
        }
    }

  # aws_route.public will be created
  + resource "aws_route" "public" {
      + destination_cidr_block = "0.0.0.0/0"
      + gateway_id             = (known after apply)
      + id                     = (known after apply)
      + instance_id            = (known after apply)
      + instance_owner_id      = (known after apply)
      + network_interface_id   = (known after apply)
      + origin                 = (known after apply)
      + route_table_id         = (known after apply)
      + state                  = (known after apply)
    }

  # aws_route_table.public will be created
  + resource "aws_route_table" "public" {
      + arn              = (known after apply)
      + id               = (known after apply)
      + owner_id         = (known after apply)
      + propagating_vgws = (known after apply)
      + route            = (known after apply)
      + tags             = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-routing-table-public"
        }
      + tags_all         = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-routing-table-public"
        }
      + vpc_id           = (known after apply)
    }

  # aws_route_table_association.public[0] will be created
  + resource "aws_route_table_association" "public" {
      + id             = (known after apply)
      + route_table_id = (known after apply)
      + subnet_id      = (known after apply)
    }

  # aws_route_table_association.public[1] will be created
  + resource "aws_route_table_association" "public" {
      + id             = (known after apply)
      + route_table_id = (known after apply)
      + subnet_id      = (known after apply)
    }

  # aws_security_group.load_balancer_security_group will be created
  + resource "aws_security_group" "load_balancer_security_group" {
      + arn                    = (known after apply)
      + description            = "Managed by Terraform"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 0
              + ipv6_cidr_blocks = [
                  + "::/0",
                ]
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "11.111.11.111/32",
                ]
              + description      = ""
              + from_port        = 443
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 443
            },
          + {
              + cidr_blocks      = [
                  + "11.111.11.111/32",
                ]
              + description      = ""
              + from_port        = 80
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 80
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-sg"
        }
      + tags_all               = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-sg"
        }
      + vpc_id                 = (known after apply)
    }

  # aws_security_group.mount_target_security_group will be created
  + resource "aws_security_group" "mount_target_security_group" {
      + arn                    = (known after apply)
      + description            = "Managed by Terraform"
      + egress                 = (known after apply)
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = []
              + description      = ""
              + from_port        = 2049
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = (known after apply)
              + self             = false
              + to_port          = 2049
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-efs-sg"
        }
      + tags_all               = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-efs-sg"
        }
      + vpc_id                 = (known after apply)
    }

  # aws_security_group.rds_sg will be created
  + resource "aws_security_group" "rds_sg" {
      + arn                    = (known after apply)
      + description            = "mage-data-prep RDS Security Group"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = []
              + description      = ""
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = true
              + to_port          = 0
            },
          + {
              + cidr_blocks      = []
              + description      = ""
              + from_port        = 5432
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = (known after apply)
              + self             = false
              + to_port          = 5432
            },
        ]
      + name                   = "mage-data-prep-production-rds-sg"
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-production-rds-sg"
        }
      + tags_all               = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-production-rds-sg"
        }
      + vpc_id                 = (known after apply)
    }

  # aws_security_group.service_security_group will be created
  + resource "aws_security_group" "service_security_group" {
      + arn                    = (known after apply)
      + description            = "Managed by Terraform"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 0
              + ipv6_cidr_blocks = [
                  + "::/0",
                ]
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "11.111.11.111/32",
                ]
              + description      = ""
              + from_port        = 6789
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = (known after apply)
              + self             = false
              + to_port          = 6789
            },
        ]
      + name                   = (known after apply)
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-service-sg"
        }
      + tags_all               = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-service-sg"
        }
      + vpc_id                 = (known after apply)
    }

  # aws_subnet.private[0] will be created
  + resource "aws_subnet" "private" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-west-2a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.32.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-private-subnet-1"
        }
      + tags_all                                       = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-private-subnet-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_subnet.private[1] will be created
  + resource "aws_subnet" "private" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-west-2b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.32.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-private-subnet-2"
        }
      + tags_all                                       = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-private-subnet-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_subnet.public[0] will be created
  + resource "aws_subnet" "public" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-west-2a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.32.100.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = true
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-public-subnet-1"
        }
      + tags_all                                       = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-public-subnet-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_subnet.public[1] will be created
  + resource "aws_subnet" "public" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-west-2b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.32.101.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = true
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-public-subnet-2"
        }
      + tags_all                                       = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-public-subnet-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_vpc.aws-vpc will be created
  + resource "aws_vpc" "aws-vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.32.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_classiclink                   = (known after apply)
      + enable_classiclink_dns_support       = (known after apply)
      + enable_dns_hostnames                 = true
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-vpc"
        }
      + tags_all                             = {
          + "Environment" = "production"
          + "Name"        = "mage-data-prep-vpc"
        }
    }

Plan: 28 to add, 0 to change, 0 to destroy.
```


# Deploy to AWS with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/setup



![](https://user-images.githubusercontent.com/78053898/198754745-9e462ffc-1214-47d2-8093-05c88949c08e.png)

## Pre-requisites

### Setup

[Download Mage maintained Terraform scripts](/production/deploying-to-cloud/using-terraform).

### Permissions

In order to create all the required resources in AWS, your IAM user account must
have the following
[permissions](/production/deploying-to-cloud/aws/terraform-apply-policy).

In order to terminate and remove all the resources created from Terraform, you
need the following
[permissions](/production/deploying-to-cloud/aws/terraform-destroy-policy).

***

## Deploy to ECS cluster

Use Terraform scripts in [aws](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/aws)
folder to provision resources for AWS ECS cluster.

***

### Terraform plan

You can run the following command to see all the resources that will be created
by Terraform:

```bash theme={"system"}
cd aws
terraform plan
```

By default, here are the **[resources](/production/deploying-to-cloud/aws/resources)** that will be
created.

***

### 1. Environment variables

If you don’t have the AWS CLI installed, you’ll need to create this file:
`~/.aws/credentials`.

In that file, add the following values:

```
[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX
```

Alternatively, you can install the AWS CLI by following
[Amazon’s instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

***

### 2. Customize Terraform settings

**Application name (optional)**

In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you can change the default application name that will appear in AWS ECS:

```
variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}
```

**Docker image (optional)**

In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you can change the default Docker image:

```
variable "docker_image" {
  description = "Docker image url used in ECS task."
  default     = "mageai/mageai:latest"
}
```

> [!NOTE]
>
> **Custom Docker images**
>
> If you previously tagged a Docker image you built when following this
> [CI/CD guide](/production/ci-cd), you must push that locally tagged Docker
> image to Amazon Elastic Container Registry (ECR) before deploying using
> Terraform. Read the
> [AWS documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
> to learn more.

1. `docker build --platform linux/amd64 -tag [image_name]:latest .`
2. `docker tag [image_name]:latest [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest`
3. `docker push [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest`

**Region (optional)**

In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you can change the region:

```
variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "us-west-2"
}
```

**Availability zones (optional)**

In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you must change the availability zones to match your region from above:

```
variable "availability_zones" {
  description = "List of availability zones"
  default     = ["us-west-2a", "us-west-2b"]
}
```

**More**

Other variables defined in
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf)
can also be customized to your needs.

***

### 3. Configurable environment variables

In the
[`./aws/env_vars.json`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/env_vars.json)
file, you can edit the following variables, which are used by the tool while
running in the cloud:

Change the value of the variables with the following names to match the actual
values you want the tool to use while running in the cloud:

* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`

These variable values are used by the tool to retrieve AWS resources like
CloudWatch events, etc.

**Other environment variables**

You can add any environment variable you want in this file. These will be set on
the running container.

> Values
>
> The values in your `env_vars.json` must be strings. Wrap each value in double
> quotes.

***

### 4. Deploy

**Using CLI**

1. Change directory into scripts folder:
   ```bash theme={"system"}
   cd aws
   ```
2. Initialize Terraform:
   ```bash theme={"system"}
   terraform init
   ```
   * If you run into errors like the following:
     ```
     │ Error: Failed to install provider
     │
     │ Error while installing hashicorp/template v2.2.0: the local package for registry.terraform.io/hashicorp/template 2.2.0 doesn't match any of the checksums previously recorded in the dependency lock file (this might be because the available checksums are for packages targeting different
     │ platforms); for more information: https://www.terraform.io/language/provider-checksum-verification
     ```
     then run the following commands to resolve this:
     ```
     brew install kreuzwerker/taps/m1-terraform-provider-helper
     m1-terraform-provider-helper activate
     m1-terraform-provider-helper install hashicorp/template -v v2.2.0
     rm .terraform.lock.hcl
     terraform init --upgrade
     ```
3. Deploy:
   ```bash theme={"system"}
   terraform apply
   ```

Once it’s finished deploying, you can access Mage in your browser.

1. Open your EC2 dashboard.
2. View all load balancers.
3. Click on the load balancer with the name `mage-data-prep` in it (if you
   changed the app name, then find the load balancer with that app name).
4. Find the public DNS name, copy that, and paste it in your browser.

***

**Using Docker**

From the root directory of Mage, run the following commands:

1. Initialize Terraform:

```bash theme={"system"}
docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
  hashicorp/terraform:latest init
```

2. Deploy:

```bash theme={"system"}
docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
  --env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  --env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  hashicorp/terraform:latest apply
```

***

### Errors

**Page isn’t loading**

If you run into connection issues, check to see if your IP is whitelisted in the
appropriate [security group](#security).

**503 Forbidden**

Check to see if the service task in your ECS cluster is running or if it
stopped.

503 typically means that the service task isn’t running and that can be caused
by a variety of things.

Open the service task that stopped running and click on the "Logs" tab to see
what issues occurred.

**`exec /bin/sh: exec format error`**

If your cluster is running and the service launches a task that eventually
stops, check the logs for the stopped task. If the logs contain the error above,
try re-building your image with the platform flag `--platform linux/amd64`.

***

### Updating Mage versions

After Mage is running in your cloud, you can update the Mage Docker image
version by running the following command in your CLI tool:

```bash theme={"system"}
aws ecs update-service --cluster [cluster_name] --service [service_name] --force-new-deployment
```

> `cluster_name`
>
> This is the name of your AWS ECS cluster. Go to the AWS ECS dashboard to view
> the clusters and find the cluster with the name that contains the `app_name`
> from `variables.tf`.

> `service_name`
>
> This is the name of your AWS ECS service running in your AWS ECS cluster with
> the name identified from above.

***

### Misc

#### Security

To enable other IP addresses access to Mage, open the security group named
`mage-data-prep-sg` to whitelist IPs.

Add a new inbound rule for HTTP port 80 and use your IP address.

#### HTTPS enabling

Terraform creates a load balancer with a HTTP listener by default.

To enable HTTPS for your cloud app, you need to have SSL certificates. You can
create a SSL certificate in
[AWS Certificate Manager](https://console.aws.amazon.com/acm/home?#/certificates/list).

After you get a certificate, add a HTTPS listener in your load balancer group.

1. Open your EC2 dashboard.
2. View all load balancers.
3. Click on the load balancer with the name `mage-data-prep` in it (if you
   changed the app name, then find the load balancer with that app name).
4. Click "Listeners and rules" tab.
5. Click "Add listener" button
6. For "Protocol" and "Port", choose HTTPS and keep the default port or enter a
   different port.
7. In "Default actions", select "Forward to target groups" and forward to the target group that
   contains your app name (`mage-data-prep-production-tg` by default)
8. In "Security Policy", keep the default security policy.
9. For "Default SSL certificate", do one of the following:
   1. If you created or imported a certificate using AWS Certificate Manager,
      choose From ACM and choose the certificate.
   2. If you uploaded a certificate using IAM, choose From IAM and choose the
      certificate.
10. Click "Add" to create the HTTPS listener.
11. Edit your HTTP listener, change the default action to redirect traffic to
    your HTTPS listener.

The detailed instructions can be found in:
[https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html)

#### Terminate all resources

If you want to delete all resources created from deploying, run the following
command:

```bash theme={"system"}
terraform destroy
```

A sample output could look like this:

```
Destroy complete! Resources: 10 destroyed.
```

### Authentication with ECS task execution role

Mage creates an ECS task execution role with the format `{app_name}-execution-task-role`.
You can grant the ECS task permissions to access other AWS services by attaching IAM policies to this ECS task execution role.

For example, if you want to list the AWS events that can be used for event triggers, you can attach IAM Policy `CloudWatchEventsReadOnlyAccess`
to your ECS task execution role.

***

## Deploy to EKS cluster

Use Terraform scripts in [aws-eks](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/aws-eks)
folder to provision resources for AWS EKS cluster.

You can run the following command to see all the resources that will be created
by Terraform:

```bash theme={"system"}
cd aws-eks
terraform plan
```

Similar with deploying to ECS cluster, you need to set up the [environment variables](/production/deploying-to-cloud/aws/setup#1-environment-variables) first, and you can customize the Terraform settings in [./aws-eks/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-eks/variables.tf) file.

Then, follow the steps below to set up EKS cluster and deploy Mage.

**1. Apply terraform**

Run `terraform apply` to provision the EKS cluster

**2. Configure kubectl**

Configure `kubectl` with command `aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)`

**3. Configure PVC**

> [!NOTE]
>
> If you are planning to use workspaces in EFS, you will need to add additional configurations to the StorageClass.
>
> ```yaml theme={"system"}
> apiVersion: storage.k8s.io/v1
> kind: StorageClass
> metadata:
>   name: efs-sc
> provisioner: efs.csi.aws.com
> # needed for EFS CSI driver dynamic provisioning
> parameters:
>   provisioningMode: efs-ap
>   fileSystemId: <EFS_VOLUME_ID>
>   directoryPerms: "700"
> ```

Update the `EFS_VOLUME_ID` in [kube/efs-pvc.yaml](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-eks/kube/efs-pvc.yaml) to the efs id in terraform output

**4. Install EFS CSI Driver**

> [!NOTE]
>
> For workspaces support, you may need to install the EFS CSI driver add-on directly from the AWS console. Make sure
> to use a driver version that supports dynamic provisioning.

Run command `kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"` to install efs csi driver

**5. Deploy app to EKS**

Run `kubectl apply -f kube/` to deploy app or use Helm to deploy Mage.

**6. Whitelist IP in EKS node's security group**

**7. Access Mage on EKS**

Get the node IP via command `kubectl get nodes -o wide` and access Mage with `http://[ip]:{node_port}`


# AWS policies for Terraform apply
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/terraform-apply-policy



```json theme={"system"}
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"application-autoscaling:DescribeScalingPolicies",
				"cloudwatch:GetMetricData",
				"ec2:AssociateRouteTable",
				"ec2:AttachInternetGateway",
				"ec2:AuthorizeSecurityGroupEgress",
				"ec2:AuthorizeSecurityGroupIngress",
				"ec2:CreateInternetGateway",
				"ec2:CreateRoute",
				"ec2:CreateRouteTable",
				"ec2:CreateSecurityGroup",
				"ec2:CreateSubnet",
				"ec2:CreateTags",
				"ec2:CreateVpc",
				"ec2:DescribeAccountAttributes",
				"ec2:DescribeAvailabilityZones",
				"ec2:DescribeInternetGateways",
				"ec2:DescribeNetworkAcls",
				"ec2:DescribeRouteTables",
				"ec2:DescribeSecurityGroups",
				"ec2:DescribeSubnets",
				"ec2:DescribeVpcAttribute",
				"ec2:DescribeVpcClassicLink",
				"ec2:DescribeVpcClassicLinkDnsSupport",
				"ec2:DescribeVpcs",
				"ec2:ModifySubnetAttribute",
				"ec2:ModifyVpcAttribute",
				"ec2:RevokeSecurityGroupEgress",
				"ec2:RevokeSecurityGroupEgress",
				"ec2:RevokeSecurityGroupIngress",
				"ecr:BatchCheckLayerAvailability",
				"ecr:CompleteLayerUpload",
				"ecr:CreateRepository",
				"ecr:DescribeImages",
				"ecr:DescribeRepositories",
				"ecr:GetAuthorizationToken",
				"ecr:GetRegistryScanningConfiguration",
				"ecr:InitiateLayerUpload",
				"ecr:ListTagsForResource",
				"ecr:PutImage",
				"ecr:TagResource",
				"ecr:UploadLayerPart",
				"ecs:CreateCluster",
				"ecs:CreateService",
				"ecs:DescribeClusters",
				"ecs:DescribeServices",
				"ecs:DescribeTaskDefinition",
				"ecs:ListAccountSettings",
				"ecs:ListClusters",
				"ecs:ListServices",
				"ecs:RegisterTaskDefinition",
				"ecs:TagResource",
				"ecs:UpdateService",
				"elasticfilesystem:CreateFileSystem",
				"elasticfilesystem:CreateMountTarget",
				"elasticfilesystem:CreateMountTarget",
				"elasticfilesystem:DescribeFileSystems",
				"elasticfilesystem:DescribeLifecycleConfiguration",
				"elasticfilesystem:DescribeMountTargetSecurityGroups",
				"elasticfilesystem:DescribeMountTargets",
				"elasticfilesystem:TagResource",
				"elasticfilesystem:UpdateFileSystem",
				"elasticloadbalancing:AddTags",
				"elasticloadbalancing:CreateListener",
				"elasticloadbalancing:CreateLoadBalancer",
				"elasticloadbalancing:CreateTargetGroup",
				"elasticloadbalancing:DescribeListeners",
				"elasticloadbalancing:DescribeLoadBalancerAttributes",
				"elasticloadbalancing:DescribeLoadBalancers",
				"elasticloadbalancing:DescribeTags",
				"elasticloadbalancing:DescribeTargetGroupAttributes",
				"elasticloadbalancing:DescribeTargetGroups",
				"elasticloadbalancing:ModifyLoadBalancerAttributes",
				"elasticloadbalancing:ModifyTargetGroupAttributes",
				"elasticloadbalancing:SetSecurityGroups",
				"iam:AttachRolePolicy",
				"iam:CreatePolicy",
				"iam:CreateRole",
				"iam:CreateServiceLinkedRole",
				"iam:GetPolicy",
				"iam:GetPolicyVersion",
				"iam:GetRole",
				"iam:ListAttachedRolePolicies",
				"iam:ListRolePolicies",
				"iam:PassRole",
				"iam:TagRole",
				"lambda:CreateFunction",
				"lambda:GetFunction",
				"lambda:GetFunctionCodeSigningConfig",
				"lambda:ListVersionsByFunction",
				"logs:CreateLogGroup",
				"logs:DescribeLogGroups",
				"logs:FilterLogEvents",
				"logs:ListTagsLogGroup",
				"logs:TagResource",
				"rds:AddTagsToResource",
				"rds:CreateDBInstance",
				"rds:CreateDBSubnetGroup",
				"rds:DescribeDBInstances",
				"rds:DescribeDBSubnetGroups",
				"rds:ListTagsForResource",
				"rds:ModifyDBInstance"
			],
			"Resource": "*"
		}
	]
}
```


# AWS policies for Terraform destroy
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/terraform-destroy-policy



```json theme={"system"}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DeleteInternetGateway",
        "ec2:DeleteRoute",
        "ec2:DeleteRouteTable",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSubnet",
        "ec2:DeleteVpc",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeNetworkAcls",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcAttribute",
        "ec2:DescribeVpcClassicLink",
        "ec2:DescribeVpcClassicLinkDnsSupport",
        "ec2:DescribeVpcs",
        "ec2:DetachInternetGateway",
        "ec2:DetachNetworkInterface",
        "ec2:DisassociateRouteTable",
        "ecs:DeleteCluster",
        "ecs:DeleteService",
        "ecs:DeregisterTaskDefinition",
        "ecs:DescribeClusters",
        "ecs:DescribeServices",
        "ecs:DescribeTaskDefinition",
        "ecs:UpdateService",
        "elasticfilesystem:DeleteFileSystem",
        "elasticfilesystem:DeleteMountTarget",
        "elasticfilesystem:DescribeFileSystems",
        "elasticfilesystem:DescribeLifecycleConfiguration",
        "elasticfilesystem:DescribeMountTargetSecurityGroups",
        "elasticfilesystem:DescribeMountTargets",
        "elasticloadbalancing:DeleteListener",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DeleteTargetGroup",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:DescribeTargetGroupAttributes",
        "elasticloadbalancing:DescribeTargetGroups",
        "iam:DeletePolicy",
        "iam:DeleteRole",
        "iam:DetachRolePolicy",
        "iam:GetPolicy",
        "iam:GetPolicyVersion",
        "iam:GetRole",
        "iam:ListAttachedRolePolicies",
        "iam:ListInstanceProfilesForRole",
        "iam:ListPolicyVersions",
        "iam:ListRolePolicies",
        "lambda:DeleteFunction",
        "lambda:GetFunction",
        "lambda:GetFunctionCodeSigningConfig",
        "lambda:ListVersionsByFunction",
        "logs:DeleteLogGroup",
        "logs:DescribeLogGroups",
        "logs:ListTagsLogGroup",
        "rds:DeleteDBInstance",
        "rds:DeleteDBSubnetGroup",
        "rds:DescribeDBInstances",
        "rds:DescribeDBSubnetGroups",
        "rds:ListTagsForResource"
      ],
      "Resource": "*"
    }
  ]
}
```


# Terraform Azure resources
Source: https://docs.mage.ai/production/deploying-to-cloud/azure/resources



Here is the default output of `terraform plan`:

```bash theme={"system"}
data.http.myip: Reading...
data.http.myip: Read complete after 0s [id=http://ipv4.icanhazip.com]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_application_gateway.network will be created
  + resource "azurerm_application_gateway" "network" {
      + id                          = (known after apply)
      + location                    = "westus2"
      + name                        = "mage-data-prep-production-app-gateway"
      + private_endpoint_connection = (known after apply)
      + resource_group_name         = "mage-data-prep-production"

      + backend_address_pool {
          + fqdns        = []
          + id           = (known after apply)
          + ip_addresses = (known after apply)
          + name         = "mage-data-prep-production-backend-pool"
        }

      + backend_http_settings {
          + cookie_based_affinity               = "Disabled"
          + id                                  = (known after apply)
          + name                                = "mage-data-prep-production-http-setting"
          + pick_host_name_from_backend_address = false
          + port                                = 6789
          + probe_id                            = (known after apply)
          + protocol                            = "Http"
          + request_timeout                     = 60
          + trusted_root_certificate_names      = []
        }

      + frontend_ip_configuration {
          + id                            = (known after apply)
          + name                          = "mage-data-prep-production-frontend-ip-config"
          + private_ip_address            = (known after apply)
          + private_ip_address_allocation = "Dynamic"
          + private_link_configuration_id = (known after apply)
          + public_ip_address_id          = (known after apply)
        }

      + frontend_port {
          + id   = (known after apply)
          + name = "mage-data-prep-production-frontend-port"
          + port = 80
        }

      + gateway_ip_configuration {
          + id        = (known after apply)
          + name      = "mage-data-prep-production-ip-configuration"
          + subnet_id = (known after apply)
        }

      + http_listener {
          + frontend_ip_configuration_id   = (known after apply)
          + frontend_ip_configuration_name = "mage-data-prep-production-frontend-ip-config"
          + frontend_port_id               = (known after apply)
          + frontend_port_name             = "mage-data-prep-production-frontend-port"
          + host_names                     = []
          + id                             = (known after apply)
          + name                           = "mage-data-prep-production-listener"
          + protocol                       = "Http"
          + ssl_certificate_id             = (known after apply)
          + ssl_profile_id                 = (known after apply)
        }

      + request_routing_rule {
          + backend_address_pool_id    = (known after apply)
          + backend_address_pool_name  = "mage-data-prep-production-backend-pool"
          + backend_http_settings_id   = (known after apply)
          + backend_http_settings_name = "mage-data-prep-production-http-setting"
          + http_listener_id           = (known after apply)
          + http_listener_name         = "mage-data-prep-production-listener"
          + id                         = (known after apply)
          + name                       = "mage-data-prep-production-routing-rule"
          + priority                   = 10
          + redirect_configuration_id  = (known after apply)
          + rewrite_rule_set_id        = (known after apply)
          + rule_type                  = "Basic"
          + url_path_map_id            = (known after apply)
        }

      + sku {
          + capacity = 2
          + name     = "Standard_v2"
          + tier     = "Standard_v2"
        }

      + ssl_policy {
          + cipher_suites        = (known after apply)
          + disabled_protocols   = (known after apply)
          + min_protocol_version = (known after apply)
          + policy_name          = (known after apply)
          + policy_type          = (known after apply)
        }
    }

  # azurerm_container_group.container_group will be created
  + resource "azurerm_container_group" "container_group" {
      + exposed_port        = (known after apply)
      + fqdn                = (known after apply)
      + id                  = (known after apply)
      + ip_address          = (known after apply)
      + ip_address_type     = "Private"
      + location            = "westus2"
      + name                = "mage-data-prep-production"
      + network_profile_id  = (known after apply)
      + os_type             = "Linux"
      + resource_group_name = "mage-data-prep-production"
      + restart_policy      = "Always"
      + tags                = {
          + "Environment" = "production"
        }

      + container {
          + commands = (known after apply)
          + cpu      = 1
          + image    = "mageai/mageai:latest"
          + memory   = 1.5
          + name     = "mage-data-prep-production-container"

          + ports {
              + port     = 6789
              + protocol = "TCP"
            }

          + volume {
              + empty_dir            = false
              + mount_path           = "/home/src"
              + name                 = "mage-data-prep-fs"
              + read_only            = false
              + share_name           = "mage-data-prep-production-data"
              + storage_account_key  = (sensitive value)
              + storage_account_name = "magedataprepstorage"
            }
        }
    }

  # azurerm_network_profile.containergroup_profile will be created
  + resource "azurerm_network_profile" "containergroup_profile" {
      + container_network_interface_ids = (known after apply)
      + id                              = (known after apply)
      + location                        = "westus2"
      + name                            = "mage-data-prep-production-profile"
      + resource_group_name             = "mage-data-prep-production"

      + container_network_interface {
          + name = "mage-data-prep-production-nic"

          + ip_configuration {
              + name      = "aciipconfig"
              + subnet_id = (known after apply)
            }
        }
    }

  # azurerm_network_security_group.nsg-aci will be created
  + resource "azurerm_network_security_group" "nsg-aci" {
      + id                  = (known after apply)
      + location            = "westus2"
      + name                = "mage-data-prep-production-nsg-aci"
      + resource_group_name = "mage-data-prep-production"
      + security_rule       = [
          + {
              + access                                     = "Allow"
              + description                                = ""
              + destination_address_prefix                 = ""
              + destination_address_prefixes               = [
                  + "10.0.10.0/24",
                ]
              + destination_application_security_group_ids = []
              + destination_port_range                     = ""
              + destination_port_ranges                    = [
                  + "22",
                  + "443",
                  + "445",
                  + "6789",
                  + "8000",
                ]
              + direction                                  = "Inbound"
              + name                                       = "from-gateway-subnet"
              + priority                                   = 100
              + protocol                                   = "Tcp"
              + source_address_prefix                      = ""
              + source_address_prefixes                    = [
                  + "10.0.20.0/24",
                ]
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
          + {
              + access                                     = "Allow"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = ""
              + destination_port_ranges                    = [
                  + "443",
                  + "445",
                  + "80",
                ]
              + direction                                  = "Outbound"
              + name                                       = "to-internet"
              + priority                                   = 100
              + protocol                                   = "Tcp"
              + source_address_prefix                      = "*"
              + source_address_prefixes                    = []
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
          + {
              + access                                     = "Deny"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = "*"
              + destination_port_ranges                    = []
              + direction                                  = "Inbound"
              + name                                       = "DenyAllInBound-Override"
              + priority                                   = 900
              + protocol                                   = "*"
              + source_address_prefix                      = "*"
              + source_address_prefixes                    = []
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
          + {
              + access                                     = "Deny"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = "*"
              + destination_port_ranges                    = []
              + direction                                  = "Outbound"
              + name                                       = "DenyAllOutBound-Override"
              + priority                                   = 900
              + protocol                                   = "*"
              + source_address_prefix                      = "*"
              + source_address_prefixes                    = []
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
        ]
    }

  # azurerm_network_security_group.nsg-public will be created
  + resource "azurerm_network_security_group" "nsg-public" {
      + id                  = (known after apply)
      + location            = "westus2"
      + name                = "mage-data-prep-production-nsg-public"
      + resource_group_name = "mage-data-prep-production"
      + security_rule       = [
          + {
              + access                                     = "Allow"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = ""
              + destination_port_ranges                    = [
                  + "80",
                ]
              + direction                                  = "Inbound"
              + name                                       = "whitelist-inbound-ip"
              + priority                                   = 100
              + protocol                                   = "Tcp"
              + source_address_prefix                      = ""
              + source_address_prefixes                    = [
                  + "11.111.11.111/32",
                ]
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
          + {
              + access                                     = "Allow"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = "65200-65535"
              + destination_port_ranges                    = []
              + direction                                  = "Inbound"
              + name                                       = "AllowInfraComms"
              + priority                                   = 200
              + protocol                                   = "Tcp"
              + source_address_prefix                      = "*"
              + source_address_prefixes                    = []
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
          + {
              + access                                     = "Deny"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = "*"
              + destination_port_ranges                    = []
              + direction                                  = "Inbound"
              + name                                       = "DenyAllInBound-Override"
              + priority                                   = 900
              + protocol                                   = "*"
              + source_address_prefix                      = "*"
              + source_address_prefixes                    = []
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
        ]
    }

  # azurerm_public_ip.public_ip will be created
  + resource "azurerm_public_ip" "public_ip" {
      + allocation_method       = "Static"
      + fqdn                    = (known after apply)
      + id                      = (known after apply)
      + idle_timeout_in_minutes = 4
      + ip_address              = (known after apply)
      + ip_version              = "IPv4"
      + location                = "westus2"
      + name                    = "mage-data-prep-production-public-ip"
      + resource_group_name     = "mage-data-prep-production"
      + sku                     = "Standard"
      + sku_tier                = "Regional"
    }

  # azurerm_resource_group.resource_group will be created
  + resource "azurerm_resource_group" "resource_group" {
      + id       = (known after apply)
      + location = "westus2"
      + name     = "mage-data-prep-production"
    }

  # azurerm_storage_account.aci_storage will be created
  + resource "azurerm_storage_account" "aci_storage" {
      + access_tier                       = (known after apply)
      + account_kind                      = "StorageV2"
      + account_replication_type          = "LRS"
      + account_tier                      = "Standard"
      + allow_nested_items_to_be_public   = true
      + cross_tenant_replication_enabled  = true
      + default_to_oauth_authentication   = false
      + enable_https_traffic_only         = true
      + id                                = (known after apply)
      + infrastructure_encryption_enabled = false
      + is_hns_enabled                    = false
      + large_file_share_enabled          = (known after apply)
      + location                          = "westus2"
      + min_tls_version                   = "TLS1_2"
      + name                              = "magedataprepstorage"
      + nfsv3_enabled                     = false
      + primary_access_key                = (sensitive value)
      + primary_blob_connection_string    = (sensitive value)
      + primary_blob_endpoint             = (known after apply)
      + primary_blob_host                 = (known after apply)
      + primary_connection_string         = (sensitive value)
      + primary_dfs_endpoint              = (known after apply)
      + primary_dfs_host                  = (known after apply)
      + primary_file_endpoint             = (known after apply)
      + primary_file_host                 = (known after apply)
      + primary_location                  = (known after apply)
      + primary_queue_endpoint            = (known after apply)
      + primary_queue_host                = (known after apply)
      + primary_table_endpoint            = (known after apply)
      + primary_table_host                = (known after apply)
      + primary_web_endpoint              = (known after apply)
      + primary_web_host                  = (known after apply)
      + public_network_access_enabled     = true
      + queue_encryption_key_type         = "Service"
      + resource_group_name               = "mage-data-prep-production"
      + secondary_access_key              = (sensitive value)
      + secondary_blob_connection_string  = (sensitive value)
      + secondary_blob_endpoint           = (known after apply)
      + secondary_blob_host               = (known after apply)
      + secondary_connection_string       = (sensitive value)
      + secondary_dfs_endpoint            = (known after apply)
      + secondary_dfs_host                = (known after apply)
      + secondary_file_endpoint           = (known after apply)
      + secondary_file_host               = (known after apply)
      + secondary_location                = (known after apply)
      + secondary_queue_endpoint          = (known after apply)
      + secondary_queue_host              = (known after apply)
      + secondary_table_endpoint          = (known after apply)
      + secondary_table_host              = (known after apply)
      + secondary_web_endpoint            = (known after apply)
      + secondary_web_host                = (known after apply)
      + shared_access_key_enabled         = true
      + table_encryption_key_type         = "Service"

      + blob_properties {
          + change_feed_enabled           = (known after apply)
          + change_feed_retention_in_days = (known after apply)
          + default_service_version       = (known after apply)
          + last_access_time_enabled      = (known after apply)
          + versioning_enabled            = (known after apply)

          + container_delete_retention_policy {
              + days = (known after apply)
            }

          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + delete_retention_policy {
              + days = (known after apply)
            }
        }

      + network_rules {
          + bypass                     = (known after apply)
          + default_action             = (known after apply)
          + ip_rules                   = (known after apply)
          + virtual_network_subnet_ids = (known after apply)

          + private_link_access {
              + endpoint_resource_id = (known after apply)
              + endpoint_tenant_id   = (known after apply)
            }
        }

      + queue_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + hour_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }

          + logging {
              + delete                = (known after apply)
              + read                  = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
              + write                 = (known after apply)
            }

          + minute_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }
        }

      + routing {
          + choice                      = (known after apply)
          + publish_internet_endpoints  = (known after apply)
          + publish_microsoft_endpoints = (known after apply)
        }

      + share_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + retention_policy {
              + days = (known after apply)
            }

          + smb {
              + authentication_types            = (known after apply)
              + channel_encryption_type         = (known after apply)
              + kerberos_ticket_encryption_type = (known after apply)
              + versions                        = (known after apply)
            }
        }
    }

  # azurerm_storage_share.container_share will be created
  + resource "azurerm_storage_share" "container_share" {
      + access_tier          = (known after apply)
      + enabled_protocol     = "SMB"
      + id                   = (known after apply)
      + metadata             = (known after apply)
      + name                 = "mage-data-prep-production-data"
      + quota                = 100
      + resource_manager_id  = (known after apply)
      + storage_account_name = "magedataprepstorage"
      + url                  = (known after apply)
    }

  # azurerm_subnet.sn-aci will be created
  + resource "azurerm_subnet" "sn-aci" {
      + address_prefixes                               = [
          + "10.0.10.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = (known after apply)
      + enforce_private_link_service_network_policies  = (known after apply)
      + id                                             = (known after apply)
      + name                                           = "aci"
      + private_endpoint_network_policies_enabled      = (known after apply)
      + private_link_service_network_policies_enabled  = (known after apply)
      + resource_group_name                            = "mage-data-prep-production"
      + service_endpoints                              = [
          + "Microsoft.Storage",
        ]
      + virtual_network_name                           = "mage-data-prep-production"

      + delegation {
          + name = "acidelegationservice"

          + service_delegation {
              + actions = [
                  + "Microsoft.Network/virtualNetworks/subnets/join/action",
                  + "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                ]
              + name    = "Microsoft.ContainerInstance/containerGroups"
            }
        }
    }

  # azurerm_subnet.sn-public will be created
  + resource "azurerm_subnet" "sn-public" {
      + address_prefixes                               = [
          + "10.0.20.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = (known after apply)
      + enforce_private_link_service_network_policies  = (known after apply)
      + id                                             = (known after apply)
      + name                                           = "public"
      + private_endpoint_network_policies_enabled      = (known after apply)
      + private_link_service_network_policies_enabled  = (known after apply)
      + resource_group_name                            = "mage-data-prep-production"
      + virtual_network_name                           = "mage-data-prep-production"
    }

  # azurerm_subnet_network_security_group_association.sn-nsg-aci will be created
  + resource "azurerm_subnet_network_security_group_association" "sn-nsg-aci" {
      + id                        = (known after apply)
      + network_security_group_id = (known after apply)
      + subnet_id                 = (known after apply)
    }

  # azurerm_subnet_network_security_group_association.sn-nsg-public will be created
  + resource "azurerm_subnet_network_security_group_association" "sn-nsg-public" {
      + id                        = (known after apply)
      + network_security_group_id = (known after apply)
      + subnet_id                 = (known after apply)
    }

  # azurerm_virtual_network.virtual_network will be created
  + resource "azurerm_virtual_network" "virtual_network" {
      + address_space       = [
          + "10.0.0.0/16",
        ]
      + dns_servers         = [
          + "10.0.0.4",
          + "10.0.0.5",
        ]
      + guid                = (known after apply)
      + id                  = (known after apply)
      + location            = "westus2"
      + name                = "mage-data-prep-production"
      + resource_group_name = "mage-data-prep-production"
      + subnet              = (known after apply)
      + tags                = {
          + "Environment" = "production"
        }
    }

Plan: 14 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + id = (known after apply)
  + ip = (known after apply)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
```


# Deploy to Azure with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/azure/setup



![](https://user-images.githubusercontent.com/78053898/198754758-13aed127-73cd-4582-8556-f4f7f84012f9.png)

## Overview

Here is a high level diagram of how Mage interacts with various Azure resources deployed using
this Terraform template.

![Mage Azure architecture](https://mage-ai.github.io/assets/production/deployment/azure/architecture.png)

## Pre-requisites

* Download Mage maintained [Terraform templates](/production/deploying-to-cloud/using-terraform).

***

## 1. Install Azure CLI

Follow these
[instructions](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) to
install the Azure CLI on your workstation.

***

## 2. Log into Azure from CLI

If you don’t already have an account in Azure, create one
[here](https://portal.azure.com/).

Once you created an account, from your terminal run the following command to log
in:

```bash theme={"system"}
az login
```

Your browser will open and have you sign in with an existing Azure account. Once
completed, your terminal will output something like this:

```json theme={"system"}
[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "7712a6c4-1684-402f-8d1a-f1bb656e5d40",
    "id": "d17fcd32-b98c-4f81-a881-42b8e811a0e8",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Azure subscription 1",
    "state": "Enabled",
    "tenantId": "7712a6c4-1684-402f-8d1a-f1bb656e5d40",
    "user": {
      "name": "eng@mage.com",
      "type": "user"
    }
  }
]
```

***

## 3. Customize Terraform settings

**Storage account name (REQUIRED)**

The `storage_account_name` must be unique globally. Before running any Terraform
commands, please change the `default` value of variable named
`storage_account_name` in the
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf)
file.

```js theme={"system"}
variable "storage_account_name" {
  description = "Storage account name. It must be globally unique across Azure."
  default     = "something_very_unique"
}
```

**Key vault name (REQUIRED)**

The `key_vault_name` must be unique globally. Before running any Terraform
commands, please change the `default` value of variable named `key_vault_name`
in the
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf)
file.

```js theme={"system"}
variable "key_vault_name" {
  description = "Key vault name. It must be globally unique across Azure."
  default     = "something_very_unique"
}
```

**Virtual network name**

In the file
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf),
you can change the `default` value under `app_name`:

```js theme={"system"}
variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}
```

**Docker image**

In the file
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf),
you can change the `default` value under `docker_image`:

```js theme={"system"}
variable "docker_image" {
  description = "Docker image url."
  default     = "mageai/mageai:latest"
}
```

> [!NOTE]
>
> **Custom Docker images**
>
> If you previously tagged a Docker image you built when following this
> [CI/CD guide](/production/ci-cd), you must push that locally tagged Docker image
> to Azure Container Registry before deploying using Terraform.

**Region**

In the file
[./azure/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/main.tf),
you can change the `location` value under `resource_group`:

```js theme={"system"}
resource "azurerm_resource_group" "resource_group" {
  name     = "${var.app_name}-${var.app_environment}"
  location = "West US 2"
}
```

### More

Other variables defined in
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf)
can also be customized to your needs.

### Environment variables for application

Set your environment variables in your running cloud environment by adding the
following under the resource named `azurerm_container_group` in the file
[./azure/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/main.tf):

```js theme={"system"}
resource "azurerm_container_group" "container_group" {
  ...

  container {
    ...

    environment_variables = {
      ...
      "ENVIRONMENT_VARIABLE_NAME" = "ENVIRONMENT_VARIABLE_VALUE"
    }
  }
}
```

***

## 4. Deploy

1. Change directory into scripts folder:

   ```bash theme={"system"}
   cd azure
   ```

2. Initialize Terraform

   ```bash theme={"system"}
   terraform init
   ```

   A sample output could look like this:

   ```
   Initializing the backend...

   Initializing provider plugins...
   - Finding latest version of hashicorp/azurerm...
   - Finding latest version of hashicorp/http...
   - Installing hashicorp/azurerm v3.24.0...
   - Installed hashicorp/azurerm v3.24.0 (signed by HashiCorp)
   - Installing hashicorp/http v3.1.0...
   - Installed hashicorp/http v3.1.0 (signed by HashiCorp)

   Terraform has created a lock file .terraform.lock.hcl to record the provider
   selections it made above. Include this file in your version control repository
   so that Terraform can guarantee to make the same selections by default when
   you run "terraform init" in the future.

   Terraform has been successfully initialized!

   You may now begin working with Terraform. Try running "terraform plan" to see
   any changes that are required for your infrastructure. All Terraform commands
   should now work.

   If you ever set or change modules or backend configuration for Terraform,
   rerun this command to reinitialize your working directory. If you forget, other
   commands will detect it and remind you to do so if necessary.
   ```

   ### Terraform plan

   You can run the following command to see all the resources that will be created
   by Terraform: `terraform plan`

   By default, here are the **[resources](/production/deploying-to-cloud/azure/resources)** that will be
   created.

3. Deploy

   ```bash theme={"system"}
   terraform apply
   ```

   A sample output could look like this:

   ```
   Apply complete! Resources: 4 added, 2 changed, 0 destroyed.

   Outputs:

   id = "/subscriptions/d17fcd12-b89c-4f81-a221-40b8e768a0e8/resourceGroups/mage-data-prep2-production/providers/Microsoft.ContainerInstance/containerGroups/mage-data-prep-production"
   ip = "20.3.85.62"
   ```

4. In your browser, go to [`http://[IP_address]/`](http://IP_address/).

   > [!NOTE]
   >
   > Change the `IP_address` to the IP address that was output in your terminal
   > after successfully running `terraform apply`.

***

## Updating Mage versions

After Mage is running in your cloud, you can update the Mage Docker image
version by running the following command in your CLI tool:

```bash theme={"system"}
az container restart --name mage-data-prep-production --resource-group mage-data-prep-production
```

***

## Misc

### Security

By default, `terraform apply` will attempt to add your current workstation’s IP
address to the above mentioned security group.

In order to access Mage on Azure from another local workstation, you must add
that IP address to the security group named
`mage-data-prep-production-nsg-public`.

> Security group name
>
> Your security group might be named differently because you may have changed
> your virtual network name (e.g. `app_name`). The security group you must add
> the IP to will end with `-nsg-public`.

### HTTPS enabling

TBD

### Terminate all resources

If you want to delete all resources created from deploying, run the following
command:

```bash theme={"system"}
terraform destroy
```

A sample output could look like this:

```
Destroy complete! Resources: 10 destroyed.
```

***


# Deploy to DigitalOcean with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/digitalocean/setup



## Pre-requisites

### Setup

[Download Mage maintained Terraform scripts](/production/deploying-to-cloud/using-terraform).

## Terraform plan

You can run the following command to see all the resources that will be created
by Terraform:

```bash theme={"system"}
cd 
terraform plan
```

***

## 1. Configure access token

Generate DigitalOcean access token by following the guide: [https://developer.hashicorp.com/terraform/tutorials/applications/digitalocean-provider#generate-digitalocean-access-token](https://developer.hashicorp.com/terraform/tutorials/applications/digitalocean-provider#generate-digitalocean-access-token)

Set the token in `DIGITALOCEAN_ACCESS_TOKEN` environment variable

***

## 2. Create ssh key

Create ssh key with command: `ssh-keygen`. In the file
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf), update the value of `ssh_pub_key` to your path of public key.

***

## 3. Customize Terraform settings

**Application name (optional)**

In the file
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf),
you can change the default application name that will appear in DigitalOcean:

```
variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}
```

**Region (optional)**

In the file
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf),
you can change the region:

```
variable "region" {
  type        = string
  description = "Region of the datacenters"
  default     = "sfo3"
}
```

**More**

Other variables defined in
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf)
can also be customized to your needs.

***

## 4. Provision resources

**Using CLI**

1. Change directory into scripts folder:
   ```bash theme={"system"}
   cd digitalocean
   ```
2. Initialize Terraform:
   ```bash theme={"system"}
   terraform init
   ```
3. Deploy:
   ```bash theme={"system"}
   terraform apply
   ```

Once it’s finished deploying, you can get the IP from the output.

## 5. Run app in the provisioned Droplet

SSH into the instance: `ssh -i [path_to_private_key] root@[ip]`

Run command to start Mage app in :

```bash theme={"system"}
docker run -d -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start default_repo
```

To update Mage to latest version, you can pull latest docker image with command:

```bash theme={"system"}
docker pull mageai/mageai:latest
```

Then, just restart the Mage app with the same `docker run` command above.

## 6. Open browser with Mage

Once you’ve launched the Droplet and started Mage, you can access Mage from your web browser by entering the following:

`http://[ip]:[port]`

Change the IP to match the IP of your Droplet and the port that you’re using for Mage.

***

## Misc

### Terminate all resources

If you want to delete all resources created from deploying, run the following
command:

```bash theme={"system"}
terraform destroy
```

A sample output could look like this:

```
Destroy complete! Resources: 10 destroyed.
```


# Push Docker Image to GCP Artifact Registry
Source: https://docs.mage.ai/production/deploying-to-cloud/gcp/gcp-artifact-registry



While you may pull an image directly from Docker Hub, here are our legacy instructions for pushing a Docker image to GCP Artifact Registry:

1. Create a repository on
   [GCP Artifact Registry](https://console.cloud.google.com/artifacts) by
   clicking the **`+ CREATE REPOSITORY`** button.

   * Fill in the **Name**
   * Choose Docker as the **Format**
   * Choose any **Location type**
   * Choose any **Encryption**
   * Click **`CREATE`**

2. Click on the newly created repository (e.g. `mage-docker`).

3. Near the top of the page, click the link **`SETUP INSTRUCTIONS`** or read these
   [instructions](https://cloud.google.com/artifact-registry/docs/docker/authentication)
   to set up authentication for Docker.

   * Run the following command in your terminal:

     ```bash theme={"system"}
     gcloud auth configure-docker [region]-docker.pkg.dev
     ```

   * An example command could look like this:

     ```bash theme={"system"}
     gcloud auth configure-docker us-west2-docker.pkg.dev
     ```

4. Pull the Mage Docker image:

   ```bash theme={"system"}
   docker pull mageai/mageai:latest
   ```

   * If your local workstation is using macOS and a silicon chip (e.g. M1, M2,
     etc), then run this command instead:

     ```bash theme={"system"}
     docker pull --platform linux/amd64 mageai/mageai:latest
     ```

5. Tag the pulled Mage docker image or use a previously tagged Docker image you
   built when following this [CI/CD guide](/production/ci-cd):

   * Sample commands if using vanilla Mage Docker image:

     ```bash theme={"system"}
     docker tag mageai/mageai:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest
     ```

     * An example command could look like this:

       ```bash theme={"system"}
       docker tag mageai/mageai:latest us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest
       ```

   * Sample commands if using previously tagged a custom Docker image using the tag `mageprod:latest`:

     ```bash theme={"system"}
     docker tag mageprod:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest
     ```

     * An example command could look like this if you previously tagged your custom Docker image with the tag `mageprod:latest`:

       ```bash theme={"system"}
       docker tag mageprod:latest us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest
       ```

6. Push the local Docker image to GCP Artifact Registry:

   ```bash theme={"system"}
   docker push [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest
   ```

   * An example command could look like this:

     ```bash theme={"system"}
     docker push us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest
     ```


# Terraform GCP resources
Source: https://docs.mage.ai/production/deploying-to-cloud/gcp/resources



Here is the default output of `terraform plan`:

```bash theme={"system"}
data.http.myip: Reading...
data.http.myip: Read complete after 0s [id=http://ipv4.icanhazip.com]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_cloud_run_service.run_service will be created
  + resource "google_cloud_run_service" "run_service" {
      + autogenerate_revision_name = true
      + id                         = (known after apply)
      + location                   = "us-west2"
      + name                       = "mage-data-prep"
      + project                    = (known after apply)
      + status                     = (known after apply)

      + metadata {
          + annotations      = {
              + "run.googleapis.com/ingress"      = "internal-and-cloud-load-balancing"
              + "run.googleapis.com/launch-stage" = "BETA"
            }
          + generation       = (known after apply)
          + labels           = (known after apply)
          + namespace        = (known after apply)
          + resource_version = (known after apply)
          + self_link        = (known after apply)
          + uid              = (known after apply)
        }

      + template {
          + metadata {
              + annotations      = (known after apply)
              + generation       = (known after apply)
              + name             = (known after apply)
              + namespace        = (known after apply)
              + resource_version = (known after apply)
              + self_link        = (known after apply)
              + uid              = (known after apply)
            }

          + spec {
              + container_concurrency = (known after apply)
              + service_account_name  = (known after apply)
              + serving_state         = (known after apply)
              + timeout_seconds       = (known after apply)

              + containers {
                  + image = "region-docker.pkg.dev/project_id/repository_name/mageai"

                  + env {
                      + name  = "FILESTORE_IP_ADDRESS"
                      + value = (known after apply)
                    }
                  + env {
                      + name  = "FILE_SHARE_NAME"
                      + value = "share1"
                    }

                  + ports {
                      + container_port = 6789
                      + name           = (known after apply)
                    }

                  + resources {
                      + limits = {
                          + "cpu"    = "2000m"
                          + "memory" = "1G"
                        }
                    }
                }
            }
        }

      + traffic {
          + latest_revision = true
          + percent         = 100
          + url             = (known after apply)
        }
    }

  # google_cloud_run_service_iam_member.run_all_users will be created
  + resource "google_cloud_run_service_iam_member" "run_all_users" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + location = "us-west2"
      + member   = "allUsers"
      + project  = (known after apply)
      + role     = "roles/run.invoker"
      + service  = "mage-data-prep"
    }

  # google_compute_backend_service.backend will be created
  + resource "google_compute_backend_service" "backend" {
      + connection_draining_timeout_sec = 300
      + creation_timestamp              = (known after apply)
      + fingerprint                     = (known after apply)
      + id                              = (known after apply)
      + load_balancing_scheme           = "EXTERNAL"
      + name                            = "mage-data-prep-backend"
      + port_name                       = "http"
      + project                         = (known after apply)
      + protocol                        = "HTTP"
      + security_policy                 = "mage-data-prep-security-policy"
      + self_link                       = (known after apply)
      + session_affinity                = (known after apply)
      + timeout_sec                     = 30

      + backend {
          + balancing_mode               = "UTILIZATION"
          + capacity_scaler              = 1
          + group                        = (known after apply)
          + max_connections              = (known after apply)
          + max_connections_per_endpoint = (known after apply)
          + max_connections_per_instance = (known after apply)
          + max_rate                     = (known after apply)
          + max_rate_per_endpoint        = (known after apply)
          + max_rate_per_instance        = (known after apply)
          + max_utilization              = (known after apply)
        }

      + cdn_policy {
          + cache_mode                   = (known after apply)
          + client_ttl                   = (known after apply)
          + default_ttl                  = (known after apply)
          + max_ttl                      = (known after apply)
          + negative_caching             = (known after apply)
          + serve_while_stale            = (known after apply)
          + signed_url_cache_max_age_sec = (known after apply)

          + cache_key_policy {
              + include_host           = (known after apply)
              + include_named_cookies  = (known after apply)
              + include_protocol       = (known after apply)
              + include_query_string   = (known after apply)
              + query_string_blacklist = (known after apply)
              + query_string_whitelist = (known after apply)
            }

          + negative_caching_policy {
              + code = (known after apply)
              + ttl  = (known after apply)
            }
        }

      + log_config {
          + enable      = (known after apply)
          + sample_rate = (known after apply)
        }
    }

  # google_compute_global_address.ip will be created
  + resource "google_compute_global_address" "ip" {
      + address            = (known after apply)
      + creation_timestamp = (known after apply)
      + id                 = (known after apply)
      + name               = "mage-data-prep-service-ip"
      + project            = (known after apply)
      + self_link          = (known after apply)
    }

  # google_compute_global_forwarding_rule.frontend will be created
  + resource "google_compute_global_forwarding_rule" "frontend" {
      + id                    = (known after apply)
      + ip_address            = (known after apply)
      + ip_protocol           = (known after apply)
      + label_fingerprint     = (known after apply)
      + load_balancing_scheme = "EXTERNAL"
      + name                  = "mage-data-prep-frontend"
      + network               = (known after apply)
      + port_range            = "80"
      + project               = (known after apply)
      + psc_connection_id     = (known after apply)
      + psc_connection_status = (known after apply)
      + self_link             = (known after apply)
      + target                = (known after apply)
    }

  # google_compute_region_network_endpoint_group.cloudrun_neg will be created
  + resource "google_compute_region_network_endpoint_group" "cloudrun_neg" {
      + id                    = (known after apply)
      + name                  = "mage-data-prep-neg"
      + network_endpoint_type = "SERVERLESS"
      + project               = (known after apply)
      + region                = "us-west2"
      + self_link             = (known after apply)

      + cloud_run {
          + service = "mage-data-prep"
        }
    }

  # google_compute_security_policy.policy will be created
  + resource "google_compute_security_policy" "policy" {
      + fingerprint = (known after apply)
      + id          = (known after apply)
      + name        = "mage-data-prep-security-policy"
      + project     = (known after apply)
      + self_link   = (known after apply)
      + type        = (known after apply)

      + advanced_options_config {
          + json_parsing = (known after apply)
          + log_level    = (known after apply)

          + json_custom_config {
              + content_types = (known after apply)
            }
        }

      + rule {
          + action      = "allow"
          + description = "Whitelist IP"
          + preview     = (known after apply)
          + priority    = 100

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "11.111.11.111/32",
                    ]
                }
            }
        }
      + rule {
          + action      = "deny(403)"
          + description = "default rule"
          + preview     = (known after apply)
          + priority    = 2147483647

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "*",
                    ]
                }
            }
        }
    }

  # google_compute_target_http_proxy.http_proxy will be created
  + resource "google_compute_target_http_proxy" "http_proxy" {
      + creation_timestamp = (known after apply)
      + id                 = (known after apply)
      + name               = "mage-data-prep-http-proxy"
      + project            = (known after apply)
      + proxy_bind         = (known after apply)
      + proxy_id           = (known after apply)
      + self_link          = (known after apply)
      + url_map            = (known after apply)
    }

  # google_compute_url_map.url_map will be created
  + resource "google_compute_url_map" "url_map" {
      + creation_timestamp = (known after apply)
      + default_service    = (known after apply)
      + fingerprint        = (known after apply)
      + id                 = (known after apply)
      + map_id             = (known after apply)
      + name               = "mage-data-prep-urlmap"
      + project            = (known after apply)
      + self_link          = (known after apply)
    }

  # google_filestore_instance.instance will be created
  + resource "google_filestore_instance" "instance" {
      + create_time = (known after apply)
      + etag        = (known after apply)
      + id          = (known after apply)
      + location    = "us-west2-a"
      + name        = "mage-data-prep"
      + project     = (known after apply)
      + tier        = "BASIC_HDD"
      + zone        = (known after apply)

      + file_shares {
          + capacity_gb = 1024
          + name        = "share1"
        }

      + networks {
          + connect_mode      = "DIRECT_PEERING"
          + ip_addresses      = (known after apply)
          + modes             = [
              + "MODE_IPV4",
            ]
          + network           = "default"
          + reserved_ip_range = (known after apply)
        }
    }

  # google_project_service.artifactregistry will be created
  + resource "google_project_service" "artifactregistry" {
      + disable_on_destroy = false
      + id                 = (known after apply)
      + project            = (known after apply)
      + service            = "artifactregistry.googleapis.com"
    }

  # google_project_service.cloudrun will be created
  + resource "google_project_service" "cloudrun" {
      + disable_on_destroy = false
      + id                 = (known after apply)
      + project            = (known after apply)
      + service            = "run.googleapis.com"
    }

  # google_project_service.iam will be created
  + resource "google_project_service" "iam" {
      + disable_on_destroy = false
      + id                 = (known after apply)
      + project            = (known after apply)
      + service            = "iam.googleapis.com"
    }

  # google_project_service.resourcemanager will be created
  + resource "google_project_service" "resourcemanager" {
      + disable_on_destroy = false
      + id                 = (known after apply)
      + project            = (known after apply)
      + service            = "cloudresourcemanager.googleapis.com"
    }

  # google_project_service.vpcaccess will be created
  + resource "google_project_service" "vpcaccess" {
      + disable_on_destroy = false
      + id                 = (known after apply)
      + project            = (known after apply)
      + service            = "vpcaccess.googleapis.com"
    }

  # google_vpc_access_connector.connector will be created
  + resource "google_vpc_access_connector" "connector" {
      + id             = (known after apply)
      + ip_cidr_range  = "10.8.0.0/28"
      + max_throughput = 300
      + min_throughput = 200
      + name           = "mage-data-prep-connector"
      + network        = "default"
      + project        = (known after apply)
      + region         = "us-west2"
      + self_link      = (known after apply)
      + state          = (known after apply)
    }

Plan: 16 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + service_ip = (known after apply)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
```


# Deploy to GCP with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/gcp/setup



![](https://user-images.githubusercontent.com/78053898/198754786-3c4618dd-d648-4af7-b288-5ff18371335b.png)

## Pre-requisites

### Setup

[Download Mage maintained Terraform scripts](/production/deploying-to-cloud/using-terraform).

### Permissions

Go to the IAM management dashboard, find the service account associated to the
account you just logged into, and then add these roles to that service account
(e.g. choose your account as the principal when adding new roles):

1. Artifact Registry Read
2. Artifact Registry Writer
3. Cloud Run Developer
4. Cloud SQL
5. Service Account Token Creator

***

## Terraform plan

You can run the following command to see all the resources that will be created
by Terraform:

```bash theme={"system"}
cd gcp
```

```bash theme={"system"}
terraform plan
```

By default, here are the **[resources](/production/deploying-to-cloud/gcp/resources)** that will be
created.

***

## 1. Install `gcloud` CLI

Follow these [instructions](https://cloud.google.com/sdk/docs/install) to
install the CLI on your workstation.

For macOS and Homebrew, you can do the following:

```bash theme={"system"}
brew install --cask google-cloud-sdk
```

***

## 2. Log into GCP from CLI

If you don’t already have an account in GCP, create one
[here](https://cloud.google.com/).

Once you created an account, from your terminal run the following command to log
in:

```bash theme={"system"}
gcloud init
```

```bash theme={"system"}
gcloud auth application-default login
```

## 3. Customize Terraform settings

**Project ID (REQUIRED)**

Before running any Terraform commands, please change the `default` value of the
variable named `project_id` in the
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
file.

```js theme={"system"}
variable "project_id" {
  type        = string
  description = "The name of the project"
  default     = "unique-gcp-project-id"
}
```

**Docker Image (REQUIRED)**

Google Cloud Run supports pulling Docker images directly from Docker Hub. To use the latest Mage image (default):

```js theme={"system"}
variable "docker_image" {
  type        = string
  description = "The docker image to deploy to Cloud Run."
  default     = "mageai/mageai:latest"
}
```

Simply change the tag to adjust the version of Mage you want to deploy. If you need to build your own image, you can push it to Docker Hub or follow our guide to use [GCP Artifact Registry](/production/deploying-to-cloud/gcp/gcp-artifact-registry).

**Application Name**

In the file
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf),
you can change the `default` value under `app_name`:

```js theme={"system"}
variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}
```

**Region**

In the file
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf),
you can change the `default` value under `region`:

```js theme={"system"}
variable "region" {
  type        = string
  description = "The default compute region"
  default     = "us-west2"
}
```

### Environment variables for application

Set your environment variables in your running cloud environment by adding the
following under the resource named `google_cloud_run_service` in the file
[./gcp/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf):

```js theme={"system"}
resource "google_cloud_run_service" "run_service" {
  ...

  template {
    spec {
      containers {
        ...
        env {
          name  = "NAME_OF_ENVIRONMENT_VARIABLE"
          value = "value of environment variable"
        }
      }
    }
  }
}
```

### Secrets

See our [docs](/production/deploying-to-cloud/secrets/GCP) for configuring secrets in Google Secret Manager and using secrets locally while developing.

#### Method A: Terraform configurations

1. Once you save a secret in Google Secret Manager, click on the
   **PERMISSIONS** tab.

2. Click the button **`+ GRANT ACCESS`**.

3. Under the field labeled **New principles**, add the service account that is
   associated to your Google Cloud Run. If you can’t find this, try deploying
   GCP with Terraform and you’ll receive an error that looks like this:

   ```
   ╷
   │ Error: resource is in failed state "Ready:False", message: Revision 'mage-2992l' is not ready and cannot serve traffic. spec.template.spec.volumes[1].secret.items[0].key: Permission denied on secret: projects/1054912425479/secrets/bigquery_credentials/versions/latest for Revision service account 1054912425479-compute@developer.gserviceaccount.com. The service account used must be granted the 'Secret Manager Secret Accessor' role (roles/secretmanager.secretAccessor) at the secret, project or higher level.
   │
   │   with google_cloud_run_service.run_service,
   │   on main.tf line 88, in resource "google_cloud_run_service" "run_service":
   │   88: resource "google_cloud_run_service" "run_service" {
   │
   ╵
   ```

   If you still get this error when re-running `terraform apply` even after granting access,
   try **Method B** down below.

   > [!NOTE]
   >
   > By default, the service account used by Cloud Run that needs access to the secret is the `Compute Engine` default service account.
   > If you want to grant access to a different service account,
   > configure the `service_account_name` argument in the `spec` block in
   > [`main.tf`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf#L99):
   >
   > ```
   > # Create the Cloud Run service
   > resource "google_cloud_run_service" "run_service" {
   >   name = var.app_name
   >   location = var.region
   >
   >   template {
   >     spec {
   >       service_account_name = "[your service account email here]"
   >     }
   >   }
   > }
   > ```

4. Under the field labeled **Select a role**, enter the value
   `Secret Manager Secret Accessor`.

5. Click the button **SAVE**.

6. Mount secrets to Google Cloud Run via Terraform in the file
   [./gcp/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf):

   ```
   resource "google_cloud_run_service" "run_service" {
     ...

     template {
       spec {
         containers {
           ...
           env {
             name = "path_to_keyfile"
             value = "/secrets/bigquery/bigquery_credentials"
           }
           volume_mounts {
             name       = "secrets-bigquery_credentials"
             mount_path = "/secrets/bigquery"
           }
         }
         volumes {
           name = "secrets-bigquery_credentials"
           secret {
             secret_name  = "bigquery_credentials"
             items {
               key  = "latest"
               path = "bigquery_credentials"
             }
           }
         }
       }
     }
   }
   ```

7. In your Python code or in any field that support variable interpolation,
   enter the following to get the value from the secret: 1

   ```python Python theme={"system"}
   import os


   with open(os.getenv('path_to_keyfile'), 'r') as f:
       print(f.read())
   ```

   ```yaml YAML theme={"system"}
   demo:
   target: prod
   outputs:
     prod_bigquery:
       dataset: mage_demo
       keyfile: "{{ env_var('path_to_keyfile') }}"
       method: service-account
       project: my_cool_gcp_project
       threads: 1
       type: bigquery
   ```

#### Method B: through GCP UI (aka console but not to be confused with CLI)

1. Go to your Google Cloud Run dashboard, find the service running Mage, and
   click on it to view its details.
2. Follow these
   [Google provided instructions](https://cloud.google.com/run/docs/configuring/secrets#mounting-secrets-service)
   to manually mount the secret to the running Mage Cloud Run service and grant
   the right permissions for that service to access those secrets.
   After you mount the secrets, you can create environment variables in your Cloud Run that reference
   your secrets. Follow the below example to setup the mounted secret and environment variable for that
   secret:

   | **Secrets** | |
   | - | - |
   | Secret name | `bigquery_credentials` |
   | Reference method | `Mounted as volume` |
   | Mount path | `/secrets/bigquery` |
   | Path 1 | `bigquery_credentials` |
   | Version 1 | `latest` |

   | **Environment variables** | |
   | - | - |
   | Name | `path_to_keyfile` |
   | Value | `/secrets/bigquery/bigquery_credentials` |

### More

Other variables defined in
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
can also be customized to your needs.

## 4. Deploy

1. Change directory into scripts folder:
   ```bash theme={"system"}
   cd gcp
   ```

2. Initialize Terraform:

   ```bash theme={"system"}
   terraform init
   ```

   A sample output could look like this:

   ```
   Initializing the backend...

   Initializing provider plugins...
   - Finding hashicorp/google versions matching ">= 3.3.0"...
   - Finding latest version of hashicorp/http...
   - Installing hashicorp/google v4.38.0...
   - Installed hashicorp/google v4.38.0 (signed by HashiCorp)
   - Installing hashicorp/http v3.1.0...
   - Installed hashicorp/http v3.1.0 (signed by HashiCorp)

   Terraform has created a lock file .terraform.lock.hcl to record the provider
   selections it made above. Include this file in your version control repository
   so that Terraform can guarantee to make the same selections by default when
   you run "terraform init" in the future.

   Terraform has been successfully initialized!

   You may now begin working with Terraform. Try running "terraform plan" to see
   any changes that are required for your infrastructure. All Terraform commands
   should now work.

   If you ever set or change modules or backend configuration for Terraform,
   rerun this command to reinitialize your working directory. If you forget, other
   commands will detect it and remind you to do so if necessary.
   ```

3. Deploy:

   ```bash theme={"system"}
   terraform apply
   ```

   A sample output could look like this:

   ```
   Apply complete! Resources: 7 added, 1 changed, 0 destroyed.

   Outputs:

   service_ip = "34.107.187.208"
   ```

   It’ll take a few minutes for GCP Cloud Run to start up and be ready to
   receive requests.

   After a few minutes, open a browser and go to `http://[IP_address]`

   > [!NOTE]
   >
   > Change the `IP_address` to the IP address that was output in your terminal
   > after successfully running `terraform apply`.

### Errors

#### 403 Forbidden

If you get this error when trying to open the above IP address in your browser,
open the security group named
[`[application_name]-security-policy`](https://console.cloud.google.com/net-security/securitypolicies/list).
Click on that security group and verify your IP address is whitelisted.

If it isn’t, add a new rule with the following values:

| Mode | Match | Action | Priority |
| - | - | - | - |
| Basic mode | [Enter your IP address](https://whatismyipaddress.com/) | Allow | 100 |

If you still get 403, check to see if you’re using `http://` and NOT `https://`.

#### Container can’t start

If Cloud Run is deployed but the service failed because the container couldn’t start,
it is usually caused by the Docker image not being built using the correct platform tag.

Pull the Mage Docker image using this command:

```
docker pull --platform linux/amd64 mageai/mageai:latest
```

Once pulled, tag your image and push to GCP Artifact Registry again.

***

## Updating Mage versions

After Mage is running in your cloud, you can update the Mage Docker image
version by running the following command in your CLI tool:

```bash theme={"system"}
gcloud run deploy [app_name] --image [docker_image]
```

> `app_name`
>
> This is the value you changed when editing the
> [./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
> file.

***

## Misc

### Security

To enable other IP addresses access to Mage, open the security group named
[`[application_name]-security-policy`](https://console.cloud.google.com/net-security/securitypolicies/list).
Click on that security group and add a new rule with the following values:

| Mode | Match | Action | Priority |
| - | - | - | - |
| Basic mode | [Enter your IP address](https://whatismyipaddress.com/) | Allow | 100 |

To enable IP addresses access to specific endpoints, you can follow
[this example](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/load_balancer.tf#L23-L33)
to add rules to security policy. Another way is to add a new rule through UI
with the following values:

| Mode | Match | Action | Priority |
| - | - | - | - |
| Advanced mode | `request.path.startsWith('/api/pipeline_schedules/123/pipeline_runs') && inIpRange(origin.ip, '[IP address]/32')` | Allow | 200 |

### HTTPS enabling

To enable HTTPS for Mage app deployed on GCP, you need to firstly make sure you
have a domain. Then follow the steps below to set up HTTPS:

1. In
   [./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
   file, update the `default` value of `ssl` to be `true`, and set the `default`
   value of `domain` variable to be the domain url you want to use.
2. Run `terraform apply` to create the HTTPS load balancer.
3. Get the `service_ip` from the output of terraform command.
4. Update the DNS A and AAAA records to point the domain url to the load
   balancer's IP address:
   [https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs#update-dns](https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs#update-dns)
5. Visit the domain url to access your Mage app.

### Terminate all resources

If you want to delete all resources created from deploying, run the following
command:

```bash theme={"system"}
terraform destroy
```

A sample output could look like this:

```
Destroy complete! Resources: 10 destroyed.
```

***

## dbt

For information about how to serve dbt docs in production,
[see here](/dbt/docs).


# AWS Secrets Manager
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/AWS

Retrieve AWS Secrets Manager values in Mage configuration files without storing them in source code.

Mage can retrieve secret strings from AWS Secrets Manager when it renders YAML and SQL templates. Use this integration for database passwords, API keys, and other credentials that should not be committed to your project.

In Mage Pro, this guide is also available from **Settings > Access & security > Secrets management**.

## Prerequisites

Before configuring Mage:

1. [Create a secret in AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) in the Region Mage will use.
2. Grant the IAM role or user running Mage permission to call `secretsmanager:GetSecretValue` for the required secrets.
3. If a secret uses a customer-managed KMS key, also grant `kms:Decrypt` for that key.

For example, scope an IAM policy to the secrets Mage needs:

```json theme={"system"}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-west-2:123456789012:secret:prod/*"
    }
  ]
}
```

## Configure AWS authentication

Mage checks environment variables first, then built-in Mage secrets with the same credential names. When no explicit access keys are configured, the underlying AWS SDK uses its default provider chain. In AWS, prefer an IAM role attached to the Mage workload so that you do not need long-lived access keys.

When a workload role is not available, set the following environment variables on every Mage process that renders templates containing AWS secret references:

| Variable | Required | Description |
| - | - | - |
| `AWS_ACCESS_KEY_ID` | For access-key authentication | AWS access key ID. |
| `AWS_SECRET_ACCESS_KEY` | For access-key authentication | AWS secret access key. |
| `AWS_SESSION_TOKEN` | For temporary credentials | Session token paired with temporary access keys. |
| `AWS_REGION_NAME` or `AWS_DEFAULT_REGION` | Recommended | Region containing the secret. Mage checks `AWS_REGION_NAME` first, then `AWS_DEFAULT_REGION`; if neither is configured, it defaults to `us-west-2`. |

These values may be provided as protected deployment environment variables. Mage also checks its built-in secret store for variables with these exact names, but that bootstrapping option requires Mage's secret database to be available before the AWS client is created.

Do not place literal AWS credentials in `io_config.yaml`, pipeline code, or version control.

## Reference a secret

Use `aws_secret_var` anywhere Mage template variables are supported:

```jinja theme={"system"}
{{ aws_secret_var('prod/postgres/password') }}
```

For example:

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_HOST: database.internal
  POSTGRES_USER: analytics
  POSTGRES_PASSWORD: "{{ aws_secret_var('prod/postgres/password') }}"
```

Pass either the secret name or its full ARN as the argument. A full ARN is useful when names are ambiguous, but it is not required for a secret in the configured Region and account.

`aws_secret_var` returns the complete `SecretString`. If the AWS secret contains JSON, extract the field you need with `json_value`:

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_USER: "{{ json_value(aws_secret_var('prod/postgres'), 'username') }}"
  POSTGRES_PASSWORD: "{{ json_value(aws_secret_var('prod/postgres'), 'password') }}"
```

Binary secrets are not supported by this template helper because it reads `SecretString` rather than `SecretBinary`.

## Cache behavior and rotation

Mage uses the AWS Secrets Manager caching client and its library-managed refresh interval for normal `aws_secret_var` lookups. A rotated value is not guaranteed to appear immediately. Bypass the cache for a lookup with:

```jinja theme={"system"}
{{ aws_secret_var('prod/postgres/password', cached=False) }}
```

An uncached lookup calls `GetSecretValue` directly and increases AWS API traffic. Existing database connections and clients can continue using an older credential until they are recreated.

## Troubleshooting

### Access denied

Confirm the active IAM identity can call `secretsmanager:GetSecretValue` on the requested secret. For a customer-managed KMS key, also confirm it can call `kms:Decrypt`. Check both the IAM policy and any resource-based policy on the secret.

### Secret not found

Verify the secret name or ARN and the configured Region. Mage uses `AWS_REGION_NAME`, then `AWS_DEFAULT_REGION`, and otherwise `us-west-2`. If the secret is in another account, use a full ARN and ensure the secret policy permits the Mage identity.

### Invalid secret identifier

Confirm that the value passed to `aws_secret_var` is a valid secret name or ARN. Mage reports invalid AWS parameters separately from missing secrets.

### Invalid credentials or expired token

For access-key authentication, confirm the access key and secret key belong together. Temporary credentials also require a current `AWS_SESSION_TOKEN`. Prefer a workload IAM role to avoid manually rotating these credentials.

### JSON secret renders as one string

`aws_secret_var` returns the full secret string. Use `json_value(aws_secret_var('secret-name'), 'field')` to select a value from a JSON object.

## Security recommendations

* Prefer workload IAM roles over long-lived access keys.
* Grant `secretsmanager:GetSecretValue` only for the secrets Mage requires.
* Use separate roles and secret paths for development, staging, and production.
* Enable AWS Secrets Manager rotation where supported by the downstream service.
* Avoid logging rendered configuration or secret values.
* Audit access with AWS CloudTrail.


# Azure Key Vault
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure



### YAML

To use secrets in YAML files (e.g. in data integration pipelines):

1. Add the following keys and values to your environment variables so that
   Mage can access the Azure Key Vault:
   1. `AZURE_KEY_VAULT_URL`: The Azure Key Vault URI.
   2. `AZURE_CLIENT_ID`: Azure Active Directory [client ID](https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-client-application-configuration#client-id).
   3. `AZURE_CLIENT_SECRET`: Azure Active Directory [client secret](https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret).
   4. `AZURE_TENANT_ID`: Azure Active Directory [tenant ID](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-ad-tenant).
2. Store a secret in Azure Key Vault.
3. Use the following syntax in your YAML file to interpolate secret values from Azure Key Vault:

   ```yaml theme={"system"}
   "{{ azure_secret_var('secret_name') }}"
   ```

   For example:

   ```yaml theme={"system"}
   api_key: "{{ azure_secret_var('API_KEY') }}"
   access_token: "{{ azure_secret_var('ACCESS_TOKEN') }}"
   ```

### Python

Here is the example python code to get secrets from Azure Key Vault:

```python theme={"system"}
from mage_ai.services.azure.key_vault.key_vault import get_secret

get_secret('secret_name')
```


# Doppler
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/Doppler

Retrieve Doppler secrets in Mage configuration files without storing credentials in source code.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=doppler-secrets)

Mage can retrieve secrets from a Doppler project and config when it renders YAML and SQL templates. Use this integration for values such as database passwords, API keys, and access tokens that should not be committed to your Mage project.

In Mage Pro, this guide is also available from **Settings > Access & security > Secrets management**.

## Prerequisites

Before configuring Mage:

1. Create a [Doppler project and config](https://docs.doppler.com/docs/enclave-projects-configs).
2. Add the secrets that Mage needs to the config.
3. Create a [service token](https://docs.doppler.com/docs/service-tokens) with access to that config. Prefer a dedicated, read-only token for each Mage environment.

## Configure Mage

Set these environment variables on every Mage process that renders configuration containing Doppler references:

| Variable | Required | Description |
| - | - | - |
| `DOPPLER_TOKEN` | Yes | Service token used to read secrets. |
| `DOPPLER_PROJECT` | Yes | Default Doppler project identifier. |
| `DOPPLER_CONFIG` | Yes | Default Doppler config name, such as `dev`, `stg`, or `prd`. |
| `DOPPLER_API_HOST` | No | Doppler API host. Defaults to `https://api.doppler.com`. |
| `DOPPLER_SECRETS_CACHE_TTL_SECONDS` | No | In-memory cache lifetime in seconds. Defaults to `10`. Use `0` to fetch on every lookup. |

For example:

```bash theme={"system"}
export DOPPLER_TOKEN="dp.st.example-token"
export DOPPLER_PROJECT="analytics"
export DOPPLER_CONFIG="prd"
```

Restart Mage after changing its environment variables. Do not put `DOPPLER_TOKEN` in `io_config.yaml`, pipeline code, or version control.

## Reference a secret

Use `doppler_secret_var` anywhere Mage template variables are supported:

```jinja theme={"system"}
{{ doppler_secret_var('POSTGRES_PASSWORD') }}
```

For example, an `io_config.yaml` profile can retrieve multiple values from the default Doppler project and config:

```yaml theme={"system"}
version: 0.1.1
default:
  POSTGRES_DBNAME: analytics
  POSTGRES_HOST: database.internal
  POSTGRES_PASSWORD: "{{ doppler_secret_var('POSTGRES_PASSWORD') }}"
  POSTGRES_USER: "{{ doppler_secret_var('POSTGRES_USER') }}"
```

Mage requests the computed value for the named secret. If the secret does not exist in the selected project and config, template rendering fails instead of returning an empty value.

## Override the project or config

The environment variables define the defaults. Pass `project` or `config` for a reference that should use a different Doppler scope:

```jinja theme={"system"}
{{ doppler_secret_var('POSTGRES_PASSWORD', project='billing', config='prd') }}
```

You can override either value independently. The runtime still uses `DOPPLER_TOKEN`, so that token must be authorized to read every referenced project and config.

## Cache behavior and secret rotation

Mage caches the complete secret set for each API host, project, and config in process memory. The default cache lifetime is 10 seconds. References to the same scope reuse that cached result until its TTL expires.

Set `DOPPLER_SECRETS_CACHE_TTL_SECONDS=0` to disable caching. This makes secret changes available on the next template render, but increases Doppler API traffic and makes rendering more sensitive to temporary network failures. A negative TTL is treated as `0`.

After the TTL expires, Mage fetches the config again on the next lookup. Existing database connections and clients may continue using an older credential until they are recreated.

## Troubleshooting

### Missing Doppler configuration

If Mage reports missing `DOPPLER_TOKEN`, `DOPPLER_PROJECT`, or `DOPPLER_CONFIG`, confirm the variables are present in the environment of the Mage process, then restart it. Setting them only in an interactive shell does not update an already-running deployment.

### Secret not found

Confirm that the secret name is exact and case-sensitive, the selected project and config contain it, and the service token can read that scope. Check per-reference `project` and `config` overrides as well as the defaults.

### Doppler API request fails

Verify that Mage can reach `https://api.doppler.com`, the service token is valid, and the project and config identifiers are correct. If your environment uses a proxy or a custom Doppler API endpoint, set `DOPPLER_API_HOST` to the reachable base URL without a trailing path.

### A rotated value is not used immediately

Wait for `DOPPLER_SECRETS_CACHE_TTL_SECONDS` to elapse and trigger another template render. Recreate any open database connection or client that was initialized with the previous value. For rotation-sensitive workloads, temporarily reduce the TTL or set it to `0`, considering the additional API traffic.

## Security recommendations

* Use a separate Doppler service token for each Mage environment.
* Grant the token access only to the projects and configs that environment requires.
* Store the token in your deployment platform's protected environment configuration, not in Mage project files.
* Rotate tokens and application credentials regularly, and revoke unused tokens.
* Avoid logging rendered configuration or secret values.


# GCP Secret Management
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/GCP



## Creating secrets

1. Go to
   [Google Secret Manager UI](https://console.cloud.google.com/security/secret-manager?referrer=search\&authuser=1).
2. Click the button at the top labeled **`+ CREATE SECRET`**.
3. Fill in the name of your secret; e.g. `bigquery_credentials`.
4. Under **Secret value**, upload your service account credentials JSON file or
   paste the JSON into the text area labeled **Secret value**.
5. Scroll all the way down and click the button **`CREATE SECRET`**.

You can mount secrets from Google Secret Manager through Terraform configurations or through the Google Console UI.

## Using secrets locally

### Download credentials from GCP UI

1. Download the credentials JSON file from GCP.
2. Run Mage and mount the secrets as a volume in Docker.
   Follow these [instructions](/getting-started/setup#using-secret-credential-files-in-docker) to learn how to do this.
3. Here are example code snippets to read from that credentials JSON file:

   ```python Python theme={"system"}
   with open('/home/secrets/gcp_credentials.json', 'r') as f:
       print(f.read())
   ```

   ```yaml YAML theme={"system"}
   demo:
   target: prod
   outputs:
     prod_bigquery:
       dataset: mage_demo
       keyfile: /home/secrets/gcp_credentials.json
       method: service-account
       project: my_cool_gcp_project
       threads: 1
       type: bigquery
   ```

   > Note
   >
   > This code example assumes your credentials JSON file downloaded from GCP is named
   > `gcp_credentials.json` and that the mount path (e.g. `-v`) you used when running Docker
   > is `/home/secrets`.

***

### Download credentials using `gcloud` CLI

1. Authenticate locally by running this command in your local terminal:

   ```
   gcloud auth application-default login
   ```

2. Create a new `.env` file in your Mage project folder with the following values:

   ```
   GOOGLE_APPLICATION_CREDENTIALS="[PATH TO YOUR USER CREDENTIALS, MOST LIKELY: ~/.config/gcloud/application_default_credentials.json]"
   GCS_BUCKET=[YOUR DEV BUCKET]
   GCLOUD_PROJECT=[YOUR PROJECT]
   ```

3. Run Mage using Docker and set the environment variable `GOOGLE_APPLICATION_CREDENTIALS`.
   Follow these [instructions](/getting-started/setup#environment-variables) to learn how to do this.
   For example, set the environment variable to:

   ```
   -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/FILE_NAME.json
   ```

4. Run Mage and mount the secrets as a volume in Docker.
   Follow these [instructions](/getting-started/setup#using-secret-credential-files-in-docker) to learn how to do this.
   For example:

   ```
   -v ~/.config/gcloud/application_default_credentials.json:/tmp/keys/FILE_NAME.json
   ```

5. Here is an example code snippet:

   ```python theme={"system"}
   from google.cloud import storage
   from pandas import DataFrame
   from datetime import datetime
   import pytz
   import os
   ​
   ​
   @data_exporter
   def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
   ​
       bucket_name = os.getenv('GCS_BUCKET')
   ​
       now = datetime.utcnow()
       pt = pytz.timezone("America/Los_Angeles")
       now_pst = pytz.utc.localize(now).astimezone(pt)
   ​
       object_key = f'test_file_{now_pst.strftime("%Y-%m-%d")}.csv'
   ​
       storage_client = storage.Client()
       bucket = storage_client.bucket(bucket_name)
       blob = bucket.blob(object_key)
   ​
       blob.upload_from_string(df.to_csv())
   ​
       print(f'df uploaded to {bucket}/{object_key}.')
   ```

## GCP Secret Manager Integration in Mage Pro

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=gcp-secret)

Securely manage and inject sensitive credentials into your Mage Pro pipelines using **Google Cloud Secret Manager**. This Pro-only feature helps enterprises meet cloud security and compliance standards by keeping secrets like API keys, database passwords, and tokens outside of source code.

### Required Environment Variables

Before accessing secrets, configure your Mage Pro cluster with the following environment variables:

* `GOOGLE_APPLICATION_CREDENTIALS`:
  Full path to your **GCP service account credentials JSON file**.
  *You can upload the credentials file directly to the Mage Pro cluster using the file browser interface.*

* `GCP_PROJECT_ID`:
  The **Google Cloud Project ID** associated with your Secret Manager secrets.

These credentials must have `Secret Manager Secret Accessor` permissions for the secrets you intend to retrieve.

### How to Use the GCP Secret Variable

You can reference GCP secrets programmatically in Python blocks or declaratively in YAML configs.

#### Python code

Use this approach inside a block in your Mage pipeline:

```python theme={"system"}
from mage_ai.services.gcp.secret_manager.secret_manager import get_secret

# Retrieve a secret by ID from GCP Secret Manager
secret_value = get_secret('secret_id')
```

#### YAML config

To inject a secret dynamically into a YAML config (e.g., for a data source, destination, or authentication setting):

```yaml theme={"system"}
{{gcp_secret_var('secret_id')}}
```

Mage will automatically resolve and substitute the secret value at runtime.


# HashiCorp Vault
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/Vault

Securely manage and inject sensitive credentials into your Mage pipelines using HashiCorp Vault. This feature helps enterprises meet security and compliance standards by keeping secrets like API keys, database passwords, and tokens outside of source code.

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=vault-secret)

## Overview

HashiCorp Vault is a secrets management tool that provides secure storage and access to sensitive data. Mage integrates with Vault to retrieve secrets at runtime, ensuring that sensitive credentials are never stored in your codebase or configuration files.

## Prerequisites

Before using Vault with Mage, ensure you have:

1. A running HashiCorp Vault instance
2. Appropriate permissions to read secrets from Vault
3. A valid Vault token or authentication method

## Configuration

### Required Environment Variables

Configure your Mage environment with the following environment variables:

* `VAULT_URL`: The URL of your Vault server (default: `http://127.0.0.1:8200`)
* `VAULT_TOKEN`: Your Vault authentication token

### Optional Environment Variables

* `VAULT_MOUNT_POINT`: The KV v2 mount point for secrets (default: `secret`)

### Example Environment Setup

```bash theme={"system"}
export VAULT_URL="https://vault.company.com"
export VAULT_TOKEN="hvs.xxxxxxxxxxxxxxxxxxxx"
export VAULT_MOUNT_POINT="secret"
```

## Using Secrets in Mage

### Python Code

Use the Vault integration directly in your Python blocks. The `get_secret()` function accepts two parameters:

* `path`: The secret path in Vault (e.g., `'mage/database'`)
* `key`: Optional specific key within the secret (e.g., `'password'`)

**Syntax:**

```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret

# Get a specific key from a secret
value = get_secret('secret/path', 'key_name')

# Get all key-value pairs from a secret
all_data = get_secret('secret/path')
```

**Example:**

```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret

# Get a specific key from a secret
db_password = get_secret('mage/database', 'password')

# Get all key-value pairs from a secret
db_config = get_secret('mage/database')
print(f"Database host: {db_config['host']}")
print(f"Database username: {db_config['username']}")
```

### YAML Configuration

Reference Vault secrets in your YAML configuration files using the `vault_secret_var()` function. This function accepts two parameters:

* `path`: The secret path in Vault (e.g., `'mage/database'`)
* `key`: The specific key within the secret (e.g., `'password'`)

**Syntax:**
Use the `"{{ vault_secret_var('secret/path', 'key_name') }}"` syntax in any YAML field that supports variable interpolation:

```yaml theme={"system"}
version: 0.1.1
default:
  VARIABLE_NAME: "{{ vault_secret_var('secret/path', 'key_name') }}"
```

**Example 1 - IO Config:**

```yaml theme={"system"}
version: 0.1.1
default:
  # PostgreSQL configuration using Vault secrets
  POSTGRES_HOST: "{{ vault_secret_var('mage/database', 'host') }}"
  POSTGRES_PORT: 5432
  POSTGRES_DBNAME: "my_database"
  POSTGRES_USER: "{{ vault_secret_var('mage/database', 'username') }}"
  POSTGRES_PASSWORD: "{{ vault_secret_var('mage/database', 'password') }}"
```

**Example 2 - Data Integration Pipeline:**

```yaml theme={"system"}
# API source configuration
url: "https://api.example.com/data"
method: GET
headers:
  Authorization: "Bearer {{ vault_secret_var('mage/api', 'token') }}"
  X-API-Key: "{{ vault_secret_var('mage/api', 'key') }}"
response_parser: data.items
```

## Creating Secrets in Vault

### Using Vault CLI

1. **Authenticate with Vault:**
   ```bash theme={"system"}
   vault auth -method=userpass username=your-username
   ```

2. **Create a secret:**
   ```bash theme={"system"}
   vault kv put secret/mage/database \
     username=dbuser \
     password=dbpassword \
     host=db.example.com
   ```

3. **Verify the secret:**
   ```bash theme={"system"}
   vault kv get secret/mage/database
   ```

### Using Vault UI

1. Navigate to your Vault UI:
   * **Local development**: `http://127.0.0.1:8200/ui`
   * **Production**: `https://vault.company.com/ui`
2. Go to **Secrets** → **KV v2**
3. Click **Create secret**
4. Enter the secret path (e.g., `mage/database`)
5. Add key-value pairs for your credentials
6. Click **Save**

## Error Handling

The Vault integration provides specific error handling for common scenarios:

```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret

try:
    secret_value = get_secret('mage/database', 'password')
except KeyError as e:
    print(f"Secret or key not found: {e}")
except PermissionError as e:
    print(f"Access denied: {e}")
except Exception as e:
    print(f"Vault error: {e}")
```

## Security Best Practices

### 1. Token Management

* Use short-lived tokens when possible
* Implement token rotation
* Store tokens securely (environment variables, not in code)

### 2. Secret Organization

* Use hierarchical paths (e.g., `mage/prod/database`, `mage/dev/database`)
* Follow consistent naming conventions
* Separate secrets by environment

### 3. Access Control

* Implement least-privilege access
* Use Vault policies to restrict access
* Regularly audit secret access

### 4. Monitoring

* Enable Vault audit logs
* Monitor secret access patterns
* Set up alerts for unusual activity

## Troubleshooting

### Common Issues

**1. Authentication Failed**

```
Exception: Unable to authenticate to Vault at https://vault.company.com
```

* Verify your `VAULT_TOKEN` is valid
* Check if the token has expired
* Ensure the Vault URL is correct

**2. Secret Not Found**

```
KeyError: Secret path 'mage/database' not found in Vault
```

* Verify the secret path exists
* Check the mount point configuration
* Ensure you have read permissions

**3. Access Denied**

```
PermissionError: Access denied to secret path 'mage/database'
```

* Check your Vault token permissions
* Verify the secret path is accessible
* Contact your Vault administrator

## Advanced Usage

### Custom Mount Points

If your Vault instance uses a different mount point:

```bash theme={"system"}
export VAULT_MOUNT_POINT="custom-secrets"
```

### Multiple Vault Instances

For different environments, you can create multiple Vault clients:

```python theme={"system"}
import os
from mage_ai.services.hashicorp.vault.vault import Vault

# Override environment variables for specific operations
os.environ['VAULT_URL'] = 'https://vault-prod.company.com'
os.environ['VAULT_TOKEN'] = 'prod-token'

prod_vault = Vault()
```

## Integration Examples

### Database Connection

```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
import psycopg2

# Get database credentials from Vault
db_config = get_secret('mage/database')

# Create database connection
conn = psycopg2.connect(
    host=db_config['host'],
    port=db_config['port'],
    database=db_config['database'],
    user=db_config['username'],
    password=db_config['password']
)
```

### API Authentication

```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
import requests

# Get API credentials from Vault
api_token = get_secret('mage/api', 'token')

# Make authenticated API request
headers = {'Authorization': f'Bearer {api_token}'}
response = requests.get('https://api.example.com/data', headers=headers)
```

### Cloud Provider Credentials

```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
import boto3

# Get AWS credentials from Vault
aws_creds = get_secret('mage/aws')

# Create AWS client
session = boto3.Session(
    aws_access_key_id=aws_creds['access_key_id'],
    aws_secret_access_key=aws_creds['secret_access_key'],
    region_name=aws_creds['region']
)

s3_client = session.client('s3')
```

## Migration from Other Secret Managers

If you're migrating from other secret management solutions:

### From Mage Built-in Secrets

```python theme={"system"}
# Old way
from mage_ai.data_preparation.shared.secrets import get_secret_value
password = get_secret_value('database_password')

# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')
```

### From Environment Variables

```python theme={"system"}
# Old way
import os
password = os.getenv('DATABASE_PASSWORD')

# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')
```

## Support

For additional help with Vault integration:

* [HashiCorp Vault Documentation](https://www.vaultproject.io/docs)
* [Vault Python Client (hvac) Documentation](https://hvac.readthedocs.io/)
* [Mage Community Forum](https://www.mage.ai/chat)


# Mage’s built-in secret manager
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/mage

Storing and using secrets through Mage.

In Mage Pro, open **Settings > Access & security > Secrets management** to compare this option with supported external secret managers. Create and manage built-in project secrets from **Secrets** in the primary side navigation.

In various surfaces in Mage, you may be asked to input config for certain integrations
such as cloud databases or services. In these cases, you may need to input a password
or an api key, but you don't want it to be shown in plain text. To get around this issue,
we created a way to store your secrets in the Mage database.

> [!WARNING]
>
> Your secrets are encrypted before being stored into the Mage database. The encryption key
> will be stored in the Mage data folder which by default will be created at `~/.mage_data`. If
> you need more secure encryption, we recommend using a secrets manager.

## Secret management

### Creating secrets

To manage your secrets in Mage OSS, select **Secrets** in the left navigation. In
Mage Pro, open **Library** and select **Secrets**. This opens the global secrets
management page where you can see all your secrets, create new ones, and delete
existing ones.

You can also manage secrets directly from the pipeline edit page. In the sidekick,
you should see a `Secrets` tab. To create a secret, press the `New` button on this tab.
Input a name for the secret and the value of the secret, and press Enter or Return to save.

![Create Secrets](https://mage-ai.github.io/assets/create-secrets.png)

The secrets can be shared across the project that they are created in.

Pipeline level secrets coming soon...

### Secrets in different environments

Secrets are stored to the Mage database. If you are using the same database across multiple
Mage environments, i.e. development and production, the secrets will not be shared across the environments
unless you shared the same Mage data directory for every environment.

Each secret will be saved to the database with a name and a uuid. The name will be used to identify
the secret, and the uuid is used to identify the environment.

Example:

```
Development environment
project
└───mage_data
│   └───secrets
│   │   │   key: abc123
│   │   │   uuid: dev-uuid

Production environment
project
└───mage_data
│   └───secrets
│   │   │   key: def456
│   │   │   uuid: prod-uuid
```

**Secrets table in database**

| Name | Uuid | Value |
| - | - | - |
| `secret1` | `dev-uuid` | sooper dooper secret string |
| `secret2` | `dev-uuid` | xXxSecreTxXx |
| `secret3` | `dev-uuid` | dev only secret |
| `secret1` | `prod-uuid` | real production secret |
| `secret2` | `prod-uuid` | m4g3r0x! |
| `secret1` | `random` | random secret |

| Command | Development Value | Production Value |
| - | - | - |
| `get_secret_value('secret1')` | sooper dooper secret string | real production secret |
| `get_secret_value('secret2')` | xXxSecreTxXx | m4g3r0x! |
| `get_secret_value('secret3')` | dev only secret | `None` |

Coming soon:

* secrets scoped to a pipeline

## Using Secrets

You can use the following syntax to have Mage interpolate the secret when reading from the config:

```yaml theme={"system"}
field_name: "{{ mage_secret_var('your_secret_name') }}"
```

![Using Secrets](https://mage-ai.github.io/assets/using-secrets.png)

You can also fetch the secret value in a Mage code block by importing a helper method:

```python theme={"system"}
from mage_ai.data_preparation.shared.secrets import get_secret_value

get_secret_value('your_secret_name')
```


# Helm
Source: https://docs.mage.ai/production/deploying-to-cloud/using-helm



![](https://avatars.githubusercontent.com/u/15859888?s=200\&v=4)

Deploy Mage to a Kubernetes cluster with Helm. Use the chart's `values.yaml` to customize deployment topology, dependencies, storage, and Mage Pro features.

***

# Prerequisites

## Install Helm

```bash From Homebrew (macOS) theme={"system"}
brew install helm
```

```bash From Apt (Debian/Ubuntu) theme={"system"}
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
```

Full installation guide: [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/)

## Setup a Kubernetes Cluster

### Local

[Enable Kubernetes](https://docs.docker.com/desktop/kubernetes/#enable-kubernetes) in Docker Desktop to start a Kubernetes cluster locally.

Other options for starting a Kubernetes cluster locally:

* [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
* [Minikube](https://minikube.sigs.k8s.io/docs/start/)

### AWS EKS

Follow the [guide](/production/deploying-to-cloud/aws/setup#deploy-to-eks-cluster) to set up the EKS cluster.

### Google Kubernetes Engine (GKE)

Follow the [guide](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#create_cluster) to create a GKE cluster.

***

# Deploy Mage using Helm

Mage Helm repo: [https://mage-ai.github.io/helm-charts/](https://mage-ai.github.io/helm-charts/)

### Add Helm repository

```bash theme={"system"}
helm repo add mageai https://mage-ai.github.io/helm-charts
```

### Install with default config

```bash theme={"system"}
helm install my-mageai mageai/mageai
```

### Get the default values

To customize the deployment, save the chart's default values and edit them:

```bash theme={"system"}
helm show values mageai/mageai > values.yaml
```

Then install or upgrade using your file:

```bash theme={"system"}
helm install --values values.yaml my-mageai mageai/mageai
# or
helm upgrade --install my-mageai mageai/mageai --values values.yaml
```

***

# Customize deployment

The chart supports many options. Key areas are summarized below; for the full reference, see the [Mage Helm Chart — Values Reference](https://github.com/mage-ai/helm-charts/blob/master/charts/mageai/values.yaml) in the chart repository.

## Deployment topology

* **`replicaCount`** — Number of Mage replicas (default: `1`). Increase for high availability or more concurrent requests and pipeline runs. When using multiple replicas, you must enable Redis (see [Dependencies](#dependencies)).
* **`standaloneScheduler`** — When `true`, splits Mage into separate **web server** and **scheduler** deployments so you can scale and schedule them independently. Use `scheduler.replicaCount` and `webServer.replicaCount` instead of `replicaCount` when enabled.

## Dependencies

* **PostgreSQL** (`postgresql.enabled`) — Required for production: pipelines, triggers, and metadata. Enable and set `auth.username`, `auth.password`, and `auth.database` (or use an external database and configure Mage accordingly).
* **Redis** (`redis.enabled`) — Required when `replicaCount` > 1, for distributed locking and caching. You can use an external Redis by setting `customRedisURL` and leaving `redis.enabled: false`.

## Container image

* **`image.repository`**, **`image.tag`**, **`image.pullPolicy`** — Control which Mage image is used (default repository: `mageai/mageai`).
* **`imagePullSecrets`** — Use for private registries or Mage Pro registry; the secret must exist in the target namespace.

## Storage and volumes

Project code must be available at `/home/src` inside the container. You can provide it in two ways:

1. **`extraVolumes` / `extraVolumeMounts`** (default) — The chart typically mounts a volume named `mage-fs` at `/home/src`. Point it to your project:
   * **Local / hostPath:** set `extraVolumes` to a `hostPath` with your Mage project path.
   * **Cloud (EKS, GKE, AKS):** provision a **ReadWriteMany** PersistentVolume (e.g. EFS, Filestore, Azure Files) and a PersistentVolumeClaim, then use that claim in `extraVolumes` / `extraVolumeMounts`.

2. **`persistence.enabled: true`** — Use the chart's built-in PersistentVolume. Set `persistence.storageClassName` and `persistence.size` (e.g. `5Gi`). For cloud filesystems (e.g. EFS), you can use `persistence.csi` if supported by the chart.

### Example: PVC for project code (EKS / GKE / Azure)

After creating a ReadWriteMany PVC (e.g. [EKS EFS PVC](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-eks/kube/efs-pvc.yaml), [GKE Filestore](https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#dynamic_provisioning), or [AKS storage](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#persistent-volume-claims)):

```yaml theme={"system"}
extraVolumeMounts:
  - name: mage-fs
    mountPath: /home/src

extraVolumes:
  - name: mage-fs
    persistentVolumeClaim:
      claimName: your-pvc-name
```

## Service and ingress

* **`service.type`** — Default is `LoadBalancer`; set to `ClusterIP` or `NodePort` if needed. **`service.port`** defaults to `6789`.
* **`service.annotations`** — Use for cloud load balancers (e.g. AWS NLB, GCP).
* **`ingress`** — Set `ingress.enabled: true` and configure `ingress.className`, `ingress.hosts`, and `ingress.tls` for external access. For production, consider the [Mage-Ingress chart](https://github.com/mage-ai/helm-charts/tree/master/charts/ingress).

## Environment and configuration

* **`config`** — Key-value environment variables for the Mage container (e.g. `USER_CODE_PATH: /home/src/default_repo`).
* **`secrets`** — Chart-created secrets injected as env vars. Prefer a secret manager (e.g. Vault, AWS Secrets Manager) in production.
* **`existingSecret`** — Use an existing Secret for env vars; takes precedence over `secrets` if both are set.
* **`extraEnvs`** — Additional env vars, including `valueFrom` (e.g. downward API for `metadata.namespace`).

## Resources and scheduling

* **`resources`** — Set CPU/memory `limits` and `requests` for the Mage container.
* **`nodeSelector`**, **`tolerations`**, **`affinity`** — Control pod placement.
* **`livenessProbe`** / **`readinessProbe`** — Default to `/api/status` on the http port; customize or replace with `customLivenessProbe` / `customReadinessProbe`.
* **`hpa`** — Horizontal Pod Autoscaler (commented out by default).

## Quick reference: common overrides

| Use case | Key values |
| - | - |
| Production with database | `postgresql.enabled: true`, `redis.enabled: true` |
| Multiple replicas | `replicaCount: 2`, `redis.enabled: true` |
| Private registry | `image.repository`, `imagePullSecrets` |
| Custom persistent volume | `extraVolumes` / `extraVolumeMounts` with your path or PVC |
| Persistent storage | `persistence.enabled: true`, `persistence.storageClassName`, `persistence.size` |
| Mage Pro workspaces | `workspaceConfig.enabled: true` |
| Mage Pro K8s executor | `k8sExecutorConfig.enabled: true` |

***

# Default workspace and K8s executor config

The workspaceConfig and k8sExecutorConfig options in the Helm chart are supported only in Mage Pro. They are ignored when running Mage OSS.

[Try Mage Pro for free](https://cloud.mage.ai/sign-up?source=kubernetes-executor)

[Open](https://cloud.mage.ai/sign-up?source=kubernetes-executor)

When using **Mage Pro**, you can configure default workspace and Kubernetes executor settings via the chart so the control plane (and, for the K8s executor, workspace pods) get the config file mounted and the corresponding environment variables set automatically.

* **`workspaceConfig`** — Default workspace configuration (e.g. `container_config`, `service_account_name`, `lifecycle_config`). The chart creates a ConfigMap, mounts it on the control plane at `/etc/mage/workspace-config.yaml`, and sets `WORKSPACE_DEFAULT_CONFIG_PATH`. See [Workspaces — Default config file](/guides/developer-ux/workspaces#default-workspace-config-file).
* **`k8sExecutorConfig`** — Default K8s executor configuration (e.g. `resource_limits`, `resource_requests`, `pod`). The chart creates a ConfigMap, mounts it at `/etc/mage/k8s-executor-config.yaml`, and sets `K8S_DEFAULT_CONFIG_PATH`. See [K8s executor — Default config file](/production/executors/k8#default-config-file).

Enable and set `enforce` as needed in `values.yaml`; do not set `WORKSPACE_DEFAULT_CONFIG_PATH` or `K8S_DEFAULT_CONFIG_PATH` manually when using these chart options.

***

# Further reading

* [Mage Helm Chart source and values](https://github.com/mage-ai/helm-charts) — full values reference in the chart repo
* [Helm chart releases](https://github.com/mage-ai/helm-charts/releases)
* [Mage Pro Workspaces](/guides/developer-ux/workspaces)
* [Kubernetes Executor](/production/executors/k8)


# Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/using-terraform



![](https://user-images.githubusercontent.com/78053898/198754720-d6b1e5bd-caa8-431e-a4cb-e26743c0ef36.png)

Deploy Mage to your cloud provider using Terraform.

Cloud providers currently supported with Mage Terraform scripts:

* AWS
* Azure
* GCP
* DigitalOcean

***

## Pre-install

Clone Mage’s
[Terraform template repository](https://github.com/mage-ai/mage-ai-terraform-templates):

```bash theme={"system"}
git clone https://github.com/mage-ai/mage-ai-terraform-templates.git
```

***

## Install Terraform CLI

```bash Using Homebrew theme={"system"}
brew tap hashicorp/tap && brew install hashicorp/tap/terraform
```

```bash Using Docker theme={"system"}
docker pull hashicorp/terraform:latest
```

### Windows

Follow [Microsoft’s instructions](https://learn.microsoft.com/en-us/azure/developer/terraform/get-started-windows-bash?tabs=bash#4-install-terraform-for-windows) to install Terraform on Windows.

### Errors

If you’re installing Terraform using Homebrew on an Apple M1 laptop, you may get this error:

```
Error: Incompatible provider version

Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.

Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. 
Other versions of this provider may have different platforms supported.
```

Follow this [guide](https://medium.com/@sssanjaya/terraform-error-incompatible-provider-version-m1-or-darwin-arm64-fae0159f913f) to fix the issue.

## Cloud specific Terraform instructions

Read the following documentation for the cloud provider of your choice:

* [Amazon Web Services (AWS)](/production/deploying-to-cloud/aws)
* [Microsoft Azure](/production/deploying-to-cloud/azure)
* [Google Cloud Platform (GCP)](/production/deploying-to-cloud/gcp)
* [DigitalOcean](/production/deploying-to-cloud/digitalocean)

***


# AWS ECS executor
Source: https://docs.mage.ai/production/executors/aws

Execute block runs in separate tasks.

You can choose to launch separate AWS ECS tasks to executor blocks by specifying
block executor\_type to be `ecs` in pipeline's metadata.yaml file.

There're 2 ways to customize the ECS executor config,

1. Specify the `ecs_config` in project's metadata.yaml file. Example config:

   ```yaml theme={"system"}
   ecs_config:
     cpu: 1024
     memory: 2048
   ```

   2. Add the `executor_config` at block level in pipeline's metadata.yaml file. Example config:

   ```yaml theme={"system"}
   blocks:
   - uuid: example_data_loader
     type: data_loader
     upstream_blocks: []
     downstream_blocks: []
     executor_type: ecs
     executor_config:
       cpu: 1024
       memory: 2048
   ```

To run the whole pipeline in one ECS executor, you can set the `executor_type` at pipeline level and set `run_pipeline_in_one_process` to true.
`executor_config` can also be set at pipeline level. Here is the example pipeline metadata.yaml:

```yaml theme={"system"}
blocks:
- ...
- ...
executor_type: ecs
run_pipeline_in_one_process: true
name: example_pipeline
...
```

## Configurations

| Field name | Description | Example values |
| - | - | - |
| assign\_public\_ip | Whether to assign public IP to the ECS task. | true/false (default: true) |
| cpu | The CPU allocated to the ECS task. | 1024 |
| enable\_execute\_command | Whether to [enable execute command for debugging](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) | true/false (default: false) |
| launch\_type | The launch type of the ECS task. | FARGATE |
| memory | The memory allocated to the ECS task. | 2048 |
| tags | The tags of the ECS task. | \['tag1', 'tag2'] |
| wait\_timeout | The maximum wait time for the ECS task (in seconds). The default wait timeout for the ECS task is 10 minutes. Setting to -1 will disable waiting. | 1200 (default: 600) |

### Example

```yaml theme={"system"}
ecs_config:
  cpu: 1024
  memory: 2048
  assign_public_ip: false
  enable_execute_command: true
  wait_timeout: 1200
```

## IAM permissions

Required IAM permissions for using ECS executor:

```json theme={"system"}
[
  "ec2:DescribeNetworkInterfaces",
  "ecs:DescribeTasks",
  "ecs:ListTasks",
  "ecs:RunTask",
  "iam:PassRole"
]
```

***

## Resource management

> **[Get started for free](https://cloud.mage.ai/sign-up?source=aws-executor)**
>
> ![](https://mage-ai.github.io/assets/pro/banner-dark.png)
>
> A fully managed service, where we maintain the infrastructure, guarantee uptime,
> automatically scale your workloads to handle any volume of pipeline runs,
> automatically upgrade new versions of Mage Pro only features,
> monitor your production pipelines, and provide enterprise level support.

[Get started with Mage Pro for free](https://cloud.mage.ai/sign-up?source=aws-executor)


# Azure Container Instance executor
Source: https://docs.mage.ai/production/executors/azure

Execute block runs in separate container instances.

If your Mage app is deployed on Microsoft Azure with Mage’s [Terraform](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/azure)
scripts, you can choose to launch separate Azure container instances to execute blocks.

How to configure pipeline to use Azure Container Instance executor:

1. Update Project's metadata.yaml

```yaml theme={"system"}
azure_container_instance_config:
  cpu: 1
  memory: 2
```

2. Update the `executor_type` of the block to `azure_container_instance` in pipeline's metadata.yaml and specify `executor_config` optionally.
   The block level executor\_config will override the global executor\_config.

```yaml theme={"system"}
blocks:
- uuid: example_data_loader
  type: data_loader
  upstream_blocks: []
  downstream_blocks: []
  executor_type: azure_container_instance
  executor_config:
    cpu: 1
    memory: 2
  ...
```

***

## Resource management

> **[Get started for free](https://cloud.mage.ai/sign-up?source=azure-executor)**
>
> ![](https://mage-ai.github.io/assets/pro/banner-dark.png)
>
> A fully managed service, where we maintain the infrastructure, guarantee uptime,
> automatically scale your workloads to handle any volume of pipeline runs,
> automatically upgrade new versions of Mage Pro only features,
> monitor your production pipelines, and provide enterprise level support.

[Get started with Mage Pro for free](https://cloud.mage.ai/sign-up?source=azure-executor)


# GCP Cloud Run executor
Source: https://docs.mage.ai/production/executors/gcp

Execute block runs in separate jobs.

If your Mage app is deployed on GCP Cloud Run, you can choose to launch separate GCP Cloud Run jobs to execute blocks.

How to configure pipeline to use GCP cloud run executor:

1. Update Project's metadata.yaml

```yaml theme={"system"}
gcp_cloud_run_config:
  path_to_credentials_json_file: "/path/to/credentials_json_file"
  project_id: project_id
  timeout_seconds: 600
```

2. Update the `executor_type` of block to `gcp_cloud_run` in pipeline's metadata.yaml:

```yaml theme={"system"}
blocks:
- uuid: example_data_loader
  type: data_loader
  upstream_blocks: []
  downstream_blocks: []
  executor_type: gcp_cloud_run
  ...
```

Customizing compute resource for GCP Cloud Run executor is coming soon.

***

## Resource management

> **[Get started for free](https://cloud.mage.ai/sign-up?source=gcp-executor)**
>
> ![](https://mage-ai.github.io/assets/pro/banner-dark.png)
>
> A fully managed service, where we maintain the infrastructure, guarantee uptime,
> automatically scale your workloads to handle any volume of pipeline runs,
> automatically upgrade new versions of Mage Pro only features,
> monitor your production pipelines, and provide enterprise level support.

[Get started with Mage Pro for free](https://cloud.mage.ai/sign-up?source=gcp-executor)


# Kubernetes executor
Source: https://docs.mage.ai/production/executors/k8

Run Mage pipeline blocks in separate Kubernetes pods for scalability, isolation, and resource control.

Use Mage Pro if you need to run your blocks in separate pods or process while developing in the code editor.

Running a block from the code editor, while developing a pipeline, won’t use this executor.
Instead, the block will be executed using the same process that the code editor is running on.

[Try Mage Pro for free](https://cloud.mage.ai/sign-up?source=kubernetes-executor)

[Open](https://cloud.mage.ai/sign-up?source=kubernetes-executor)

If your Mage app is deployed in a **Kubernetes cluster**, you can configure the **Kubernetes Executor** (`executor_type: k8s`) to run each block in its own Kubernetes pod.

**Defaults in Mage Pro:**

* Namespace: `default`
* **Job name format:** `mage-data-prep-block-{block_run_id}`

You can override these defaults at the **project level**, **pipeline level**, **block level**, or with a **full Kubernetes job template**.

## Basic Setup

To configure a pipeline block to use Kubernetes executor, you simply just need to update the `executor_type` of the block to `k8s` in pipeline's metadata.yaml:

```yaml theme={"system"}
blocks:
- uuid: example_data_loader
  type: data_loader
  upstream_blocks: []
  downstream_blocks: []
  executor_type: k8s
  ...
```

By default, Mage uses `default` as the Kubernetes namespace. You can customize the namespace by setting the `KUBE_NAMESPACE` environment variable.

```bash theme={"system"}
export KUBE_NAMESPACE=my-namespace
```

## Configuration Methods

You can configure Kubernetes Executor in several places. Mage builds the
effective config by deep-merging the less specific config first, then applying
more specific config on top:

1. Environment variable defaults and hardcoded defaults
2. Project-level `k8s_executor_config`
3. Pipeline-level `executor_config`
4. Block-level `executor_config`

For a normal block-level Kubernetes executor run, block settings win over
pipeline settings, pipeline settings win over project settings, and project
settings win over environment defaults. For `run_pipeline_in_one_process: true`,
Mage uses the pipeline-level Kubernetes executor and merges only project-level
and pipeline-level settings because the whole pipeline runs in one pod.

Most nested dictionaries are merged, so you can set broad defaults at the
project level and override only the keys that need to change at the pipeline or
block level. Resource requests and limits are applied to the job container after
the pod template is built, so `executor_config.resource_requests` and
`executor_config.resource_limits` take precedence over `container.resources`.

### 1. Environment Variable Defaults

Use environment variables to set default resource requests, resource limits,
maximum allowed resource requests, maximum allowed resource limits, and service
account for all Kubernetes executor jobs. Default values are only applied when a
more specific project, pipeline, or block config does not set the same resource
group. Maximum values are Mage Pro-only guardrails: after environment defaults,
project config, pipeline config, and block config are merged, Mage fails the job
before creating the Kubernetes Job if the effective request or limit exceeds a
configured maximum.

```bash theme={"system"}
export K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_CPU=250m
export K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_MEMORY=512Mi
export K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_CPU=500m
export K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_MEMORY=1Gi
export K8S_EXECUTOR_MAX_RESOURCE_REQUESTS_CPU=2000m
export K8S_EXECUTOR_MAX_RESOURCE_REQUESTS_MEMORY=4Gi
export K8S_EXECUTOR_MAX_RESOURCE_LIMITS_CPU=4000m
export K8S_EXECUTOR_MAX_RESOURCE_LIMITS_MEMORY=8Gi
export K8S_EXECUTOR_DEFAULT_SERVICE_ACCOUNT_NAME=default
```

✅ **Use when:**
You want a cluster-wide fallback for jobs that do not define their own
Kubernetes resources.

### 2. Project-Level Configuration

Set `k8s_executor_config` in the project's `metadata.yaml` to apply defaults to
all blocks and pipeline runs that use the Kubernetes Executor in that project:

```yaml theme={"system"}
k8s_executor_config:
  job_name_prefix: data-prep
  namespace: default
  resource_limits:
    cpu: 1000m
    memory: 2048Mi
  resource_requests:
    cpu: 500m
    memory: 1024Mi
  service_account_name: default
  pod:
    node_selector:
      workload: batch
    tolerations:
      - key: "workload"
        operator: "Equal"
        value: "batch"
        effect: "NoSchedule"
```

✅ **Use when:**
You want consistent Kubernetes settings across a project, such as default
CPU/RAM, namespace, service account, or node scheduling.

### 3. Pipeline-Level Configuration

Add `executor_type: k8s` and `executor_config` to the pipeline's
`metadata.yaml`. Pipeline-level config applies to the pipeline executor when
`run_pipeline_in_one_process: true`; it also acts as a shared default for block
runs in that pipeline before block-level overrides are applied.

```yaml theme={"system"}
executor_type: k8s
run_pipeline_in_one_process: true
executor_config:
  namespace: default
  resource_limits:
    cpu: 2000m
    memory: 4096Mi
  resource_requests:
    cpu: 1000m
    memory: 2048Mi
```

✅ **Use when:**
You want every block run in one pipeline to inherit the same Kubernetes config,
or you run the entire pipeline in one Kubernetes pod with
`run_pipeline_in_one_process: true`.

### 4. Block-Level Configuration

Add `executor_type: k8s` and `executor_config` to a block in the pipeline's
`metadata.yaml`:

```yaml theme={"system"}
blocks:
- uuid: example_data_loader
  type: data_loader
  executor_type: k8s
  executor_config:
    namespace: default
    resource_limits:
      cpu: 1000m
      memory: 2048Mi
    resource_requests:
      cpu: 500m
      memory: 1024Mi
```

✅ **Use when:**
You only want to:

* Run certain blocks in the Kubernetes executor

* Override the Kubernetes executor config for specific blocks

* The Kubernetes job name is in this format: `mage-{job_name_prefix}-block-{block_run_id}`.
  The default `job_name_prefix` is `data-prep`.
  You can customize it in the `k8s_executor_config`.
  You can interpolate the trigger name in the `job_name_prefix` field with the format `{trigger_name}`.

- **GPU Support:**
  ```yaml theme={"system"}
  k8s_executor_config:
    resource_limits:
      nvidia.com/gpu: 1  # request 1 GPU
    pod:
      node_selector:
        accelerator: nvidia-tesla-v100
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
  ```
  Make sure [GPU device plugins](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#using-device-plugins) are installed.

- **Custom container & job spec:**
  ```yaml theme={"system"}
  k8s_executor_config:
    container_config:
      image: mageai/mageai:0.9.7
      env:
      - name: USER_CODE_PATH
        value: /home/src/k8s_project
    job:
      active_deadline_seconds: 120
      backoff_limit: 3
      ttl_seconds_after_finished: 86400
  ```

✅ **Use when:**
You need specialized resources or scheduling for one expensive block without
changing the rest of the pipeline.

### 5. Full Kubernetes Job Template (maximum control)

Set the `K8S_CONFIG_FILE` environment variable to the path of a YAML
configuration file. This is the raw job template for Kubernetes executor jobs.
Use `K8S_DEFAULT_CONFIG_PATH` instead when you want a reusable policy that
merges with user config as described in [Default config file](#default-config-file).

**Example template:**

```yaml theme={"system"}
metadata:
  annotations:
    application: "mage"
    component: "executor"
  labels:
    application: "mage"
    type: "spark"
  namespace: "default"

pod:
  service_account_name: "mage-service-account"
  image_pull_secrets: "my-registry-secret"
  node_selector:
    node-type: "compute"
    instance-type: "c5.xlarge"
    zone: "us-west-1a"
  scheduler_name: "custom-scheduler"
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "spot-instance"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "node-type"
                operator: "In"
                values: ["compute", "gpu"]
              - key: "instance-type"
                operator: "In"
                values: ["c5.xlarge", "c5.2xlarge"]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: "zone"
                operator: "In"
                values: ["us-west-1a", "us-west-1b"]
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: "mage-pipeline"
          topologyKey: "kubernetes.io/hostname"
  volumes:
  - name: data-pvc
    persistent_volume_claim:
      claim_name: mage-data-pvc
  - name: config-volume
    config_map:
      name: mage-config
  - name: secret-volume
    secret:
      secret_name: mage-secrets
  - name: empty-dir-volume
    empty_dir: {}

container:
  name: "mage-data"
  image: "mageai/mageai:latest"
  image_pull_policy: "IfNotPresent"
  env:
    - name: "KUBE_NAMESPACE"
      value: "default"
    - name: "secret_key"
      value: "somesecret"
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
      nvidia.com/gpu: 1
    requests:
      cpu: "0.1"
      memory: "0.5Gi"
      nvidia.com/gpu: 1
  volume_mounts:
    - mount_path: "/tmp/data"
      name: "data-pvc"
    - mount_path: "/etc/config"
      name: "config-volume"
    - mount_path: "/etc/secrets"
      name: "secret-volume"
      read_only: true
```

## Node Scheduling Configuration

### Node Selector

Use `node_selector` to schedule pods on specific nodes based on labels:

```yaml theme={"system"}
k8s_executor_config:
  pod:
    node_selector:
      node-type: "gpu"           # Schedule on GPU nodes
      instance-type: "c5.large"  # Schedule on specific instance types
      zone: "us-west-1a"         # Schedule in specific availability zones
```

### Tolerations

Use `tolerations` to allow pods to be scheduled on tainted nodes:

```yaml theme={"system"}
k8s_executor_config:
  pod:
    tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      - key: "spot-instance"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
```

### Affinity Rules

Use `affinity` for advanced scheduling rules:

```yaml theme={"system"}
k8s_executor_config:
  pod:
    affinity:
      # Node affinity - prefer specific node types
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: "node-type"
                  operator: "In"
                  values: ["compute", "gpu"]
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
                - key: "zone"
                  operator: "In"
                  values: ["us-west-1a", "us-west-1b"]

      # Pod anti-affinity - avoid co-locating similar pods
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: "mage-pipeline"
            topologyKey: "kubernetes.io/hostname"
```

### Custom Scheduler

Use a custom scheduler for advanced scheduling logic:

```yaml theme={"system"}
k8s_executor_config:
  pod:
    scheduler_name: "custom-scheduler"
```

✅ **Use when:**
You need full control over pod specs, annotations, volume mounts, and other Kubernetes settings.

## Multi-Container Pods

If your Mage deployment runs in a **multi-container pod**, set the `MAGE_CONTAINER_NAME` environment variable to specify which container runs Mage:

```yaml theme={"system"}
env:
  - name: MAGE_CONTAINER_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
```

If not set, Mage defaults to the **first container** in the pod.

## Default config file

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=kubernetes-executor)

You can provide a **default K8s executor config file** that Mage reads from the path in `K8S_DEFAULT_CONFIG_PATH` environment variable. The file is loaded as a policy and merged with block-level and project-level config. Use this to enforce cluster-wide defaults (e.g. resource limits, service account) without editing every project.

**Helm chart:** When deploying **Mage Pro** with the [Mage Helm chart](https://github.com/mage-ai/helm-charts), configure the default via **`k8sExecutorConfig`** in `values.yaml`. The chart creates a ConfigMap from your config, mounts it into the control-plane (and workspace pods), and sets `K8S_DEFAULT_CONFIG_PATH` automatically. Do not set the env var manually when using the chart. See [Using Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) for details. For non-Helm deployments, set the env var and ensure the file exists at that path (e.g. via your own ConfigMap or volume mount).

**File format:**

```yaml theme={"system"}
enabled: true   # Set to false to disable this policy
enforce: true   # If true, policy overrides user config. If false, user config overrides.

resource_limits:
  cpu: "500m"
  memory: "1Gi"
resource_requests:
  cpu: "250m"
  memory: "512Mi"
service_account_name: default

pod:
  node_selector:
    node-type: compute
  tolerations: []
  volumes: []

container:
  env: []
  volume_mounts: []
```

**Policy fields:**

| Field | Description |
| - | - |
| `enabled` | If `false`, the policy is not applied. |
| `enforce` | Controls how the default file is merged with **user config** (block + pipeline + project). See [Priority when merging](#priority-when-merging-default-file-with-user-config) below. |
| *(other keys)* | Same structure as [project-level k8s\_executor\_config](/production/executors/k8#2-project-level-configuration) (e.g. `resource_limits`, `resource_requests`, `pod`, `container`, `metadata`, `job`). |

The file must exist at the given path on the Mage control-plane. Use a ConfigMap or volume mount to provide it. When **workspaces** are created, only the K8s default config file is injected (volume + volume mount + env var) so workspace pods can read it; see [Workspaces](/guides/developer-ux/workspaces#default-workspace-config-file).

**Priority when merging default file with user config**

The default file is **merged** with the combined user config (block + pipeline + project). The **effective priority order** (who wins for a given key) depends on `enforce`:

| `enforce` | Effective priority (highest → lowest) | Result |
| - | - | - |
| **`true`** | **1. Default config file** → 2. Block → 3. Pipeline → 4. Project → 5. Env vars → 6. Hardcoded | **Policy is highest priority.** Values from the default file override block, pipeline, and project for the same keys. Use this to enforce cluster-wide defaults (e.g. resource limits, service account) that users cannot override. Conflicting values can cause an error on update when strict conflict checking is used. |
| **`false`** | **1. Block** → 2. Pipeline → 3. Project → 4. Default config file (fills gaps only) → 5. Env vars → 6. Hardcoded | **User wins.** Block, pipeline, and project override the policy for the same keys. The policy only fills in keys that user config did not set. Use this to provide optional defaults while allowing per-block, per-pipeline, or per-project overrides. |

So when **`enforce: true`**, the policy has the **highest** priority; when **`enforce: false`**, block, pipeline, and project config are higher and the policy only supplies missing values.

## Environment Variables Reference

### General Configuration

| Variable | Purpose | Default |
| - | - | - |
| `KUBE_NAMESPACE` | Namespace for job execution | default |
| `K8S_CONFIG_FILE` | Path to full Kubernetes job config file | — |
| `MAGE_CONTAINER_NAME` | Container to run Mage in multi-container pods | — |

### Resource Usage Metrics

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=kubernetes-resource-usage)

In Mage Pro, completed Kubernetes executor runs can store CPU and memory usage
summaries for the run detail **Resources** tab and the pipeline monitoring
dashboard. Mage uses Prometheus first when
`K8S_RESOURCE_USAGE_PROMETHEUS_URL` is configured; otherwise it uses the
Kubernetes Metrics API fallback when available.

Pipeline-level recommendations are available in Mage Pro after completed K8s
executor runs have stored CPU or memory samples. Mage calculates the suggested
`resource_requests` and `resource_limits` from observed usage, then shows the
values in the pipeline monitoring page. Open **Monitoring** for a pipeline, then
review **Pipeline resource usage** for compute metrics history and **Pipeline
Recommended K8s executor config** for the recommended Kubernetes executor
config. Use **Apply** to write the recommendation to the pipeline-level
`executor_config`, or copy the values into block-level `executor_config` or
project-level `k8s_executor_config` when you want a narrower or broader scope.

**How recommendations are calculated:**

| Resource | Basis | Request | Limit |
| - | - | - | - |
| CPU | `p95_cores`, then `max_cores`, then latest `current_cores` | `max(basis * 0.75, 50m)` | `max(basis * 1.5, request * 2)` |
| Memory | `p95_bytes`, then `max_bytes`, then latest `current_bytes` | `basis * 1.1` | `basis * 1.3` |

Mage marks the recommendation as `low` confidence when the run has fewer than
three samples or an OOM warning, `medium` when enough samples exist from the
Kubernetes Metrics API fallback, and `high` when Prometheus provides at least
six samples. Prometheus samples are queried with
`K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` and Mage can wait once after a run
completes, controlled by
`K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS`, before retrying sparse
completed runs. When unset, the completion delay defaults to
`K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS`.

The Kubernetes Metrics API fallback samples active pods only. To use it, install
Metrics Server, confirm `v1beta1.metrics.k8s.io` is available, and grant the
Mage service account `get` and `list` access to `pods.metrics.k8s.io` in the K8s
executor job namespace. Because the Metrics API does not keep historical pod
samples after pods finish, short runs can still have sparse metrics; use
Prometheus when you need reliable completed-run history.

| Variable | Purpose | Default |
| - | - | - |
| `K8S_RESOURCE_USAGE_PROMETHEUS_URL` | Prometheus base URL for K8s resource usage summaries | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN` | Bearer token for Prometheus requests | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME` | Username for Prometheus basic authentication | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD` | Password for Prometheus basic authentication | — |
| `K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS` | Timeout for each Prometheus query | `5` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` | Query range step for CPU and memory samples | `15` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS` | Delay before retrying Prometheus for completed runs with too few samples | `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS` | Set to `0`, `false`, `no`, or `off` to skip TLS certificate verification | `true` |

See [Monitoring with Prometheus](/observability/external/prometheus#kubernetes-resource-usage-in-mage-pro) for Prometheus query behavior, authentication details, and Kubernetes Metrics API fallback setup.

### Default Resource Configuration

Only in Mage Pro.

Try our fully managed solution to access this advanced feature.

[Get started for free](https://cloud.mage.ai/sign-up?source=kubernetes-executor)

These environment variables set default resource limits and requests, maximum
allowed resource limits and requests, and the default service account for all
K8s executor jobs. Defaults are applied when project-level, pipeline-level, or
block-level configuration does not specify resource requests or limits.
Maximums are Mage Pro-only guardrails enforced after environment defaults,
project-level config, pipeline-level config, and block-level config are merged.

| Variable | Purpose | Example Value |
| - | - | - |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_CPU` | Default CPU request for executor jobs | `250m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_MEMORY` | Default memory request for executor jobs | `512Mi` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_CPU` | Default CPU limit for executor jobs | `500m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_MEMORY` | Default memory limit for executor jobs | `1Gi` |
| `K8S_EXECUTOR_MAX_RESOURCE_REQUESTS_CPU` | Maximum CPU request allowed for executor jobs | `2000m` |
| `K8S_EXECUTOR_MAX_RESOURCE_REQUESTS_MEMORY` | Maximum memory request allowed for executor jobs | `4Gi` |
| `K8S_EXECUTOR_MAX_RESOURCE_LIMITS_CPU` | Maximum CPU limit allowed for executor jobs | `4000m` |
| `K8S_EXECUTOR_MAX_RESOURCE_LIMITS_MEMORY` | Maximum memory limit allowed for executor jobs | `8Gi` |
| `K8S_EXECUTOR_DEFAULT_SERVICE_ACCOUNT_NAME` | Default service account for executor jobs | `default` |

**Priority order without a default config file** (highest to lowest):

1. Block-level `executor_config` in the pipeline's `metadata.yaml`
2. Pipeline-level `executor_config` in the pipeline's `metadata.yaml`
3. Project-level `k8s_executor_config` in the project's `metadata.yaml`
4. Environment variable defaults (listed above)
5. Hardcoded defaults

When Mage Pro resource usage recommendations suggest new Kubernetes
`resource_requests` or `resource_limits`, apply the values at the narrowest
level you want to tune: block-level for one block, pipeline-level for all runs
of one pipeline, or project-level for all Kubernetes executor jobs in the
project.


# Local python executor
Source: https://docs.mage.ai/production/executors/python

Local python executors are ran within the same container as the scheduler service.

You can customize the compute resource with the same way mentioned in this
[document](/production/configuring-production-settings/compute-resource).
section.

### Force local Python executor

If you want to use `local_python` executor when `DEFAULT_EXECUTOR_TYPE` is set to another executor type,
you can set the `executor_type` to `local_python_force`.

In Mage Pro, sensor blocks with no executor type, or with `executor_type: local_python`, use
`local_python_force` automatically. This keeps polling in the scheduler service instead of
creating a remote executor for each sensor block run. To run a sensor remotely, set its
executor type explicitly, such as `executor_type: k8s`. In a PySpark pipeline, a sensor whose
code uses Spark continues to use the PySpark executor.


# PySpark executor
Source: https://docs.mage.ai/production/executors/spark



If the pipeline type is `pyspark`, we use PySpark executors for pipeline and
block executions.

You can customize the compute resource of PySpark executor by
updating the instance types of `emr_config` in project's metadata.yaml file.

## Example:

```yaml theme={"system"}
emr_config:
  ec2_key_name: "xxxxx"
  master_instance_type: "r5.2xlarge"
  slave_instance_type: "r5.2xlarge"
```

## Spark compute resource manager

Manage your Spark compute resources and track Spark pipeline execution metrics.

[Learn more](/integrations/compute/management)


# Mage Pro
Source: https://docs.mage.ai/production/mage/pro

A powerful, cloud-native solution to orchestrate and scale your data pipelines — available as fully managed, private, or hybrid cloud deployments.

Our fully managed solution for teams is now available!

[Try Mage Pro for free](https://cloud.mage.ai/sign-up?source=cloud-infrastructure)

[Open](https://cloud.mage.ai/sign-up?source=cloud-infrastructure)

## What is Mage Pro?

**Mage Pro** is a production-grade platform for building, running, and scaling data pipelines — with zero infrastructure hassle. You can deploy it as:

* A **fully managed cloud service** (Mage-hosted)
* A **private cloud** installation (your cloud, your control)
* A **hybrid cloud** setup (Mage manages control plane; you host workloads)

Mage Pro supports all major cloud providers including **AWS**, **GCP**, and **Azure**, and is designed for teams that need high scalability, security, and reliability.

***

## Deployment Options

### ☁️ Fully Managed Cloud (Mage-hosted)

Let Mage handle everything: we manage infrastructure, upgrades, availability, and monitoring.

### 🔒 Private Cloud (Your Infrastructure)

Run Mage Pro securely in your own cloud environment. Ideal for teams with data residency or compliance requirements.

### 🔁 Hybrid Cloud (Split Control)

Keep data and compute in your environment while Mage manages the control plane. The best of both worlds — reduced overhead, full data privacy.

***

## Migrating to Mage Pro

**1. Create multiple environments**

![Mage Pro Cloud Create multiple environments](https://mage-ai.github.io/assets/pro/cloud/cloud--cluster_new.jpg)

Visit [Mage Pro](https://cloud.mage.ai) and sign in. Spin up dedicated environments/clusters for dev, staging, and prod — with isolated workspaces, users, and resources.

**2. Customize and manage each environment**

![Mage Pro Cloud Customize and manage each environment](https://mage-ai.github.io/assets/pro/cloud/cloud--cluster_detail.jpg)

Supports **multi-tenant workspaces**, **RBAC**, and custom compute/resource settings per team or project.

**3. Monitor data pipeline CPU utilization**

![Mage Pro Cloud Monitor data pipeline CPU utilization](https://mage-ai.github.io/assets/pro/cloud/cloud--cluster_usage_cpu.jpg)

Get real-time insights into pipeline performance with built-in **CPU utilization charts** and system metrics.

**4. Monitor data pipeline memory usage**

![Mage Pro Cloud Monitor data pipeline memory usage](https://mage-ai.github.io/assets/pro/cloud/cloud--cluster_usage_memory.jpg)

Monitor and alert on memory usage across pipelines to avoid resource exhaustion and bottlenecks.

**5. Migrate existing projects instantly into Mage Pro**

![Mage Pro deployments page](https://mage-ai.github.io/assets/pro/features/deployments/deployments-page.png)

Migrate your existing projects into Mage Pro using our **Git-based import tool**.
You can connect to a remote Git repository to keep your pipeline code in sync across environments.

👉 Follow the [Version Control & Sync Guide](/guides/data-sync/version-control-guide) to learn how to integrate Mage with your Git workflows.

***

## Why Choose Mage Pro?

Mage Pro is more than just a hosted version of Mage — it’s a powerful upgrade with exclusive features, deeper integrations, and a refined developer experience for teams running data pipelines at scale.

### 🚀 Mage Pro–Only Features

#### 🧠 AI & Developer Experience

* **AI Sidekick**: Get context-aware code suggestions, explanations, and debugging for every block.
* **Enhanced Code Editor**: Smarter autocomplete, live error checking, and documentation hints.
* **Personalized UI Themes & Light Mode**: Match your workflow preferences.
* **VSCode & Cursor Integration**: Edit and sync directly with your local IDE.

#### 🛠️ Advanced Orchestration & Infrastructure

* **Workspaces**: Create isolated environments for dev, staging, and prod.
* **Pipeline-Level Resource Monitoring**: Track memory and CPU usage per pipeline.
* **Stateful Streaming Pipelines**: Run real-time pipelines with stateful checkpointing.
* **Deployment App & Version Control Terminal**: Seamlessly deploy and manage pipeline code using built-in Git tools.
* **More Variable Interpolation Syntax**: Dynamically pass values with support for env vars, secrets, and config vars.

#### 🔌 Pipeline Engine & Execution Enhancements

* **Stateful Streaming Pipelines**: Run real-time pipelines with built-in state management and checkpointing.
* **Unified IO Config Framework**: Use a single config structure across batch pipelines, streaming pipelines, and data integration pipelines.
* **New Data Sources & Destinations**: Access exclusive connectors not available in Mage OSS.
* **Enhanced Existing Integrations**: Benefit from optimized, production-ready versions of popular sources and destinations.
* **Simplified Spark & Databricks Integration**: Run distributed compute jobs with less setup and config.
* **Native Iceberg Support**: Work with open table formats natively for modern data lakes.

### 🏢 Enterprise-Grade Platform

* ☁️ **Flexible Deployments**: Fully managed, private cloud (your VPC), or hybrid cloud.
* 🔒 **Security Built-In**: RBAC, workspace isolation, secrets vault, audit logs.
* 📊 **Full Observability**: Built-in dashboards for logs, CPU/memory, errors, and alerts.
* 📈 **Autoscaling & High Availability**: Handle bursty, concurrent workloads across projects.
* 🤝 **Dedicated Support & SLAs**: White-glove onboarding, expert guidance, and 24/7 help if needed.
* 🔁 **Instant Migration**: Import OSS pipelines via Git and sync across environments.

***

## 🚀 Get Started with Mage Pro

> **[Get started for free](https://cloud.mage.ai/sign-up?source=cloud-infrastructure)**
>
> ![](https://mage-ai.github.io/assets/pro/banner-dark.png)
>
> A fully managed service, where we maintain the infrastructure, guarantee uptime,
> automatically scale your workloads to handle any volume of pipeline runs,
> automatically upgrade new versions of Mage Pro only features,
> monitor your production pipelines, and provide enterprise level support.