# Changelog
Source: https://docs.mage.ai/about/changelog
Latest new features and changelog.
Learn more about the latest features and improvements
# Code of Conduct
Source: https://docs.mage.ai/about/code-of-conduct
We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a *magical* โจ experience for everyone.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* ๐ช Deliver magical experiences: demonstrating empathy and kindness toward others.
Being respectful of differing opinions, viewpoints, and experiences
* ๐ Give people power-ups: in data engineering *and* open-source development,
doing your best to help others learn and grow
* ๐ฌ Victorious as a team: working together to build the best product possible,
recognizing that teamwork can amplify outcomes beyond that of an individual
* ๐ โโ๏ธ No Ego: giving and gracefully accepting constructive feedback
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcementโ please reach
out to [the Mage team](mailto:hello@mage.ai) with any concerns.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate.
### 2. Warning
**Community Impact**: A violation through a single incident or series of
actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the
community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code\_of\_conduct.html][v2.1].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
# Features
Source: https://docs.mage.ai/about/features
## Data pipeline management
๐ See more
[details here](/design/data-pipeline-management).
## Notebook for building data pipelines
### 1. Data centric editor
An interactive coding experience designed for preparing data to train ML models.
Visualize the impact of your code every time you load, clean, and transform
data.
### 2. Production ready code
No more writing throw away code or trying to turn notebooks into scripts.
Each block (aka cell) in this editor is a modular file that can be tested,
reused, and chained together to create an executable data pipeline locally or in
any environment.
Read more about [blocks](/design/blocks) and how they work.
Run your data pipeline end-to-end using the command line function:
`$ mage run [project] [pipeline]`
You can run your pipeline in production environments with the orchestration
tools
* [Airflow](/guides/integrate-mage-airflow)
* [Prefect](/integrations/prefect)
### 3. Extensible
Easily add new functionality directly in the source code or through plug-ins
(coming soon).
Adding new API endpoints ([Tornado](https://www.tornadoweb.org/en/stable/)),
transformations (Python, PySpark, SQL), and charts (using
[React](https://reactjs.org/)) is easy to do (tutorial coming soon).
# Frequently Asked Questions
Source: https://docs.mage.ai/about/frequently-asked-questions
Here are some frequently asked questions about Mage and our best answers.
Mage is an open-source data pipeline tool for transforming and integrating data.
๐ง A mage is someone who uses magic. Advanced technology is indistinguishable from magic.
Weโre on a mission to make AI technology more accessible by building data tools for engineers and scientists.
Find out more about our story: [https://www.mage.ai/blog/mage-heros-journey-fantasy-epic-on-how-a-startup-rose-from-the-ashes](https://www.mage.ai/blog/mage-heros-journey-fantasy-epic-on-how-a-startup-rose-from-the-ashes)
Our tool was built with data engineers and data scientists in mind, but is not limited to those roles. Other data professionals could find value in the tool.
You can quickly and easily get started by installing Mage using Docker (recommended),ย **`pip`**, orย **`conda`**. Click [here](https://docs.mage.ai/introduction/overview#quick-start) for details.
Mage is free as long as you are self-hosted (AWS, GCP, Azure, or Digital Ocean).
Our 4 [core design principles](/design/core-design-principles) that differentiate ourselves are:
1. Easy developer experience
2. Engineering best practices built-in
3. Data is a first-class citizen
4. Scaling is made simple
Features that set us apart (some of the others might eventually have these features):
1. Mix and match SQL and Python in data pipeline tasks.
2. UI/IDE for building and managing data pipelines.
3. Data centric: we designed and built a pipeline engine ONLY for moving and transforming data. This makes it possible for us to make datasets a 1st class citizen; enabling native features such as partitioning, versioning, backfilling, data validation, testing, and data quality monitoring.
4. Extensible: we designed and built the tool with developers in mind, making sure itโs really easy to add new functionality to the source code or through plug-ins.
5. Scalable: the tool can handle very, very large datasets while transforming the data or charting it.
6. Production ready: when you build your data pipeline, it runs exactly the same in development as it does in production. Deploying the tool and managing the infrastructure in production is very easy and simple, unlike Airflow.
7. Modular: every block/cell you write is a standalone file that is interoperable; meaning it can be used in other pipelines or in other code bases.
Check out our blog [Mage vs. Fivetran](https://www.mage.ai/blog/mage-vs-fivetran).
Check out our blog [Mage vs Airbyte](https://www.mage.ai/blog/mage-vs-airbyte).
Mage provides an interactive notebook with built-in engineering best practices for building pipelines, which makes prototyping and building production-ready pipelines much easier.
Mage supports writing pipelines in multiple languages which include Python, SQL, and R.
Mage supports multiple types of pipelines natively such as:
* Standard batch pipelines
* Data integration pipelines
* Streaming pipelines
* Spark pipelines
* DBT pipelines
We currently support SQL, Python, R, and PySpark. Coming soon: Spark SQL.
Yes! [Here](https://docs.mage.ai/integrations/spark-pyspark) is a step-by-step tutorial to use Mage withย Sparkย on EMR.
Sagemaker is used to train machine learning models and serve them via api.
Mage is an engine for running data pipelines that can move and transform data. That data can then be stored anywhere (e.g. S3) and used to train models in Sagemaker.
Databricks provides infrastructure to run Spark. They also provide notebooks that can run your code in Spark as well.
Mage can execute your code in a Spark cluster, managed by AWS, GCP, or even Databricks.
[Here is a doc](/integrations/observability/alerting-slack) to help you set up alerting for pipeline status updates in [Slack](https://www.mage.ai/chat).
We love and welcome community contributions! [Here is a doc](/contributing/overview) to get you started.
To request features, add a โFeature requestโ using the `New issue` button in GitHub from this [link](https://github.com/mage-ai/mage-ai/issues), or join our [feature-request](https://www.mage.ai/chat) Slack channel.
*Canโt find what youโre looking for? [Ask a question here](https://github.com/mage-ai/mage-ai/discussions/categories/q-a) or [join our slack](https://mage.ai/chat) for additional support!*
# Releases
Source: https://docs.mage.ai/about/releases
Latest new features and changelog.
Install the newest version of Mage
***
Install the newest version of Mage
Read full release notes
### โ๏ธ Google cloud storage source
You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.
by @TalaatHasanin in [https://github.com/mage-ai/mage-ai/pull/5334](https://github.com/mage-ai/mage-ai/pull/5334)
### ๐ฌ๏ธ ๐ฝ๏ธ Airtable Integration
Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.
by @TalaatHasanin in [https://github.com/mage-ai/mage-ai/pull/5404](https://github.com/mage-ai/mage-ai/pull/5404)
### โ๏ธ Add trim reformat action to transformer block
A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.
by @cristopheridlc in [https://github.com/mage-ai/mage-ai/pull/5321](https://github.com/mage-ai/mage-ai/pull/5321)
### ๐ปโโ๏ธ Enable polars dataframe in GCS data exporter
This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.
by @TalaatHasanin in [https://github.com/mage-ai/mage-ai/pull/5348](https://github.com/mage-ai/mage-ai/pull/5348)
### Add support to scheduler name on k8s executor
This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.
by @messerzen in [https://github.com/mage-ai/mage-ai/pull/5412](https://github.com/mage-ai/mage-ai/pull/5412)
Read full release notes
### ๐ง Memory management upgrade
This feature enhances system performance and stability by optimizing allocation, deallocation, and overall utilization of RAM resources, resulting in improved efficiency and reduced memory-related errors.
Check out the [doc](https://www.notion.so/mageai/Memory-management-upgrades-e6c5e6e360ce410094091e00c46e3bb6?pvs=4) for details.
### ๐ Dynamic blocks 2.0
Introducing an enhanced Dynamic blocks feature for creating dynamic content blocks that can adapt based on user input or data changes, offering improved flexibility and interactivity for users. This update aims to streamline workflows and enhance the overall user experience by making content more responsive and customizable.
Learn more in this [doc](https://www.notion.so/mageai/Dynamic-Blocks-2-0-d2fc11a4f48148d68b817e9bfb38a732).
### ๐ Azure DB connection via Key Vault
Securely retrieve a database connection URL stored in Azure Key Vault using environment variables for authentication.
```
AZURE_KEY_VAULT_URL
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID
```
by [@wangxiaoyou1993](https://github.com/wangxiaoyou1993) in [#5302](https://github.com/mage-ai/mage-ai/pull/5302)
### ๐๏ธ Workspace monitoring
Add an "Overview" page and "Pipeline runs" page to the Workspace Management UI. This provides some overall monitoring for all of the pipeline runs in the different workspaces without having to individually open up each workspace instance.
by @johnson-mage in [https://github.com/mage-ai/mage-ai/pull/5311](https://github.com/mage-ai/mage-ai/pull/5311)
Read full release notes
Read full release notes
Read full release notes
Read full release notes
* [0.9.66 | Shogun Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.66)
* [0.9.65 | Demon Slayer Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.65)
* [0.9.64 | Maestro Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.64)
* [0.9.63 | Halo Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.63)
* [0.9.62 | The Beekeeper Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.62)
* [0.9.60 | Yusuke Urameshi Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.60)
* [0.9.59 | Year of the Dragon Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.59)
* [0.9.50 | Wonka Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.50)
* [0.9.48 | The Boy and the Heron Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.48)
* [0.9.46 | Wish Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.46)
* [0.9.45 | Yuji Itadori Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.45)
* [0.9.43 | Attack on Titan Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.43)
* [0.9.41 | Halloween Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.41)
* [0.9.38 | Goosebumps Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.38)
* [0.9.35 | Loki Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.35)
* [0.9.34 | C-3PO Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.34)
* [0.9.30 | Cowboy Bebop Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.30)
* [0.9.28 | The Creator Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.28)
* [0.9.26 | Expend4bles Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.26)
* [0.9.23 | One Piece Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.23)
* [0.9.21 | Ahsoka Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.21)
* [0.9.19 | The Equalizer Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.19)
* [0.9.16 | Gran Turismo Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.16)
* [0.9.14 | Blue Beetle Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.14)
* [0.9.11 | Mutant Mayhem Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.11)
* [0.9.10 | Haunted Mansion Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.10)
* [0.9.80 | Dead Reckoning Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.8)
* [0.9.40 | Solar Flare Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.4)
* [0.9.00 | The Dial of Destiny Release](https://github.com/mage-ai/mage-ai/releases/tag/0.9.0)
* [0.8.93 | Rise of the Beasts Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.93)
* [0.8.86 | Fast Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.86)
* [0.8.83 | Fury of the Gods Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.83)
* [0.8.78 | Rise of the Machines Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.78)
* [0.8.75 | Guardians Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.75)
* [0.8.69 | Hairy Otter Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.69)
* [0.8.58 | Once & Always Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.58)
* [0.8.52 | The Super Mario Bros. Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.52)
* [0.8.44 | Dungeons and Dragons Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.44)
* [0.8.37 | Foundation Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.37)
* [0.8.29 | Wick Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.29)
* [0.8.27 | Shazam Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.27)
* [0.8.24 | Merlin Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.24)
* [0.8.15 | Creed Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.15)
* [0.8.11 | The Mandalorian Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.11)
* [0.8.3 | Everything Everywhere All at Once Release](https://github.com/mage-ai/mage-ai/releases/tag/0.8.3)
* [0.7.98 | Quantumania Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.98)
* [0.7.90 | That '90s Show Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.90)
* [0.7.84 | Rabbit Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.84)
* [0.7.74 | Lunar Release](https://github.com/mage-ai/mage-ai/releases/tag/0.7.74)
# Roadmap
Source: https://docs.mage.ai/about/roadmap
Mage's 2024 Roadmap coming soon!
# Help improve the tool
Source: https://docs.mage.ai/about/statistics
Please contribute usage statistics to help improve the developer experience for you and everyone in the community ๐ค.
All usage statistics are completely anonymous.
Itโs impossible for Mage to know which statistics belongs to whom.
## ๐ Why is this important?
By opting into sending usage statistics to [Mage](https://www.mage.ai/),
itโll help the team and community of contributors (Magers) learn whatโs going wrong with the tool
and what improvements can be made.
In addition to helping reduce potential errors,
youโll help inform which features are useful and which need work.
## ๐ค What usage statistics am I sending?
### Project UUID
Each project will have a universally unique identifier.
This will help Mage count how many projects have been created.
Itโs impossible to associate a UUID with a project without knowing the pair together.
Your project UUID is stored in the projectโs `metadata.yaml` file,
located at the root of your project: `[project_name]/metadata.yaml`.
Here is an example of what it could look like:
```yaml theme={"system"}
variables_dir: ~/.mage_data
# ...
project_uuid: 4279d28ab1f64644b1f2f4f779be7b7e
```
### Number of pipelines
Sending usage statistics will include the number of pipelines in a single project.
This will help improve the coding experience when building pipelines.
### Number of pipeline runs
Sending usage statistics will include the number of times any pipeline has ran in a single project.
This will help add better pipeline management features.
### Number of users
Sending usage statistics will include the number of users in a single project.
This will help improve the collaboration capabilities of the tool.
This usage statistic is only included if
[user authentication](/production/authentication/overview) is enabled.
### Errors
When an application error occurs in Mage, the error type, error message, and offending line of code
will be included in the usage statistics.
This will help fix bugs and improve the developer experience.
### Platform
The operating system, release, version, etc of the machine that Mage is running on.
This information will help reproduce errors.
## ๐คทโโ๏ธ How does this work?
Usage statistics are anonymously sent to Mageโs online server.
Hereโs a sample of the JSON payload containing usage statistics that could be sent:
```json theme={"system"}
{
"usage_statistic": {
"project_uuid": "4279d28ab1f64644b1f2f4f779be7b7e",
"pipelines": 40,
"pipeline_runs": 357,
"users": 13,
"platform": "Linux-5.15.49-linuxkit-aarch64-with-glibc2.31",
"version": "0.8.70",
"error": {
"message": "...",
"traceback": "..."
}
}
}
```
### Enable
To enable sending usage statistics, add a key in the projectโs `metadata.yaml` file called
`help_improve_mage` with the value `true`.
Here is an example:
```yaml theme={"system"}
project_uuid: 4279d28ab1f64644b1f2f4f779be7b7e
help_improve_mage: true
```
### Disable
To disable, change the value of `help_improve_mage` to `false`.
Here is an example:
```yaml theme={"system"}
project_uuid: 4279d28ab1f64644b1f2f4f779be7b7e
help_improve_mage: false
```
# Authentication
Source: https://docs.mage.ai/agent/authentication
Log in to your Mage Pro cluster and manage credentials.
## Interactive login
```bash theme={"system"}
mage-agent login
```
The CLI prompts for:
1. Your Mage Pro cluster URL (e.g. `https://app.mage.ai`)
2. Email
3. Password
On success, mage-agent writes your credentials to `~/.mage-agent/config.json` and fetches remote skill files from the cluster.
## Non-interactive login
Pass all values as flags to skip prompts โ useful in CI or scripted environments:
```bash theme={"system"}
mage-agent login \
--cluster-url https://your-mage-url \
--email YOUR_EMAIL \
--password YOUR_PASSWORD
```
## Credential file
Credentials are stored at `~/.mage-agent/config.json` with `600` permissions (owner read/write only):
```json theme={"system"}
{
"cluster_url": "https://your-mage-url",
"oauth_token": "YOUR_OAUTH_TOKEN"
}
```
Do not commit `~/.mage-agent/config.json` to version control or share it. The OAuth token provides full access to your Mage Pro cluster.
You can also create this file manually if you already have an OAuth token โ mage-agent will pick it up automatically.
## Re-authenticating
If you see `AUTH_EXPIRED` errors, your session token has expired. Run `mage-agent login` again to get a fresh token.
## How authentication is used
All MCP tool calls and sync operations send the OAuth token as a bearer credential to your cluster. You can override the default API key by adding an `api_key` field to the config file:
```json theme={"system"}
{
"cluster_url": "https://your-mage-url",
"oauth_token": "YOUR_OAUTH_TOKEN",
"api_key": "YOUR_API_KEY"
}
```
# CLI Reference
Source: https://docs.mage.ai/agent/cli-reference
All mage-agent commands and their options.
## Global flags
These flags apply to every command:
| Flag | Default | Description |
| ----------------- | ------- | -------------------------------------------------- |
| `--log-level` | `INFO` | Log verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `--format-output` | `json` | Output format: `json` or `human` |
You can also set `MAGE_AGENT_LOG_LEVEL` as an environment variable:
```bash theme={"system"}
MAGE_AGENT_LOG_LEVEL=DEBUG mage-agent sync-status
```
***
## `mage-agent login`
Authenticate with your Mage Pro cluster and save credentials.
```bash theme={"system"}
mage-agent login [OPTIONS]
```
| Option | Description |
| --------------------- | -------------------- |
| `--cluster-url URL` | Mage Pro cluster URL |
| `--email EMAIL` | Account email |
| `--password PASSWORD` | Account password |
If any option is omitted, the CLI prompts interactively.
On success, writes `~/.mage-agent/config.json` and fetches remote skill files from the cluster.
***
## `mage-agent mcp`
Start the MCP stdio server. This is the command your IDE or AI agent launches.
```bash theme={"system"}
mage-agent mcp
```
No options. The server reads JSON-RPC requests from stdin and writes responses to stdout. Logs go to stderr.
***
## `mage-agent sync-project-to-local`
Pull all remote project files to a local directory.
```bash theme={"system"}
mage-agent sync-project-to-local [--local-project-dir PATH]
```
| Option | Default | Description |
| --------------------- | ------------------------ | --------------------- |
| `--local-project-dir` | `.mage-agent/local_sync` | Destination directory |
***
## `mage-agent sync-status`
Show what has changed between local and remote.
```bash theme={"system"}
mage-agent sync-status [--local-project-dir PATH]
```
Output groups files into: `remote_to_local`, `local_to_remote`, `conflicts`, `in_sync`.
Use `--format-output human` for a readable table.
***
## `mage-agent sync-local-to-remote`
Push locally changed files to the cluster.
```bash theme={"system"}
mage-agent sync-local-to-remote [--local-project-dir PATH]
```
Only files with `local_to_remote` status are uploaded.
***
## `mage-agent sync`
Bidirectional sync with interactive conflict resolution.
```bash theme={"system"}
mage-agent sync [--local-project-dir PATH]
```
For each conflicting file, prompts: `[r]emote / [l]ocal / [s]kip`.
# IDE Integration
Source: https://docs.mage.ai/agent/ide-integration
Connect Cursor, Claude Code, or OpenAI Codex CLI to your Mage Pro cluster via MCP.
mage-agent exposes a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server over stdio. Any MCP-compatible AI client can connect by launching `mage-agent mcp` as a subprocess.
## Prerequisites
Complete [Installation](/agent/installation) and [Authentication](/agent/authentication) before setting up IDE integration.
***
## Cursor
In Cursor, go to **Settings โ MCP** (or open the Command Palette and search for "MCP Servers").
Add the following server config:
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "mage-agent",
"args": ["mcp"]
}
}
}
```
Save the config and reload the window. Confirm that `mage-agent` tools appear in the MCP tool list.
***
## Claude Code
```bash theme={"system"}
claude mcp add mage-agent -- mage-agent mcp
```
Or add it manually to `.claude/settings.json` in your project (or `~/.claude/settings.json` globally):
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "mage-agent",
"args": ["mcp"]
}
}
}
```
Restart the Claude Code session. Run a quick sanity check:
> "Use the `pipeline_list` tool to list my Mage pipelines"
***
## OpenAI Codex CLI
Add `mage-agent` to the `mcpServers` section of your Codex config file:
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "mage-agent",
"args": ["mcp"]
}
}
}
```
Restart Codex CLI and verify tools and resources are visible.
***
## PATH issues
If `mage-agent` is not found when the IDE launches the MCP server, use the absolute path to the executable:
```bash theme={"system"}
which mage-agent
# e.g. /Users/you/.venv/bin/mage-agent
```
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "/Users/you/.venv/bin/mage-agent",
"args": ["mcp"]
}
}
}
```
***
## Skill resources
When connected, your AI assistant can also read four Markdown guidance resources that describe how to interact with Mage:
| Resource URI | Content |
| -------------------- | -------------------------------------- |
| `skill://pipeline` | Pipeline CRUD patterns and conventions |
| `skill://blocks` | Block inspection and editing guidance |
| `skill://executions` | Run triggers and log retrieval |
| `skill://sync` | Sync operations and conflict handling |
These are loaded automatically from your cluster at login time (with bundled fallbacks if the cluster is unreachable).
# Installation
Source: https://docs.mage.ai/agent/installation
System requirements and install options for mage-agent.
## Requirements
* Python **3.10** or newer
* A Mage Pro cluster (URL + account credentials)
## Download from Mage Pro
In Mage Pro, open **Settings** > **Platform** > **Mage Agent** to download the latest Mage Agent installer for Codex or Claude Code.
The page lists the latest GitHub release assets for macOS and Windows, including file sizes and checksums. Download links use GitHub's stable latest-release asset URLs, so Mage Pro does not need source changes when release tags or asset URLs change. Use the download that matches your operating system and AI coding client, then follow the prompts from the installer.
## Install from PyPI
```bash theme={"system"}
pip install mage-agent
```
We recommend installing inside a virtual environment:
```bash theme={"system"}
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install mage-agent
```
Verify:
```bash theme={"system"}
mage-agent --version
```
## Install from source
```bash theme={"system"}
git clone https://github.com/mage-ai/mage-agent.git
cd mage-agent
python -m venv .venv
source .venv/bin/activate
pip install -e .
```
## IDE PATH requirement
When you configure your IDE to launch `mage-agent mcp`, the IDE may use a different `PATH` than your terminal. If the MCP server fails to start, use the absolute path to the executable:
```bash theme={"system"}
which mage-agent # e.g. /Users/you/.venv/bin/mage-agent
```
Then update your IDE config:
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "/Users/you/.venv/bin/mage-agent",
"args": ["mcp"]
}
}
}
```
# Introduction
Source: https://docs.mage.ai/agent/introduction
Build and manage Mage data pipelines directly from your AI coding assistant.
**Mage Agent** connects your local development environment to your Mage Pro cluster through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io). Your AI assistant can inspect, create, modify, and run Mage pipelines using natural language โ and a built-in sync engine keeps local files in step with the cluster.
Get connected in under five minutes.
Connect Cursor, Claude Code, or Codex CLI.
Mirror your Mage project to local files.
Full reference for every tool your AI can call.
## How it works
```
Your AI assistant โโMCPโโโบ mage-agent โโRESTโโโบ Mage Pro cluster
โ โ
โ sync engine
โโโ local files โโโโโโโโโโโโ
```
1. **MCP server** โ mage-agent runs as a background stdio process. Your IDE or AI agent sends tool calls over JSON-RPC; mage-agent translates them into Mage Pro REST API requests.
2. **Skill resources** โ the server exposes Markdown guidance files that give your AI assistant context on how to interact with pipelines, blocks, runs, and sync operations.
3. **Sync engine** โ a separate CLI layer lets you pull the full remote project tree to a local directory, push local edits back, and resolve conflicts file by file.
## Supported clients
Any MCP-compatible client that can launch a subprocess will work with the same configuration.
# Mage Agent Plugin
Source: https://docs.mage.ai/agent/mage-agent-plugin
Install the Mage Agent Plugin and connect it to a Mage cluster.
# Mage Agent Plugin Installation Guide
## What It Is
The Mage Agent Plugin connects an AI agent client to Mage so the agent can work
with the Mage clusters you authorize. After installation and sign-in, the agent
can help inspect and operate Mage workflows, including pipelines, blocks,
schedules, runs, logs, outputs, files, and other Mage project resources.
## Where To Download It
Download the installer from the public Mage Agent Plugin releases repository:
```text theme={"system"}
https://github.com/mage-ai/agent-plugin/releases
```
The `mage-ai/agent-plugin` repository is the public release-asset repository.
The plugin source and release automation are maintained in the Mage plugin
source under:
```text theme={"system"}
mage-pro/plugins/mage-agent-plugin
```
## Choose The Correct Installer
Use the package that matches your agent client and operating system.
macOS:
```text theme={"system"}
Mage-Agent-MacOS-Codex.pkg
Mage-Agent-MacOS-Claude.pkg
```
Windows:
```text theme={"system"}
Mage-Agent-Windows-Codex.zip
Mage-Agent-Windows-Claude.zip
```
Use the Codex package if you are installing the plugin for Codex. Use the
Claude package if you are installing it for Claude Code.
## Prerequisites
Before installing, make sure you have:
* Access to the Mage cluster you want the agent to use.
* The cluster name or cluster URL.
* The correct agent client installed, such as Codex or Claude Code.
* For Claude Code, the Claude Code CLI must be installed and support plugins.
For Claude Code, verify:
```bash theme={"system"}
claude plugin --help
```
## Install On macOS
1. Open the GitHub Releases page:
```text theme={"system"}
https://github.com/mage-ai/agent-plugin/releases
```
2. Download the correct `.pkg` file:
```text theme={"system"}
Mage-Agent-MacOS-Codex.pkg
Mage-Agent-MacOS-Claude.pkg
```
3. Open the downloaded package.
4. Follow the installer prompts.
5. When the installer finishes, the Mage plugin is installed for the selected
agent client.
The macOS installer starts the Mage OAuth sign-in helper after installing the
plugin.
## Install On Windows
1. Open the GitHub Releases page:
```text theme={"system"}
https://github.com/mage-ai/agent-plugin/releases
```
2. Download the correct `.zip` file:
```text theme={"system"}
Mage-Agent-Windows-Codex.zip
Mage-Agent-Windows-Claude.zip
```
3. Extract the zip file.
4. Run the installer command inside the extracted folder.
For Codex:
```powershell theme={"system"}
.\Install-Codex.cmd
```
For Claude Code:
```powershell theme={"system"}
.\Install-Claude-Code.cmd
```
## Sign In To Mage
After installing, sign in to Mage and connect the plugin to a cluster.
If the installer opens the sign-in flow automatically, complete the browser
login and select or provide the Mage cluster.
If you need to sign in manually from the plugin source checkout, run:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-login
```
Example:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-login mageai-1-tutorials
```
You can also use a full cluster URL:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-login https://your-mage-cluster-url
```
## Switch Clusters
To connect the plugin to a different Mage cluster, run:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-switch-cluster
```
To skip the cluster selection screen, pass the cluster name or URL directly:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-switch-cluster
```
Example:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-switch-cluster mageai-1-tutorials
```
## Verify The Active Cluster
Check which Mage cluster is currently connected:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-current-cluster
```
List saved local cluster state:
```bash theme={"system"}
./plugins/mage-agent-plugin/scripts/mage-list-clusters
```
## What Gets Saved Locally
After sign-in, the plugin stores the active Mage connection locally at:
```text theme={"system"}
~/.mage/agent/config.json
```
That file stores the selected `cluster_url` and the token needed for the plugin
to call Mage on your behalf.
OAuth helper logs are written to:
```text theme={"system"}
~/.mage/agent/codex-oauth.log
```
## Confirm It Works
After installation and sign-in:
1. Open your agent client.
2. Ask the agent to use Mage.
3. Try a read-only request first, such as listing available Mage pipelines or
checking the current cluster.
4. If the agent asks you to authenticate, complete the browser sign-in and retry
the request.
## Quick Checklist
* Download installer from `https://github.com/mage-ai/agent-plugin/releases`.
* Pick Codex or Claude package.
* Pick macOS `.pkg` or Windows `.zip`.
* Install the package.
* Sign in to Mage.
* Provide or select the Mage cluster name or URL.
* Verify the active cluster.
* Ask the agent to run a simple Mage read-only request.
# MCP Tools
Source: https://docs.mage.ai/agent/mcp-tools
Full reference for every tool your AI assistant can call via mage-agent.
Once mage-agent is connected to your IDE, your AI assistant has access to the following tools. All tools require valid authentication.
In MCP clients that support rendered tool output, mage-agent returns compact
resource widgets for supported resource actions. Widgets are currently
available for pipelines, blocks, pipeline runs, block runs, block outputs,
pipeline schedules, and pipeline triggers. They do not change the tool
arguments or authentication flow; clients that do not support rendered output
still receive the normal structured tool response.
## Resource Widget Contract
The widget path is an additive MCP Apps enhancement for read-only resource
tools. A widget-enabled tool descriptor includes both `_meta.ui.resourceUri` and
`_meta["openai/outputTemplate"]` pointing at the registered HTML resource. When
the tool succeeds, mage-agent returns:
* `structuredContent.kind = "mage_resource_widget"`
* `structuredContent.schema_version` for renderer compatibility
* `structuredContent.resource` and `structuredContent.action`
* `structuredContent.state` as `ok`, `empty`, `loading`, or `error`
* sanitized `records`, `summary`, `columns`, `filters`, and `pagination`
* optional sanitized `mutation` metadata for create, update, or delete results
* optional sanitized `graph` data for pipeline-oriented detail widgets, using
normalized `nodes` and `edges` instead of raw pipeline YAML
* `content[0].text` as a JSON fallback for clients that do not render widgets
* `_meta["openai/widgetData"]` with widget-only rendering hints
List and get widgets render the normal table or detail view. Mutating actions
may render the returned record with a mutation summary banner and footer, such as
`Operation: Create`; delete results without a returned record render a compact
success state. The adapter must redact sensitive fields before they enter
`structuredContent`, fallback `content`, widget `_meta`, or activity logs.
Pipeline-oriented widgets may include a compact graph preview when Mage returns
block dependency metadata. The graph shape is:
```json theme={"system"}
{
"nodes": [
{
"id": "load_customers",
"label": "Load customers",
"type": "data_loader",
"status": "completed",
"selected": false
}
],
"edges": [
{
"source": "load_customers",
"target": "segment_customers"
}
],
"totalNodes": 12,
"truncated": false
}
```
Renderers should treat graph data as optional. If it is missing, malformed, or
too large, the widget must fall back to the existing list/detail table without
blocking the tool response.
## Adding Widget Support
To add another Mage resource widget:
1. Add the resource/action pair to `widget_contracts.py`.
2. Add presentation columns, summary fields, empty/error copy, and guidance in
`widget_registry.py`.
3. Add adapter tests that cover populated, empty, malformed, large-result, and
secret-bearing payloads.
4. Add renderer coverage in `widgets/resource_widget` and the app-shell preview
route under `/apps/test/resource-widgets`.
5. Run the MCP protocol tests and an EKS mocked app-shell visual check before
calling the change QA-ready.
For normal Mage trigger inventory, active trigger counts, API/webhook trigger
lists, run-once schedules, and next run dates, use `pipeline_schedules`. Use
`pipeline_triggers` only when the user explicitly asks for the advanced
`PipelineTriggerResource` config surface scoped to a pipeline.
***
## Codex chat context
Read the visible non-chat text that Mage Pro AI Chat captured from the user's current browser page for the active chat turn.
This tool is only registered when Mage Pro AI Chat starts Codex with captured page context. Standard IDE MCP sessions do not expose it unless `MAGE_AGENT_CURRENT_PAGE_CONTEXT_PATH` points to a valid context sidecar file.
| Parameter | Type | Required | Description |
| -------------- | ------- | -------- | ---------------------------------------------------------------------- |
| `include_text` | boolean | no | Return the captured page text. Set to `false` to return metadata only. |
The context is scoped to the authenticated user and chat session, excludes the Codex chat UI, and should only be used for page-relative requests such as "explain this page" or "what am I looking at?"
***
## Pipelines
List all pipelines in the project.
**Arguments:** none
**Example prompt:** *"List all my Mage pipelines"*
Get details for a single pipeline.
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ------------- |
| `uuid` | string | yes | Pipeline UUID |
Create a new pipeline.
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ------------------------------------------------- |
| `name` | string | yes | Pipeline name |
| `type` | string | no | `python` (default), `streaming`, or `integration` |
**Example prompt:** *"Create a new streaming pipeline called `realtime_events`"*
Update fields on an existing pipeline.
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | ------------------- |
| `uuid` | string | yes | Pipeline UUID |
| `name` | string | no | New name |
| `description` | string | no | Updated description |
Delete a pipeline by UUID.
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ------------- |
| `uuid` | string | yes | Pipeline UUID |
Deletion is permanent and cannot be undone.
***
## Blocks
List all blocks in a pipeline.
| Parameter | Type | Required | Description |
| --------------- | ------ | -------- | ------------- |
| `pipeline_uuid` | string | yes | Pipeline UUID |
Get block details including source code.
| Parameter | Type | Required | Description |
| --------------- | ------ | -------- | ------------- |
| `pipeline_uuid` | string | yes | Pipeline UUID |
| `uuid` | string | yes | Block UUID |
Create a new block in a pipeline.
| Parameter | Type | Required | Description |
| ----------------- | ------ | -------- | -------------------------------------------------- |
| `pipeline_uuid` | string | yes | Pipeline UUID |
| `name` | string | yes | Block name |
| `type` | string | yes | Block type (see below) |
| `language` | string | no | `python` (default), `sql`, `r`, `yaml`, `markdown` |
| `content` | string | no | Initial source code |
| `upstream_blocks` | array | no | UUIDs of upstream blocks |
| `config` | object | no | Additional block configuration |
**Supported block types:**
`data_loader`, `transformer`, `data_exporter`, `sensor`, `callback`, `chart`, `conditional`, `custom`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`
If no `config` is provided, mage-agent automatically matches a block template for you. For `integration` pipelines, it also auto-detects available sources and destinations.
**Example prompt:** *"Add a SQL transformer block to the `customer_etl` pipeline that filters rows where revenue > 1000"*
Update a block's code or configuration.
| Parameter | Type | Required | Description |
| ----------------- | ------ | -------- | ---------------------------- |
| `pipeline_uuid` | string | yes | Pipeline UUID |
| `uuid` | string | yes | Block UUID |
| `content` | string | no | New source code |
| `name` | string | no | New name |
| `upstream_blocks` | array | no | Updated upstream block UUIDs |
List available block templates.
| Parameter | Type | Required | Description |
| --------------- | ------- | -------- | ------------------------------------------------ |
| `force_refresh` | boolean | no | Bypass the daily cache and re-fetch from cluster |
Templates are cached for 24 hours at `~/.mage-agent/cache/block_templates.json`.
***
## Runs & Logs
Trigger a pipeline run.
| Parameter | Type | Required | Description |
| --------------- | ------ | -------- | -------------------------------------- |
| `pipeline_uuid` | string | yes | Pipeline UUID |
| `payload` | object | no | Optional run payload (variables, etc.) |
**Example prompt:** *"Run the `daily_refresh` pipeline"*
Fetch logs for a specific pipeline run.
| Parameter | Type | Required | Description |
| ----------------- | ------ | -------- | -------------------------------------------- |
| `pipeline_run_id` | string | yes | Pipeline run ID (returned by `run_pipeline`) |
**Example prompt:** *"Show me the logs for the last run"*
***
## Sync
Pull all remote project files to a local directory.
| Parameter | Type | Required | Description |
| ------------------- | ------ | -------- | -------------------------------------------------------- |
| `local_project_dir` | string | no | Local directory path (default: `.mage-agent/local_sync`) |
Show what has changed between local and remote.
| Parameter | Type | Required | Description |
| ------------------- | ------ | -------- | -------------------- |
| `local_project_dir` | string | no | Local directory path |
Push locally changed files to the cluster.
| Parameter | Type | Required | Description |
| ------------------- | ------ | -------- | -------------------- |
| `local_project_dir` | string | no | Local directory path |
***
## Mage Cloud Backend
Mage Cloud Backend tools are exposed only for installed-agent sessions that have
Cloud handoff credentials. These tools are generated from the Cloud Backend
resource inventory, so call `mage_tool_contract` before invoking a Cloud tool
when payload or query shape matters.
Cloud Backend create and update requests use the same resource-root envelope
that the backend API expects. Do not send flat mutation payloads.
| Tool | Mutation payload root |
| ------------------ | --------------------- |
| `teams` | `team` |
| `cloud_users` | `user` |
| `team_memberships` | `team_membership` |
| `invitations` | `invitation` |
Role fields use Cloud Backend integer values: owner `0`, admin `1`,
editor `2`, viewer `3`.
For example, creating a team invitation uses:
```json theme={"system"}
{
"action": "create",
"identifiers": {
"team_id": "42"
},
"payload": {
"invitation": {
"invitee_email": "user@example.com",
"role": 3
}
}
}
```
`cluster_stats` requires `query.cluster_uuid`; `cluster_uuid` is not a path
identifier for that tool.
`cloud_users` supports `list`, `get`, and `update`. It does not expose
`create`, because Cloud Backend treats `users.create` as the public signup
path rather than a private team-admin user creation API.
***
## Error responses
All tool errors return a structured envelope:
```json theme={"system"}
{
"status": "error",
"error": {
"code": "AUTH_EXPIRED",
"message": "Session token expired. Run `mage-agent login` to re-authenticate."
}
}
```
| Error code | Cause |
| ---------------- | ------------------------------------------------ |
| `CONFIG_INVALID` | Missing or malformed `~/.mage-agent/config.json` |
| `AUTH_REQUIRED` | No credentials found |
| `AUTH_EXPIRED` | 401/403 from cluster โ run `mage-agent login` |
| `API_ERROR` | Cluster API returned an error |
| `TOOL_NOT_FOUND` | Requested tool name is not registered |
| `INTERNAL_ERROR` | Unexpected internal failure |
# Quickstart
Source: https://docs.mage.ai/agent/quickstart
Install mage-agent, log in, and make your first AI-assisted pipeline call in under five minutes.
```bash theme={"system"}
pip install mage-agent
```
Verify the install:
```bash theme={"system"}
mage-agent --version
```
```bash theme={"system"}
mage-agent login
```
The CLI prompts for your cluster URL, email, and password. On success, credentials are saved to `~/.mage-agent/config.json`.
To skip the prompts, pass flags directly:
```bash theme={"system"}
mage-agent login \
--cluster-url https://your-mage-url \
--email YOUR_EMAIL \
--password YOUR_PASSWORD
```
Add the following to your IDE's MCP server config and restart the client:
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "mage-agent",
"args": ["mcp"]
}
}
}
```
See [IDE Integration](/agent/ide-integration) for client-specific setup instructions.
Ask your AI assistant:
> "List all my Mage pipelines"
> "Create a new Python transformer block in the `customer_etl` pipeline"
> "Run the `daily_refresh` pipeline and show me the logs"
The assistant calls the appropriate MCP tool automatically โ no manual API calls required.
## Next steps
Mirror your Mage project to your machine for local editing.
See every tool available to your AI assistant.
# Syncing Local Files
Source: https://docs.mage.ai/agent/sync
Mirror your Mage Pro project to a local directory and push edits back to the cluster.
The sync engine lets you work on pipeline files in your local IDE and keep them in sync with your Mage Pro cluster โ pulling remote changes down and pushing local edits back up.
## Supported file types
`.py`, `.sql`, `.r`, `.yaml`, `.yml`, `.md`, `.txt`
***
## Pull remote files to local
Download the full remote project tree to a local directory:
```bash theme={"system"}
mage-agent sync-project-to-local --local-project-dir ./my-mage-project
```
If `--local-project-dir` is omitted, the CLI prompts for a path (default: `.mage-agent/local_sync`).
***
## Check sync status
See what has changed between your local copy and the cluster:
```bash theme={"system"}
mage-agent sync-status --local-project-dir ./my-mage-project
```
Files are grouped into four categories:
| Status | Meaning |
| ----------------- | -------------------------------------------- |
| `remote_to_local` | Remote changed since last sync โ pull needed |
| `local_to_remote` | Local changed since last sync โ push needed |
| `conflicts` | Both sides changed since last sync |
| `in_sync` | No changes on either side |
For a human-readable table instead of JSON:
```bash theme={"system"}
mage-agent sync-status --local-project-dir ./my-mage-project --format-output human
```
***
## Push local changes to the cluster
```bash theme={"system"}
mage-agent sync-local-to-remote --local-project-dir ./my-mage-project
```
Only files flagged as `local_to_remote` are uploaded.
***
## Bidirectional sync with conflict resolution
Run a full two-way sync in one command:
```bash theme={"system"}
mage-agent sync --local-project-dir ./my-mage-project
```
For each conflicting file, mage-agent prompts:
```
pipelines/customer_etl/transformers/clean_names.py โ CONFLICT
Remote modified: 2024-11-01 14:32
Local modified: 2024-11-01 15:10
Keep [r]emote / [l]ocal / [s]kip?
```
Choose `r`, `l`, or `s` for each file. Conflicts are never silently overwritten.
***
## Sync via AI assistant
You can also trigger sync operations through your AI assistant using MCP tools:
| Tool | What it does |
| ----------------------- | ------------------------------------ |
| `sync_project_to_local` | Pull remote files to local directory |
| `sync_status` | Show remote/local diff |
| `sync_local_to_remote` | Push local changes to cluster |
Example prompt:
> "Pull the latest remote changes to my local project at `./my-mage-project`"
***
## How sync state is tracked
mage-agent stores a state file at `/.mage-agent-sync-state.json` after each sync. This file records the last-known timestamps for all tracked files, enabling accurate detection of what changed on each side. Do not delete or edit it manually.
Files inside hidden directories (`.git`, `.venv`, etc.) and `__pycache__` folders are automatically excluded from sync.
# Troubleshooting
Source: https://docs.mage.ai/agent/troubleshooting
Solutions for common mage-agent issues.
## Authentication errors
Your session token has expired or is missing.
```bash theme={"system"}
mage-agent login
```
This fetches a new token and overwrites the existing config.
The config file at `~/.mage-agent/config.json` is missing or malformed.
Re-run login to recreate it:
```bash theme={"system"}
mage-agent login
```
Or create the file manually:
```json theme={"system"}
{
"cluster_url": "https://your-mage-url",
"oauth_token": "YOUR_OAUTH_TOKEN"
}
```
***
## MCP / IDE issues
1. Confirm `mage-agent` is accessible from the IDE's runtime PATH โ or use an absolute path (see [Installation](/agent/installation#ide-path-requirement)).
2. Save the MCP config and **restart** the IDE or agent client.
3. Test the server directly in a terminal:
```bash theme={"system"}
mage-agent mcp
```
If it starts without errors, the issue is the IDE config. If it errors, check authentication first.
The IDE cannot find the executable. Run:
```bash theme={"system"}
which mage-agent
```
Copy the full path and use it in your IDE config:
```json theme={"system"}
{
"mcpServers": {
"mage-agent": {
"command": "/full/path/to/mage-agent",
"args": ["mcp"]
}
}
}
```
Run login again โ the MCP server may have started before credentials were saved, or the token expired during the session:
```bash theme={"system"}
mage-agent login
```
Then restart the IDE to pick up fresh credentials.
***
## Sync issues
Conflicts are detected by comparing timestamps on each side. Small clock differences between your machine and the cluster can cause false positives.
Check what the conflict looks like before choosing a side:
```bash theme={"system"}
mage-agent sync-status --local-project-dir ./my-project --format-output human
```
Then run `mage-agent sync` and choose `r` (remote) or `l` (local) for each affected file.
Only these file extensions are tracked: `.py`, `.sql`, `.r`, `.yaml`, `.yml`, `.md`, `.txt`.
Also check that the files are not inside a hidden directory (`.git`, `.venv`) or a `__pycache__` folder โ those are always excluded.
The state file lives at `/.mage-agent-sync-state.json`. If it is missing, mage-agent treats all files as new and will re-pull the full project on the next `sync-project-to-local`. Delete it intentionally only if you want a clean baseline.
***
## Enable debug logging
Add `--log-level DEBUG` to any command to see full request/response traces:
```bash theme={"system"}
mage-agent --log-level DEBUG sync-status --local-project-dir ./my-project
mage-agent --log-level DEBUG mcp
```
Or set the environment variable permanently in your shell profile:
```bash theme={"system"}
export MAGE_AGENT_LOG_LEVEL=DEBUG
```
***
## Still stuck?
Open an issue at [github.com/mage-ai/mage-agent](https://github.com/mage-ai/mage-agent/issues) and include the debug log output.
# AI Clients
Source: https://docs.mage.ai/ai/ai-client
Use and enable different AI clients.
## AI Client
You now have the option to select from various AI clients to harness
[the capabilities of Mage AI](https://docs.mage.ai/guides/ai/overview), as detailed in the
Mage AI capabilities documentation. Currently, we offer support for OpenAI and Hugging Face,
with the promise of additional AI clients being added in the future.
## Use Hugging Face Client
### Setup
#### Hugging Face Inference Endpoint
In order to utilize the Hugging Face AI Client, it is necessary to establish a Hugging Face inference endpoint.
You can set it up following [this guide](https://huggingface.co/inference-endpoints).
This process is quite straightforward. It entails
* selecting the specific model you wish to use,
* determining the hosting environment (AWS or Azure),
* specifying the geographical region,
* choosing the type of GPU.
For your convenience and based on our testing, we recommend using the
"mistralai/Mistral-7B-Instruct-v0.1" model.
Once the Inference endpoint is operational, it will provide you with an API URL and a corresponding
token for establishing a secure connection.
#### Mage Project Setup
Within your Mage project's metadata YAML configuration, please include the subsequent "ai\_config" section:
```
ai_config:
mode: 'hugging_face'
open_ai_config:
openai_api_key: key
hugging_face_config:
huggingface_api: api_url
huggingface_inference_api_token: api_token
```
The "mode" parameter determines your selection of the AI client to be employed.
It can be specified as either "open\_ai" or "hugging\_face," with the default value being set to "open\_ai."
"hugging\_face\_config" as a mandatory configuration if you choose to use the hugging face client.
This configuration includes the two essential elements obtained from the Hugging Face inference
endpoint, namely, the API and Token.
You are ready to go once the "ai\_config" is setup. At this point,
you can fully leverage Mage AI's capabilities, such as generating blocks
with text description, automatically write comments for your functions, etc.
## How to add a new AI Client
You may find it necessary to employ an AI client other than those offered by OpenAI and Hugging Face.
Additionally, you might wish to make direct calls to your Language Model (LLM).
This can be accomplished by enabling a new AI client for your specific needs.
This is an [example PR](https://github.com/mage-ai/mage-ai/pull/3850).
### Create new AI config
Create a dedicated configuration to save the params required to connect to LLM in the [config.py](https://github.com/mage-ai/mage-ai/blob/92c372b24e08148863d799d9afcdd44483c11c89/mage_ai/orchestration/ai/config.py#L19).
For instance, when using the Hugging Face client, the LLM is hosted within the
inference endpoint, mandating both the API and Token for invoking the service
for inference. In the OpenAI client, the OpenAI key is required to facilitate model inference.
### Create dedicated AI Client
Inherit the [AIClient interface](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/ai/ai_client.py)
and implements the two required functions: โinference\_with\_promptโ and โfind\_block\_paramsโ.
* Inference\_with\_prompt function does the LLM model inference.
It takes the prompt template, required variables being used in the prompt and
return the inference result.
* Find\_block\_params function does a multi classification based on code description
to generate required types including block\_type, pipeline\_type, language,
action type and data source.
You can read your configuration in the Setup function and initialize the client
to talk to your service.
### Enable in llm\_pipeline\_wizard
The last action to take is modifying the Setup function within "[mage\_ai/ai/llm\_pipeline\_wizard.py](https://github.com/mage-ai/mage-ai/blob/92c372b24e08148863d799d9afcdd44483c11c89/mage_ai/ai/llm_pipeline_wizard.py#L195)"
to introduce a new mode of your client and initialize your AI client.
# Building, Configuring, and Automating with LLMs
Source: https://docs.mage.ai/ai/blocks
Learn how to create, configure, and run AI Blocks in Mage to generate text, produce structured outputs, write and validate code, and trigger other pipelinesโempowering you to build advanced RAG and automation workflows with ease.
## 1. What is an AI Block?
An **AI block** is a special type of block in a Mage pipeline that uses a Large Language Model (LLM) to:
* Generate text or code
* Return structured data in a defined schema
* Trigger other pipelines as tools
* Validate outputs before passing them downstream
AI blocks are configured in **YAML** and can be designed to:
* Write and validate executable code
* Produce structured outputs for downstream blocks
* Decide which other pipelines to trigger and pass them variables
They integrate directly with Mageโs orchestration engine, allowing AI-generated outputs to be chained together, validated, and reused.
***
## 2. How to Use an AI Block
1. **Add a new block** in your Mage pipeline and set its type to `ai`.
2. **Write a prompt** โ the main instruction for the AI.
3. **Optionally define output settings**:
* **Code generation** (`output.code`)
* **Structured JSON schema** (`output.format`)
4. **(Optional) Configure tools** to allow the AI to trigger other pipelines.
5. **Run the pipeline** โ Mage will send your prompt and configuration to the AI model, validate the output (if specified), and pass it to the next blocks.
**Minimal example:**
```yaml theme={"system"}
prompt: Summarize the input text in two sentences
```
***
## 3. AI Block Configuration
### 3.1 Basic Structure
```yaml theme={"system"}
prompt: |
Enter what you want AI to do
output:
# Choose either `code` or `format`, not both
code:
save: true
validation: Ensure this code runs without errors
format:
name:
type: string
description: The name of the person
age:
type: integer
minimum: 0
validation: Validation prompt for output
tools:
prompt: A prompt to decide which pipeline(s) to trigger
required: false
pipelines:
- uuid: pipeline_uuid
description: Description of the pipeline
blocks:
- uuid: block_uuid
variables:
user_id:
type: string
```
***
### 3.2 Fields
#### **prompt** (Required)
The instruction to send to the AI model.
* Supports multi-line strings with `|`.
* Example:
```yaml theme={"system"}
prompt: |
Write a haiku about the moon.
```
#### **output**
Controls how the AIโs response is validated and structured.
1. **Code Output**
```yaml theme={"system"}
output:
code:
save: true # Save generated code for reuse
validation: Ensure no syntax errors
language: sql # Optional, e.g., python or sql
profile: default # SQL connection profile from io_config.yaml
client: postgres # SQL client type
```
2. **Structured Format Output**
```yaml theme={"system"}
output:
format:
title:
type: string
description: Blog post title
tags:
type: array
items:
type: string
published:
type: boolean
validation: Ensure all fields are present and valid
```
3. **Unstructured Output**
```yaml theme={"system"}
output: ''
```
***
#### **tools**
Lets the AI trigger other pipelines as part of execution.
```yaml theme={"system"}
tools:
prompt: Select a pipeline to process this request
required: true
pipelines:
- uuid: abc123
description: Cleans and normalizes customer data
blocks:
- uuid: block_1
variables:
customer_id:
type: string
```
***
## 4. JSON Schema Support for Structured Outputs
The `output.format` and `variables` fields use **JSON Schema draft-07** (subset) to define:
* `type` โ string, number, integer, boolean, array, object
* `description` โ field description
* `enum` โ allowed values
* `items` โ item schema for arrays
* `minItems`, `maxItems` โ constraints for arrays
* `pattern` โ regex validation for strings
* `required` โ list of required keys
* `oneOf`, `anyOf`, `allOf` โ branching validation
* `$ref`, `$defs` โ recursive or modular schemas
**Example:**
```yaml theme={"system"}
output:
format:
email:
type: string
format: email
signup_date:
type: string
format: date-time
```
***
## 5. Best Practices
* Use **`output.code`** only when expecting executable code.
* Use **`output.format`** for predictable structured outputs.
* Avoid defining both `code` and `format` in the same block.
* Use **multi-line prompts** with `|` for readability.
* Only include fields you need โ minimal configs run faster.
* Always include a `validation` prompt for critical outputs.
* When chaining pipelines, define `variables` clearly for tool execution.
***
## 6. Example Configurations
**Minimal AI Block**
```yaml theme={"system"}
prompt: Summarize this document in bullet points
```
**AI Block with Structured Output**
```yaml theme={"system"}
prompt: Extract customer details from the text
output:
format:
name:
type: string
description: Customer's full name
email:
type: string
format: email
phone:
type: string
pattern: "^[0-9\\-]+$"
validation: Ensure all fields are filled
```
**AI Block Triggering Another Pipeline**
```yaml theme={"system"}
prompt: Analyze sales data and decide next action
tools:
prompt: Choose a data processing pipeline
required: true
pipelines:
- uuid: sales_analysis_001
description: Processes daily sales data
blocks:
- uuid: clean_data_block
variables:
date:
type: string
format: date
```
# Customized AI resources for training and inference
Source: https://docs.mage.ai/ai/custom-resources
## Custom resources
Customized GPU accelerated resources for running AI/ML/LLM pipelines.
## Inference endpoints
Deploy high-performance, low-latency API endpoints for executing blocks and returning output data,
such as inference endpoints.
# Machine learning pipeline tutorial
Source: https://docs.mage.ai/ai/ml/train-model
Build a machine learning pipeline to train a model on the Titanic dataset.
In this tutorial, weโll create a pipeline that does the following:
1. Load data from an online endpoint
2. Select columns and fill in missing values
3. Train a model to predict which passengers will survive
If you prefer to skip the tutorial and view the finished code, follow
[this guide](/guides/train/complete-project).
If you havenโt setup a project before, check out the [setup guide](/getting-started/setup)
before starting.
## 1. Setup
### 1a. Add Python packages to project
In the left sidebar (aka file browser), click on the `requirements.txt` file
under the `demo_project/` folder.
Then add the following dependencies to that file:
```text theme={"system"}
matplotlib
requests
scikit-learn
```
Then, save the file by pressing `โ + S`.
### 2a. Install dependencies
The simplest way is to run pip install from the tool.
Add a scratchpad block by pressing the `+ Scratchpad` button. Then run the
following command:
```bash theme={"system"}
pip install -r demo_project/requirements.txt
```
Alternatively, here are other ways of installing dependencies (depending on if
you are using Docker or not):
#### Docker
Get the name of the container that is running the tool:
```bash theme={"system"}
docker ps
```
Sample output:
```bash Sample output theme={"system"}
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
214e1155f5c3 mage/data "python mage_ai/commโฆ" 5 seconds ago Up 2 seconds mage-ai_server_run_6f8d367ac405
```
The container name in the above sample output is
`mage-ai_server_run_6f8d367ac405`.
Then run this command to install Python packages in the
`demo_project/requirements.txt` file:
```bash theme={"system"}
docker exec [container_name] pip3 install -r demo_project/requirements.txt
```
### pip
If you arenโt using Docker, just run the following command in your terminal:
```bash theme={"system"}
pip3 install -r demo_project/requirements.txt
```
## 2. Create new pipeline
In the top left corner, click `File > New pipeline`. Then, click the name of the
pipeline next to the green dot to rename it to `titanic survivors`.
## 3. Play around with scratchpad
There are 4 buttons, click on the `+ Scratchpad` button to add a block.
Paste the following sample code in the block:
```python theme={"system"}
import matplotlib.pyplot as plt
import numpy as np
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t, s)
plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('About as simple as it gets, folks')
plt.grid(True)
plt.show()
```
Then click the `Play button` on the right side of the block to run the code.
Alternatively, you can use the following keyboard shortcuts to execute code in
the block:
* โ + Enter
* Control + Enter
* Shift + Enter (run code and add a new block)
Now that weโre done with the scratchpad, we can leave it there or delete it. To
delete a block, click the trash can icon on the right side or use the keyboard
shortcut by typing the letter D and then D again.
## 4. Load data
1. Click the `+ Data loader` button, select `Python`, then click the template
called `API`.
2. Rename the block to `load dataset`.
3. In the function named `load_data_from_api`, set the `url` variable to:
`https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv`.
4. Run the block by clicking the play icon button or using the keyboard
shortcuts `โ + Enter`, `Control + Enter`, or `Shift + Enter`.
After you run the block (โ + Enter), you can immediately see a sample of the
data in the blockโs output.
Here is what the code should look like:
```python theme={"system"}
import io
import pandas as pd
import requests
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_api(**kwargs) -> DataFrame:
"""
Template for loading data from API
"""
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
@test
def test_output(df) -> None:
"""
Template code for testing the output of the block.
"""
assert df is not None, 'The output is undefined'
```
## 5. Transform data
Weโre going to select numerical columns from the original dataset, then fill in
missing values for those columns (aka impute).
1. Click the `+ Transformer` button, select `Python`, then click
`Generic (no template)`.
2. Rename the block to `extract and impute numbers`.
3. Paste the following code in the block:
```python theme={"system"}
from pandas import DataFrame
import math
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
def select_number_columns(df: DataFrame) -> DataFrame:
return df[['Age', 'Fare', 'Parch', 'Pclass', 'SibSp', 'Survived']]
def fill_missing_values_with_median(df: DataFrame) -> DataFrame:
for col in df.columns:
values = sorted(df[col].dropna().tolist())
median_age = values[math.floor(len(values) / 2)]
df[[col]] = df[[col]].fillna(median_age)
return df
@transformer
def transform_df(df: DataFrame, *args) -> DataFrame:
return fill_missing_values_with_median(select_number_columns(df))
```
After you run the block (โ + Enter), you can immediately see a sample of the
data in the blockโs output.
## 6. Train model
In this part, weโre going to accomplish the following:
1. Split the dataset into a training set and a test set.
2. Train logistic regression model.
3. Calculate the modelโs accuracy score.
4. Save the training set, test set, and model artifact to disk.
Here are the steps to take:
1. Add a new data exporter block by clicking `+ Data exporter` button, select
`Python`, then click `Generic (no template)`.
2. Rename the block to `train model`.
3. Paste the following code in the block:
```python theme={"system"}
from pandas import DataFrame
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
import os
import pickle
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
LABEL_COLUMN = 'Survived'
def build_training_and_test_set(df: DataFrame) -> None:
X = df.drop(columns=[LABEL_COLUMN])
y = df[LABEL_COLUMN]
return train_test_split(X, y)
def train_model(X, y) -> None:
model = LogisticRegression()
model.fit(X, y)
return model
def score_model(model, X, y) -> None:
y_pred = model.predict(X)
return accuracy_score(y, y_pred)
@data_exporter
def export_data(df: DataFrame) -> None:
X_train, X_test, y_train, y_test = build_training_and_test_set(df)
model = train_model(X_train, y_train)
score = score_model(model, X_test, y_test)
print(f'Accuracy: {score}')
cwd = os.getcwd()
filename = f'{cwd}/finalized_model.lib'
print(f'Saving model to {filename}')
pickle.dump(model, open(filename, 'wb'))
print(f'Saving training and test set')
X_train.to_csv(f'{cwd}/X_train')
X_test.to_csv(f'{cwd}/X_test')
y_train.to_csv(f'{cwd}/y_train')
y_test.to_csv(f'{cwd}/y_test')
```
Run the block (โ + Enter).
## 7. Run pipeline
We can now run the entire pipeline end-to-end. In your terminal, execute the
following command:
```bash Docker theme={"system"}
./scripts/run.sh demo_project titanic_survivors
```
```bash pip theme={"system"}
bash mage run demo_project titanic_survivors
```
You can also run the pipeline from the UI. Click on the Execute pipeline
from right bottom panel.
Your output should look something like this:
```text theme={"system"}
Executing data_loader block: load_dataset...DONE
Executing transformer block: extract_and_impute_numbers...DONE
Executing data_exporter block: train_model...Accuracy: 0.757847533632287
Saving model to /home/src/finalized_model.lib
Saving training and test set
DONE
```
***
## Congratulations!
Youโve successfully built an ML pipeline that consists of modular code blocks
and is reproducible in any environment.
If you have more questions or ideas, please live chat with us in
[Slack](https://www.mage.ai/chat).
# Retrieval Augmented Generation (RAG) pipeline builder
Source: https://docs.mage.ai/ai/rag-pipeline
Build and deploy RAG pipelines with ease.
# Using AI (artificial intelligence)
Source: https://docs.mage.ai/ai/setup
Use AI to perform various actions in Mage.
Mage uses AI to help you build data pipelines fasterโ with our OpenAI integration, you can automate the tedious parts of pipeline development and focus on the fun stuff.
You will need to add an OpenAI API key to your project before you can use AI for various actions.
We currently support the following AI features:
* Generate pipelines
* Generate blocks
* Chat with Codex in Mage Pro
* Document code
* Block documentation
* Block comments
* Pipeline documentation
* Simultaneously document *all* blocks in a pipeline
To see examples of how to use AI in Mage, check out our [AI guides](/guides/ai/overview).
## Setup
You need to add an OpenAI API key to your project before you can use AI for various actions.
Read [OpenAIโs documentation](https://help.openai.com/en/articles/4936850-where-do-i-find-my-secret-api-key)
to get your API key.
Once you have your OpenAI API key, go to project settings (click the "wizard" in the top right > Workspace > Preferences) and enter the OpenAI API key under the section labeled OpenAI.
## AI Chat in Mage Pro
Mage Pro includes AI Chat for Codex-backed workspace assistance. Open AI Chat from Mage Pro, or visit `/ai/chat` in an authenticated workspace as a user with the Editor role or higher.
For each chat turn, Mage captures the visible non-chat text from the current browser page, along with the page title and URL, so Codex can answer page-relative prompts such as "explain this page" or "what am I looking at?" The captured context is scoped to the authenticated user and chat session, excludes the Codex chat UI, and is capped at 20,000 characters.
The AI Chat sidebar lists active and archived chats, lets you search chat history by title, and loads older chats with **Show more** when more results are available. Mage Pro remembers the selected chat and whether you were viewing active or archived chats for the current workspace, then restores that state when you refresh or reopen AI Chat.
AI Chat supports session-scoped file attachments. Attachment uploads require a chat session and are limited by the `MAGE_CODEX_CHAT_ATTACHMENT_MAX_BYTES`, `MAGE_CODEX_CHAT_ATTACHMENT_MAX_COUNT`, and `MAGE_CODEX_CHAT_ATTACHMENT_MAX_SESSION_BYTES` environment variables.
When Codex returns a project-local PNG, JPEG, GIF, or WebP path, AI Chat renders the image inline. The path can be on its own line, embedded in prose, such as `Here it is: /home/src/demo/cyborg_mage.png`, or used as a Markdown link target, such as `[output/anime_styled_mage.png](output/anime_styled_mage.png)`. The preview is served through a short-lived authenticated URL, so the image file does not need a public URL.
For direct model authentication, configure `MAGE_CODEX_CHAT_OPENAI_API_KEY` or store the key in a Mage secret named by `MAGE_CODEX_CHAT_OPENAI_API_KEY_SECRET`. `OPENAI_API_KEY` is also accepted as a fallback.
For Mage Pro proxy authentication, set `MAGE_CODEX_MODEL_AUTH_MODE=proxy` and configure either `MAGE_CODEX_MODEL_PROXY_BASE_URL` or `MAGE_PRO_API_ENDPOINT` with `MAGE_PRO_API_API_KEY`. The chat model, reasoning effort, service tier, and concurrency defaults can be configured with the `MAGE_CODEX_CHAT_*` environment variables listed in [Environment variables](/development/variables/environment-variables).
## Generate pipeline using AI
When creating a new pipeline, select the option labeled Using AI.
Then, type the description of what the pipeline should do.
For example:
*Load data from an API,
then clean the column names,
and finally export the dataframe to PostgreSQL.*
## Generate block using AI
You must turn on the feature named `add_new_block_v2`
in your project settings. (click the "wizard" in the top right > Workspace > Preferences)
When adding a new block, type in the description of what you want the block to do.
In the autocomplete dropdown list, select the 1st option with a label starting with
Generate block using AI: ...
## Document code using AI
### Add documentation for a block
On the edit pipeline page in the top right corner of a block, click the AI actions icon.
Select the option labeled Document block.
### Add documentation for a pipeline and all its blocks
On the edit pipeline page in the top right corner of a block, click the AI actions icon.
Select the option labeled Document pipeline and all blocks.
### Add comments in a block
On the edit pipeline page in the top right corner of a block, click the AI actions icon.
Select the option labeled Add comments in code.
# Your AI data engineer
Source: https://docs.mage.ai/ai/sidekick/index
Upgrade from a co-pilot to a co-commander with your very own AI sidekick for pipelines, analytics, and machine learning.
**Mage Pro AI Sidekick is your intelligent assistant for data pipelines, analytics, and ML workflows.**
It helps you design, debug, and optimize your pipelines โ faster and smarter โ using natural language. Whether you're building ETL processes, exploring data, or resolving code issues, Mage Pro AI understands your intent and delivers code, context, and insights on the spot.
Use it to:
* Generate production-grade pipelines with a simple prompt
* Fix broken blocks or refactor complex logic
* Analyze pipeline behavior and explain errors
* Visualize outputs and transform them into shareable, interactive data products
Itโs like having a senior data engineer available 24/7 โ inside your repo.
## AI-Powered Answers for Data Engineering, Analytics, and ML
**Mage Pro AI** is your always-on data expert. Whether youโre building data pipelines, designing warehouse schemas, troubleshooting integrations, or making sense of your analytics stack, Mage Pro AI has the answer. Itโs trained to understand the language of modern data teams โ from ETL to orchestration to compliance โ so you can move faster, with confidence.
### How it works
Just type your question โ natural language is all it takes. Mage Pro AI instantly parses your query and provides accurate, actionable responses across a wide range of data topics: infrastructure, modeling, governance, ML, and more. Itโs built into the Mage Pro platform and fine-tuned specifically for data engineers, analysts, and architects. No docs to dig through. No syntax to memorize. Just ask.
### Why it matters
Data teams lose countless hours searching for answers across scattered documentation, Slack threads, and tribal knowledge. Mage Pro AI changes that. With expert-level responses at your fingertips, your team can solve problems faster, onboard more easily, and stay focused on high-impact work. Itโs like having a senior engineer on call โ 24/7.
***
## Design Complex Data Pipelines in Seconds
Stop wasting hours piecing together pipelines by hand. Mage Pro AI can design and build complex data workflows in seconds. Just describe your goal in plain English, and Mage Pro AI auto-generates the full pipelineโevery block, every dependency, every line of code.
### How it works
Tell Mage Pro AI what you want to buildโingest from S3, transform with Polars, join with a Delta Lake table, run daily, alert on failure. Mage instantly maps out the entire pipeline and generates working code for each step, fully integrated with your existing tools and infrastructure.
### Why it matters
Manually building pipelines is slow, error-prone, and often blocks teams from experimenting or scaling quickly. Mage Pro AI turns hours of setup into seconds of generationโso engineers can focus on what matters: logic, not boilerplate.
***
## AI-Generated Code Blocks for Data Pipelines
Design your pipeline one step at a timeโwithout writing a single line of code. Mage Pro AI can generate complete, production-ready code blocks for every stage of your data workflow.
### How it works
Tell Mage what you need:
> โLoad this CSV from S3,โ\
> โJoin with this Delta Lake table,โ\
> โClean nulls in this column,โ\
> โCalculate weekly aggregates.โ
### Why it matters
Writing every block by hand is slow and repetitive. Mage Pro AI speeds up development by giving you smart, reusable code blocks that integrate cleanly with upstream and downstream steps.
***
## Smarter, Faster Code for Every Pipeline Block
Mage Pro AI helps you evolve your workflows by intelligently updating, refactoring, and improving code blocks in place.
### How it works
Highlight a block. Ask Mage Pro AI to optimize it, update logic, or align with new requirements. The AI understands your pipeline context and rewrites the code accordingly.
### Why it matters
Mage Pro AI handles updates safely, intelligently, and instantlyโsaving hours of debugging while ensuring consistency and performance across your pipeline.
***
## Fix Pipeline Code Errors with AI-Powered Debugging
Mage Pro AI helps you resolve issues in your pipeline code with intelligent, context-aware fixes.
### How it works
Highlight the problematic block or describe the issue. Mage Pro AI analyzes the code and suggests a corrected version that works within the full pipeline context.
### Why it matters
Debugging pipeline code wastes valuable time. Mage Pro AI reduces that friction by helping your team quickly find and fix issues.
***
## Self-Healing Data Pipelines with AI
Mage Pro AI makes it possible to build pipelines that detect failures, diagnose errors, and fix themselves.
### How it works
When a pipeline breaks, Mage Pro AI iterates through debugging cycles, rewriting blocks and rerunning them until validation checks pass.
### Why it matters
Mage Pro AI automatically fixes pipelines, ensuring that data quality checks are met without wasting hours chasing down errors.
***
## Understand Your Pipeline Data with AI-Powered Q\&A and Analysis
With Mage Pro AI, you can ask questions about your pipeline data and get clear answers.
### How it works
Ask Mage Pro AI questions like โWhy did this block fail?โ or โAre there outliers in this column?โ It responds with insights, context, and suggested actions.
### Why it matters
Mage Pro AI eliminates the need to query logs or write SQLโhelping you understand your data instantly and take informed action.
***
## AI-Powered Data Visualizations for Every Step in Your Pipeline
Mage Pro AI transforms pipeline outputs into insightful, ready-to-use visualizations.
### How it works
After a block runs, Mage Pro AI analyzes its output and auto-generates the most appropriate visualizationโbar charts, line graphs, histograms, heatmaps, and more.
### Why it matters
Visual feedback helps you debug faster and understand your data more deeply at every stepโwithout needing to configure or write plotting code.
***
## Build Interactive Data Products Instantly with AI-Generated Tables
Mage Pro AI lets you turn raw pipeline outputs into smart, explorable tablesโinstantly.
### How it works
Mage Pro AI creates interactive tables with sorting, filtering, and searching automatically after your pipeline runsโno front-end or dashboard setup needed.
### Why it matters
Donโt stop at raw output. Instantly turn your data into interactive, insightful products for analysts, stakeholders, or pipelines.
## Enable RAG
**Empower Mage Pro AI to give smarter, context-aware answers by enabling RAG (Retrieval-Augmented Generation).**
When RAG is enabled, Mage Pro AI can reference your actual project files โ including pipeline code, configurations, and documentation โ to provide deeper, more accurate answers. This dramatically improves the AIโs ability to help with debugging, code generation, and pipeline understanding.
### Why Enable RAG
Without RAG, Mage Pro AI answers based solely on general knowledge and standard behaviors. With RAG enabled:
* โ The AI can **reference your exact project structure and code**
* โ Responses are **context-specific** and align with your real pipelines
* โ You get **accurate fixes, relevant suggestions, and smarter debugging**
* โ Answers reflect your own naming conventions, tools, and logic
If RAG is not enabled, the AI will ask for permission before reading any of your files each time you make a request.
### How to Enable RAG
Create a file at the root of your project named `ai_config.yaml`. This file defines which folders the AI is allowed to read and which files to exclude.
#### Option 1: Enable RAG for the entire project
Create a file at the root of your project named `ai_config.yaml`
```yaml theme={"system"}
rag:
folders:
include:
- /home/src/default_repo
files:
exclude:
# gitignore patterns
- secrets.json
# Add this if you permit the AI to read your data when asking it questions.
# If this is not present, then the AI will ask for your permission each time.
data:
all: true
```
#### Option 2: Enable RAG for selected folders only
```yaml theme={"system"}
rag:
folders:
include:
- /home/projects/mage-pro-tutorials/callbacks
- /home/projects/mage-pro-tutorials/conditionals
- /home/projects/mage-pro-tutorials/custom
- /home/projects/mage-pro-tutorials/data_exporters
- /home/projects/mage-pro-tutorials/data_loaders
- /home/projects/mage-pro-tutorials/dbt
- /home/projects/mage-pro-tutorials/extensions
- /home/projects/mage-pro-tutorials/sensors
- /home/projects/mage-pro-tutorials/transformers
- /home/projects/mage-pro-tutorials/services
- /home/projects/mage-pro-tutorials/utils
files:
exclude:
# gitignore patterns
- .env
```
### How to Trigger Reindexing
Once you've updated your `ai_config.yaml`, call the following API to reindex your project files into the vector store:
```python theme={"system"}
import requests
url = "https://cluster.mage.ai/[cluster UUID]/api/vector_store"
headers = {
"Authorization": "Bearer IwMQiEhGerIOt2DxJNiCwOlSiRigpda2FM4mTfjphq8",
"Content-Type": "application/json",
}
payload = {
"vector_store": {
"path": "/home/src/default_repo",
},
"api_key": "[API key]",
}
response = requests.post(url, headers=headers, json=payload)
```
After reindexing, Mage Pro AI will be able to answer questions using your real project dataโenabling more personalized, precise assistance.
***
## Real-World Use Cases for Mage Pro AI
### Build AI-Generated Data Pipelines
Create complete data pipelines from a single prompt. Describe your data goals in plain English and Mage Pro AI generates end-to-end ETL workflows with correct dependencies and block-level logic.
### Automate Machine Learning Pipelines
Use Mage Pro AI to generate data preparation, model training, and evaluation pipelines in seconds. Easily experiment and scale machine learning workflows without writing boilerplate code.
### Generate Code Blocks On Demand
Need to load a CSV, clean nulls, or join with a Delta Lake table? Mage Pro AI generates clean, reusable code blocks tailored to your infrastructure, tools, and pipeline logic.
### Debug and Fix Pipelines with AI
Highlight a broken step or describe the issueโMage Pro AI suggests fixes, explains errors, and updates the code in context. Eliminate hours of debugging time.
### Build Self-Healing Pipelines
Mage Pro AI can detect failed blocks, rerun them, and rewrite logic to meet validation rulesโcreating fault-tolerant pipelines that repair themselves automatically.
### Ask Questions About Your Data
Want to understand why a block failed or check for anomalies? Mage Pro AI provides contextual answers and analysisโno SQL or log digging required.
### Auto-Generate Visualizations and Tables
Mage Pro AI analyzes pipeline outputs and automatically creates helpful visualizations and interactive tablesโready for review or sharing with stakeholders.
### Rapidly Prototype and Iterate
Test new data workflows, transform logic, or refactor pipelines in seconds using natural language. Mage Pro AI makes it easy to try new ideas without slowing down.
# Blog
Source: https://docs.mage.ai/community/blog
Want to be entertained?
# Contact
Source: https://docs.mage.ai/community/contact
Send us an email
# GitHub
Source: https://docs.mage.ai/community/github
Open-source project
# LinkedIn
Source: https://docs.mage.ai/community/linkedin
Letโs connect professionally
# Slack
Source: https://docs.mage.ai/community/slack
Instantly chat with the team and other community members
# Twitter
Source: https://docs.mage.ai/community/twitter
Ready for funny memes?
# Adding an IO class
Source: https://docs.mage.ai/contributing/backend/io/adding-a-class
IO classes power Mage sources and destinations. Read on to learn how you can contribute an IO class to Mage.
## What's an IO class?
IO classes are at the heart of Mageโ they're the code that enables Mage to read and write data. When you configure your [`io_config`](https://docs.mage.ai/development/io_config) to use a source or destination, you're telling Mage to use a specific IO class.
IO classes are defined [here](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/io) in the Mage repository. You can see that there are a few different types of IO classesโ some are for reading data, some are for writing data, and some are for both.
## Why should I contribute?
If you have a favorite IO class *that's not currently supported*, you can reach out *or* contribute it to Mage! Contributing is a great way to build your skills, become a part of the Mage community, and help others.
## How do I contribute?
### Configure your development environment
The first step to contributing an IO class is to configure your development environment. You can find instructions for doing so [here](https://docs.mage.ai/contributing/development-environment).
### Create a new IO class
Once configured, you'll want to create a new file in the `mage_ai/io` directory. The file should be named after the IO class you're contributing. For example, if you're contributing an IO class called `MyIOClass`, you should create a file called `my_io_class.py`.
Some IO classes are relatively straightforward, like [exporting to local files](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/file.py) or [writing to Google Sheets](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/google_sheets.py). Others, like full database integrations, can be pretty complex. [Google BigQuery](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/bigquery.py) is a good example of a complex IO class.
Most classes are platform specific, but for a database, you might need the following methods:
* `alter_table`: Alter the table schema
* `load`: Load data from the database
* `export`: Export data to the database
* `execute`: Execute a query on the database
* `execute_queries`: Execute multiple queries on the database
Additionally, every class will need to have the following methods:
* `__init__`: Initialize the class
* `with_config`: Initialize the database client from the configuration loader. You'll notice that this method is used in *all* of our templates.
### Test your IO class
In order to test your class, you should perform all of the methods you defined in the class itself. Once you pass our linting checks (outlined in the development environment link above) *and* your tests pass, you're ready to submit a pull request! Learn more about pull requests on the contribution.
## Create a pull request
# Contributing to the backend server
Source: https://docs.mage.ai/contributing/backend/overview
Mage backend code is written in Python ๐ and our server uses the Tornado ๐ช๏ธ framework. Here are some guides on adding features to the Mage backend.
## Guides
### API
* [Overview](/api-reference/overview)
### Data integrations
* [Add a new source to the data integration pipeline](/contributing/data-integrations/add-new-source)
* [Add a new destination to the data integration pipeline](/contributing/data-integrations/add-new-destination)
### Streaming pipelines
* [Add a new source or destination to the streaming pipeline](/contributing/backend/streaming/sources-and-destinations)
### IO classes
* [Add a new IO class](/contributing/backend/io/adding-a-class)
## Style guide
### Linter
Install `flake8` in your IDE to lint the Python code.
To run the linter locally, execute this script:
```bash theme={"system"}
./scripts/server/lint.sh
```
## Testing
### Unit tests
Add unit tests for the feature in
[mage\_ai/tests](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/tests) directory.
To run the tests locally, execute this script:
```bash theme={"system"}
./scripts/server/test.sh
```
It is also possible to run unit tests directly in a live docker instance, as given in the following steps.
1. Find out the backend server container name, with the command `docker container ls` in a terminal:
```bash theme={"system"}
% docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8dbcfbe41755 mage/data "./scripts/install_aโฆ" 5 days ago Up 38 minutes 0.0.0.0:3000->3000/tcp mage-ai-app-1
b9d811e1e3e8 mage/data "python mage_ai/servโฆ" 5 days ago Up 38 minutes 0.0.0.0:6789->6789/tcp mage-ai-server-1
```
2. Start an interactive `bash` session with the backend server container:
```bash theme={"system"}
% docker exec -it mage-ai-server-1 /bin/bash
```
3. Run unit tests with the following command:
```bash theme={"system"}
root@b9d811e1e3e8:/home/src# python3 -m pytest mage_ai -x
```
## Debugging
# Contributing
Source: https://docs.mage.ai/contributing/backend/streaming/sources-and-destinations
This doc talks about how to add a new source or destination to the streaming pipeline.
## Prerequisites
Follow this [doc](/contributing/overview) to set up the development environment for Mage.
## Add a new source
Example PR for adding a source: [https://github.com/mage-ai/mage-ai/pull/1953](https://github.com/mage-ai/mage-ai/pull/1953)
* Add the new source type to [mage\_ai/streaming/constants.py](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/constants.py).
* Add the source file to the folder [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/streaming/sources](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/streaming/sources)
* Define the source class which inherits from the `BaseSource`.
* Define the source config class.
* Implement the methods:
* `init_client`: Initialize the client used to connect to the source.
* `batch_read`: Batch read messages from the source and use `handler` method to process the messages.
* Add the new source to the [SourceFactory](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/sources/source_factory.py).
* Add a template file with example config to the folder: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/data\_preparation/templates/data\_loaders/streaming](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/data_preparation/templates/data_loaders/streaming)
* Add new source type to the frontend code.
* [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L31](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L31)
* [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L3](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L3)
* [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L22](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L22)
* Add unit tests: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/tests/streaming/sources](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/tests/streaming/sources)
* Add doc: [https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/sources](https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/sources)
***
## Add a new destination (sink)
Example PR for adding a sink: [https://github.com/mage-ai/mage-ai/pull/2121](https://github.com/mage-ai/mage-ai/pull/2121)
* Add the new sink type to [mage\_ai/streaming/constants.py](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/constants.py).
* Add the sink file to the folder [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/streaming/sinks](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/streaming/sinks)
* Define the sink class which inherits from the `BaseSink`.
* Define the sink config class.
* Implement the methods:
* `init_client`: Initialize the client used to connect to the destination.
* `batch_write`: Batch write messages to destination.
* Add the new sink to the [SinkFactory](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/streaming/sinks/sink_factory.py).
* Add a template file with example config to the folder: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/data\_preparation/templates/data\_exporters/streaming](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/data_preparation/templates/data_exporters/streaming)
* Add new source type to the frontend code.
* [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L37](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/components/PipelineDetail/AddNewBlocks/utils.tsx#L37)
* [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L3](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L3)
* [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/frontend/interfaces/DataSourceType.ts#L22](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/DataSourceType.ts#L22)
* Add unit tests: [https://github.com/mage-ai/mage-ai/tree/master/mage\_ai/tests/streaming/sinks](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/tests/streaming/sinks)
* Add doc: [https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/destinations](https://github.com/mage-ai/mage-ai/tree/master/docs/guides/streaming/destinations)
# Testing
Source: https://docs.mage.ai/contributing/backend/testing/overview
Making Mage so invincible that nothing can break it, not even itself.
## Errors
### SQL Alchemy migration error
Delete the `test.db` file located in the directory where you are running the
`python3 -m pytest ... -x` command.
Example:
```
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: index ix_pipeline_run_execution_date already exists
...
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) index ix_pipeline_run_execution_date already exists
[SQL: CREATE INDEX ix_pipeline_run_execution_date ON pipeline_run (execution_date)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
----------------------------------------------------------------------
Ran 0 tests in 0.038s
FAILED (errors=1)
```
# How to add a chart
Source: https://docs.mage.ai/contributing/charts/how-to-add
## 1. Add a new chart type
Add a new type in [`mage_ai/data_preparation/models/widget/constants.py`](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/models/widget/constants.py):
```python theme={"system"}
class ChartType(str, Enum):
# ...
PIE_CHART = 'pie_chart'
```
Add a new type in [`mage_ai/frontend/interfaces/ChartBlockType.ts`](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/frontend/interfaces/ChartBlockType.ts):
```javascript theme={"system"}
export enum ChartTypeEnum {
// ...
PIE_CHART = 'pie_chart',
}
export const CHART_TYPES = [
// ...
ChartTypeEnum.PIE_CHART,
];
```
## 2. Define the configuration options for the chart
In the file `mage_ai/frontend/components/ChartBlock/constants.ts`,
add a list of options for the user to configure their chart with:
```javascript theme={"system"}
export const CONFIGURATIONS_BY_CHART_TYPE = {
[ChartTypeEnum.PIE_CHART]: [
{
label: () => 'variable name of values',
monospace: true,
uuid: 'super_cool_uuid',
},
],
};
```
Also, add the new variable name to the following constant in the
file `mage_ai/frontend/interfaces/ChartBlockType.ts`,:
```javascript theme={"system"}
export const VARIABLE_NAMES = [
'super_cool_uuid',
];
```
If 1 or more of the configuration options is for a variable name that can be defined in the
chart blockโs code and referenced at presentation time, then you must add the following
in the file `mage_ai/data_preparation/models/widget/constants.py`:
```python theme={"system"}
VARIABLE_NAMES_BY_CHART_TYPE = {
ChartType.PIE_CHART: [
'super_cool_uuid',
],
}
```
### Add default settings to the configuration
When a user selects a chart type, you can set default values for these options.
In the file `mage_ai/frontend/interfaces/ChartBlockType.ts`, add the following:
```javascript theme={"system"}
export const DEFAULT_SETTINGS_BY_CHART_TYPE = {
[ChartTypeEnum.PIE_CHART]: {
configuration: () => ({
[VARIABLE_NAME_X]: 'x',
}),
content: ({
upstream_blocks: upstreamBlocks = [],
}: BlockType) => {
const uuid = upstreamBlocks[0];
return `x = ${uuid}[${uuid}.columns[0]]`;
},
},
};
```
### Add helpful information about the variable type that the chart expects
Each chart requires 1 or more input values.
The chart knows how to access those input values by referencing a variable name that the user
inputs into the configuration options.
Each variable can be of a certain type; array/list of strings, integers, etc.
To help the user know how to set the variables properly,
add the following information into the file `mage_ai/frontend/components/ChartBlock/constants.ts`:
```javascript theme={"system"}
export const VARIABLE_INFO_BY_CHART_TYPE = {
[ChartTypeEnum.PIE_CHART]: {
[VARIABLE_NAME_X]: (): string => 'must be a list of booleans, dates, integers, floats, or strings.',
},
};
```
## 3. Define post processing logic for chart data
In the file `mage_ai/data_preparation/models/widget/__init__.py`,
add additional logic for parsing the input values from the
variables defined in the chart blockโs code.
The output of this method is provided to the front-end React components for rendering.
```python theme={"system"}
class Widget(Block):
def post_process_variables(self, variables):
if ChartType.PIE_CHART == self.chart_type:
variables = {}
return variables
```
## 4. Create React component for rendering chart
In the file `mage_ai/frontend/components/ChartBlock/ChartController.tsx`,
import your React component for your new chart and render it.
For example:
```javascript theme={"system"}
function ChartController({
block,
data,
width,
}: ChartControllerProps) {
const {
configuration,
} = block;
const chartType = configuration?.chart_type;
if (ChartTypeEnum.PIE_CHART === chartType) {
return ;
}
}
```
# Adapt an existing source
Source: https://docs.mage.ai/contributing/data-integrations/adapt-existing-source
Mage builds data integrations from [Singer taps](https://www.singer.io/#taps), so if there's a Singer source you want to use that isn't supported yet, you can adapt it yourself!
## Getting started
Mage sources live in the `/mage_integrations/mage_integrations` folder. To add a new source, you'll need to create a new folder in the `/sources` subdirectory.
The folder name should be the name of the tap. For existing sources, that starts with pulling in the GitHub repo for the tap. For example, for the [GitHub tap](https://github.com/MeltanoLabs/tap-github), we would:
```bash theme={"system"}
cd mage_integrations/mage_integrations
git clone https://github.com/MeltanoLabs/tap-github.git sources/github
```
The result:
```
.
โโโ LICENSE
โโโ README.md
โโโ config-sample.json
โโโ config.json
โโโ meltano.yml
โโโ poetry.lock
โโโ pyproject.toml
โโโ pytest.ini
โโโ tap-github.sh
โโโ tap_github
โย ย โโโ __init__.py
โย ย โโโ authenticator.py
โย ย โโโ client.py
โย ย โโโ organization_streams.py
โย ย โโโ repository_streams.py
โย ย โโโ schema_objects.py
โย ย โโโ scraping.py
โย ย โโโ streams.py
โย ย โโโ tap.py
โย ย โโโ tests
โย ย โโโ user_streams.py
โย ย โโโ utils
โโโ tox.ini
```
## Configuring the source
We now have a folder with our tap and a bunch of miscellaneous files. Our first step is to clear out everything we *don't* need. We can delete all files/folders other than `tap_your_source` and `tests`, but hold onto anything that looks like a `config.json` file or a sample config.
Next, we need to create a configuration file and an init file:
```bash theme={"system"}
mkdir sources/github/templates \
&& touch sources/github/templates/config.json \
&& touch sources/github/__init__.py
```
We can now populate our `config.json` file with the sample from the tap. In the GitHub example, this is `config.sample.json`. It looks like this:
```json theme={"system"}
{
"access_token": "abcdefghijklmnopqrstuvwxyz1234567890ABCD",
"repository": "singer-io/target-stitch",
"start_date": "2021-01-01T00:00:00Z",
"request_timeout": 300,
"base_url": "https://api.github.com"
}
```
These are the parameters we'll need to configure our tap. After cleanup, our tree looks like:
```
.
โโโ __init__.py
โโโ tap_github
โย ย โโโ __init__.py
โย ย โโโ authenticator.py
โย ย โโโ client.py
โย ย โโโ organization_streams.py
โย ย โโโ repository_streams.py
โย ย โโโ schema_objects.py
โย ย โโโ scraping.py
โย ย โโโ streams.py
โย ย โโโ tap.py
โย ย โโโ tests
โย ย โโโ user_streams.py
โย ย โโโ utils
โโโ templates
โโโ config.json
```
Now, we need to override some methods in the class to make it work with Mage. This is where things get fun! Copy and paste the following template into `your_tap/__init__.py`:
```python theme={"system"}
from typing import List
from mage_integrations.sources.base import Source, main
from mage_integrations.sources.catalog import Catalog
class YourSource(Source):
def discover(self, streams: List[str] = None) -> Catalog:
pass
def sync(self, catalog: Catalog) -> None:
pass
def test_connection(self) -> None:
client = YourSource(YOUR_CONFIG)
client.close()
if __name__ == "__main__":
main(YourSource, schemas_folder="tap_your_tap/schemas")
```
Replace YourSource with the name of your source, e.g. `Github` for Github, and `tap_your_tap` with the name of your tap, e.g. `tap_github`.
### Overwrite tap methods
Our goal is to overwrite the `discover` and `sync` methods to make them work with Mage. Discover is likely in `tap_your_tap/discover.py` and sync is likely in `tap_your_tap/sync.py`. It requires a bit of nuance, but our job is to look through the existing Singer tap, understand how the `sync` and `discover` methods were implemented, then execute them in Mage `__init__.py` under the corresponding method. This is necessarily different for every tap, since each project is entirely independent.
Don't forget to use absolute imports for these files. [HubSpot](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/hubspot) is another good example of a completed tap.
### Add test connection method
### Overwrite `write_schema` method
Finally, we need to overwrite the `write_schema` method in the `tap_your_tap/sync.py` file. Add the following import:
`from mage_integrations.sources.messages import write_schema`
Now, switch out the default function (likely `singer.write_schema`) for the mage function using the default arguments. Here's an example:
```python theme={"system"}
write_schema(
stream_name=catalog['tap_stream_id'],
schema=schema,
key_properties=catalog.get('key_properties', ['id']),
bookmark_properties=catalog.get('bookmark_properties'),
disable_column_type_check=catalog.get('disable_column_type_check'),
partition_keys=catalog.get('partition_keys'),
replication_method=catalog.get('replication_method'),
stream_alias=catalog.get('stream_alias'),
unique_conflict_method=catalog.get('unique_conflict_method'),
unique_constraints=catalog.get('unique_constraints'),
)
```
### Add logging
To add verbose logging to Mage, add a logger argument each tap file, `tap_github/sync.py`, `tap_github/discover.py`:
```python theme={"system"}
def discover(client, logger=None):
"""
Run the discovery mode, prepare the catalog file and return catalog.
"""
mage_logger = logger or LOGGER
```
Then replace usage of `LOGGER` with `mage_logger`.
```python theme={"system"}
for stream_name, schema_dict in schemas.items():
try:
schema = Schema.from_dict(schema_dict)
mdata = field_metadata[stream_name]
except Exception as err:
# These were LOGGER previously
mage_logger.error(err)
mage_logger.error('stream_name: %s', stream_name)
mage_logger.error('type schema_dict: %s', type(schema_dict))
raise err
```
This will enable Mage to better log Singer taps! ๐ฅณ
## Add source to the Mage UI
Add the new source to the `SOURCES` list constant in [this](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_integrations/sources/constants.py) file
This will make your new source visible in the Mage UI.
## Test your source
Sources should be tested both in the console and via the UI. Follow the directions [here](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/README.md#testing-sync-locally) to configure a lightweight test of your sources taps via a bash terminal and the Mage UI in the dev docker instance. This entails:
* Setting up a sample source and connection locally.
* Running the tap directly, via Python.
* Writing to an intermediate file to inspect the output.
* Performing an end-to-end sync.
## Populate `README.md`
Finally, populate a README file in the source folder, e.g. `/sources/github/README.md`. This should include:
* The name/logo of the tap.
* A configuration section.
* The method for accessing/creating an API key to use with the folder.
* See [HubSpot](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/hubspot) or [GitHub](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/github) for examples.
# Create a new destination
Source: https://docs.mage.ai/contributing/data-integrations/add-new-destination
Here is an example PR for adding a new destination: [https://github.com/mage-ai/mage-ai/pull/2277](https://github.com/mage-ai/mage-ai/pull/2277)
In this guide, we'll build a sample destination called `SampleDest`. When you're building your own destination, you can swap out the `SampleDest` name for the real name of your new destination.
1. Create a directory for the new destination
2. Define custom destination class
3. Add main function
4. Add your destination to the UI (optional)
5. Test your destination
## 1. Create a directory for the new destination
In the `mage_integrations/destinations/` directory, add a new directory named after your destination.
Use snake case and lowercase for your directory name: `mage_integrations/destinations/sample_dest/`.
In this new directory, create the following subdirectories and files:
* `mage_integrations/destinations/sample_dest/templates/config.json`
* `mage_integrations/destinations/sample_dest/__init__.py`
* `mage_integrations/destinations/sample_dest/README.md`
The directory structure should look like this:
```
mage_integrations/
| destinations/
| | sample_dest/
| | | templates/
| | | | config.json
| | | __init__.py
| | | README.md
```
### Templates folder
This folder contains a sample configuration JSON file thatโs
displayed to the user when they are setting up a synchronization using this destination.
The `config.json` file contains keys and values that are used to configure the
behavior of the destination as well as credentials to authenticate requests to the destination.
#### Naming convention
You must use the exact filename `config.json`.
#### Examples
```json theme={"system"}
{
"api_key": "",
"secret_key": ""
}
```
### `__init__.py`
This is where majority of the destination logic will exist.
#### Examples
[`mage_integrations/destinations/amazon_s3/__init__.py`](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/amazon_s3/__init__.py)
### `README.md`
Document how to configure and use your destination in the `README.md` file.
## 2. Define custom destination class
In the `mage_integrations/destinations/sample_dest/__init__.py`,
create a new class named after your destination and subclass the
[base destination class](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/base.py).
If you're adding a destination for a SQL data warehouse or database, you can subclass the
[base sql destination class](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/sql/base.py)
```python theme={"system"}
from mage_integrations.destinations.base import Destination
class SampleDest(Destination):
pass
```
### Override the `export_batch_data` method
The base `Destination` class has an instance method called `export_batch_data`. Here is the interface:
```python theme={"system"}
def export_batch_data(self, record_data: List[Dict], stream: str) -> None:
...
```
Override this method to contain the logic for exporting data that is specific to your destination.
## 3. Add main function
In the file `mage_integrations/destinations/sample_dest/__init__.py`, add the main function to the bottom of the file
like below:
```python theme={"system"}
if __name__ == '__main__':
destination = SampleDest(
argument_parser=argparse.ArgumentParser(),
batch_processing=True,
)
destination.process(sys.stdin.buffer)
```
## 4. Add your destination to the UI (optional)
Add the new destination to the `DESTINATIONS` list constant in this file: [https://github.com/mage-ai/mage-ai/blob/master/mage\_ai/data\_integrations/destinations/constants.py](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_integrations/destinations/constants.py)
## 5. Test your destination
Follow this [doc](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations) to test your new destination.
To test the destination in the UI, you can install your updated `mage_integrations` module by running the following commands in Mage terminal:
```bash theme={"system"}
pip uninstall -y mage_integrations
pip install "git+https://github.com/your_repo.git@your_branch#egg=mage-integrations&subdirectory=mage_integrations"
```
# Create a new source
Source: https://docs.mage.ai/contributing/data-integrations/add-new-source
This guide details adding a new source to Mage. If your source already exists as a [Singer tap](https://www.singer.io/#taps), check out [our guide](/contributing/data-integrations/adapt-existing-source) for adapting an existing source instead.
Mage Sources are built on Singer Taps. You can read more about the Singer spec [here](https://github.com/singer-io/getting-started). In this guide, we'll copy over our sample source to get started and build from there.
If you haven't already, start by cloning the Mage repo and navigating to the folder:
```bash theme={"system"}
git clone https://github.com/mage-ai/mage-ai.git
cd mage-ai
```
For our source, we'll need:
* A schema that represents the source data
* A template configuration file
* An `__init__.py` file that contains the logic for fetching data from the source.
* A `test_connection` method that tests the connection to our source.
* A `discover` method that returns a [Catalog](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/catalog.py) of streams.
* A `load_data` method that yields data from the source as a dictionary.
We'll start from a template to make things easy.
## Copy the template source
Navigate to `/mage-ai/mage_integrations/mage_integrations/sources` and create a copy of the `titanic` directory:
```bash theme={"system"}
cd ./mage_integrations/mage_integrations/sources
cp -R titanic MY_SOURCE
```
Our titanic source simply reads data from a CSV file, but we'll modify it to do much more. You'll notice the following files and folders in the `MY_SOURCE` directory:
```
mage_integrations/sources/MY_SOURCE
โโโ README.md
โโโ __init__.py
โโโ schemas
โ โโโ passengers.json
โโโ templates
โโโ config.json
```
So we'll need to update or overwrite each of these.
## Update schemas
This folder contains all the known schemas from your source. For sources that have dynamic schemas (e.g. database tables from MySQL), this folder may be empty since the schema is dependent on the individual's source data.
The JSON format of these schema files follows the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#schemas).
When naming schemas, use the plural name of the object you're referencing. This plural name will be displayed to the individual who is setting up a synchronization using this source. Take the [sample "passengers" schema](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/titanic/schemas/passengers.json) for example:
```json theme={"system"}
{
"properties": {
"Survived": {
"type": ["null", "integer"]
},
"Name": {
"type": ["null", "string"]
}
},
"type": ["null", "object"]
}
```
## Add templates
This folder contains a sample configuration JSON file that's displayed to the user when they are setting up a synchronization using this source.
The `config.json` file contains keys and values that are used to configure the behavior of the source as well as credentials to authenticate requests to the source. You must use the exact filename `config.json`, regardless of your source's name.
The following simple example is present in the [Titanic source](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/titanic/templates/config.json).
```json theme={"system"}
{
"api_key": "",
"secret_key": ""
}
```
## Update `__init__.py`
A majority of source logic lives in the `__init__.py` file. Most of the work in adding your own source will involve creating/overwriting methods from `__init__.py`. [Our sample](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/titanic/__init__.py) contains the following:
```python theme={"system"}
from mage_integrations.sources.base import Source, main
from typing import Dict, Generator, List
import requests
URL = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'
class Titanic(Source):
def load_data(
self,
**kwargs,
) -> Generator[List[Dict], None, None]:
text = requests.get(URL).text
rows = []
lines = text.rstrip().split('\n')
columns = lines[0].split(',')
for line in lines[1:]:
values = line.split(',')
rows.append({col: values[idx] for idx, col in enumerate(columns)})
yield rows
def test_connection(self):
request = requests.get(URL)
if request.status_code != 200:
raise Exception('Could not fetch titanic data')
if __name__ == '__main__':
main(Titanic)
```
### Rename source class
First, rename the source class to match your new source
```python theme={"system"}
from mage_integrations.sources.base import Source
class MY_SOURCE(Source):
pass
```
### Edit methods
#### The `load_data()` method
Override this method to contain the logic for fetching data specific to your source. The `Titanic` source's `load_data` method reads data from a CSV file and yields a list of dictionaries:
```python theme={"system"}
def load_data(
self,
**kwargs,
) -> Generator[List[Dict], None, None]:
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'
text = requests.get(url).text
rows = []
lines = text.rstrip().split('\n')
columns = lines[0].split(',')
for line in lines[1:]:
values = line.split(',')
rows.append({col: values[idx] for idx, col in enumerate(columns)})
yield rows
```
Your `load_data` method should *also* yield a list of dictionaries. There is a keyword argument named `query` in the `load_data` method that is a dictionary. When Mage runs a source, the following keys and values are automatically available on each run:
| Key | Description | Sample value |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| `_execution_date` | The date and time (in ISO format) of when the pipeline started running. | `2022-10-21T17:24:49.443559` |
| `_execution_partition` | An automatically formatted partition of the pipeline run using the execution date. | `20221021T172557` (e.g. format `%Y%m%dT%H%M%S`) |
| `_start_date` | You can define this variable as a [runtime variable](/getting-started/runtime-variable) in your pipeline or it'll be automatically filled in using the date and time your pipeline runs minus 1 hour, day, week, etc (based on your schedule's interval). | `2022-10-01T00:00:00.000000` |
| `_end_date` | You can define this variable as a [runtime variable](/getting-started/runtime-variable) in your pipeline or it'll be automatically filled in using the date and time your pipeline runs. | `2022-10-02T00:00:00.000000` |
#### The `discover()` method
The discover method should return a [Catalog](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/catalog.py) of streams that define the data in your source. For example discover methods, see our [Google Sheets](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/google_sheets/__init__.py) source or our [DynamoDB source](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/dynamodb/__init__.py).
#### The `test_connection()` method
### Update `main()` function
Change the source name in the `main()` function to match your new source:
```python theme={"system"}
if __name__ == '__main__':
main(MY_SOURCE)
```
## Add your source to the UI
The list of sources is available [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_integrations/sources/constants.py), add yours to make it accessible via the UI.
## Document your source
Document how to configure and use your source in the `README.md` file at the source's root directory. Documentation helps Magers and future contributor understand exactly how your source works! Be sure to be thorough and descriptive. [Here's an example](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/salesforce/README.md) of documentation done well.
## Test your source
You'll first need to configure your [development environment](https://docs.mage.ai/contributing/development-environment). Once complete, follow this [doc](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations) to test your new source.
# Configure your development environment
Source: https://docs.mage.ai/contributing/development-environment
Want to get started contributing code? You're in the right place! Read on to learn how to set up your development environment.
## Getting started
Configuring a development environment can be *hard*, but at Mage we're trying to make things as simple and straightforward as possible.
To get started, you'll need Docker installed on your machine. You can read more about Docker [here](https://www.docker.com/).
## Environment setup
Next, you'll need to configure your development environment. This should be as simple as running a few commands. You can watch the video below or skip to the numbered instructions.
Configure your coding environment for local development: `pyenv`, pre-commit hooks, etc. Follow the instructions [here](https://github.com/mage-ai/mage-ai/blob/master/README_dev.md).
```bash theme={"system"}
./scripts/init.sh [project_name]
```
```bash theme={"system"}
./scripts/dev.sh [project_name]
```
If you only work on backend, use the command below for better performance
```bash theme={"system"}
./scripts/start.sh [project_name]
```
When developing frontend, use port 3000. If you only need to develop backend, you can use port 6789. Find these ports at [http://localhost:3000](http://localhost:3000) & [http://localhost:6789](http://localhost:6789).
## Push code changes
Once your environment is configured, you can make edits and see changes in real time! Now you're ready to contribute to the Mage repo. This is very similar to contributing to other open source projects on GitHub.
If you *don't* have any experience contributing code and this is confusing, feel free to reach out in Slack and we'll help you get started.
Otherwise, follow the steps below to push your code changes to the Mage repo.
## Troubleshoot
### Permission denied
If you get an error relating to permission denied when trying to execute a script,
it might have something to do with SELinux.
Two or more containers are trying to mount the same volume. With SELinux itโs not possible without
the `z` option.
For more information about using the `z` option,
read [Dockerโs documentation](https://docs.docker.com/storage/bind-mounts/).
### No such file or directory for "install\_and\_run.sh" file
If you are using a Windows machine and run into the following error while executing the
`docker-compose.yml` file: `exec ./scripts/install_and_run.sh: no such file or directory`,
it may be due to a difference in line-endings used on Windows versus Macs.
You can check the `mage_ai/frontend/scripts/install_and_run.sh` file in VSCode to see if
it is using CRLFs (carriage return line feeds), and if it is, change it to LFs (line feeds)
and save the file. Then try running the `docker-compose` command again.
### ModuleNotFoundError: No module named 'x'
If there were added new libraries you should manually handle new dependencies. It can be done in 2 ways:
1. `docker-compose build` from project root will fully rebuild an image with new dependencies - it can take lots of time
2. `pip install x` from inside the container will only install the required dependency - it should be much faster
# Documentation
Source: https://docs.mage.ai/contributing/documentation/overview
Like every other part of Mage, our documentation is open-source. Read on to learn how to make edits or even write entirely new docs for Mage!
## Overview
Mage uses [Mintlify](https://www.mintlify.com/) as our documentation framework. Mintlify provides hosting and deployment for our docs, which makes editing quite simple. In this guide, we'll discuss different "levels" of contributing to our docs, starting from simplest to most advanced.
All of our docs are written in plain text as Markdown files. If you're not familiar with Markdown, check out [this guide](https://www.markdownguide.org/basic-syntax/) to get started. You can take a look at the structure and contents of our docs by navigating this site and checking out the [docs directory](https://github.com/mage-ai/mage-ai) on GitHub.
## Level 1: Suggesting Edits ๐ฆธ๐ปโโ๏ธ
The very simplest way to contribute to the Mage docs is directly in the browser, using GitHub's file editing feature. This is a great way to suggest small edits to existing docs, but it's not the best way to add entirely new docs or complex formatting, images, etc.
To make edits, simply click on the "Suggest Edit" button at the bottom of every page. This will open a new tab with the file you're editing. Make your changes, then click "Commit changes." This will open a new pull request with your changes. Be sure to add appropriate details to your pull request & tag `@wangxiaoyou1993`. Once the pull request is merged, your changes will be live on the site!
## Level 2: Add complex components ๐ช
### Images
To add images to the Mage docs, you'll need to contact a team member or host the images in GitHub. Our assets currently live in a private repo to keep the docs repo lightweight.
Reach out in [Slack](https://www.mage.ai/chat) if you're editing documentation that requires additional assets and we'll get them hosted for you.
When adding images, we use centered divs or the Mintlify frame component. You can see the code in this page for examples of both.
### Mintlify elements
To learn more about Mintlify-specific elements, visit the Mintlify docs [here](https://mintlify.com/docs/content/page). You can add things like accordion groups, interactive lists, html elements, and more. Anything you can do in Mintlify, you can do in our docs.
## Level 3: Add entirely new docs ๐ช
### Adding a new page
Each page is an MDX file that should begin with the following metadata syntax with `---` at the start and end. Here are basics to get you started:
1. [Page basics](https://mintlify.com/docs/content/page)
2. [Headers and text](https://mintlify.com/docs/content/text)
3. [Images and embeds](https://mintlify.com/docs/content/image-embeds)
Once you add a new page in the appropriate folder in our `docs` directory, you'll need to add it to our site navigation. That is accomplished through the `mint.json` file in the root of our `docs` directory.
You'll need to add a new entry to the `pages` array in that file. The `path` should be the relative path to your new page, and the `title` should be the title of your page. Paths in `mint.json` should *not* include file extensions.
The best way to understand the structure is to read about it in the [Mintlify docs](https://mintlify.com/docs/settings/global) and to take a look at our `mint.json` file [here](https://github.com/mage-ai/mage-ai/blob/master/docs/mint.json).
### Developing locally in Mintlify
If you *really* want to level up your docs contributions, you can create a local development environment. This will allow you to see changes *live* as you make them on your machine.
Follow the instructions from Mintlify [here](https://mintlify.com/docs/quickstart#2-updating-your-docs) to get started. Once configured, you'll be able to run a local server that will allow you to see your changes in real-time.
Simply `cd mage-ai/docs` + `mintlify dev` and you'll be able to navigate to a live version at `localhost:3000`.
### Adding icons
Some elements in Mintlify have icons. Those are sourced from [fontawesome](https://fontawesome.com/). Any icon you find there can be added as an icon to the Mage docs, where appropriate. We only add icons to top-level items in our sidebar.
## Level 4: Adding entirely new sections ๐ง๐ปโโ๏ธ
If you'd like to contribute a larger, more structured section of our documentation, please reach out to a member of our team via Slack. They'll be able to guide you through the process of making sweeping changes.
# Contributing to the front-end client
Source: https://docs.mage.ai/contributing/frontend/overview
Guides on adding features to the front-end client.
| | |
| -------------- | ------------------- |
| Language | TypeScript |
| Framework | React, Next.js |
| Code directory | `mage_ai/frontend/` |
## Style guide
Mage follows the [Airbnb JavaScript style guide](https://airbnb.io/javascript/react/).
### Linter
To run the linter locally, execute this script:
```bash theme={"system"}
./scripts/test.sh
```
## Setup
```bash theme={"system"}
yarn global add eslint-config-next
yarn global add eslint_d
```
# Frontend testing
Source: https://docs.mage.ai/contributing/frontend/testing
Making Mage invincible so that nothing can break it, not even itself.
## Prerequisites
* It is not recommended to run tests on your working project, since failed tests can leave behind
empty pipelines, triggers, etc. Set up your
[development environment](https://docs.mage.ai/contributing/development-environment).
* Make sure [user authentication](https://docs.mage.ai/production/authentication/overview) is enabled.
* Make sure your development server is running.
## Configuration
* Tests default to "fail" if a client error occurs. Add the following to the top of the test file to disable this check.
```ts theme={"system"}
test.use({ failOnClientError: false });
```
* Certain setting features are enabled by default. Refer to `base.ts` for more information, and use the following snippet when needed.
```ts theme={"system"}
test.use({ settingFeaturesToDisable: { [FeatureUUIDEnum.LOCAL_TIMEZONE]: true } });
```
## Running the tests
You have the option to just run one specific test or all tests at once. There are
[several ways](https://playwright.dev/docs/running-tests) to run specific tests.
```bash theme={"system"}
# Run everything, headless by default.
yarn test
# Run everything with a browser.
yarn test:h
# Run test files that have the word "pipelines" in their names.
yarn test pipelines
```
You can run the frontend tests using the CI/CD Playwright configuration with the following command:
```bash theme={"system"}
yarn test:ci
```
## Creating a test
Refer to the [Playwright docs](https://playwright.dev/docs/writing-tests) for examples on writing frontend tests.
## Debugging
The easiest way to debug is to run tests in UI mode so you can see what went wrong.
```bash theme={"system"}
yarn test --ui
# Or
yarn test:ci --ui
```
If you are running into issues with the app and would like to report it, you can tell us about it either on [GitHub](https://github.com/mage-ai/mage-ai/issues/new/choose) or [Slack](https://www.mage.ai/chat).
# Contributing
Source: https://docs.mage.ai/contributing/overview
We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!
## ๐ฅพ Choose your adventure
There *so* many ways to contribute to Mage! We welcome all contributions, no matter how small.
Start your contribution journey below, whether you're an *apprentice* or a *master of spells*. Once you feel confident in your path, [read on](/contributing/development-environment) to learn how to get started.
Thanks for leading the charge! The easiest way to contribute is to report bugs or suggest new features.
### Report bugs
Head on over to our [Slack](https://mage.ai/chat) and let us know what's up. We recommend the `#bugs-troubleshoot-questions` channel.
### Squash bugs
We're always trying to make our project better, but we need your help! Take a look at our [GitHub issues](https://github.com/mage-ai/mage-ai/issues) for small to medium complexity projects and bugs that need fixing.
### Suggest new features
We *love* your suggestions. First take a look at our [roadmap](/about/roadmap) to see if your idea is already planned. You might also consider checking out open issues on [GitHub](https://github.com/mage-ai/mage-ai).
Otherwise, drop a line in our [Slack](https://mage.ai/chat) and let us know what you're thinking. We recommend the `#feature-requests` channel.
### Write documentation
We have *a lot* of docs. We really hope they're fun to read, but we know they can be improved.
The great news is that they're fun to write, too! The *easiest* way to contribute is by scrolling to the bottom of the page you'd like to edit (maybe this one) and clicking the "Edit this page" link. You'll be taken to GitHub where you can make your changes and submit a pull request.
If you like working in code, that's great too! All of our docs are written in markdown, so you can just pull Mage from [GitHub](https://github.com/mage-ai/mage-ai) and start writing.
### Community Support
If you're already an adept Mager, we need your help! We're looking for community members to help answer questions in our [Slack](https://mage.ai/chat). Check out the `#bugs-troubleshoot-questions`, `#general`, and `#feature-requests` channels to get started. Anything helps!
If you want to take it up a notch, check out our GitHub issues! We have a lot of [open issues](https://github.com/mage-ai/mage-ai/issues) that need your help.
### Backend development
We're always looking for help with the Mage backendโ that might look like refactoring code, adding new features, or fixing bugs.
[Here's](https://github.com/mage-ai/mage-ai/pull/3497) an example of an *amazing* backend PR that drastically improved dbt functionality in Mage.
Check out our [roadmap](https://airtable.com/shrJctaGt4tvU27uF/tbl5bIlFkk2KhCpL4) for small to large complexity projects that need your help!
### Data integration
Mage supports [data integration](/data-integrations/overview) pipelines via the Singer spec. We're always looking for help adding/updating our sources and destinations.
Here's an example of a [great community PR](https://github.com/mage-ai/mage-ai/pull/2277) & here's our [docs on adding a source](/contributing/data-integrations/adapt-existing-source).
### IO operations (SQL block) and templates
Want to help build *awesome* tools *inside* the pipeline builder page? Help out with our IO operations by adding new SQL blocks! Here's a PR that implements a [DuckDB IO Class](https://github.com/mage-ai/mage-ai/pull/3463).
The simplest way you can help is by adding data loader/exporter templates. Here's an example PR that adds a few for [DuckDB](https://github.com/mage-ai/mage-ai/pull/3553).
### Frontend development
Want to help build the *future* of the Mage UI? Help out with our frontend development! We're always looking for help with UI/UX, accessibility, and more. Checkout our [frontend](/contributing/frontend/overview) page for more info.
Got questions? Live chat with us in [Slack](https://www.mage.ai/chat)
Anything you contribute, the Mage team and community will maintain. Weโre in it
together!
# Styling
Source: https://docs.mage.ai/contributing/styling
```
eslint_d --fix --ext .js,.jsx,.ts,.tsx
git status --porcelain | grep '^[AM?]' | cut -c4- | grep '\.py$' | xargs black
git diff --name-status master | grep '^[AM?]' | awk '{$1=""; sub("^[ \t]+", ""); print}' | grep '\.\(js\|jsx\|ts\|tsx\)$' | xargs eslint_d --config mage_ai/frontend/.eslintrc.js --fix --ext .js,.jsx,.ts,.tsx
```
# Using data integrations in batch pipelines (in Beta)
Source: https://docs.mage.ai/data-integrations/batch-pipelines
Load data from data integration sources and export to data integration destinations as blocks inside batch pipelines.
Available in version `0.9.35` and greater.
## Overview
You can use a [data integration source](/data-integrations/sources/overview) (e.g. Stripe)
and/or a [data integration destination](/data-integrations/destinations/overview) (e.g. Trino)
as a block in batch pipelines.
### Why is this important?
Combine the low-code aspect and capabilities of data integrations with the flexibility and
dynamic capabilities of a batch pipeline.
### Example
1. Use a Python data loader block to write custom code for fetching Stripe credentials.
2. Use a SQL data loader block to fetch data from a table in PostgreSQL.
3. Add a data integration source block to fetch data from Stripe.
4. Use the output from the Python data loader block and interpolate the Stripe credentials into the data integration source block.
5. Add a Python transformer block to merge, clean, and aggregate data fetched from PostgreSQL and Stripe.
6. Add a data integration destination block that writes the output data from the Python transformer block and export it to SalesForce.
The image below doesnโt reflect the above example exactly.
The image shows a different set of blocks than the example above.
***
## Features
* Use any data integration source.
* Use any data integration destination.
* Sources and destinations have capabilities of a normal Python block in a batch pipeline.
* Dynamically calculate the source and destination credentials at runtime.
* Dynamically build the source and destination catalog schema at runtime.
* Source blocks can output its fetched data to any downstream block.
* Any block can output its data to a destination block and have it exported to that destination.
* Support incremental sync
* `[Coming soon]` *Support log based sync*.
* `[Coming soon]` *Replicate source and destination blocks*.
* `[Coming soon]` *Support source and destination blocks as dynamic blocks*.
***
## How to use
1. Go to the [project preferences](http://localhost:6789/settings/workspace/preferences) and
enable the feature labeled `Data integration in batch pipeline`.
This feature requires the `Add new block v2` feature to be enabled.
2. Create a new batch pipeline.
3. Add a new Data loader block by selecting:
1. Templates โ
2. Data loader โ
3. Sources โ
4. Select the source to add to the pipeline.
4. Configure the source:
1. Add credentials.
2. Select 1 or more streams to sync.
3. Setup stream settings.
4. Select 1 or more columns to sync from stream.
5. Add a new Data exporter block by selecting:
1. Templates โ
2. Data exporter โ
3. Destinations โ
4. Select the source to add to the pipeline.
6. Configure the destination:
1. Add credentials.
2. Select 1 or more stre.ams to export.
3. Setup stream settings.
4. Select 1 or more columns to include in the destination table when exporting stream.
***
## Configure source and destination
For more information on how to configure and setup a data integration source and destination,
refer back to the original [data integration guide](/guides/data-integration-pipeline).
### Credentials
### Upstream block settings
### Select streams
### Streams overview with bulk editing
### Stream detail overview
For more information on how to configure settings for a stream,
refer back to the original [data integration guide](/guides/data-integration-pipeline).
### Stream detail schema properties with bulk editing
For more information on how to configure schema properties for a stream,
refer back to the original [data integration guide](/guides/data-integration-pipeline).
### Stream detail sample data
***
## Python blocks
When adding a data integration source or destination block to a batch pipeline,
you can choose the language to be YAML or Python (default option is YAML).
If you choose Python as the language, you programmatically define the following settings of the
data integration:
| Configuration | Example | Decorator | Description |
| ---------------- | ---------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| Source | `stripe` | `@data_integration_source` | The source to load data from. |
| Destination | `salesforce` | `@data_integration_destination` | The destination to export data to. |
| Config | `database='...'` | `@data_integration_config` | A dictionary containing the credentials and other settings specific to the source or destination. |
| Selected streams | `['account']` | `@data_integration_selected_streams` | A list of stream names to be selected from the catalog. |
| Catalog | `{...}` | `@data_integration_catalog` | A dictionary containing a list of streams and the streamโs settings, schema properties, and metadata. |
### `@data_integration_source`
This decorated function is required.
```python theme={"system"}
@data_integration_source
def source(*args, **kwargs) -> str:
return 'postgresql'
```
### `@data_integration_destination`
This decorated function is required.
```python theme={"system"}
@data_integration_destination
def destination(*args, **kwargs) -> str:
return 'postgresql'
```
### `@data_integration_config`
This decorated function is required.
```python theme={"system"}
@data_integration_config
def config(*args, **kwargs) -> Dict:
destination: str = kwargs.get('destination', None)
return {
'database': '',
'host': '',
'password': '',
'port': 5432,
'schema': '',
'table': '',
'username': '',
}
```
#### Keyword arguments inside the decorated function
The following keyword arguments are only available in the decorated functionโs body for sources.
| | |
| ------------------- | ---------------------- |
| Keyword | `source` |
| Sample code | `kwargs['source']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the source |
| Available in | Sources only |
| | |
| ------------------- | --------------------------- |
| Keyword | `destination` |
| Sample code | `kwargs['destination']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the destination |
| Available in | Destinations only |
### `@data_integration_selected_streams`
This decorated function is optional.
```python theme={"system"}
@data_integration_selected_streams(discover_streams: bool = False)
def selected_streams(*args, **kwargs) -> List[str]:
config: Dict = kwargs.get('config', None)
discover_streams_func: Callable = kwargs.get('discover_streams_func', None)
source: str = kwargs.get('source', None)
return []
```
#### Decorator arguments
The following arguments are only available in the decorator `@data_integration_selected_streams`
for sources.
| | |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Argument | `discover_streams` |
| Argument values | `False`, `True` |
| Description | If `True`, there will be a list of stream names available in the functionโs body via the keyword argument `selected_streams`; e.g. `kwargs['selected_streams']`. This list of stream names are retrieved by the sourceโs underlying code in the [Mage Integrations library](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources). |
| Available in | Sources only |
#### Keyword arguments inside the decorated function
The following keyword arguments are only available in the decorated functionโs body.
| | |
| ------------------- | ---------------------- |
| Keyword | `source` |
| Sample code | `kwargs['source']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the source |
| Available in | Sources only |
| | |
| ------------------- | --------------------------- |
| Keyword | `destination` |
| Sample code | `kwargs['destination']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the destination |
| Available in | Destinations only |
| | |
| ------------------- | ------------------------------------------------------------------- |
| Keyword | `config` |
| Sample code | `kwargs['config']` |
| Return value type | `dict` |
| Sample return value | `{ 'database': '...' }` |
| Description | The dictionary containing the credentials and other configurations. |
| Available in | Sources and destinations |
| | |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| Keyword | `selected_streams` |
| Sample code | `kwargs['selected_streams']` |
| Return value type | `List[str]` |
| Sample return value | `['account', 'users']` |
| Description | A list of strings containing all the available stream names for the source. This is only available if `discover_streams` is `True`. |
| Available in | Sources only |
| | |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Keyword | `discover_streams_func` |
| Sample code | `kwargs['discover_streams_func']` |
| Return value type | `Callable` |
| Sample return value | If function is invoked, the return value is the same as `selected_streams` above. |
| Description | A list of stream names are retrieved by the sourceโs underlying code in the [Mage Integrations library](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources). |
| Available in | Sources only |
### `@data_integration_catalog`
The returned dictionary will override the catalog setup through the
user interface.
```python theme={"system"}
@data_integration_catalog(discover: bool = False, select_all: bool = True)
def catalog(*args, **kwargs) -> Dict:
config: Dict = kwargs.get('config', None)
selected_streams: List[str] = kwargs.get('selected_streams', None)
source: str = kwargs.get('source', None)
# catalog_from_discover is None unless discover=True
catalog_from_discover: Dict = kwargs.get('catalog', None)
# Executing this function will fetch and return the catalog
discover_func: Callable = kwargs.get('discover_func', None)
return {
'streams': [],
}
```
#### Decorator arguments
The following arguments are only available in the decorator `@data_integration_catalog`
for sources.
| | |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Argument | `discover` |
| Argument values | `False`, `True` |
| Description | If `True`, the source will fetch the catalog and all stream schema properties then pass the returned value to the decorated functionโs keyword arguments. |
| Available in | Sources only |
| | |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Argument | `select_all` |
| Argument values | `True`, `False` |
| Description | If `False`, the streams in the catalog wonโt be selected by default and youโll have to manually select the stream via the streamโs metadata. |
| Available in | Sources only |
#### Keyword arguments inside the decorated function
The following keyword arguments are only available in the decorated functionโs body.
| | |
| ------------------- | ---------------------- |
| Keyword | `source` |
| Sample code | `kwargs['source']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the source |
| Available in | Sources only |
| | |
| ------------------- | --------------------------- |
| Keyword | `destination` |
| Sample code | `kwargs['destination']` |
| Return value type | `str` |
| Sample return value | `'postgresql'` |
| Description | The name of the destination |
| Available in | Destinations only |
| | |
| ------------------- | ------------------------------------------------------------------- |
| Keyword | `config` |
| Sample code | `kwargs['config']` |
| Return value type | `dict` |
| Sample return value | `{ 'database': '...' }` |
| Description | The dictionary containing the credentials and other configurations. |
| Available in | Sources and destinations |
| | |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Keyword | `selected_streams` |
| Sample code | `kwargs['selected_streams']` |
| Return value type | `List[str]` |
| Sample return value | `['account', 'users']` |
| Description | A list of strings containing the available stream names returned from the decorated function under `@data_integration_selected_streams`. |
| Available in | Sources and destinations |
| | |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Keyword | `catalog` |
| Sample code | `kwargs['catalog']` |
| Return value type | `dict` |
| Sample return value | `{ 'streams': [] }` |
| Description | A dictionary with a key named `streams` that is a list of dictionaries containing the stream schema properties and metadata. This is only available if `discover` is `True`. |
| Available in | Sources only |
| | |
| ------------------- | ------------------------------------------------------------------------------------------- |
| Keyword | `discover_func` |
| Sample code | `kwargs['discover_func']` |
| Return value type | `Callable` |
| Sample return value | If function is invoked, the return value is the same as `catalog` above. |
| Description | If function is invoked, the source will fetch the catalog and all stream schema properties. |
| Available in | Sources only |
***
## Data from upstream blocks
The data integration source and destination block can depend on upstream blocks.
Those upstream blocks can pass its output data into the data integration source and destination,
which then can be used as:
* Input arguments to interpolate information in the credentials
(or other decorated functions if source or destination block is a Python block).
* Data to be exported to a destination.
### Input arguments
Follow these steps to use the output of an upstream block as an input:
1. Open the data integration blockโs configuration and click on the tab
labeled Upstream block settings.
2. Toggle the name of the block you want as an input.
3. Check the box labeled Use the blockโs output data as an input.
Inputs are available as positional arguments for all decorated functions in Python blocks.
#### Other options
If checked, then the upstream blockโs output data is only used as inputs and wonโt be ingested
and exported to the destination. This is only applicable to destination blocks.
If checked, then the upstream blockโs catalog will be included as part of the input argument(s)
for the data integration block.
#### Examples using upstream data output as inputs
If you have an upstream block, named `pg_credentials`, that has the following code:
```python theme={"system"}
@custom
def get_credentials(*args, **kwargs):
return dict(
database='dangerous',
host='host.docker.internal',
password='postgres',
port=5432,
schema='mage',
username='postgres',
)
```
Here are examples on how to use the upstream blockโs output to configure credentials at runtime:
If you have a data integration source or
destination block that depends on the above upstream block,
you can interpolate the output of the upstream block in the source or destination blockโs
credentials like this:
```yaml theme={"system"}
database: |
{{ block_output('pg_credentials', parse='lambda x: x["database"]') }}
host: |
{{ block_output('pg_credentials', parse='lambda x: x["host"]') }}
password: |
{{ block_output('pg_credentials', parse='lambda x: x["password"]') }}
port: |
{{ block_output('pg_credentials', parse='lambda x: x["port"]') }}
schema: |
{{ block_output('pg_credentials', parse='lambda x: x["schema"]') }}
username: |
{{ block_output('pg_credentials', parse='lambda x: x["username"]') }}
```
Depending on how many upstream blocks your source or destination block uses as an input,
itโll determine how many positional arguments are available in any of your decorated functions.
```python theme={"system"}
@data_integration_config
def config(credentials, *args, **kwargs) -> Dict:
return {
'database': credentials['database'],
'host': credentials['host'],
'password': credentials['password'],
'port': credentials['port'],
'schema': credentials['schema'],
'username': credentials['username'],
}
```
#### Examples using upstream blockโs catalog as input
If you have an upstream block, named `titanic_survival`, that has the following code:
```python theme={"system"}
import io
import pandas as pd
import requests
@data_loader
def load_data_from_api(*args, **kwargs):
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/titanic_survival.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
```
And its output data has the following Pandas data types:
| Column | Data type | Sample data |
| ----------- | --------- | ------------------------------------------ |
| PassengerId | `int64` | 2 |
| Survived | `int64` | 1 |
| Pclass | `int64` | 1 |
| Name | `object` | Mrs. John Bradley (Florence Briggs Thayer) |
| Sex | `object` | female |
| Age | `float64` | 38 |
| SibSp | `int64` | 1 |
| Parch | `int64` | 0 |
| Ticket | `object` | PC 17599 |
| Fare | `float64` | 71.2833 |
| Cabin | `object` | C85 |
| Embarked | `object` | C |
Here are examples on how to use the upstream blockโs catalog as an input to the data integrationโs
decorated functions:
Depending on how many upstream blocks your source or destination block uses as an input,
itโll determine how many positional arguments are available in any of your decorated functions.
```python theme={"system"}
@data_integration_catalog
def catalog(titanic_survival_data_and_catalog, *args, **kwargs) -> Dict:
_data, catalog = titanic_survival_data_and_catalog
return catalog
```
The positional argument correlated to the upstream block that youโre using as an input
will be either a single object or a tuple. Itโs a tuple if you checked the box
labeled Use catalog as an input in the data integration blockโs configuration on
the tab labeled Upstream block settings.
In this example, the last item in the tuple from the 1st positional argument (
e.g. `titanic_survival_data_and_catalog`) contains a Python dictionary that represents
the catalog from the upstream block named `titanic_survival`.
This catalog contains a key named `streams` that is a list of dictionaries containing
stream settings, metadata, and schema properties of the output data from the upstream
block `titanic_survival`.
Here is an example of that catalog (a lot of the metadata and schema properties are omitted
from this example so that itโs not too lengthy):
```python theme={"system"}
{
'streams': [
{
'metadata': [
{
'breadcrumb': [
'properties',
'Embarked',
],
'metadata': {
'inclusion': 'available',
'selected': True,
},
},
],
'schema': {
'properties': {
'PassengerId': {
'type': [
'null',
'integer',
],
},
},
'type': 'object',
},
'stream': 'titanic',
'type': 'SCHEMA',
'replication_method': 'FULL_TABLE',
},
],
}
```
### Data to be exported
Outputs from upstream blocks, as data to be exported, can only be used by destination blocks.
The outputs from upstreams blocks will not only be available as an input,
but itโll also be used as a data stream when ingesting and exporting to the destination.
You can control which upstream blockโs data is ingested and exported by configuring the
destination blockโs streams.
***
## Output data from source blocks
Destination blocks donโt output any data to downstream blocks.
A data integration source block will output the data fetched from the source and pass that
data to its downstream blocks.
A downstream data integration destination block will automatically know how to ingest the data
and process it for exporting.
### Shape of output data
All other types of blocks will receive the data integration source blockโs output data either as
a dictionary or as a list containing 2 or more dictionaries.
The output data will be a dictionary if the source block is only fetching 1 stream.
The output data will be a list of dictionaries if the source block is fetching 2 or more streams.
The number of items in the list corresponds to the number of streams that the source block is
configured to sync.
#### Shape of dictionary
Each item in the list is a dictionary with the following keys:
| Key | Value data type | Sample value | Description |
| --------- | ----------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stream` | `dict` | `{ 'stream': 'accounts' }` | The dictionary for key `stream` contains information about the streamโs name (via the `stream` key), settings (e.g. replication method), metadata, and schema properties. |
| `columns` | `List[str]` | `['id', 'power', 'created_at']` | A list of column names from the data that was fetched for the stream. |
| `rows` | `List[List[Any]]` | `[[1, 3.5, '3000-01-01']]` | A list of lists where each item in the outer list is a row that was fetched from the stream. Each item in the inner list is a value from a column. The column that the value corresponds to is determined by the index of that value in its list and the column name at that same index in the `columns` list. |
### Sample output data
```json theme={"system"}
{
"columns": [
"passengerid",
"survived",
"pclass",
"name",
"sex",
"age",
"sibsp",
"parch",
"ticket",
"fare",
"cabin",
"embarked",
"_mage_created_at",
"_mage_updated_at"
],
"stream": {
"tap_stream_id": "titanic",
"replication_method": "FULL_TABLE",
"key_properties": [],
"schema": {
"properties": {
"passengerid": {
"type": [
"null",
"integer"
]
}
},
"type": "object"
},
"stream": "titanic",
"metadata": [
{
"breadcrumb": [],
"metadata": {
"table-key-properties": [],
"forced-replication-method": "FULL_TABLE",
"valid-replication-keys": [],
"inclusion": "available",
"schema-name": "titanic",
"selected": true
}
},
{
"breadcrumb": [
"properties",
"passengerid"
],
"metadata": {
"inclusion": "available",
"selected": true
}
}
],
"auto_add_new_fields": false,
"unique_conflict_method": "UPDATE",
"run_in_parallel": false
},
"rows": [
[
1,
0,
3,
"Braund, Mr. Owen Harris",
"male",
22.0,
1,
0,
"A/5 21171",
7.25,
null,
"S",
"2023-09-23T04:14:44.850136-07:00",
"2023-09-23T04:14:44.850136-07:00"
],
[
1,
0,
3,
"Braund, Mr. Owen Harris",
"male",
22.0,
1,
0,
"A/5 21171",
7.25,
null,
"S",
"2023-09-23T04:37:30.366570-07:00",
"2023-09-23T04:37:30.366570-07:00"
],
[
2,
1,
1,
"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",
"female",
38.0,
1,
0,
"PC 17599",
71.2833,
"C85",
"C",
"2023-09-23T04:14:44.850153-07:00",
"2023-09-23T04:14:44.850153-07:00"
],
[
2,
1,
1,
"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",
"female",
38.0,
1,
0,
"PC 17599",
71.2833,
"C85",
"C",
"2023-09-23T04:37:30.366592-07:00",
"2023-09-23T04:37:30.366592-07:00"
],
[
3,
1,
3,
"Heikkinen, Miss. Laina",
"female",
26.0,
0,
0,
"STON/O2. 3101282",
7.925,
null,
"S",
"2023-09-23T04:14:44.850155-07:00",
"2023-09-23T04:14:44.850155-07:00"
],
[
3,
1,
3,
"Heikkinen, Miss. Laina",
"female",
26.0,
0,
0,
"STON/O2. 3101282",
7.925,
null,
"S",
"2023-09-23T04:37:30.366594-07:00",
"2023-09-23T04:37:30.366594-07:00"
],
[
4,
1,
1,
"Futrelle, Mrs. Jacques Heath (Lily May Peel)",
"female",
35.0,
1,
0,
"113803",
53.1,
"C123",
"S",
"2023-09-23T04:14:44.850157-07:00",
"2023-09-23T04:14:44.850157-07:00"
]
]
}
```
***
## Load/Export stream data in parallel
For each stream in a data integration source block,
you can enable that stream to have its data fetched in parallel
while other streams for that source is being fetched.
For each stream in a data integration destination block,
you can enable that stream to have its data exported in parallel
while other streams for that destination is being exported.
This option can be turned on by editing a specific streamโs settings.
This option can also be turned on or off for multiple streams at once by bulk editing all the
streams at once in the Overview section:
***
## Incremental sync
Synchronize your data incrementally after the 1st sync.
When you choose 1 or more bookmark values in a stream, the value from that property will be
saved after each successful sync. The next sync will use that value to compare to the next
batch of records that need to be synced.
### Override bookmark values
If you want to change the last saved bookmark value for the next sync, you can do this in 2 ways:
### Example of bookmark values editor
# Configure data integration pipelines
Source: https://docs.mage.ai/data-integrations/configuration
There are many ways to configure your data integration pipeline.
## Interpolation
You can interpolate pipeline variables, environment variables, secrets, and more into the source
config file, schema settings, and destination config file.
### Pipeline variables
```
{{ variables('variable_name') }}
```
#### Variable names
You can add your own custom variables to the pipeline. For more information,
read the [documentation](/getting-started/runtime-variable#using-runtime-variables).
Here are the default variables available to interpolate:
| Name | Description | Sample value |
| --------------------- | -------------------------------------------------------------------------------- | ----------------- |
| `pipeline.name` | Name of the current pipeline. | `ETL demo` |
| `pipeline.uuid` | UUID of the current pipeline. | `etl_demo` |
| `ds` | Execution date string. | `2023-12-25` |
| `hr` | Execution hour string from `00` to `23`. | `12` |
| `env` | Environment Mage is running in. | `dev`, `prod` |
| `execution_partition` | The current partition of the execution: datetime in this format `%Y%m%dT%H%M%S`. | `20231225T122520` |
### Common prefixes
Mage supports table prefixes for integration sources. To add a prefix, add the following to the source configuration for a data integration pipeline:
```yaml theme={"system"}
_patterns:
destination_table: MY_PREFIX_{{ variables('stream') }}
```
For example, configuring a PostgreSQL source:
### Environment variables
```
{{ env_var('environment_variable_name') }}
```
### Secrets
Use the following syntax to interpolate secrets:
```
{{ mage_secret_var('your_secret_name') }}
```
For more information on secrets, read the [documentation](/development/variables/secrets).
# Airtable
Source: https://docs.mage.ai/data-integrations/destinations/airtable
How to configure and authenticate the Airtable destination for loading data into Airtable via the API.
Connect Mage to Airtable as a **destination** to load data into your bases. This guide explains how to configure authentication using a Personal Access Token and the required base ID.
***
## Configuration Parameters
To use Airtable as a destination, provide the following configuration values:
| Key | Description | Example Value | Required |
| --------- | ------------------------------------------------------------------------ | ---------------- | -------- |
| `token` | Airtable Personal Access Token with the required scopes | `pat1234abcd...` | โ |
| `base_id` | The unique ID of the Airtable base you're writing to (starts with `app`) | `app5678efgh...` | โ |
***
## Generating Credentials
To generate your Airtable Personal Access Token:
1. Visit the [Airtable Developer Hub](https://airtable.com/developers/web/guides/personal-access-tokens).
2. Click **Create token**.
3. Assign the appropriate scopes (listed below).
4. Save the token securely and include it in your Mage destination configuration.
***
## Required Scopes
Make sure your token includes the following scopes:
* `schema.bases:write` โ Required for schema access
* `data.records:write` โ Required to insert, update, or delete records in Airtable
***
## FAQs
### What is `base_id` in Airtable?
The `base_id` is a unique identifier for your Airtable base. It typically starts with `app` and can be found in the baseโs URL or retrieved using the Airtable API.
### How should I store my Airtable token securely?
Use environment variables or Mageโs secrets manager to store your token. Avoid hardcoding tokens in your configuration files or version control.
### Can Mage write data to multiple Airtable bases?
Yes, but you must configure each destination with its corresponding `base_id` and ensure the access token has permission for each base.
***
## Related Resources
* [Airtable API Docs](https://airtable.com/developers/web/api/introduction)
* [Schema Guide](https://airtable.com/developers/web/guides/schema)
* [Personal Access Token Guide](https://airtable.com/developers/web/guides/personal-access-tokens)
# Amazon S3
Source: https://docs.mage.ai/data-integrations/destinations/amazon_s3
How to configure Amazon S3 as a destination in Mage to write data to your S3 bucket in CSV or Parquet format.
Configure **Amazon S3** as a Mage destination to write data files in CSV or Parquet format directly into your AWS S3 bucket.
***
## Configuration Parameters
The following parameters are required to set up Amazon S3 as a destination:
| Key | Description | Example Value | Required |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -------- |
| `aws_access_key_id` | Your AWS access key ID. | `AKIA...` | โ |
| `aws_secret_access_key` | Your AWS secret access key. | `xyz456...` | โ |
| `aws_region` | AWS region where your S3 bucket is located. | `us-west-2` | โ |
| `role_arn` | (Optional) ARN of an IAM role to assume for access. Useful for temporary or cross-account access. | `arn:aws:iam::111111:role/example-role` | โ |
| `bucket` | The name of the S3 bucket where data will be written. | `user_generated_content` | โ |
| `file_type` | The file format to use when writing to S3. Supported values: `csv`, `parquet`. | `parquet` | โ |
| `object_key_path` | The S3 object key prefix (path) to write files under. Do **not** include `s3://`, the bucket name, or table name in this path. | `users/ds/20221225` | โ |
| `column_header_format` | (Optional) Format for column headers. Will be applied before writing to S3. Options: `upper`, `lower`, or `null` (no formatting). | `lower` | โ |
| `date_partition_format` | (Optional) Format for date-based partitioning in file paths. If null, partitioning is skipped. Examples: `%Y%m%d`, `%Y%m%dT%H`. | `%Y%m%d` | โ |
| `aws_endpoint` | (Optional) Custom endpoint URL for non-AWS S3-compatible services (e.g., MinIO). | `https://play.min.io` | โ |
***
## Credential Requirements
To use Amazon S3, you need an AWS IAM user or role with permissions to:
* Put objects (`s3:PutObject`)
* Create folders (`s3:PutObject`)
* Optionally assume roles (`sts:AssumeRole`) if `role_arn` is used
Ensure the credentials provided have appropriate access to the target bucket and path.
***
## FAQs
### Can I write to MinIO or other S3-compatible storage?
Yes. Set the `aws_endpoint` field to your custom endpoint URL (e.g., `https://play.min.io`). Ensure your credentials and permissions are configured accordingly.
### How are files named in S3?
Mage automatically handles file naming using table names and batch metadata. Your `object_key_path` acts as the prefix under which the files are written.
### What happens if I donโt set `date_partition_format`?
Files will be written directly into the specified path without date-based folder partitioning.
### What if both `role_arn` and access keys are set?
Mage will attempt to assume the IAM role using the provided access key credentials. This is helpful in cross-account setups or for short-lived access via STS.
***
## Related Resources
* [Amazon S3 Documentation](https://docs.aws.amazon.com/s3/)
* [IAM Policy Reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html)
* [MinIO Documentation](https://min.io/docs/minio/linux/index.html)
# BigQuery
Source: https://docs.mage.ai/data-integrations/destinations/bigquery
How to configure BigQuery as a destination in Mage for loading data into your Google Cloud BigQuery datasets.
Use **BigQuery** as a Mage destination to load structured data into Google Cloudโs fully-managed, serverless data warehouse.
***
## Configuration Parameters
When configuring BigQuery as a destination, provide the following parameters:
| Key | Description | Example Value | Required |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | -------- |
| `path_to_credentials_json_file` | Absolute path to your Google Cloud service account credentials file (JSON). If Mage is running on a GCP instance, leave this as `null` to use the instance's service account. | `/path/to/credentials.json` | โ |
| `project_id` | The GCP project ID where your BigQuery dataset exists. | `example_project` | โ |
| `dataset` | Name of the BigQuery dataset where data will be written. | `example_dataset` | โ |
| `location` | (Optional) The region of your BigQuery dataset. If not set, BigQuery may infer it automatically. | `us-west1` | โ |
| `disable_update_column_types` | If `false`, Mage will alter existing BigQuery column types to match the schema when necessary. If `true`, existing types will not be modified. | `false` (default) | โ |
| `use_batch_load` | Use BigQuery **load jobs** instead of the streaming API. Recommended for better performance and cost efficiency. | `true` (default) | โ |
| `credentials_info` | Alternative to `path_to_credentials_json_file`. Directly embed your credentials JSON as a dictionary (useful for environment-based configs). | See structure below | โ |
### `credentials_info` Structure
If you're passing credentials inline instead of a path, structure it like this:
```yaml theme={"system"}
auth_provider_x509_cert_url: str
auth_uri: str
client_email: str
client_id: str
client_x509_cert_url: str
private_key: str
private_key_id: str
project_id: str
token_uri: str
type: str
```
***
## Optional Configuration
| Key | Description | Example Value | Default |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------- |
| `skip_schema_creation` | If `true`, Mage will **not** run a `CREATE SCHEMA` command. Useful if the schema already exists. See [issue #3416](https://github.com/mage-ai/mage-ai/issues/3416). | `true` | `false` |
| `lower_case` | If `true`, column names will be automatically converted to lowercase before writing to BigQuery. | `true` | `true` |
## Mage Pro Features
The following features are available only in [**Mage Pro**](https://cloud.mage.ai):
### Advanced Configuration (Mage Pro only)
| Key | Description | Default |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `truncate_table_before_full_table_sync` | If `true`, Mage will truncate (empty) the destination table before performing a full table sync. This ensures no old data remains. | `false` |
***
## Required Permissions
Your GCP service account or user must have the following permissions to use BigQuery as a destination:
```text theme={"system"}
bigquery.datasets.create
bigquery.jobs.create
bigquery.readsessions.create
bigquery.readsessions.getData
```
Additionally, the account must have the **BigQuery Data Editor** role on the target dataset:
* Role: `roles/bigquery.dataEditor`
***
## Related Resources
* [BigQuery API Overview](https://cloud.google.com/bigquery/docs/reference/rest)
* [Google Cloud IAM Roles](https://cloud.google.com/bigquery/docs/access-control)
* [Mage GitHub Issue #3416](https://github.com/mage-ai/mage-ai/issues/3416)
# ClickHouse
Source: https://docs.mage.ai/data-integrations/destinations/clickhouse
How to configure ClickHouse as a destination in Mage to write data to a ClickHouse database.
Configure **ClickHouse** as a Mage destination to write tabular data into a ClickHouse database using the SQLAlchemy driver.
***
## Configuration Parameters
Provide the following parameters to use ClickHouse as a destination:
| Key | Description | Example Value | Required |
| ---------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------- | -------- |
| `sqlalchemy_url` | SQLAlchemy connection string used to connect to ClickHouse. Includes host, user, password, etc. | `clickhouse://default:@localhost/default` | โ |
| `table_name` | (Optional) Name of the destination table to write data into. If omitted, the stream name will be used. | `abc123` | โ |
***
## SQLAlchemy URL Format
The `sqlalchemy_url` should follow this structure:
```
clickhouse://:@:/
```
**Examples:**
* Without password (using default database):\
`clickhouse://default:@localhost/default`
* With credentials:\
`clickhouse://user:pass123@127.0.0.1:9000/analytics_db`
Ensure the database exists in ClickHouse before running the Mage pipeline.
***
## Requirements
* The ClickHouse database must be reachable from the Mage environment.
* Mage requires the [`clickhouse-connect`](https://pypi.org/project/clickhouse-connect/) Python package to be installed.
* The target table should match the schema or allow dynamic schema creation depending on your setup.
***
## FAQs
### Can I use a secure connection?
Yes. To use HTTPS or TLS-secured connections, ensure your SQLAlchemy URL includes the appropriate protocol and port, and that your ClickHouse server supports it.
### What happens if `table_name` is not provided?
Mage will automatically use the stream name as the destination table name.
***
## Related Resources
* [ClickHouse SQLAlchemy Docs](https://clickhouse.com/docs/en/integrations/sqlalchemy)
* [ClickHouse Official Documentation](https://clickhouse.com/docs/en/)
* [clickhouse-connect PyPI Package](https://pypi.org/project/clickhouse-connect/)
# Databricks
Source: https://docs.mage.ai/data-integrations/destinations/databricks
How to configure Databricks as a destination in Mage to write pipeline data to Databricks SQL warehouses using Unity Catalog.
Use **Databricks** as a destination in Mage to load structured data into your Databricks workspace. Mage integrates with Databricks SQL warehouses and uses **Unity Catalog** for table management.
This destination is ideal for exporting transformed pipeline data to Databricks for analytics, ML workloads, or data lakehouse operations.
***
## Required Configuration
Provide the following credentials when configuring Databricks as a destination:
| Key | Description | Example Value | Required |
| ----------------- | ----------------------------------------------------- | --------------------------------- | -------- |
| `access_token` | Personal access token to authenticate with Databricks | `dapi123abc...` | โ |
| `server_hostname` | Hostname of your Databricks workspace | `dbc-123456.cloud.databricks.com` | โ |
| `http_path` | HTTP path of the Databricks SQL warehouse or cluster | `/sql/1.0/warehouses/abc123` | โ |
| `schema` | Name of the schema (database) within the catalog | `analytics` | โ |
| `table` | Name of the table to write data into | `user_events` | โ |
***
## Optional Configuration
| Key | Description | Example Value | Default |
| ---------------------- | ------------------------------------------------------------------------------- | ------------- | ------- |
| `catalog` | Name of the Unity Catalog to use | `workspace` | - |
| `skip_schema_creation` | If `true`, Mage won't run `CREATE SCHEMA`. Useful if the schema already exists. | `true` | `false` |
| `lower_case` | If `true`, all column names will be lowercased. | `true` | `true` |
| `allow_reserved_words` | If `true`, Mage will allow use of SQL reserved words as column names. | `false` | `false` |
***
## Notes
* **Unity Catalog**: Set `catalog` to your Unity Catalog name. This is an optional field for table management in Databricks.
* **Access Token**: Generate a personal access token from your Databricks workspace settings.
* **HTTP Path**: Find the HTTP path in your SQL warehouse or cluster connection details.
* **Permissions**: Ensure your access token has permissions to create schemas and write to tables in the specified catalog and schema.
***
## Related Resources
* [Databricks SQL Documentation](https://docs.databricks.com/sql/)
* [Unity Catalog Overview](https://docs.databricks.com/data-governance/unity-catalog/index.html)
* [Databricks Personal Access Tokens](https://docs.databricks.com/dev-tools/auth/pat.html)
# Delta Lake (Azure)
Source: https://docs.mage.ai/data-integrations/destinations/delta_lake_azure
How to configure Delta Lake on Azure as a destination in Mage to write data to a Delta table using ABFS.
## Configuration Parameters
To use **Delta Lake on Azure** as a destination, provide the following parameters:
| Key | Description | Example Value | Required |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------- |
| `account_name` | Your Azure Storage account name. This is the name of the account that hosts your Delta table. | `myazurestorage` | โ |
| `access_key` | The access key for your Azure Storage account. Used to authenticate writes to your Delta table. | `ABC123...XYZ` | โ |
| `table_uri` | URI for the Delta table. Must follow the `abfs:///` format. For more options, refer to the [delta-rs docs](https://delta-io.github.io/delta-rs/python/usage.html). | `abfs://my-container/delta/my-table` | โ |
| `mode` | Write mode. - `append`: Adds rows to the table. Errors if schema changes. - `overwrite`: Replaces all existing rows and recreates the table. | `append` (default) or `overwrite` | โ |
***
## Notes
* The table URI must point to a valid **Delta Lake table path** accessible via Azure Data Lake Storage (Gen2) using the **ABFS** protocol.
* Mage uses the [delta-rs](https://delta-io.github.io/delta-rs/python/usage.html) engine for interacting with Delta Lake.
* If using `overwrite` mode, ensure downstream systems tolerate table recreation.
***
## Related Resources
* [Delta Lake Python API (`delta-rs`)](https://delta-io.github.io/delta-rs/python/usage.html)
* [Delta Lake Overview](https://delta.io/)
* [Azure Storage ABFS URI Format](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri)
# Delta Lake (S3)
Source: https://docs.mage.ai/data-integrations/destinations/delta_lake_s3
How to configure Delta Lake on AWS S3 as a destination in Mage to write data to a Delta table using S3 URIs.
## Configuration Parameters
To use **Delta Lake on AWS S3** as a destination in Mage, provide the following parameters:
| Key | Description | Example Value | Required |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | -------- |
| `aws_access_key_id` | Your AWS access key ID. Required to authenticate with S3. | `AKIAIOSFODNN7EXAMPLE` | โ |
| `aws_secret_access_key` | Your AWS secret access key. Used for writing to the S3 bucket. | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` | โ |
| `aws_region` | AWS region where your S3 bucket is located. | `us-west-2` *(default)* | โ |
| `bucket` | Name of the S3 bucket where the Delta table is stored. | `mage-spark-data` | โ |
| `table` | Logical name of the Delta table that Mage will write to. | `dim_profiles_v1` | โ |
| `mode` | Write mode: - `append`: Adds rows to the table. Fails if schema changes. - `overwrite`: Replaces all existing rows and recreates the table. | `append` (default) or `overwrite` | โ |
| `object_key_path` | Path inside the S3 bucket where the Delta table is written. Exclude `s3://`, bucket name, and table name from this path. | `users/ds/20250708` | โ |
***
## Notes
* The object key path is relative to the S3 bucket and should not include the scheme (`s3://`), bucket name, or table name.
* Mage uses the [delta-rs](https://delta-io.github.io/delta-rs/python/usage.html) engine for writing data to Delta Lake on S3.
* Use `overwrite` mode with caution: it deletes all existing rows and recreates the table.
***
## Related Resources
* [Delta Lake Python API (`delta-rs`)](https://delta-io.github.io/delta-rs/python/usage.html)
* [Delta Lake Overview](https://delta.io/)
* [Working with S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html)
# Doris
Source: https://docs.mage.ai/data-integrations/destinations/doris
How to configure Doris as a destination in Mage to write data to a Doris database table, including SSH tunnel and optional connection settings.
## Configuration Parameters
To use **Doris** as a destination in Mage, provide the following parameters:
| Key | Description | Example Value | Required |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | -------- |
| `database` | Name of the Doris database to export data to. | `demo` | โ |
| `host` | Hostname or IP address of your Doris database server. | `mage.abc.us-west-2.rds.amazonaws.com` | โ |
| `port` | Port number on which Doris is listening. The default is usually `9030`. | `9030` | โ |
| `username` | Username used to connect to the Doris database. Must have permission to read and write data. | `root` | โ |
| `password` | Password for the above user. | `abc123...` | โ |
| `table` | Name of the destination table in Doris that Mage will create or write to. | `dim_users_v1` | โ |
| `connection_method` | Method used to connect to Doris. Choose between `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` | โ |
| `ssh_host` | *(Optional)* Bastion host IP or DNS if using SSH tunnel. | `123.45.67.89` | โ |
| `ssh_port` | *(Optional)* Port of the bastion host. Defaults to `22`. | `22` | โ |
| `ssh_username` | *(Optional)* Username to authenticate with the bastion server. | `bastion_user` | โ |
| `ssh_password` | *(Optional)* Password for SSH authentication. Required if not using a private key. | `password` | โ |
| `ssh_pkey` | *(Optional)* Path to or contents of private key file used for SSH authentication. | `/path/to/private/key.pem` | โ |
| `conn_kwargs` | *(Optional)* Additional [MySQL connection options](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) as a dictionary. | `{"ssl_ca": "CARoot.pem"}` | โ |
| `use_lowercase` | *(Optional)* Whether to convert column names to lowercase. | `true` or `false` | โ |
***
## Optional Settings
| Key | Description | Example Value | Required |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -------- |
| `skip_schema_creation` | If `true`, Mage will not run the `CREATE SCHEMA` command. Useful if the schema already exists. [Related issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` | โ |
| `lower_case` | If `true`, Mage will enforce lowercase column names. Default is `true`. | `true` | โ |
***
## Notes
* **SSH tunneling** is useful for secure access to Doris behind firewalls or private networks.
* `conn_kwargs` is especially helpful if you need to enable SSL or set advanced connection options.
* Doris uses a MySQL-compatible protocol, so many standard MySQL configurations also apply.
***
## Frequently Asked Questions
### Can I connect to Doris using an SSH tunnel?
Yes. Set `connection_method` to `ssh_tunnel` and provide the appropriate `ssh_host`, `ssh_username`, and either `ssh_password` or `ssh_pkey` depending on your authentication method.
### Can I use a private key for SSH authentication?
Yes. Use the `ssh_pkey` parameter to provide the file path or content of the SSH private key. This is recommended over using `ssh_password`.
### Does Doris support SSL connections?
Yes. Use the `conn_kwargs` parameter to pass MySQL-compatible SSL options like `ssl_ca`, `ssl_cert`, and `ssl_key`.
### What happens if I use `overwrite` mode?
Mage will drop the existing table (if it exists) and recreate it, replacing all existing data. Make sure your downstream systems can handle this reset.
### Whatโs the difference between `use_lowercase` and `lower_case`?
Both settings control whether column names are converted to lowercase. `use_lowercase` applies during export, while `lower_case` can be used to enforce this behavior globally. We recommend keeping them consistent.
***
## Related Resources
* [Doris Official Documentation](https://doris.apache.org/)
* [MySQL Connector Python Docs](https://dev.mysql.com/doc/connector-python/en/)
* [How to Configure SSH Tunnel Connections](https://www.ssh.com/academy/ssh/tunneling)
# DuckDB
Source: https://docs.mage.ai/data-integrations/destinations/duckdb
How to configure DuckDB as a destination in Mage to write pipeline data to local DuckDB files for fast, embedded analytics.
***
## Overview
Use **DuckDB** as a destination in Mage to write structured data to local DuckDB database files. DuckDB is an embedded analytical database that runs in-process, making it ideal for fast local analytics, data staging, or lightweight data processing workflows.
This destination is perfect for scenarios where you need to quickly analyze data locally before pushing to a larger data warehouse, or for building lightweight analytics pipelines.
***
## Required Configuration
You must enter the following configuration when writing to a DuckDB destination:
| Key | Description | Example Value | Required |
| ----------- | ----------------------------------------------------------------------- | ------------------------------------------- | -------- |
| `file_path` | Absolute file path to the `.duckdb` file where data will be saved. | `/home/src/default_repo/mage_output.duckdb` | โ |
| `if_exists` | What to do if the table already exists: `replace`, `append`, or `fail`. | `replace` | โ |
***
## Configuration Options
### `if_exists` Values
| Value | Description |
| --------- | ---------------------------------------------------------------------------------------------- |
| `replace` | Overwrites the existing table with new data. The table will be recreated on each pipeline run. |
| `append` | Adds new data to the existing table. Existing rows remain unchanged. |
| `fail` | Raises an error if the table already exists. Useful for preventing accidental overwrites. |
***
## Notes
* **Embedded Database**: DuckDB is an embedded database. The file is local to the machine or container where Mage is running.
* **Pandas Integration**: Supports writing pandas DataFrames directly to tables in a `.duckdb` file.
* **File Path**: Use absolute paths to ensure the file is created in the correct location.
* **Performance**: DuckDB is optimized for analytical queries and can handle large datasets efficiently.
* **Use Cases**: Ideal for fast, local analytics or staging data before pushing to a warehouse.
***
## Sample Configuration
```yaml theme={"system"}
file_path: /home/src/default_repo/mage_output.duckdb
if_exists: replace
```
***
## Related Resources
* [DuckDB Documentation](https://duckdb.org/docs/)
* [DuckDB Python API](https://duckdb.org/docs/api/python/overview)
* [DuckDB GitHub Repository](https://github.com/duckdb/duckdb)
# Elasticsearch
Source: https://docs.mage.ai/data-integrations/destinations/elasticsearch
How to configure Elasticsearch as a destination in Mage to stream or batch index records into an Elasticsearch cluster using custom index metadata, authentication, and bulk operations.
***
## Configuration Parameters
By default, the Mage `table` name is used to set the Elasticsearch **index** name.
| Key | Description | Default | Required |
| --------------------- | ---------------------------------------------------------------------------------------------------------------- | ----------- | -------- |
| `scheme` | HTTP scheme used to connect to Elasticsearch (`http` or `https`). | `http` | โ |
| `host` | Elasticsearch host. Can be a domain or IP address. | `localhost` | โ |
| `port` | Elasticsearch port. | `9200` | โ |
| `username` | Username for basic authentication. | `None` | โ |
| `password` | Password for basic authentication. | `None` | โ |
| `bearer_token` | Bearer token for token-based authentication. | `None` | โ |
| `api_key_id` | API key ID for key-based authentication. | `None` | โ |
| `api_key` | API key for key-based authentication. | `None` | โ |
| `ssl_ca_file` | Path to SSL certificate file used for TLS verification. | `None` | โ |
| `verify_certs` | Whether to verify SSL certificates when using HTTPS. | `True` | โ |
| `index_schema_fields` | JSONPath mapping for selecting values from the record to generate index names dynamically. | `None` | โ |
| `_op_type` | Operation type for the Elasticsearch request. Typically `index`, but can be `create`, `update`, or `delete`. | `index` | โ |
| `metadata_fields` | Dictionary used to map fields in the input record to metadata (e.g. `_id`) in the Elasticsearch index request. | `None` | โ |
| `bulk_kwargs` | Settings that control Elasticsearch `bulk` behavior. Useful for tuning performance and retries. See table below. | `None` | โ |
***
## Metadata Field Mapping
Use `metadata_fields` to extract values from incoming records and apply them to Elasticsearch document metadata (e.g., `_id`):
```yaml theme={"system"}
metadata_fields:
:
:
```
**Example:**
```yaml theme={"system"}
metadata_fields:
stream_name:
_id: my_id
```
> This will set the `_id` of each indexed document to the value of the `my_id` field in the input record.
***
## Bulk Operation Settings (`bulk_kwargs`)
All options below are optional, but useful for controlling indexing performance and fault tolerance.
| Key | Description | Default |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| `use_parallel` | If `true`, enables parallel bulk indexing using multiple threads. Can significantly improve throughput. | `false` |
| `chunk_size` | Max number of records per bulk request. | `500` |
| `max_chunk_bytes` | Max size in bytes of each bulk request payload. | `104857600` |
| `max_retries` | *(Only with `use_parallel: false`)* Number of retry attempts on HTTP 429 (too many requests) errors. | `0` |
| `initial_backoff` | *(Only with `use_parallel: false`)* Initial wait time (in seconds) before retry. Each subsequent retry backs off exponentially. | `2` |
| `max_backoff` | *(Only with `use_parallel: false`)* Maximum time to wait before retrying. | `600` |
| `thread_count` | *(Only with `use_parallel: true`)* Number of threads to use in the bulk operation thread pool. | `4` |
| `queue_size` | *(Only with `use_parallel: true`)* Size of the task queue buffering chunks between the main thread and processing threads. | `4` |
***
## Example Configuration
```yaml theme={"system"}
scheme: http
host: localhost
port: 9200
metadata_fields:
stream_name:
_id: my_id
bulk_kwargs:
use_parallel: true
chunk_size: 500
```
***
## Notes
* You can authenticate with **basic auth**, **bearer tokens**, or **API keys**.
* Set `_op_type: create` if you want to avoid overwriting documents with the same `_id`.
* For production environments, it's recommended to use `https`, enable `verify_certs`, and configure `ssl_ca_file`.
* Performance tuning with `bulk_kwargs` is especially helpful for large datasets or real-time indexing.
***
## Related Resources
* [Elasticsearch Bulk API Docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)
* [Elasticsearch Python Client (elasticsearch-py)](https://elasticsearch-py.readthedocs.io/en/v8.11.1/)
* [Elasticsearch Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)
# Google Cloud Storage
Source: https://docs.mage.ai/data-integrations/destinations/google_cloud_storage
How to configure Google Cloud Storage (GCS) as a destination in Mage to write data files in Parquet or CSV format with optional date partitioning.
## Overview
Use **Google Cloud Storage (GCS)** as a destination in Mage to write data from your pipelines directly into a GCS bucket. This is ideal for exporting pipeline outputs in formats like **Parquet** or **CSV** for downstream analytics, storage, or ML workflows.
Mage supports native integration with GCS using either a service account key file or automatic authentication via GCP instance metadata.
***
## Configuration Parameters
You must provide the following parameters when configuring GCS as a destination:
| Key | Description | Example Value | Required |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | -------- |
| `bucket` | Name of the target GCS bucket where files will be stored. | `user_generated_content` | โ |
| `google_application_credentials` | Path to the GCP service account JSON key file. Leave as `null` if Mage is running on GCP and should use the default instance service account. | `/path/to/key.json` | โ |
| `file_type` | Format of the files to be written. Supported values: `parquet`, `csv`. | `parquet` or `csv` | โ |
| `object_key_path` | Relative path inside the bucket where files will be stored. Do **not** include `gs://`, bucket name, or table name. | `users/ds/20221225` | โ |
| `date_partition_format` | *(Optional)* Datetime format to use for partitioned folders. If `null`, partitioning will be skipped. Examples: `%Y%m%d`, `%Y%m%dT%H`. | `null` or `%Y%m%d` | โ |
***
## Notes
* The `object_key_path` defines where in the GCS bucket the data will be written. Mage will automatically append file extensions based on the selected `file_type`.
* If you're running Mage inside a GCP service (like Cloud Run, GKE, or Compute Engine), and you leave `google_application_credentials` as `null`, Mage will use the instance's IAM identity for authentication.
* Use `date_partition_format` to enable directory-based partitioning by time, which is useful for managing time-series datasets or incremental loads.
***
## Related Resources
* [Google Cloud Storage Documentation](https://cloud.google.com/storage/docs/introduction)
* [GCS URI Naming Guidelines](https://cloud.google.com/storage/docs/gsutil/addlhelp/HowNamingWorks)
* [GCP Service Account Authentication](https://cloud.google.com/docs/authentication/getting-started)
***
## Frequently Asked Questions
### Can Mage write to a GCS bucket if it's running inside GCP?
Yes. If Mage is running inside a GCP environment (like GKE, Cloud Run, or Compute Engine), you can leave `google_application_credentials` as `null`. Mage will use the default service account attached to the instance for authentication.
***
### What file formats are supported when exporting data to GCS?
Mage currently supports `parquet` and `csv` as the `file_type` options when writing data to Google Cloud Storage.
***
### How does `object_key_path` work?
`object_key_path` specifies the path inside the bucket where your files will be stored. It should exclude the bucket name and should not include `gs://`. For example, `users/2025_07_08`.
***
### What does `date_partition_format` do?
When provided, this setting tells Mage to create date-based subfolders (e.g., `20250708`, `20250708T13`) based on the run time. If set to `null`, partitioning is disabled and all files will go into the specified `object_key_path`.
***
### Do I need to manually set file extensions?
No. Mage automatically adds the appropriate extension (`.parquet` or `.csv`) to each file based on the `file_type` you select.
***
### Can I write to a multi-region or dual-region GCS bucket?
Yes. Mage can write to any standard, multi-region, or dual-region GCS bucket as long as the configured credentials have write access.
# Iceberg S3
Source: https://docs.mage.ai/data-integrations/destinations/iceberg_s3
How to configure Apache Iceberg on S3 as a destination in Mage to write data to ACID-compliant Iceberg tables stored in AWS S3.
Use **Apache Iceberg on S3** as a destination in Mage to write data to ACID-compliant Iceberg tables stored in AWS S3.
***
## Overview
This destination writes data to Apache Iceberg tables stored in AWS S3. Iceberg provides:
* **ACID Properties**: Full ACID compliance for data consistency
* **Schema Evolution**: Add, remove, or modify columns without breaking existing queries
* **Time Travel**: Query data as it was at any point in time
* **Partitioning**: Automatic partitioning for better query performance
* **Compaction**: Automatic file compaction to optimize storage
***
## Required Configuration
| Parameter | Description | Required | Default |
| ----------------------- | -------------------- | -------- | ------- |
| `aws_access_key_id` | S3 access key ID | โ | - |
| `aws_secret_access_key` | S3 secret access key | โ | - |
| `bucket` | S3 bucket name | โ | - |
| `namespace` | Iceberg namespace | โ | - |
| `table_name` | Iceberg table name | โ | - |
***
## Optional Configuration
| Parameter | Description | Required | Default |
| ------------ | ----------- | -------- | ----------- |
| `aws_region` | AWS region | โ | `us-west-2` |
***
## Configuration Example
```yaml theme={"system"}
aws_access_key_id: your_access_key
aws_region: us-west-2
aws_secret_access_key: your_secret_key
bucket: your-bucket-name
namespace: your_namespace
table_name: your_table_name
```
***
## Table Structure
Iceberg tables are created with the following structure:
```
s3://bucket/namespace.db/table_name/
โโโ metadata/
โ โโโ v1.metadata.json
โ โโโ v2.metadata.json
โ โโโ ...
โโโ data/
โ โโโ 00000-0-abc123.parquet
โ โโโ 00001-0-def456.parquet
โ โโโ ...
โโโ snapshots/
โโโ 1234567890.avro
โโโ ...
```
***
## Usage Examples
### Creating a New Table
The destination automatically creates the table if it doesn't exist, using the schema from the incoming data.
### Appending Data
New data is automatically appended to the existing Iceberg table.
### Schema Evolution
If the incoming data has new columns, the table schema is automatically updated.
***
## Troubleshooting
### Common Issues
1. **Access Denied**: Check your S3 credentials and bucket permissions
2. **Region Mismatch**: Ensure the region matches your S3 bucket configuration
### Debugging
Enable debug logging to see detailed information about table creation and data writing operations.
***
## Related Resources
* [Apache Iceberg Documentation](https://iceberg.apache.org/docs/)
* [Iceberg S3 Configuration](https://iceberg.apache.org/docs/latest/aws/)
* [AWS S3 Documentation](https://docs.aws.amazon.com/s3/)
# Kafka
Source: https://docs.mage.ai/data-integrations/destinations/kafka
How to configure Apache Kafka as a destination in Mage to publish JSON records to a Kafka topic using optional message keys.
***
## Overview
Use **Apache Kafka** as a destination in Mage to stream records from your pipeline into Kafka topics in real time. Each record is serialized as a JSON object and sent to the configured `bootstrap_server`.
If a `Key Property` is defined in the Source Schema, Mage will use the corresponding field value as the Kafka message keyโenabling partitioned message routing and supporting downstream systems that rely on message keys.
This is ideal for:
* Real-time data streaming
* Event-driven microservices
* Kafka-based data pipelines and messaging queues
***
## Configuration Parameters
| Key | Description | Default | Required |
| ------------------ | --------------------------------------------------------------------------------------------------- | ------- | -------- |
| `bootstrap_server` | Comma-separated list of Kafka brokers with ports (e.g., `kafka:9093`, `broker1:9092,broker2:9092`). | None | โ |
| `topic` | Kafka topic to publish messages to. | None | โ |
| `api_version` | *(Optional)* Kafka API version to use. If unspecified, Mage will auto-detect the server version. | None | โ |
***
## Notes
* Records are serialized as **JSON** using the [Kafka Python client](https://kafka-python.readthedocs.io/en/master/).
* To set Kafka message keys, define a `Key Property` in your source schema.
* You can explicitly set the `api_version` to match your Kafka broker version if needed.
* Compatible with self-hosted Kafka, Confluent Cloud, Amazon MSK, and other Kafka-compatible platforms.
***
## Example Configuration
```yaml theme={"system"}
bootstrap_server: broker1.kafka.internal:9092,broker2.kafka.internal:9092
topic: mage_data_stream
api_version: (0, 10, 1)
```
***
## Related Resources
* [Apache Kafka Documentation](https://kafka.apache.org/documentation/)
* [Kafka Python Client (kafka-python)](https://kafka-python.readthedocs.io/en/master/)
* [Confluent Kafka Topic Design](https://developer.confluent.io/learn/kafka/topic-design/)
# MongoDB
Source: https://docs.mage.ai/data-integrations/destinations/mongodb
How to configure MongoDB as a destination in Mage to write data to a MongoDB database using a connection string.
## Overview
Use **MongoDB** as a destination in Mage to export records from your pipelines directly into a MongoDB collection. This is ideal for persisting semi-structured JSON data, powering APIs, or integrating with operational data systems.
Each record from your pipeline will be inserted as a document in the specified MongoDB collection. If no collection name is provided, Mage will default to using the stream name as the collection name.
***
## Configuration Parameters
| Key | Description | Example Value | Required |
| ------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------- | -------- |
| `connection_string` | MongoDB connection URI string, including hostname and port. | `mongodb://mongodb0.example.com:27017` | โ |
| `db_name` | Name of the MongoDB database where documents will be stored. | `admin` | โ |
| `table_name` | *(Optional)* Name of the MongoDB collection to write to. If omitted, the stream name is used. | `user_profiles` | โ |
***
## Notes
* The `connection_string` should include any authentication, cluster details, or replica set info as needed (e.g., `mongodb+srv://user:pass@cluster0.mongodb.net/`).
* If `table_name` is not set, Mage automatically uses the name of the stream that produced the data.
* Mage inserts each record as a single MongoDB document in the target collection.
***
## Example Configuration
```yaml theme={"system"}
connection_string: mongodb://mongodb0.example.com:27017
db_name: analytics
```
***
## Related Resources
* [MongoDB Connection String URI Format](https://www.mongodb.com/docs/manual/reference/connection-string/)
* [MongoDB Python Driver (`pymongo`)](https://pymongo.readthedocs.io/en/stable/)
* [MongoDB Collections vs Tables](https://www.mongodb.com/docs/manual/core/databases-and-collections/)
# MSSQL (Microsoft SQL Server)
Source: https://docs.mage.ai/data-integrations/destinations/mssql
How to configure Microsoft SQL Server (MSSQL) as a destination in Mage to write pipeline data into a SQL Server database using SQL authentication or Azure AD.
## Overview
Use **Microsoft SQL Server (MSSQL)** as a destination in Mage to export data from your pipelines into relational tables for analytics, reporting, or operational workloads. Mage supports both on-premise SQL Server deployments and cloud-hosted options like **Azure SQL Database**.
***
## Configuration Parameters
You must provide the following credentials to configure the MSSQL destination:
| Key | Description | Example Value | Required |
| ---------- | ----------------------------------------------------------------------------- | ------------- | -------- |
| `database` | Name of the target database where data will be written. | `msdb` | โ |
| `host` | Hostname or IP address of your MSSQL server. | `172.20.0.2` | โ |
| `port` | Port used by the MSSQL service. Typically `1433`. | `1433` | โ |
| `username` | Username with permission to read and write to the specified schema and table. | `guest` | โ |
| `password` | Password for the MSSQL user. | `abc123...` | โ |
| `schema` | Target schema inside the database. | `public` | โ |
| `table` | Name of the destination table Mage will create or write to. | `users` | โ |
***
## Optional Parameters
| Key | Description | Example Value |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `authentication` | Authentication method. Set to `ActiveDirectoryServicePrincipal` to use [Azure AD authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver16). | `ActiveDirectoryServicePrincipal` |
| `driver` | Name of the [ODBC driver for SQL Server](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16). | `ODBC Driver 18 for SQL Server` |
| `skip_schema_creation` | If `true`, Mage will skip the `CREATE SCHEMA` command during initialization. [See GitHub issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, all column names will be converted to lowercase. Default is `true`. | `true` |
| `connection_method` | Connection method. Set to `ssh_tunnel` to connect via SSH tunnel. Default: `direct` | `ssh_tunnel` |
| `ssh_host` | (SSH tunnel only) The host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (SSH tunnel only) The port of the intermediate bastion server. Default: `22` | `22` |
| `ssh_username` | (SSH tunnel only) The username used to connect to the bastion server. | `username` |
| `ssh_password` | (SSH tunnel only, optional) The password used to connect to the bastion server. Required if not using `ssh_pkey`. | `password` |
| `ssh_pkey` | (SSH tunnel only, optional) The path to the private key used to connect to the bastion server. Required if not using `ssh_password`. | `/path/to/private/key` |
SSH tunneling is a Mage Pro only feature.
***
## Notes
* Mage uses the **SQLAlchemy MSSQL dialect** to connect and write data into Microsoft SQL Server.
* Ensure the user has `INSERT` and `CREATE TABLE` permissions if writing to new tables.
* Azure-hosted SQL Servers may require SSL or Azure AD authentication depending on your network configuration.
* Make sure any required firewall rules or virtual network settings are properly configured for external access.
***
## Related Resources
* [Microsoft SQL Server Docs](https://learn.microsoft.com/en-us/sql/sql-server/)
* [ODBC Driver for SQL Server](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server)
* [Azure Active Directory Authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver16)
# MySQL
Source: https://docs.mage.ai/data-integrations/destinations/mysql
How to configure MySQL as a destination in Mage to write pipeline data to a MySQL table with optional SSH tunneling and connection parameters.
***
## Overview
Use **MySQL** as a destination in Mage to write pipeline data into structured tables within a MySQL database. This integration supports both direct connections and **SSH tunneling** for secure access to remote or private databases.
Each record from the Mage pipeline is inserted as a row in the configured table. Mage supports optional connection customization and SSL settings through connection keyword arguments.
***
## Configuration Parameters
You must provide the following parameters when configuring the MySQL destination:
| Key | Description | Example Value | Required |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------- | -------- |
| `database` | Name of the MySQL database where data will be written. | `demo` | โ |
| `host` | Hostname or IP address of the MySQL server. | `mage.abc.us-west-2.rds.amazonaws.com` | โ |
| `port` | Port number used by MySQL (typically `3306`). | `3306` | โ |
| `username` | Username with permission to access the database and target schema. | `root` | โ |
| `password` | Password for the MySQL user. | `abc123...` | โ |
| `table` | Name of the destination table that Mage will write to. | `dim_users_v1` | โ |
| `connection_method` | Method used to connect to the MySQL server: `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` | โ |
| `ssh_host` | *(Optional)* Host of the bastion or jump server for SSH tunnel access. | `123.45.67.89` | โ |
| `ssh_port` | *(Optional)* SSH port on the bastion host. Default is `22`. | `22` | โ |
| `ssh_username` | *(Optional)* SSH username for authentication. | `username` | โ |
| `ssh_password` | *(Optional)* SSH password. Required if not using a private key. | `password` | โ |
| `ssh_pkey` | *(Optional)* Path to or content of the private key file used for SSH authentication. | `/path/to/private/key.pem` | โ |
| `conn_kwargs` | *(Optional)* Additional [MySQL connection parameters](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) in dictionary format. | `{"ssl_ca": "CARoot.pem"}` | โ |
| `use_lowercase` | *(Optional)* Whether to convert all column names to lowercase. | `true` or `false` | โ |
***
## Optional Configs
| Key | Description | Example Value |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `skip_schema_creation` | If `true`, Mage will skip running the `CREATE SCHEMA` command. Useful when the schema already exists. [See related issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, all column names will be forced to lowercase. Default is `true`. | `true` |
***
## Notes
* For cloud-hosted MySQL (e.g., Amazon RDS, Azure Database for MySQL), ensure proper firewall rules and access control.
* Use `ssh_tunnel` mode when connecting to databases hosted in private networks or VPCs.
* The `conn_kwargs` parameter is useful for enabling SSL/TLS and advanced options (e.g., `ssl_ca`, `ssl_cert`, `connect_timeout`).
* Mage automatically creates the destination table unless `skip_schema_creation` is set to `true`.
***
## Related Resources
* [MySQL Documentation](https://dev.mysql.com/doc/)
* [MySQL Connector/Python Connect Arguments](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html)
* [SSH Tunneling Explained](https://www.ssh.com/academy/ssh/tunneling)
# OpenSearch
Source: https://docs.mage.ai/data-integrations/destinations/opensearch
How to configure OpenSearch as a destination in Mage to stream pipeline data into OpenSearch indices using JSON records with support for authentication, ECS metadata, and SSL.
## Overview
Use **OpenSearch** as a destination in Mage to publish JSON-formatted records into an OpenSearch index. This is ideal for search indexing, log aggregation, and real-time analytics.
By default, Mage uses the pipelineโs `table` name to set the OpenSearch **index name**. You can also define dynamic index patterns using JSONPath expressions.
***
## Configuration Parameters
| Key | Description | Default | Required |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -------- |
| `scheme` | HTTP scheme used for connecting to OpenSearch. Use `http` or `https`. | `http` | โ |
| `host` | Hostname or IP address of the OpenSearch node or cluster. | `localhost` | โ |
| `port` | Port number for the OpenSearch HTTP endpoint. | `9200` | โ |
| `username` | *(Optional)* Username for basic authentication. | `None` | โ |
| `password` | *(Optional)* Password for basic authentication. | `None` | โ |
| `bearer_token` | *(Optional)* Bearer token for token-based authentication. | `None` | โ |
| `api_key_id` | *(Optional)* API key ID for key-based authorization. | `None` | โ |
| `api_key` | *(Optional)* API key secret for key-based authorization. | `None` | โ |
| `ssl_ca_file` | *(Optional)* Path to the SSL CA certificate file for verifying HTTPS connections. | `None` | โ |
| `index_schema_fields` | *(Optional)* JSONPath mapping used to dynamically generate the index name based on record values. | `None` | โ |
| `metadata_fields` | *(Optional)* Dictionary mapping fields (using JSONPath) to populate ECS-style metadata fields in the index request (e.g., `_id`, `@timestamp`). | `None` | โ |
***
## Notes
* Mage automatically converts each record into JSON format and indexes it into the specified OpenSearch index.
* If `index_schema_fields` is configured, Mage will use the JSONPath-extracted values from each record to generate dynamic index names.
* `metadata_fields` is useful for setting OpenSearch ECS-compatible fields like `_id`, `@timestamp`, etc., from the incoming data stream.
* Authentication can be configured using **basic auth**, **bearer tokens**, or **API key pairs**.
* For production use, we recommend enabling `https` and providing a trusted `ssl_ca_file`.
***
## Related Resources
* [OpenSearch Documentation](https://opensearch.org/docs/)
* [OpenSearch Python Client](https://opensearch-project.github.io/opensearch-py/)
* [ECS (Elastic Common Schema) Fields](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html)
* [OpenSearch Security Configuration](https://opensearch.org/docs/latest/security-plugin/)
# OracleDB
Source: https://docs.mage.ai/data-integrations/destinations/oracledb
How to configure Oracle Database (OracleDB) as a destination in Mage to write data to a specified schema using the thin or thick client.
## Overview
Use **OracleDB** as a destination in Mage to write pipeline data directly into a table within an Oracle database. Mage supports both **thin** and **thick** connection modes, allowing flexibility for local clients, instant clients, or full Oracle installations.
This destination is ideal for enterprise data integration with existing Oracle systems, including Oracle XE, Oracle Cloud Infrastructure (OCI), and on-premise Oracle servers.
***
## Configuration Parameters
You must provide the following credentials when configuring the OracleDB destination:
| Key | Description | Example Value | Required |
| ---------- | ------------------------------------------------------------------------------------------------------------------ | ---------------------- | -------- |
| `host` | Hostname or IP address of the Oracle database server. | `oracledb.example.com` | โ |
| `port` | Port on which the Oracle listener accepts client connections. Typically `1521`. | `1521` | โ |
| `service` | Oracle **service name** (not SID). The listener uses this to direct connections to the correct pluggable database. | `xepdb1` | โ |
| `user` | Username with permission to connect and write to the target schema. | `oracle_user` | โ |
| `password` | Password for the Oracle user. | `xyz123` | โ |
| `database` | Name of the Oracle database where the table will be created or updated. | `orcl` | โ |
| `mode` | Oracle client mode. Set to either `thin` (default) or `thick` depending on the installed driver. | `thin` or `thick` | โ |
***
## Optional Parameters
| Key | Description | Example Value |
| ------------ | ------------------------------------------------------------------------------ | ------------- |
| `lower_case` | If `true`, all column names will be converted to lowercase. Default is `true`. | `true` |
***
## Notes
* The `mode` parameter determines how the Oracle client connects:
* Use `thin` for lightweight connections (e.g., with the Oracle Instant Client).
* Use `thick` when a full Oracle client is installed and native libraries are required.
* Ensure the Oracle listener is correctly configured to accept service nameโbased connections.
* Use of the `service` name is preferred over SID in modern Oracle setups, especially with pluggable databases (PDBs).
***
## Related Resources
* [Oracle Net Services Overview](https://docs.oracle.com/en/database/oracle/oracle-database/19/netag/introduction-to-oracle-net-services.html)
* [Oracle Instant Client Installation Guide](https://www.oracle.com/database/technologies/instant-client.html)
* [Oracle Database Connection Strings](https://docs.oracle.com/en/database/oracle/oracle-database/19/jjdbc/data-sources-and-url-connection-strings.html)
# Destinations technical documentation
Source: https://docs.mage.ai/data-integrations/destinations/overview
A destination is a system where you want the data to be exported. A destination can be a 3rd party API, SaaS, database, data warehouse, or a data lake.
## Singer spec
Mage uses the data engineering community standard for data integrations called the
[Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
In addition,
Mage further standardizes the spec and provides common classes and methods to make implementing them easier and faster.
## Folder structure
Destinations are defined in the `mage_integrations/destinations/` directory.
All destinations should be a subclass of the `Destination` class defined in
`mage_integrations/destinations/base.py`.
## Configs
* `maximum_batch_size_mb`: Destination processes records the batches. The maximum batch size can be configured.
## Contribute
Want to learn about contributing a destination? Read on [here](/contributing/data-integrations/add-new-destination)!
# PostgreSQL
Source: https://docs.mage.ai/data-integrations/destinations/postgresql
How to configure PostgreSQL as a destination in Mage to write pipeline data into a PostgreSQL table using direct database credentials and optional schema/table controls.
***
## Overview
Use **PostgreSQL** as a destination in Mage to export structured data from your pipelines directly into a PostgreSQL table. Mage supports both self-hosted and cloud-hosted PostgreSQL instances such as **bit.io**, **Supabase**, **Render**, and **AWS RDS**.
Mage will automatically create the target table and insert pipeline output rows into it, with support for schema selection, lowercase column naming, and reserved keyword handling.
***
## Configuration Parameters
Provide the following credentials when setting up PostgreSQL as a destination:
| Key | Description | Example Value | Required |
| ---------- | ----------------------------------------------------------- | ------------------------------ | -------- |
| `database` | Name of the PostgreSQL database where data will be written. | `demo` | โ |
| `host` | Hostname or IP address of the PostgreSQL server. | [`db.bit.io`](https://bit.io/) | โ |
| `port` | Port used by the PostgreSQL service. Default is `5432`. | `5432` | โ |
| `username` | Username with access to the target database and schema. | `guest` | โ |
| `password` | Password for the PostgreSQL user. | `abc123...` | โ |
| `schema` | Name of the schema where the table will be created. | `public` | โ |
| `table` | Name of the table Mage will write records into. | `dim_users_v1` | โ |
***
## Optional Parameters
| Key | Description | Example Value |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `skip_schema_creation` | If `true`, Mage will skip running `CREATE SCHEMA`. Use this if the schema already exists. [See issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will convert all column names to lowercase. Default is `true`. | `true` |
| `allow_reserved_words` | If `true`, Mage will allow column names that are PostgreSQL reserved words. Default is `false`. | `true` or `false` |
***
## Notes
* PostgreSQL is a widely supported open-source relational databaseโideal for analytics, app backends, and transformations.
* Ensure the user has `CREATE TABLE`, `INSERT`, and optionally `CREATE SCHEMA` privileges.
* Mage supports exporting data to **existing tables** or **creating new tables** based on the pipeline output schema.
* If your database is hosted on platforms like Heroku, AWS RDS, or bit.io, make sure the network settings allow inbound connections from Mage.
***
## Related Resources
* [PostgreSQL Documentation](https://www.postgresql.org/docs/)
* [bit.io Free PostgreSQL Hosting](https://bit.io/)
* [PostgreSQL Reserved Keywords](https://www.postgresql.org/docs/current/sql-keywords-appendix.html)
* [SQLAlchemy PostgreSQL Dialect](https://docs.sqlalchemy.org/en/20/dialects/postgresql.html)
# Redshift
Source: https://docs.mage.ai/data-integrations/destinations/redshift
How to configure Amazon Redshift as a destination in Mage to write data into Redshift tables using user credentials or IAM-based authentication.
***
## Overview
Use **Amazon Redshift** as a destination in Mage to export pipeline data into a highly scalable, cloud-based data warehouse. Mage supports both **standard username/password authentication** and **IAM database authentication** using AWS access keys.
Mage will automatically create and write to the specified table, with support for optional schema management and merge-based updates.
***
## Configuration Parameters
You must provide the following credentials when configuring Redshift as a destination:
| Key | Description | Example Value | Required |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -------------------- |
| `database` | Name of the Redshift database where data will be written. | `demo` | โ |
| `host` | Hostname of the Redshift cluster endpoint. | `mage-prod.3.us-west-2.redshift.amazonaws.com` | โ |
| `port` | Redshift port (default is `5439`). | `5439` | โ |
| `user` | Redshift username with access to the database and schema. | `awsuser` | โ (if not using IAM) |
| `password` | Password for the Redshift user. | `abc123...` | โ (if not using IAM) |
| `schema` | Schema within the database where the table will be created or written to. | `public` | โ |
| `table` | Name of the destination table to store data from your pipeline. | `dim_users_v1` | โ |
| `region` | AWS region where the Redshift cluster is hosted. | `us-west-2` | โ |
| `use_merge_load` | If `true`, Mage will attempt to merge incoming data with existing records (UPSERT-like behavior). | `false` *(default)* | โ |
| `use_s3_copy` | If `true`, Mage will upload data to S3 and load into Redshift using the COPY command (Pro feature). | `false` *(default)* | โ |
| `type_mapping` | Custom mapping to override default column type conversions. Useful for specifying custom Redshift data types like SUPER for JSON data (Pro feature). | see below | โ (Pro) |
> **Note**: `use_s3_copy` and `type_mapping` are Mage **Pro-only** features. `use_s3_copy` enables faster bulk loads via S3 and supports IAM roles or access key authentication.
***
## IAM-Based Authentication (Alternative to User/Password)
If you prefer to use **IAM credentials** instead of traditional authentication, you can use the following fields:
| Key | Description | Example Value | Required (if using IAM) |
| -------------------- | ------------------------------------------------------------------------ | ------------- | ----------------------- |
| `access_key_id` | AWS access key ID for an IAM user or role with Redshift database access. | `AKIA...` | โ |
| `secret_access_key` | AWS secret access key corresponding to the above access key ID. | `xyz123...` | โ |
| `cluster_identifier` | Identifier of the Amazon Redshift cluster. | `mage-prod` | โ |
| `db_user` | IAM-authenticated Redshift user. | `admin` | โ |
***
## S3 Configuration (Required if `use_s3_copy` is true)
| Key | Description | Required | Sample Value |
| ----------------------- | ------------------------------------------------------------------- | ------------------------- | ----------------------------------------------- |
| `aws_access_key_id` | AWS access key for uploading data to S3. Ignored if using IAM role. | โ (unless using IAM role) | `AKIA...` |
| `aws_secret_access_key` | AWS secret key for uploading data to S3. Ignored if using IAM role. | โ (unless using IAM role) | `xyz...` |
| `aws_region` | Region where the S3 bucket is located. | โ | `us-west-2` |
| `bucket` | S3 bucket to temporarily stage Parquet files for Redshift COPY. | โ | `mage-feature-sets` |
| `redshift_iam_role` | IAM role ARN with S3 read access attached to your Redshift cluster. | Optional | `arn:aws:iam::123456789012:role/MyRedshiftRole` |
Configure these inside a nested `s3_creds` config object:
```yaml theme={"system"}
use_s3_copy: true
s3_creds:
aws_access_key_id: AKIA...
aws_secret_access_key: xyz...
aws_region: us-west-2
bucket: mage-feature-sets
redshift_iam_role: arn:aws:iam::123456789012:role/MyRedshiftRole
```
***
## Type Mapping
The `type_mapping` configuration allows you to override default column type conversions. This is useful for advanced cases like storing JSON in Redshiftโs `SUPER` type or adjusting column widths.
### Example Configuration
**YAML Format:**
```yaml theme={"system"}
type_mapping:
object: SUPER
array: SUPER
string: VARCHAR(1000)
```
**Complete YAML Example:**
```yaml theme={"system"}
database: demo
host: mage-prod.3.us-west-2.redshift.amazonaws.com
password: abc123...
port: 5439
region: us-west-2
schema: public
user: awsuser
use_merge_load: false
use_s3_copy: false
lower_case: true
type_mapping:
object: SUPER
array: SUPER
string: VARCHAR(1000)
datetime: TIMESTAMPTZ
number: BIGINT
```
### Supported Overrides
* **`object`** โ Redshift type for JSON objects (e.g., `SUPER`)
* **`array`** โ Redshift type for JSON arrays (e.g., `SUPER`)
* **`string`** โ Custom VARCHAR or other string type
* **`datetime`** โ Custom timestamp type (e.g., `TIMESTAMPTZ`)
* **`number`** โ Custom numeric type (e.g., `BIGINT`)
> When using `SUPER`, Mage automatically applies `JSON_PARSE()` for correct Redshift insertion.
***
## Optional Parameters
| Key | Description | Example Value |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `skip_schema_creation` | If `true`, Mage will skip running the `CREATE SCHEMA` command. Useful if the schema already exists. [See issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will automatically convert column names to lowercase. Default is `true`. | `true` |
***
## Grant Permissions
To allow Mage to create schemas and insert data, grant the following permissions to your Redshift user or IAM role:
```sql theme={"system"}
GRANT CREATE ON DATABASE your_database TO your_user;
GRANT CREATE ON ALL TABLES IN SCHEMA your_schema TO your_user;
```
Refer to the [Redshift GRANT command documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) for details.
***
## Related Resources
* [Amazon Redshift Documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html)
* [Redshift IAM Database Authentication](https://docs.aws.amazon.com/redshift/latest/mgmt/guid-iam-db-authentication.html)
* [Redshift SQL GRANT Syntax](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html)
* [Redshift JDBC and ODBC Drivers](https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html)
# Salesforce
Source: https://docs.mage.ai/data-integrations/destinations/salesforce
How to configure Salesforce as a destination in Mage to write records using OAuth or username/password authentication with support for insert, update, upsert, and bulk actions.
***
## Overview
Use **Salesforce** as a destination in Mage to export records into standard or custom Salesforce objects. Mage supports both **OAuth 2.0** and **username/password/token** authentication methods, and enables `insert`, `update`, `upsert`, `delete`, and `hard_delete` actions.
You can also configure behavior for bulk failures, define external ID fields for upserts, and optionally override the source-to-object mapping using `table_name`.
***
## Configuration Parameters
| Key | Description | Required |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- | -------- |
| `client_id` | OAuth client ID from your Salesforce connected app. | Optional |
| `client_secret` | OAuth client secret from your connected app. | Optional |
| `refresh_token` | OAuth refresh token. Used to obtain new access tokens. | Optional |
| `username` | Username used for Salesforce **username/password** authentication. | Optional |
| `password` | Corresponding Salesforce password. | Optional |
| `security_token` | Salesforce security token (needed with password-based login). | Optional |
| `domain` | Salesforce domain to authenticate against. Use `login` (default), `test` (sandbox), or a custom My Domain. | โ |
| `action` | Default operation to perform on incoming records: `insert`, `update`, `upsert`, `delete`, or `hard_delete`. | โ |
| `external_id_name` | External ID field for `upsert` operations. Default is `"Id"`. | Optional |
| `allow_failures` | If `true`, failed records will be skipped and processing will continue. Default is `false`. | Optional |
| `table_name` | Override the source stream name with a specific Salesforce object name (e.g., `Account`, `Contact`, `Opportunity`). | Optional |
***
## Authentication Options
You can authenticate to Salesforce using either of the following methods:
### Option 1: OAuth 2.0
* Set `client_id`, `client_secret`, and `refresh_token`.
* Recommended for connected apps and secure access with refreshable tokens.
* [How to obtain credentials โ](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/salesforce/README.md#how-to-get-your-client_id-client_secret-and-refresh_token)
### Option 2: Username / Password
* Set `username`, `password`, and `security_token`.
* Recommended for sandbox/testing environments.
* Ensure that IP restrictions are disabled or add your Mage IP to the trusted IP list.
***
## Limitations
To map a **non-default source stream** to a specific Salesforce object, you must explicitly set the `table_name` parameter in your configuration.
* The value of `table_name` **must** match the API name of a Salesforce object (e.g., `Account`, `Lead`, `Opportunity`, or custom objects like `MyObject__c`).
***
## Allow Failures Behavior
When `allow_failures` is set to `true`:
* Records that are **eligible** (i.e., match the Salesforce object schema and are valid for the selected action) but **fail to commit** will be skipped.
* The pipeline will continue processing other records.
When `allow_failures` is set to `false`:
* Any failed record will raise an error and halt the pipeline.
***
## Related Resources
* [Salesforce API Documentation](https://developer.salesforce.com/docs)
* [Salesforce Connected App Setup](https://help.salesforce.com/s/articleView?id=sf.connected_app_create.htm)
* [Salesforce Bulk API](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/)
* [Salesforce Object Reference](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_concepts.htm)
# Snowflake
Source: https://docs.mage.ai/data-integrations/destinations/snowflake
How to configure Snowflake as a destination in Mage to write pipeline data to Snowflake using batch loads, optional key-pair auth, and custom warehouse settings.
***
## Overview
Use **Snowflake** as a destination in Mage to load structured data into your cloud data warehouse. Mage supports both **row-by-row insertion** and **batch upload** modes, and integrates with Snowflake using **username/password** or **key-pair authentication**.
This destination is ideal for exporting transformed pipeline data for analytics, BI tools, or data lakehouse modeling.
***
## Required Configuration
Provide the following credentials when configuring Snowflake as a destination:
| Key | Description | Example Value | Required |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------- | -------- |
| `account` | Your Snowflake [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). | `abc1234.us-east-1` | โ |
| `database` | Name of the Snowflake database to write to. | `DEMO_DB` | โ |
| `schema` | Schema within the database where the table will be created. | `PUBLIC` | โ |
| `table` | Target table that Mage will create or insert into. | `dim_users_v1` | โ |
| `username` | Snowflake user with `INSERT` and `CREATE TABLE` permissions. | `guest` | โ |
| `warehouse` | Virtual warehouse used to execute queries and ingest data. | `COMPUTE_WH` | โ |
| `use_batch_load` | If `true`, enables batch loading instead of row-by-row inserts. Recommended for performance. | `true` *(default)* | โ |
| `disable_double_quotes` | If `true`, column and table names will not be wrapped in double quotes. Disabling may reduce case-sensitivity issues. | `false` *(default)* | โ |
***
## Optional Connection Parameters
| Key | Description | Example Value |
| ---------------------- | ------------------------------------------------------------------------------------- | ----------------- |
| `password` | Password for the Snowflake user. | `abc123...` |
| `private_key_file` | Path to the private key file for key-pair authentication (requires version โฅ 0.9.76). | `/path/to/key.p8` |
| `private_key_file_pwd` | Passphrase for the private key file. | `abc123...` |
| `role` | Snowflake role to use for the session. | `ANALYST_ROLE` |
๐ [Enable key-pair authentication โ](https://docs.snowflake.com/en/user-guide/key-pair-auth)
***
## Optional Mage Settings
| Key | Description | Example Value |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `skip_schema_creation` | If `true`, Mage will skip running the `CREATE SCHEMA` command. Useful if the schema already exists. [See issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will convert all column names to lowercase. Default is `true`. | `true` |
***
## Notes
* Set `use_batch_load: true` for large datasets to improve performance and reduce API calls.
* When using **key-pair authentication**, ensure the private key is generated and stored securely.
* If `disable_double_quotes` is `false`, column names will retain their case. Otherwise, names will be treated as case-insensitive.
* Make sure your user has permission to create tables and write to the specified schema in the selected database and warehouse.
***
## Related Resources
* [Snowflake Account Identifier Format](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html)
* [Snowflake Key-Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth)
* [Snowflake Warehouses Overview](https://docs.snowflake.com/en/user-guide/warehouses.html)
* [Snowflake SQL Reference](https://docs.snowflake.com/en/sql-reference/)
# Teradata
Source: https://docs.mage.ai/data-integrations/destinations/teradata
How to configure Teradata as a destination in Mage to write pipeline data to a Teradata database using standard host, port, and user credentials.
## Overview
Use **Teradata** as a destination in Mage to write structured pipeline data into a Teradata database table. This is ideal for analytics, reporting, or operational workloads in enterprise environments using Teradataโs scalable architecture.
Mage connects using direct credentials and supports exporting to tables in any schema the user has access to.
***
## Configuration Parameters
Provide the following credentials when configuring Teradata as a **destination**:
| Key | Description | Example Value | Required |
| ---------- | ------------------------------------------------------------------------------- | ------------------------------ | -------- |
| `database` | Name of the Teradata database where the table will be created or written to. | `demo` | โ |
| `host` | Hostname or IP address of the Teradata server. | `test.clearscape.teradata.com` | โ |
| `port` | Port used by the Teradata database (commonly `1025`, depending on environment). | `1025` | โ |
| `username` | Teradata username with write access to the specified database and schema. | `demo_user` | โ |
| `password` | Password for the Teradata user. | `abc123...` | โ |
***
## Notes
* Mage will automatically create the target table if it doesn't exist, unless configured otherwise.
* Ensure the user has `CREATE TABLE`, `INSERT`, and `SELECT` privileges in the target schema.
* Teradata uses ANSI SQL but has some syntax differences (e.g., date handling, identity columns) to be aware of.
* This integration is well-suited for batch ingestion pipelines and bulk exports into data marts.
***
## Related Resources
* [Teradata Documentation](https://docs.teradata.com/)
* [Teradata SQL Reference](https://docs.teradata.com/r/Teradata-SQL)
* [Teradata Vantage Overview](https://www.teradata.com/Products/Vantage)
* [Clearscape Analytics](https://www.teradata.com/Products/Clearscape-Analytics)
# Trino
Source: https://docs.mage.ai/data-integrations/destinations/trino
How to configure Trino as a destination in Mage to write pipeline data to Trino-compatible catalogs using supported connectors such as PostgreSQL, Iceberg, Delta Lake, and more.
***
## Overview
Use **Trino** as a destination in Mage to write data to a variety of underlying data sources including **PostgreSQL**, **Delta Lake**, **Iceberg**, **S3**, and moreโvia Trinoโs supported catalog and connector system.
Trino enables federated query and write access across databases, data warehouses, and data lakes, all through a unified SQL engine.
***
## Configuration Parameters
You must provide the following parameters when configuring Trino as a destination:
| Key | Description | Example Value | Required |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------- | -------- |
| `catalog` | Trino catalog used to access the target data source (e.g., PostgreSQL, Iceberg, Delta Lake). | `my_prod_pg` | โ |
| `connector` | Trino connector name. Determines how Trino interacts with the catalog's backend. | `postgresql` | โ |
| `host` | Hostname or IP address of the Trino coordinator. | `127.0.0.1` | โ |
| `port` | Port where the Trino coordinator is running. Default is `8080`. | `8080` | โ |
| `username` | Username for authenticating with the Trino coordinator. | `admin` | โ |
| `password` | *(Optional)* Password for Trino authentication. | `abc123...` | โ |
| `schema` | Schema name where the target table will be created or written to. | `public` | โ |
| `table` | Name of the destination table to write data into. | `dim_users_v1` | โ |
| `query_max_length` | Maximum character length allowed for Trino SQL query payloads. | `1000000` | โ |
| `ssl` | Whether to disable SSL certificate verification. Set `false` to disable. | `false` | โ |
| `location` | (Delta Lake only) URI location of the target table or storage bucket. | `s3://my-delta-bucket/` | โ |
| `ignore_location_for_temp_tables` | Prevents Trino from setting `WITH LOCATION` for temp tables. Recommended when using Delta Lake with Glue metastore. | `false` *(default)* | โ |
***
## Notes on Delta Lake + Glue Integration
When using the **Delta Lake connector** with a **Glue metastore**, tables created with the `WITH LOCATION` property will **not delete underlying data** when dropped.
To avoid residual files when Mage creates temp tables, set:
```yaml theme={"system"}
ignore_location_for_temp_tables: true
```
# Overview
Source: https://docs.mage.ai/data-integrations/overview
Data integration is the process of synchronizing data between two systems.
## Requirements
Data integrations are only supported when running Mage using [Docker](/getting-started/setup#using-docker).
## Use cases
### Syncing external data โ internal systems
If you have data in 3rd party systems (e.g other SaaS tools) and you want that
data available in your database, data warehouse, or data lake, you can use Mage
to sync data between those systems.
### Syncing internal data โ external systems (reverse)
Your internal database or data warehouse may contain application data generated
by your users or customers. This data can be valuable if used in conjunction
with 3rd party SaaS tools.
You can use Mage to sync data from your internal systems, and have that data
appear in various 3rd party SaaS tools.
***
## How to build a data integration pipeline
[Documentation here](/guides/data-integration-pipeline)
***
## Sources
A source is a system that you want to load data from and synchronize it into
another system. A source can be a 3rd party API, SaaS, database, data warehouse,
or a data lake.
Read more about Sources:
* [Technical documentation](/data-integrations/sources/overview)
* [Add a new source](/contributing/data-integrations/add-new-source)
### Available sources
* [Airtable](/data-integrations/sources/airtable)
* [Amazon S3](/data-integrations/sources/amazon_s3)
* [Amplitude](/data-integrations/sources/amplitude)
* [API](/data-integrations/sources/api)
* [Azure Blob Storage](/data-integrations/sources/azure_blob_storage)
* [BigQuery](/data-integrations/sources/bigquery)
* [Chargebee](/data-integrations/sources/chargebee)
* [Commercetools](/data-integrations/sources/commercetools)
* [Couchbase](/data-integrations/sources/couchbase)
* [Datadog](/data-integrations/sources/datadog)
* [Dremio](/data-integrations/sources/dremio)
* [DynamoDB](/data-integrations/sources/dynamodb)
* [Facebook Ads](/data-integrations/sources/facebook_ads)
* [Freshdesk](/data-integrations/sources/freshdesk)
* [Front](/data-integrations/sources/front)
* [Google Ads](/data-integrations/sources/google_ads)
* [Google Analytics](/data-integrations/sources/google_analytics)
* [Google Cloud Storage](/data-integrations/sources/google_cloud_storage)
* [Google Search Console](/data-integrations/sources/google_search_console)
* [Google Sheets](/data-integrations/sources/google_sheets)
* [HubSpot](/data-integrations/sources/hubspot)
* [Intercom](/data-integrations/sources/intercom)
* [Knowi](/data-integrations/sources/knowi)
* [LinkedIn Ads](/data-integrations/sources/linkedin_ads)
* [Microsoft SQL Server](/data-integrations/sources/mssql)
* [Mode](/data-integrations/sources/mode)
* [Monday](/data-integrations/sources/monday)
* [MongoDB](/data-integrations/sources/mongodb)
* [MySQL](/data-integrations/sources/mysql)
* [OracleDB](/data-integrations/sources/oracledb)
* [Outreach](/data-integrations/sources/outreach)
* [Paystack](/data-integrations/sources/paystack)
* [Pipedrive](/data-integrations/sources/pipedrive)
* [PostgreSQL](/data-integrations/sources/postgresql)
* [Postmark](/data-integrations/sources/postmark)
* [PowerBI](/data-integrations/sources/powerbi)
* [Redshift (Amazon Web Services)](/data-integrations/sources/redshift)
* [Salesforce](/data-integrations/sources/salesforce)
* [SFTP](/data-integrations/sources/sftp)
* [Snowflake](/data-integrations/sources/snowflake)
* [Stripe](/data-integrations/sources/stripe)
* [Tableau](/data-integrations/sources/tableau)
* [Twitter Ads](/data-integrations/sources/twitter_ads)
* [Zendesk](/data-integrations/sources/zendesk)
* *Shopify (coming soon)*
### Sources available in Mage Pro
* [Facebook Pages](/data-integrations/sources/facebook_pages)
* [Google Campaign Manager 360](/data-integrations/sources/google_campaign_manager_360)
* [Google Display & Video 360](/data-integrations/sources/google_display_video_360)
* [Instagram](/data-integrations/sources/instagram)
* [Microsoft Ads](/data-integrations/sources/microsoft_ads)
* [QuickBooks](/data-integrations/sources/quickbooks)
* [Salesforce Marketing Cloud](/data-integrations/sources/salesforce_marketing_cloud)
* [TikTok Ads](/data-integrations/sources/tiktok_ads)
* [Instagram](/data-integrations/sources/instagram)
* [QuickBooks](/data-integrations/sources/quickbooks)
* [Xero](/data-integrations/sources/xero)
* [Youtube Analytics](/data-integrations/sources/youtube_analytics)
Mage adheres to the data engineering standard for data integrations called the
[Singer Spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
All sources (aka taps) that has been developed by the open-source community can
be adapted to use in Mage. Here is a
[short list of available ones](https://www.singer.io/#taps).
Donโt see a source you need?
Check out the next section on Add a new source or tell us on slack and we
can help you build it.
[Join us on Slack](https://www.mage.ai/chat)
***
## Destinations
A destination is a system that you want to export data into. A destination can
be a 3rd party API, SaaS, database, data warehouse, or a data lake.
Read more about destinations:
* [Technical documentation](/data-integrations/destinations/overview)
* [Add a new destination](/contributing/data-integrations/add-new-destination)
### Available destinations
* [Amazon Web Services (AWS) Redshift](/data-integrations/destinations/redshift)
* [Amazon Web Services (AWS) S3](/data-integrations/destinations/amazon_s3)
* [BigQuery](/data-integrations/destinations/bigquery)
* [Clickhouse](/data-integrations/destinations/clickhouse)
* [Delta Lake (S3)](/data-integrations/destinations/delta_lake_s3)
* [Delta Lake (Azure)](/data-integrations/destinations/delta_lake_azure)
* [Elasticsearch](/data-integrations/destinations/elasticsearch)
* [Google Cloud Storage](/data-integrations/destinations/google_cloud_storage)
* [Kafka](/data-integrations/destinations/kafka)
* [Microsoft SQL Server](/data-integrations/destinations/mssql)
* [MongoDB](/data-integrations/destinations/mongodb)
* [MySQL](/data-integrations/destinations/mysql)
* [Opensearch](/data-integrations/destinations/opensearch)
* [OracleDB](/data-integrations/destinations/oracledb)
* [PostgreSQL](/data-integrations/destinations/postgresql)
* [Salesforce](/data-integrations/destinations/salesforce)
* [Snowflake](/data-integrations/destinations/snowflake)
* [Trino](/data-integrations/destinations/trino)
* *Azure Blob Storage (coming soon)*
* *HubSpot (coming soon)*
### Destinations available in Mage Pro
* [Databricks](/data-integrations/destinations/databricks)
* [DuckDB](/data-integrations/destinations/duckdb)
* [Iceberg S3](/data-integrations/destinations/iceberg_s3)
Mage adheres to the data engineering standard for data integrations called the
[Singer Spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
All destinations (aka targets) that has been developed by the open-source
community can be adapted to use in Mage. Here is a
[short list of available ones](https://www.singer.io/#targets).
> Donโt see a destination you need?
>
> Check out the next section on Add a new destination or tell us on
> [Slack](https://www.mage.ai/chat). and we can help you build it.
[Join us on Slack](https://www.mage.ai/chat)
# Airtable
Source: https://docs.mage.ai/data-integrations/sources/airtable
## Configuration
To set up the Airtable source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ------------ | ---------------------------------------------------------------------------- | ------------ | -------- |
| `token` | Airtable personal access token with access to your desired tables. | `pat...` | โ |
| `base_id` | The ID of the Airtable base (application) you want to access. | `app...` | โ |
| `table_name` | The name or ID of the table to sync. If not specified, all tables will sync. | `tbl...` | |
> **Find your Personal Access Token:**\
> See Airtableโs [guide](https://airtable.com/developers/web/guides/personal-access-tokens) for generating a personal access token.
## Required Scopes
Ensure your Airtable personal access token has the following scopes:
* `data.records:read`
* `schema.bases:read`
# Amazon S3
Source: https://docs.mage.ai/data-integrations/sources/amazon_s3
## Configuration
To set up the Amazon S3 source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | -------- |
| `aws_access_key_id` | AWS access key ID. | `abc123` | โ |
| `aws_secret_access_key` | AWS secret access key. | `xyz456` | โ |
| `aws_region` | Region where the S3 bucket is located. | `us-west-2` | |
| `role_arn` | (Optional) ARN of the IAM role to assume for accessing the S3 bucket. | `arn:aws:iam::111111:role/example-role` | |
| `bucket` | Name of the AWS S3 bucket to read data from. | `user_generated_content` | โ |
| `file_type` | Type of files in the bucket. Supported values: `parquet`, `csv`, `infer`. If `infer`, the type is detected based on file extension. | `infer` | |
| `prefix` | Path within the bucket where files are located. Do not include `s3://`, the bucket name, or the table name in this path. | `users/ds/20221225` | |
| `search_pattern` | Regular expression to match files within the prefix. | `test_table\\/.*\\.csv` | |
| `single_stream_in_prefix` | If `true`, all files in the prefix are treated as a single stream. Defaults to `false`. | `false` | |
| `table_configs` | A list of table configurations to configure multiple streams. Each table config must include `prefix`, `search_pattern`, and `table_name`. | `[{"prefix": "users/", "search_pattern": "test_table\\/.*\\.csv", "table_name": "test_table"}]` | |
| `aws_endpoint` | (Optional) Custom S3-compatible endpoint (e.g., for MinIO). | `https://play.min.io` | |
# Amplitude
Source: https://docs.mage.ai/data-integrations/sources/amplitude
## Configuration
To set up the Amplitude source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ------------ | -------------------------------------- | ------------ | -------- |
| `api_key` | Amplitude project's unique API key. | `73bb...` | โ |
| `secret_key` | Amplitude project's unique secret key. | `ABC1...` | โ |
> **Find your API credentials:**\
> See Amplitudeโs [documentation](https://www.docs.developers.amplitude.com/analytics/find-api-credentials/).
### Optional Configuration
| Key | Description | Sample Value |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `host` | Amplitude host URL. Standard server: `https://amplitude.com`. EU Residency server: `https://analytics.eu.amplitude.com`. Defaults to `https://amplitude.com`. | `https://analytics.eu.amplitude.com` |
## Query Behavior
Loading data from Amplitude requires a start date and an end date.
Mage automatically passes the start date and end date based on the scheduled trigger you create.\
For example, if your pipeline runs daily, the start date will be 1 day ago, and the end date will be today.
If you prefer to hardcode and customize these dates, follow the [instructions to override runtime variables](https://docs.mage.ai/production/configuring-production-settings/runtime-variable).
Add the following keys and values to your pipelineโs runtime variables:
| Key | Description | Sample Value |
| ------------- | ------------------------------ | ------------ |
| `_start_date` | Date to start fetching events. | `2022-10-01` |
| `_end_date` | Date to stop fetching events. | `2022-10-03` |
## Schema
The following resources are synced:
* [Events](https://www.docs.developers.amplitude.com/analytics/apis/export-api/#response-schema)
# API Source Configuration
Source: https://docs.mage.ai/data-integrations/sources/api
Learn how to configure API sources in Mage to extract data from RESTful and file-based endpoints. This guide covers authentication, pagination, response parsing, and advanced options available in Mage Pro.
Mage supports integrating with any RESTful or file-based API. This allows you to extract data from JSON, CSV, TSV, XLSX responses, with support for pagination, authentication, and schema inference.
๐ Sample public APIs for testing: [https://apipheny.io/free-api/](https://apipheny.io/free-api/)
## Basic Features (OSS)
### Configuration Fields
You must enter the following credentials when configuring this source:
| Key | Description | Required or optional |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| `url` | API URL | Required |
| `method` | `GET` or `POST` | Optional: `GET` is default |
| `query` | URL query parameters. | Optional |
| `payload` | When method is `POST`, this payload is used in the request body. | Optional |
| `headers` | Request headers. | Optional |
| `response_parser` | Parse the API response using dot notation. The final result must be an array. | Optional |
| `columns` | If the final data returned from the API or from the `response_parser` is not a JSON object (e.g. itโs an array of strings or an array of array of strings), then you must define the columns. | Required (conditionally) |
| `separator` | If the file extension is TSV, XLSX (or CSV with specific separator), then you can define it | Optional: ',' is default |
| `has_header` | If the file extension is TSV, XLSX or CSV, and contains a header, then you can define it as True | Optional: False is default |
***
## Mage Pro Features
The following features are available only in [**Mage Pro**](https://cloud.mage.ai) and are designed for advanced API integrations at scale:
### Advanced Configuration (Mage Pro only)
These configuration parameters give you greater control over how data is fetched, parsed, and validated from APIsโespecially useful for large files, custom encodings, or complex schemas.
| Key | Description | Default |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `encoding` | Specifies the character encoding of the API response, such as `"utf-8"` or `"iso-8859-1"`. Use this to avoid decoding issues with special characters. | `"utf-8"` |
| `verify` | Whether to verify SSL certificates for HTTPS requests. Disabling this is useful in internal or staging environments with self-signed certs. | `True` |
| `schema_discovery_sample_rows` | The number of rows to sample when inferring the schema from API responses or files. A higher value improves schema accuracy for inconsistent data. | `100` |
| `force_all_columns_type` | Forces all columns to be interpreted as a single type: `"string"`, `"integer"`, or `"number"`. Useful for normalizing schema across variable data sources. | *None* |
| `infer_schema_when_syncing` | Enables schema inference at the time of sync (rather than preview), which is ideal for streaming or large file responses. | `False` |
These options enhance your ability to handle diverse data formats and edge cases in production environments, giving you full control over data ingestion quality and reliability.
### Multi-Stream Configuration (Mage Pro only)
Mage Pro supports defining multiple independent API configs using the `streams` key. Each key in the `streams` map represents a separate stream and can define its own URL, headers, parser, and pagination settings.
```yaml theme={"system"}
streams:
users:
url: https://api.example.com/users
headers:
x-api-key: user-key
response_parser: data.users
orders:
url: https://api.example.com/orders
headers:
x-api-key: order-key
response_parser: data.orders
```
Each stream entry can independently use any configuration field supported in the single-stream format.
### Pagination Configuration
To sync paginated API responses, the `pagination_config` config block supports two types:
* **`"cursor"`** โ follows a token provided in the API response (e.g., `next_cursor`)
* **`"page"`** โ iterates using a page number or offset (e.g., `page=1`, `page=2`, โฆ)
***
#### Cursor-Based Pagination
Use when the API returns a **next cursor/token** in the response body.
| Key | Description | Required |
| ---------------------- | --------------------------------------------------------------------------------------------------- | -------- |
| `type` | Must be `"cursor"` | Yes |
| `cursor_param` | Name of the query parameter for the cursor (single cursor only) | Yes\* |
| `cursor_params` | List of query parameter names used for multiple cursors | Yes\* |
| `initial_cursor_value` | Initial value(s) to start pagination. Can be a string, list, or dictionary. | No |
| `next_cursor_path` | Dot path(s) in response to extract next cursor. String for single, dictionary for multiple cursors. | Yes |
> \*Either `cursor_param` or `cursor_params` must be provided.
> If `cursor_params` is used, `next_cursor_path` must be a dictionary mapping each param to its response path.
**Example: Single cursor**
```yaml theme={"system"}
pagination_config:
type: cursor
cursor_param: next
initial_cursor_value: null
next_cursor_path: meta.next_cursor
```
**Example: Multiple cursors**
```yaml theme={"system"}
pagination_config:
type: cursor
cursor_params:
- start
- end
initial_cursor_value:
start: null
end: null
next_cursor_path:
start: meta.start_cursor
end: meta.end_cursor
```
#### Page-Based Pagination
Use when the API accepts **page numbers** or **offsets** and optionally supports a `limit` parameter.
| Key | Description | Required |
| ------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------ |
| `type` | `"page"` | โ |
| `param` | Name of the query parameter that controls page or offset (e.g., `"page"`, `"offset"`) | โ |
| `initial_value` | Starting value for the page or offset | โ |
| `increment_by` | Value to increase per request (e.g., `1` for pages, `100` for offset) | โ |
| `limit_param` | Name of the limit parameter (e.g., `"limit"`, `"per_page"`) | Optional |
| `limit_value` | Limit value per page | Optional |
| `stop_condition.type` | When to stop: `"empty"` (no results) or `"max_value"` (based on total pages) | โ |
| `stop_condition.max_value_path` | (If using `max_value`) Dot-notation path to total pages or count (e.g., `meta.totalPages`) | Required for `max_value` |
***
**Example: Page + Total Pages (stop when last page reached)**
```json theme={"system"}
pagination_config:
type: page
param: page
initial_value: 1
increment_by: 1
stop_condition:
type: max_value
max_value_path: total_pages
```
**Example: Offset + Limit (stop when response is empty)**
```json theme={"system"}
pagination_config:
type: page
param: offset
initial_value: 0
increment_by: 100
limit_param: limit
limit_value: 100
stop_condition:
type: empty
```
## Other Example Configs
### Example GET API 1
```json theme={"system"}
url: https://api.plos.org/search
query:
q: "title:DNA"
headers:
Content-Type: application/json
response_parser: response.docs[0].author_display
columns:
- author_display
```
The response from the above endpoint is:
```json theme={"system"}
{
"response": {
"numFound": 5669,
"start": 0,
"maxScore": 6.7217336,
"docs": [
{
"id": "10.1371/journal.pone.0000290",
"journal": "PLoS ONE",
"eissn": "1932-6203",
"publication_date": "2007-03-14T00:00:00Z",
"article_type": "Research Article",
"author_display": [
"Rayna I. Kraeva",
"Dragomir B. Krastev",
"Assen Roguev",
"Anna Ivanova",
"Marina N. Nedelcheva-Veleva",
"Stoyno S. Stoynov"
],
"abstract": [
"Nucleic acids, due to their structural and chemical properties, can form double-stranded secondary structures that assist the transfer of genetic information and can modulate gene expression. ",
"However, the nucleotide sequence alone is insufficient in explaining phenomena like intron-exon recognition during RNA processing. This raises the question whether nucleic acids are endowed with other attributes that can contribute to their biological functions. ",
"In this work, we present a calculation of thermodynamic stability of DNA/DNA and mRNA/DNA duplexes across the genomes of four species in the genus Saccharomyces by nearest-neighbor method. ",
"The results show that coding regions are more thermodynamically stable than introns, 3โฒ-untranslated regions and intergenic sequences. Furthermore, open reading frames have more stable sense mRNA/DNA duplexes than the potential antisense duplexes, a property that can aid gene discovery. ",
"The lower stability of the DNA/DNA and mRNA/DNA duplexes of 3โฒ-untranslated regions and the higher stability of genes correlates with increased mRNA level. These results suggest that the thermodynamic stability of DNA/DNA and mRNA/DNA duplexes affects mRNA transcription."
],
"title_display": "Stability of mRNA/DNA and DNA/DNA Duplexes Affects mRNA Transcription",
"score": 6.7217336
}
]
}
}
```
However, with the `response_parser` value of `response.docs[0].author_display`,
the data that is extracted from the APIโs response is:
```json theme={"system"}
[
"Rayna I. Kraeva",
"Dragomir B. Krastev",
"Assen Roguev",
"Anna Ivanova",
"Marina N. Nedelcheva-Veleva",
"Stoyno S. Stoynov"
]
```
Since each item in the final data is not a JSON object, the `columns` configuration value is required.
The final data is converted into a JSON object before being outputted to its destination:
```json theme={"system"}
[
{
"author_display": "Rayna I. Kraeva"
},
{
"author_display": "Dragomir B. Krastev"
},
{
"author_display": "Assen Roguev"
},
{
"author_display": "Anna Ivanova"
},
{
"author_display": "Marina N. Nedelcheva-Veleva"
},
{
"author_display": "Stoyno S. Stoynov"
}
]
```
### Example GET API 2
```json theme={"system"}
url: https://api.coingecko.com/api/v3/coins/markets
query:
vs_currency: usd
```
The response from the above endpoint is:
```json theme={"system"}
[
{
"id": "bitcoin",
"symbol": "btc",
"name": "Bitcoin",
"image": "https://assets.coingecko.com/coins/images/1/large/bitcoin.png?1547033579",
"current_price": 17154.23,
"market_cap": 329331527834,
"market_cap_rank": 1,
"fully_diluted_valuation": 359638803581,
"total_volume": 14339246353,
"high_24h": 17227.56,
"low_24h": 17107.75,
"price_change_24h": 18.57,
"price_change_percentage_24h": 0.10839,
"market_cap_change_24h": -425260465.15130615,
"market_cap_change_percentage_24h": -0.12896,
"circulating_supply": 19230300.0,
"total_supply": 21000000.0,
"max_supply": 21000000.0,
"ath": 69045,
"ath_change_percentage": -75.16662,
"ath_date": "2021-11-10T14:24:11.849Z",
"atl": 67.81,
"atl_change_percentage": 25185.95387,
"atl_date": "2013-07-06T00:00:00.000Z",
"roi": null,
"last_updated": "2022-12-11T00:32:01.695Z"
}
]
```
Because there is no `response_parser`, the final data matches the exact response from the API.
Since each item in the final data is a JSON object, the `columns` configuration value isnโt required.
### Example POST API
```yaml theme={"system"}
url: https://api.something.com/users
method: POST
payload:
user:
first_name: Urza
power: 10
headers:
"Content-Type": "application/json"
token: abc123
response_parser: "user"
```
### Example XLSX or CSV
```yaml theme={"system"}
url: https://docs.google.com/spreadsheets/d/e/2PACX-1vTLcLUBAJAWf-8NQSjlbB3E4LR6DWk5QIZC-KtRb1j2CXXcgY6mE6vOJAW8PoJ1BAOgjXYpE4tY1LAD/pub?output=xlsx
method: GET
has_header: True
```
The response from the above endpoint is:
```
teste,first_name,second_name,email
1,Sadella,Tythacott,stythacott0@sina.com.cn
2,Melessa,Flaune,mflaune1@si.edu
3,Caroljean,Filipowicz,cfilipowicz2@guardian.co.uk
4,Doll,Wannan,dwannan3@people.com.cn
5,Nancy,Giraudy,ngiraudy4@pagesperso-orange.fr
6,Dominic,Bimson,dbimson5@vinaora.com
7,Kikelia,Bishopp,kbishopp6@cdc.gov
8,Andrus,Pomfrett,apomfrett7@wikipedia.org
9,Wildon,Fillingham,wfillingham8@google.co.uk
10,Alfonse,Leechman,aleechman9@jalbum.net
11,Phil,Emblem,pemblema@opera.com
12,Eyde,Brewer,ebrewerb@istockphoto.com
....
```
Data can also be checked by clicking on the "Load Sample data" button.
# Azure Blob Storage
Source: https://docs.mage.ai/data-integrations/sources/azure_blob_storage
## Configuration
To set up the Azure Blob Storage source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------- | ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `connection_string` | Storage account connection string. | `BlobEndpoint=https://xxx.blob.core.windows.net/;yyyyyy&sig=testsig` |
| `container_name` | The name of the Blob storage container where the data is stored. | `testcontainer` |
| `prefix` | Path within the container where files are located. Do not include the container name or table name in this path. | `users/ds/20221225` |
## How to Find Connection String
* **Option 1:** Get the connection string from your Storage Account's Access Keys:\
[Microsoft documentation](https://learn.microsoft.com/en-us/answers/questions/1071173/where-can-i-find-storage-account-connection-string)
* **Option 2:** If you want to restrict access to the storage account, generate a **Shared Access Signature (SAS)** token and build the connection string:\
[Microsoft documentation](https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/create-sas-tokens?view=form-recog-3.0.0#create-your-sas-tokens)
# BigQuery
Source: https://docs.mage.ai/data-integrations/sources/bigquery
## Configuration
To set up the BigQuery source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. If Mage is running on GCP, you can leave this `null` to use the instanceโs service account. | `/path/to/service_account_credentials.json` |
| `dataset` | BigQuery dataset you want to read data from. | `example_dataset` |
| `credentials_info` | An alternative way to specify Google service account credentials directly as a dictionary. | Structure shown below |
`credentials_info` structure:
```yaml theme={"system"}
auth_provider_x509_cert_url: str
auth_uri: str
client_email: str
client_id: str
client_x509_cert_url: str
private_key: str
private_key_id: str
project_id: str
token_uri: str
type: str
```
### Optional Configs
| Key | Description | Sample value |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| `skip_schema_creation` | If `true`, Mage won't run CREATE SCHEMA command. For more information, see this [issue](https://github.com/mage-ai/mage-ai/issues/3416) | `true` |
| `lower_case` | If `true`, Mage will set all columns name as lower case. Default is `true` | `true` |
### Required permissions
Your BigQuery account should at least have the below permissiont to use the BigQuery destination
```
bigquery.datasets.create
bigquery.jobs.create
bigquery.readsessions.create
bigquery.readsessions.getData
```
Your account should also have "BigQuery Data Editor" role for the BigQuery dataset you specified in the config.
# Chargebee
Source: https://docs.mage.ai/data-integrations/sources/chargebee
## Configuration
To set up the Chargebee source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ----------------- | -------------------------------------------------------------------------------------------------------- | ---------------------- |
| `api_key` | API key for your Chargebee site. | `fake_api_key...` |
| `site` | Name of your Chargebee site. (For `mage.chargebee.com`, enter `mage`.) | `your-site-name` |
| `start_date` | The date in ISO (`YYYY-MM-DDTHH:MM:SSZ`) format to start pulling data (for resources that support this). | `2010-01-01T00:00:00Z` |
| `include_deleted` | (Optional) Whether to include deleted records in all streams. Defaults to `True`. | `True` |
| `request_timeout` | (Optional) Timeout for requests to the Chargebee API (in seconds). Defaults to `300`. | `300` |
## Streams
The following streams are supported in the Chargebee source:
๐ [List of Supported Streams](https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/sources/chargebee/streams)
# Commercetools
Source: https://docs.mage.ai/data-integrations/sources/commercetools
## Configuration
To set up the Commercetools source, provide the following configuration parameters:
| Key | Description | Sample Value |
| --------------- | ----------------------------------------------------------------------- | ---------------------------- |
| `client_id` | `client_id` of the API client. Used to obtain the access token. | `abcdefg` |
| `client_secret` | `client_secret` of the API client. Used to obtain the access token. | `abcdefghijklmnopqrstuvwxyz` |
| `host` | The host in your Commercetools URL, e.g., `aws`, `gcp`. | `gcp` |
| `project_key` | The project key associated with your project. | `projectkey` |
| `region` | The region in your Commercetools URL, e.g., `us-central1`, `us-east-2`. | `us-central1` |
## How to Get Your `client_id` and `client_secret`
Follow [Commercetools's documentation](https://docs.commercetools.com/api/authorization):
* Create an API client.
* Retrieve `client_id`, `client_secret`, and `project_key` from the API client credentials page.
# Couchbase
Source: https://docs.mage.ai/data-integrations/sources/couchbase
## Configuration
To set up the Couchbase source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| `connection_string` | Connection string for your Couchbase database. [More info](https://docs.couchbase.com/kotlin-sdk/current/howtos/connecting.html#connection-string-scheme). | `couchbase://my_instance.cloud.couchbase.com` |
| `username` | Username to access the database (must have permissions to read from the specified bucket and scope). | `username` |
| `password` | Password for the user. | `password` |
| `bucket` | Name of the Couchbase bucket containing your data. | `my_bucket` |
| `scope` | Name of the Couchbase scope. Only collections within this scope will be available in Mage. | `my_scope` |
| `strategy` | (Optional) Strategy for schema inference. See below. | `infer` |
## Strategy Details
By default, Mage will attempt to infer your dataโs schema from a sample of documents.
* `strategy: infer` โ Mage will try to infer a unified schema from the data. If multiple schemas exist, Mage will pick the one covering the most data points.
* `strategy: combine` โ Mage will combine the data into a single `_document` column as an object.
# Custom source
Source: https://docs.mage.ai/data-integrations/sources/custom_source
Easily plug in your own data integration sources and destinations with Mage Pro.
Mage Pro makes it easy to integrate your own custom data integration sources and destinations.
To get started, follow the [adapt an existing source](/contributing/data-integrations/adapt-existing-source) guide or the [add a new source](/contributing/data-integrations/add-new-source) guide to create your custom integration.
Once your custom source or destination is ready, simply include it in your Mage Pro project by adding it to the appropriate folder as shown below.
## Folder structure
To register custom sources and destinations in your Mage Pro project, organize your project directory as follows:
```
.
โโโ mage_data
โโโ project_name
โโโ charts
โโโ custom
โโโ data_exporters
โโโ data_loaders
โโโ dbt
โโโ extensions
โโโ pipelines
โโโ data_integrations # โ important
โ โโโ sources
โ โ โโโ custom_source_1
โ โ โ โโโ templates
โ โ โ โ โโโ config.json
โ โ โ โโโ __init__.py
โ โ โ โโโ README.md
โ โ โโโ custom_source_2
โ โ โโโ metadata.yaml # Register all source definitions here
โ โโโ destinations
โ โโโ custom_destination_1
โ โโโ custom_destination_2
โ โโโ metadata.yaml # Register all destination definitions here
โโโ scratchpads
โโโ transformers
โโโ utils
โโโ __init__.py
โโโ io_config.yaml
โโโ metadata.yaml
โโโ requirements.txt
```
**Instructions**
1. Create a `data_integrations` folder inside your project directory.
2. Inside data\_integrations, create two subfolders:
* `sources`: for your custom data source integrations
* `destinations`: for your custom data destination integrations
3. Each source/destination should have its own folder (e.g., `custom_source_1`) containing:
* An `__init__.py` file with the integration class
* A `README.md` for documentation
* A `templates/` directory with a `config.json` file for default configuration
4. Define a `metadata.yaml` file in each of the sources/ and destinations/ folders to register the integrations with Mage Pro.
This structure allows Mage Pro to discover and load your custom integrations dynamically.
## Register custom sources and destinations
To register your custom sources or destinations, create or update the `metadata.yaml` file inside the
respective `sources/` or `destinations/` subfolders under `data_integrations`.
๐ Example: `sources/metadata.yaml`
```yaml theme={"system"}
sources:
- name: Source Name
module_name: SourceModuleName
uuid: source_uuid
```
๐ Example: `destinations/metadata.yaml`
```yaml theme={"system"}
destinations:
- name: Source Name
module_name: SourceModuleName
uuid: source_uuid
```
Field descriptions
* `name`: Display name shown in the Mage UI dropdown.
* `module_name` (optional): The class name defined in the source's `__init__.py` file. Defaults to the `name` field with spaces removed.
* `uuid` (optional): Folder name of the source. Defaults to the lowercased `name`, with spaces replaced by underscores.
## Code modification
Unlike the [add a new source guide](/contributing/data-integrations/add-new-source) (which adds sources to the Mage core),
using custom sources in your project requires updating import paths to point to your project-level modules.
Update any imports from `mage_integrations` to `data_integrations` like so:
```python theme={"system"}
# Before
from mage_integrations.sources.custom_source.module import SomeClass
# After
from data_integrations.sources.custom_source.module import SomeClass
```
This change ensures that Mage loads and uses your custom integration from the project folder instead of the built-in Mage package.
## Testing Your Custom Source
To test your custom data source integration:
1. Create a new data integration pipeline.
2. **Select your custom source** from the source dropdown.
* Custom sources appear at the **bottom** of the list in the UI.
3. **Enter your source configuration**.
4. Click "**Test connection**" if your custom source class implements the test\_connection method.
5. Click "**View and select stream**" to verify that the expected streams are returned.
6. **Add a destination** and run the pipeline using a trigger to test end-to-end data flow.
### Troubleshooting: Source Not Appearing in UI?
If your custom source isn't showing up in the Data Integration source dropdown, follow these steps to debug:
1. Create a pipeline and add a **scratchpad block**.
2. **Run the following code** in the block to inspect the custom sources:
```python theme={"system"}
# Inspect custom source registration
from mage_ai.server.api.integration_sources import get_custom_collection
custom_sources = get_custom_collection('sources', debug=True)
print(f'Custom sources detected: {custom_sources}')
```
3. **Try importing your custom module manually** to ensure it's discoverable and error-free:
```python theme={"system"}
from data_integrations.sources.source_uuid import SourceModuleName
```
Replace `source_uuid` and `SourceModuleName` with the actual folder name and class name of your custom source.
### Troubleshooting: "View and select stream" Not Working?
If clicking "**View and select stream**" in the UI doesnโt return any streams, you can manually debug the discovery step using a **scratchpad block** in your pipeline.
Run the following code to test your custom source's `discover_streams` method:
```python theme={"system"}
from data_integrations.sources.source_uuid import SourceModuleName
# Replace with your actual config keys and values
config = dict(
key1='value1',
key2='value2',
)
# Initialize the source class with config
source = SourceModuleName(config=config)
# Run stream discovery
streams = source.discover_streams()
# Print discovered streams
print(f'Discovered streams: {streams}')
```
**What to check**
* Ensure the `config` you provide contains all required fields expected by your source class.
* If `discover_streams()` returns an empty list or raises an error, check your source implementation or credentials.
# Datadog
Source: https://docs.mage.ai/data-integrations/sources/datadog
## Configuration
To set up the Datadog source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `api_key` | Datadog API key. | `abc123...` |
| `application_key` | Datadog application key. | `def456...` |
| `start_date` | The date in ISO (`YYYY-MM-DDTHH:MM:SSZ`) format to start pulling data (for streams that support this). If a bookmark property is used, it will prioritize the latest bookmark date. | `2010-01-01T00:00:00Z` |
| `query` | Some streams support a `query` parameter to narrow down the data (currently `audit_logs` and `logs` streams). | `query string...` |
## Query Configuration
To set up a query for a specific stream, provide the stream name as a key in the `query` field, and the query string as the value.
Example:
```yaml theme={"system"}
query:
audit_logs: your query here...
```
# Doris
Source: https://docs.mage.ai/data-integrations/sources/doris
## Configuration
To set up the Doris source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| `database` | The name of the database you want to read data from. | `demo` |
| `host` | Host name of your Doris database. | `mage.abc.us-west-2.rds.amazonaws.com` |
| `port` | Port where the database is running (typically 9030). | `9030` |
| `username` | Username with permissions to read and write to the specified schema. | `root` |
| `password` | Password for the user. | `abc123...` |
| `connection_method` | Connection method to Doris server: `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` |
| `conn_kwargs` | (Optional) Extra [connection keyword arguments](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) in dictionary format. | `{"ssl_ca": "CARoot.pem", "ssl_cert": "certificate.pem", "ssl_key": "key.pem"}` |
| `ssh_host` | (Optional) Host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (Optional) Port of the bastion server. Default is 22. | `22` |
| `ssh_username` | (Optional) Username used to connect to the bastion server. | `username` |
| `ssh_password` | (Optional) Password used to connect to the bastion server (if authenticating with password). | `password` |
| `ssh_pkey` | (Optional) Path to the private key used to connect to the bastion server (if authenticating with private key). | `/path/to/private/key` |
### Optional Configuration
| Key | Description | Sample Value |
| ------------------- | ----------------------------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch in each batch (defaults to 50,000). Adjust higher if your instance has more memory. | `50000` |
# Dremio
Source: https://docs.mage.ai/data-integrations/sources/dremio
## Configuration
This source uses the Dremio Arrow Flight Client to connect and execute SQL queries and shares most of its settings.\
For more details, refer to the [Dremio Python Client documentation](https://docs.dremio.com/current/sonar/client-applications/clients/python).
> **Note:** Mage's implementation of the Dremio source does **not** allow the `query` connection parameter.
### Required Parameters
| Key | Description | Sample Value |
| ---------------- | ----------------------------------------------------------------------------- | ------------ |
| `username` | Username of the Dremio account used for authentication. | `ABC123` |
| `schema` | Dremio source table schema. | `ABC123` |
| `source_backend` | SQL source backend used to query data from Dremio (see section below). | `postgresql` |
| `password` | Password for the Dremio account. (Required if no token is provided.) | `ABC123` |
| `token` | Personal Access Token or OAuth2 Token. (Required if no password is provided.) | `ABC123` |
### Optional Parameters
| Key | Description | Sample Value |
| ---------------------------------- | ------------------------------------------------------------------------- | -------------------- |
| `hostname` | Hostname or IP address of the coordinator node (defaults to `localhost`). | `localhost` |
| `port` | Dremio's Arrow Flight server port (defaults to `32010`). | `32010` |
| `tls` | Enables TLS encryption for the connection. | `false` |
| `disable_certificate_verification` | Disables TLS server certificate verification. | `true` |
| `path_to_certs` | Path to trusted certificates for TLS connections. | `/path/to/certs` |
| `session_properties` | Key-value pairs for session properties. | `{ "key": "value" }` |
### Source Backend
To maintain consistent data extraction quality, this source allows specifying a preferred SQL source backend.\
Currently supported backend sources:
* `postgresql`
* `mysql`
* `mssql`
> **Note:**\
> OracleDB and Snowflake do **not** require a `source_backend`.\
> Internal tables created by S3 or other object storage also do **not** require a backend source.
# Amazon DynamoDB
Source: https://docs.mage.ai/data-integrations/sources/dynamodb
Use Mage to extract data from Amazon DynamoDB tables using AWS credentials or IAM roles.
## Overview
Amazon DynamoDB is a fully managed NoSQL database service that delivers single-digit millisecond performance at scale. With Mage, you can integrate DynamoDB as a source to extract and transform structured data from your tables into your pipelines.
## Configuration
To connect Mage to DynamoDB, add a source block with the following configuration in your data integration pipeline.
| Key | Description | Example Value |
| ----------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------ |
| `aws_access_key_id` | Your AWS access key ID. *(Not required if using `role_arn` and Mage is running on AWS.)* | `AKIA...` |
| `aws_secret_access_key` | Your AWS secret access key. *(Not required if using `role_arn` and Mage is running on AWS.)* | `abc123xyz789...` |
| `aws_region` | AWS region where your DynamoDB tables are located. Defaults to `us-west-2`. | `us-west-2` |
| `role_arn` | (Optional) ARN of an IAM role Mage should assume when accessing DynamoDB. | `arn:aws:iam::123456789012:role/mage-role` |
If running Mage on EC2, EKS, or ECS, you can skip `aws_access_key_id` and `aws_secret_access_key` by granting access via IAM roles (via instance profiles or IRSA).
## Permissions
Ensure your AWS IAM user or role has the following permissions:
* `dynamodb:Scan`
* `dynamodb:Query`
* `dynamodb:DescribeTable`
* `dynamodb:GetItem` *(if using key-based lookups)*
* `dynamodb:ListTables` *(optional โ for dynamic table discovery)*
These permissions are required for Mage to:
* Read data from DynamoDB tables
* Apply filters or projections
* Automatically detect schema or validate table access
# Facebook Ads
Source: https://docs.mage.ai/data-integrations/sources/facebook_ads
## Configuration
To set up the Facebook Ads source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| -------------- | ------------------------------------------------------------------ | -------------------------------------- | -------- |
| `account_id` | The Facebook Ad account ID used when extracting data. | `"abc123..."` (double quotes required) | โ |
| `access_token` | OAuth token to access API endpoints. | `abc123...` | โ |
| `start_date` | Fetch data newer than the specified `start_date`. Format: ISO8601. | `2022-01-01T00:00:00Z` | โ |
### `account_id`
Visit [Facebook Ads Manager Accounts](https://www.facebook.com/adsmanager/manage/accounts) to find your account ID.\
It should look like: `13261011608127020`.
> **Note**\
> Your account ID is **not** the same as your App ID.
### `access_token`
1. [Create a Facebook App](https://developers.facebook.com/docs/development/create-an-app/).
2. Get the access token:
1. Go to: `https://developers.facebook.com/apps/[app_id]/marketing-api/tools`
* Replace `[app_id]` with the ID of the app you created.
2. Under **Select Token Permissions**, check the following:
* `ads_management`
* `ads_read`
* `read_insights`
3. Click **Get Token**.
3. Copy the generated `access_token`.
## Errors
### API Rate Limit
If you encounter API rate limits, refer to this [Facebook Community Thread](https://developers.facebook.com/community/threads/158172292247085/) for more information.
# Facebook Pages
Source: https://docs.mage.ai/data-integrations/sources/facebook_pages
## Configuration
Connect to **Facebook Pages** and sync posts, insights, and engagement metrics. Provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | -------- |
| `access_token` | OAuth access token for Facebook Graph API with the required permissions (`pages_read_engagement`, `pages_read_user_content`). | `EAABsbCS1i...` | โ |
| `page_ids` | List of Facebook Page IDs you want to extract data from. | `["1234567890", "9876543210"]` | โ |
| `start_date` | Start date (ISO 8601 format) to begin fetching posts and engagement data. | `2025-01-01T00:00:00Z` | โ |
## How to Get Your `access_token`
You have two options to generate an access token:
### 1. Use the OAuth Button (Recommended)
* Click the **OAuth Authorize** button in your Mage workspace.
* Approve the required Facebook permissions:
* `pages_show_list`
* `pages_read_engagement`
* `pages_read_user_content`
* `read_insights`
* The access token will be securely generated and auto-filled in the configuration.
> **Tip**: Use the OAuth flow for a smoother, safer authentication process without manually handling tokens.
### 2. Manually Generate Access Token
* Create a Facebook App via [Facebook Developer Portal](https://developers.facebook.com/).
* Configure the app and get the permissions listed above.
* Generate a **User Access Token** or **Page Access Token** via [Graph API Explorer](https://developers.facebook.com/tools/explorer/).
* (Optional) Exchange for a **long-lived token** for production environments.
## What is Facebook Pages API?
The [Facebook Pages API](https://developers.facebook.com/docs/pages) allows programmatic access to posts, insights, page metadata, and engagement data for Facebook Pages. Businesses use it to monitor social presence, measure engagement, and automate reporting workflows.
## Why Integrate Facebook Pages with Mage?
* **Automate data extraction**: Fetch page posts, reactions, and comments for analysis.
* **Engagement tracking**: Monitor likes, shares, comments, and post performance.
* **Insights at scale**: Integrate page metrics into BI dashboards for better reporting.
* **Marketing analytics**: Correlate Facebook engagement with other marketing activities.
## Supported Streams
The Facebook Pages source extracts the following resources:
* **page**: Basic page details like name, ID, and metadata.
* **page\_insights**: Aggregated metrics like page impressions, engagement, and follower growth.
* **post\_attachments**: Media and links attached to posts.
* **post\_insights**: Detailed insights for each post, including reach and engagement breakdown.
* **post\_tagged\_profile**: Profiles tagged in the posts.
* **posts**: Content posted on the page โ text, images, videos, and links.
# Freshdesk
Source: https://docs.mage.ai/data-integrations/sources/freshdesk
## Configuration
To set up the Freshdesk source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------ | ------------------------------------------------------------------------------------------- | ---------------------- |
| `api_key` | Freshdesk API key. | `abc123` |
| `domain` | Your Freshdesk domain. For `https://mage.freshdesk.com`, the domain is `mage`. | `mage` |
| `start_date` | Fetch data newer than the specified `start_date`. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2021-01-01T00:00:00Z` |
## How to Get Your `api_key`
Follow [Freshdeskโs documentation](https://support.freshdesk.com/en/support/solutions/articles/215517-how-to-find-your-api-key)\
to locate your API key.
# Front
Source: https://docs.mage.ai/data-integrations/sources/front
## Configuration
To set up the Front source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------- |
| `token` | The API token of the Front app. | `abc123` |
| `start_date` | Used for the `analytics` stream. Fetch data newer than this `start_date`. Format: ISO8601. | `2023-01-01T00:00:00Z` |
| `end_date` | Used for the `analytics` stream. Fetch data older than this `end_date`. If `null`, current date is used. | `2023-01-05T00:00:00Z` |
| `incremental_range` | Used for the `analytics` stream. Defines the incremental extract range: `daily` or `hourly`. | `daily`, `hourly` |
## How to Get Your `token`
Follow [Frontโs documentation](https://dev.frontapp.com/docs/create-and-revoke-api-tokens#create-an-api-token)\
to create and retrieve your API token.
# GitHub
Source: https://docs.mage.ai/data-integrations/sources/github
## Configuration
To set up the GitHub source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ----------------- | ------------------------------------------------------------------------------- | ------------------------------------------ | -------- |
| `access_token` | A GitHub access token containing credentials to access your desired repository. | `abcdefghijklmnopqrstuvwxyz1234567890ABCD` | โ |
| `repository` | The repository path (everything after `github.com`). | `mage-ai/mage-ai` | โ |
| `start_date` | A sync start date. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2021-01-01T00:00:00Z` | โ |
| `request_timeout` | Request timeout value in seconds. | `300` | โ |
| `base_url` | The base URL before the repository path (defaults to `https://api.github.com`). | `https://api.github.com` | โ |
## Generating Credentials
Follow [this guide](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) to generate a Personal Access Token (PAT).
## Supported Endpoints
This source pulls raw data from the GitHub REST API and extracts the following resources for a single repository:
* Assignees
* Collaborators
* Commits
* Commit Comments
* Events
* Issues
* Issue Events
* Issue Milestones
* Projects
* Project Cards
* Project Columns
* Pull Requests
* PR Commits
* Releases
* Comments
* Reviews
* Review Comments
* Stargazers
* Teams
* Team Members
* Team Memberships
It outputs the schema for each resource.
> **Note:**\
> Incremental sync is currently **not supported** for any endpoint.\
> To add support, the `IncrementalStream` class in `./tap_github/streams.py` needs modifications to correctly handle bookmark values.
# Google Ads
Source: https://docs.mage.ai/data-integrations/sources/google_ads
Sync campaign, ad group, keyword, and performance report data from the Google Ads API v23 using OAuth 2.0 authentication.
## Overview
This source syncs campaign, ad group, ad, keyword, and performance report data from the [Google Ads API](https://developers.google.com/google-ads/api/docs/start) (**v23**). Authentication uses user OAuth (not service accounts) โ the connector requires a `refresh_token` alongside a `developer_token`.
Incremental state is handled inside the connector; **stream-level INCREMENTAL selection is not available** in the UI, but bookmarks still resume progress across syncs.
## Authentication
This connector uses **two kinds** of credentials:
* **Application credential** (`developer_token`) โ identifies your app to the Google Ads API. Issued per manager account via [API Center](https://ads.google.com/aw/apicenter). This is not an OAuth token.
* **User OAuth** (`refresh_token`, `oauth_client_id`, `oauth_client_secret`) โ proves the user has consented to read their Ads data. The required OAuth scope is `https://www.googleapis.com/auth/adwords`.
### Manual setup vs Mage OAuth Authorize
| Method | What to set |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Mage OAuth Authorize** (recommended) | Use the **OAuth Authorize** button on the source block. Mage injects `refresh_token` automatically. You only need `developer_token` and `login_customer_ids`. You may omit `oauth_client_id` and `oauth_client_secret`. |
| **Manual** | Create an OAuth 2.0 client in Google Cloud (Web or Desktop app type), obtain a `refresh_token` for the Ads scope, and fill in all three OAuth fields. |
## Service accounts and Application Default Credentials are **not** supported.
## INCREMENTAL Sync
This source handles state internally, meaning **INCREMENTAL sync selection is not supported**.
## Configuration
| Key | Required | Default | Description |
| --------------------- | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `developer_token` | โ | โ | Google Ads API application token from [API Center](https://ads.google.com/aw/apicenter). Not a user OAuth token. |
| `oauth_client_id` | Conditional | โ | Google Cloud OAuth client ID. Required for manual OAuth setup; optional when using Mage OAuth Authorize. |
| `oauth_client_secret` | Conditional | โ | Google Cloud OAuth client secret. Same conditions as `oauth_client_id`. |
| `refresh_token` | โ | โ | Long-lived user refresh token for the `https://www.googleapis.com/auth/adwords` scope. Injected automatically by Mage OAuth Authorize. |
| `login_customer_ids` | โ | โ | List of `{ customerId, loginCustomerId }` pairs. `customerId` is the account to read; `loginCustomerId` is the manager (MCC) account used to access it, or `null` for direct access. |
| `start_date` | โ | โ | Fetch data newer than this date. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). |
| `end_date` | | now | Upper bound for report syncs. |
| `request_timeout` | | `900` | HTTP timeout in seconds for API calls. |
| `conversion_window` | | `30` | Conversion attribution window in days. Allowed: `1`โ`30`, `60`, `90`. |
| `query_limit` | | `1000000` | Max rows per query for core streams. |
## Sample configs
**Using Mage OAuth Authorize** (recommended โ no OAuth client credentials needed):
```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
refresh_token: YOUR_REFRESH_TOKEN
start_date: '2023-01-01T00:00:00Z'
login_customer_ids:
- customerId: '1234567890'
loginCustomerId: null
```
**Manual OAuth setup with manager account**:
```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
oauth_client_id: YOUR_CLIENT_ID.apps.googleusercontent.com
oauth_client_secret: YOUR_CLIENT_SECRET
refresh_token: YOUR_REFRESH_TOKEN
start_date: '2023-01-01T00:00:00Z'
login_customer_ids:
- customerId: '1234567890'
loginCustomerId: '0987654321'
```
## How to get `login_customer_ids`
`customerId` is the account whose data you want to sync. `loginCustomerId` is the manager (MCC) account you use to access it โ leave it `null` when there is no manager hop.
1. Sign in to [Google Ads](https://ads.google.com/).
2. Open the account switcher in the top bar.
3. Copy the numeric ID for each account (strip dashes: `123-456-7890` โ `'1234567890'`).
## How to get a `developer_token`
A developer token belongs to a [manager (MCC) account](https://support.google.com/google-ads/answer/6139186). Client-only accounts cannot open API Center.
1. Sign in to a manager account.
2. Open [API Center](https://ads.google.com/aw/apicenter) (**Tools & Settings โ Setup โ API Center**).
3. Complete the API sign-up. Your token may start as **Test** tier; request **Basic** or **Standard** access when you are ready for production volume.
4. Set the token as `developer_token` in your config.
## Related links
* [Google Ads API โ Get Started](https://developers.google.com/google-ads/api/docs/start)
* [OAuth with Google Ads API](https://developers.google.com/google-ads/api/docs/oauth/overview)
* [Developer token access levels](https://developers.google.com/google-ads/api/docs/access-levels)
* [Manager accounts overview](https://support.google.com/google-ads/answer/6139186)
To set up the Google Ads source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| --------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------- |
| `login_customer_ids` | A list containing both `customerId` and `loginCustomerId` from Google Ads. | `"login_customer_ids": [{"customerId": "1234567890", "loginCustomerId": "0987654321"}]` | โ |
| `oauth_client_id` | OAuth client ID from Google Cloud. | `XXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYY.apps.googleusercontent.com` | โ |
| `oauth_client_secret` | OAuth client secret from Google Cloud. | `abc...` | โ |
| `refresh_token` | Google API refresh token, obtainable from [Google OAuth Playground](https://developers.google.com/oauthplayground). | `abc...` | โ |
| `developer_token` | Google Ads developer token. | `abc...` | โ |
| `start_date` | Fetch data newer than the specified `start_date`. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2022-01-01T00:00:00Z` | โ |
## Generating Credentials
Check out [this article](https://articles.wesionary.team/how-to-implement-google-ads-api-ff69f628d4ac)\
by **Sudeep Timalsina** for a great guide on generating credentials for Google Ads API access.
# Google Analytics
Source: https://docs.mage.ai/data-integrations/sources/google_analytics
## Configuration
To set up the Google Analytics source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. | `/path/to/service_account_credentials.json` |
| `property_id` | Google Analytics GA4 Property ID. [Find your Property ID here](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id). | `12345678` |
| `start_date` | The start date for the Google Analytics query. Format: `YYYY-MM-DD`. | `2022-01-01` |
| `end_date` | The end date for the Google Analytics query. Format: `YYYY-MM-DD`. | `2022-01-01` |
### Prerequisites
1. Enable the **Google Analytics API** in your Google Cloud Console:\
[Google Cloud API Dashboard](https://console.cloud.google.com/apis/dashboard)
2. The service account must have access to the Google Analytics property:
* Go to the [Google Analytics Admin Dashboard](https://analytics.google.com/analytics)
* Click **Admin** in the bottom left corner.
* Click **Account Access Management**.
* Click **Property Access Management**.
* Add the service account email address (from your JSON credentials) with the appropriate permissions.
## Schema
Google Analytics data includes two types of columns:
* [Dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema#custom_dimensions)
* [Metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema#custom_dimensions)
### Limitations
* **Maximum number of dimensions:** 9\
([See more on Dimensions](https://developers.google.com/analytics/devguides/reporting/data/v1/rest/v1beta/Dimension))
* **Maximum number of metrics:** 10\
([See more on Metrics](https://developers.google.com/analytics/devguides/reporting/data/v1/rest/v1beta/Metric))
# Google Campaign Manager 360
Source: https://docs.mage.ai/data-integrations/sources/google_campaign_manager_360
Sync entity data and Report Builder exports from Campaign Manager 360 using the DFA Reporting & Trafficking API v5 and a service account.
## Overview
This source syncs entity data and report exports from [Campaign Manager 360](https://campaignmanager.google.com/) using the [Campaign Manager 360 API (dfareporting v5)](https://developers.google.com/doubleclick-advertisers). API v4 is sunset; this connector targets **v5 only**.
There are two categories of streams:
* **Entity streams** (`user_profiles`, `advertisers`, `campaigns`, `reports`) โ list resources via the DFA Reporting & Trafficking API. Each row includes a stable `entity_id`, flattened top-level fields, and `_raw` (full API object as JSON).
* **Report export stream** (`report_rows`) โ runs saved **Report Builder** reports by ID, waits for completion, downloads the CSV file via the API, and emits one record per data row. Columns vary by report definition and are packed into a `row_data` JSON string.
Authentication uses a **Google Cloud service account** JSON key. Domain-wide delegation is supported via the optional `email` field.
All streams use **full table** replication.
## Configuration
| Key | Required | Default | Description |
| ------------------------------- | ---------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path_to_credentials_json_file` | โ | โ | Path to a Google Cloud **service account** JSON key with access to Campaign Manager 360. |
| `email` | | โ | Google user to impersonate when using **domain-wide delegation**. Leave blank for direct service-account access. |
| `profile_id` | โ | โ | Campaign Manager **user profile ID**. Required for `advertisers`, `campaigns`, `reports`, and `report_rows`. Run the `user_profiles` stream (or use **Load sample data**) to discover available profile IDs. |
| `advertiser_ids` | | auto-discover | Comma-separated DCM advertiser IDs. If omitted, the `campaigns` stream auto-discovers advertisers by calling `advertisers.list` for `profile_id`. |
| `report_ids` | โ (`report_rows` only) | โ | Comma-separated IDs of saved **Report Builder** reports. Required for `report_rows`; ignored by all other streams. Each report must be configured to output **CSV** (not Excel). |
| `report_use_latest_file` | | `false` | When `true`, reads the most recently completed (`REPORT_AVAILABLE`) file for each report instead of triggering a new execution. Use this for idempotent syncs or when you lack permission to trigger new runs. Raises an error if no completed file exists for a report ID. |
| `report_run_synchronous` | | `false` | Passes `synchronous=true` to [`reports.run`](https://developers.google.com/doubleclick-advertisers/rest/v5/reports/run). May complete faster for small reports when supported. Has no effect when `report_use_latest_file` is `true`. |
| `report_poll_interval_seconds` | | `10` | Seconds between [`files.get`](https://developers.google.com/doubleclick-advertisers/rest/v5/files/get) status polls while the report is `QUEUED` or `PROCESSING`. Has no effect when `report_use_latest_file` is `true`. |
| `report_poll_timeout_seconds` | | `3600` | Maximum seconds to wait for `REPORT_AVAILABLE` before failing the sync. Has no effect when `report_use_latest_file` is `true`. |
## Sample configs
**Entity streams only** โ user profiles, advertisers, campaigns, reports (no report download):
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
profile_id: "1234567"
```
**Report rows โ trigger a new run and wait for completion**:
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
profile_id: "1234567"
report_ids: "987654321,876543210"
report_poll_interval_seconds: 30
report_poll_timeout_seconds: 1800
```
**Report rows โ read the most recently completed file** (no new execution triggered):
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
profile_id: "1234567"
report_ids: "987654321"
report_use_latest_file: true
```
**Campaigns scoped to specific advertisers with domain-wide delegation**:
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/cm360_service_account.json
email: analyst@company.com
profile_id: "1234567"
advertiser_ids: "111111111,222222222"
```
## Streams
| Stream | Description |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `user_profiles` | DCM user profiles accessible to the credential. A useful starting point โ run this stream to discover your `profile_id` value. |
| `advertisers` | Advertisers visible to `profile_id` via [`advertisers.list`](https://developers.google.com/doubleclick-advertisers/rest/v5/advertisers/list). |
| `campaigns` | Campaigns per advertiser. Uses `advertiser_ids` if set; otherwise auto-discovers advertisers first. |
| `reports` | Saved Report Builder report definitions for `profile_id`. Returns report metadata only โ use `report_rows` to execute and download data. |
| `report_rows` | Executes each configured `report_id`, waits for the file to reach `REPORT_AVAILABLE`, downloads the CSV, and emits one record per data row. Empty rows (e.g. summary/footer rows) are skipped. Each record contains `_report_id`, `_file_id`, `_row_number`, and `row_data` (a JSON string mapping CSV header names to cell values). |
**`report_rows` and sample data:** When loading sample data, only the first `report_id` is run and at most 25 CSV rows are returned, so the UI preview does not trigger every report or download large files.
## How `report_rows` works
1. For each `report_id`, the connector calls [`reports.run`](https://developers.google.com/doubleclick-advertisers/rest/v5/reports/run) to start a new execution (or, if `report_use_latest_file: true`, looks up the most recent completed file via `reports.files.list`).
2. It polls [`files.get`](https://developers.google.com/doubleclick-advertisers/rest/v5/files/get) every `report_poll_interval_seconds` until the status reaches `REPORT_AVAILABLE`, `FAILED`, or `CANCELLED`, or until `report_poll_timeout_seconds` is exceeded.
3. Once available, the file is downloaded in streaming chunks via the `files.get_media` endpoint and written to a local temp file.
4. The temp file is parsed as a UTF-8 CSV. Blank rows are skipped. Each data row is emitted as a record.
## Prerequisites
1. Enable **Campaign Manager 360 API** (`dfareporting.googleapis.com`) in [Google Cloud Console](https://console.cloud.google.com/apis/dashboard).
2. Create a service account and download its JSON key. Set `path_to_credentials_json_file` to the local path.
3. In Campaign Manager 360, grant the service account access to the account (via **Admin โ User Profiles**). Use **domain-wide delegation** and set `email` if your organization requires impersonation.
4. For `report_rows`: create and save the report in Campaign Manager's [Report Builder](https://support.google.com/campaignmanager/answer/2823849) with **CSV** output format. Copy the numeric report ID from the URL or Report Builder UI and add it to `report_ids`.
### OAuth scopes used
| Scope | Used by |
| ------------------------------------------------ | ---------------------------------------------------------------------- |
| `https://www.googleapis.com/auth/dfareporting` | All streams (reporting API) |
| `https://www.googleapis.com/auth/dfatrafficking` | Entity streams (trafficking API โ required for advertisers, campaigns) |
## Related links
* [Campaign Manager 360 API (v5) overview](https://developers.google.com/doubleclick-advertisers)
* [reports.run reference](https://developers.google.com/doubleclick-advertisers/rest/v5/reports/run)
* [File status lifecycle](https://developers.google.com/doubleclick-advertisers/rest/v5/File#Status)
* [Service account authentication](https://cloud.google.com/iam/docs/service-account-overview)
# Google Cloud Storage
Source: https://docs.mage.ai/data-integrations/sources/google_cloud_storage
## Configuration
To set up the Google Cloud Storage source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. If Mage is running on GCP, you can leave this `null` to use the instanceโs service account. | `/path/to/service_account_credentials.json` |
| `bucket` | Name of the Google Cloud Storage bucket to read data from. | `my_bucket` |
| `file_type` | Type of the files in the bucket. Supported values: `parquet`, `csv`. | `parquet` |
| `credentials_info` | Alternative way to specify Google service account credentials directly as a dictionary. | Structure shown below |
| `prefix` | A prefix used to filter blobs in the bucket. Only blobs whose names start with this prefix will be included. | `sales_data_` |
**`credentials_info` structure:**
```yaml theme={"system"}
type: str
project_id: str
private_key_id: str
private_key: str
client_email: str
client_id: str
auth_uri: str
token_uri: str
auth_provider_x509_cert_url: str
client_x509_cert_url: str
universe_domain: str
```
# Google Display & Video 360
Source: https://docs.mage.ai/data-integrations/sources/google_display_video_360
Sync DV360 entity metadata and Bid Manager performance report data from Google's Display & Video 360 platform using a service account.
## Overview
This source syncs entity metadata from [Display & Video 360](https://displayvideo.google.com/) using the [Display Video API v4](https://developers.google.com/display-video/api/reference/rest/v4) and performance report data using the [DoubleClick Bid Manager API v2](https://developers.google.com/bid-manager/reference/rest).
**Entity streams** cover the full DV360 resource hierarchy: partners โ advertisers โ campaigns, insertion orders, line items, creatives, channels, ad groups, and more, including nested resources such as assigned targeting options and YouTube asset associations.
**Bid Manager streams** (`bid_manager_queries`, `bid_manager_reports`, `bid_manager_report_rows`) list saved queries and their completed reports, and download CSV report data directly from the GCS path returned by the Bid Manager API.
Authentication uses a **Google Cloud service account** JSON key (not user OAuth). Domain-wide delegation is supported via the optional `email` field.
All streams use **full table** replication. Each entity row includes `entity_id`, `_raw` (full API object as JSON), and flattened top-level fields.
## Configuration
| Key | Required | Default | Description |
| ------------------------------- | -------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path_to_credentials_json_file` | โ | โ | Path to a Google Cloud **service account** JSON key with access to Display & Video 360. |
| `email` | | โ | Google user to impersonate when using **domain-wide delegation**. Leave blank for direct service-account access. |
| `partner_id` | โ | โ | DV360 **partner ID**. Required for `advertisers` and any advertiser-scoped stream when `advertiser_ids` is not set. Not required for `partners` or `bid_manager_*` streams. Optional for **Test connection** (falls back to `partners.list` without it). |
| `advertiser_ids` | | auto-discover | Comma-separated DV360 advertiser IDs. If set, all advertiser-scoped streams use only these IDs and `partner_id` is not needed for those streams. If omitted, advertiser IDs are discovered by calling `advertisers.list` under `partner_id`. |
| `bid_manager_query_ids` | | all queries | Comma-separated Bid Manager query IDs. If omitted, all queries from `queries.list` are used โ this can be slow when there are many queries. |
## Sample configs
**DV360 entity streams only** (no Bid Manager):
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
partner_id: "123456"
```
**Scoped to specific advertisers** (`partner_id` not required for advertiser-scoped streams):
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
advertiser_ids: "111111111,222222222"
```
**Bid Manager streams with explicit query IDs** (disable entity streams in catalog):
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
bid_manager_query_ids: "123456789,987654321"
```
**Full config โ DV360 entities + Bid Manager + domain-wide delegation**:
```yaml theme={"system"}
path_to_credentials_json_file: /secrets/dv360_service_account.json
email: analyst@company.com
partner_id: "123456"
advertiser_ids: "111111111,222222222"
bid_manager_query_ids: "123456789"
```
## Streams
### DV360 entity streams
| Stream | API resource |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `partners` | [`partners.list`](https://developers.google.com/display-video/api/reference/rest/v4/partners/list) โ partners visible to the credential. No `partner_id` required. |
| `advertisers` | [`advertisers.list`](https://developers.google.com/display-video/api/reference/rest/v4/advertisers/list) โ advertisers under `partner_id`. |
| `campaigns` | `advertisers.campaigns.list` |
| `insertion_orders` | `advertisers.insertionOrders.list` |
| `line_items` | `advertisers.lineItems.list` |
| `ad_groups` | `advertisers.adGroups.list` |
| `ad_group_ads` | `advertisers.adGroupAds.list` |
| `ad_assets` | `advertisers.adAssets.list` |
| `creatives` | `advertisers.creatives.list` |
| `channels` | `advertisers.channels.list` |
| `channel_sites` | `advertisers.channels.sites.list` โ walks each channel per advertiser. |
| `invoices` | `advertisers.invoices.list` |
| `location_lists` | `advertisers.locationLists.list` |
| `location_list_assigned_locations` | `advertisers.locationLists.assignedLocations.list` |
| `negative_keyword_lists` | `advertisers.negativeKeywordLists.list` |
| `negative_keyword_list_negative_keywords` | `advertisers.negativeKeywordLists.negativeKeywords.list` |
| `advertiser_assigned_targeting_options` | `advertisers.targetingTypes.assignedTargetingOptions.list` โ one request per targeting type enum value. |
| `ad_group_assigned_targeting_options` | `advertisers.adGroups.targetingTypes.assignedTargetingOptions.list` โ one request per ad group per targeting type. |
| `ad_group_youtube_asset_associations` | `advertisers.adGroups.youtubeAssetTypes.youtubeAssetAssociations.list` |
| `line_item_assigned_targeting_options` | `advertisers.lineItems.targetingTypes.assignedTargetingOptions.list` |
| `line_item_youtube_asset_associations` | `advertisers.lineItems.youtubeAssetTypes.youtubeAssetAssociations.list` |
### Bid Manager streams
| Stream | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bid_manager_queries` | Saved Bid Manager query definitions from [`queries.list`](https://developers.google.com/bid-manager/reference/rest/v2/queries/list). |
| `bid_manager_reports` | Report metadata per query from [`queries.reports.list`](https://developers.google.com/bid-manager/reference/rest/v2/queries.reports/list). |
| `bid_manager_report_rows` | For each completed (`DONE`) CSV report, streams rows from the GCS path in `metadata.googleCloudStoragePath`. Each record contains `entity_id`, `_query_id`, `_report_id`, `_row_number`, and `row_data` (JSON map of CSV column โ value). Reports with status other than `DONE`, XLSX format, or no GCS path are skipped. |
**Sample data limits:** `bid_manager_queries` returns at most 25 rows; `bid_manager_reports` uses only the first query and one report; `bid_manager_report_rows` returns at most 25 CSV rows from a single report.
## Prerequisites
1. Enable the **Display & Video 360 API** (`displayvideo.googleapis.com`) in [Google Cloud Console](https://console.cloud.google.com/apis/dashboard).
2. For Bid Manager streams, also enable the **DoubleClick Bid Manager API** (`doubleclickbidmanager.googleapis.com`) and grant the service account read access to the GCS buckets where report files are delivered.
3. Create a service account and download its JSON key. Set `path_to_credentials_json_file` to the local path.
4. In the DV360 UI, grant the service account access at the partner or advertiser level as appropriate.
5. If your organization uses **domain-wide delegation**: configure it in Google Workspace Admin for the service account, include the DV360 and Bid Manager scopes, and set `email` to the user to impersonate.
### OAuth scopes used
| Scope | Used by |
| ------------------------------------------------------- | ----------------------------------------- |
| `https://www.googleapis.com/auth/display-video` | All DV360 entity streams |
| `https://www.googleapis.com/auth/doubleclickbidmanager` | Bid Manager streams |
| `https://www.googleapis.com/auth/devstorage.read_only` | Bid Manager report file download from GCS |
## Related links
* [Display & Video 360 API (v4) reference](https://developers.google.com/display-video/api/reference/rest/v4)
* [DoubleClick Bid Manager API (v2) reference](https://developers.google.com/bid-manager/reference/rest)
* [Service account authentication](https://cloud.google.com/iam/docs/service-account-overview)
* [Domain-wide delegation](https://developers.google.com/workspace/guides/create-credentials#optional_set_up_domain-wide_delegation_for_a_service_account)
# Google Search Console
Source: https://docs.mage.ai/data-integrations/sources/google_search_console
## Configuration
To set up the Google Search Console source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. | `/path/to/service_account_credentials.json` |
| `email` | If using domain-wide delegation, enter the user email to impersonate. | `test@xyz.com` |
| `site_urls` | Website URLs separated by commas (e.g., `https://www.example.com`, `sc-domain:example.com`). | `https://www.mage.ai, sc-domain:example.com` |
| `start_date` | Start date for the Google Search Console query. Format: `YYYY-MM-DD`. | `2022-01-01` |
## Prerequisites
* [Enable the Google Search Console API](https://console.cloud.google.com/apis/dashboard).
* Grant the service account access to your Search Console property:
* Go to [Google Search Console](https://search.google.com/search-console).
* Navigate to **Settings** โ **Users and permissions**.
* Add your service account email with the appropriate permissions.
# Google Sheets
Source: https://docs.mage.ai/data-integrations/sources/google_sheets
## Configuration
To set up the Google Sheets source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------- |
| `path_to_credentials_json_file` | Path to your Google service account credentials JSON file. | `/path/to/service_account_credentials.json` |
| `spreadsheet_id` | The unique identifier of a spreadsheet (found in the spreadsheet URL after `/spreadsheets/d/`). | `abcdefg123456` |
| `selected_sheet_names` | A list of sheet names to filter and extract. | `["sheet1", "sheet2"]` |
## Prerequisites
* [Enable the Google Sheets API](https://console.cloud.google.com/apis/library/sheets.googleapis.com).
* Share your spreadsheet with the service account:
* Open the spreadsheet.
* Click **Share**.
* Add the service accountโs email with **Editor** or **Viewer** permission.
## Sheet Requirements
Each sheet must meet the following requirements:
* Sheet is not empty.
* The first row must contain column names (no duplicate column names).
* The following rows must contain only valid cell values:
* No [errorType](https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/other#ExtendedValue) or formulaType cells.
# HubSpot
Source: https://docs.mage.ai/data-integrations/sources/hubspot
## Configuration
To set up the HubSpot source, provide the following configuration parameters:
| Key | Description | Sample Value |
| -------------------- | ---------------------------------------------------------------------------------- | ---------------------- |
| `access_token` | Secret access token used to make authenticated API requests. | `my_token` |
| `disable_collection` | If `false`, disables the collection of anonymous usage metrics. | `false` |
| `request_timeout` | Timeout (in seconds) for API requests. | `300` |
| `start_date` | Cutoff date for syncing historical data. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2023-01-01T00:00:00Z` |
## How to Get `access_token`
Follow HubSpotโs [documentation](https://developers.hubspot.com/docs/api/private-apps) to create a Private App and generate your `access_token`.
### Scopes
Select all the **Read** scopes (except `crm.objects.feedback_submissions`) under the **CRM** section.
| Scope | Read |
| ------------------------------ | ---- |
| `crm.lists` | โ |
| `crm.objects.companies` | โ |
| `crm.objects.contacts` | โ |
| `crm.objects.custom` | โ |
| `crm.objects.deals` | โ |
| `crm.objects.line_items` | โ |
| `crm.objects.marketing_events` | โ |
| `crm.objects.owners` | โ |
| `crm.objects.quotes` | โ |
| `crm.schemas.companies` | โ |
| `crm.schemas.contacts` | โ |
| `crm.schemas.custom` | โ |
| `crm.schemas.deals` | โ |
| `crm.schemas.line_items` | โ |
| `crm.schemas.quotes` | โ |
# Instagram
Source: https://docs.mage.ai/data-integrations/sources/instagram
## Configuration
Connect to **Instagram Graph API** to sync media, stories, and insights for business accounts. Provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ------------------------------ | ----------------------------------------------------------------------- | ------------------------------ | -------- |
| `access_token` | OAuth access token with Instagram Graph API permissions. | `EAAGm0PX4ZCpsBA...` | โ |
| `ig_user_ids` | List of Instagram Business Account IDs to fetch data for. | `["1234567890", "9876543210"]` | โ |
| `media_insights_lookback_days` | Number of days to look back for media insights. | `60` | Optional |
| `start_date` | Start date (ISO 8601 format) to begin fetching media and insights data. | `2025-01-01T00:00:00Z` | โ |
| `metrics_log_level` | Log level for API metrics (e.g., `INFO`, `DEBUG`, `WARNING`). | `INFO` | Optional |
## How to Get Your `access_token`
You have two options to generate an access token:
### 1. Use the OAuth Button (Recommended)
* Click the **OAuth login** button in your Mage workspace.
* Approve the required permissions for Instagram:
* `instagram_basic`
* `instagram_manage_insights`
* `pages_show_list`
* The access token will be securely generated and automatically populated in the configuration.
> **Tip**: Use the OAuth flow for a smoother and safer authentication process without manually handling tokens.
### 2. Manually Generate Access Token
* Create a Facebook App via [Facebook Developer Portal](https://developers.facebook.com/).
* Add Instagram Basic Display API and Instagram Graph API products to your app.
* Generate a **User Access Token** via [Graph API Explorer](https://developers.facebook.com/tools/explorer/).
* Make sure to select the correct Instagram-related permissions.
* Convert the short-lived token to a **long-lived token** for production use.
## What is Instagram Graph API?
The [Instagram Graph API](https://developers.facebook.com/docs/instagram-api) is designed for businesses and creators to access media, insights, and metrics on Instagram accounts. It enables data-driven marketing and content analysis.
## Why Integrate Instagram with Mage?
* **Automate content extraction**: Sync media posts and stories.
* **Measure engagement**: Track likes, comments, saves, impressions, and reach.
* **Audience insights**: Understand follower growth and demographics.
* **Marketing analytics**: Integrate with other platforms for a 360-degree marketing view.
## Supported Streams
The Instagram source extracts the following resources:
* **media**: All published media objects (images, videos, carousels).
* **media\_children**: Children of carousel media posts.
* **media\_insights**: Metrics like reach, impressions, engagement for each media post.
* **stories**: Instagram Stories posted by the account.
* **story\_insights**: Insights for each Story, including taps forward, exits, impressions.
* **user\_insights\_28day**: Aggregated user insights over a 28-day window.
* **user\_insights\_daily**: Daily user insights.
* **user\_insights\_followers**: Follower count and demographic data.
* **user\_insights\_online\_followers**: Times of day when followers are online.
* **user\_insights\_weekly**: Weekly user insights.
* **users**: Profile data of the Instagram Business accounts.
# Intercom
Source: https://docs.mage.ai/data-integrations/sources/intercom
## Configuration
To set up the Intercom source, provide the following configuration parameters:
| Key | Description | Sample Value |
| -------------- | ------------------------------------------------------------------------------------ | --------------- |
| `access_token` | The access token used to authenticate your Intercom account. | `abcdefg123456` |
| `start_date` | Filters results to fetch only records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |
### How to Get `access_token`
Follow Intercomโs [instructions](https://developers.intercom.com/building-apps/docs/authentication-types#section-how-to-get-your-access-token)\
to obtain your `access_token`.
# Knowi
Source: https://docs.mage.ai/data-integrations/sources/knowi
## Configuration
To set up the Knowi source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ----------------- | ----------------------------------------------------------------------------------------------- | --------------- |
| `access_token` | (REQUIRED) The access token used to authenticate your Knowi account. | `abcdefg123456` |
| `request_timeout` | (OPTIONAL) The amount of time before the request times out (default: 300 seconds). | `300` |
| `user_agent` | (OPTIONAL) The User Agent string to send in the request header. | `my-app-v1.0` |
| `start_date` | (OPTIONAL) Filters results to fetch only records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |
### How to Get Your `access_token`
Follow Knowiโs [instructions](https://www.knowi.com/docs/managementAPI.html#APIAuthentication)\
to generate and retrieve your `access_token`.
# LinkedIn Ads
Source: https://docs.mage.ai/data-integrations/sources/linkedin_ads
## Configuration
To set up the LinkedIn Ads source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ----------------- | ----------------------------------------------------------------------------------------------- | ---------------------------- | ---------------------- |
| `accounts` | Comma-separated list of account IDs. Required if syncing `accounts` or `account_users` streams. | `"id1, id2, id3"` | Required (conditional) |
| `access_token` | Access token for authentication. | `def789...` | โ |
| `request_timeout` | Timeout (in seconds) for API requests. | `300` | |
| `start_date` | Fetch data newer than this `start_date`. Format: ISO8601 (`YYYY-MM-DDTHH:MM:SSZ`). | `2023-01-01T00:00:00Z` | โ |
| `user_agent` | User agent string (usually your email address). | `your_email@your_domain.com` | โ |
Instead of providing an `access_token`, you can also use the following OAuth credentials:
| Key | Description | Sample Value | Required |
| --------------- | -------------------- | ------------ | -------- |
| `client_id` | OAuth client ID. | `abc123...` | โ |
| `client_secret` | OAuth client secret. | `xyz456...` | โ |
| `refresh_token` | OAuth refresh token. | `def789...` | โ |
### How to Get `access_token`
1. Create a LinkedIn App: [LinkedIn Developers](https://www.linkedin.com/developers/apps).
2. Enable the **Marketing Developer Platform** product:\
[Request Access](https://learn.microsoft.com/en-us/linkedin/shared/authentication/getting-access?context=linkedin%2Fcontext#marketing).
3. Submit the access form and wait for approval (a few business days).
4. Once approved, follow these [instructions](https://www.linkedin.com/developers/tools/oauth) to generate an `access_token`.
# Microsoft Ads
Source: https://docs.mage.ai/data-integrations/sources/microsoft_ads
Sync performance reports from the Microsoft Advertising API (Bing Ads v13) using OAuth 2.0 and the official bingads Python SDK.
## Overview
This source syncs performance reports from the [Microsoft Advertising API](https://learn.microsoft.com/en-us/advertising/guides/get-started) (Bing Ads API **v13**) using the official [`bingads`](https://github.com/BingAds/BingAds-Python-SDK) Python SDK.
Reports are requested asynchronously: the connector submits a report request, polls until the result is ready, downloads the data, and emits one record per row. All three streams use **daily aggregation** by default and sync **incrementally** on `TimePeriod` per ad account, so only new or updated days are fetched on subsequent runs.
Authentication uses **OAuth 2.0** with a `refresh_token` (no service accounts). You provide an Azure AD application's `client_id` (and optionally `client_secret`) alongside a `developer_token` that identifies your API application.
## Configuration
| Key | Required | Default | Description |
| ----------------------- | -------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `developer_token` | โ | โ | Microsoft Advertising developer token. Identifies your application to the API. Obtained from [Microsoft Advertising API Center](https://developers.ads.microsoft.com/Account). |
| `client_id` | โ | โ | Azure AD **Application (client) ID** for your registered app. |
| `client_secret` | | โ | Azure AD client secret. Required for **confidential/web app** OAuth flows. Leave empty for **public/desktop app** flows. |
| `refresh_token` | โ | โ | OAuth refresh token scoped to `https://ads.microsoft.com/msads.manage`. See [Authentication with OAuth](https://learn.microsoft.com/en-us/advertising/guides/authentication-oauth). |
| `oauth_redirection_uri` | | `https://login.microsoftonline.com/common/oauth2/nativeclient` | Redirect URI registered in Azure AD. Must match the URI used when obtaining the refresh token. Required when `client_secret` is set. |
| `customer_id` | โ | โ | Microsoft Advertising **manager account (customer) ID** the user operates under. |
| `account_ids` | โ | โ | Comma-separated **advertiser account IDs** to sync. Find these in the Microsoft Advertising UI under account settings. |
| `start_date` | โ | โ | Reporting lower bound. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). |
| `end_date` | | today (UTC) | Optional reporting upper bound. |
| `environment` | | `production` | API environment: `production` or `sandbox`. |
| `aggregation` | | `Daily` | Report aggregation period. Common values: `Daily`, `Weekly`, `Monthly`. |
| `report_time_zone` | | โ | [ReportTimeZone](https://learn.microsoft.com/en-us/advertising/reporting-service/reporttimezonetype) value. Leave empty to use the account's default time zone. |
| `poll_interval_ms` | | `5000` | Milliseconds between status polls while the async report is generating. |
| `report_timeout_ms` | | `3600000` | Maximum milliseconds to wait for report download before failing (default 1 hour). |
> **Legacy key:** `accounts` is accepted as an alias for `account_ids` for backwards compatibility.
## Sample configs
**Minimal โ production environment, daily reports**:
```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
refresh_token: YOUR_REFRESH_TOKEN
customer_id: "12345678"
account_ids: "111111111,222222222"
start_date: "2025-01-01T00:00:00Z"
```
**Confidential app flow (with client secret)**:
```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
client_secret: YOUR_CLIENT_SECRET
refresh_token: YOUR_REFRESH_TOKEN
oauth_redirection_uri: https://login.microsoftonline.com/common/oauth2/nativeclient
customer_id: "12345678"
account_ids: "111111111"
start_date: "2025-01-01T00:00:00Z"
```
**Sandbox testing**:
```yaml theme={"system"}
developer_token: YOUR_DEVELOPER_TOKEN
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
refresh_token: YOUR_SANDBOX_REFRESH_TOKEN
customer_id: "12345678"
account_ids: "111111111"
start_date: "2025-01-01T00:00:00Z"
environment: sandbox
```
## Streams
| Stream | Report type | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| `campaign_performance_report` | [CampaignPerformanceReportRequest](https://learn.microsoft.com/en-us/advertising/reporting-service/campaignperformancereportrequest) | Daily campaign-level impressions, clicks, spend, and conversions. |
| `ad_group_performance_report` | [AdGroupPerformanceReportRequest](https://learn.microsoft.com/en-us/advertising/reporting-service/adgroupperformancereportrequest) | Daily ad group-level performance metrics. |
| `ad_performance_report` | [AdPerformanceReportRequest](https://learn.microsoft.com/en-us/advertising/reporting-service/adperformancereportrequest) | Daily ad-level performance metrics. |
All streams sync **incrementally** on `TimePeriod` (one bookmark per ad account). On the first run the full history from `start_date` is fetched; subsequent runs fetch only new or updated days.
## How to get credentials
### 1. Developer token
1. Sign in to [Microsoft Advertising](https://ads.microsoft.com/) as the account owner or a user with API access.
2. Go to **Tools โ API Center** (or visit the [Microsoft Advertising API Center](https://developers.ads.microsoft.com/Account) directly).
3. Request and copy your developer token.
### 2. Azure AD app (`client_id` / `client_secret`)
1. Open [Azure Portal โ App registrations](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade) and click **New registration**.
2. Choose **Public client / native** for desktop flows (no `client_secret` required) or **Web** for server flows.
3. Add `https://login.microsoftonline.com/common/oauth2/nativeclient` as a redirect URI (or your own URI for web flows).
4. Copy the **Application (client) ID** โ `client_id`.
5. For web flows: go to **Certificates & secrets** and create a client secret โ `client_secret`.
### 3. Refresh token
Use the [Microsoft Advertising OAuth guide](https://learn.microsoft.com/en-us/advertising/guides/authentication-oauth) to complete the authorization code flow with the scope `https://ads.microsoft.com/msads.manage offline_access`. Copy the resulting `refresh_token` into your config.
### 4. `customer_id` and `account_ids`
Find these in the [Microsoft Advertising UI](https://ui.ads.microsoft.com):
* **Customer ID** โ shown in the top-right account selector next to your account name.
* **Account IDs** โ visible in **Accounts Summary** or each account's settings page.
## Related links
* [Microsoft Advertising API โ Get Started](https://learn.microsoft.com/en-us/advertising/guides/get-started)
* [Authentication with OAuth](https://learn.microsoft.com/en-us/advertising/guides/authentication-oauth)
* [Reporting Service overview](https://learn.microsoft.com/en-us/advertising/reporting-service/reporting-service-reference)
* [ReportTimeZone values](https://learn.microsoft.com/en-us/advertising/reporting-service/reporttimezonetype)
# Mode
Source: https://docs.mage.ai/data-integrations/sources/mode
## Configuration
To set up the Mode source, provide the following configuration parameters:
| Key | Description | Sample Value |
| -------------- | ------------------------------------------------------------------------------------ | --------------------- |
| `access_token` | Access token used to authenticate your Mode account. | `abcdefg123456` |
| `password` | Mode token password. | `supersecurepassword` |
| `workspace` | Workspace ID in Mode. | `ws-12345` |
| `user_agent` | User agent string to send in the request header. | `my-app-v1.0` |
| `start_date` | Filters results to only fetch records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |
### How to Get Your `access_token` and `password`
Follow Modeโs [instructions](https://mode.com/developer/api-reference/authentication/)\
to generate your `access_token` and `password`.
# Monday
Source: https://docs.mage.ai/data-integrations/sources/monday
## Configuration
To set up the Monday source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------- | ----------------------------------------------------------- | ---------------------------- |
| `api_token` | API token used for authentication. | `abcdefghijklmnopqrstuvwxyz` |
| `board_id` | ID of the board to query board-related data. | `12345678` |
| `board_limit` | Number of boards returned per page for the `boards` stream. | `25` |
## How to Get Your `api_token`
Follow [Mondayโs documentation](https://developer.monday.com/api-reference/docs/authentication#accessing-api-tokens)\
to retrieve your API token.
# MongoDB
Source: https://docs.mage.ai/data-integrations/sources/mongodb
## Configuration
To set up the MongoDB source, provide the following configuration parameters:
| Key | Description | Sample Value |
| --------------- | ------------------------------------------------------------------------------- | --------------- |
| `database` | Name of the database you want to read data from. | `company` |
| `host` | Hostname of your MongoDB server. | `127.0.0.1` |
| `password` | Password for the user to access the database. | `abc123...` |
| `port` | Port on which MongoDB is running (typically `27017`). | `27017` |
| `user` | Username that will access the database. | `admin` |
| `replica_set` | (Optional) Name of the replica set, if any. | `optional` |
| `ssl` | (Optional) Use SSL for the connection. Default is `false`. | `false` |
| `authSource` | (Optional) Authentication source database name. | `the_database` |
| `authMechanism` | (Optional) Authentication mechanism. Default is none. Example: `SCRAM-SHA-256`. | `SCRAM-SHA-256` |
## Run MongoDB Locally
You can run a local instance of MongoDB using Docker:
```bash theme={"system"}
docker run -p 27017:27017 \
-e MONGO_INITDB_ROOT_USERNAME=admin \
-e MONGO_INITDB_ROOT_PASSWORD=password \
--name mongo \
mongo:latest
```
# MSSQL (Microsoft SQL Server)
Source: https://docs.mage.ai/data-integrations/sources/mssql
## Configuration
To set up the MSSQL source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ---------- | ------------------------------------------------------------------------------------------------------ | ----------------- |
| `database` | Name of the database you want to read data from. | `msdb` |
| `host` | Hostname of the MSSQL database, including port number at the end (e.g., `host,port`). | `172.20.0.2,1433` |
| `password` | Password for the MSSQL user to access the database. | `abc123...` |
| `port` | Port where the database is running (typically `1433`). | `1433` |
| `schema` | Schema of the database you want to read data from. | `public` |
| `username` | Username that will access the database (must have read and write permissions to the specified schema). | `guest` |
### Optional Configuration
| Key | Description | Sample Value |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `authentication` | Authentication mode for connecting to MSSQL with [Azure Active Directory authentication](https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver16). | `ActiveDirectoryServicePrincipal` |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Adjust if your instance has more memory. | `50000` |
| `driver` | ODBC [driver](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16) for SQL Server. | `ODBC Driver 18 for SQL Server` |
| `connection_method` | Connection method. Set to `ssh_tunnel` to connect via SSH tunnel. Default: `direct` | `ssh_tunnel` |
| `ssh_host` | (SSH tunnel only) The host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (SSH tunnel only) The port of the intermediate bastion server. Default: `22` | `22` |
| `ssh_username` | (SSH tunnel only) The username used to connect to the bastion server. | `username` |
| `ssh_password` | (SSH tunnel only, optional) The password used to connect to the bastion server. Required if not using `ssh_pkey`. | `password` |
| `ssh_pkey` | (SSH tunnel only, optional) The path to the private key used to connect to the bastion server. Required if not using `ssh_password`. | `/path/to/private/key` |
SSH tunneling is a Mage Pro only feature.
# MySQL
Source: https://docs.mage.ai/data-integrations/sources/mysql
## Configuration
To set up the MySQL source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| `database` | Name of the database you want to read data from. | `demo` |
| `host` | Hostname of your MySQL server. | `mage.abc.us-west-2.rds.amazonaws.com` |
| `port` | Port on which the database is running (typically `3306`). | `3306` |
| `username` | Username with permissions to read and write to the specified schema. | `root` |
| `password` | Password for the user to access the database. | `abc123...` |
| `connection_method` | Method used to connect to MySQL server, either `direct` or `ssh_tunnel`. | `direct` or `ssh_tunnel` |
| `conn_kwargs` | (Optional) Extra [connection keyword arguments](https://dev.mysql.com/doc/connector-python/en/connector-python-connectargs.html) in dictionary format. | `{"ssl_ca": "CARoot.pem", "ssl_cert": "certificate.pem", "ssl_key": "key.pem"}` |
| `ssh_host` | (Optional) Host of the intermediate bastion server. | `123.45.67.89` |
| `ssh_port` | (Optional) Port of the bastion server. Default is `22`. | `22` |
| `ssh_username` | (Optional) Username used to connect to the bastion server. | `username` |
| `ssh_password` | (Optional) Password for bastion authentication (if using password). | `password` |
| `ssh_pkey` | (Optional) Path to the private key file for bastion authentication (if using a private key). | `/path/to/private/key` |
### Optional Configuration
| Key | Description | Sample Value |
| ------------------- | ------------------------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Adjust higher if your instance has more memory. | `50000` |
# OracleDB
Source: https://docs.mage.ai/data-integrations/sources/oracledb
## Configuration
To set up the OracleDB source, provide the following configuration parameters:
| Key | Description | Sample Value |
| ---------- | ----------------------------------------------------------------------------- | ---------------------- |
| `host` | OracleDB host. | `oracledb.example.com` |
| `port` | OracleDB port (defaults to `1521`). | `1521` |
| `service` | Listener service name on the database server for client connections. | `xepdb1` |
| `password` | Password for the user. | `xyz123` |
| `user` | Username with permission to connect and query the database. | `xyz123` |
| `mode` | Connection mode for the Oracle client. Accepted values are `thin` or `thick`. | `thick` |
### Optional Configuration
| Key | Description | Sample Value |
| ------------------- | ------------------------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Adjust higher if your instance has more memory. | `50000` |
# Outreach
Source: https://docs.mage.ai/data-integrations/sources/outreach
## Config
You must enter the following credentials when configuring this source:
| Key | Description | Sample value | Required |
| --------------- | ------------------------------------------ | ---------------------- | -------- |
| `client_id` | ... | `abc123...` | โ |
| `client_secret` | ... | `xyz456...` | โ |
| `page_size` | ... | ... | |
| `quota_limit` | ... | ... | |
| `redirect_uri` | ... | `https://www.mage.ai` | โ |
| `refresh_token` | ... | `def789...` | โ |
| `start_date` | A cutoff date for syncing historical data. | `2023-01-01T00:00:00Z` | โ |
### How to get `refresh_token`
Follow [Outreachโs instructions](https://api.outreach.io/api/v2/docs).
#### Get `code`
```
https://api.outreach.io/oauth/authorize?client_id=&redirect_uri=&response_type=code&scope=
```
Sample URL:
```
https://accounts.outreach.io/oauth/authorize?client_id=...&redirect_uri=https%3A%2F%2Fwww.mage.ai&response_type=code&scope=accounts.read+calls.read+duties.read+events.read+mailboxes.read+mailings.read+opportunities.read+personas.read+prospects.read+sequences.read+stages.read+tasks.read+teams.read+users.read
```
#### Exchange `code` for `refresh_token`
```curl theme={"system"}
curl https://api.outreach.io/oauth/token \
-X POST \
-d client_id= \
-d client_secret= \
-d redirect_uri= \
-d grant_type=authorization_code \
-d code=
```
Sample response:
```json theme={"system"}
{
"access_token": ,
"token_type": "bearer",
"expires_in": 7200,
"refresh_token": ,
"scope": ,
"created_at": 1503301100
}
```
## Streams
### Prospects
The column `contactHistogram` in the stream `prospects` have been removed because the
column type (array of arrays) isnโt supported in destinations at the moment.
# Sources technical documentation
Source: https://docs.mage.ai/data-integrations/sources/overview
A source is a system that you want to load data from and synchronize it into another system. A source can be a 3rd party API, SaaS, database, data warehouse, or a data lake.
## Singer spec
Mage uses the data engineering community standard for data integrations called the
[Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
In addition,
Mage further standardizes the spec and provides common classes and methods to make implementing them easier and faster.
## Folder structure
Sources are defined in the `mage_integrations/sources/` directory.
All sources should be a subclass of the `Source` class defined in
`mage_integrations/sources/base.py`.
## Contribute
Want to learn about contributing a source? Read on [here](/contributing/data-integrations/add-new-source)!
# Paystack
Source: https://docs.mage.ai/data-integrations/sources/paystack
## Configuration
Connect to **Paystack**, a leading payments gateway in Africa, and automate your payment data extraction with the following configuration parameters:
| Key | Description | Sample Value |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `start_date` | Start date in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) to begin pulling transaction and payment data. If bookmarks are available, Mage prioritizes the latest bookmark. | `2010-01-01T00:00:00Z` |
| `secret_key` | Secret key from your Paystack dashboard used to authenticate API requests. | `fake_api_key...` |
| `request_timeout` | (Optional) Timeout (in seconds) for API requests to Paystack (default: 300 seconds). | `300` |
## What is Paystack?
[Paystack](https://paystack.com/) is a modern payments platform helping businesses in Africa collect online and offline payments seamlessly. With Paystack, businesses can accept payments via credit card, debit card, bank transfers, and mobile money.
## Why Integrate Paystack with Mage?
* **Automated financial data ingestion**: Pull transaction records, payment events, and customer data directly into your data warehouse.
* **Real-time analytics**: Enable up-to-date reporting and insights on payment performance, transaction trends, and customer behavior.
* **Secure and compliant**: OAuth and API key authentication ensure secure access to sensitive payment data.
* **Scalable extraction**: Handle large volumes of payment data efficiently for growing businesses.
# Pipedrive
Source: https://docs.mage.ai/data-integrations/sources/pipedrive
## Configuration
To connect to the Pipedrive CRM API and set up the Pipedrive source in Mage, provide the following configuration parameters:
| Key | Description | Sample Value |
| ------------ | --------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `start_date` | The start date in ISO format (`YYYY-MM-DDTHH:MM:SSZ`) to begin syncing data from Pipedrive (for supported resources). | `2010-01-01T00:00:00Z` |
| `api_key` | API key used to authenticate your Pipedrive account and access CRM data. | `fake_api_key...` |
## What is Pipedrive?
[Pipedrive](https://www.pipedrive.com/) is a leading customer relationship management (CRM) platform designed to help sales teams manage their pipelines efficiently. Integrating Pipedrive with Mage allows you to automate the extraction of CRM data like deals, leads, contacts, and activities for reporting and analytics.
## How to Get Your `api_key`
1. Log in to your [Pipedrive account](https://app.pipedrive.com/).
2. Click on your profile picture (top right) โ **Company Settings** โ **Personal Preferences** โ **API**.
3. Copy your personal API token.
For more information, refer to [Pipedrive's API documentation](https://pipedrive.readme.io/docs/core-api-concepts-authentication).
## Why Integrate Pipedrive with Mage?
* **Automate data extraction**: Schedule and sync CRM data without manual intervention.
* **Custom analytics**: Build advanced reports using Pipedrive data.
* **Sales performance insights**: Analyze deals, stages, activities, and forecasts.
* **Unified data warehouse**: Combine Pipedrive data with other business data for holistic reporting.
## Supported Streams in Pipedrive
Mageโs Pipedrive integration extracts data from the following objects (streams):
* Activities
* Activity Types
* Currency
* Deal Products
* Deals
* Dealflow
* Filters
* Files
* Notes
* Organizations
* Persons
* Pipelines
* Products
* Stages
* Users
# PostgreSQL
Source: https://docs.mage.ai/data-integrations/sources/postgresql
## Configuration
To set up the PostgreSQL source, provide the following connection parameters:
| Key | Description | Sample Value |
| ------------------ | ---------------------------------------------------------------------- | ------------ |
| `database` | Name of the database you want to extract data from. | `demo` |
| `host` | Hostname of your PostgreSQL database. | `db.bit.io` |
| `password` | Password for the PostgreSQL user. | `abc123...` |
| `port` | Port number where the database is accessible (typically `5432`). | `5432` |
| `schema` | Schema containing the tables you want to extract. | `public` |
| `username` | Username with permissions to read and write from the specified schema. | `guest` |
| `replication_slot` | Logical replication slot name for Change Data Capture (CDC). | `mage_slot` |
| `publication_name` | Logical replication publication name for CDC. | `mage_pub` |
### Optional Configuration
| Key | Description | Sample Value |
| ------------------- | ----------------------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch per batch (default is 50,000). Adjust if your instance has sufficient memory. | `50000` |
## Why Use PostgreSQL as a Source in Mage?
* **Real-time change data capture (CDC)**: Capture and sync only changed rows.
* **Reliable large-scale data ingestion**: Load millions of rows efficiently.
* **Flexible schema support**: Work with multiple schemas in your PostgreSQL instance.
* **Secure and scalable**: Support for SSL connections, logical replication slots, and publication management.
## Change Data Capture (CDC)
Mage supports 2 types of change data capture with PostgreSQL:
1. Batch query
2. Log replication
### Batch query
Mage will query PostgreSQL in batches using `SELECT`, `WHERE`, and `ORDER BY`
statements.
### Log replication
Mage supports **Change Data Capture (CDC)** for PostgreSQL using **logical replication**,
enabling real-time tracking of inserts, updates, and deletes without requiring full table reloads.
By reading PostgreSQLโs Write-Ahead Logs (WAL), Mage captures changes efficiently and applies them to the
destination system โ inserting new records, updating existing ones, or deleting as needed โ
ensuring low-latency, incremental data synchronization.
#### Setup in PostgreSQL
1. Enable logical replication in Postgres config
1. Local Postgres
1. Open the `postgresql.conf` file. Here is an example location on Mac OSX:
`/Users/Mage/Library/Application Support/Postgres/var-14/postgresql.conf`.
2. Under the settings section, change the value of `wal_level` to `logical`. The line in your
`postgresql.conf` file should look like this:
```text theme={"system"}
wal_level = logical
```
3. Restart the PostgreSQL service or database. You can do this via the PostgreSQL app or if youโre
on Linux, run the following commands:
```bash theme={"system"}
sudo service postgresql stop
sudo service postgresql start
```
4. Run the following query in your PostgreSQL database: `SHOW wal_level`. The result should be:
| `wal_level` |
| ----------- |
| `logical` |
2. AWS RDS/Aurora
1. Follow this [doc](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.html#AuroraPostgreSQL.Replication.Logical.Configure)
to set up PostgreSQL logical replication for an Aurora PostgreSQL DB cluster
2. Logical replication for AWS RDS (PostgreSQL) can be set up in a similar way with the Aurora cluster.
3. Azure Database for PostgreSQL
1. Azure Database for PostgreSQL supports logical replication for **Flexible Server** and **Single Server** deployments.
2. Follow this [Azure guide](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical-replication) to enable logical replication on Azure PostgreSQL Flexible Server.
3. Make sure the following parameters are set in the Azure portal:
* `wal_level` = `logical`
* `max_replication_slots` โฅ 1
* `max_wal_senders` โฅ 1
4. Grant replication privileges to your user:
```sql theme={"system"}
ALTER ROLE WITH REPLICATION;
```
5. Azure also requires that you configure firewall rules to allow your Mage server or IP range to access the PostgreSQL server.
6. For more detailed instructions, see: [Set up logical replication on Azure PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical-replication).
2. Run the following command in PostgreSQL to create a replication slot:
```sql theme={"system"}
SELECT pg_create_logical_replication_slot('mage_slot', 'pgoutput');
```
The replication slot name `mage_slot` quoted is used as an example in the guide. The actual replication slot name should be unique for each pipeline source.
The result should looking something like this:
| `pg_create_logical_replication_slot` |
| ------------------------------------ |
| `(mage_slot,0/51A80778)` |
3. Create a publication for all tables or for 1 specific table using the following commands:
```sql theme={"system"}
CREATE PUBLICATION mage_pub FOR ALL TABLES;
```
`mage_pub` is used in Mageโs [code](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/sources/postgresql/__init__.py#L126)
or for 1 table:
```sql theme={"system"}
CREATE PUBLICATION mage_pub FOR TABLE some_schema.some_table_name;
```
Replace `some_schema` with the schema of the table and `some_table_name` with the name
of the table you want to replicate.
Run the following command to add or remove a table from publication.
```sql theme={"system"}
ALTER PUBLICATION mage_pub ADD/DROP TABLE table_name;
```
4. Verify that the publication was created successfully by running the following command in PostgreSQL:
```sql theme={"system"}
SELECT * FROM pg_publication_tables;
```
The result should looking something like this:
| `pubname` | `schemaname` | `tablename` |
| ---------- | ------------ | ----------- |
| `mage_pub` | `public` | `users` |
5. Grant the database user permission to read the replication slot:
```sql theme={"system"}
ALTER ROLE WITH REPLICATION;
```
#### Create data integration pipeline in Mage
Follow this [guide to create a data integration pipeline](/guides/data-integration-pipeline)
in Mage.
However, choose PostgreSQL as the source and choose `LOG_BASED` as the
replication method.
#### Testing pipeline end-to-end
Once youโve created the pipeline, add a few rows into your PostgreSQL table that you just
created a logical replication for.
You can use the `INSERT` command to add rows. For example:
```sql theme={"system"}
INSERT INTO some_schema.some_table_name
VALUES (1, 2, 3)
```
Replace `some_schema` with the schema of the table and `some_table_name` with the name
of the table you want to replicate.
Change the `VALUES` to match the columns in your table.
##### Verify replication logs being created
Run the following commands in PostgreSQL to check for new logs:
```sql theme={"system"}
SELECT
*
FROM pg_logical_slot_peek_binary_changes('mage_slot', null, null, 'proto_version', '1', 'publication_names', 'mage_pub');
```
##### Run sync
After you added a few new rows, [create a trigger](/guides/triggering-pipelines#trigger)
to start running your pipeline and begin syncing data.
## What is PostgreSQL?
[PostgreSQL](https://www.postgresql.org/) is an open-source, enterprise-grade relational database system. Known for its robustness, extensibility, and SQL compliance, PostgreSQL powers thousands of applications worldwide in both small businesses and large enterprises.
Common use cases include:
* Transactional applications
* Analytical workloads
* Hybrid OLTP + OLAP systems
* Data warehousing
* Business intelligence (BI) and reporting pipelines
# Postmark
Source: https://docs.mage.ai/data-integrations/sources/postmark
## Configuration
To connect to the Postmark API and extract email event data, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| ----------------------- | -------------------------------------------------------------------- | ------------ | -------- |
| `postmark_server_token` | Your Postmark server's API token used to authenticate requests. | `abc123...` | โ |
| `start_date` | Cutoff date to start syncing historical data (format: `YYYY-MM-DD`). | `2023-01-01` | โ |
## What is Postmark?
[Postmark](https://postmarkapp.com/) is a reliable transactional email service for sending and tracking important application-generated emails, such as password resets, receipts, welcome emails, and more.\
Integrating Postmark with Mage allows you to automate the extraction of email delivery events, bounce information, and open/click tracking into your data warehouse for advanced analytics.
## How to Get Your `postmark_server_token`
1. Log in to your [Postmark account](https://postmarkapp.com/).
2. Navigate to **Servers** โ **API Tokens**.
3. Copy the **Server API Token** you want to use.
For more information, see Postmark's [API token documentation](https://postmarkapp.com/support/article/1008-what-are-the-account-and-server-api-tokens).
## Why Integrate Postmark with Mage?
* **Automated email tracking**: Extract delivery, open, click, and bounce events.
* **Centralized analytics**: Consolidate Postmark email data with other sources in your data warehouse.
* **Improve deliverability**: Analyze bounce reasons and optimize your sending practices.
* **Real-time monitoring**: Stay updated on the performance of transactional emails.
* **Scalable architecture**: Sync millions of email events without manual intervention.
## Supported Streams from Postmark
When integrated with Mage, the following Postmark data streams are available:
* **messages\_opens** โ Track when recipients open your emails.
* **messages\_outbound** โ Retrieve details of outbound messages sent via Postmark.
* **stats\_outbound\_bounces** โ Statistics on email bounces (hard, soft bounces).
* **stats\_outbound\_clients** โ Email client analytics (e.g., Gmail, Outlook, Apple Mail).
* **stats\_outbound\_overview** โ High-level summary of outbound email performance.
* **stats\_outbound\_platform** โ Platform analytics showing devices used by recipients.
# Power BI
Source: https://docs.mage.ai/data-integrations/sources/powerbi
## Configuration
To set up the Power BI source and connect to your workspace, provide the following configuration parameters:
| Key | Description | Sample Value |
| ----------------- | --------------------------------------------------------------------------------------------- | --------------- |
| `access_token` | (REQUIRED) Access token used to authenticate with your Power BI account. | `abcdefg123456` |
| `request_timeout` | (OPTIONAL) Timeout (in seconds) for API requests (default: 300). | `300` |
| `user_agent` | (OPTIONAL) User Agent string to send in the request header. | `my-app-v1.0` |
| `start_date` | (OPTIONAL) Filter results to only sync records updated after this date. Format: `YYYY-MM-DD`. | `2023-01-01` |
## What is Power BI?
[Power BI](https://powerbi.microsoft.com/) is Microsoftโs leading business intelligence (BI) platform, enabling organizations to visualize data, create interactive dashboards, and generate real-time insights.\
Integrating Power BI with Mage allows you to automate data extraction, sync analytics reports, and merge insights across your enterprise data ecosystem.
## How to Get Your `access_token`
Follow Power BIโs [official instructions](https://community.fabric.microsoft.com/t5/Developer/REST-API-Get-Access-Token/m-p/1895937/) to generate an OAuth `access_token` via the Azure Active Directory (Azure AD) app registration process.
## Why Integrate Power BI with Mage?
* **Automate report extraction**: Fetch dashboards, reports, and datasets seamlessly.
* **Centralize business analytics**: Combine Power BI data with data warehouses and lakes.
* **Real-time decision-making**: Access up-to-date analytics without manual refreshes.
* **Enhanced scalability**: Manage large-scale BI deployments with streamlined data syncs.
* **Boost productivity**: Eliminate manual export processes for business users and analysts.
## Supported Streams from Power BI
When integrated with Mage, Power BI enables access to the following data streams:
* **apps** โ Metadata about published Power BI apps in your organization.
* **dashboards** โ Details of dashboards created and shared within workspaces.
* **datasets** โ Information about datasets, including refresh schedules and models.
* **groups** โ Workspaces and group information for collaborative projects.
* **reports** โ Metadata on reports built and published in Power BI.
* **tiles** โ Tiles on dashboards representing visuals and pinned content.
# QuickBooks
Source: https://docs.mage.ai/data-integrations/sources/quickbooks
## Configuration
To set up the QuickBooks source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| --------------- | ------------------------------------------------------------------------------------------------ | ---------------------- | -------- |
| `client_id` | Client ID associated with your QuickBooks app. | `123456` | โ |
| `client_secret` | Client secret generated when you create your QuickBooks app. | `abcdefg` | โ |
| `start_date` | The start date for syncing data. Format: `YYYY-MM-DDTHH:MM:SSZ`. | `2021-01-01T00:00:00Z` | โ |
| `user_agent` | User agent name registered for your QuickBooks app. | `agent_name` | โ |
| `realm_id` | Realm ID associated with your QuickBooks company account. | `123456` | โ |
| `refresh_token` | Refresh token used to renew the access token after expiration (requires `offline_access` scope). | `abcdefg` | โ |
## How to Generate Credentials
Follow QuickBooks' [OAuth 2.0 Authentication Playground guide](https://developer.intuit.com/app/developer/qbo/docs/develop/authentication-and-authorization/oauth-2.0-playground) to generate the required credentials:
1. Register a new QuickBooks app to get your `client_id` and `client_secret`.
2. Configure the OAuth scopes required by your app.
3. Use the OAuth 2.0 playground to complete authentication and retrieve your `refresh_token` and `realm_id`.
## Additional Notes
* The `start_date` determines how far back data will be synced.
* Ensure that the `offline_access` scope is enabled to obtain a `refresh_token`.
* Tokens expire after a set period; use the `refresh_token` to obtain new access credentials automatically.
# Amazon Redshift
Source: https://docs.mage.ai/data-integrations/sources/redshift
## Configuration
To connect to your Amazon Redshift cluster and extract data, provide the following configuration parameters:
| Key | Description | Sample Value |
| ---------- | -------------------------------------------------------------------- | ---------------------------------------------- |
| `database` | Name of the database you want to load data from. | `demo` |
| `host` | Hostname of your Redshift cluster. | `mage-prod.3.us-west-2.redshift.amazonaws.com` |
| `password` | Password for the database user. | `abc123...` |
| `port` | Port where Redshift is running (typically `5439`). | `5439` |
| `region` | AWS region where your Redshift cluster is located. | `us-west-2` |
| `schema` | Schema containing the tables you want to extract. | `public` |
| `user` | Username with permissions to read and write to the specified schema. | `awsuser` |
### IAM Authentication (Optional)
Alternatively, use IAM database authentication by providing the following parameters:
| Key | Description | Sample Value |
| -------------------- | --------------------------------------------------- | ------------ |
| `access_key_id` | AWS access key for the IAM role or IAM user. | `abc123...` |
| `cluster_identifier` | Cluster identifier of the Amazon Redshift cluster. | `mage-prod` |
| `db_user` | Database user for Redshift IAM authentication. | `admin` |
| `secret_access_key` | AWS secret access key for the IAM role or IAM user. | `xyz123` |
### Optional Configuration
| Key | Description | Sample Value |
| ------------------- | ---------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch per batch (defaults to 50,000). Tune higher for larger clusters. | `50000` |
## What is Amazon Redshift?
[Amazon Redshift](https://aws.amazon.com/redshift/) is a fully managed, petabyte-scale cloud data warehouse solution designed for large-scale data storage and analytics. Redshift delivers fast query performance by using columnar storage, massively parallel processing (MPP), and advanced compression.
Integrating Amazon Redshift with Mage allows you to automate data extraction from Redshift clusters, streamline ETL pipelines, and centralize data for analytics and business intelligence workloads.
## Why Integrate Amazon Redshift with Mage?
* **High-performance data extraction**: Load massive datasets at scale.
* **Flexible authentication**: Connect using either traditional credentials or secure IAM authentication.
* **Automated ETL pipelines**: Move data from Redshift to your data lake or warehouse with zero manual effort.
* **Cost-efficient analytics**: Optimize Redshift query performance and storage.
* **Unified data workflows**: Combine Redshift data with other sources to power real-time dashboards and reports.
# Salesforce
Source: https://docs.mage.ai/data-integrations/sources/salesforce
## Config
You must enter the following settings when configuring this source:
| Key | Description | Sample value |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `api_type` | The `api_type` is used to switch the behavior of the tap between using Salesforce's "REST" and "BULK" APIs. When new fields are discovered in Salesforce objects, the `select_fields_by_default` key describes whether or not the tap will select those fields by default. | `REST`, `BULK` |
| `domain` | Your Salesforce instance domain. Use 'login' (default) or 'test' (sandbox), or Salesforce My domain. | `test`, `login` |
| `select_fields_by_default` | If `true`, the fields in a schema of a stream will all be selected by default when setting up a synchronization. | `true`, `false` |
| `start_date` | The `start_date` is used by the tap as a bound on SOQL queries when searching for records. This should be this exact format `YYYY-mm-ddTHH:MM:SSZ`. | `2022-11-30T21:31:20Z` |
If using `OAuth` based authentication:
| Key | Description | Sample value |
| --------------- | ------------------------------------------------------------- | ------------ |
| `client_id` | OAuth Salesforce App secrets. | `ABC1...` |
| `client_secret` | OAuth Salesforce App secrets. | `ABC1...` |
| `refresh_token` | The `refresh_token` is a secret created during the OAuth flow | `ABC1...` |
If using `Password` based authentication:
| Key | Description | Sample value |
| ---------------- | --------------------------------- | ------------ |
| `username` | Salesforce account username | `ABC1...` |
| `password` | Salesforce account password | `ABC1...` |
| `security_token` | Salesforce account security token | `ABC1...` |
Optional settings:
| Key | Description | Sample value |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `threshold` | When running `INCREMENTAL` sync runs, threshold is used to throttle how often STATE messages are generated (in `REST` api\_type).This can be useful to minimize the amount of `STATE` messages being generated. | Defaults to 1000 |
| `streams` | List of stream names to be discovered inside the salesforce tap. if none is given, the tap will search for all available streams, which can take several minutes. | \["Account"] |
### How to get your `client_id`, `client_secret`, and `refresh_token`
#### Find your `client_id` and `client_secret`
1. Sign into your Salesforce account.
2. Create a Connected App and Enable OAuth Settings for API Integration
by following this [Salesforce documentation](https://help.salesforce.com/s/articleView?id=sf.connected_app_create_api_integration.htm\&type=5).
1. Use the following URL as the callback URL: [https://login.salesforce.com/services/oauth2/success](https://login.salesforce.com/services/oauth2/success)
3. When you are at the step to choose which OAuth Scopes to authorize, you must give your
Connected App at least these 2 scopes:
1. Manage (`api`)
2. Perform requests at any time (`refresh_token`)
4. After you created a Connected App, go to your Setup and click on the Home tab.
5. On the left sidebar under the Platform tools > Apps > App Manager, click on App Manager.
6. Find the row that contains the app you just created. On the furthest right column, click the dropdown caret icon and click View.
7. Under the API (Enable OAuth Settings) section, find the subsection labeled Consumer Key and Secret and click the button Manage Consumer Details.
8. A new page will open, displaying your Consumer Key (which is your `client_id`) and Consumer Secret (which is your `client_secret`).
#### Authorize your Connected App
1. Construct a URL with the following format:
`https://[your_salesforce_domain].my.salesforce.com/services/oauth2/authorize?client_id=[client_id]&redirect_uri=https://login.salesforce.com/services/oauth2/success&response_type=code`
| Variable | Description | Sample value |
| -------------------------- | -------------------------------------- | ------------- |
| `[your_salesforce_domain]` | The domain of your Salesforce account. | `mage-dev-ed` |
| `[client_id]` | Your Consumer Key | `ABC123...` |
2. Open that URL in a browser.
3. Authorize your Connected App to have API access to your Salesforce account.
4. Once you authorize, youโll be redirect to a URL like this:
`https://login.salesforce.com/services/oauth2/success?code=[code]`.
5. Note the value of the `code` URL parameter, itโll be used to get a `refresh_token`.
#### Get a `refresh_token`
Open a terminal and make the following POST request:
```bash theme={"system"}
curl -X POST https://[your_salesforce_domain].my.salesforce.com/services/oauth2/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=authorization_code&code=[code]&client_id=[client_id]&client_secret=[client_secret]&redirect_uri=https://login.salesforce.com/services/oauth2/success"
```
Change the following URL parameters to match your credentials:
| URL parameter | Description | Sample value |
| -------------------------- | ----------------------------------------------------------------- | ------------- |
| `[your_salesforce_domain]` | The domain of your Salesforce account. | `mage-dev-ed` |
| `[code]` | The code you received after authorizing your Connected App | `ABC123...` |
| `[client_id]` | Your Consumer Key | `ABC123...` |
| `[client_secret]` | Your Consumer Secret | `ABC1...` |
Once you execute the above CURL command successfully,
youโll receive a response that could look like this:
```json theme={"system"}
{
"access_token": "...",
"refresh_token": "...",
"signature": "...",
"scope": "...",
"id_token": "...",
"instance_url": "https://[your_salesforce_domain].my.salesforce.com",
"id": "https://login.salesforce.com/id/.../...",
"token_type": "Bearer",
"issued_at": ""
}
```
Take note of the `refresh_token`, itโll be used when you configure this source.
### Additional resources in case you get lost
* [https://stackoverflow.com/questions/12794302/salesforce-authentication-failing/29112224#29112224](https://stackoverflow.com/questions/12794302/salesforce-authentication-failing/29112224#29112224)
* [https://developer.salesforce.com/docs/atlas.en-us.api\_rest.meta/api\_rest/intro\_oauth\_and\_connected\_apps.htm](https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_oauth_and_connected_apps.htm)
* [https://help.salesforce.com/s/articleView?id=sf.remoteaccess\_oauth\_web\_server\_flow.htm\&type=5](https://help.salesforce.com/s/articleView?id=sf.remoteaccess_oauth_web_server_flow.htm\&type=5)
## Schema
The available schema depends on the objects in your Salesforce account.
Some example schema can include the following (depending on your account):
* AIApplication
* AIApplicationConfig
* AIInsightAction
* AIInsightFeedback
* AIInsightReason
* AIInsightValue
* AIPredictionEvent
* AIRecordInsight
* AcceptedEventRelation
* Account
* ...
* WorkTypeFeed
* WorkTypeGroup
* WorkTypeGroupFeed
* WorkTypeGroupHistory
* WorkTypeGroupMember
* WorkTypeGroupMemberFeed
* WorkTypeGroupMemberHistory
* WorkTypeGroupShare
* WorkTypeHistory
* WorkTypeShare
* *and more...*
# Salesforce Marketing Cloud
Source: https://docs.mage.ai/data-integrations/sources/salesforce_marketing_cloud
Sync campaigns, journeys, assets, subscribers, and engagement events from Salesforce Marketing Cloud using the REST and SOAP APIs.
## Overview
This source syncs marketing data from [Salesforce Marketing Cloud](https://www.salesforce.com/products/marketing-cloud/overview/) (SFMC) using both the **Marketing Cloud Engagement REST API** and the **SOAP API**. Authentication uses the OAuth 2.0 `client_credentials` flow via an installed package.
## Authentication
Create an **Installed Package** in Marketing Cloud:
1. In Marketing Cloud, go to **Setup โ Platform Tools โ Apps โ Installed Packages**.
2. Click **New** and give the package a name.
3. Under **Components**, click **Add Component โ API Integration โ Server-to-Server**.
4. Grant the scopes listed below and save.
5. Copy the **Client ID**, **Client Secret**, and the tenant subdomain from the **Authentication Base URI** (e.g. `mc563885gzs27c5t9-63k636ttgm` from `https://mc563885gzs27c5t9-63k636ttgm.auth.marketingcloudapis.com/`).
### Required scopes
| Area | Scope |
| ------------------------ | --------------------------------------------------------------------------- |
| Campaigns | `campaigns_read` |
| Content Builder (assets) | `documents_and_images_read`, `saved_content_read` |
| Journey Builder | `journeys_read` |
| Automation Studio | `automations_read` |
| Email Studio / SOAP | `list_and_subscribers_read`, `tracking_events_read`, `data_extensions_read` |
## Configuration
| Key | Required | Default | Description |
| --------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `client_id` | โ | โ | OAuth 2.0 client ID from the installed package. |
| `client_secret` | โ | โ | OAuth 2.0 client secret from the installed package. |
| `subdomain` | โ | โ | Tenant-specific subdomain. Accepts a bare subdomain (`mc563885gzs27c5t9-63k636ttgm`) or a full URL โ the connector parses the subdomain automatically. |
| `account_id` | | โ | MID of the child business unit to target. Omit to use the top-level account. |
| `start_date` | | โ | ISO 8601 lower bound for incremental streams. **Required** for event streams (`sent_events`, `open_events`, `click_events`, `bounce_events`, `unsub_events`). Example: `2024-01-01T00:00:00Z`. |
## Sample configs
**Minimal โ REST streams only:**
```yaml theme={"system"}
client_id: abcdef1234567890abcdef1234567890
client_secret: ABCDEF1234567890abcdef1234567890
subdomain: mc563885gzs27c5t9-63k636ttgm
```
**With child business unit and event streams:**
```yaml theme={"system"}
client_id: abcdef1234567890abcdef1234567890
client_secret: ABCDEF1234567890abcdef1234567890
subdomain: mc563885gzs27c5t9-63k636ttgm
account_id: "7281234"
start_date: "2024-01-01T00:00:00Z"
```
## Streams
### REST streams
| Stream | Replication | Key | Description |
| -------------------- | ---------------------------- | ----------- | ----------------------------------------------------------------------------------- |
| `campaigns` | Full table | `entity_id` | Marketing campaigns. |
| `assets` | Incremental (`modifiedDate`) | `entity_id` | Content Builder assets โ emails, images, and templates. |
| `journeys` | Full table | `entity_id` | Journey Builder definitions including triggers, goals, and activities. |
| `journey_statistics` | Full table | `entity_id` | Aggregate send/open/click/bounce metrics per journey, plus per-activity breakdowns. |
| `automations` | Full table | `entity_id` | Automation Studio definitions including schedule and steps. |
### SOAP streams
| Stream | Replication | Key | Description |
| ------------------ | ------------------------- | ----------- | ------------------------------------------------------------------------------ |
| `subscriber_lists` | Full table | `entity_id` | Email Studio subscriber lists. |
| `data_extensions` | Full table | `entity_id` | Data Extension definitions โ name, key, sendable flag, and retention settings. |
| `subscribers` | Full table | `entity_id` | All subscribers with email address and status. |
| `sends` | Full table | `entity_id` | Email send job summaries with delivered/open/click/bounce counts. |
| `sent_events` | Incremental (`EventDate`) | `entity_id` | Individual email send records per subscriber. Requires `start_date`. |
| `open_events` | Incremental (`EventDate`) | `entity_id` | Email open tracking records per subscriber. Requires `start_date`. |
| `click_events` | Incremental (`EventDate`) | `entity_id` | Link click records including URL. Requires `start_date`. |
| `bounce_events` | Incremental (`EventDate`) | `entity_id` | Bounce records with category, type, and SMTP details. Requires `start_date`. |
| `unsub_events` | Incremental (`EventDate`) | `entity_id` | Unsubscribe records per subscriber. Requires `start_date`. |
Event streams use **30-day date windowing** to avoid SFMC server-side timeouts on large datasets. The `start_date` config value (or the `EventDate` bookmark from a previous sync) is used as the window start.
## Related links
* [SFMC Installed Packages](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/packages-overview.html)
* [Marketing Cloud Engagement REST API](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/rest-api-overview.html)
* [Marketing Cloud SOAP API](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/web_service_guide.html)
* [OAuth 2.0 Server-to-Server Integration](https://developer.salesforce.com/docs/marketing/marketing-cloud/guide/integration-s2s-client-credentials.html)
# SFTP
Source: https://docs.mage.ai/data-integrations/sources/sftp
## Configuration
Connect securely to an SFTP server and load files into your Mage pipelines by providing the following configuration parameters:
| Key | Description | Sample Value |
| ------------------- | --------------------------------------------------------------------- | --------------------------- |
| `host` | Hostname or IP address of the SFTP server. | `sftp.example.com` |
| `port` | Port number for the SFTP server. | `22` |
| `username` | Username to authenticate with the SFTP server. | `myuser` |
| `password` | Password for authentication (optional if using private key). | `mypassword` |
| `private_key_file` | Path to the private key file for key-based authentication (optional). | `/home/user/.ssh/id_rsa` |
| `start_date` | Earliest file date to synchronize (`YYYY-MM-DD` format). | `2021-01-28` |
| `tables` | An array defining table configurations for loading files (required). | See table config below |
| `decryption_config` | Configuration for decrypting GPG-encrypted files (optional). | See decryption config below |
## Tables Configuration
Define how files are processed and structured into tables:
### Required Parameters
| Key | Description | Sample Value |
| ---------------- | --------------------------------------------------- | --------------------------- |
| `table_name` | Name assigned to the table (stream). | `customer_orders` |
| `search_prefix` | Directory path where the target files are located. | `/Export/SubFolder` |
| `search_pattern` | Regex pattern to match filenames for ingestion. | `MyExportData.*\\.zip.gpg$` |
| `delimiter` | Delimiter used to separate fields (default is `,`). | `,` |
### Optional Parameters
| Key | Description | Sample Value |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | ---------------------- |
| `key_properties` | Array of unique keys for the table. Defaults to `['_sdc_source_file', '_sdc_source_lineno']`. | `["_sdc_source_file"]` |
| `encoding` | Character encoding of the file (default: `utf-8`). | `utf-8` |
| `sanitize_header` | Clean up header names for SQL compatibility (default: `False`). | `True` or `False` |
| `skip_rows` | Number of header or metadata rows to skip at the top of the file (default: `0`). | `0` |
| `auto_generate_field_names` *(Pro only)* | Automatically generate column names (`col_0`, `col_1`, etc.) if headers are missing (default: `false`). | `true` or `false` |
| `schema_discovery_sample_rows` *(Pro only)* | Number of rows to sample to infer schema (default: `100`). Not applicable for encrypted files. | `100` |
## Decryption Configuration (Optional)
If your files are encrypted with GPG, specify the following:
| Key | Description | Sample Value |
| -------------- | ------------------------------------------------------------------ | ------------------------ |
| `SSM_key_name` | AWS SSM parameter key name storing the GPG private key passphrase. | `SSM_PARAMETER_KEY_NAME` |
| `gnupghome` | Path to your `.gnupg` directory containing GPG keys. | `/home/.gnupg` |
| `passphrase` | Passphrase for decrypting GPG files. | `mysecretpassphrase` |
## Mage Pro Enhancements ๐
With **Mage Pro**, the SFTP source is enhanced for improved performance, usability, and scalability.\
The following features are **available in Mage Pro only**:
* **Auto-Generate Field Names**:\
Automatically generate column names (`col_0`, `col_1`, ..., `col_N`) when headers are missing by setting `auto_generate_field_names: true`.
* **Schema Discovery Sample Rows**:\
Speed up schema inference by limiting how many rows are sampled with `schema_discovery_sample_rows`.
* **Improved Logging**:\
Enhanced, structured logging for better traceability and debugging.
* **TableSpec Schema Support**:\
Define detailed table specifications for schema generation and validation.
* **Selective Column Extraction**:\
Write only the selected columns to the schema to reduce size and improve performance.
* **Better Code Quality and Documentation**:\
Cleaner, more robust code and fully updated documentation for enterprise use.
## What is SFTP?
**SFTP (Secure File Transfer Protocol)** is a secure method for transferring files over an encrypted SSH connection.\
It provides encrypted file access, transfer, and management across a reliable, secure channel.
## Why Use SFTP with Mage?
* **Secure and encrypted**: Safely ingest sensitive files over SSH-secured channels.
* **Seamless ETL workflows**: Automate file ingestion into your data warehouse or lake.
* **Support for encrypted files**: Easily decrypt GPG-protected files during ingestion.
* **High performance**: Faster schema discovery, selective column loading, and large file handling.
* **Enterprise-ready**: Built for scale with advanced logging, security, and performance features.
# Snowflake
Source: https://docs.mage.ai/data-integrations/sources/snowflake
## Configuration
Connect to your Snowflake data warehouse and extract data by providing the following configuration parameters:
| Key | Description | Sample Value |
| ----------- | ------------------------------------------------------------------------------------------------------------ | ------------------- |
| `account` | Your [Snowflake account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). | `abc1234.us-east-1` |
| `database` | The name of the database you want to query data from. | `DEMO_DB` |
| `schema` | Schema containing the tables you want to extract data from. | `PUBLIC` |
| `username` | Username for accessing the database (must have read/write permissions). | `guest` |
| `warehouse` | Name of the warehouse for executing queries. | `COMPUTE_WH` |
## Optional Connection Configs
| Key | Description | Sample Value |
| ---------------------- | ------------------------------------------------------------------------ | ------------------------ |
| `password` | Password for the database user. | `abc123...` |
| `private_key_file` | Path to the Snowflake private key file (supported in version >= 0.9.76). | `/path/to/snowflake_key` |
| `private_key_file_pwd` | Passphrase for the private key file (supported in version >= 0.9.76). | `abc123...` |
| `role` | Role of the user to access the database. | `ROLE` |
> For setting up key-pair authentication, follow [this guide](https://docs.snowflake.com/en/user-guide/key-pair-auth).
## Other Optional Configs
| Key | Description | Sample Value |
| ------------------- | -------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch per batch (default: 50,000). Increase for high-memory systems. | `50000` |
## What is Snowflake?
[Snowflake](https://www.snowflake.com/) is a leading cloud-based data warehouse platform that provides fast, scalable, and flexible data storage and analytics. It separates compute and storage for cost efficiency and performance, supporting structured and semi-structured data with SQL-based querying.
## Why Integrate Snowflake with Mage?
* **Seamless ETL pipelines**: Automate extraction of large datasets from Snowflake into your data lakes or warehouses.
* **Scalable performance**: Handle millions of rows with ease using high-speed, cloud-native architecture.
* **Flexible authentication**: Use either password-based or secure key-pair authentication.
* **Cost-effective**: Optimize compute and storage separately for your data workflows.
* **Enterprise-ready security**: Built-in role-based access control and encryption.
# Stripe
Source: https://docs.mage.ai/data-integrations/sources/stripe
## Configuration
Connect to Stripeโs powerful payment platform and automate your data extraction by providing the following configuration parameters:
| Key | Description | Sample Value |
| --------------- | ----------------------------------------------- | ------------ |
| `account_id` | Your Stripe accountโs unique identifier. | `73bb...` |
| `client_secret` | Secret key used to authenticate API requests. | `ABC1...` |
| `start_date` | Date to start syncing data from (`YYYY-MM-DD`). | `2022-01-01` |
> **Find your API keys**:\
> Refer to Stripeโs [official documentation](https://stripe.com/docs/keys) to locate your accountโs API keys.
## What is Stripe?
[Stripe](https://stripe.com/) is a leading payment processing platform that enables businesses of all sizes to accept online payments, manage subscriptions, send invoices, and handle financial operations with powerful APIs and a seamless developer experience.
## Why Integrate Stripe with Mage?
* **Automated data ingestion**: Sync financial and customer data from Stripe into your data warehouse or lake.
* **Comprehensive financial insights**: Access detailed records of transactions, payouts, subscriptions, and more.
* **Real-time analytics**: Stay updated with payment activities without manual exports.
* **Scalable extraction**: Handle high volumes of transactions efficiently with robust API integrations.
* **Secure and compliant**: Stripeโs infrastructure ensures PCI compliance and high security for payment data.
## Supported Schemas
The following Stripe data objects are extracted and loaded:
* Balance transactions
* Charges
* Coupons
* Customers
* Disputes
* Events
* Invoice items
* Invoice line\_items
* Invoices
* Payment intents
* Payout transactions
* Payouts
* Plans
* Products
* Subscription items
* Subscriptions
* Transfers
# Tableau
Source: https://docs.mage.ai/data-integrations/sources/tableau
## Configuration
Connect to your Tableau Cloud environment and extract data using the following configuration parameters:
| Key | Description | Sample Value |
| ----------------- | ---------------------------------------------------------------------- | ---------------------------------------------- |
| `access_token` | (REQUIRED) Access token used to authenticate with Tableau API. | `abcdefg123456` |
| `base_url` | (REQUIRED) Base API URL including your `` and ``. | `https:///api/3.20/sites/` |
| `request_timeout` | (OPTIONAL) Timeout for API requests (default: 300 seconds). | `300` |
| `user_agent` | (OPTIONAL) Custom User-Agent string for API requests. | `my-app-v1.0` |
| `start_date` | (OPTIONAL) Fetch records updated after this date for incremental sync. | `2023-01-01` |
> **Note**:\
> This connector has been tested with **Tableau Cloud**. It may not work with Tableau Server deployments.
### How to get your `access_token`
Follow Tableauโs [official documentation](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_auth.htm) to generate an API `access_token` for authentication.
## What is Tableau?
[Tableau](https://www.tableau.com/) is a leading business intelligence and data visualization platform. It helps organizations turn raw data into interactive dashboards and actionable insights without requiring deep technical expertise.
## Why Integrate Tableau with Mage?
* **Automate data extraction**: Streamline the process of pulling analytics data from Tableau Cloud.
* **Enable real-time updates**: Keep your datasets synced with the latest visualizations and reports.
* **Enhance data workflows**: Integrate Tableau data into broader ETL pipelines for advanced analytics.
* **Scalable and secure**: Handle large datasets securely via Tableauโs API with token-based authentication.
# Teradata
Source: https://docs.mage.ai/data-integrations/sources/teradata
## Configuration
Connect to your Teradata database and extract data by providing the following configuration parameters:
| Key | Description | Sample Value |
| ---------- | ----------------------------------------------------------------------- | ------------------------------ |
| `database` | The name of the database you want to read data from. | `demo` |
| `host` | Hostname or IP address of your Teradata database. | `test.clearscape.teradata.com` |
| `port` | Port number for the database connection (default is 1025 for Teradata). | `1025` |
| `username` | Username with permissions to access the database and schema. | `demo_user` |
| `password` | Password for the user to authenticate. | `abc123...` |
### Optional Configs
| Key | Description | Sample Value |
| ------------------- | -------------------------------------------------------------------------------------- | ------------ |
| `batch_fetch_limit` | Number of rows to fetch per batch (default: 50,000). Adjust based on available memory. | `50000` |
## What is Teradata?
[Teradata](https://www.teradata.com/) is a powerful enterprise-grade relational database management system (RDBMS) optimized for high-performance data warehousing and large-scale analytics. It supports complex queries and advanced analytics across massive datasets.
## Why Integrate Teradata with Mage?
* **High-performance analytics**: Leverage Teradataโs parallel processing to extract large volumes of data efficiently.
* **Scalable ETL workflows**: Integrate Teradata data into your data pipelines for advanced transformation and analysis.
* **Enterprise-ready**: Built for high security, reliability, and compliance standards.
* **Batch optimization**: Customize batch fetch limits to tune performance based on workload and memory.
# TikTok Ads
Source: https://docs.mage.ai/data-integrations/sources/tiktok_ads
Sync ad management objects and performance insight reports from the TikTok Marketing API using a long-lived access token.
## Overview
This source syncs ad management objects and performance insight reports from the [TikTok Marketing API](https://ads.tiktok.com/marketing_api/). It covers advertisers, campaigns, ad groups, ads, and a set of insight breakdowns (by age/gender, country, platform, and province).
Authentication is via a **long-lived access token** issued by the TikTok Marketing API portal. The token is scoped to one or more advertiser accounts, which you list in `accounts`.
Insight streams (`ad_insights`, `ad_insights_by_*`) sync **incrementally** from `start_date`. Entity streams (`advertisers`, `campaigns`, `adgroups`, `ads`) sync **full table** by default.
## Configuration
| Key | Required | Default | Description |
| ----------------- | -------- | --------- | ------------------------------------------------------------------------------------------------- |
| `access_token` | โ | โ | TikTok Marketing API access token. Obtained from the TikTok for Business API portal. |
| `accounts` | โ | โ | Comma-separated advertiser account IDs, or a YAML list. All streams sync data for these accounts. |
| `start_date` | โ | โ | Start of incremental extraction. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`). |
| `end_date` | | now | Optional upper bound for extraction. |
| `sandbox` | | `"false"` | Set to `"true"` to use TikTok sandbox API hosts instead of production. |
| `user_agent` | | โ | User-Agent header for API calls. TikTok recommends setting this to identify your integration. |
| `request_timeout` | | `300` | Request timeout in seconds. |
| `page_size` | | `1000` | Number of results per API page. |
| `include_deleted` | | `"false"` | Set to `"true"` to include deleted rows in the `ads`, `adgroups`, and `campaigns` streams. |
## Sample configs
**Minimal โ entity and insight streams for one account**:
```yaml theme={"system"}
access_token: YOUR_ACCESS_TOKEN
accounts: "1234567890123456789"
start_date: "2024-01-01T00:00:00Z"
```
**Multiple accounts with deleted rows included**:
```yaml theme={"system"}
access_token: YOUR_ACCESS_TOKEN
accounts: "1234567890123456789,9876543210987654321"
start_date: "2024-01-01T00:00:00Z"
include_deleted: "true"
```
**Sandbox testing**:
```yaml theme={"system"}
access_token: YOUR_SANDBOX_TOKEN
accounts: "1234567890123456789"
start_date: "2024-01-01T00:00:00Z"
sandbox: "true"
```
## Streams
| Stream | Type | Description |
| ------------------------------- | ----------- | ------------------------------------------------------------------------------------- |
| `advertisers` | Full table | Advertiser account details for the configured `accounts`. |
| `campaigns` | Full table | Campaigns per advertiser. Set `include_deleted: "true"` to include deleted campaigns. |
| `adgroups` | Full table | Ad groups per campaign. |
| `ads` | Full table | Individual ads per ad group. |
| `ad_insights` | Incremental | Daily ad-level performance metrics from `start_date`. |
| `ad_insights_by_age_and_gender` | Incremental | Ad performance broken down by age and gender. |
| `ad_insights_by_country` | Incremental | Ad performance broken down by country. |
| `ad_insights_by_platform` | Incremental | Ad performance broken down by platform (iOS, Android, etc.). |
| `campaign_insights_by_province` | Incremental | Campaign performance broken down by province/region. |
## How to get an access token
1. Go to the [TikTok for Business Marketing API portal](https://ads.tiktok.com/marketing_api).
2. Create a new app (use the **Marketing API** app flow, not the consumer "TikTok for Developers" Login Kit/Share Kit flow).
3. Under app permissions, request **read** access for:
* **Ad Account Management**
* **Ads Management**
* **Reporting**
4. Optionally add a **sandbox advertiser** for testing without spending budget.
5. Authorize the app against your TikTok Ads account and generate an access token.
6. Copy the `access_token` and your advertiser ID(s) into the source config.
> **Token expiry:** TikTok Marketing API access tokens are long-lived but can expire. If syncs fail with auth errors, regenerate the token in the portal and update your config.
## Related links
* [TikTok Marketing API documentation](https://ads.tiktok.com/marketing_api/docs)
* [TikTok Ads Manager](https://ads.tiktok.com/)
* [Reporting API reference](https://ads.tiktok.com/marketing_api/docs?id=1738864835805185)
# Twitter Ads
Source: https://docs.mage.ai/data-integrations/sources/twitter_ads
## Configuration
Connect to the Twitter Ads API and extract campaign and ad performance data by providing the following configuration parameters:
| Key | Description | Sample Value |
| --------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `start_date` | Absolute beginning date for syncing bookmarked endpoints (`YYYY-MM-DDTHH:MM:SSZ`). | `2023-01-01T00:00:00Z` |
| `consumer_key` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) consumer key. | `YOUR_TWITTER_ADS_CONSUMER_KEY` |
| `consumer_secret` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) consumer secret. | `YOUR_TWITTER_ADS_CONSUMER_SECRET` |
| `access_token` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) access token. | `YOUR_TWITTER_ADS_ACCESS_TOKEN` |
| `access_token_secret` | [OAuth 1.0a](https://developer.twitter.com/en/docs/authentication/oauth-1-0a) access token secret. | `YOUR_TWITTER_ADS_ACCESS_TOKEN_SECRET` |
| `account_ids` | Comma-delimited list of Twitter Ad Account IDs. | `id1, id2, id3` |
| `attribution_window` | Number of days for latency look-back period to stabilize analytics reporting. | `14` |
| `with_deleted` | Include logically deleted records in the results (`true` or `false`). | `true` |
| `country_codes` | Comma-delimited list of ISO 2-letter country codes for targeting and segmentation. | `US, CA, MX, DE` |
| `page_size` | Optional: custom `page_size` for pagination. | `1000` |
| `reports` | List of reports with `name`, `entity`, `segment`, and `granularity` details. | `[{"name": "campaigns_genders_hourly_report", "entity": "CAMPAIGN", "segment": "GENDER", "granularity": "HOUR"}]` |
| `request_timeout` | Timeout for the Twitter Ads API client (default: 300 seconds). | `300` |
## How to Get Access to the Twitter Ads API
To use the Twitter Ads source, you must first get access to the Twitter Ads API. Follow [this guide](https://developer.twitter.com/en/docs/twitter-ads-api/getting-started) to apply for access and configure your app credentials.
## What is Twitter Ads?
[Twitter Ads](https://ads.twitter.com/) is an advertising platform that allows businesses to promote content, grow followers, and drive engagement through highly targeted ad campaigns across the Twitter network.
## Why Integrate Twitter Ads with Mage?
* **Automated marketing insights**: Extract detailed campaign, ad group, and ad-level performance metrics.
* **Advanced segmentation**: Access granular breakdowns by gender, location, device, and more.
* **Real-time reporting**: Keep your analytics and BI dashboards updated with the latest ad performance data.
* **Scalable API extraction**: Efficiently handle large ad accounts and high-volume data reporting.
* **Secure and reliable**: OAuth 1.0a authentication ensures secure API access.
## Supported Streams
The following Twitter Ads objects are extracted:
* account\_media
* accounts
* advertiser\_business\_categories
* campaigns
* cards
* cards\_image\_conversation
* cards\_poll
* cards\_video\_conversation
* content\_categories
* funding\_instruments
* iab\_categories
* line\_item\_apps
* line\_items
* media\_creatives
* preroll\_call\_to\_actions
* promotable\_users
* promoted\_accounts
* promoted\_tweets
* scheduled\_promoted\_tweets
* tailored\_audiences
* targeting\_app\_store\_categories
* targeting\_conversations
* targeting\_criteria
* targeting\_devices
* targeting\_events
* targeting\_interests
* targeting\_languages
* targeting\_locations
* targeting\_network\_operators
* targeting\_platform\_versions
* targeting\_platforms
* targeting\_tv\_markets
* targeting\_tv\_shows
* tracking\_tags
* tweets
# Xero
Source: https://docs.mage.ai/data-integrations/sources/xero
Sync accounting data from Xero including invoices, contacts, payments, and more. Supports incremental syncing with automatic OAuth token refresh.
## Configuration
To set up the Xero source, provide the following configuration parameters:
| Key | Description | Sample Value | Required |
| --------------------------- | ---------------------------------------------------------------------------------- | ---------------------- | -------- |
| `client_id` | Client ID issued when you create your Xero app. | `abcdefg` | โ |
| `client_secret` | Client secret generated when you create your Xero app. | `abcdefg` | โ |
| `tenant_id` | Tenant ID associated with your Xero organization. | `abcdefg` | โ |
| `start_date` | The start date for syncing data. Format: `YYYY-MM-DDTHH:MM:SSZ`. | `2021-01-01T00:00:00Z` | โ |
| `refresh_token` | OAuth refresh token (required if `refresh_token_secret_name` is not provided). | `abcdefg` | โ ๏ธ |
| `refresh_token_secret_name` | Name of the Mage secret containing the refresh token (recommended for production). | `XERO_REFRESH_TOKEN` | โ ๏ธ |
You must provide either `refresh_token` or `refresh_token_secret_name`. Using `refresh_token_secret_name` is recommended because Xero uses rotating refresh tokens (each token can only be used once), and this option automatically persists the new token after each refresh.
### Option 1: Direct Refresh Token
```yaml theme={"system"}
client_id: YOUR_CLIENT_ID
client_secret: YOUR_CLIENT_SECRET
tenant_id: YOUR_TENANT_ID
start_date: "2021-01-01T00:00:00Z"
refresh_token: YOUR_REFRESH_TOKEN
```
Xero refresh tokens are single-use. Each time the token is refreshed, a new one is issued and the old one is invalidated. With this option, the new token is only stored in memory during the sync, so subsequent pipeline runs may fail if the token has been rotated.
### Option 2: Mage Secret (Recommended)
```yaml theme={"system"}
client_id: YOUR_CLIENT_ID
client_secret: YOUR_CLIENT_SECRET
tenant_id: YOUR_TENANT_ID
start_date: "2021-01-01T00:00:00Z"
refresh_token_secret_name: XERO_REFRESH_TOKEN
```
First, create a secret in Mage. You can choose any name for your secret:
1. Open a pipeline edit page
2. Expand the right side panel
3. Click the **"Secrets"** tab
4. Click **"Add new secret"**
5. Enter your chosen secret name (e.g., `XERO_REFRESH_TOKEN`) and your initial refresh token as the value
6. Copy the secret name to your Xero config's `refresh_token_secret_name` field
The tap will automatically update the secret with the new refresh token after each sync, ensuring reliable long-term operation.
## Supported Streams
The Xero source supports the following streams:
### Incremental Streams
These streams support incremental syncing based on the `start_date` configuration and bookmark state:
| Stream | Primary Key | Replication Key |
| --------------------- | ------------------- | --------------- |
| `accounts` | AccountID | UpdatedDateUTC |
| `bank_transactions` | BankTransactionID | UpdatedDateUTC |
| `bank_transfers` | BankTransferID | CreatedDateUTC |
| `contacts` | ContactID | UpdatedDateUTC |
| `credit_notes` | CreditNoteID | UpdatedDateUTC |
| `employees` | EmployeeID | UpdatedDateUTC |
| `expense_claims` | ExpenseClaimID | UpdatedDateUTC |
| `invoices` | InvoiceID | UpdatedDateUTC |
| `items` | ItemID | UpdatedDateUTC |
| `journals` | JournalID | JournalNumber |
| `linked_transactions` | LinkedTransactionID | UpdatedDateUTC |
| `manual_journals` | ManualJournalID | UpdatedDateUTC |
| `overpayments` | OverpaymentID | UpdatedDateUTC |
| `payments` | PaymentID | UpdatedDateUTC |
| `prepayments` | PrepaymentID | UpdatedDateUTC |
| `purchase_orders` | PurchaseOrderID | UpdatedDateUTC |
| `quotes` | QuoteID | UpdatedDateUTC |
| `receipts` | ReceiptID | UpdatedDateUTC |
| `users` | UserID | UpdatedDateUTC |
### Full Table Streams
These streams are fully synced on each run (no incremental support):
| Stream | Primary Key |
| --------------------- | ------------------ |
| `branding_themes` | BrandingThemeID |
| `contact_groups` | ContactGroupID |
| `currencies` | Code |
| `organisations` | OrganisationID |
| `repeating_invoices` | RepeatingInvoiceID |
| `tax_rates` | TaxType |
| `tracking_categories` | TrackingCategoryID |
## How to Generate Credentials
To use this source, you need to create a Xero OAuth app and complete the OAuth flow.
### 1. Create a Xero App
Go to the [Xero Developer Portal](https://developer.xero.com/app/manage) and create a new app. Note down your **Client ID** and **Client Secret**.
### 2. Configure OAuth2 Scopes
When setting up your Xero app, grant the following OAuth2 scopes:
| Scope | Required For |
| ------------------------------ | --------------------------------------- |
| `accounting.settings.read` | Organisation settings, currencies, etc. |
| `accounting.transactions.read` | Invoices, bank transactions, payments |
| `accounting.contacts.read` | Contacts |
| `accounting.reports.read` | Reports |
| `accounting.journals.read` | Journals |
| `accounting.attachments.read` | Attachments |
| `offline_access` | Long-lived access via refresh tokens |
* The minimum scope required for discovery is `accounting.settings.read`.
* You only need scopes for the streams you plan to sync.
* Missing scopes for a selected stream will result in `401` or `403` errors.
**Example Scope String:**
```
accounting.settings.read accounting.transactions.read accounting.contacts.read accounting.reports.read accounting.journals.read accounting.attachments.read offline_access
```
### 3. Complete the OAuth2 Authorization Flow
Follow [Xero's OAuth2 Auth Flow Guide](https://developer.xero.com/documentation/guides/oauth2/auth-flow/) to obtain your initial `access_token` and `refresh_token`.
Make sure to request the `offline_access` scope to receive a refresh token.
### 4. Get Your Tenant ID
The `tenant_id` uniquely identifies your Xero organization. After completing OAuth authentication:
1. Call the connections endpoint:
```bash theme={"system"}
curl -X GET "https://api.xero.com/connections" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json"
```
2. Extract the `tenantId` from the response:
```json theme={"system"}
[
{
"id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"tenantType": "ORGANISATION",
"tenantName": "Your Company Name"
}
]
```
If your app is connected to multiple Xero organizations, the response will contain multiple entries. Choose the `tenantId` for the organization you want to sync data from.
| Key | Description | Sample Value | Required |
| --------------- | ------------------------------------------------------------------------------------------------ | ---------------------- | -------- |
| `access_token` | OAuth access token used to authenticate API requests. | `abcdefg` | โ |
| `id_token` | Token containing user identity details (if OpenID Connect scopes are requested). | `abcdefg` | โ |
| `start_date` | The start date for syncing data. Format: `YYYY-MM-DDTHH:MM:SSZ`. | `2021-01-01T00:00:00Z` | โ |
| `client_id` | Client ID issued when you create your Xero app. | `abcdefg` | โ |
| `client_secret` | Client secret generated when you create your Xero app. | `abcdefg` | โ |
| `tenant_id` | Tenant ID associated with your Xero organization. | `abcdefg` | โ |
| `refresh_token` | Refresh token used to renew the access token after expiration (requires `offline_access` scope). | `abcdefg` | โ |
## Required Xero OAuth2 Scopes
To use this source, you must grant your Xero app the following OAuth2 scopes.
### Minimum Scope for Discovery
* `accounting.settings.read`
Required for the `/Currencies` endpoint, which is called during discovery to verify:
* Your `access_token` is valid
* Your `tenant_id` is authorised
* The app has the correct scopes
If this scope is missing, discovery will fail before any sync begins.
### Scopes for Syncing All Streams
To sync all available streams, grant the following scopes:
| Scope | Required For |
| ------------------------------ | --------------------------------------- |
| `accounting.settings.read` | Organisation settings, currencies, etc. |
| `accounting.transactions.read` | Invoices, bank transactions, payments |
| `accounting.contacts.read` | Contacts |
| `accounting.reports.read` | Reports |
| `accounting.journals.read` | Journals |
| `accounting.attachments.read` | Attachments |
| `offline_access` | Long-lived access via refresh tokens |
**Note:**
* You only need scopes for the streams you plan to sync.
* Missing scopes for a selected stream will result in `401` or `403` errors.
### Example Scope String
```
accounting.settings.read
accounting.transactions.read
accounting.contacts.read
accounting.reports.read
accounting.journals.read
accounting.attachments.read
offline_access
```
## How to Generate Credentials
Follow Xero's [OAuth 2.0 Authentication Flow guide](https://developer.xero.com/documentation/guides/oauth2/auth-flow/) to generate the required credentials:
1. Register a new Xero app to get your `client_id` and `client_secret`.
2. Set up the OAuth scopes your app requires.
3. Complete the OAuth 2.0 authorization flow to obtain your `access_token`, `refresh_token`, `id_token`, and `tenant_id`.
4. Store these securely in your configuration.
## Additional Notes
* The `start_date` determines how far back data will be synced.
* Ensure that the `offline_access` scope is enabled to receive a `refresh_token`.
* Tokens expire after a set period; use the `refresh_token` to obtain new `access_token` and `id_token` values automatically.
* During setup, the connector calls the `/Currencies` endpoint to validate platform access before syncing.
# YouTube Analytics API Source
Source: https://docs.mage.ai/data-integrations/sources/youtube_analytics
This guide explains how to configure the YouTube Analytics API as a data source in Mage. Learn how to authenticate with Google Cloud, request the necessary OAuth scopes, and sync data from specific channels.
Mage supports ingesting data from YouTube Analytics by connecting to the YouTube Analytics API and YouTube Data API v3 using OAuth 2.0 credentials.
This integration allows you to extract rich channel analytics, video performance, and viewer engagement metrics directly into your Mage pipelines.
***
## ๐ง Configuration
To connect to the YouTube Analytics API, provide the following parameters in your source configuration:
| Key | Description | Sample Value | Required |
| --------------- | ------------------------------------------------------------- | --------------------------------------------- | -------- |
| `client_id` | OAuth 2.0 Client ID from Google Cloud Console | `123456789-abcdef.apps.googleusercontent.com` | โ |
| `client_secret` | OAuth 2.0 Client Secret from Google Cloud Console | `abcdefghijklmnopqrstuvwxyz` | โ |
| `refresh_token` | OAuth 2.0 refresh token for authenticating requests | `1//abcdefghijklmnopqrstuvwxyz` | โ |
| `start_date` | Start date for pulling analytics data | `2021-01-01T00:00:00Z` | โ |
| `user_agent` | User agent string identifying the integration | `example@example.com` | โ |
| `channel_ids` | Comma-separated list of channel IDs to retrieve analytics for | `UC1234..., UC5678...` | โ |
***
## ๐ How to Generate Credentials
Follow these steps to generate the required credentials from the Google Cloud Console:
### 1. Set Up OAuth Credentials
1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project (or use an existing one)
3. Enable the following APIs:
* **YouTube Data API v3**
* **YouTube Analytics API v2**
4. Navigate to โAPIs & Services > Credentialsโ
5. Create a new **OAuth 2.0 Client ID** (select "Web application" or "Desktop app")
### 2. Generate a Refresh Token
Use [OAuth 2.0 Playground](https://developers.google.com/oauthplayground/) to get a `refresh_token`:
* Click the gear icon and check **"Use your own OAuth credentials"**, then enter your Client ID and Secret.
* Select the following scopes:
* `https://www.googleapis.com/auth/youtube.readonly`
* `https://www.googleapis.com/auth/yt-analytics.readonly`
* Follow the steps to **authorize**, **exchange code for tokens**, and **copy the refresh token**.
## How It Works
This integration connects to two key Google APIs to extract YouTube Analytics data and video metadata:
### APIs Used
* **[YouTube Reporting API](https://developers.google.com/youtube/reporting/v1/reports)**\
Used to retrieve bulk analytics reports for YouTube channels and content owners.
* [Jobs](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/jobs/list): List available report jobs
* [Reports](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/jobs.reports/list): Retrieve generated reports
* [Report Types](https://developers.google.com/youtube/reporting/v1/reference/rest/v1/reportTypes/list): View available report categories
* Report examples:
* [Channel Reports](https://developers.google.com/youtube/reporting/v1/reports/channel_reports)
* [Content Owner Reports](https://developers.google.com/youtube/reporting/v1/reports/content_owner_reports)
* **[YouTube Data API v3](https://developers.google.com/youtube/v3/docs)**\
Used to fetch real-time metadata and performance metrics for channels, videos, and playlists.
* [Channels](https://developers.google.com/youtube/v3/docs/channels/list)
* [Playlists](https://developers.google.com/youtube/v3/docs/playlists/list)
* [Playlist Items](https://developers.google.com/youtube/v3/docs/playlistItems/list)
* [Videos](https://developers.google.com/youtube/v3/docs/videos/list)
## Supported Streams
The following streams are available in the YouTube Analytics integration:
| Stream | Description |
| ---------------- | ---------------------------------------------------------- |
| `channels` | Metadata and statistics for authenticated YouTube channels |
| `playlist_items` | Videos and items within playlists |
| `playlists` | Metadata about playlists created by the authenticated user |
| `reports` | Bulk analytics data from YouTube Reporting API |
| `videos` | Video metadata, engagement, and performance statistics |
## Sync Behavior
* **Incremental Syncs**\
The integration supports stateful incremental loading, so only new or modified records are fetched during each sync run.
* **Schema Output**\
Mage automatically generates the output schema for each stream, making it easier to understand and work with the resulting data.
# Zendesk
Source: https://docs.mage.ai/data-integrations/sources/zendesk
## Config
You must enter the following credentials when configuring this source:
| Key | Description | Sample value |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `access_token` | OAuth token to access API endpoints. | `abc123...` |
| `request_timeout` | Number of milliseconds until the API request should timeout and raise an error. | 300 |
| `start_date` | Fetch data that is newer than the `start_date`. | `2022-01-01T00:00:00Z` |
| `subdomain` | If you access Zendesk from a URL like this [https://magesupport.zendesk.com/](https://magesupport.zendesk.com/), then the `domain` is `magesupport`. | `magesupport` |
## How to get your `access_token`
Follow [Zendeskโs documentation](https://support.zendesk.com/hc/en-us/articles/4408845965210)
to get your access token.
Example Admin Center URL: [https://magesupport.zendesk.com/admin/home](https://magesupport.zendesk.com/admin/home)
(subdomain in example is `magesupport`).
### Example values for getting access token
Zendesk authorization page
[https://magesupport.zendesk.com/oauth/authorizations/new?client\_id=mage-ai\&response\_type=code\&redirect\_uri=https://www.mage.ai\&scope=read](https://magesupport.zendesk.com/oauth/authorizations/new?client_id=mage-ai\&response_type=code\&redirect_uri=https://www.mage.ai\&scope=read)
Authorize redirect URL with code
[https://www.mage.ai/?code=6b21972c79e12cbda37947df4ba22d4cb394da752e955c6dac81361d5f453695](https://www.mage.ai/?code=6b21972c79e12cbda37947df4ba22d4cb394da752e955c6dac81361d5f453695)
Example cURL command to exchange code for access token
```curl theme={"system"}
curl --request POST \
--url https://magesupport.zendesk.com/oauth/tokens \
--header 'Content-Type: application/json' \
--data '{
"grant_type": "authorization_code",
"code": "6b21972c79e12cbda37947df4ba22d4cb394da752e955c6dac81361d5f453695",
"client_id": "mage-ai",
"client_secret": "your_oauth_client_secret",
"redirect_uri": "https://www.mage.ai",
"scope": "read"
}'
```
Example OAuth token exchange response
```json theme={"system"}
{
"access_token": "cdaae5613efca304972cd8175adf51b560691f51e9a6331a986a772801a5d898",
"token_type": "bearer",
"scope": "read"
}
```
# Project Structure
Source: https://docs.mage.ai/design/abstractions/project-structure
Details about how the Mage file directory works and how typical projects are structured.
## Overview
A Mage project lives inside it's own folder. It's there where mage data is stored (`mage_data`) and individual projects are housed. You can think of a project as an environment: it contains configurations, pipelines, and other project-specific data.
## Project structure
Here is a [sample project](https://github.com/mage-ai/demo%5Fetl%5Fpipeline) and a sample folder structure:
```
.
โโโ mage_data
โโโ my-first-project
โโโ charts
โโโ custom
โโโ data_exporters
โโโ data_loaders
โโโ dbt
โโโ extensions
โโโ pipelines
โ โโโ demo
โ โโโ __init__.py
โ โโโ metadata.yaml
โโโ scratchpads
โโโ transformers
โโโ utils
โโโ __init__.py
โโโ io_config.yaml
โโโ metadata.yaml
โโโ requirements.txt
```
Let's walk through each folder:
* Each [Block](/design/core-abstractions#block) type has it's own directory:
* `charts`
* [`custom`](/guides/blocks/custom-blocks)
* [`data_loaders`](/design/blocks/data-loader)
* [`data_exporters`](/design/blocks/data-exporter)
* [`extensions`](/design/blocks/extension)
* [`sensors`](/design/blocks/sensor)
* [`scratchpads`](/design/blocks/scratchpad)
* [`transformers`](/design/blocks/transformer)
* dbt assets are stored in the [dbt directory](/dbt/overview).
* Each [pipeline](/design/core-abstractions#pipeline) is represented by a YAML file in the `pipelines` folder under the Mage project directory: `[project_dir]/pipelines/[pipeline_name]/metadata.yaml`
* The `utils` folder is meant to hold custom utilities for your project. For example, [Python scripts](/development/dependencies/custom-files).
* The `extensions` folder is used for Mage extensions that integrate other data tools, like [Great Expectations](/development/testing/great-expectations).
* Data loader and exporter [configurations](/development/io_config) are stored in the `io_config.yaml` file.
* The `metadata.yaml` file contains project-level metadata. There's a metadata file in each pipeline as well.
Code can be shared across the entire project.
## Project organization
We know that you're hard at work creating blocks and building pipelines, that's why we've made it easy to organize your project.
Blocks can be organized in subdirectories within their respective foldersโ no additional configuration required. For example, you might have a `data_loaders` folder with subdirectories for `api`, `database`, and `file` data loaders.
You can do the same when creating blocks via the UIโ simply name the block `subfolder_name/block_name` and the subfolder will be automatically created.
This might look like:
```
.
โโโ data_loaders
โโโ api
โย ย โโโ load_bamboo.py
โย ย โโโ load_recurly.py
โย ย โโโ load_stripe.py
โโโ database
| โโโ load_duckdb.py
โย ย โโโ load_influxdb.py
โโโ file
โโโ load_avro.py
โโโ load_parquet.py
```
## Project data
Project-level data is stored in the `mage_data` directoryโ located at the same level as your project folder.
`mage_data` holds project-level cache data and stores the result of Block runs, which are then returned to the user via the UI. Here are the locations for various components of `mage_data`:
* SQLite DB: `mage_data/[project_folder]/mage-ai.db`
* The SQLite DB stores project-level metadata, such as pipeline and block run data. Here's a sample tree tables in the `tables` schema:
```
โโโ tables
โโโ backfill
โโโ block_run
โโโ event_matcher
โโโ oauth2_access_token
โโโ oauth2_application
โโโ permission
โโโ pipeline_run
โโโ pipeline_schedule
โโโ pipeline_schedule_event_matcher_association
โโโ role
โโโ secret
โโโ sqlite_master
โโโ user
โโโ user_role
```
* You can connect to the SQLite DB like any other database. The JDBC URL `jdbc:sqlite:PATH`, where PATH is `/home/src/mage_data/[project_folder]/mage-ai.db` for Docker installs or `~/.mage_data/[project_folder]/mage-ai.db` for pip installs.
* Block output: `mage_data/[project_folder]/pipelines/[pipeline_uuid]/.variables/`
* Pipeline and block logs: `mage_data/[project_folder]/pipelines/[pipeline_uuid]/.logs/`
* Cache: `mage_data/[project_folder]/.cache`
To clear project block output, cache data, or log data, consider performing routine operations on this folder. This might be helpful for reducing storage size, optimizing cost, or otherwise maintaining your Mage project.
## Changing the name of your project
### Docker
If youโre running Mage in Docker, change the environment variable `USER_CODE_PATH` to the absolute path where your project lives.
For example, if your original project name was `demo_magic` and the directory is located at `/home/src/demo_magic`,
then the value of `USER_CODE_PATH` should be: `/home/src/demo_magic`.
### Not using Docker
When you run `mage start [project_name]` on the command line to start running Mage,
the 1st command line argument value is used for your projectโs name.
# Overview
Source: https://docs.mage.ai/design/blocks
Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.
## Unique block paradigm
### Modularity
Mage breaks down complex data engineering tasks into smaller, manageable units (blocks). Each block in a pipeline is a standalone file that can be executed independently.
Modular design makes it easier to maintain and update data pipelines by modifying a specific block or creating new ones without affecting other parts of the pipeline.
This isolation reduces the risk of introducing new bugs into the system when making changes. The modularity of blocks leads to cleaner, more maintainable codebases.
Modular design enables fault-tolerant pipelines so that when one or more blocks fail, the rest can keep running. Debugging modular blocks is faster than debugging a monolithic code base. Once resolved, failed blocks can be retried without re-running the entire pipeline.
### Reusability
This design encapsulates each step of a data pipeline into reusable blocks that perform a specific function; such as data loading, transformation, or exporting.
Developers can build complex pipelines faster by leveraging existing blocks across different projects, different pipelines, or within the same pipeline.
Designing blocks to be reusable significantly reduces the time and effort required to develop new data pipelines, as developers can reuse blocks instead of writing new code from scratch.
### Scalability
Developers can optimize individual blocks for handling large data volumes or more complex processing tasks efficiently, without the need for extensive redesigns or refactoring of the entire pipeline.
Dynamic blocks adapt their behavior based on input data or runtime conditions, enabling the creation of flexible and complex data pipelines that can easily accommodate varying scalability requirements all without the need to write duplicate code.
***
## Block types
* [Data loader](/design/blocks/data-loader)
* [Transformer](/design/blocks/transformer)
* [Data exporter](/design/blocks/data-exporter)
* [Scratchpad](/design/blocks/scratchpad)
* [Sensor](/design/blocks/sensor)
* [dbt](/dbt/overview)
* [Extensions](/design/blocks/extension)
* [Callbacks](/design/blocks/callbacks)
# Callback blocks
Source: https://docs.mage.ai/design/blocks/callbacks
A callback block is associated to another block. When the parent block succeeds or fails, the callback block functions are executed.
Requires version `0.8.61` or greater.
Callback blocks are part of your pipeline but donโt run as individual steps in your pipeline
like data loader blocks, transformer blocks, etc.
However, callback blocks are associated to other blocks within the pipeline
(e.g. data loaders, transformers, data exporters, etc).
After those other blocks are executed,
any associated callback blocks will also be executed depending on the status of the parent block.
***
## Example
1. Your pipeline has the following blocks:
1. `load_data_from_api` (data loader)
2. `clean_column_names` (transformer)
3. `save_data` (data exporter)
2. Then, you add the following callback block to your pipeline:
```python theme={"system"}
@callback('success')
def hello_world(parent_block_data, **kwargs):
# some code...
@callback('failure')
def alert_me(parent_block_data, **kwargs):
# some code...
@callback('cancelled')
def notify_cancelled(parent_block_data, **kwargs):
# some code...
```
3. You associate the above callback block to the following 2 blocks:
* `load_data_from_api` (data loader)
* `clean_column_names` (transformer)
4. When you run the pipeline, itโll execute the `load_data_from_api` (data loader) block first.
It successfully completes. Then, the callback block will run and execute the function `hello_world`.
5. Next, the pipeline will execute the `clean_column_names` (transformer) block.
It fails due to some error. The callback block will run and execute the function `alert_me`.
***
## How to add callbacks to your pipeline
1. Create a new pipeline or open an existing pipeline.
2. Edit the pipeline.
3. On the right side of the page, click the Add-ons icon in the navigation.
If you donโt see it, try expanding the right area of the page.
4. Click the button Callbacks.
5. Click the button + Callback block.
6. In the callback blockโs code, add a function and decorate it with `@callback`.
You can have as many functions in a callback block as you want.
7. Use `@callback` or `@callback('success')` to decorate a function that should only run when the
parent block successfully completes.
8. Use `@callback('failure')` to decorate a function that should only run when the
parent block fails.
### Supported callbacks
#### `success`
Execute callback function when block runs successfully.
```python theme={"system"}
@callback('success')
def only_run_this_function_on_success(parent_block_data, **kwargs):
pass
```
#### `failure`
Execute callback function when block fails.
```python theme={"system"}
@callback('failure')
def only_run_this_function_on_failure(parent_block_data, **kwargs):
pass
```
#### `cancelled`
Available in versions `>= 0.9.77`.
Execute callback function when block is cancelled (manually or by system interruption).
```python theme={"system"}
@callback('cancelled')
def only_run_this_function_on_cancelled(parent_block_data, **kwargs):
pass
```
### Advanced parameters
In **Mage Pro**, you can pass additional parameters to the `@callback` decorator.
#### `fetch_inputs`
By default, Mage will fetch upstream inputs before executing a callback.
If your callback doesnโt need input data (e.g. sending alerts, posting logs), you can **disable input fetching** to improve performance.
```python theme={"system"}
@callback('failure', fetch_inputs=False)
def alert_me_without_inputs(*args, **kwargs):
# Runs faster since upstream block inputs are not fetched
send_alert("Parent block failed!")
```
* **`fetch_inputs=True`** (default) โ Fetch parent block inputs before executing callback
* **`fetch_inputs=False`** โ Skip fetching inputs, speeding up execution
### Positional arguments
The 1st positional argument is the data output of the callback blockโs parent block.
Positional arguments are only supported in Mage versions `>= 0.8.81`.
### Keyword arguments
Here are the keyword arguments that are available in each callback function:
| Name | Description | Sample value |
| --------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `block_uuid` | Callback block UUID. | `'fireball_callback'` |
| `parent_block_uuid` | The block UUID of the parent block. | `'parent_block'` |
| `ds` | Date string when the parent block started executing. | `'2023-12-25'` |
| `event` | A dictionary containing metadata from an event triggered pipeline. | `{}` |
| `execution_date` | Python datetime object for when the parent block started executing. | `datetime.datetime(2023, 4, 26, 20, 28, 17, 335254, tzinfo=datetime.timezone.utc)` |
| `execution_partition` | Partition used for the parent block when it was executed. | `'207/20230426T202817'` |
| `hr` | Hour string when the parent block started executing. | `'20'` |
| `pipeline_run` | Python pipeline run object associated to the current run. | `PipelineRun(id=2357, pipeline_uuid=fire_etl, execution_date=2023-04-26 20:28:17.335254+00:00)` |
| `pipeline_uuid` | UUID of the current pipeline. | `'fire_etl'` |
| `__input` | For version `>= 0.8.81`. See below for more details. | `{ 'some_block_uuid': {} }` |
#### `__input` keyword argument
For versions `>= 0.8.81`.
If your callback function is defined to only accept keyword arguments and no positional arguments,
you can access the callback blockโs parent blockโs data output by accessing the key
labeled `__input` in the callback functionโs keyword arguments.
The value of the `__input` key is a dictionary. The keys in that dictionary are named after the
callback blockโs parent blockโs UUID.
For example, if the callback blockโs parent blockโs UUID is `load_api_data`, then this is how
you would access `load_api_data`โs data output:
```python theme={"system"}
@callback('success')
def callback_for_load_api_data(**kwargs):
data_from_parent_block = kwargs['__input']['load_api_data']
```
#### Data integration pipelines only
The following keyword arguments are only available in callback functions with parent blocks in a
data integration pipelines:
| Name | Description | Sample value |
| ------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------------- |
| `destination_table` | The table name that the source data is being exported to. | `'dim_users_v1'` |
| `index` | The order in which the parent block executed compared to other blocks of the same type. | `0` |
| `is_last_block_run` | Boolean value for whether or not the parent block was the last block ran for the block type in the pipeline. | `True` or `False` |
| `stream` | The name of the source stream (or table) that the parent block is syncing for. | `'stripe_transactions'` |
***
## Legacy
For versions `< 0.8.61`.
### Adding a callback to your block
1. Create a block as normal.
2. In the top right of the block, click the triple dot icon which should open up a menu
with more options. Click "Add callback".
3. You should now see another code editor under the main editor for your block.
4. You can add an `on_success` callback and/or a `on_failure` callback. These callbacks
will run after your block run completes or fails. When you run your block in the
pipeline edit page, it will also run the callback depending on the status of the block.
5. The callback function will be passed your pipeline's runtime variables as keyword
arguments, so you can use the same `kwargs.get('')` syntax in your callback.
# Conditional blocks
Source: https://docs.mage.ai/design/blocks/conditionals
A conditional block is an 'Add-on Block' associated with another block. The condition will be evaluated before the parent block is executed, determining if the parent block gets executed.
Requires version `0.8.93` or greater.
Conditional blocks are part of your pipeline but don't run as individual steps in your pipeline like data loader blocks, transformer blocks, etc.
However, conditional blocks are associated to other blocks within the pipeline (e.g. data loaders, transformers, data exporters, etc).
Before those other blocks are executed, any associated conditional blocks will be executed. If any conditional block returns False, the parent block will not be executed, and its status will be set as `condition_failed`.
### An example condition
Take the following example:
The `morpheus` block returns a variableโ `pill`, which is either red or blue. Both downstream blocks have conditionals associated with them. The `red_pill` block will only execute if `pill` is red, and the `blue_pill` block will only execute if `pill` is blue.
Here's what the conditionals look like for each:
Because `pill=red`, the `red_pill` block will execute, and the `blue_pill` block will not. Additionally, any *downstream* blocks of the `blue_pill` block will not execute.
## Add conditionals to your pipeline
Conditional blocks are "add-on blocks" in Mageโ that means they're added through the side panel on the right in the pipeline editor.
To add a condition to your block:
1. On the right side of the page, expand the side panel by hovering, click the Add-ons icon in the navigation.
2. Click the button Conditionals.
3. Click the button + Conditional block.
4. In the conditional block's code, add a function that returns a boolean value. Be sure it's decorated with `@condition`.
5. Select the magnifiying class icon at the bottom of the block to associate your conditional with a block.
Looking for a video walkthrough? Check out our [tips and tricks](/guides/tips-and-tricks) page!
### Positional arguments
The positional arguments to the conditional block will be the same as the positional
arguments to the parent block. You can use the positional arguments to get the output
of the upstream blocks.
### Keyword arguments
Here are the keyword arguments that are available in each conditional function:
| Name | Description | Sample value |
| --------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| `block_uuid` | Conditional block UUID. | `'fireball_conditional'` |
| `ds` | Date string when the parent block started executing. | `'2023-12-25'` |
| `event` | A dictionary containing metadata from an event triggered pipeline. | `{}` |
| `execution_date` | Python datetime object for when the trigger run started executing. | `datetime.datetime(2023, 4, 26, 20, 28, 17, 335254, tzinfo=datetime.timezone.utc)` |
| `execution_partition` | Partition used for the trigger run when it was executed. | `'207/20230426T202817'` |
| `hr` | Hour string when the trigger run started executing. | `'20'` |
| `pipeline_run` | Python pipeline run object associated to the current run. | `PipelineRun(id=2357, pipeline_uuid=fire_etl, execution_date=2023-04-26 20:28:17.335254+00:00)` |
| `pipeline_uuid` | UUID of the current pipeline. | `'fire_etl'` |
# Data exporter
Source: https://docs.mage.ai/design/blocks/data-exporter
After completing data transformations, utilize the data exporter blocks to either load the processed data or store a machine learning model in an external data storage system.
Mage natively supports integration with a variety of data storage systems. However, these integrations often
require specific configurations in both the exporter block and the `io_config.yml` file to ensure seamless operation.
The `io_config.yml` file typically includes connection details such as host, port, database name, username, and password.
Meanwhile, the exporter block needs to be configured with the appropriate export parameters, such as target table names,
schema details, and conflict resolution strategies.
### Technical Details
1. **Data Exporter Blocks**:
* These blocks are designed to facilitate the movement of transformed data or trained models to external systems.
* Configuration parameters might include destination paths, file formats, table names, schemas, and update strategies.
* Most data exporters include a `config_profile` parameter set to `'default'` by default. This parameter can be customized
to use different configuration profiles if you have multiple profiles or have renamed them.
2. **Supported Data Storage Systems**:
* Mage supports a wide range of storage systems including PostgreSQL, MySQL, AWS S3, Google Cloud Storage, Azure Blob Storage, and many more.
* Each system may have unique requirements and configurations to ensure compatibility and optimal performance.
3. **Configuration in `io_config.yml`**:
* This file serves as the central configuration hub for defining connection parameters.
* Typical parameters include:
* `host`: The server address of the storage system.
* `port`: The port number for the connection.
* `database`: The name of the target database or data storage container.
* `username` and `password`: Authentication credentials.
* Additional parameters as required by specific storage systems (e.g., SSL settings, API tokens).
### Examples
Configure the `io_config.yml` file to connect your Mage pipeline to a snowflake data warehouse. While optional,
depending on how your Snowflake DW is configured you may need to enter all information into the .yml file.
Itโs recommended to store sensitive information as Secrets. See the general [Secrets documentation](https://docs.mage.ai/development/variables/secrets)
for more information.
```YAML theme={"system"}
SNOWFLAKE_USER: username
SNOWFLAKE_PASSWORD: password
SNOWFLAKE_ACCOUNT: account_id.region
SNOWFLAKE_DEFAULT_WH: null # Optional default warehouse
SNOWFLAKE_DEFAULT_DB: null # Optional default database
SNOWFLAKE_DEFAULT_SCHEMA: null # Optional default schema
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null # Optional private key passphrase
SNOWFLAKE_PRIVATE_KEY_PATH: null # Optional private key path
SNOWFLAKE_ROLE: null # Optional role name
SNOWFLAKE_TIMEOUT: null # Optional timeout in seconds
```
Enter information for the following in the Data Exporter block
1. table\_name - requires developers enter the name of their destination table
2. database - requires developers enter the name of their destination data base
3. schema - requires developers enter the name of their destination schema
All other information is handled in the 'io\_config.yml' file.
Example Code:
```Python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Snowflake warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#snowflake
"""
table_name = 'your_table_name'
database = 'your_database_name'
schema = 'your_schema_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
database,
schema,
if_exists='replace', # Specify resolution policy if table already exists
)
```
Configure the `io_config.yml` file to connect your Mage pipeline to a Azure Blob Storage. Configure some Secrets and enter them into `io_config.yml` file.
If you need more information on entering secrets see this [documentation](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure).
```YAML theme={"system"}
AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
```
Enter information for the following in the Data Exporter block
1. container\_name - requires developers enter the name of their destination container
2. blob\_path - requires developers enter the name of their destination blob path
All other information is handled in the `io_config.yml` file.
Example Code:
```Python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_azure_blob_storage(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Azure Blob Storage.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = 'your_container_name'
blob_path = 'your_blob_path'
AzureBlobStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
container_name,
blob_path,
)
```
Configure the `io_config.yml` file to connect your Mage pipeline to a PostgreSQL database. Configure some Secrets and enter them into `io_config.yml` file.
If you need more information on entering secrets see this [documentation](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure).
```YAML theme={"system"}
POSTGRES_CONNECT_TIMEOUT: 10
POSTGRES_DBNAME: postgres
POSTGRES_SCHEMA: public # Optional
POSTGRES_USER: username
POSTGRES_PASSWORD: password
POSTGRES_HOST: hostname
POSTGRES_PORT: 5432
```
If exporting from Docker to an external machine use the host.docker.internal for `POSTGRES_HOST:`
Enter information for the following in the Data Exporter block
1. schema\_name - requires developers enter the name of their destination container
2. table\_name - requires developers enter the name of their destination blob path
All other information is handled in the `io_config.yml` file.
Example Code:
```Python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a PostgreSQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#postgresql
"""
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
)
```
Unlike other data exporters, delta lake exporters do not currently configure through the `io_config.yml` file.
They contain the necessary configurations within the exporter block itself. Lets break that down.
**Storage Options**
* `'AWS_ACCESS_KEY_ID'`: Your AWS access key ID.
* `'AWS_SECRET_ACCESS_KEY'`: Your AWS secret access key.
* `'AWS_REGION'`: The AWS region where your S3 bucket is located.
* `'AWS_S3_ALLOW_UNSAFE_RENAME'`: This option allows unsafe rename operations on S3, which might be necessary for some workflows.
Remember Secrets can be stored in Mageโs internal Secrets Manager, .YAML files, or sync directly with Cloud Secret Managers.
**Additional Configurations**
* `uri`: The S3 URI where the Delta Table is stored.
Example Code:
```Python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
"""
Export data to a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
"""
storage_options = {
'AWS_ACCESS_KEY_ID': '',
'AWS_SECRET_ACCESS_KEY': '',
'AWS_REGION': '',
'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
}
uri = 's3://[bucket]/[key]'
write_deltalake(
uri,
data=df,
mode='append', # append or overwrite
overwrite_schema=False, # set True to alter the schema when overwriting
partition_by=[],
storage_options=storage_options,
)
```
By correctly configuring these components, you can effectively streamline the data loading process into your chosen storage system,
whether it be a relational database, a data lake, or a machine learning model repository.
# Data loader
Source: https://docs.mage.ai/design/blocks/data-loader
Write your code for fetching data from a remote source or loading it from disk.
## Example
```python theme={"system"}
import io
import pandas as pd
import requests
@data_loader
def load_data_from_api(*args, **kwargs):
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
```
## How data is passed to other blocks
The object that is returned from the decorated function (e.g. `load_data_from_api`)
will be made available in all blocks that depend on this block.
## Templates
To read about Mage's data loading clients that interface with popular data
storage systems, see docs on [Data loading](/design/data-loading).
# Dynamic blocks
Source: https://docs.mage.ai/design/blocks/dynamic-blocks
Use the output of a block to dynamically create more blocks.
A dynamic block will create multiple downstream blocks at runtime.
The number of blocks it creates is equal to the number of items in the output data of the
dynamic block multiplied by the number of its downstream blocks.
For example, the following data loader block will output a list of 2 lists.
The 1st item in the list contains a list of 3 dictionaries containing user information.
```python theme={"system"}
from typing import Dict, List
@data_loader
def load_data(*args, **kwargs) -> List[List[Dict]]:
users = []
metadata = []
for i in range(3):
i += 1
users.append(dict(id=i, name=f'user_{i}'))
metadata.append(dict(block_uuid=f'for_user_{i}'))
return [
users,
metadata, # optional
]
```
If the above data loader has 2 downstream blocks (e.g. 2 blocks that depend on this data loader),
then this dynamic block will create 6 total downstream blocks at runtime
(3 users multiplied by 2 downstream blocks).
> This feature saves me when I need to securely transfer multiple media files from one location to another (SCP), especially when the files are spread across different directories and across the network. Typically, one would have to loop through each file, but secure copy (SCP) operations can fail for various reasons, and dealing with exceptions can be error-prone.
>
> An alternative approach would be to use an API trigger (like the one provided by Mage AI) to receive POST requests for each file that needs to be securely copied. However, this method can be highly resource-intensive, particularly when dealing with thousands of pipeline runs.
>
> Dynamic block, which allows me to manage source and destination specifications in a DataFrame and then dynamically execute SCP operations without having to handle exceptions manually (thanks to the "retry incomplete block" feature), is a game-changer. Not only does it save me from the hassle of exception handling, but it also keeps computational resources in check by avoiding unnecessary pipeline triggers.
>
> Thank you for your world-class project. Keep up the excellent work.
>
>
### Data output shape of a dynamic block
A dynamic block must return a list of 2 lists of dictionaries
(e.g. `List[List[Dict]]`).
For example, a data loader block returns the following output:
```json theme={"system"}
[
[
{
"id": 1,
"name": "user_1"
},
{
"id": 2,
"name": "user_2"
},
{
"id": 3,
"name": "user_3"
}
],
[
{
"block_uuid": "for_user_1"
},
{
"block_uuid": "for_user_2"
},
{
"block_uuid": "for_user_3"
}
]
]
```
#### Input data for downstream blocks
The 1st item in the list is a list of dictionaries.
These dictionaries contain data that will be passed down to all its downstream blocks.
```json theme={"system"}
[
{
"id": 1,
"name": "user_1"
},
{
"id": 2,
"name": "user_2"
},
{
"id": 3,
"name": "user_3"
}
]
```
The number of items in this list of dictionaries will correspond to how many downstream blocks get
created at runtime.
For example, if this dynamic block has a downstream transformer block with the following code:
```python theme={"system"}
from typing import Dict, List
@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
data['id'] = int(data['id']) * 100
return [data]
```
The return output data of this downstream transformer block will be:
```json theme={"system"}
[
{
"id": 100,
"name": "user_1"
}
]
```
> Note
>
> The transformer block is returning a list as its output data.
> The list of items is used as positional arguments in any downstream block of this transformer block.
#### Metadata for downstream blocks (optional)
The 2nd item in the list is a list of dictionaries.
These dictionaries contain metadata for each downstream block.
```json theme={"system"}
[
{
"block_uuid": "for_user_1"
},
{
"block_uuid": "for_user_2"
},
{
"block_uuid": "for_user_3"
}
]
```
This metadata is used to uniquely identify and create each dynamically created downstream block.
The keys in each metadata dictionary are:
| Key | Description | Required |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `block_uuid` | This value is used in combination with the downstream blockโs original UUID to construct a unique UUID across all dynamic blocks. This value must be unique within the same list of metadata dictionaries. | Yes |
If the above transformer block UUID is `anonymize_user_data` and the metadata for the 1st
dynamically created block is `{ "block_uuid": "for_user_1" }`, then the dynamically created blockโs
UUID is `anonymize_user_data:for_user_1`.
The convention is `[original_block_uuid]:[metadata_block_uuid]`.
## Dynamically created blocks
Every downstream block from a dynamic block is referred to as a dynamically created block.
The number of these blocks created are determined by the return output data of a dynamic block.
For example, if a dynamic block returns the following data:
```json theme={"system"}
[
[
{
"id": 1,
"name": "user_1"
},
{
"id": 2,
"name": "user_2"
},
{
"id": 3,
"name": "user_3"
}
],
[
{
"block_uuid": "for_user_1"
},
{
"block_uuid": "for_user_2"
},
{
"block_uuid": "for_user_3"
}
]
]
```
Then there will be 3 dynamically created blocks for every downstream block of that dynamic block.
If the dynamic block had 1 downstream block, then 3 blocks will be dynamically created.
For example, if the downstream block was a transformer with UUID `anonymize_user_data`
and the following code:
```python theme={"system"}
from typing import Dict, List
@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
data['id'] = int(data['id']) * 100
return [data]
```
There will be 3 dynamically created blocks with the following UUID and return output data:
| Dynamic block UUID | Return output data |
| -------------------------------- | ----------------------------------- |
| `anonymize_user_data:for_user_1` | `[{ "id": 100, "name": "user_1" }]` |
| `anonymize_user_data:for_user_2` | `[{ "id": 200, "name": "user_2" }]` |
| `anonymize_user_data:for_user_3` | `[{ "id": 300, "name": "user_3" }]` |
### Downstream blocks of dynamically created blocks
If a dynamically created block has downstream blocks,
those downstream blocks will be created multiple times. The number of times those are created
correspond to how many blocks were created dynamically from the dynamic block.
In the current example, 3 blocks were created dynamically
(3 users, 1 downstream block from the dynamic block).
If the above transformer block has 2 downstream blocks, then 6 more blocks will be created.
For example, if the transformer block `anonymize_user_data` has 2 downstream blocks with UUID
`clean_column_names` and `compute_engagement_score`, the following blocks will also be
dynamically created:
1. `anonymize_user_data:for_user_1`
```
โ clean_column_names:for_user_1
โ compute_engagement_score:for_user_1
```
2. `anonymize_user_data:for_user_2`
```
โ clean_column_names:for_user_2
โ compute_engagement_score:for_user_2
```
3. `anonymize_user_data:for_user_3`
```
โ clean_column_names:for_user_3
โ compute_engagement_score:for_user_3
```
The data that is passed into each of these downstream blocks
(e.g. `clean_column_names`, `compute_engagement_score`) will come from its upstream block
(e.g. `anonymize_user_data:for_user_1`, `anonymize_user_data:for_user_2`, `anonymize_user_data:for_user_3`).
For example, the block `clean_column_names:for_user_1` and `compute_engagement_score:for_user_1`
will have input data with the following shape and value:
```json theme={"system"}
{
"id": 100,
"name": "user_1"
}
```
### Reduce output
By default, dynamically created blocks will create more dynamically created blocks from their own
downstream blocks.
A dynamically created block will pass its return output data to its downstream blocks and so on.
However, a dynamically created block can be configured to reduce all the outputs across each
dynamically created block and combine them into a single list of items.
If we configure the above transformer block `anonymize_user_data` to reduce its output, then
there will only be 2 downstream blocks instead of 6.
The 2 downstream blocks will be `clean_column_names` and `compute_engagement_score`.
The input data to each of these 2 downstream blocks will be:
```json theme={"system"}
[
{
"id": 100,
"name": "user_1"
},
{
"id": 200,
"name": "user_2"
},
{
"id": 300,
"name": "user_3"
}
]
```
***
## Dynamic SQL blocks
SQL blocks can be dynamic or be a child of a dynamic block.
Here is an example pipeline with code:
```
[data_loader]
โ
[dynamic_sql_loader]
โ
[transformer]
```
`data_loader.py`: dynamic block
```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
return [[dict(uuid=1), dict(uuid=2)]]
```
`dynamic_sql_loader`: dynamic block, dynamic child
```sql theme={"system"}
SELECT * FROM {{ df_1 }}
UNION ALL
SELECT 2 AS uuid
```
`transformer`: dynamic child
```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
return data
```
When this pipeline is triggered, the following block runs will be created:
```
[data_loader]
โ โ
[dynamic_sql_loader:0] [dynamic_sql_loader:1]
โ โ โ โ
[transformer:0:0] [transformer:0:1] [transformer:1:0] [transformer:1:1]
```
***
## Tutorial guide
Follow this [guide](/guides/blocks/dynamic-blocks) to create a simple pipeline with a dynamic block,
dynamically created blocks, and a block that reduces its output.
# Scratchpad
Source: https://docs.mage.ai/design/blocks/scratchpad
Use these blocks to experiment and write throw away code.
Scratchpad blocks aren't used when executing a pipeline.
## Getting data from other blocks
If you want to read the output data from another block, use the following sample code:
```python theme={"system"}
from mage_ai.data_preparation.variable_manager import get_variable
df = get_variable('pipelined_uuid', 'block_uuid', 'output_0')
```
* Change the value of `pipelined_uuid` to match your current pipeline UUID.
* Change the value of `block_uuid` to match the UUID of the block you want to read data from.
# Sensor
Source: https://docs.mage.ai/design/blocks/sensor
A sensor is a block that continuously evaluates a condition until itโs met or until a period of time has elapsed.
In your pipeline, you can add sensors that run continuously and only complete
when an external pipeline or external block is successfully ran for a particular
partition.
Then, you can add a block in your pipeline that depends on that sensor.
If there is a block with a sensor as an upstream dependency, that block wonโt start
running until the sensor has evaluated its condition successfully.
***
## Setup
1. [Add a sensor to your pipeline](/design/blocks/sensor)
2. [Configuring a sensor](/design/core-abstractions#sensor)
***
## Example
You can add sensors to a pipeline the same way you add other types of blocks.
```python theme={"system"}
from mage_ai.orchestration.run_status_checker import check_status
@sensor
def check_condition(*args, **kwargs) -> bool:
return check_status(
'pipeline_uuid',
kwargs['execution_date'],
block_uuid='block_uuid',
hours=24,
)
```
In the above code, change `pipeline_uuid` to the actual pipeline UUID you want this sensor to keep
checking whether or not itโs finished running successfully.
The `block_uuid` positional argument is optional. If you add a value for the `block_uuid`
positional argument, then the sensor will check the status of the block in that pipeline.
Only when that specific block is finished running successfully, then the sensor will be complete.
The 'hours' positional argument is optional. If you add a value for the 'hours' positional argument,
then the sensor will check the status of the pipeline for that many hours in past from execution\_date.
The default value is 24 hours.
Sensors can also have upstream block dependencies. The output of the upstream blocks will be passed
into the sensor in the `args` parameter like in other Mage block types.
Currently, the sensor makes a check every 60 seconds.
## Templates
When adding a sensor block, here are the available templates to choose from:
1. Google BigQuery
2. MySQL
3. PostgreSQL
4. Amazon Redshift
5. Amazon S3
6. Snowflake
Below are the code examples in those templates.
### Google BigQuery
Check the results of a BigQuery query.
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path
@sensor
def query_bigquery_and_check_condition(*args, **kwargs) -> bool:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your BigQuery query' # Specify your SQL query here
loader = BigQuery.with_config(ConfigFileLoader(config_path, config_profile))
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
### MySQL
Check the results of a MySQL query.
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path
@sensor
def query_mysql_and_check_condition(*args, **kwargs) -> bool:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your MySQL query' # Specify your SQL query here
with MySQL.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
### PostgreSQL
Check the results of a PostgreSQL query.
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
@sensor
def query_postgres_and_check_condition(*args, **kwargs) -> bool:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your Postgres query' # Specify your SQL query here
with Postgres.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
### Amazon Redshift
Check the results of a Redshift query.
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path
@sensor
def query_redshift_and_check_condition(*args, **kwargs) -> bool:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your Redshift query' # Specify your SQL query here
with Redshift.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
### Amazon S3
Check if a file or folder exists in a S3 bucket.
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path
@sensor
def check_condition(*args, **kwargs) -> bool:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
s3_path = 'path/to/folder/or/file'
config_file_loader = ConfigFileLoader(config_path, config_profile)
return S3.with_config(config_file_loader).exists(
bucket_name, s3_path
)
```
### Snowflake
Check the results of a Snowflake query.
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path
@sensor
def query_snowflake_and_check_condition(*args, **kwargs) -> bool:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your Snowflake query' # Specify your SQL query here
with Snowflake.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
# Transformer
Source: https://docs.mage.ai/design/blocks/transformer
Use these blocks to clean, transform, and enhance data from other blocks.
## Example
```python theme={"system"}
from pandas import DataFrame
@transformer
def transform_df(df: DataFrame, *args, **kwargs) -> DataFrame:
return df.iloc[:10]
```
## How data is received from upstream blocks
All blocks (except Scratchpads and Sensors) pass their data from the return statement in their
decorated function to all their downstream blocks.
In order to access the data from an upstream block within a current block,
use the positional arguments of the decorated function.
For example, in the above function named `transform_df`, the output data from its upstream block
is accessed by the positional argument named `df`.
## Getting data from multiple upstream blocks
If a block has more than 1 upstream block,
then each upstream blockโs output is accessible via positional arguments.
For example, if you have 3 upstream blocks,
then there will be 3 positional arguments available within the `transform_df` function.
See below for an example:
```python theme={"system"}
from pandas import DataFrame
import pandas as pd
@transformer
def transform_df(df1, df2, df3, **kwargs) -> DataFrame:
return pd.concat([df1, df2, df3])
```
## Handling column renames in data integration pipeline
Since data passed to transform blocks is a pandas DataFrame, many operations are available on each row of your data.
It's possible to do a few dangerous things with this, for example if you have a target destination that needs it's columns renamed to match existing schemas.
In this case, there can be edge case mismatches between the data types in the source columns, and the expected data types in the destination. This is particularly important for SQL columns where the INSERT commands generated by the pipelines will use CAST expressions to try and make sure the writes succeed. However, if a column is renamed in the transformer, we will use our best guess by looking at values to decide what to CAST the column values to.
For example, a TIMESTAMPTZ value of `2023-01-09T20:01:00`, once it's renamed, can look like a string, but using an INSERT with `CAST('2023-01-09T20:01:00' AS TEXT)` to a column with data type `TIMESTAMPTZ` will fail.
### Using the `rename_columns` decorator parameter (Recommended)
The recommended way to rename columns in data integration pipelines is to use the `rename_columns` parameter in the `@transformer` decorator. This approach automatically preserves original column types and updates schema metadata including `unique_constraints` and `key_properties`.
**Benefits:**
* **Type preservation**: Original column types are maintained after renaming
* **Schema integrity**: `unique_constraints` and `key_properties` are automatically updated
* **Declarative**: Column renaming is specified at the decorator level, making intent clear
* **Only renames existing columns**: Columns that don't exist in the DataFrame are silently skipped
**Example:**
```python theme={"system"}
from pandas import DataFrame
@transformer(rename_columns={
'impressions': 'field_5627',
'clicks': 'field_5628',
'new_free_signups': 'field_5629',
'new_paid_signups': 'field_5626',
'event_timestamp': 'field_5630'
})
def transform(data, *args, **kwargs):
# Your transformation logic here
# Columns are automatically renamed after this function executes
data.insert(0, 'trashed', False)
return data
```
The `rename_columns` parameter accepts a dictionary where:
* **Keys** are the original column names (must exist in the DataFrame)
* **Values** are the new column names
**Note:** If a column in `rename_columns` doesn't exist in the DataFrame, it will be silently skipped. All target column names must be unique.
### Using manual column renaming with aliases (Legacy approach)
If you need more control or are using an older version, you can manually rename columns in your transformer function and add column "aliases" in the pipeline's data integration catalog.
1. Have a transformer block in your integration pipeline that does some column renaming:
```python theme={"system"}
from pandas import DataFrame
import pandas as pd
@transformer
def transform(data, *args, **kwargs):
# keys = source column names, values = destination column names
COLUMN_MAP = {
'impressions': 'field_5627',
'clicks': 'field_5628',
'new_free_signups': 'field_5629',
'new_paid_signups': 'field_5626',
'event_timestamp': 'field_5630'
}
# the pandas rename in place to match destination schema
data.rename(columns=COLUMN_MAP, inplace=True)
data.insert(0, 'trashed', False)
return data
```
2. Add 'aliases' property to each renamed field in your pipeline's `data_integration_catalog.json` file.
```json theme={"system"}
{
"catalog": {
"streams": [
{
"tap_stream_id": "marketing_data",
"replication_method": "INCREMENTAL",
"key_properties": [
"id"
],
"schema": {
"properties": {
"id": {
"type": [
"integer"
]
},
"impressions": {
"type": [
"null",
"number"
],
"aliases": [
"field_5627"
]
},
"clicks": {
"type": [
"null",
"number"
],
"aliases": [
"field_5628"
]
},
"new_free_signups": {
"type": [
"null",
"number"
],
"aliases": [
"field_5629"
]
},
"new_paid_signups": {
"type": [
"null",
"number"
],
"aliases": [
"field_5626"
]
},
"event_timestamp": {
"format": "date-time",
"type": [
"null",
"string"
],
"aliases": [
"field_5630"
]
}
},
"type": "object"
},
"stream": "marketing_data",
"metadata": [...],
...
}
]
}
}
```
Now, pipeline runs will use the correct data types for INSERT commands, thanks to transformer column aliases defined in the schema.
**Note:** When using the `rename_columns` decorator parameter, you don't need to manually add aliases to the catalog - the schema metadata is automatically updated. The manual approach with aliases is only needed when renaming columns inside the transformer function code.
# Transformer blocks
Source: https://docs.mage.ai/design/blocks/transformer-blocks
Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.
## Core Concepts
The core object of any transformer action is the **payload**, a JSON object that describes the transformation to perform alongside settings for that transformation. A sample transformer action payload is shown below:
```json theme={"system"}
{
"action_type": "impute",
"action_arguments": ["age", "name"],
"action_options": {
"strategy": "constant",
"value": "32"
},
"axis": "column",
"action_variables": {
"age": {
"uuid": "age",
"column_type": "number"
},
"name": {
"uuid": "hourly_salary",
"column_type": "text"
}
}
}
```
The above transformer action imputes missing values in the "age" and "hourly\_salary" columns with the constant value `32`. Here is a list of the keys used in transformer action payloads:
| Key | Description |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| axis | Axis to perform the transformation on. Either 'row' or 'column'. |
| action\_type | Type of action to perform. See [Transformer Action Reference](#transformer-action-reference) for the list of action types. |
| action\_code | Text field for specifying any code such as filters. |
| action\_arguments | The columns to perform the transformation on. |
| action\_options | Other settings to provide to the transformer action. See [Transformer Action Reference](#transformer-action-reference) for specific details on which actions require which options. |
| action\_variables | Variable metadata for each input argument (column). Specifies column name and type. |
| outputs | Variable metadata for each output column. Specifies column name and type. See [Column Types](#appendix-column-types) for information on possible column types to specify. |
This JSON formatted payload is difficult to generate so it is recommended to use the `build_transformer_action` factory to generate the payload. The above snippet generates the same payload as above:
```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
df = pd.DataFrame({...}) # your input data frame
payload = build_transformer_action(
df,
action_type=ActionType.IMPUTE,
arguments=['age'],
axis=Axis.COLUMN,
options={'strategy': 'constants', 'values': 32},
)
```
See [Building Transformer Actions](#building-transformer-actions) for detailed information on using `build_transformer_action`.
The transformer action payload, once built, can be used to construct a `BaseAction` object. This object processes the payload to identify the transformations to apply a data frame. Through this action object, the transformer action can be applied to a data frame using the `execute` function:
```python theme={"system"}
payload = build_transformer_action(...)
transformed_df = BaseAction(payload).execute(df)
```
# Building Transformer Actions
***build\_transformer\_action*** - `build_transformer_action(df: DataFrame, action_type: ActionType | str, arguments: List[str], action_code: str, options: Dict, axis: Axis | str, outputs: List[Dict]) -> Dict:`
Source: `mage_ai.data_cleaner.transformer_actions.utils.build_transformer_action`
Builds transformer action payload from arguments. The output of this function can be passed
as input to `transformer_actions.base.BaseAction` in order to perform the requested transformation.
Note: Action variables are inferred from `arguments` and `df`.
* **Args**:
* `df (DataFrame)`: The data frame to build a transformer action payload for.
* `action_type (ActionType | str)`: Transformer action to perform.
* `arguments (List[str], optional)`: Columns to perform this action on. Defaults to `[]`.
* `action_code (str, optional)`: Special code or query to execute with action. Defaults to `''`.
* `options (Dict, optional)`: Options specifying behavior of action. Defaults to `{}`.
* `axis (Union[Axis, str], optional)`: Axis of the data frame to apply the action to. Defaults to `Axis.COLUMN`.
* `outputs (List[Dict], optional)`: Specifies metadata of newly created columns. Defaults to `[]`.
* **Returns**: (`Dict`) Transformer action payload to use with `transformer_actions.base.BaseAction`
* **Example**:
```python theme={"system"}
action = build_transformer_action(
df,
action_type=ActionType.FIRST,
action_code='retailer != null and retail_price >= 0', # Enter filtering condition on rows before aggregation
arguments=['retail_price', 'review'], # Enter the columns to compute aggregate over
axis=Axis.COLUMN,
options={'groupby_columns': ['retailer']}, # Enter columns to group by
outputs=[
# The number of outputs below must match the number of arguments
{'uuid': 'first_retail_price', 'column_type': 'number_with_decimals'},
{'uuid': 'first_review', 'column_type': 'category'},
],
)
```
# Transformer Action Reference
* [Building Transformer Actions](#building-transformer-actions)
* [Transformer Action Reference](#transformer-action-reference)
* [Column Actions](#column-actions)
* [Aggregation Actions](#aggregation-actions)
* [Formatting Actions](#formatting-actions)
* [Clean Column Names](#clean-column-names)
* [Fix Syntax Errors](#fix-syntax-errors)
* [Reformat Values](#reformat-values)
* [Column Removal Actions](#column-removal-actions)
* [Select Columns](#select-columns)
* [Remove Columns](#remove-columns)
* [Row Shifting Actions](#row-shifting-actions)
* [Shift Up](#shift-up)
* [Shift Down](#shift-down)
* [Feature Scaling Actions](#feature-scaling-actions)
* [Normalize Data](#normalize-data)
* [Standardize Data](#standardize-data)
* [Other Column Actions](#other-column-actions)
* [Difference](#difference)
* [Fill In Missing Values](#fill-in-missing-values)
* [Remove Outliers](#remove-outliers)
* [Row Actions](#row-actions)
* [Drop Duplicates](#drop-duplicates)
* [Filter](#filter)
* [Remove Rows](#remove-rows)
* [Sort](#sort)
* [Appendix: Filter Syntax](#appendix-filter-syntax)
* [Appendix: Column Types](#appendix-column-types)
# Column Actions
## Aggregation Actions
Applies some aggregation function over groups determined by values in some columns. A set of grouping columns are provided, whose unique combinations of values are used to form groups. For example, consider the dataset below:
| Paid Dues | Due Amount |
| --------- | ---------- |
| Yes | \$5.00 |
| No | \$15.20 |
| Yes | \$3.50 |
| Yes | \$1.28 |
| No | \$25.03 |
The data could be grouped by "Paid Dues" to form the following groups:
| Paid Dues | Due Amount |
| --------- | ---------- |
| Yes | \$5.00 |
| Yes | \$3.50 |
| Yes | \$1.28 |
| Paid Dues | Due Amount |
| --------- | ---------- |
| No | \$15.20 |
| No | \$25.03 |
Then an aggregation function is applied to each group over the input columns. In the example, we can compute the total dues by group by aggregating over "Due Amount":
| Paid Dues | Total Due Amount |
| --------- | ---------------- |
| Yes | \$9.78 |
| No | \$40.23 |
Then the output of this aggregation is used to create a new column for each input column, where each row receives the corresponding aggregate of the group it falls into. In the example above, the final output data frame would be:
| Paid Dues | Due Amount | Total Due Amount by Payment Status |
| --------- | ---------- | ---------------------------------- |
| Yes | \$5.00 | \$9.78 |
| No | \$15.20 | \$40.23 |
| Yes | \$3.50 | \$9.78 |
| Yes | \$1.28 | \$9.78 |
| No | \$25.03 | \$40.23 |
All of these transformer actions have the same set of arguments when using `build_transformer_action`, but have different types corresponding to the different aggregation functions that could be applied to the groups.
**Example**
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.AVERAGE,
action_code='brand != null',
arguments=['revenue', 'cost'],
axis=Axis.COLUMN,
options={'groupby_columns': ['brand']},
outputs=[
{'uuid': 'avg_rev_per_brand', 'column_type': 'number_with_decimals'},
{'uuid': 'avg_cost_per_brand', 'column_type': 'number_with_decimals'},
],
)
```
**Args**
* ***action\_code:*** Filter for selecting rows before aggregation is performed. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.
* ***arguments:*** Columns to compute aggregate values over. For each argument a new column will be created to store the aggregates values over the column.
* If no arguments are provided, no aggregation is performed
* ***options*:**
* `groupby_columns` (optional) - list of columns to form groups with when aggregating. A group consists of each unique combination of values chosen from each of the columns specified in this option.
* If this is empty, aggregation is performed over the first column specified in *arguments* (only if *arguments* is nonempty - if empty no aggregation is performed) and only one column is added
```
The following options are used as a shortcut to simplify filtering over time windows.
Let $A$ and $B$ be datetime columns, and let $\text{window}$ be an integer describing the length of
a time window in seconds. If the following options are specified, rows for which the time value for
$B \in [A, A + \text{window}]$ are selected (All rows where value for $B$ must (a) be greater than
or equal to the value for $A$ and (b) less than or equal to the value $A+window$).
```
* `timestamp_feature_a` (optional) - time feature A (a column in the data frame)
* `timestamp_feature_b` (optional) - time feature B (a column in the data frame)
* `window` (optional) - max time window between feature A and feature B in seconds
If some (or none) of these options are specified, then the filtering is not performed. Regardless if this filtering is performed, *action\_code* filtering is still performed.
* ***outputs*:** Specifies the metadata for each output column of the aggregation (corresponding to the aggregation created from input argument in *arguments*). This array must be the same length as *arguments*, and each entry must contain the following metadata:
* `uuid (string)`: Name of the new column
* `column_type (string | ColumnType)`: Type of the new column ( See `mage_ai.data_cleaner.column_types.constants.ColumnType` for options)
**Aggregation Functions**
These are the possible aggregations that can be applied to each group per each input column. To use the aggregation function specify the corresponding action type in `build_transformer_action`.
| Name | Action Type Enum | Description |
| -------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------- |
| ***Average*** | `ActionType.AVERAGE` | Averages input column value over each group |
| ***Count Distinct*** | `ActionType.COUNT_DISTINCT` | Counts the number of distinct rows in each group per input column |
| ***Count*** | `ActionType.COUNT` | Counts the number of rows in each group per input column |
| ***First*** | `ActionType.FIRST` | For each input column, selects first item in each group (as appearing in the order of the original data frame) |
| ***Last*** | `ActionType.LAST` | For each input column, selects last item in each group (as appearing in the order of the original data frame) |
| ***Maximum*** | `ActionType.MAX` | Gets the maximum value per group for each input column |
| ***Median*** | `ActionType.MEDIAN` | Gets the median value per group for each input column |
| ***Minimum*** | `ActionType.MIN` | Gets the maximum value per group for each input column |
| ***Sum*** | `ActionType.SUM` | Sums over all values in each group per input column |
## Formatting Actions
These transformer actions involve reformatting column names or the values in columns
### Clean Column Names
Cleans column names according to the following rules:
1. Names are converted to snake case
2. All wrapping whitespace or underscores are removed
3. All non-alphanumeric characters (except "\_") are removed
4. Numbers are prefixed with "number\_"
5. All Python keywords are postfixed by "\_"
Snake case is chosen as
1. all characters are lower case
2. no whitespace is used (which makes referring to columns in code easier)
| Original Name | Cleaned Name | Notes |
| ----------------------- | -------------------- | ------------------------------------------------- |
| good\_name | good\_name | columns in snake case are left unchanged |
| bad case | bad\_case | |
| %@#342%34@@#342 | number\_34234342 | Symbols are removed first, then number is cleaned |
| PascalCaseColumn | pascal\_case\_column | |
| camelCaseText | camel\_case\_text | |
| \_\_\_snake\_case\_\_\_ | snake\_case | Surrounding underscores are stripped |
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.CLEAN_COLUMN_NAME,
arguments=df.columns,
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** Columns whose name to clean. If empty, no columns names are cleaned.
### Fix Syntax Errors
Marks syntax errors in column values. Syntax errors are defined as values that are improperly formatted or of the incorrect type. For example:
* A number in a text column
* An improperly formatted email, phone number, or zip code
* An improperly formatted date
Values that break these syntax errors are currently marked as:
* `"invalid"` if column is of categorical or string type
* `pandas.NaT` if column is a datetime
* `np.nan` if column is a number type
*WIP*: Intelligently fix syntax errors in these columns automatically
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.FIX_SYNTAX_ERRORS,
arguments=df.columns,
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** Columns to mark syntax errors for. If empty no columns are checked.
### Reformat Values
Reformats values in column based on requested action. The currently supported reformats are:
* Standardize capitalization (`reformat = 'caps_standardization'`): Forces text column to follow a single capitalization strategy (either lowercase or uppercase)
* Convert currencies to a number (`reformat = 'currency_to_num'`): Converts a currency value (by default stored as a string) to a decimal number. If unable to convert, doesn't perform conversion
* Supported currencies: **\$, CAD, ยฃ, โฌ, ยฅ, Rs, ๅ **
| Original String | Parsed Decimal | Notes |
| ---------------- | -------------- | ----------------------------------------------- |
| " \$ 10000" | 10000 | |
| "- ยฅ 22.324523" | -22.324523 | Negation in front of symbol supported |
| " 10000 ๅ " | 10000 | Currencies with postfixed symbols are supported |
| "0.42 โฌ" | 0.42 | |
| "- 3.42032 CAD" | -3.42032 | |
| "Rs - 100000.23" | -100000.23 | Negation after symbol is supported |
* Convert string to datetime (`reformat = 'date_time_conversion'`): Converts a string value representing a datetime to a `pandas.Timestamp` object if correctly formatted, else converts to `None`.
* Trim whitespace (`reformat = 'trim'`): Removes leading and trailing whitespace from text columns.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REFORMAT,
arguments=[],
axis=Axis.COLUMN,
options={'reformat': 'caps_standardization', 'capitalization': 'lowercase'},
)
```
**Args**
* ***arguments:*** Columns to apply the reformat action to
* ***options:***
* `reformat` (optional): The reformat action to apply to the specified columns. Can be:
* `None` - no reformat is applied
* `'caps_standardization'` - standardize capitalization
* `'currency_to_num'` - convert currency string to number
* `'date_format_conversion'` - convert datetime string to `pandas.Timestamp`
* `'trim'` - remove leading and trailing whitespace from text columns
* `capitalization` (optional): Specifies the capitalization strategy to use when standardizing capitalization. This argument is ignored unless `reformat = "caps_standardization"`. Options are `['lowercase', 'uppercase']`.
## Column Removal Actions
### Select Columns
Keeps only the specified columns; removes all other columns from data frame.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SELECT,
arguments=[],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to keep in the data frame. If empty, all columns are removed.
### Remove Columns
Drops the specified columns.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REMOVE,
arguments=[],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to remove in the data frame. If empty, no columns are removed.
## Row Shifting Actions
These actions copy a column with all rows shifted up or down by a certain amount. When rows are shifted, one or more values are deleted and some new values have to be filled in. The fill-in data types are:
| Type | Fill Value |
| ----------- | ------------ |
| Numeric | `np.nan` |
| Datetime | `pandas.NaT` |
| Other Types | `None` |
### Shift Up
Shifts all rows in a column up by one. The first entry is deleted and the last entry is turned to a null type.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SHIFT_UP,
arguments=['Name'],
axis=Axis.COLUMN,
outputs=[{'uuid': 'shifted_name', 'type': 'text'}],
)
```
**Args**
* ***arguments:*** The column to shift up. At most one column can be shifted at once:
* If multiple columns are provided as arguments, only the first is shifted up
* If no columns are provided then no shifting is performed
* ***outputs:*** Metadata for the newly added upshifted column. Must have at most a single entry containing the following information:
* `uuid`: Name of the new upshifted column
* `type`: Data type of the new upshifted column
### Shift Down
Shifts all rows up by *periods* rows. The last *periods* entries are deleted and the first *periods* are converted to a null type.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SHIFT_DOWN,
arguments=['created_time'],
axis=Axis.COLUMN,
options={'periods': 3},
outputs=[{'uuid': 'downshifted_created_time', 'type': 'datetime'}],
)
```
**Args**
* ***arguments:*** The column to shift down. At most one column can be shifted at once:
* If multiple columns are provided as arguments, only the first is shifted down
* If no columns are provided then no shifting is performed
* ***options:***
* `periods` (optional): Specifies how many rows to shift down. Defaults to 1.
* `groupby_columns` (optional): Specifies columns to group by. If specified, shifts each *group* by *periods* rows instead of the entire data frame. The resulting column is then returned. If unspecified no grouping is performed. Multiple columns can be specified to group by.
* ***outputs:*** Metadata for the newly added downshifted column. Must have at most a single entry containing the following information:
* `uuid`: Name of the new downshifted column
* `type`: Data type of the new downshifted column
## Feature Scaling Actions
These actions are used to normalize the range of independent variables or features of data. These actions are generally performed during the data preprocessing step.
| Type | Formula |
| --------------- | ---------------------- |
| Normalization | `(X-Xmin)/(Xmax-Xmin)` |
| Standardization | `(X-Xmean)/Xstd` |
### Normalize Data
Perform Normalization on the desired columns of a dataset.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.NORMALIZE,
arguments=['Column1','Column2'],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to Normalize.
* If no columns are provided then no Normalization is performed
### Standardize Data
Perform Standardization on the desired columns of a dataset.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.STANDARDIZE,
arguments=['Column1','Column2'],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to Standardize.
* If no columns are provided then no Standardization is performed
## Other Column Actions
### Difference
Adds a new column where each row is the difference between every consecutive two rows in the original column.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.DIFF,
arguments=['position'],
axis=Axis.COLUMN,
outputs=[{'uuid': 'delta_position', 'column_type': 'number_with_decimals'}],
)
```
**Args**
* ***arguments:*** The column to calculate difference on. At most one column can be specified:
* If multiple columns are provided as arguments, only the first is used in calculated
* If no columns are provided then no calculation
* ***outputs:*** Metadata for the newly added difference column. Must have at most a single entry containing the following information:
* `uuid`: Name of the difference column
* `type`: Data type of the difference column (most often will be a number)
### Fill In Missing Values
Imputes missing values using different strategies:
| Strategy | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Average | Fill missing values with the average per column. |
| Column | Fill missing values with values from another column. |
| Constant | Fill missing values with a placeholder constant. Can fill with a specific value; otherwise a default placeholder is chosen by column type. |
| Median | Fill missing values with the median per column. |
| Mode | Fill missing values with the most frequent value per column. |
| Random | Fill missing values with a randomly sampled non-null value from same column. |
| Sequential | Fill missing values with the previous row. Should only be used with timeseries columns. |
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.IMPUTE,
arguments=df.columns, # Specify columns to impute
axis=Axis.COLUMN,
options={'strategy': ImputationStrategy.CONSTANT}, # Specify imputation strategy
)
```
**Args**
* ***arguments:*** The columns to impute (fill in the missing values).
* ***options:***
* `strategy` (required): Specifies strategy to use. See table above or enum `ImputationStrategy`
* `value` (optional): Value to fill in missing value with. This argument is ignored unless `strategy = 'constant'`, in which case the constant value used to impute is `value`.
* `timeseries_cols = []` (optional): Columns which specify the time series in the data frame. This argument is ignored unless `strategy = 'sequential'`, in which case the data frame is sorted along these columns to sequentially impute data in the order of the time series.
### Remove Outliers
Removes multidimensional outliers in the data frame by analyzing numerical columns. If there are no numerical columns, no check is performed. Null values are ignored when checking for outliers.
The following methods are supported for detecting outliers:
| Method | Description |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `'itree'` | Uses the [Isolation Tree](https://scikit-learn.org/stable/modules/outlier_detection.html#isolation-forest) algorithm. Number of trees is set to 100. |
| `'lof'` | Uses the [Local Outlier Factor](https://scikit-learn.org/stable/modules/outlier_detection.html#local-outlier-factor) algorithm. Number of neighbors for computing local density scales with amount of data and contamination rate. |
| `'auto'` | Automatically determines from above methods based on number of data points and dimensionality of the data. |
Notes:
* The contamination rate for both algorithms is estimated via the per-dimension IQR (which discounts multidimensional interactions but still estimates how many outliers there likely are)
* If the input data has more than 20 dimensions, the dimensionality is reduced using PCA
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REMOVE_OUTLIERS,
arguments=df.columns, # Specify columns to remove outliers from
axis=Axis.COLUMN,
options={'method': 'auto'}, # Specify algorithm to use for outlier removal
)
```
**Args**
* ***arguments:*** The set of columns to consider when checking for outliers. Of these columns, only numerical columns will be used to calculate outliers.
* ***options:***
* `method` (optional): Specifies outlier detection method to use:
* `'itree'`
* `'lof'`
* `'auto'`
Defaults to `'auto'`.
# Row Actions
These are transformer actions that can be applied to each row of a data frame, and so have `axis = Axis.ROW`.
### Drop Duplicates
Drops duplicate rows from the data frame.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.DROP_DUPLICATE,
arguments=[],
axis=Axis.ROW,
options={'keep': 'first'},
)
```
**Args**
* ***arguments:*** the columns along which comparisons will be made to check for duplicate rows. If empty, all columns will be used to perform comparison.
* ***options:***
* `keep` (optional): specifies policy on keeping duplicates:
* `'first'` - keeps the first occurrence of the row
* `'last'` - keeps the last occurrence of the row
* `False` - removes occurrences of duplicates
Defaults to `'last'`.
### Filter
Filters row by boolean condition. Selects all rows that meets the filter condition.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.FILTER,
axis=Axis.ROW,
action_code='column1 != null and column2 <= 5'
)
```
**Args**
* ***action\_code:*** filter to apply to select rows. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.
### Remove Rows
Removes rows specified by their index.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REMOVE,
axis=Axis.ROW,
options={'rows': []},
)
```
**Args**
* ***options:***
* `rows`: List of the indices corresponding to the rows to remove. If empty, no rows are removed.
### Sort
Sorts rows in the specified order.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SORT,
arguments=[],
axis=Axis.ROW,
options={'ascending': True},
)
```
**Args**
* ***arguments:*** the columns whose value to consider while sorting rows. If empty then sort is not performed.
* ***options:***
* `ascending` (optional): Specifies how to sort rows. If `True`, then rows are sorted in ascending order; if `False` sorted in descending order. Defaults to `True`.
* If sorted in ascending order, null values are placed at the top of the data frame in the same order as they appeared in the original data frame
* If sorted in descending order, null values are placed at the bottom of the data frame in the same order as they appeared in the original data frame
* `ascendings` (optional): A list of booleans specifying if each column specified in *arguments* should be sorted in ascending order (must be the same length). If specified, this list will take precedence over the `ascending` option.
# Appendix: Filter Syntax
Valid filters are composed of clauses of the form:
```
[column_name] [op] [value or column_name]
```
* List of valid operations: `==, !=, <=, <, >, >=, contains, not contains`
* `contains` (and `not contains`) checks if the value is a substring (or not a substring) of values in the column specified.
* The right hand side of each clause can either be a literal value (boolean, number, string) or another column to compare to.
Clauses can be connected using the `and` and `or` operators and grouped with parentheses.
```
(brand != null and value > 60) or (discounted == null)
```
**Rules on Naming Columns**
* If column names contain only alphanumeric characters and the special character '\_', then the raw column name can used to filter code.
Example: `age != null and price_of_item >= 50.23`
* If columns contain whitespace, newlines, and some of the following special characters: `+=-*&^%$! ?~|<>(){}[],.`, then the quoted column name can be used to filter code. Any quote character can be used.
Example: `("+= -*&^%$! ?~ |<>" == False and ' kas22d fe ($)' > 50.23) or ("dis>>> ??cou nted" contains "partial")`
* If columns contain any other special characters apart from those listed above, the column cannot be filtered. This is a fundamental limitation of pandas, see [here](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html?highlight=query#pandas.DataFrame.query:~:text=During%20parsing%20a,not%20be%20escaped) for more information.
# Appendix: Column Types
These are list of column types used in the Mage tool. One of these column types should be specified for every entry in the *options* argument of a transformer action. The enum source is `mage_ai.data_cleaner.column_types.constants.ColumnType`.
| Type | Enum | String Value | Description |
| ------------------------- | -------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| Category | `ColumnType.CATEGORY` | "category" | Categorical variable that takes one of some finite set of values. This set has at most 255 entries. |
| High Cardinality Category | `ColumnType.CATEGORY_HIGH_CARDINALITY` | "category\_high\_cardinality" | Categorical variable that takes one of some finite set of values. This set has more than 255 entries. |
| Datetime | `ColumnType.DATETIME` | "datetime" | Datetime objects or strings representing a datetime. |
| Email | `ColumnType.EMAIL` | "email" | Strings that contains emails. |
| List | `ColumnType.LIST` | "list" | Lists of primitives (integers, strings, booleans, floats) of variable length. |
| Number | `ColumnType.NUMBER` | "number" | Integer numbers. Can be a Python integer, NumPy integer or a stringified integer. |
| Floating Point Number | `ColumnType.NUMBER_WITH_DECIMALS` | "number\_with\_decimals" | Floating point numbers. Can be a Python float, NumPy float, or a stringified float. |
| Phone Number | `ColumnType.PHONE_NUMBER` | "phone\_number" | Phone numbers (either in string form or integer form). |
| Text | `ColumnType.TEXT` | "text" | Text values, such as sentences or paragraphs. |
| Binary | `ColumnType.TRUE_OR_FALSE` | "true\_or\_false" | A variable that takes on two different values (including booleans). |
| Zip Code | `ColumnType.ZIP_CODE` | "zip\_code" | Zip codes (either in string form or integer form). |
# Abstractions
Source: https://docs.mage.ai/design/core-abstractions
These are the fundamental concepts that Mage uses to operate.
## Project
A Mage project forms the basis for all the work you do in Mage. It contains the code for your data pipelines, blocks, and other project-specific data. You can think of it like a GitHub repoโ it can even track your [file history, too](https://docs.mage.ai/guides/version-control/file-versions#file-versioning-and-history).
Check out our detailed [project structure](/design/abstractions/project-structure) docs for more info.
## Pipeline
A pipeline contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.
Each pipeline is represented by a YAML file (here's an [example](https://github.com/mage-ai/demo_etl_pipeline/blob/master/pipelines/etl_demo/metadata.yaml)) in the `pipelines` folder under the Mage project directory.
Pipelines can be filtered, tagged, and grouped by various properties. See our [docs page](/pipelines/pipeline-tagging) for more info.
## Block
A block is a file with code that can be executed independently or within a
pipeline. Together, blocks form a Directed Acyclic Graph (DAG), which we call pipelines. A block won't start running in a pipeline until all its upstream dependencies are met.
There are 8 types of blocks.
1. Data loader
2. Transformer
3. Data exporter
4. Scratchpad
5. Sensor
6. dbt
7. Extensions
8. Callbacks
For more information, please see the [documentation on blocks](/design/blocks)
Here is an example of a
[data loader block](https://github.com/mage-ai/demo%5Fetl%5Fpipeline/blob/master/data%5Floaders/load%5Fdataset.py)
and a snippet of its code:
```bash theme={"system"}
@data_loader
def load_data_from_api() -> DataFrame:
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
```
Each block file is stored in a folder that matches its respective type (e.g.
transformers are stored in `[project_name]/transformers/`.
## Sensor
A sensor is a block that continuously evaluates a condition until it's met or
until a period of time has elapsed.
If there is a block with a sensor as an upstream dependency, that block won't
start running until the sensor has evaluated its condition successfully.
Sensors can check for anything. Examples of common sensors check for:
* Does a table exist (e.g. `mage.users_v1`)?
* Does a partition of a table exist (e.g. `ds = 2022-12-31`)?
* Does a file in a remote location exist (e.g. `S3`)?
* Has another pipeline finished running successfully?
* Has a block from another pipeline finished running successfully?
* Has a pipeline run or block run failed?
Here is an example of a sensor that will keep checking to see if pipeline
`transform_users` has finished running successfully for the current execution
date:
```bash theme={"system"}
from mage_ai.orchestration.run_status_checker import check_status
@sensor
def check_condition(**kwargs) -> bool:
return check_status(
'pipeline_uuid',
kwargs['execution_date'],
)
```
This example is using a helper function called `check_status` that handles the
logic for retrieving the status of a pipeline run for `transform_users` on the
current execution date. You can optionally pass block uuid and hours
parameters to `check_status` to check the status of a block run or within a
interval of time.
## Data product
Every block produces data after it's been executed. These are called **data
products** in Mage.
Data validation occurs whenever a block is executed.
Additionally, each data product produced by a block can be automatically
partitioned, versioned, and backfilled.
Some examples of data products produced by blocks:
* ๐ Dataset/Table in a database, data warehouse, etc.
* ๐ผ๏ธ Image
* ๐น Video
* ๐ Text file
* ๐ง Audio file
## Trigger
A trigger is a set of instructions that determine when or how a pipeline should
run. A pipeline can have 1 or more triggers.
There are 3 types of triggers:
1. Schedule
2. Event
3. API
### Schedule
A schedule-type trigger will instruct the pipeline to run after a start date and
on a set interval.
Currently, the frequency pipelines can be scheduled for include:
* Run exactly once
* Hourly
* Daily
* Weekly
* Monthly
* Every N minutes (coming soon)
### Event
An event-type trigger will instruct the pipeline to run whenever a specific
event occurs.
For example, you can have a pipeline start running when a database query is
finished executing or when a new object is created in Amazon S3 or Google
Storage.
You can also trigger a pipeline using your own custom event by making a `POST`
request to the `http://localhost/api/events` endpoint with a custom event
payload.
Check out this
[**tutorial**](/orchestration/triggers/events/aws) on how
to create an event trigger.
### API
An API-type trigger will instruct the pipeline to run after a specific API call
is made.
You can make a POST request to an endpoint provided in the UI when creating or
editing a trigger. You can optionally include runtime variables in your request
payload.
## Run
A run record stores information about when it was started, its status, when it
was completed, any runtime variables used in the execution of the pipeline or
block, etc.
Every time a pipeline or a block is executed (outside of the notebook while
building the pipeline and block), a run record is created in a database.
There are 2 types of runs:
### Pipeline run
This contains information about the entire pipeline execution.
### Block run
Every time a pipeline is executed, each block in the pipeline will be executed
and potentially create a block run record.
## Log
A log is a file that contains system output information.
It's created whenever a pipeline or block is ran.
Logs can contain information about the internal state of a run, text that is
outputted by loggers or `print` statements in blocks, or errors and stack traces
during code execution.
Here is an example of a log in the
[**Data pipeline management**](/about/features) UI:
Logs are stored on disk wherever Mage is running. However, you can configure
where you want log files written to (e.g. Amazon S3, Google Storage, etc).
## Backfill
A backfill creates 1 or more pipeline runs for a pipeline.
# Core Design Principles
Source: https://docs.mage.ai/design/core-design-principles
Every user experience and technical design decision adheres to these principles.
## ๐ป Easy developer experience
Open-source engine that comes with a custom notebook UI for building data
pipelines.
* Mage comes with a specialized notebook UI for building data pipelines.
* Use Python and SQL (more languages coming soon) together in the same pipeline
for ultimate flexibility.
* Set up locally and get started developing with a single command.
* Deploying to production is fast using native integrations with major cloud
providers.
## ๐ข Engineering best practices built-in
Build and deploy data pipelines using modular code. No more writing throwaway
code or trying to turn notebooks into scripts.
* Writing reusable code is easy because every block in your data pipeline is a
standalone file.
* Data validation is written into each block and tested every time a block is
run.
* Operationalizing your data pipelines is easy with built-in observability, data
quality monitoring, and lineage.
* Each block of code has a single responsibility: load data from a source,
transform data, or export data anywhere.
## ๐ณ Data is a first class citizen
Designed from the ground up specifically for running data-intensive workflows.
* Every block run produces a data product (e.g. dataset, unstructured data,
etc.)
* Every data product can be automatically partitioned.
* Each pipeline and data product can be versioned.
* Backfilling data products is a core function and operation.
## ๐ช Scaling is made simple
Analyze and process large data quickly for rapid iteration.
* Transform very large datasets through a native integration with Spark.
* Handle data intensive transformations with built-in distributed computing
(e.g. Dask, Ray) \[coming soon].
* Run thousands of pipelines simultaneously and manage transparently through a
collaborative UI.
* Execute SQL queries in your data warehouse to process heavy workloads.
# Data loader utilities
Source: https://docs.mage.ai/design/data-loading
Mage provides data loading clients that simplify loading and exporting data in your pipelines, allowing you to spend more time analyzing and transforming your data for ML tasks. Currently, Mage includes clients for the following data sources:
**File Storage:**
* [AWS S3](#s3)
* [Azure Blob Storage](#azureblobstorage)
* [Azure Data Lake Storage](#azure-data-lake-storage) (Pro only)
* [Filesystem](#fileio)
* [Google Cloud Storage](#googlecloudstorage)
**Databases:**
* [AWS Redshift](#redshift)
* [ClickHouse](#clickhouse)
* [Databricks SQL](#databricks-sql) (Pro only)
* [DuckDB](#duckdb)
* [MySQL](#mysql)
* [MSSQL](#mssql)
* [MongoDB](#mongodb)
* [OracleDB](#oracledb)
* [PostgreSQL](#postgresql)
* [SQLite](#sqlite)
* [Snowflake](#snowflake)
* [Spark](#spark)
* [Trino](#trino)
**Data Warehouses:**
* [Google BigQuery](#bigquery)
* [Microsoft Fabric Warehouse](#microsoft-fabric-warehouse) (Pro only)
**APIs & Services:**
* [Google Sheets](#googlesheets)
Mage's data loading clients fall into two categories:
* **File-Loading Clients** - import/export data between files and your pipeline. Includes both local filesystem storage and external filesystem storage (like AWS S3)
* **Database Clients** - imports data from an external database into your pipeline and exports data frames back into that database. These database clients include the `execute` method which execute your queries in the connected database. The Google BigQuery client follows this structure.
* A subcategory of database clients are **connection-based clients**, which wrap a connection to the database. This connection is used to execute transactions (sets of queries) on the database, which are either committed (saved to database) or rolled-back (deleted). Clients for PostgreSQL, Redshift, and Snowflake follow this structure.
## Example: Loading data from a file
While traditional Pandas IO procedures can be utilized to load files into your pipeline, Mage provides the `mage_ai.io.file.FileIO` client as a convenience wrapper.
The following code uses the `load` function of the `FileIO` client to load the Titanic survival dataset from a CSV file into a Pandas DataFrame for use in your pipeline. All data loaders can be initialized with the `verbose = True` parameter, which will print the current action the data loading client is performing. This parameter defaults to `False`.
```python theme={"system"}
loader = FileIO(verbose=True)
df = loader.load(
'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
)
```
Then the `export` function is used to save this data frame back to file, this time in JSON format
```python theme={"system"}
loader.export(df, './titanic_survival.json', orient='records')
```
Any formatting settings (such as specifying the data frame *orient*) can be passed as keyword arguments to `load` and `export`. These arguments are passed to Pandas IO procedures for the requested file format, enabling fine-grained control over how your data is loaded and exported.
As the data loader was constructed with the verbose parameter set to `True`, the above operations would print the following output describing the actions of the data loader.
```bash theme={"system"}
FileIO initialized
โโ Loading data frame from 'https://raw.githubusercontent.com/datasciencedojodatasets/master/titanic.csv'...DONE
โโ Exporting data frame to './titanic_survival.json'...DONE
```
See the [FileIO](#fileio) API to learn more about loading data from filesystem.
## Example: Loading data from Snowflake warehouse
Loading data from a Snowflake data warehouse is made easy using the `mage_ai.io.snowflake.Snowflake` data loading client. In order to authenticate access to a Snowflake warehouse, the client requires the associated Snowflake credentials:
* Account Username
* Account Password
* Account ID (including your region) (Ex: *example.us-west-2*)
These parameters can be manually specified as input to the data loading client (see [Snowflake](#snowflake) API). However we recommend using a [Configuration Loader](#configuration-settings) to handle loading these secrets. If you used `mage init` to create your project repository, you can store these values in your `io_config.yaml` file and use `mage_ai.io.config.ConfigFileLoader` to construct the data loader client.
For detailed information about setting up `io_config.yaml`, see the [IO Config Setup documentation](/development/io_config_setup).
An example `io_config.yaml` file in this instance would be:
```yaml theme={"system"}
default:
SNOWFLAKE_USER: my_username
SNOWFLAKE_PASSWORD: my_password
SNOWFLAKE_ACCOUNT: example.us-west-2
```
with which a Snowflake data loading client can be constructed as:
```python theme={"system"}
config = ConfigFileLoader('io_config.yaml', 'default')
loader = Snowflake.with_config(config, verbose=True)
```
To learn more about using configuration settings, see [Configuration Settings](#configuration-settings).
Then the following code uses the functions:
* `execute()` - executes an arbitrary query on your data warehouse. In this case, the warehouse, database, and schema to use are selected.
* `load()` - loads the results of a `SELECT` query into a Pandas DataFrame.
* `export()` - stores data frame as a table in your data warehouse. If the table exists, then the data is appended by default (and can be configured with other behavior, see [Snowflake](#snowflake) API). If the table doesn't exist, then the table is created with the given schema name and table name (the table data types are inferred from the Python data type).
```python theme={"system"}
with loader:
loader.execute('USE WAREHOUSE my_warehouse;')
loader.execute('USE DATABASE my_database;')
loader.execute('USE SCHEMA my_schema;')
df = loader.load('SELECT * FROM test_table;')
loader.export(df, 'my_schema', 'test_table')
```
The `loader` object manages a direct connection to your Snowflake data warehouse, so it is important to make sure that your connection is closed once your operations are completed. You can manually use `loader.open()` and `loader.close()` to open and close the connection to your data warehouse or automatically manage the connection with a context manager.
To learn more about loading data from Snowflake, see the [Snowflake](#snowflake) API for more details on member functions and usage.
# Client APIs
This section covers the API for using the following data loaders.
## FileIO
`mage_ai.io.file.FileIO`
Handles data transfer between the filesystem and the Mage app. The `FileIO` client currently supports importing and exporting with the following file formats:
* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel
Constructor
`__init__(self, verbose: bool)`
Initializes the FileIO data loading client.
* **Args**
* `verbose (bool)`: Enables verbose output printing. Defaults to False.
Methods
***export*** - `export(df: DataFrame, filepath: os.PathLike, format: FileFormat | str = None, **kwargs) -> None:`
Exports the input data frame to the file specified.
If the format is HDF5, the default key under which the data frame is stored is the stem of the filename. For example, if the file to write the data frame to is 'storage/my\_data\_frame.hdf5', the key would be 'my\_data\_frame'. This can be overridden using the `key` keyword argument.
* **Args**:
* `df (DataFrame)`: Data frame to export.
* `filepath (os.PathLike)`: Filepath to export data frame to.
* `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Writer |
| ------- | ---------------------------------------------------------------------------------------------------------------------- |
| CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
| JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
| Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
| HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
| XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
| Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |
***load*** - `load(filepath: os.PathLike, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT **kwargs) -> DataFrame`
Loads the data frame from the filepath specified. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).
* **Args**:
* `filepath (os.PathLike)`: Filepath to load data frame from.
* `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
* `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Reader |
| ------- | ------------------------------------------------------------------------------------------------------ |
| CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
| JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
| Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
| HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
| XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
| Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |
* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.
## S3
`mage_ai.io.s3.S3`
Handles data transfer between an S3 bucket and the Mage app. The `S3` client supports importing and exporting with the following file formats:
* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel
If an IAM profile is not set up using `aws configure`, then AWS credentials for accesses the S3 bucket can be manually specified through either Mage's [configuration loader](#configuration-settings) system or through keyword arguments in the constructor (see constructor).
If using configuration settings, specify the following keys:
```yaml theme={"system"}
AWS_ACCESS_KEY_ID: AWS Access Key ID credential
AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
AWS_REGION: AWS Region
```
Constructor
`__init__(self, verbose: bool)`
Initializes the S3 data loading client.
* **Args**
* `verbose (bool)`: Enables verbose output printing. Defaults to False.
* If IAM profile is not set up using `aws configure` and Mage's configuration loader is not used, then specify your AWS credentials through the following keyword arguments:
* `aws_access_key_id (str)`: AWS Access Key ID credential
* `aws_secret_access_key (str)`: AWS Secret Access Key credential
* `aws_region (str)`: Region associated with AWS IAM profile
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> S3`
Creates S3 data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`S3`) The constructed dataloader using this method
Methods
***export*** - `export(df: DataFrame, bucket_name: str, object_key: str, format: FileFormat | str = None, **kwargs) -> None:`
Exports data frame to an S3 bucket.
If the format is HDF5, the default key under which the data frame is stored is the stem of the filename. For example, if the file to write the data frame to is 'storage/my\_data\_frame.hdf5', the key would be 'my\_data\_frame'. This can be overridden using the `key` keyword argument.
* **Args**:
* `data (Union[DataFrame, str])`: Data frame or file path to export.
* `bucket_name (str)`: Name of the bucket to export data frame to.
* `object_key (str)`: Object key in S3 bucket to export data frame to.
* `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Writer |
| ------- | ---------------------------------------------------------------------------------------------------------------------- |
| CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
| JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
| Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
| HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
| XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
| Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |
***load*** - `load(bucket_name: str, object_key: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT **kwargs) -> DataFrame`
Loads data from object in S3 bucket into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).
* **Args**:
* `bucket_name (str)`: Name of the bucket to load data from.
* `object_key (str)`: Key of the object in S3 bucket to load data from.
* `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
* `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Reader |
| ------- | ------------------------------------------------------------------------------------------------------ |
| CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
| JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
| Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
| HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
| XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
| Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |
* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.
## FileIO
`mage_ai.io.file.FileIO`
Handles data transfer between the filesystem and the Mage app. The `FileIO` client currently supports importing and exporting with the following file formats:
* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel
Constructor
`__init__(self, verbose: bool)`
Initializes the FileIO data loading client.
* **Args**
* `verbose (bool)`: Enables verbose output printing. Defaults to False.
Methods
***export*** - `export(df: DataFrame, filepath: os.PathLike, format: FileFormat | str = None, **kwargs) -> None:`
Exports the input data frame to the file specified.
If the format is HDF5, the default key under which the data frame is stored is the stem of the filename. For example, if the file to write the data frame to is 'storage/my\_data\_frame.hdf5', the key would be 'my\_data\_frame'. This can be overridden using the `key` keyword argument.
* **Args**:
* `df (DataFrame)`: Data frame to export.
* `filepath (os.PathLike)`: Filepath to export data frame to.
* `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Writer |
| ------- | ---------------------------------------------------------------------------------------------------------------------- |
| CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
| JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
| Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
| HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
| XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
| Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |
***load*** - `load(filepath: os.PathLike, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT **kwargs) -> DataFrame`
Loads the data frame from the filepath specified. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).
* **Args**:
* `filepath (os.PathLike)`: Filepath to load data frame from.
* `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
* `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Reader |
| ------- | ------------------------------------------------------------------------------------------------------ |
| CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
| JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
| Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
| HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
| XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
| Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |
* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.
## GoogleCloudStorage
`mage_ai.io.google_cloud_storage.GoogleCloudStorage`
Handles data transfer between a Google Cloud Storage bucket and the Mage app. Supports loading files of any of the following types:
* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel
If `GOOGLE_APPLICATION_CREDENTIALS` environment is set, no further arguments are needed other than those specified below. Otherwise, use the factory method `with_config` to construct the data loader using manually specified credentials.
To authenticate (and authorize) access to Google Cloud Storage, credentials must be provided.
Below are the different ways to access those credentials:
* Define the `GOOGLE_APPLICATION_CREDENTIALS` environment variable holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY_FILEPATH` key with your [configuration loader](#configuration-settings) or the `path_to_credentials` keyword argument with the client constructor holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY` key with your [configuration loader](#configuration-settings) or the `credentials_mapping` keyword argument with the client constructor holding a mapping sharing the same contents as your service key
* if using a configuration file, be careful to wrap your service key values in quotes so the YAML parser reads the settings correctly
* Manually pass the `google.oauth2.service_account.Credentials` object with the keyword argument `credentials`
Constructor
`__init__(self, verbose: bool = False, **kwargs)`
Initializes the GoogleCloudStorage data loading client.
* **Args**
* `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
* `credentials_mapping (Mapping[str, str])` - Mapping object corresponding to your service account key. See instructions above on when to use this keyword argument
* `path_to_credentials (str)` - Filepath to service account key. See instructions above on when to use this keyword argument.
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> GoogleCloudStorage`
Creates GoogleCloudStorage data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`GoogleCloudStorage`) The constructed dataloader using this method
Methods
***export*** - `export(df: DataFrame, bucket_name: str, object_key: str, format: FileFormat | str = None, **kwargs) -> None:`
Exports data frame to a GoogleCloudStorage bucket.
* **Args**:
* `data (Union[DataFrame, str])`: Data frame or file path to export.
* `bucket_name (str)`: Name of the bucket to export data frame to.
* `object_key (str)`: Object key in GoogleCloudStorage bucket to export data frame to.
* `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Writer |
| ------- | ---------------------------------------------------------------------------------------------------------------------- |
| CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
| JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
| Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
| HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
| XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
| Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |
***load*** - `load(bucket_name: str, object_key: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT, **kwargs) -> DataFrame`
Loads data from object in GoogleCloudStorage bucket into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).
* **Args**:
* `bucket_name (str)`: Name of the bucket to load data from.
* `object_key (str)`: Key of the object in GoogleCloudStorage bucket to load data from.
* `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
* `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Reader |
| ------- | ------------------------------------------------------------------------------------------------------ |
| CSV | [*read\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) |
| JSON | [*read\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html) |
| Parquet | [*read\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html) |
| HDF5 | [*read\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html) |
| XML | [*read\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_xml.html) |
| Excel | [*read\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) |
* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.
## AzureBlobStorage
`mage_ai.io.azure_blob_storage.AzureBlobStorage`
Handles data transfer between an Azure Blob Storage container and Mage. Supports loading files of the following types:
* CSV
* JSON
* Parquet
* HDF5
* XML
* Excel
To authenticate access to Azure Blob Storage, the following environment variables must be set:
* `AZURE_CLIENT_ID`
* `AZURE_CLIENT_SECRET`
* `AZURE_TENANT_ID`
* `AZURE_STORAGE_ACCOUNT_NAME`
By default, these correspond to variables of the same name in `io_config.yaml`.
Constructor
Initializes data loader from an Azure Blob Storage container.
* **Args**
* `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> AzureBlobStorage`
Creates AzureBlobStorage data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`AzureBlobStorage`) The constructed dataloader using this method
Methods
***export*** - `export(df: DataFrame, bucket_name: str, object_key: str, format: FileFormat | str = None, **kwargs) -> None:`
Exports data frame to an Azure Blob Storage container.
* **Args**:
* `df (DataFrame)`: Data frame to export.
* `container_name (str)`: Name of the Azure container to export data to.
* `blob_path (str)`: The desired output path of the data in your Azure Blob
* `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:
| Format | Pandas Writer |
| ------- | ---------------------------------------------------------------------------------------------------------------------- |
| CSV | [*DataFrame.to\_csv*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) |
| JSON | [*DataFrame.to\_json*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) |
| Parquet | [*DataFrame.to\_parquet*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) |
| HDF5 | [*DataFrame.to\_hdf*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html) |
| XML | [*DataFrame.to\_xml*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html) |
| Excel | [*DataFrame.to\_excel*](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html) |
***load*** - `load(container_name: str, blob_path: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT, **kwargs) -> DataFrame`
Loads data from object in Azure Blob Storage into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).
## GoogleSheets
`mage_ai.io.google_sheets.GoogleSheets`
Handles data transfer between a Google Sheets spreadsheet and the Mage app.
Authentication is the same for our other Google Cloud data loaders. Read more about configuration [here](integrations/databases/GoogleSheets#configuration).
The Google Sheets class is defined [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/google_sheets.py).
## BigQuery
`mage_ai.io.bigquery.BigQuery`
Handles data transfer between a BigQuery data warehouse and the Mage app.
Authentication with a Google BigQuery warehouse requires specifying the service account key for the service account that has access to the BigQuery warehouse. There are four ways to provide this service key:
* Define the `GOOGLE_APPLICATION_CREDENTIALS` environment variable holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY_FILEPATH` key with your [configuration loader](#configuration-settings) or the `path_to_credentials` keyword argument with the client constructor holding the filepath to your service account key
* Define the `GOOGLE_SERVICE_ACC_KEY` key with your [configuration loader](#configuration-settings) or the `credentials_mapping` keyword argument with the client constructor holding a mapping sharing the same contents as your service key
* if using a configuration file, be careful to wrap your service key values in quotes so the YAML parser reads the settings correctly
* Manually pass the `google.oauth2.service_account.Credentials` object with the keyword argument `credentials`
Constructor
`__init__(self, **kwargs)`
Initializes the BigQuery data loading client.
* **Args**
* `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
* `credentials_mapping (Mapping[str, str])` - Mapping object corresponding to your service account key. See instructions above on when to use this keyword argument
* `path_to_credentials (str)` - Filepath to service account key. See instructions above on when to use this keyword argument.
All other keyword arguments can be found in the [Google BigQuery Python Client docs](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html?highlight=client#google-cloud-bigquery-client-client)
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> BigQuery`
Creates BigQuery data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`BigQuery`) BigQuery data loading client constructed using this method
***with\_credentials\_file*** - `with_credentials_file(cls, path_to_credentials: str, **kwargs) -> BigQuery`
Constructs BigQuery data loader using the file containing the service account key.
* **Args**:
* `path_to_credentials (str)`: Path to the credentials file.
* `**kwargs`: Additional parameters to pass to BigQuery client constructor.
* **Returns**: (`BigQuery`) BigQuery data loading client constructed using this method
***with\_credentials\_object*** - `with_credentials_object(cls, credentials: Mapping[str, str], **kwargs) -> BigQuery`
Constructs BigQuery data loader using manually specified service account key credentials.
* **Args**:
* `credentials (Mapping[str, str])`: Mapping containing same key-value pairs as a service account key.
* `**kwargs`: Additional parameters to pass to BigQuery client constructor.
* **Returns**: (`BigQuery`) BigQuery data loading client constructed using this method
Methods
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected BigQuery warehouse.
* **Args**:
* `query_string (str)`: Query to execute on the BigQuery warehouse.
* `**kwargs`: Additional arguments to pass to query, such as query configurations. See [*Client.query()*](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html?highlight=client#google.cloud.bigquery.client.Client.query) docs for additional arguments.
***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, **kwargs) -> None`
Exports a data frame to a Google BigQuery warehouse. If table doesn't exist, the table is automatically created.
* **Args**:
* `df (DataFrame)`: Data frame to export
* `table_id (str)`: ID of the table to export the data frame to. If of the format
`"your-project.your_dataset.your_table_name"`. If this table exists, the table schema must match the data frame schema. If this table doesn't exist, the table schema is automatically inferred.
* `if_exists (str)`: Specifies export policy if table exists. Either - `'fail'`: throw an error. - `'replace'`: drops existing table and creates new table of same name. - `'append'`: appends data frame to existing table. In this case the schema must match the original table.
Defaults to `'replace'`. If `write_disposition` is specified as a keyword argument, this parameter
is ignored (as both define the same functionality).
* `**configuration_params`: Configuration parameters for export job. See valid configuration parameters at [*LoadJobConfig*](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.LoadJobConfig.html#google.cloud.bigquery.job.LoadJobConfig) docs.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
When a select query is provided, this function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional parameters to pass to the query. See [Google BigQuery Python client](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html?highlight=client#google.cloud.bigquery.client.Client.query) docs for additional arguments.
***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`
Sample data from a table in the BigQuery warehouse. Sample is not guaranteed to be random.
* **Args**:
* `schema (str)`: The schema to select the table from.
* `size (int)`: The number of rows to sample. Defaults to 10,000,000.
* `table (str)`: The table to sample from in the connected database.
* **Returns**: (`DataFrame`) Sampled data from the table.
## PostgreSQL
`mage_ai.io.postgres.Postgres`
Handles data transfer between a PostgreSQL database and the Mage app. The `Postgres` client utilizes the following keys to connect the PostgreSQL database.
```yaml theme={"system"}
POSTGRES_DBNAME: PostgreSQL database name
POSTGRES_USER: PostgreSQL database login username
POSTGRES_PASSWORD: PostgreSQL database login password
POSTGRES_HOST: PostgreSQL database hostname
POSTGRES_PORT: PostgreSQL database port
```
Constructor
`__init__(self, **kwargs)`
Initializes the Postgres data loading client.
* **Args**:
* `dbname (str)`: The name of the database to connect to.
* `user (str)`: The user with which to connect to the database with.
* `password (str)`: The login password for the user.
* `host (str)`: Host address for database.
* `port (str)`: Port on which the database is running.
* `verbose (bool)`: Enables verbose output printing. Defaults to `False`.
* `**kwargs`: Additional settings for creating psycopg2 connection
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Postgres`
Creates Postgres data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Postgres`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to PostgreSQL database.
***commit*** - `commit()`
Saves all changes made to the database since the previous transaction.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected PostgreSQL database. Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the PostgreSQL database.
* `**kwargs`: Additional parameters to pass to the query. See [`psycopg2` docs](https://www.psycopg.org/docs/usage.html#query-parameters) for configuring query parameters.
***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, index: bool, **kwargs) -> None`
Exports data frame to the PostgreSQL database from a Pandas data frame. If table doesn't exist, the table is automatically created. If the schema doesn't exist, the schema is also created.
Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to the PostgreSQL database.
* `table_name (str)`: Name of the table to export data to (excluding database and schema).
* `database (str)`: Name of the database in which the table is located.
* `schema (str)`: Name of the schema in which the table is located.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
* `**kwargs`: Additional arguments to pass to writer. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for details on UPSERT (`unique_constraints`, `unique_conflict_method`) and other advanced options like `overwrite_types`, `query_string`, `auto_clean_name`, etc.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional parameters to pass to the query. See [`psycopg2` docs](https://www.psycopg.org/docs/usage.html#query-parameters) for configuring query parameters.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to PostgreSQL database.
***rollback*** - `rollback()`
Rolls back (deletes) all database changes made since the last transaction.
***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`
Sample data from a table in the PostgreSQL database. Sample is not guaranteed to be random.
* **Args**:
* `schema (str)`: The schema to select the table from.
* `size (int)`: The number of rows to sample. Defaults to 10,000,000.
* `table (str)`: The table to sample from in the connected database.
* **Returns**: (`DataFrame`) Sampled data from the table.
## Advanced Export Parameters for SQL Databases
The following advanced export parameters are available for all SQL database clients (PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, ClickHouse, Trino, Databricks SQL, SQLite, etc.) that inherit from `BaseSQL`. These parameters provide fine-grained control over how data is exported.
> **Note:** UPSERT (UPDATE or INSERT) functionality is only supported on a subset of SQL databases (e.g., PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, SQLite). It is **not** supported on Redshift, ClickHouse, Trino or Databricks SQL. See below for details.
### UPSERT (UPDATE or INSERT)
UPSERT allows you to update existing rows when they match unique constraints, or insert new rows if they don't exist. This is useful for maintaining data integrity and avoiding duplicate entries.
To enable UPSERT functionality, you need to specify both `unique_constraints` and `unique_conflict_method` parameters:
* `unique_constraints (List[str])`: A list of column names that form the unique constraint. These columns are used to identify existing rows for updates.
* `unique_conflict_method (str)`: Specifies how to handle conflicts when unique constraints are violated. Can be either:
* `'UPDATE'` (or use constant `UNIQUE_CONFLICT_METHOD_UPDATE`): Updates existing rows that match the unique constraints with new values from the data frame, and inserts new rows that don't match.
* `'IGNORE'` (or use constant `UNIQUE_CONFLICT_METHOD_IGNORE`): Ignores rows that would violate unique constraints (does not update or insert).
**Example: UPSERT with PostgreSQL**
```python theme={"system"}
from mage_ai.io.postgres import Postgres
from mage_ai.io.constants import UNIQUE_CONFLICT_METHOD_UPDATE
loader = Postgres(...)
with loader:
loader.export(
df,
table_name='users',
schema='public',
if_exists='append',
unique_constraints=['user_id', 'email'], # Columns that form unique constraint
unique_conflict_method=UNIQUE_CONFLICT_METHOD_UPDATE, # Update existing, insert new
)
loader.commit()
```
In this example:
* If a row with the same `user_id` and `email` already exists, it will be updated with the new values from the data frame.
* If no matching row exists, a new row will be inserted.
**Example: Ignore duplicates**
```python theme={"system"}
from mage_ai.io.postgres import Postgres
from mage_ai.io.constants import UNIQUE_CONFLICT_METHOD_IGNORE
loader = Postgres(...)
with loader:
loader.export(
df,
table_name='users',
schema='public',
if_exists='append',
unique_constraints=['user_id'],
unique_conflict_method=UNIQUE_CONFLICT_METHOD_IGNORE, # Skip rows that already exist
)
loader.commit()
```
**Note**: UPSERT functionality is supported on PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, and SQLite. The exact SQL syntax used may vary by database (e.g., PostgreSQL uses `ON CONFLICT`, MySQL uses `ON DUPLICATE KEY UPDATE`, MSSQL uses `MERGE`), but the interface remains consistent across all supported databases.
UPSERT functionality is **not supported** on Redshift, ClickHouse, Trino and Databricks SQL.
### Additional Export Parameters
The following parameters provide additional control over the export process:
* `allow_reserved_words (bool)`: If `True`, allows column names that are SQL reserved words. Defaults to `False`.
* `auto_clean_name (bool)`: If `True`, automatically cleans column and table names (removes special characters, handles reserved words). Defaults to `True`.
* `case_sensitive (bool)`: If `True`, preserves case sensitivity for identifiers. Defaults to `False`.
* `cascade_on_drop (bool)`: If `True`, adds `CASCADE` when dropping tables. Defaults to `False`.
* `drop_table_on_replace (bool)`: If `True`, drops the table when using `if_exists='replace'` instead of deleting all rows. Defaults to `False`.
* `overwrite_types (Dict[str, str])`: A dictionary mapping column names to SQL data types to override automatic type inference. Example: `{'price': 'DECIMAL(10,2)', 'description': 'TEXT'}`.
* `query_string (str)`: A custom SQL query string to use for creating the table. If provided, the table is created using `CREATE TABLE AS SELECT ...` syntax.
* `skip_check_table_exists (bool)`: If `True`, skips checking if the table exists before export. Defaults to `False`.
* `skip_create_schema (bool)`: If `True`, skips creating the schema if it doesn't exist. Defaults to `False`.
* `skip_semicolon_at_end (bool)`: If `True`, omits the semicolon at the end of SQL statements. Defaults to `False`.
**Example: Using overwrite\_types**
```python theme={"system"}
loader.export(
df,
table_name='products',
schema='public',
overwrite_types={
'price': 'DECIMAL(10,2)',
'description': 'TEXT',
'created_at': 'TIMESTAMP',
}
)
```
**Example: Using query\_string**
```python theme={"system"}
loader.export(
df,
table_name='filtered_data',
schema='public',
query_string='SELECT * FROM source_table WHERE status = \'active\'',
)
```
## OracleDB
```yaml theme={"system"}
ORACLEDB_USER: OracleDB username
ORACLEDB_PASSWORD: OracleDB password
ORACLEDB_HOST: OracleDB host
ORACLEDB_PORT: OracleDB port
ORACLEDB_SERVICE: OracleDB service
ORACLEDB_MODE: OracleDB mode
```
Constructor
`__init__(self, **kwargs)`
Initializes the OracleDB data loading client.
* **Args**:
* `user (str)`: The user for connecting to the database with.
* `password (str)`: The login password for the user.
* `host (str)`: Host address for database.
* `port (int)`: Port on which the database is running Defaults to 3306.
* `service (str)`: OracleDB service
* `mode (str)`: switch between OracleDB mode of `thin` or `thick`
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* `**kwargs`: Additional settings for creating psycopg2 connection
To switch OracleDB mode to `thick`, it is required to use the customized oracle Dockerfile in `integrations/oracle/Dockerfile`.
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Postgres`
Creates OracleDB data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`OracleDB`) The constructed dataloader using this method
Methods
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected OracleDB database. Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the OracleDB database.
* `**kwargs`: Additional parameters to pass to the query.
***export*** - `export(df: DataFrame, schema_name: str, table_name: str, if_exists: str, index: bool, **kwargs) -> None`
Exports data frame to the OracleDB database from a Pandas data frame. If table doesn't exist, the table is automatically created. The `schema_name` can be set as `None` since it is not required for OracleDB loaders.
Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to the OracleDB database.
* `schema_name (str)`: Not required for OracleDB loaders. Set to `None`.
* `table_name (str)`: Name of the table to export data to (excluding database).
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
* `**kwargs`: Additional arguments to pass to writer.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to OracleDB database.
## MySQL
`mage_ai.io.mysql.MySQL`
Handles data transfer between a MySQL database and the Mage app. The `MySQL` client utilizes the following keys to connect the MySQL database.
```yaml theme={"system"}
MYSQL_DATABASE: MySQL database name
MYSQL_HOST: MySQL database hostname
MYSQL_PASSWORD: MySQL database login password
MYSQL_PORT: MySQL database port
MYSQL_USER: MySQL database login username
MYSQL_ALLOW_LOCAL_INFILE: Pass boolean value to `allow_local_infile` param during connection to enable the capability to load local data file.
```
Constructor
`__init__(self, **kwargs)`
Initializes the MySQL data loading client.
* **Args**:
* `database (str)`: The name of the database to connect to.
* `user (str)`: The user for connecting to the database with.
* `password (str)`: The login password for the user.
* `host (str)`: Host address for database.
* `port (int)`: Port on which the database is running Defaults to 3306.
* `allow_local_infile (bool)`: Enables verbose capability to load local file. Defaults to `False`.
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* `**kwargs`: Additional settings for creating psycopg2 connection
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MySQL`
Creates MySQL data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MySQL`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to MySQL database.
***commit*** - `commit()`
Saves all changes made to the database since the previous transaction.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected MySQL database. Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the MySQL database.
* `**kwargs`: Additional parameters to pass to the query.
***export*** - `export(df: DataFrame, schema_name: str, table_name: str, if_exists: str, index: bool, **kwargs) -> None`
Exports data frame to the MySQL database from a Pandas data frame. If table doesn't exist, the table is automatically created. The `schema_name` can be set as `None` since it is not required for MySQL loaders.
Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to the MySQL database.
* `schema_name (str)`: Not required for MySQL loaders. Set to `None`.
* `table_name (str)`: Name of the table to export data to (excluding database).
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
* `**kwargs`: Additional arguments to pass to writer. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for details on UPSERT (`unique_constraints`, `unique_conflict_method`) and other advanced options.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to MySQL database.
***sample*** - `sample(database: str, table: str, size: int, **kwargs) -> DataFrame`
Sample data from a table in the MySQL database. Sample is not guaranteed to be random.
* **Args**:
* `database (str)`: The database to select the table from.
* `table (str)`: The table to sample from in the connected database.
* `size (int)`: The number of rows to sample. Defaults to 10,000,000.
* **Returns**: (`DataFrame`) Sampled data from the table.
## DuckDB
`mage_ai.io.duckdb`
Handles data transfer between DuckDB database and the Mage app.
DuckDB client utilizes the following keys to connect:
```yaml theme={"system"}
DUCKDB_DATABASE: database # Database name
DUCKDB_SCHEMA: main # Schema name. Default is main.
```
Constructor
```
__init__(self,
database: str
schema: str = None,
verbose: bool = True,
**kwargs
)
```
Initializes the DuckDB data loading client.
* **Args**:
* `database (str)`: The name of the database to connect to.
* `schema`: Schema to use.
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* `**kwargs`: Additional settings for creating psycopg2 connection
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> DuckDB`
Creates DuckDB data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`DuckDB`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to DuckDB database.
***open*** - `open()`
Opens a connection to DuckDB database.
***table\_exists*** - `table_exists()`
Build query to check if a table existing in the DB.
* **Args**:
* `schema_name`: Name of the schema to use.
* `table_name`: Name of the table.
* **Returns**: true or false.
***upload\_dataframe*** - `upload_dataframe()`
Build query to insert data into table.
* **Args**:
* `df`: Data in dataframe type.
***get\_type*** - `get_type()`
Convert data from input type into DuckDB type.
* **Args**:
* `column`: Metadata associated with the column.
* `dtype`: Input data type.
* **Returns**: DuckDB data type in string format.
## Snowflake
`mage_ai.io.snowflake.Snowflake`
Handles data transfer between a Snowflake data warehouse and the Mage app. The `Snowflake` client utilizes the following keys to authenticate access and connect to Snowflake servers.
```yaml theme={"system"}
SNOWFLAKE_USER: username
SNOWFLAKE_PASSWORD: password
SNOWFLAKE_ACCOUNT: account_id.region
SNOWFLAKE_DEFAULT_WH: null # Optional default warehouse
SNOWFLAKE_DEFAULT_DB: null # Optional default database
SNOWFLAKE_DEFAULT_SCHEMA: null # Optional default schema
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null # Optional private key passphrase
SNOWFLAKE_PRIVATE_KEY_PATH: null # Optional private key path
SNOWFLAKE_ROLE: null # Optional role name
SNOWFLAKE_TIMEOUT: null # Optional timeout in seconds
```
Constructor
`__init__(self, **kwargs)`
Initializes settings for connecting to Snowflake data warehouse.
The following arguments must be provided to the connector, all other
arguments are optional.
*Required Arguments*:
* `user (str)`: Username for the Snowflake user.
* `password (str)`: Login Password for the user.
* `account (str)`: Snowflake account identifier (including region, excluding `snowflake-computing.com` suffix).
*Optional Arguments*:
* `verbose (bool)`: Specify whether to print verbose output.
* `database (str)`: Name of the default database to use. If unspecified no database is selected on login.
* `schema (str)`: Name of the default schema to use. If unspecified no schema is selected on login.
* `warehouse (str)`: Name of the default warehouse to use. If unspecified no warehouse is selected on login.
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Snowflake`
Creates Snowflake data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `verbose (bool)`: Enables verbose output printing. Defaults to False.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Snowflake`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to Snowflake server.
***commit*** - `commit()`
Saves all changes made to the warehouse since the previous transaction.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected Snowflake warehouse. Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the Snowflake warehouse.
* `**kwargs`: Additional parameters to pass to the query. See [Snowflake Connector Docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#execute) for additional parameters.
***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, **kwargs) -> None`
Exports a Pandas data frame to a Snowflake warehouse based on the table name. If table doesn't exist, the table is automatically created.
Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to a Snowflake warehouse.
* `table_name (str)`: Name of the table to export data to (excluding database and schema).
* `database (str)`: Name of the database in which the table is located.
* `schema (str)`: Name of the schema in which the table is located.
* `if_exists (str, optional)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'append'`.
* `**kwargs`: Additional arguments to pass to writer. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for details on UPSERT (`unique_constraints`, `unique_conflict_method`) and other advanced options.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from Snowflake into a Pandas data frame based on the query given. This will fail unless a `SELECT` query is provided.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse using `execute`.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `*args, **kwargs`: Additional parameters to pass to the query. See [Snowflake Connector Docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#execute) for additional parameters.
* **Returns**: (`DataFrame`) Data frame containing the queried data
***open*** - `open()`
Opens a connection to Snowflake servers.
***rollback*** - `rollback()`
Rolls back (deletes) all database changes made since the last transaction.
***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`
Sample data from a table in the Snowflake warehouse. Sample is not guaranteed to be random.
* **Args**:
* `schema (str)`: The schema to select the table from.
* `size (int)`: The number of rows to sample. Defaults to 10,000,000.
* `table (str)`: The table to sample from in the connected database.
* **Returns**: (`DataFrame`) Sampled data from the data frame.
## Redshift
`mage_ai.io.redshift.Redshift`
Handles data transfer between a Redshift cluster and the Mage app. Mage uses temporary credentials to authenticate access to a Redshift cluster. There are two ways to specify these credentials:
1. Pre-generate temporary credentials and specify them in the configuration settings. Add the following keys to the configuration settings for `Redshift` to use the temporary credentials:
```yaml theme={"system"}
REDSHIFT_DBNAME: Name of Redshift database to connect to
REDSHIFT_HOST: Redshift Cluster hostname
REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
```
2. Provide an IAM Profile to automatically generate temporary credentials for connection. The IAM profile is read from `~/.aws/` and is used with the `GetClusterCredentials` endpoint to generate temporary credentials. Add the following keys to the configuration settings for `Redshift` to generate temporary credentials
```yaml theme={"system"}
REDSHIFT_DBNAME: Name of Redshift database to connect to
REDSHIFT_DBUSER: Redshift database user to generate credentials for
REDSHIFT_CLUSTER_ID: Redshift cluster ID
REDSHIFT_IAM_PROFILE: Name of the IAM profile to generate temp credentials with
```
If an IAM profile is not setup using `aws configure`, manually specify the AWS credentials in the configuration settings as well.
```yaml theme={"system"}
AWS_ACCESS_KEY_ID: AWS Access Key ID credential
AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
AWS_SESSION_TOKEN: AWS Session Token (used to generate temp DB credentials)
AWS_REGION: AWS Region
```
Constructor
`__init__(**kwargs)`
* **Args**
* `verbose`: Enables verbose output. Defaults to `False`.
See other keyword arguments in [Redshift's Python connector](https://docs.aws.amazon.com/redshift/latest/mgmt/python-configuration-options.html) docs.
Attributes
* *conn* (`redshift_connector.Connection`) - the underlying [Redshift Connection](https://docs.aws.amazon.com/redshift/latest/mgmt/python-api-reference.html#python-api-connection) object.
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Redshift`
Initializes Redshift client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor.
* **Returns**: (`Redshift`) Redshift data loading client constructed using this method
***with\_temporary\_credentials*** - `with_temporary_credentials(database: str, host: str, user: str, password: str, port: int = 5439, **kwargs) -> Redshift`
Creates a Redshift data loader with temporary database credentials.
* **Args**:
* `database (str)`: Name of the database to connect to.
* `host (str)`: The hostname of the Redshift cluster which the database belongs to.
* `user (str)`: Temporary credentials username for use in authentication.
* `password (str)`: Temporary credentials password for use in authentication.
* `port (int, optional)`: Port number of the Redshift cluster. Defaults to 5439.
* `**kwargs`: Additional parameters passed to the loader constructor.
* **Returns**: (`Redshift`) Redshift data loading client constructed using this method
***with\_iam*** - `with_iam(cluster_identifier: str, database: str, db_user: str, profile: str, **kwargs) -> Redshift`
Creates a Redshift data loader using an IAM profile from `~/.aws`.
The IAM Profile settings can also be manually specified as keyword arguments to this constructor, but is not recommended. If credentials are manually specified, the region of the Redshift cluster must also be specified.
* **Args**:
* `cluster_identifier (str)`: Identifier of the cluster to connect to.
* `database (str)`: The database to connect to within the specified cluster.
* `db_user (str)`: Database username
* `profile (str, optional)`: The profile to use from stored credentials file.
Defaults to 'default'.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Redshift`) Redshift data loading client constructed using this method
Methods
***close*** - `close()`
Closes connection to the Redshift cluster specified in the loader configuration.
***commit*** - `commit()`
Saves all changes made to the database since the last transaction.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected Redshift cluster. Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the Redshift cluster.
* `**kwargs`: Additional parameters to pass to the query. See [`redshift-connector` docs](https://github.com/aws/amazon-redshift-python-driver#configuring-cursor-paramstyle) for configuring query parameters.
***export*** - `export(df: DataFrame, table_name: str) -> None`
Exports a Pandas data frame to a Redshift cluster under the specified table. The changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to database in Redshift cluster.
* `table_name (str)`: Name of the table to export the data to. Table must already exist.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse using `execute`.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `*args, **kwargs`: Additional parameters to send to query, including parameters
for use with format strings. See [`redshift-connector` docs](https://github.com/aws/amazon-redshift-python-driver#configuring-cursor-paramstyle) for configuring query parameters.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to the Redshift cluster specified in the loader configuration.
***rollback*** - `rollback()`
Rolls back (deletes) all database changes made since the last transaction.
***sample*** - `sample(schema: str, table: str, size: int, **kwargs) -> DataFrame`
Sample data from a table in the selected database in the Redshift cluster. Sample is not guaranteed to be random.
* **Args**:
* `schema (str)`: The schema to select the table from.
* `size (int)`: The number of rows to sample. Defaults to 10,000,000.
* `table (str)`: The table to sample from in the connected database.
* **Returns**: (`DataFrame`) Sampled data from the table.
## MSSQL
`mage_ai.io.mssql.MSSQL`
Handles data transfer between a Microsoft SQL Server database and the Mage app. The `MSSQL` client utilizes the following keys to connect to the database.
```yaml theme={"system"}
MSSQL_DATABASE: MSSQL database name
MSSQL_USER: MSSQL database login username
MSSQL_PASSWORD: MSSQL database login password
MSSQL_HOST: MSSQL database hostname
MSSQL_PORT: MSSQL database port (defaults to 1433)
MSSQL_SCHEMA: MSSQL database schema (defaults to 'dbo')
MSSQL_AUTHENTICATION: Authentication method (optional)
MSSQL_CONNECTION_METHOD: Connection method ('direct' or 'ssh_tunnel')
MSSQL_SSH_HOST: SSH tunnel host (if using SSH tunnel)
MSSQL_SSH_PORT: SSH tunnel port
MSSQL_SSH_USERNAME: SSH tunnel username
MSSQL_SSH_PASSWORD: SSH tunnel password
MSSQL_SSH_PKEY: SSH tunnel private key path
```
Constructor
`__init__(self, **kwargs)`
Initializes the MSSQL data loading client.
* **Args**:
* `database (str)`: The name of the database to connect to.
* `user (str)`: The user for connecting to the database with.
* `password (str)`: The login password for the user.
* `host (str)`: Host address for database.
* `port (int)`: Port on which the database is running. Defaults to 1433.
* `schema (str)`: Schema name. Defaults to 'dbo'.
* `authentication (str, optional)`: Authentication method.
* `connection_method (str)`: Connection method ('direct' or 'ssh\_tunnel'). Defaults to 'direct'.
* `ssh_host (str, optional)`: SSH tunnel host.
* `ssh_port (int, optional)`: SSH tunnel port.
* `ssh_username (str, optional)`: SSH tunnel username.
* `ssh_password (str, optional)`: SSH tunnel password.
* `ssh_pkey (str, optional)`: SSH tunnel private key path.
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* `**kwargs`: Additional settings for creating connection
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MSSQL`
Creates MSSQL data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MSSQL`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to MSSQL database.
***commit*** - `commit()`
Saves all changes made to the database since the previous transaction.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected MSSQL database. Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the MSSQL database.
* `**kwargs`: Additional parameters to pass to the query.
***export*** - `export(df: DataFrame, schema_name: str, table_name: str, if_exists: str, index: bool, **kwargs) -> None`
Exports data frame to the MSSQL database from a Pandas data frame. If table doesn't exist, the table is automatically created.
Any changes made to the database will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to the MSSQL database.
* `schema_name (str)`: Name of the schema in which the table is located. Defaults to 'dbo'.
* `table_name (str)`: Name of the table to export data to.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
* `**kwargs`: Additional arguments to pass to the writer. See the [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) section for details on using options such as `unique_constraints`, `unique_conflict_method`, and other advanced export options.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to MSSQL database.
***rollback*** - `rollback()`
Rolls back (deletes) all database changes made since the last transaction.
## ClickHouse
`mage_ai.io.clickhouse.ClickHouse`
Handles data transfer between a ClickHouse data warehouse and the Mage app. The `ClickHouse` client utilizes the following keys to authenticate access and connect to ClickHouse servers.
```yaml theme={"system"}
CLICKHOUSE_HOST: ClickHouse hostname
CLICKHOUSE_PORT: ClickHouse port
CLICKHOUSE_USERNAME: ClickHouse username
CLICKHOUSE_PASSWORD: ClickHouse password
CLICKHOUSE_DATABASE: ClickHouse database name
CLICKHOUSE_INTERFACE: ClickHouse interface (optional)
CLICKHOUSE_SSL_CA_CERT: Path to CA certificate for SSL (optional)
```
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> ClickHouse`
Creates ClickHouse data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`ClickHouse`) The constructed dataloader using this method
Methods
***execute*** - `execute(command_string: str, **kwargs) -> None`
Sends command to the connected ClickHouse warehouse.
* **Args**:
* `command_string (str)`: Command to execute on the ClickHouse warehouse.
* `**kwargs`: Additional arguments to pass to command.
***load*** - `load(query_string: str, limit: int, **kwargs) -> DataFrame`
Loads data from ClickHouse into a Pandas data frame based on the query given. This will fail if the query returns no data from the database. When a select query is provided, this function will load at maximum 10,000,000 rows of data. To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded dataframe to. Defaults to 10,000,000.
* `**kwargs`: Additional arguments to pass to query.
* **Returns**: (`DataFrame`) Data frame associated with the given query.
***export*** - `export(df: DataFrame, table_name: str, database: str, if_exists: str, **kwargs) -> None`
Exports a Pandas data frame to a ClickHouse warehouse. If table doesn't exist, the table is automatically created.
* **Args**:
* `df (DataFrame)`: Data frame to export to ClickHouse warehouse.
* `table_name (str)`: Name of the table to export data to.
* `database (str)`: Name of the database in which the table is located.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `**kwargs`: Advanced export parameters. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases). (UPSERT not supported.)
## SQLite
`mage_ai.io.sqlite.SQLite`
Handles data transfer between a SQLite database and the Mage app.
Constructor
`__init__(self, database: str, verbose: bool = True, **kwargs)`
Initializes the SQLite data loading client.
* **Args**:
* `database (str)`: Path to the SQLite database file.
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* `**kwargs`: Additional settings for creating connection
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> SQLite`
Creates SQLite data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`SQLite`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to SQLite database.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected SQLite database.
* **Args**:
* `query_string (str)`: The query to execute on the SQLite database.
* `**kwargs`: Additional parameters to pass to the query.
***export*** - `export(df: DataFrame, table_name: str, if_exists: str, index: bool, **kwargs) -> None`
Exports data frame to the SQLite database from a Pandas data frame. If table doesn't exist, the table is automatically created.
* **Args**:
* `df (DataFrame)`: Data frame to export to the SQLite database.
* `table_name (str)`: Name of the table to export data to.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `index (bool)`: If `True`, the data frame index is also exported alongside the table. Defaults to `False`.
* `**kwargs`: Advanced export options such as `unique_constraints`, `unique_conflict_method`, `overwrite_types`, `auto_clean_name`, etc. See the [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) section for details.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from the results of a query into a Pandas data frame. This will fail if the query returns no data from the database.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to SQLite database.
## Trino
`mage_ai.io.trino.Trino`
Handles data transfer between a Trino data warehouse and the Mage app. The `Trino` client utilizes the following keys to authenticate access and connect to Trino servers.
```yaml theme={"system"}
TRINO_CATALOG: Trino catalog name
TRINO_HOST: Trino hostname
TRINO_PORT: Trino port (defaults to 8080)
TRINO_USER: Trino username
TRINO_PASSWORD: Trino password
TRINO_SCHEMA: Trino schema name
```
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Trino`
Creates Trino data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Trino`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to Trino warehouse.
***commit*** - `commit()`
Saves all changes made to the warehouse since the previous transaction.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected Trino warehouse. Any changes made to the warehouse will not be saved unless `commit()` is called afterward.
* **Args**:
* `query_string (str)`: The query to execute on the Trino warehouse.
* `**kwargs`: Additional parameters to pass to the query.
***export*** - `export(df: DataFrame, table_name: str, schema: str, if_exists: str, **kwargs) -> None`
Exports a Pandas data frame to a Trino warehouse. If table doesn't exist, the table is automatically created.
Any changes made to the warehouse will not be saved unless `commit()` is called afterward.
* **Args**:
* `df (DataFrame)`: Data frame to export to Trino warehouse.
* `table_name (str)`: Name of the table to export data to.
* `schema (str)`: Name of the schema in which the table is located.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `**kwargs`: Advanced export parameters. See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases). (UPSERT not supported.)
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from Trino into a Pandas data frame based on the query given. This will fail if the query returns no data from the warehouse.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to Trino warehouse.
***rollback*** - `rollback()`
Rolls back (deletes) all warehouse changes made since the last transaction.
## MongoDB
`mage_ai.io.mongodb.MongoDB`
Handles data transfer between a MongoDB database and the Mage app. The `MongoDB` client utilizes the following keys to connect to the database.
```yaml theme={"system"}
MONGODB_CONNECTION_STRING: MongoDB connection string (alternative to individual settings)
MONGODB_HOST: MongoDB hostname
MONGODB_PORT: MongoDB port (defaults to 27017)
MONGODB_USER: MongoDB username
MONGODB_PASSWORD: MongoDB password
MONGODB_DATABASE: MongoDB database name
MONGODB_COLLECTION: MongoDB collection name
```
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MongoDB`
Creates MongoDB data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MongoDB`) The constructed dataloader using this method
Methods
***load*** - `load(collection: str = None, query: Dict = None, **kwargs) -> DataFrame`
Loads the data frame from the MongoDB collection.
* **Args**:
* `collection (str, optional)`: MongoDB collection name. Defaults to collection specified in constructor.
* `query (Dict, optional)`: Filter the result by using a query object. Examples:
`{ "address": "Park Lane 38" }`, `{ "address": { "$gt": "S" } }`
* **Returns**: (`DataFrame`) Data frame object loaded from the MongoDB collection.
***export*** - `export(data: Union[DataFrame, List[Dict]], collection: str = None, **kwargs) -> None`
Exports the input dataframe to the MongoDB collection.
* **Args**:
* `data (Union[DataFrame, List[Dict]])`: Data frame or List of Dictionary to export.
* `collection (str, optional)`: MongoDB collection name. Defaults to collection specified in constructor.
## Databricks SQL
`mage_ai.io.databricks_sql.DatabricksSQL`
Handles data transfer between a Databricks SQL warehouse and the Mage app. The `DatabricksSQL` client utilizes the following keys to authenticate access and connect to Databricks.
```yaml theme={"system"}
DATABRICKS_ACCESS_TOKEN: Databricks access token
DATABRICKS_HOST: Databricks workspace hostname
DATABRICKS_HTTP_PATH: Databricks SQL warehouse HTTP path
DATABRICKS_DATABASE: Databricks database/catalog name
DATABRICKS_SCHEMA: Databricks schema name
```
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> DatabricksSQL`
Creates Databricks SQL data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`DatabricksSQL`) The constructed dataloader using this method
Methods
***close*** - `close()`
Closes connection to Databricks SQL warehouse.
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected Databricks SQL warehouse.
* **Args**:
* `query_string (str)`: The query to execute on the Databricks SQL warehouse.
* `**kwargs`: Additional parameters to pass to the query.
***export*** - `export(df: DataFrame, table_name: str, database: str, schema: str, if_exists: str, **kwargs) -> None`
Exports a Pandas data frame to a Databricks SQL warehouse. If table doesn't exist, the table is automatically created.
* **Args**:
* `df (DataFrame)`: Data frame to export to Databricks SQL warehouse.
* `table_name (str)`: Name of the table to export data to.
* `database (str)`: Name of the database/catalog in which the table is located.
* `schema (str)`: Name of the schema in which the table is located.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `**kwargs`: See [Advanced Export Parameters](#advanced-export-parameters-for-sql-databases) for available options.
***load*** - `load(query_string: str, limit: int, *args, **kwargs) -> DataFrame`
Loads data from Databricks SQL into a Pandas data frame based on the query given. This will fail if the query returns no data from the warehouse.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in warehouse.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional parameters to pass to the query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***open*** - `open()`
Opens a connection to Databricks SQL warehouse.
## Spark
`mage_ai.io.spark.Spark`
Handles data transfer between a Spark session and the Mage app. The `Spark` client utilizes the following keys to connect to Spark.
```yaml theme={"system"}
SPARK_HOST: Spark host/master URL
SPARK_METHOD: Spark connection method
SPARK_SCHEMA: Spark database/schema name
```
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> Spark`
Creates Spark data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`Spark`) The constructed dataloader using this method
Methods
***execute*** - `execute(query_string: str, **kwargs) -> None`
Sends query to the connected Spark session.
* **Args**:
* `query_string (str)`: Query to execute on the Spark session.
* `**kwargs`: Additional arguments to pass to query.
***load*** - `load(query_string: str, limit: int, **kwargs) -> DataFrame`
Loads data from Spark into a Pandas data frame based on the query given. This will fail if the query returns no data.
This function will load at maximum 10,000,000 rows of data (this limit is configurable). To operate on more data, consider performing data transformations in Spark.
* **Args**:
* `query_string (str)`: Query to fetch a table or subset of a table.
* `limit (int, Optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs`: Additional arguments to pass to query.
* **Returns**: (`DataFrame`) Data frame containing the queried data.
***export*** - `export(df: DataFrame, table_name: str, database: str, if_exists: str, **kwargs) -> None`
Exports a Pandas data frame to a Spark session. If table doesn't exist, the table is automatically created.
* **Args**:
* `df (DataFrame)`: Data frame to export to Spark session.
* `table_name (str)`: Name of the table to export data to.
* `database (str)`: Name of the database in which the table is located.
* `if_exists (ExportWritePolicy)`: Specifies export policy if table exists. Either
* `'fail'`: throw an error.
* `'replace'`: drops existing table and creates new table of same name.
* `'append'`: appends data frame to existing table.
Defaults to `'replace'`.
* `**kwargs`: Additional arguments to pass to writer.
## Azure Data Lake Storage
`mage_ai.io.azure_data_lake_storage.AzureDataLakeStorage`
Handles data transfer between an Azure Data Lake Storage Gen2 container and the Mage app. Supports loading files of the following types:
* CSV
* JSON
* Parquet
To authenticate access to Azure Data Lake Storage, the following environment variables must be set:
* `AZURE_CLIENT_ID`
* `AZURE_CLIENT_SECRET`
* `AZURE_TENANT_ID`
* `AZURE_STORAGE_ACCOUNT_NAME`
By default, these correspond to variables of the same name in `io_config.yaml`.
Constructor
`__init__(self, verbose: bool = True, **kwargs)`
Initializes data loader from an Azure Data Lake Storage Gen2 container.
* **Args**:
* `storage_account_name (str)`: Azure Storage account name (required).
* `verbose (bool)`: Enables verbose output printing. Defaults to `True`.
* `**kwargs`: Additional settings. If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are provided, they will be used for authentication. Otherwise, DefaultAzureCredential will be used.
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> AzureDataLakeStorage`
Creates AzureDataLakeStorage data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`AzureDataLakeStorage`) The constructed dataloader using this method
Methods
***export*** - `export(df: DataFrame, container_name: str, file_path: str, format: FileFormat | str = None, **kwargs) -> None`
Exports data frame to an Azure Data Lake Storage Gen2 container.
* **Args**:
* `df (DataFrame)`: Data frame to export.
* `container_name (str)`: Name of the Azure container to export data to.
* `file_path (str)`: The desired output path of the data in your Azure Data Lake Storage.
* `format (FileFormat | str, Optional)`: Format of the file to export data frame to. Defaults to `None`, in which case the format is inferred.
* `**kwargs` - Additional keyword arguments to pass to the file writer. See possible arguments by file formats in [FileIO](#fileio) section.
***load*** - `load(container_name: str, file_path: str, format: FileFormat | str = None, limit: int = QUERY_ROW_LIMIT, **kwargs) -> DataFrame`
Loads data from object in Azure Data Lake Storage Gen2 into a Pandas data frame. This function will load at maximum 10,000,000 rows of data from the specified file (this limit is configurable).
* **Args**:
* `container_name (str)`: Name of the container to load data from.
* `file_path (str)`: Path of the file in Azure Data Lake Storage to load data from.
* `format (FileFormat | str, Optional)`: Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
* `limit (int, optional)`: The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
* `**kwargs` - Additional keyword arguments to pass to the file reader. See possible arguments by file formats in [FileIO](#fileio) section.
* **Returns**: (`DataFrame`) Data frame object loaded from data in the specified file.
## Microsoft Fabric Warehouse
`mage_ai.io.microsoft_fabric_warehouse.MicrosoftFabricWarehouse`
Handles data transfer between a Microsoft Fabric Warehouse and the Mage app. This client extends the MSSQL client and uses the same connection methods.
Constructor
`__init__(self, **kwargs)`
Initializes the Microsoft Fabric Warehouse data loading client. Uses the same parameters as [MSSQL](#mssql).
Factory Methods
***with\_config*** - `with_config(config: BaseConfigLoader, **kwargs) -> MicrosoftFabricWarehouse`
Creates Microsoft Fabric Warehouse data loading client from configuration settings.
* **Args**:
* `config (BaseConfigLoader)`: Configuration loader object.
* `**kwargs`: Additional parameters passed to the loader constructor
* **Returns**: (`MicrosoftFabricWarehouse`) The constructed dataloader using this method
# Configuration Settings
Connections to third-party data storage require you to specify confidential information such as login information or access keys. While you can manually specify this information code while constructing data loading clients, it is recommended to not store the secrets directly in code.
Instead, Mage provides **configuration loaders** which allow data loading clients to use your secrets without explicitly writing them in code.
Currently Mage only supports AWS Secrets Manager as a configuration loader. If you need to use secrets from Azure Key Vault
or GCP Secret Manager, you can reference the following docs:
* [Azure Key Vault](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure)
* [GCP Secret Manager](https://docs.mage.ai/production/deploying-to-cloud/secrets/GCP)
Currently, the following sources (and their corresponding configuration loader) can be used to load configuration settings:
* Configuration File - `ConfigFileLoader`
* Environment Variables - `EnvironmentVariableLoader`
* AWS Secrets Manager - `AWSSecretLoader`
For example, the code below constructs a Redshift data loading client using secrets stored in AWS Secrets Manager
```python theme={"system"}
from mage_ai.io.config import AWSSecretLoader
from mage_ai.io.redshift import Redshift
config = AWSSecretLoader()
loader = Redshift.from_config(config)
```
The following are the set of allowed key names that you must name your secrets with in order for Mage's configuration loaders to recognize your secrets. In code you can refer to these keys by their string name or using the `mage_ai.io.config.ConfigKey` enum. Not all keys need be specified at once - only use the keys related to the services you utilize.
| Key Name | Service | Client Constructor Parameter | Description | Notes |
| -------------------------------------- | ----------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| **AWS General** | | | | |
| AWS\_ACCESS\_KEY\_ID | AWS General | - | AWS Access Key ID credential | Used by [Redshift](#redshift) and [S3](#s3) |
| AWS\_SECRET\_ACCESS\_KEY | AWS General | - | AWS Secret Access Key credential | Used by [Redshift](#redshift) and [S3](#s3) |
| AWS\_SESSION\_TOKEN | AWS General | - | AWS Session Token (used to generate temporary DB credentials) | Used by Redshift |
| AWS\_REGION | AWS General | - | AWS Region | Used by [Redshift](#redshift) and [S3](#s3) |
| AWS\_ENDPOINT | AWS General | - | AWS endpoint URL | Used by [S3](#s3). Optional |
| **AWS Redshift** | | | | |
| REDSHIFT\_DBNAME | [AWS Redshift](#redshift) | database | Name of Redshift database to connect to | |
| REDSHIFT\_HOST | [AWS Redshift](#redshift) | host | Redshift Cluster hostname | Use with temporary credentials |
| REDSHIFT\_PORT | [AWS Redshift](#redshift) | port | Redshift Cluster port. Optional, defaults to 5439. | Use with temporary credentials |
| REDSHIFT\_SCHEMA | [AWS Redshift](#redshift) | schema | Redshift schema name | Optional |
| REDSHIFT\_TEMP\_CRED\_USER | [AWS Redshift](#redshift) | user | Redshift temporary credentials username. | Use with temporary credentials |
| REDSHIFT\_TEMP\_CRED\_PASSWORD | [AWS Redshift](#redshift) | password | Redshift temporary credentials password. | Use with temporary credentials |
| REDSHIFT\_DBUSER | [AWS Redshift](#redshift) | db\_user | Redshift database user to generate credentials for. | Use to generate temporary credentials |
| REDSHIFT\_CLUSTER\_ID | [AWS Redshift](#redshift) | cluster\_identifier | Redshift cluster ID | Use to generate temporary credentials |
| REDSHIFT\_IAM\_PROFILE | [AWS Redshift](#redshift) | profile | Name of the IAM profile to generate temporary credentials with | Use to generate temporary credentials |
| **PostgreSQL** | | | | |
| POSTGRES\_DBNAME | [PostgreSQL](#postgresql) | dbname | Database name | |
| POSTGRES\_USER | [PostgreSQL](#postgresql) | user | Database login username | |
| POSTGRES\_PASSWORD | [PostgreSQL](#postgresql) | password | Database login password | |
| POSTGRES\_HOST | [PostgreSQL](#postgresql) | host | Database hostname | |
| POSTGRES\_PORT | [PostgreSQL](#postgresql) | port | PostgreSQL database port | |
| POSTGRES\_SCHEMA | [PostgreSQL](#postgresql) | schema | PostgreSQL schema name | Optional |
| POSTGRES\_CONNECTION\_METHOD | [PostgreSQL](#postgresql) | connection\_method | Connection method ('direct' or 'ssh\_tunnel') | Optional |
| POSTGRES\_SSH\_HOST | [PostgreSQL](#postgresql) | ssh\_host | SSH tunnel host | Optional |
| POSTGRES\_SSH\_PORT | [PostgreSQL](#postgresql) | ssh\_port | SSH tunnel port | Optional |
| POSTGRES\_SSH\_USERNAME | [PostgreSQL](#postgresql) | ssh\_username | SSH tunnel username | Optional |
| POSTGRES\_SSH\_PASSWORD | [PostgreSQL](#postgresql) | ssh\_password | SSH tunnel password | Optional |
| POSTGRES\_SSH\_PKEY | [PostgreSQL](#postgresql) | ssh\_pkey | SSH tunnel private key path | Optional |
| POSTGRES\_SSL\_MODE | [PostgreSQL](#postgresql) | sslmode | SSL mode | Optional |
| POSTGRES\_SSL\_ROOTCERT | [PostgreSQL](#postgresql) | sslrootcert | SSL root certificate path | Optional |
| POSTGRES\_SSL\_CERT | [PostgreSQL](#postgresql) | sslcert | SSL certificate path | Optional |
| POSTGRES\_SSL\_KEY | [PostgreSQL](#postgresql) | sslkey | SSL key path | Optional |
| POSTGRES\_CONNECT\_TIMEOUT | [PostgreSQL](#postgresql) | connect\_timeout | Connection timeout in seconds | Optional |
| **Snowflake** | | | | |
| SNOWFLAKE\_USER | [Snowflake](#snowflake) | user | Snowflake username | |
| SNOWFLAKE\_PASSWORD | [Snowflake](#snowflake) | password | Snowflake password | |
| SNOWFLAKE\_ACCOUNT | [Snowflake](#snowflake) | account | Snowflake account ID (including region) | |
| SNOWFLAKE\_DEFAULT\_DB | [Snowflake](#snowflake) | database | Default database to use. Optional, no database chosen if unspecified. | |
| SNOWFLAKE\_DEFAULT\_SCHEMA | [Snowflake](#snowflake) | schema | Default schema to use. Optional, no schema chosen if unspecified. | |
| SNOWFLAKE\_DEFAULT\_WH | [Snowflake](#snowflake) | warehouse | Default warehouse to use. Optional, no warehouse chosen if unspecified. | |
| SNOWFLAKE\_PRIVATE\_KEY\_PASSPHRASE | [Snowflake](#snowflake) | private\_key\_passphrase | Private key passphrase for key pair authentication | Optional |
| SNOWFLAKE\_PRIVATE\_KEY\_PATH | [Snowflake](#snowflake) | private\_key\_path | Path to private key file for key pair authentication | Optional |
| SNOWFLAKE\_ROLE | [Snowflake](#snowflake) | role | Snowflake role name | Optional |
| SNOWFLAKE\_TIMEOUT | [Snowflake](#snowflake) | timeout | Query timeout in seconds | Optional |
| **Google BigQuery** | | | | |
| GOOGLE\_SERVICE\_ACC\_KEY | [Google BigQuery](#bigquery) | credentials\_mapping | Service account key | |
| GOOGLE\_SERVICE\_ACC\_KEY\_FILEPATH | [Google BigQuery](#bigquery) | path\_to\_credentials | Path to service account key | |
| GOOGLE\_LOCATION | [Google BigQuery](#bigquery) | location | Google Cloud location | Optional |
| **MySQL** | | | | |
| MYSQL\_DATABASE | [MySQL](#mysql) | database | MySQL database name | |
| MYSQL\_USER | [MySQL](#mysql) | user | MySQL username | |
| MYSQL\_PASSWORD | [MySQL](#mysql) | password | MySQL password | |
| MYSQL\_HOST | [MySQL](#mysql) | host | MySQL hostname | |
| MYSQL\_PORT | [MySQL](#mysql) | port | MySQL port. Defaults to 3306. | |
| MYSQL\_CONNECTION\_METHOD | [MySQL](#mysql) | connection\_method | Connection method ('direct' or 'ssh\_tunnel') | Optional |
| MYSQL\_SSH\_HOST | [MySQL](#mysql) | ssh\_host | SSH tunnel host | Optional |
| MYSQL\_SSH\_PORT | [MySQL](#mysql) | ssh\_port | SSH tunnel port | Optional |
| MYSQL\_SSH\_USERNAME | [MySQL](#mysql) | ssh\_username | SSH tunnel username | Optional |
| MYSQL\_SSH\_PASSWORD | [MySQL](#mysql) | ssh\_password | SSH tunnel password | Optional |
| MYSQL\_SSH\_PKEY | [MySQL](#mysql) | ssh\_pkey | SSH tunnel private key path | Optional |
| **MSSQL** | | | | |
| MSSQL\_DATABASE | [MSSQL](#mssql) | database | MSSQL database name | |
| MSSQL\_USER | [MSSQL](#mssql) | user | MSSQL username | |
| MSSQL\_PASSWORD | [MSSQL](#mssql) | password | MSSQL password | |
| MSSQL\_HOST | [MSSQL](#mssql) | host | MSSQL hostname | |
| MSSQL\_PORT | [MSSQL](#mssql) | port | MSSQL port. Defaults to 1433. | |
| MSSQL\_SCHEMA | [MSSQL](#mssql) | schema | MSSQL schema name. Defaults to 'dbo'. | |
| MSSQL\_AUTHENTICATION | [MSSQL](#mssql) | authentication | Authentication method | Optional |
| MSSQL\_CONNECTION\_METHOD | [MSSQL](#mssql) | connection\_method | Connection method ('direct' or 'ssh\_tunnel') | Optional |
| MSSQL\_SSH\_HOST | [MSSQL](#mssql) | ssh\_host | SSH tunnel host | Optional |
| MSSQL\_SSH\_PORT | [MSSQL](#mssql) | ssh\_port | SSH tunnel port | Optional |
| MSSQL\_SSH\_USERNAME | [MSSQL](#mssql) | ssh\_username | SSH tunnel username | Optional |
| MSSQL\_SSH\_PASSWORD | [MSSQL](#mssql) | ssh\_password | SSH tunnel password | Optional |
| MSSQL\_SSH\_PKEY | [MSSQL](#mssql) | ssh\_pkey | SSH tunnel private key path | Optional |
| MSSQL\_DRIVER | [MSSQL](#mssql) | driver | ODBC driver name | Optional |
| **ClickHouse** | | | | |
| CLICKHOUSE\_HOST | [ClickHouse](#clickhouse) | host | ClickHouse hostname | |
| CLICKHOUSE\_PORT | [ClickHouse](#clickhouse) | port | ClickHouse port | |
| CLICKHOUSE\_USERNAME | [ClickHouse](#clickhouse) | username | ClickHouse username | |
| CLICKHOUSE\_PASSWORD | [ClickHouse](#clickhouse) | password | ClickHouse password | |
| CLICKHOUSE\_DATABASE | [ClickHouse](#clickhouse) | database | ClickHouse database name | |
| CLICKHOUSE\_INTERFACE | [ClickHouse](#clickhouse) | interface | ClickHouse interface | Optional |
| CLICKHOUSE\_SSL\_CA\_CERT | [ClickHouse](#clickhouse) | ca\_cert | Path to CA certificate for SSL | Optional |
| **Trino** | | | | |
| TRINO\_CATALOG | [Trino](#trino) | catalog | Trino catalog name | |
| TRINO\_HOST | [Trino](#trino) | host | Trino hostname | |
| TRINO\_PORT | [Trino](#trino) | port | Trino port. Defaults to 8080. | |
| TRINO\_USER | [Trino](#trino) | user | Trino username | |
| TRINO\_PASSWORD | [Trino](#trino) | password | Trino password | Optional |
| TRINO\_SCHEMA | [Trino](#trino) | schema | Trino schema name | Optional |
| **Databricks SQL** | | | | |
| DATABRICKS\_ACCESS\_TOKEN | [Databricks SQL](#databricks-sql) | access\_token | Databricks access token | |
| DATABRICKS\_HOST | [Databricks SQL](#databricks-sql) | host | Databricks workspace hostname | |
| DATABRICKS\_HTTP\_PATH | [Databricks SQL](#databricks-sql) | http\_path | Databricks SQL warehouse HTTP path | |
| DATABRICKS\_DATABASE | [Databricks SQL](#databricks-sql) | database | Databricks database/catalog name | Optional |
| DATABRICKS\_SCHEMA | [Databricks SQL](#databricks-sql) | schema | Databricks schema name | Optional |
| **Spark** | | | | |
| SPARK\_HOST | [Spark](#spark) | host | Spark master URL | |
| SPARK\_METHOD | [Spark](#spark) | method | Spark connection method | Optional |
| SPARK\_SCHEMA | [Spark](#spark) | database | Spark database/schema name | Optional |
| **MongoDB** | | | | |
| MONGODB\_CONNECTION\_STRING | [MongoDB](#mongodb) | connection\_string | MongoDB connection string | Alternative to individual settings |
| MONGODB\_HOST | [MongoDB](#mongodb) | host | MongoDB hostname | |
| MONGODB\_PORT | [MongoDB](#mongodb) | port | MongoDB port. Defaults to 27017. | |
| MONGODB\_USER | [MongoDB](#mongodb) | user | MongoDB username | |
| MONGODB\_PASSWORD | [MongoDB](#mongodb) | password | MongoDB password | |
| MONGODB\_DATABASE | [MongoDB](#mongodb) | database | MongoDB database name | |
| MONGODB\_COLLECTION | [MongoDB](#mongodb) | collection | MongoDB collection name | Optional |
| **OracleDB** | | | | |
| ORACLEDB\_USER | [OracleDB](#oracledb) | user | OracleDB username | |
| ORACLEDB\_PASSWORD | [OracleDB](#oracledb) | password | OracleDB password | |
| ORACLEDB\_HOST | [OracleDB](#oracledb) | host | OracleDB hostname | |
| ORACLEDB\_PORT | [OracleDB](#oracledb) | port | OracleDB port | |
| ORACLEDB\_SERVICE | [OracleDB](#oracledb) | service | OracleDB service name | |
| ORACLEDB\_MODE | [OracleDB](#oracledb) | mode | OracleDB mode ('thin' or 'thick') | Optional |
| **DuckDB** | | | | |
| DUCKDB\_DATABASE | [DuckDB](#duckdb) | database | DuckDB database path | |
| DUCKDB\_SCHEMA | [DuckDB](#duckdb) | schema | DuckDB schema name. Defaults to 'main'. | Optional |
| MOTHERDUCK\_TOKEN | [DuckDB](#duckdb) | - | MotherDuck authentication token | Optional |
| **SQLite** | | | | |
| SQLITE | [SQLite](#sqlite) | database | SQLite database file path | |
| **Azure** | | | | |
| AZURE\_CLIENT\_ID | Azure Services | client\_id | Azure AD application client ID | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| AZURE\_CLIENT\_SECRET | Azure Services | client\_secret | Azure AD application client secret | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| AZURE\_TENANT\_ID | Azure Services | tenant\_id | Azure AD tenant ID | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| AZURE\_STORAGE\_ACCOUNT\_NAME | Azure Services | storage\_account\_name | Azure Storage account name | Used by [Azure Blob Storage](#azureblobstorage), [Azure Data Lake Storage](#azure-data-lake-storage) |
| **Microsoft Fabric** | | | | |
| MICROSOFT\_FABRIC\_WAREHOUSE\_NAME | [Microsoft Fabric](#microsoft-fabric-warehouse) | warehouse\_name | Microsoft Fabric warehouse name | |
| MICROSOFT\_FABRIC\_WAREHOUSE\_ENDPOINT | [Microsoft Fabric](#microsoft-fabric-warehouse) | endpoint | Microsoft Fabric warehouse endpoint | |
| MICROSOFT\_FABRIC\_WAREHOUSE\_SCHEMA | [Microsoft Fabric](#microsoft-fabric-warehouse) | schema | Microsoft Fabric warehouse schema | Optional |
## Configuration Loader APIs
This section contains the exact APIs and more detailed information on the configuration loaders. Every configuration loader has two functions:
* `contains` - checks if the configuration source contains the requested key. Commonly, the `in` operation is used to check for setting existence (but is not always identical as `contains` can accept multiple parameters while the `in` keyword only accepts the key).
```python theme={"system"}
if config.contains(ConfigKey.POSTGRES_PORT):
...
# alternatively
if ConfigKey.POSTGRES_PORT in config:
...
```
* `get` - gets the configuration setting associated with the given key. If the key doesn't exist, returns None. Commonly, the data model overload `__getitem__` is used to fetch a configuration setting (but is not always identical as `get` can accept multiple parameters while `__getitem__` does not).
```python theme={"system"}
user = config.get(ConfigKey.REDSHIFT_DBUSER)
# alternatively
user = config[ConfigKey.REDSHIFT_DBUSER]
```
These functions are shared among all configuration loaders, but depending on the source some function signatures may differ.
### Configuration File
Loads configuration settings from a configuration file.
For detailed information about creating and managing `io_config.yaml` files, see the [IO Config Setup documentation](/development/io_config_setup).
For detailed information about creating and managing `io_config.yaml` files, see the [IO Config Setup documentation](/development/io_config_setup).
**Example**:
```python theme={"system"}
from mage_ai.io.config import ConfigKey, ConfigFileLoader
config = ConfigFileLoader('path/to/my/io_config.yaml', 'my_profile')
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
```
**Constructor**: `__init__(filepath: os.PathLike, profile: str)`
Initializes IO Configuration loader. Input configuration file can have two formats:
* *Standard*: contains a subset of the configuration keys specified in `ConfigKey`. This
is the default and recommended format. Below is an example configuration file using this format.
```yaml theme={"system"}
version: 0.1.0
default:
AWS_ACCESS_KEY_ID: AWS Access Key ID credential
AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
AWS_REGION: AWS Region
REDSHIFT_DBNAME: Name of Redshift database to connect to
REDSHIFT_HOST: Redshift Cluster hostname
REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
```
The above configuration file has a single **profile** named `'default'`. Each profile organizes a set of keys to use (for example, distinguishing production keys versus development keys). A configuration file can have multiple profiles.
* *Verbose*: Instead of configuration keys, each profile stores an object of settings associated with
each data migration client. This format was used in previous versions of this tool, and exists
for backwards compatibility. Below is an example configuration file using this format.
```yaml theme={"system"}
version: 0.0.0
default:
AWS:
Redshift:
database: Name of Redshift database to connect to
host: Redshift Cluster hostname
port: Redshift Cluster port. Optional, defaults to 5439
user: Redshift temp credentials username
password: Redshift temp credentials password
access_key_id: AWS Access Key ID credential
secret_access_key: AWS Secret Access Key credential
region: AWS Region
```
Use handlebars and `env_var` syntax to reference environment variables in either configuration file format.
```yaml theme={"system"}
version: 0.1.0
default:
GOOGLE_SERVICE_ACC_KEY_FILEPATH: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"
```
**Args**:
* `filepath (os.PathLike, optional)`: Path to IO configuration file. Defaults to `'[repo_path]/io_config.yaml'`
* `profile (str, optional)`: Profile to load configuration settings from. Defaults to `'default'`.
**Methods**
***contains*** - `contains(self, key: ConfigKey | str) -> Any`
Checks if the configuration setting stored under `key` is contained.
* **Args**:
* `key (str)`: Name of the configuration setting to check.
* **Returns** (`bool`) Returns `True` if configuration setting exists, otherwise returns `False`.
***get*** - `get(self, key: ConfigKey | str) -> Any`
Loads the configuration setting stored under `key`.
* **Args**:
* `key (str)`: Key name of the configuration setting to load
* **Returns**: (`Any`) Configuration setting corresponding to the given key
### Environment Variables
Loads configuration settings from environment variables in your current environment.
**Example**:
```python theme={"system"}
from mage_ai.io.config import ConfigKey, EnvironmentVariableLoader
config = EnvironmentVariableLoader()
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
```
**Constructor** : `__init__(self)` - no parameters for construction.
**Methods**:
***contains*** - `contains(env_var: ConfigKey | str) -> bool`
Checks if the environment variable given by `env_var` exists.
* **Args**:
* `key (ConfigKey | str)`: Name of the configuration setting to check existence of.
* **Returns** (`bool`) Returns `True` if configuration setting exists, otherwise returns `False`.
***get*** - `get(env_var: ConfigKey | str) -> Any`
Loads the config setting stored under the environment variable `env_var`.
* **Args**:
* `env_var (str)`: Name of the environment variable to load configuration setting from
* **Returns**: (`Any`) The configuration setting stored under `env_var`
### AWS Secret Loader
Loads secrets from AWS Secrets Manager. To authenticate access to AWS Secrets Manager, either
* Configure your AWS profile using the AWS CLI
```bash theme={"system"}
aws configure
```
* Manually specify your AWS Credentials when constructing the configuration loader
`python config = AWSSecretLoader( aws_access_key_id = 'your access key id', aws_secret_access_key = 'your secret key', region_name = 'your region' ) `
**Example**:
```python theme={"system"}
from mage_ai.io.config import ConfigKey, AWSSecretLoader
config = AWSSecretLoader()
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
# with finer control on version
postgres_db = config.get(ConfigKey.POSTGRES_DBNAME, version_id='my_version_id')
```
**Constructor** : `__init__(self, **kwargs)`:
* **Keyword Arguments**:
* `aws_access_key_id (str, Optional)`: AWS access key ID credential.
* `aws_secret_access_key (str, Optional)`: AWS secret access key credential.
* `region_name (str, Optional)`: AWS region which Secrets Manager is created in.
**Methods**:
***contains*** - `contains( secret_id: ConfigKey | str, version_id: str, version_stage_label : str) -> bool`
Check if there is a secret with ID `secret_id` contained. Can also specify the version of the
secret to check. If
* both `version_id` and `version_stage_label` are specified, both must agree on the secret version.
* neither of `version_id` or `version_stage_label` are specified, any version is checked.
* one of `version_id` and `version_stage_label` are specified, the associated version is checked.
When using the `in` operator, comparisons to specific versions are not allowed.
* **Args**:
* `secret_id (str)`: ID of the secret to load
* `version_id (str, Optional)`: ID of the version of the secret to load. Defaults to `None`.
* `version_stage_label (str, Optional)`: Staging label of the version of the secret to load. Defaults to `None`.
* **Returns**: (`bool`) Returns true if secret exists, otherwise returns false.
***get*** - `get(secret_id: ConfigKey | str, version_id: str, version_stage_label : str) -> bytes | str`
Loads the secret stored under `secret_id`. Can also specify the version of the
secret to fetch. If
* both `version_id` and `version_stage_label` are specified, both must agree on the secret version.
* neither of `version_id` or `version_stage_label` are specified, the current version is loaded.
* one of `version_id` and `version_stage_label` are specified, the associated version is loaded.
When using the `__getitem__` overload, comparisons to specific versions are not allowed.
* **Args**:
* `secret_id (str)`: ID of the secret to load
* ` version_id (str, Optional)`: ID of the version of the secret to load. Defaults to `None`.
* `version_stage_label (str, Optional)`: Staging label of the version of the secret to load. Defaults to `None`.
* **Returns**: (`bytes | str`) The secret stored under `secret_id` in AWS secret manager. If secret is a binary value, returns a `bytes` object; else returns a `string` object
# Pipeline Management
Source: https://docs.mage.ai/design/data-pipeline-management
## Setup
If you havenโt setup a project before, check out the
[setup guide](/getting-started/setup) before starting.
***
## Pipelines
`http://localhost:3000/pipelines`
This page will show all the pipelines in your project.
> Core abstraction: [Pipeline](/design/core-abstractions#pipeline)
>
> A pipeline contains references to all the blocks of code you want to run,
> charts for visualizing data, and organizes the dependency between each block
> of code.
Learn more about
[**projects and pipelines here**](/design/core-abstractions#project).
From this page, you can also create a new pipeline by clicking the
**`[+ New pipeline]`** button.
#### Creating new pipeline
Creating a new pipeline will take you to the **Pipeline edit page**; a
notebook-like experience for adding blocks, creating dependencies between
blocks, testing code, and visualizing data with charts.
Learn more about the [Notebook for building data
pipelines](/about/features#notebook-for-building-data-pipelines)
***
## Pipeline runs
[`http://localhost:3000/pipeline-runs`](http://localhost:3000/pipeline-runs)
View all the runs for every pipeline in your current project.
> Core abstraction: [Run](/design/core-abstractions#run)
>
> A run record stores information about when it was started, its status, when it
> was completed, any runtime variables used in the execution of the pipeline or
> block, etc.
***
## Pipeline detail
`http://localhost:3000/pipelines/[uuid]`
This page contains all the information and history for a single pipeline:
1. [Triggers](/design/core-abstractions#trigger)
2. [Runs](/design/core-abstractions#run)
3. [Logs](/design/core-abstractions#log)
***
## Triggers
[`http://localhost:3000/pipelines/example_pipeline/triggers`](http://localhost:3000/pipelines/example_pipeline/triggers)
This page shows all the active and inactive triggers for a single pipeline.
> Core abstraction: [Trigger](/design/core-abstractions#trigger)
>
> A trigger is a set of instructions that determine when or how a pipeline
> should run.
***
#### Create trigger
`http://localhost:3000/pipelines/[uuid]/triggers/[id]/edit`
Create a new trigger for this pipeline by clicking the **`[+ Create]`** button
near the top of the page.
You can configure the trigger to run the pipeline on a schedule, when an event
occurs, or when an API is called.
> Core abstraction: [Schedule](/design/core-abstractions#schedule)
>
> A schedule type trigger will instruct the pipeline to run after a start date
> and on a set interval.
***
> Core abstraction: [Event](/design/core-abstractions#event)
>
> An event type trigger will instruct the pipeline to run whenever a specific
> event occurs.
***
> Core abstraction: [API](/design/core-abstractions#api)
>
> An API trigger will instruct the pipeline to run whenever a specific API
> is called.
Example page for creating a schedule type trigger.
***
#### Trigger detail
On this page, you can start or pause the trigger. Starting the trigger will make
it active. Pausing the trigger will prevent it from running the pipeline.
If you have other triggers for this pipeline, pausing 1 trigger may not stop
the pipeline from running since other triggers can also run the pipeline.
***
You can also edit the trigger after creating it by clicking the
**`[Edit trigger]`** button.
***
## Runs
[`http://localhost:3000/pipelines/example_pipeline/runs`](http://localhost:3000/pipelines/example_pipeline/runs)
View the pipeline runs and block runs for a pipeline.
> Core abstraction: [Run](/design/core-abstractions#run)
>
> A run record stores information about when it was started, its status, when it
> was completed, any runtime variables used in the execution of the pipeline or
> block, etc.
#### Retry run
*WIP*
***
## Logs
[`http://localhost:3000/pipelines/example_pipeline/logs`](http://localhost:3000/pipelines/example_pipeline/logs)
Browse all logs for a pipeline. You can search and filter logs by log level,
block type, block UUID, and more.
> Core abstraction: [**Log**](/design/core-abstractions#log)
>
> A log is a file that contains system output information.
***
## Backfill
*WIP*
***
## Monitor
You can monitor many metrics for each of your pipelines and blocks. Soon, youโll
be able to monitor aggregate metrics across all pipelines and blocks.
Read more [here](/development/observability/monitoring).
## Concurrency
You can limit the concurrency of the block execution to reduce resource consumption.
### Global concurrency
You can configure the maximum number of concurrent block runs in projectโs `metadata.yaml` via `queue_config`.
```yaml theme={"system"}
queue_config:
concurrency: 100
```
The default value of concurrency is 20.
### Global concurrency by block uuid
You can configure the global block run concurrency by block uuid in project's `metadata.yaml` via `concurrency_config`.
**Example**
```yaml theme={"system"}
concurrency_config:
block_run_limit_per_block_uuid:
block_uuid_1: 1
block_uuid_2: 5
```
In the example above:
* `block_uuid_1` can run at most 1 concurrent execution.
* `block_uuid_2` can run up to 5 concurrent executions.
**Limitations**
This feature is not supported in the following scenarios:
* Replica blocks: Each replica is assigned a unique UUID for block run, so concurrency limits are not enforced across replicas.
* Dynamic blocks: Each dynamic child block has a unique UUID for block run, preventing concurrency limits from applying across them.
* Single-process mode: When `run_pipeline_in_one_process` is set to `true` in the pipelineโs `metadata.yaml`, global concurrency is not enforced.
### Pipeline level concurrency
You can edit the `concurrency_config` in each pipeline's metadata.yaml file to enforce pipeline level concurrency.
Here is the example:
```yaml theme={"system"}
concurrency_config:
block_run_limit: 5
pipeline_run_limit: 3
```
The default value of `block_run_limit` and `pipeline_run_limit` can be set via environment variables: `CONCURRENCY_CONFIG_BLOCK_RUN_LIMIT` and `CONCURRENCY_CONFIG_PIPELINE_RUN_LIMIT`
* `block_run_limit`: limit the concurrent blocks runs in one pipeline run.
* `pipeline_run_limit`: limit the concurrent pipeline runs in one pipeline trigger.
* `pipeline_run_limit_all_triggers`: limit the concurrent pipeline runs across all triggers in a pipeline.
* `on_pipeline_run_limit_reached`: choose whether to `wait` or `skip` when the pipeline run limit is reached.
## Variable storage
Mage automatically persists the output of block runs on disk. You can specify the path or the storage for block output variables
in the following ways.
* Specify the data directory path via `MAGE_DATA_DIR` environment variable. If you use Mage docker image, this environment
variable is set to `/home/src/mage_data` by default.
* If the `MAGE_DATA_DIR` environment variable is not set, you can set the `variables_dir` path in project's metadata.yaml. Here is
an example:
```yaml theme={"system"}
variables_dir: /home/src/mage_data
```
* You can also use an external storage to store the block output variables by specifying the `remote_variables_dir` path in project's
metadata.yaml.
* AWS S3 storage:
```yaml theme={"system"}
variables_dir: /home/src/mage_data
remote_variables_dir: s3://bucket/path_prefix
```
* Google Cloud Storage:
```yaml theme={"system"}
variables_dir: /home/src/mage_data
remote_variables_dir: gs://bucket/path_prefix
```
When using GCS for your remote variables directory, if you run into a "Your default credentials were not found"
error, you may need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of your
`application_default_credentials.json` file with your Google Cloud credentials and the `GOOGLE_CLOUD_PROJECT`
environment variable to your Google Cloud project ID.
## Variable retention
If you want to clean up the old variables in your variable storage, you can set the `variables_retention_period`
config in project's metadata.yaml. The valid period should end with "d", "h", or "w".
Example config:
```yaml theme={"system"}
variables_retention_period: 30d
```
After configuring `variables_retention_period` in project's metadata.yaml, you can run the following command
to clean up old variables:
```bash theme={"system"}
mage clean-cached-variables [project_path]
```
## Cache block output in memory
By default, Mage persists block output on disk. In pipeline's metadata.yaml, you have the option to configure the pipeline to cache
the block output in memory instead of persisting the block output on disk. The feature is only supported in standard batch
pipeline (without dynamic blocks) for now.
Example config:
```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```
***
## Operation history
Keeps track of recently viewed pipelines so you can easily navigate back to them
in the Pipelines Dashboard (`/pipelines`). Adds a `RECENTLY VIEWED` tab to the Pipelines
Dashboard that lists these pipelines.
# Callback templates
Source: https://docs.mage.ai/development/blocks/callbacks/templates
## Base
```python theme={"system"}
if 'callback' not in globals():
from mage_ai.data_preparation.decorators import callback
@callback('success')
def success_callback(**kwargs):
pass
@callback('failure')
def failure_callback(**kwargs):
pass
```
# Conditional templates
Source: https://docs.mage.ai/development/blocks/conditionals/templates
## Base
```python theme={"system"}
if 'condition' not in globals():
from mage_ai.data_preparation.decorators import condition
@condition
def evaluate_condition(*args, **kwargs) -> bool:
return True
```
# Data exporter templates
Source: https://docs.mage.ai/development/blocks/data_exporters/templates
## Generic
```python theme={"system"}
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(*args, **kwargs):
"""
Exports data to some source
Args:
args: The input variables from upstream blocks
Output (optional):
Optionally return any object and it'll be logged and
displayed when inspecting the block run.
"""
# Specify your data exporting logic here
```
## Local file
```python theme={"system"}
from mage_ai.io.file import FileIO
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_file(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to filesystem.
Docs: /design/data-loading#fileio
"""
filepath = 'path/to/write/dataframe/to.csv'
FileIO().export(df, filepath)
```
## Azure Blob Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_azure_blob_storage(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Azure Blob Storage.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = 'your_container_name'
blob_path = 'your_blob_path'
AzureBlobStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
container_name,
blob_path,
)
```
## Google BigQuery
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_big_query(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a BigQuery warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#bigquery
"""
table_id = 'your-project.your_dataset.your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
table_id,
if_exists='replace', # Specify resolution policy if table name already exists
)
```
## Google Cloud Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Google Cloud Storage bucket.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#googlecloudstorage
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
object_key = 'your_object_key'
GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
bucket_name,
object_key,
)
```
## MySQL
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a MySQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#mysql
"""
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
None,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
)
```
## PostgreSQL
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a PostgreSQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#postgresql
"""
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
)
```
## Amazon Redshift
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_redshift(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Redshift cluster.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#redshift
"""
table_name = 'your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
if_exists='replace', # Specify resolution policy if table already exists
)
```
## Amazon S3
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_s3(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a S3 bucket.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#s3
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
object_key = 'your_object_key'
S3.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
bucket_name,
object_key,
)
```
## Snowflake
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Snowflake warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#snowflake
"""
table_name = 'your_table_name'
database = 'your_database_name'
schema = 'your_schema_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
database,
schema,
if_exists='replace', # Specify resolution policy if table already exists
)
```
## Delta Lake
### Amazon S3
```python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
"""
Export data to a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
"""
storage_options = {
'AWS_ACCESS_KEY_ID': '',
'AWS_SECRET_ACCESS_KEY': '',
'AWS_REGION': '',
'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
}
uri = 's3://[bucket]/[key]'
write_deltalake(
uri,
data=df,
mode='append', # append or overwrite
overwrite_schema=False, # set True to alter the schema when overwriting
partition_by=[],
storage_options=storage_options,
)
```
### Azure Blob Storage
```python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
"""
Export data to a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
"""
storage_options = {
'AZURE_STORAGE_ACCOUNT_NAME': '',
'AZURE_STORAGE_ACCOUNT_KEY': '',
'AZURE_STORAGE_ACCESS_KEY': '',
'AZURE_STORAGE_MASTER_KEY': '',
'AZURE_STORAGE_CLIENT_ID': '',
'AZURE_STORAGE_CLIENT_SECRET': '',
'AZURE_STORAGE_TENANT_ID': '',
'AZURE_STORAGE_SAS_KEY': '',
'AZURE_STORAGE_TOKEN': '',
'AZURE_STORAGE_USE_EMULATOR': '',
}
uri = 'az://[container]/[key]'
write_deltalake(
uri,
data=df,
mode='append', # append or overwrite
overwrite_schema=False, # set True to alter the schema when overwriting
partition_by=[],
storage_options=storage_options,
)
```
### Google Cloud Storage
```python theme={"system"}
from deltalake.writer import write_deltalake
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
"""
Export data to a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
"""
storage_options = {
'GOOGLE_SERVICE_ACCOUNT': '',
'GOOGLE_SERVICE_ACCOUNT_PATH': '',
'GOOGLE_SERVICE_ACCOUNT_KEY': '',
'GOOGLE_BUCKET': '',
}
uri = 'gs://[bucket]/[key]'
write_deltalake(
uri,
data=df,
mode='append', # append or overwrite
overwrite_schema=False, # set True to alter the schema when overwriting
partition_by=[],
storage_options=storage_options,
)
```
## Orchestration
### Trigger pipeline
```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def trigger(*args, **kwargs):
"""
Trigger another pipeline to run.
Documentation: /orchestration/triggers/trigger-pipeline
"""
trigger_pipeline(
'pipeline_uuid', # Required: enter the UUID of the pipeline to trigger
variables={}, # Optional: runtime variables for the pipeline
check_status=False, # Optional: poll and check the status of the triggered pipeline
error_on_failure=False, # Optional: if triggered pipeline fails, raise an exception
poll_interval=60, # Optional: check the status of triggered pipeline every N seconds
poll_timeout=None, # Optional: raise an exception after N seconds
verbose=True, # Optional: print status of triggered pipeline run
)
```
# Data loader templates
Source: https://docs.mage.ai/development/blocks/data_loaders/templates
## Generic
```python theme={"system"}
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data(*args, **kwargs):
"""
Template code for loading data from any source.
Returns:
Anything (e.g. data frame, dictionary, array, int, str, etc.)
"""
# Specify your data loading logic here
return {}
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Local file
```python theme={"system"}
from mage_ai.io.file import FileIO
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_file(*args, **kwargs):
"""
Template for loading data from filesystem.
Load data from 1 file or multiple file directories.
For multiple directories, use the following:
FileIO().load(file_directories=['dir_1', 'dir_2'])
Docs: /design/data-loading#fileio
"""
filepath = 'path/to/your/file.csv'
return FileIO().load(filepath)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## API
```python theme={"system"}
import io
import pandas as pd
import requests
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_api(*args, **kwargs):
"""
Template for loading data from API
"""
url = ''
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Azure Blob Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_from_azure_blob_storage(*args, **kwargs):
"""
Template for loading data from a Azure Blob Storage.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#azureblobstorage
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = 'your_container_name'
blob_path = 'your_blob_path'
return AzureBlobStorage.with_config(ConfigFileLoader(config_path, config_profile)).load(
container_name,
blob_path,
)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Google BigQuery
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_big_query(*args, **kwargs):
"""
Template for loading data from a BigQuery warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#bigquery
"""
query = 'your_gbq_query'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
return BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Google Cloud Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_from_google_cloud_storage(*args, **kwargs):
"""
Template for loading data from a Google Cloud Storage bucket.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#googlecloudstorage
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
object_key = 'your_object_key'
return GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).load(
bucket_name,
object_key,
)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## MySQL
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_mysql(*args, **kwargs):
"""
Template for loading data from a MySQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#mysql
"""
query = 'Your MySQL query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## PostgreSQL
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_postgres(*args, **kwargs):
"""
Template for loading data from a PostgreSQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#postgresql
"""
query = 'your PostgreSQL query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Amazon Redshift
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_redshift(*args, **kwargs):
"""
Template for loading data from a Redshift cluster.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#redshift
"""
query = 'your_redshift_selection_query'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Amazon S3
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_from_s3_bucket(*args, **kwargs):
"""
Template for loading data from a S3 bucket.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#s3
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
object_key = 'your_object_key'
return S3.with_config(ConfigFileLoader(config_path, config_profile)).load(
bucket_name,
object_key,
)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Snowflake
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_snowflake(*args, **kwargs):
"""
Template for loading data from a Snowflake warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: /design/data-loading#example-loading-data-from-snowflake-warehouse
"""
query = 'your_snowflake_query'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Delta Lake
### Amazon S3
```python theme={"system"}
from deltalake import DeltaTable
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_table(*args, **kwargs):
"""
Load a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table
"""
storage_options = {
'AWS_ACCESS_KEY_ID': '',
'AWS_SECRET_ACCESS_KEY': '',
'AWS_REGION': '',
'AWS_S3_ALLOW_UNSAFE_RENAME': 'true',
}
uri = 's3://[bucket]/[key]'
dt = DeltaTable(uri, storage_options=storage_options)
return dt.to_pandas()
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
### Azure Blob Storage
```python theme={"system"}
from deltalake import DeltaTable
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_table(*args, **kwargs):
"""
Load a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table
"""
storage_options = {
'AZURE_STORAGE_ACCOUNT_NAME': '',
'AZURE_STORAGE_ACCOUNT_KEY': '',
'AZURE_STORAGE_ACCESS_KEY': '',
'AZURE_STORAGE_MASTER_KEY': '',
'AZURE_STORAGE_CLIENT_ID': '',
'AZURE_STORAGE_CLIENT_SECRET': '',
'AZURE_STORAGE_TENANT_ID': '',
'AZURE_STORAGE_SAS_KEY': '',
'AZURE_STORAGE_TOKEN': '',
'AZURE_STORAGE_USE_EMULATOR': '',
}
uri = 'az://[container]/[key]'
dt = DeltaTable(uri, storage_options=storage_options)
return dt.to_pandas()
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
### Google Cloud Storage
```python theme={"system"}
from deltalake import DeltaTable
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_table(*args, **kwargs):
"""
Load a Delta Table
Docs: https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table
"""
storage_options = {
'GOOGLE_SERVICE_ACCOUNT': '',
'GOOGLE_SERVICE_ACCOUNT_PATH': '',
'GOOGLE_SERVICE_ACCOUNT_KEY': '',
'GOOGLE_BUCKET': '',
}
uri = 'gs://[bucket]/[key]'
dt = DeltaTable(uri, storage_options=storage_options)
return dt.to_pandas()
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Druid
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.druid import Druid
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_druid(*args, **kwargs):
"""
Template for loading data from a Druid warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#druid
"""
query = 'your Druid query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Druid.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Pinot
```python theme={"system"}
from mage_ai.data_preparation.repo_manager import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.pinot import Pinot
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_pinot(*args, **kwargs):
"""
Template for loading data from a Pinot warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#pinot
"""
query = 'your Pinot query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Pinot.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
## Orchestration
### Trigger pipeline
```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def trigger(*args, **kwargs):
"""
Trigger another pipeline to run.
Documentation: /orchestration/triggers/trigger-pipeline
"""
trigger_pipeline(
'pipeline_uuid', # Required: enter the UUID of the pipeline to trigger
variables={}, # Optional: runtime variables for the pipeline
check_status=False, # Optional: poll and check the status of the triggered pipeline
error_on_failure=False, # Optional: if triggered pipeline fails, raise an exception
poll_interval=60, # Optional: check the status of triggered pipeline every N seconds
poll_timeout=None, # Optional: raise an exception after N seconds
verbose=True, # Optional: print status of triggered pipeline run
)
```
# Sensor templates
Source: https://docs.mage.ai/development/blocks/sensors/templates
## Generic
```python theme={"system"}
from mage_ai.orchestration.run_status_checker import check_status
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def check_condition(**kwargs) -> bool:
"""
Template code for checking if block or pipeline run completed.
"""
return check_status(
'pipeline_uuid',
kwargs['execution_date'],
block_uuid='block_uuid', # optional if you want the sensor to wait on a specific block
hours=24, # optional if you want to check for a specific number of hours. Default is 24
)
```
## Google BigQuery
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def query_bigquery_and_check_condition(**kwargs) -> bool:
"""
Template code for checking the results of a BigQuery query.
Specify your configuration settings in 'io_config.yaml'.
Return: True if the sensor should complete, False if it should
keep waiting
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your BigQuery query' # Specify your SQL query here
loader = BigQuery.with_config(ConfigFileLoader(config_path, config_profile))
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
## MySQL
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def query_mysql_and_check_condition(**kwargs) -> bool:
"""
Template code for checking the results of a MySQL query.
Specify your configuration settings in 'io_config.yaml'.
Return: True if the sensor should complete, False if it should
keep waiting
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your MySQL query' # Specify your SQL query here
with MySQL.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
## PostgreSQL
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def query_postgres_and_check_condition(**kwargs) -> bool:
"""
Template code for checking the results of a Postgres query.
Specify your configuration settings in 'io_config.yaml'.
Return: True if the sensor should complete, False if it should
keep waiting
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your Postgres query' # Specify your SQL query here
with Postgres.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
## Amazon Redshift
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def query_redshift_and_check_condition(**kwargs) -> bool:
"""
Template code for checking the results of a Redshift query.
Specify your configuration settings in 'io_config.yaml'.
Return: True if the sensor should complete, False if it should
keep waiting
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your Redshift query' # Specify your SQL query here
with Redshift.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
## Amazon S3
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def check_condition(**kwargs) -> bool:
"""
Template code for checking if a file or folder exists in a S3 bucket
You will also need to fill out the following AWS related fields
in `io_config.yaml`:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_REGION
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
s3_path = 'path/to/folder/or/file'
config_file_loader = ConfigFileLoader(config_path, config_profile)
return S3.with_config(config_file_loader).exists(
bucket_name, s3_path
)
```
## Google Cloud Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def check_condition(*args, **kwargs) -> bool:
"""
Template code for checking if a file or folder exists in a Google Cloud Storage bucket.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#googlecloudstorage
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = 'your_bucket_name'
object_key = 'your_object_key'
return GoogleCloudStorage.with_config(ConfigFileLoader(config_path, config_profile)).exists(
bucket_name,
object_key,
)
```
## Snowflake
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path
if 'sensor' not in globals():
from mage_ai.data_preparation.decorators import sensor
@sensor
def query_snowflake_and_check_condition(**kwargs) -> bool:
"""
Template code for checking the results of a Snowflake query.
Specify your configuration settings in 'io_config.yaml'.
Return: True if the sensor should complete, False if it should
keep waiting
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = 'Your Snowflake query' # Specify your SQL query here
with Snowflake.with_config(
ConfigFileLoader(config_path, config_profile)) as loader:
df = loader.load(query)
# Add your checks here
if df.empty:
return False
return True
```
# Transformer templates
Source: https://docs.mage.ai/development/blocks/transformers/templates
## Generic
```python theme={"system"}
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@transformer
def transform(*args, **kwargs):
"""
Template code for a transformer block.
Add more parameters to this function if this block has multiple parent blocks.
There should be one parameter for each output variable from each parent block.
Args:
args: The input variables from upstream blocks
Returns:
Anything (e.g. data frame, dictionary, array, int, str, etc.)
"""
# Specify your transformation logic here
return {}
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
# CLI Commands Reference
Source: https://docs.mage.ai/development/cli-commands
Complete reference guide for all Mage CLI commands, including usage instructions and OSS vs Pro availability.
Mage provides a command-line interface (CLI) for managing projects, running pipelines, and performing administrative tasks. The CLI is built using [Typer](https://typer.tghoward.com/) and supports both interactive and scripted usage.
## Quick Reference
### Command Availability
| Command | Description | OSS | Pro |
| -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | --- | --- |
| [`init`](#init-initialize-mage-project) | Initialize a new Mage project at the specified path | โ | โ |
| [`start`](#start-start-mage-server-and-ui) | Start the Mage server and web UI | โ | โ |
| [`run`](#run-run-pipeline) | Execute a pipeline or a specific block within a pipeline | โ | โ |
| [`clean-cached-variables`](#clean-cached-variables-clean-cached-variables) | Clean up old pipeline run outputs based on retention period | โ | โ |
| [`clean-old-logs`](#clean-old-logs-clean-old-logs) | Clean up old pipeline run logs based on retention period | โ | โ |
| [`clean-workspaces`](#clean-workspaces-clean-workspaces) | Clean up old pipeline run outputs and logs inside workspace pods | โ | โ |
| [`clean-unused-blocks`](#clean-unused-blocks-clean-unused-blocks) | Clean up unused blocks from the project (interactive) | โ | โ |
| [`create-secret`](#create-secret-create-or-update-secret) | Create or update a secret for the given Mage project | โ | โ |
| [`run-post-start-hooks`](#run-post-start-hooks-execute-post-start-hooks) | Execute post-start hooks with graceful error handling | โ | โ |
| [`restart-jobs`](#restart-jobs-restart-failed-or-cancelled-pipeline-runs) | Restart failed or cancelled pipeline runs with optional filters | โ | โ |
| [`update-workspace`](#update-workspace-update-workspace-configuration) | Update a workspace configuration synchronously | โ | โ |
| [`update-all-workspaces`](#update-all-workspaces-update-all-workspaces) | Update all workspaces in the cluster synchronously | โ | โ |
| [`re-add-workspace-routes`](#re-add-workspace-routes-re-add-workspace-routes-to-ingress) | Re-add workspace routes to the ingress (Kubernetes) | โ | โ |
| [`recover-pipelines-from-file-versions`](#recover-pipelines-from-file-versions-recover-pipelines-from-file-versions) | Recover pipelines from file versions in `.file_versions` folder | โ | โ |
### Getting Help
To see all available commands:
```bash theme={"system"}
mage --help
```
To get help for a specific command:
```bash theme={"system"}
mage --help
```
***
## Command Categories
* **[Project Management](#project-management)** - Initialize and start Mage projects
* **[Pipeline Execution](#pipeline-execution)** - Run pipelines and blocks
* **[Maintenance Commands](#maintenance-commands)** - Clean up variables and logs
* **[Mage Pro Only Commands](#mage-pro-only-commands)** - Workspace management and advanced features
***
## Available Commands
### Project Management
#### `init` - Initialize Mage Project
**Available in:** โ OSS, โ Pro
Initialize a new Mage project at the specified path.
**Usage:**
```bash theme={"system"}
mage init [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project to be created
* `--project-type` (optional): Type of project to create. Options: `main`, `sub`, or `standalone` (default: `standalone`)
* `--cluster-type` (optional): Type of instance to create for workspace management (Pro only)
* `--project-uuid` (optional): Project UUID for the new project
**Example:**
```bash theme={"system"}
mage init my_project --project-type standalone
```
***
#### `start` - Start Mage Server and UI
**Available in:** โ OSS, โ Pro
Start the Mage server and web UI.
**Usage:**
```bash theme={"system"}
mage start [project_path] [OPTIONS]
```
**Parameters:**
* `project_path` (optional): Path of the Mage project to be loaded (default: current directory)
* `--host` (optional): Specify the host (default: `localhost`)
* `--port` (optional): Specify the port (default: `6789`)
* `--manage-instance` (optional): Enable manage instance (default: `0`)
* `--dbt-docs-instance` (optional): Enable dbt docs instance (default: `0`)
* `--instance-type` (optional): Specify the instance type (default: `server_and_scheduler`)
* `--project-type` (optional): Create project of this type if it does not exist. Options: `main`, `sub`, or `standalone` (default: `standalone`)
* `--cluster-type` (optional): Type of instance to create for workspace management (Pro only)
* `--project-uuid` (optional): Set project UUID for the repo that is being started
**Example:**
```bash theme={"system"}
mage start . --host 0.0.0.0 --port 6789
```
***
### Pipeline Execution
#### `run` - Run Pipeline
**Available in:** โ OSS, โ Pro
Execute a pipeline or a specific block within a pipeline.
**Usage:**
```bash theme={"system"}
mage run [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project that contains the pipeline
* `pipeline_uuid` (required): UUID of the pipeline to be run
* `--test` (optional): Specify if tests should be run (default: `False`)
* `--block-uuid` (optional): UUID of the block to be run (if not specified, runs entire pipeline)
* `--execution-partition` (optional): Execution partition identifier
* `--executor-type` (optional): Executor type to use
* `--callback-url` (optional): Callback URL for block execution
* `--block-run-id` (optional): Block run ID
* `--pipeline-run-id` (optional): Pipeline run ID
* `--runtime-vars` (optional): Runtime variables in JSON format. These will overwrite the pipeline global variables
* `--skip-sensors` (optional): Specify if the sensors should be skipped (default: `False`)
* `--template-runtime-configuration` (optional): Runtime configuration of data integration block runs (JSON format)
**Examples:**
```bash theme={"system"}
# Run entire pipeline
mage run /path/to/project my_pipeline_uuid
# Run specific block
mage run /path/to/project my_pipeline_uuid --block-uuid my_block_uuid
# Run with runtime variables
mage run /path/to/project my_pipeline_uuid --runtime-vars '{"key": "value"}'
# Run with tests
mage run /path/to/project my_pipeline_uuid --test
```
***
### Maintenance Commands
#### `clean-cached-variables` - Clean Cached Variables
**Available in:** โ OSS, โ Pro
Clean up old pipeline run outputs based on retention period.
**Usage:**
```bash theme={"system"}
mage clean-cached-variables [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project to clean variables
* `--pipeline-uuid` (optional): UUID of the pipeline to clean (if not specified, cleans all pipelines)
* `--retention-period` (optional): Default retention period, e.g., `7d`, `2w` (days or weeks)
**Example:**
```bash theme={"system"}
mage clean-cached-variables /path/to/project --retention-period 7d
```
***
#### `clean-old-logs` - Clean Old Logs
**Available in:** โ OSS, โ Pro
Clean up old pipeline run logs based on retention period.
**Usage:**
```bash theme={"system"}
mage clean-old-logs [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project to clean old logs
* `--pipeline-uuid` (optional): UUID of the pipeline to clean (if not specified, cleans all pipelines)
* `--retention-period` (optional): Default retention period, e.g., `7d`, `2w` (days or weeks)
**Example:**
```bash theme={"system"}
mage clean-old-logs /path/to/project --retention-period 30d
```
***
#### `clean-workspaces` - Clean Workspaces
**Available in:** โ OSS, โ Pro
Clean up old pipeline run outputs and logs inside Kubernetes workspace pods.
**Usage:**
```bash theme={"system"}
mage clean-workspaces [OPTIONS]
```
**Parameters:**
* `--cluster-type` (optional): Cluster type for workspace cleanup. Currently only `k8s` is supported
* `--workspace-name` (optional): Workspace name to clean. If not specified, cleans all workspaces
* `--pipeline-uuid` (optional): UUID of the pipeline to clean
* `--retention-period` (optional): Default retention period, e.g., `7d`, `2w` (days or weeks)
* `--skip-unavailable / --no-skip-unavailable` (optional): Skip workspaces without a running pod instead of failing the cleanup job
**Example:**
```bash theme={"system"}
mage clean-workspaces --retention-period 30d
```
***
### Mage Pro Only Commands
The following commands are **exclusively available in Mage Pro**:
#### `clean-unused-blocks` - Clean Unused Blocks
**Available in:** โ OSS, โ Pro
Clean up unused blocks from the given Mage project. This command interactively prompts you to delete blocks that are not referenced by any pipeline.
**Usage:**
```bash theme={"system"}
mage clean-unused-blocks [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project
* `--block-type` (optional): The type of block to clean up (e.g., `data_loader`, `transformer`, `data_exporter`)
**Example:**
```bash theme={"system"}
mage clean-unused-blocks /path/to/project
```
**Interactive Options:**
* `y`: Yes, delete the block
* `N`: No, skip the block
* `c`: Check content of the block before deciding
* `q`: Quit the operation
***
#### `create-secret` - Create or Update Secret
**Available in:** โ OSS, โ Pro
Create or update a secret for the given Mage project.
**Usage:**
```bash theme={"system"}
mage create-secret [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project
* `name` (required): Name of the secret
* `value` (required): Value of the secret
* `--update-exists` (optional): Whether to update the secret if it already exists (default: `True`)
**Example:**
```bash theme={"system"}
mage create-secret /path/to/project my_secret_name "my_secret_value"
```
***
#### `run-post-start-hooks` - Execute Post-Start Hooks
**Available in:** โ OSS, โ Pro
Execute post-start hooks with graceful error handling. This command reads `POST_START_COMMAND` and `POST_START_HOOK_PATH` from environment variables if not provided as arguments.
**Usage:**
```bash theme={"system"}
mage run-post-start-hooks [OPTIONS]
```
**Parameters:**
* `--post-start-command` (optional): Post-start command to execute (JSON array or shell command string). Can also be set via `POST_START_COMMAND` environment variable
* `--post-start-hook-path` (optional): Path to post-start hook script file. Can also be set via `POST_START_HOOK_PATH` environment variable
**Example:**
```bash theme={"system"}
mage run-post-start-hooks --post-start-command '["pip", "install", "package"]'
```
***
#### `restart-jobs` - Restart Failed or Cancelled Pipeline Runs
**Available in:** โ OSS, โ Pro
Restart failed or cancelled pipeline runs with optional filters.
**Usage:**
```bash theme={"system"}
mage restart-jobs [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project to restart jobs from
* `--pipeline-uuid` (optional): UUID of the pipeline to filter jobs by
* `--status` (optional): Status filter: `failed`, `cancelled`, or `both` (default: `failed`)
* `--start-time` (optional): Start time filter (ISO format: `2024-01-01T00:00:00`)
* `--end-time` (optional): End time filter (ISO format: `2024-01-01T23:59:59`)
* `--limit` (optional): Maximum number of jobs to restart (default: `100`)
* `--skip-confirmation` (optional): Skip confirmation prompt (default: `False`)
**Example:**
```bash theme={"system"}
mage restart-jobs /path/to/project --status failed --limit 50
```
***
#### `update-workspace` - Update Workspace Configuration
**Available in:** โ OSS, โ Pro
Update a workspace configuration synchronously. The cluster type is determined by the `CLUSTER_TYPE` environment variable, defaulting to `k8s` if not set.
**Usage:**
```bash theme={"system"}
mage update-workspace [OPTIONS]
```
**Parameters:**
* `workspace_name` (required): Name of the workspace to update
* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--container-config` (optional): YAML string for container configuration updates
* `--update-workspace-settings` / `--no-update-workspace-settings` (optional): Whether to update workspace settings with current cluster environment variables (default: `False`)
**Example:**
```bash theme={"system"}
mage update-workspace my_workspace --container-config "memory: 4Gi"
```
***
#### `update-all-workspaces` - Update All Workspaces
**Available in:** โ OSS, โ Pro
Update all workspaces in the cluster synchronously.
**Usage:**
```bash theme={"system"}
mage update-all-workspaces [OPTIONS]
```
**Parameters:**
* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--container-config` (optional): YAML string for container configuration updates
* `--update-workspace-settings` / `--no-update-workspace-settings` (optional): Whether to update workspace settings with current cluster environment variables (default: `False`)
* `--cluster-type` (optional): Type of cluster: `k8s` or `docker` (default: `k8s`)
**Example:**
```bash theme={"system"}
mage update-all-workspaces --cluster-type k8s --update-workspace-settings
```
***
#### `re-add-workspace-routes` - Re-add Workspace Routes to Ingress
**Available in:** โ OSS, โ Pro
Re-add workspace routes to the ingress. This is useful for Kubernetes deployments where workspace routes need to be refreshed.
**Usage:**
```bash theme={"system"}
mage re-add-workspace-routes
```
**Parameters:**
* `project_path` (required): Path of the Mage project
**Example:**
```bash theme={"system"}
mage re-add-workspace-routes /path/to/project
```
***
#### `recover-pipelines-from-file-versions` - Recover Pipelines from File Versions
**Available in:** โ OSS, โ Pro
Recover pipelines from file versions. This command lists all pipelines that have file versions in the `.file_versions` folder and allows you to recover pipeline files from those versions.
**Usage:**
```bash theme={"system"}
mage recover-pipelines-from-file-versions [OPTIONS]
```
**Parameters:**
* `project_path` (required): Path of the Mage project
* `--pipeline-uuid` (optional): UUID of the pipeline to recover. If not specified, all pipelines are processed
* `--all` / `--no-all` (optional): Recover all files automatically without confirmation. Uses the most recent version for each file (default: `False`)
* `--verbose` / `-v` (optional): Show detailed pipeline information including individual files and blocks (default: `False`)
**Examples:**
```bash theme={"system"}
# Interactive recovery
mage recover-pipelines-from-file-versions /path/to/project
# Recover specific pipeline
mage recover-pipelines-from-file-versions /path/to/project --pipeline-uuid my_pipeline
# Recover all automatically
mage recover-pipelines-from-file-versions /path/to/project --all
# Verbose output
mage recover-pipelines-from-file-versions /path/to/project --verbose
```
***
## Common Use Cases
### Initializing a New Project
```bash theme={"system"}
mage init my_data_project --project-type standalone
cd my_data_project
mage start
```
### Running a Pipeline with Variables
```bash theme={"system"}
mage run /path/to/project my_pipeline_uuid \
--runtime-vars '{"date": "2024-01-01", "environment": "production"}'
```
### Cleaning Up Old Data
```bash theme={"system"}
# Clean variables older than 7 days
mage clean-cached-variables /path/to/project --retention-period 7d
# Clean logs older than 30 days
mage clean-old-logs /path/to/project --retention-period 30d
```
### Pro: Managing Secrets
```bash theme={"system"}
mage create-secret /path/to/project database_password "secure_password_123"
```
### Pro: Updating Workspace Configuration
```bash theme={"system"}
# Update a single workspace
mage update-workspace dev_workspace --update-workspace-settings
# Update all workspaces
mage update-all-workspaces --cluster-type k8s
```
***
## Environment Variables
Some commands can be configured via environment variables:
* `CLUSTER_TYPE`: Cluster type for workspace operations (default: `k8s`)
* `POST_START_COMMAND`: Post-start command for `run-post-start-hooks`
* `POST_START_HOOK_PATH`: Post-start hook script path for `run-post-start-hooks`
* `ENV_VAR_INSTANCE_TYPE`: Instance type override for `start` command
***
## Additional Resources
* [Mage OSS CLI Reference](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/cli/main.py) - Source code for OSS CLI commands
* [Project Setup Guide](/development/project/setup) - Learn how to set up a Mage project
* [Workspaces Documentation](/guides/developer-ux/workspaces) - Learn about Mage Pro workspaces
***
## Notes
* All paths can be absolute or relative to the current working directory
* JSON parameters should be properly escaped when used in shell commands
* Pro-only commands require appropriate cluster/workspace configuration
* Some commands may require database connections to be properly configured
# Database
Source: https://docs.mage.ai/development/databases/default
By default, Mage uses sqlite to store orchestration data (trigger, pipeline run, and block run). To use a different database, you can set the `MAGE_DATABASE_CONNECTION_URL` environment variable.
In production, you can set the environment variable in the corresponding Terraform script.
## PostgreSQL
`export MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://user:password@host:port/dbname`
To use a specific `schema_name` in PostgreSQL:
`export MAGE_DATABASE_CONNECTION_URL="postgresql+psycopg2://user:password@host:port/dbname?options=-c%20search_path%3Dschema_name"`
### Get credentials from AWS Secret Manager
You can also instruct Mage to fetch your Postgres DB credentials from AWS Secret Manager. You will need to
set the following environment variables:
| Variable | Value |
| ----------------------- | -------------------------------------------------------------------- |
| `AWS_DB_SECRETS_NAME` | name of secret in AWS Secret Manager |
| `AWS_DEFAULT_REGION` | AWS region the secret is stored in |
| `AWS_ACCESS_KEY_ID` | AWS access key id (not needed if using IAM role to authenticate) |
| `AWS_SECRET_ACCESS_KEY` | AWS secret access key (not needed if using IAM role to authenticate) |
The expected format of the secret is JSON string with the following keys:
* username
* password
* engine (must be `postgres`)
* host
* port
* dbname
## MSSQL (not officially supported)
`export MAGE_DATABASE_CONNECTION_URL="mssql+pyodbc://?odbc_connect=DRIVER={ODBC Driver 18 for SQL Server};SERVER=host;DATABASE=dbname;UID=user;PWD=password;ENCRYPT=yes;TrustServerCertificate=yes;"`
## Troubleshoot database migration failures
Mage uses Alembic migrations for orchestration tables such as triggers, pipeline runs, block runs, users, secrets, and audit events. If the server fails during startup with an error like `alembic`, `sqlalchemy.exc`, `DROP INDEX`, `CREATE INDEX`, `alembic_version`, or `Can't locate revision`, use the Mage database CLI before manually editing the database.
Start with a read-only diagnosis:
```bash theme={"system"}
mage db diagnose
```
If your project uses the default SQLite database, run the command from the project directory or pass the project path:
```bash theme={"system"}
mage db diagnose --project-path /path/to/project
```
The command prints the database dialect, redacted database URL, revision stored in the `alembic_version` table, migration head in the installed Mage code, recommendations, and any pending revisions that have not been migrated yet.
To list every Alembic revision available in the installed Mage code:
```bash theme={"system"}
mage db history
```
This is read-only and does not connect to or change the database.
To inspect the current schema for a specific database table:
```bash theme={"system"}
mage db schema
```
If your database uses a non-default schema, pass it explicitly:
```bash theme={"system"}
mage db schema
--schema
```
This prints the table columns, types, nullability, primary key columns, defaults, indexes, and foreign keys. Use it to confirm whether a failed migration already created or removed the table objects referenced by the error before you decide to roll back or stamp a revision.
### Run pending migrations
If the database is behind the installed Mage code, run:
```bash theme={"system"}
mage db migrate
```
To migrate to a specific Alembic revision:
```bash theme={"system"}
mage db migrate --revision
```
This runs migration SQL and uses Mage's database migration lock so multiple processes do not migrate the same database at the same time.
### Roll back a failed or incompatible migration
If a migration was applied and you need to revert it, roll back one migration:
```bash theme={"system"}
mage db rollback --revision -1
```
Or roll back to a specific revision:
```bash theme={"system"}
mage db rollback --revision
```
Review the migration file before rolling back in production. A rollback can drop columns, indexes, tables, or enum values depending on the migration.
Rollback statements often need exclusive locks on orchestration tables. For example, a downgrade that drops columns from the `user` table can wait while the running Mage server has active sessions using that table. The CLI applies a PostgreSQL lock timeout so the command fails instead of waiting forever:
```bash theme={"system"}
mage db rollback --revision -1 --lock-timeout-seconds 10
```
To inspect active PostgreSQL sessions that may be blocking the rollback:
```bash theme={"system"}
mage db locks
```
For an overall statement cap, also pass:
```bash theme={"system"}
mage db rollback --revision -1 --lock-timeout-seconds 10 --statement-timeout-seconds 120
```
If rollback times out waiting for locks, stop the Mage server or other workers connected to the orchestration database, then retry the rollback. After the rollback finishes, run `mage db diagnose` before restarting Mage.
### Stamp a revision without running SQL
Use `stamp` only when the database schema already matches the target revision but the `alembic_version` metadata is missing or wrong:
```bash theme={"system"}
mage db stamp
```
For non-interactive recovery:
```bash theme={"system"}
mage db stamp --yes
```
`stamp` changes Alembic metadata without creating, dropping, or altering database objects. It can help when a migration partially completed outside Alembic, but it can also hide real schema drift if used incorrectly. If you see an error such as `index "..." does not exist`, confirm whether the rest of the migration's expected schema changes are already present before stamping past that revision.
### Typical recovery flow
1. Back up the database before making changes in production.
2. Run `mage db diagnose`.
3. Run `mage db history` to find the target revision and surrounding migrations.
4. If the error names a table, index, or constraint, run `mage db schema
` to inspect the live table.
5. If the database is behind, run `mage db migrate`.
6. If the latest migration is bad for your deployment, run `mage db rollback --revision -1`, then pin or deploy a Mage version with a compatible migration chain.
7. If the schema already matches the intended revision but `alembic_version` is wrong, run `mage db stamp `.
8. Run `mage db diagnose` again and restart Mage.
# Import your own Python code
Source: https://docs.mage.ai/development/dependencies/custom-files
You can import any Python code from any file that is in your Mage project directory.
## Importing custom Python code
When youโre writing code in Mage, you can import any other Python files using this format:
```python theme={"system"}
from [project_name].foo import bar
```
For example, if your project name is `demo_project` and you have the following folder structure:
```
demo_project/
| pipelines/
| utils/
| | helpers/
| | | shared.py
| | | __init__.py
| | __init__.py
```
You can write the following import statement to use functions in the
`demo_project/utils/helpers/shared.py` file:
```python theme={"system"}
from demo_project.utils.helpers import shared
```
# Dependency Management
Source: https://docs.mage.ai/development/dependencies/dependency-management
Advanced dependency management with isolated virtual environments, automatic caching, and intelligent package resolution in Mage Pro.
Mage Pro includes an advanced dependency management system that provides isolated, reproducible dependency environments for your projects. This system automatically creates and manages virtual environments based on your project's dependencies, ensuring consistent execution across different environments.
## Overview
The dependency management system provides:
* **Isolated environments**: Each unique combination of dependencies gets its own virtual environment
* **Automatic caching**: Environments are reused based on content fingerprinting
* **Fast installation**: Uses `uv` for blazing-fast package installation with `pip` fallback
* **Smart compatibility checking**: Detects and reuses compatible packages from the global environment
* **Safe concurrency**: File-based locking prevents race conditions when multiple processes create environments
## Using Dependency Management
You can manage dependencies in three ways:
### 1. Via UI (Recommended for Quick Updates)
The easiest way to manage your project dependencies is through the Mage UI:
1. **Navigate to the Editor page** in your Mage workspace
2. **Open your requirements file**: Browse to `/home/src/[project_path]/requirements.txt` and open it in the editor
3. **Edit your dependencies**: Add, remove, or update package requirements in the `requirements.txt` file
```txt theme={"system"}
# Example requirements.txt
pandas==2.0.0
numpy>=1.24.0
scikit-learn==1.3.0
```
4. **Click the "Install dependencies" button**: This will automatically:
* Open a terminal at the bottom of the page
* Execute `mage deps install` command
* Show real-time installation progress and output
The system will intelligently install only the packages that need to be added or updated, reusing compatible packages from the global environment when possible.
### 2. Via Mage Terminal
For more control, you can run `mage deps` commands directly in the Mage terminal:
1. **Open the Terminal page** in your Mage workspace
2. **Run dependency commands**:
```bash theme={"system"}
# Ensure virtual environment exists
mage deps ensure
# Install dependencies from requirements.txt
mage deps install
# Install from custom requirements file
mage deps install -r production_requirements.txt
# List all environments
mage deps list
# Clean up old environments
mage deps cleanup
```
See the [CLI Commands](#cli-commands) section below for complete command documentation.
### 3. Via Cluster Restart (Automatic)
When you restart your Mage cluster, the dependency management system automatically:
1. **Reads your latest `requirements.txt`**: Checks for any changes since the last environment was created
2. **Creates or updates the virtual environment**:
* If the requirements haven't changed, reuses the existing environment
* If requirements have changed, creates a new environment with updated dependencies
3. **Activates the environment**: Makes it available for all pipeline executions
This ensures your dependencies are always synchronized with your `requirements.txt` after each cluster restart, with no manual intervention required.
## How It Works
### Automatic Server-Side Management
When the Mage server starts, the dependency management system automatically handles virtual environment setup:
* **If no environment exists**: A new virtual environment is automatically created based on your project's `requirements.txt`
* **If an environment exists**: The matching environment is automatically activated for use
This seamless integration means you don't need to manually manage virtual environments for normal server operation. The system ensures your dependencies are always properly configured without manual intervention.
For more control over environment creation, activation, and management, you can use the [CLI commands](#cli-commands) described below.
### Environment Fingerprinting
The system computes a deterministic fingerprint hash for each environment based on:
* Python version (e.g., `3.11.8`)
* Normalized `requirements.txt` content
* `uv` version (if available)
If any of these change, a new environment is created. If they match an existing environment, that environment is reused.
### Directory Structure
Virtual environments are stored under your Mage data directory:
```
/home/src/mage_data/
โโโ venvs/
โโโ .cleanup.lock # Global cleanup lock
โโโ / # e.g., 3.11.8
โโโ env-/ # Virtual environment
โ โโโ .mage_env.json # Environment metadata
โ โโโ bin/ # Python executables
โ โโโ lib/ # Installed packages
โโโ .locks/ # Lock files for concurrency
โโโ .install.lock # Installation lock
โโโ .activate.lock # Activation lock
โโโ global.lock # Global package versions
โโโ venv.lock # Installed package versions
```
Each virtual environment includes a `.mage_env.json` metadata file that tracks:
* `fingerprint`: Environment fingerprint hash
* `python_version`: Python version
* `created_at`: Creation timestamp (ISO 8601 format)
* `last_used_at`: Last usage timestamp (ISO 8601 format)
* `dependency_source`: Source of dependencies (e.g., `requirements.txt`)
### Package Installation Strategy
The system uses a smart installation strategy to minimize redundant installations:
1. **Global package detection**: Scans globally installed packages
2. **Compatibility filtering**: Filters out requirements already satisfied by global packages
3. **Constrained installation**: Installs remaining packages with global package versions as constraints
4. **Fallback handling**: Automatically retries without constraints if dependency conflicts occur
### Installation Tools
* **Primary**: `uv pip install` - Fast, modern Python package installer
* **Fallback**: `pip install` - Standard Python package installer
* **Virtual environment creation**: `uv venv --system-site-packages`
The `--system-site-packages` flag allows the virtual environment to inherit global packages while still allowing user packages to override them when needed.
## CLI Commands
### `mage deps ensure`
Create or reuse a virtual environment for your project's dependencies.
**Usage:**
```bash theme={"system"}
mage deps ensure [PROJECT_PATH] [OPTIONS]
```
**Parameters:**
* `project-path` (optional): Path to the Mage project (defaults to current directory)
* `--python-version` (optional): Python version to use (defaults to current Python version)
* `--output-file` (optional): File to write environment path to
**Examples:**
```bash theme={"system"}
# Ensure environment for current project
mage deps ensure
# Ensure environment for specific project
mage deps ensure /path/to/project
# Write environment path to file
mage deps ensure --output-file /tmp/venv_path.txt
# Use specific Python version
mage deps ensure --python-version 3.11.8
```
**Output:**
When successful, the command outputs an environment variable assignment:
```bash theme={"system"}
export MAGE_VENV_PATH="/path/to/venv"
```
You can activate this environment in your shell:
```bash theme={"system"}
eval $(mage deps ensure)
source $MAGE_VENV_PATH/bin/activate
```
### `mage deps install`
Install dependencies from a requirements file using the best available installer.
**Usage:**
```bash theme={"system"}
mage deps install [PROJECT_PATH] [OPTIONS]
```
**Parameters:**
* `project-path` (optional): Path to the Mage project (defaults to current directory)
* `-r, --requirements-file` (optional): Path to requirements file (defaults to `requirements.txt` in project)
* `--override-global-version` (optional): Skip global package constraints (default: `False`)
**Examples:**
```bash theme={"system"}
# Install dependencies for current project
mage deps install
# Install from specific requirements file
mage deps install -r custom_requirements.txt
# Install without global package constraints
mage deps install --override-global-version
# Install for specific project
mage deps install /path/to/project
```
**Behavior:**
1. Checks if running in an existing virtual environment (`VIRTUAL_ENV`)
2. If yes: Installs packages using advanced features (filtering, constraints, diff analysis)
3. If no: Installs packages system-wide using standard pip
**Conflict Resolution:**
If installation fails due to conflicting dependencies with global packages, the CLI will prompt:
```
Installation failed due to conflicting dependencies with global package versions.
Would you like to override global package versions and install without constraints? (Y/Yes/yes):
```
Answering "yes" retries the installation without global package constraints.
### `mage deps list`
List all dependency virtual environments with their metadata.
**Usage:**
```bash theme={"system"}
mage deps list [OPTIONS]
```
**Parameters:**
* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--python-version` (optional): Filter by Python version (e.g., `3.11.8`)
* `--verbose` (optional): Show detailed information including full paths
**Examples:**
```bash theme={"system"}
# List all environments
mage deps list
# List environments for specific Python version
mage deps list --python-version 3.11.8
# Show detailed information
mage deps list --verbose
```
**Output:**
```
Virtual Environments:
Python 3.11.8:
env-abc123def456... (created: 2024-01-15T10:30:00Z, last used: 2024-01-20T14:25:00Z)
env-789ghi012jkl... (created: 2024-01-10T08:00:00Z, last used: 2024-01-18T16:45:00Z)
Total: 2 environment(s)
```
### `mage deps cleanup`
Clean up stale virtual environments to free disk space.
**Usage:**
```bash theme={"system"}
mage deps cleanup [OPTIONS]
```
**Parameters:**
* `--project-path` (optional): Path to the Mage project (defaults to current directory)
* `--ttl-seconds` (optional): Time-to-live in seconds (defaults to 604800 = 7 days)
* `--max-retention` (optional): Maximum number of environments to retain across all Python versions
* `--dry-run` (optional): Show what would be deleted without actually deleting (default: `False`)
* `--verbose` (optional): Show detailed information about cleanup process
**Examples:**
```bash theme={"system"}
# Dry run to see what would be deleted
mage deps cleanup --dry-run
# Cleanup with custom TTL (14 days)
mage deps cleanup --ttl-seconds 1209600
# Limit retention to 10 most recent environments
mage deps cleanup --max-retention 10
# Combine options
mage deps cleanup --ttl-seconds 604800 --max-retention 5 --dry-run
```
**How it works:**
1. Identifies environments unused for longer than TTL
2. Skips environments with active installation or activation locks
3. If `max_retention` is set, keeps the N most recently used environments even if they exceed TTL
4. Removes eligible environments and their metadata
**Output:**
```
Cleanup completed:
- Deleted: 3 environment(s)
- Skipped: 2 environment(s)
- Scanned: 5 environment(s)
```
**Configuration:**
You can configure cleanup defaults in your project's `metadata.yaml`:
```yaml theme={"system"}
dependency_venvs_config:
cleanup_config:
ttl_seconds: 1209600 # 14 days
max_retention: 10 # Keep 10 most recent
```
## Features in Detail
### Automatic Package Filtering
The system intelligently filters out packages that are already satisfied by globally installed packages:
```
Checking requirements against global packages...
โ Skipping pandas>=1.3.0 - existing package version 1.5.0 is compatible
โ Installing numpy>=1.22.0 - version 1.21.0 does not satisfy requirement
โ Installing scikit-learn - not found in global packages
Compatibility check complete: 1/3 package(s) satisfied by global packages, 2 will be installed in venv
```
This reduces installation time and disk usage by reusing compatible packages.
### Global Package Constraints
To ensure consistency with the base environment, the system creates a `global.lock` file containing all globally installed package versions and uses it as a constraint file during installation:
```bash theme={"system"}
uv pip install -r requirements.txt -c global.lock
```
This ensures that when a package from `requirements.txt` doesn't specify an exact version, the installer uses the same version as the global environment (if compatible), maintaining consistency.
### Installation Diff Analysis
After installation, the system analyzes the difference between global and virtual environment packages:
```
============================================================
Installation Diff Analysis
============================================================
โ Reused from global: 45 package(s)
โฌ๏ธ Upgraded: 2 package(s)
- numpy: 1.21.0 โ 1.22.0
- pandas: 1.3.0 โ 1.5.0
โ Newly added: 3 package(s)
- scikit-learn==1.2.0
- scipy==1.10.0
- joblib==1.2.0
============================================================
```
This helps you understand what was installed, upgraded, or reused.
### Concurrency Safety
The system uses cross-platform file locking to prevent race conditions:
* **POSIX systems** (Linux, macOS): Uses `fcntl.flock()`
* **Windows**: Uses `msvcrt.locking()`
* **Lock timeout**: 10 minutes (configurable)
**Lock Types:**
1. **Installation lock** (`.install.lock`): Exclusive lock held during environment creation and dependency installation
2. **Activation lock** (`.activate.lock`): Shared lock held when environment is being used
3. **Cleanup lock** (`.cleanup.lock`): Global lock preventing simultaneous cleanup operations
This ensures that if multiple processes try to create the same environment simultaneously, only one creates it while others wait and then reuse it. Environments with active locks are protected from cleanup.
### Environment Validation
After creation, environments are validated to ensure they're properly initialized:
```python theme={"system"}
def validate_venv(venv_path: str) -> bool:
"""Validate that a virtual environment is properly initialized."""
python_exe = _get_venv_python_exe(venv_path)
return os.path.exists(python_exe) and os.path.isfile(python_exe)
```
Invalid environments are automatically cleaned up and recreated.
## Error Handling
The system provides detailed error messages for common issues:
### Lock Acquisition Timeout
```
Could not acquire lock /path/to/lock within 600s timeout
This may indicate:
- Another process is creating the same environment
- A previous process crashed while holding the lock
- System I/O issues
Try waiting a few minutes and retrying, or check for stale lock files.
```
### Installation Failure
```
Failed to install dependencies using uv.
Error output:
[detailed error message]
This may be due to:
- Network connectivity issues
- Invalid package specifications in requirements.txt
- Incompatible package versions
- Missing system dependencies
Please check your requirements.txt file and try again.
```
### Requirements File Not Found
```
Requirements file not found: /path/to/requirements.txt
Ensure that:
- The file exists at the specified path
- The path is correct and accessible
- You have read permissions for the file
```
## Best Practices
### 1. Pin Package Versions
For reproducibility, pin exact versions in `requirements.txt`:
```
# Good - exact versions
pandas==1.5.0
numpy==1.22.0
scikit-learn==1.2.0
# Less predictable - version ranges
pandas>=1.3.0
numpy>=1.20.0
```
### 2. Use Virtual Environments in Development
Always activate the virtual environment when developing:
```bash theme={"system"}
eval $(mage deps ensure)
source $MAGE_VENV_PATH/bin/activate
```
### 3. Update Dependencies Carefully
When updating dependencies:
1. Update `requirements.txt`
2. Run `mage deps ensure` to create a new environment
3. Test thoroughly before deploying
### 4. Monitor Disk Usage
Virtual environments can consume significant disk space. Use the cleanup command to remove stale environments:
```bash theme={"system"}
# See what would be deleted
mage deps cleanup --dry-run
# Clean up environments unused for 7+ days
mage deps cleanup
# Or configure automatic cleanup in metadata.yaml
```
### 5. Handle Dependency Conflicts
If you encounter dependency conflicts:
1. Try without `--override-global-version` first (uses global constraints)
2. If that fails, retry with `--override-global-version`
3. Review the conflict and adjust `requirements.txt` if needed
## Troubleshooting
### Environment Not Found After Creation
**Symptom**: `mage deps ensure` succeeds but environment doesn't exist
**Solution**: Check for validation failures in logs. The environment may have been created but failed validation and was cleaned up.
### Slow Environment Creation
**Symptom**: Creating environments takes a long time
**Possible causes**:
* Network issues downloading packages
* Many packages to install
* Slow disk I/O
**Solutions**:
* Install `uv` for faster installation: `pip install uv`
* Check network connectivity
* Review `requirements.txt` for unnecessary packages
### Permission Errors
**Symptom**: Cannot create virtual environment due to permission errors
**Solution**: Ensure you have write permissions to the Mage data directory:
```bash theme={"system"}
ls -ld $(python -c "from mage_ai.settings.repo import get_data_dir; print(get_data_dir())")
```
### Stale Lock Files
**Symptom**: Lock acquisition timeout errors persist
**Solution**: Manually remove stale lock files:
```bash theme={"system"}
# Find lock files
find /home/src/mage_data/venvs -name "*.lock"
# Remove stale locks (be careful!)
rm /home/src/mage_data/venvs/*/\.locks/*.lock
```
## Technical Details
### Fingerprint Algorithm
The fingerprint is computed as follows:
```python theme={"system"}
components = [
f"python:{python_version}",
f"requirements:{requirements_content or ''}",
f"uv:{uv_version or 'no-uv'}",
]
fingerprint_input = '\n'.join(components)
fingerprint = hashlib.sha256(fingerprint_input.encode('utf-8')).hexdigest()
```
### Requirements Normalization
Requirements are normalized before fingerprinting:
* Leading/trailing whitespace stripped from each line
* Empty lines removed
* Comments (lines starting with `#`) removed
* Order preserved
This ensures that cosmetic changes to `requirements.txt` don't trigger new environment creation.
### Lock File Semantics
* **.install.lock**: Exclusive lock preventing concurrent installation/creation of environment
* **.activate.lock**: Shared lock indicating environment is actively being used
* **.cleanup.lock**: Global lock preventing simultaneous cleanup operations across all environments
* **global.lock**: Snapshot of globally installed packages at environment creation time (used as constraints)
* **venv.lock**: Snapshot of packages installed in the virtual environment (for diff analysis)
## See Also
* [Python Packages](/development/dependencies/requirements) - Basic requirements.txt usage
* [Custom Files](/development/dependencies/custom-files) - Managing custom Python files
* [CLI Commands](/development/cli-commands) - Complete CLI reference
# Install Python packages using `requirements.txt`
Source: https://docs.mage.ai/development/dependencies/requirements
Mage allows you to easily integrate third-party Python packages into your project and pipelines. This guide will walk you through the process of adding and managing external dependencies.
**Mage Pro: Advanced Dependency Management**
Mage Pro includes an advanced dependency management system with isolated virtual environments, automatic caching, and intelligent package resolution. Features include:
* **Isolated environments** for each unique combination of dependencies
* **Automatic caching** with fingerprint-based environment reuse
* **Fast installation** using `uv` with `pip` fallback
* **Smart compatibility checking** to reuse global packages
* **No cluster restarts required** for dependency updates
* **Easy UI-based installation** via Terminal page
See the [Advanced Dependency Management](/development/dependencies/dependency-management) guide for complete documentation, or the [Quick Start guide](/getting-started/install-dependencies) for UI-based installation.
## Add Python packages to `requirements.txt`
### Location
At the root of your Mage project directory, you'll find a file named `requirements.txt`.
This file is used to specify all the Python packages that your project depends on.
### Path
At the root of your project directory, there is a file named `requirements.txt`. Open this file
and add the Python package you want installed and available throughout your entire project.
For example, if your project is named `demo_project` then the file will be located at
`demo_project/requirements.txt`.
### Adding packages
To add a package, simply open the `requirements.txt` file and list each package on a new line. You can also specify version requirements if needed.
### Example `requirements.txt`:
```
matplotlib==3.5.2
tqdm
```
In this example the `requirements.txt` file will install:
* matplotlib version 3.5.2
* tqdm
## Automatic installation
When installing Mage using Docker, Mage provides two main methods for automatically installing third-party packages at runtime.
These methods utilize the `USER_CODE_PATH` environment variable, which defines your project path (e.g. `/home/src/project_name`),
to locate your `requirements.txt` file and install the packages listed there.
## Using command line flags
If you installed Mage using Docker, the `USER_CODE_PATH` can be set using command line flags. You will need to add these command line flags at every run when running a new Docker container or installing a new version of Mage.
### Setting `USER_CODE_PATH` in Docker
The `USER_CODE_PATH` environment variable tells Mage where your project code is located within the Docker container. This is essential for features like automatic package installation from `requirements.txt`.
#### Docker run command
To set the `USER_CODE_PATH` when running a Mage Docker container, use the following command structure:
```Shell theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
-e USER_CODE_PATH=/home/src/project_name \
mageai/mageai /app/run_app.sh mage start [project_name]
```
Let's break down this command:
* `it`: Runs the container interactively with a pseudo-TTY.
* `p 6789:6789`: Maps port 6789 from the container to your host machine.
* `v $(pwd):/home/src`: Mounts your current directory to `/home/src` in the container.
* `e USER_CODE_PATH=/home/src/project_name`: Sets the `USER_CODE_PATH` environment variable.
* `mageai/mageai`: Specifies the Mage Docker image to use.
* `/app/run_app.sh mage start [project_name]`: The command to run inside the container.
### Important Notes
1. **Persistence**: The `USER_CODE_PATH` setting is not persistent across container restarts or when installing new versions of Mage. You must include these flags each time you run a new Docker container.
2. **Project Name**: Replace `[project_name]` with your actual Mage project name in both the `USER_CODE_PATH` and the start command.
3. **Path Consistency**: Ensure that the path in `USER_CODE_PATH` matches the mount point specified in the `v` flag. In this example, we mount to `/home/src`, so the `USER_CODE_PATH` starts with `/home/src`.
4. **Working Directory**: The `$(pwd)` in the `v` flag assumes you're running the Docker command from your project's parent directory. Adjust this path if you're running from a different location.
For more detailed information on environment variables and their impact on Mage's functionality, please refer to our comprehensive environment variable [documentation](https://docs.mage.ai/development/variables/environment-variables).
## Using Dockerfiles
One important aspect of configuring Mage is setting the `USER_CODE_PATH`, which tells Mage where your project code is located.
While this can be done using command-line flags when running Docker containers, it's also possible to set the `USER_CODE_PATH`
directly in your Dockerfile. This approach can be more convenient for consistent deployments and when building custom images.
```Bash theme={"system"}
FROM mageai/mageai:latest
### Set up project-specific arguments
ARG PROJECT_NAME=[project_name]
ARG MAGE_CODE_PATH=/home/mage_code
ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}
### Set working directory
WORKDIR ${MAGE_CODE_PATH}
### Copy project files
COPY ${PROJECT_NAME} ${PROJECT_NAME}
### Set the command to run on container startup
CMD ["/bin/sh", "-c", "/app/run_app.sh"]
```
Breakdown of the Dockerfile:
1. `FROM mageai/mageai:latest`
* Specifies the base image as the latest version of the official Mage AI image.
2. Project-specific arguments:
```Bash theme={"system"}
ARG PROJECT_NAME=[project_name]
ARG MAGE_CODE_PATH=/home/mage_code
ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}
```
* `PROJECT_NAME`: Should be replaced with your actual project name.
* `MAGE_CODE_PATH`: Sets the base path for Mage code.
* `USER_CODE_PATH`: Constructs the full path to your project, effectively setting the `USER_CODE_PATH` for Mage.
3. `WORKDIR ${MAGE_CODE_PATH}`
* Sets the working directory inside the container to the path specified by `MAGE_CODE_PATH`.
4. `COPY ${PROJECT_NAME} ${PROJECT_NAME}`
* Copies your project files from the host machine into the container.
5. `CMD ["/bin/sh", "-c", "/app/run_app.sh"]`
* Specifies the command to run when the container starts, which launches the Mage application.
To use this Dockerfile:
1. Replace `[project_name]` with your actual project name.
2. Ensure your project files are in a directory named after your project.
3. Build the Docker image using this Dockerfile.
4. Run the container to start your Mage environment.
This Dockerfile demonstrates how to set the `USER_CODE_PATH` using build arguments. By defining it this way, you ensure that Mage will
consistently know where to find your project code within the container, without needing to specify it each time you run the container.
Follow the [Repository setup](https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup) documentation to deploy this
Dockerfile to fully automate the installation of third-party Python packages.
If youโre running Mage usingย pipย orย conda, your local machineโs environment variables are accessible within the running Mage app.
### Prerequisites
1. Ensure that the environment variable `USER_CODE_PATH` is set to your project path, using
the command line flags or the Dockerfile methods.
Example: `/home/src/project_name`
2. Mage uses a script named `run_app.sh` to manage the application startup process.
### Installation process
When you start your Mage server:
1. The `run_app.sh` script checks the `USER_CODE_PATH` environment variable.
2. It locates the `requirements.txt` file within the specified project directory.
3. All packages listed in `requirements.txt` are automatically installed before the server starts.
This ensures that all required packages are available for your project and pipelines.
# Connect a local db in Docker
Source: https://docs.mage.ai/development/docker/connecting-a-database
Here's how you can connect a local database to Mage in Docker for development.
## Overview
When you develop in Docker, you can connect a database to your Mage container. This allows you to use the database in your data pipelines.
A Docker Network is a virtual network that allows containers to communicate with each other.
By creating a network, you can connect database containers to your Mage instances.
Read more about Docker Networks [here](https://docs.docker.com/network/).
The following guides detail various methods for creating databases and connecting them to your Mage project.
We recommend starting with a [docker compose](https://docs.docker.com/compose/) file, as it affords the most flexibility and customizability. You might
choose to use `docker run` instead, as it's a bit simpler.
## Guides
If you're using Mage in docker compose, you can easily connect a PostgreSQL database to your Mage container. To do so, first define a `.env` file with the following variables:
* `POSTGRES_DB`
* `POSTGRES_USER`
* `POSTGRES_PASSWORD`
* `POSTGRES_HOST`
* `PG_HOST_PORT`
Then, add the following to your `docker-compose.yml` file:
```yaml theme={"system"}
version: '3'
services:
mage:
image: mageai/mageai:latest
container_name: magic
depends_on:
- postgres
command: mage start magic
env_file:
- .env
environment:
ENV: dev
POSTGRES_DB: ${POSTGRES_SCHEMA}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_HOST: ${POSTGRES_HOST}
PG_HOST_PORT: ${PG_HOST_PORT}
ports:
- 6789:6789
volumes:
- .:/home/src/
restart: on-failure:5
postgres:
image: postgres:14
restart: on-failure
container_name: postgres-magic
build:
context: ./db
env_file:
- .env
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
ports:
- "${PG_HOST_PORT}:5432"
```
The following steps will create a PostgreSQL database container and connect it to a Mage container.
```bash theme={"system"}
docker network create mage-app
```
```bash theme={"system"}
docker run --network mage-app --network-alias postgres_db \
-it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
-e POSTGRES_USER= -e POSTGRES_PASSWORD= \
-e POSTGRES_DB= -e PG_DATA=/var/lib/postgresql/data/pgdata \
postgres:13-alpine3.17 postgres
```
```bash theme={"system"}
docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
-e MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://:@postgres_db:5432/ \
mageai/mageai \
/app/run_app.sh mage start [project_name]
```
To use a specific schema in the PostgreSQL database, the connection string could contain the schema name as a parameter:
```bash theme={"system"}
docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
-e MAGE_DATABASE_CONNECTION_URL="postgresql+psycopg2://:@postgres_db:5432/?options=-c%%20search_path%%3D" \
mageai/mageai \
/app/run_app.sh mage start [project_name]
```
```bash theme={"system"}
docker network create mage-app
```
```bash theme={"system"}
docker run --network mage-app --network-alias mssql_db \
-it -p 1433:1433 -e "ACCEPT_EULA=Y" \
-e "MSSQL_SA_PASSWORD=test_Password123" \
mcr.microsoft.com/mssql/server:2022-latest
```
```bash theme={"system"}
docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
-e MAGE_DATABASE_CONNECTION_URL="mssql+pyodbc://?odbc_connect=DRIVER={ODBC Driver 18 for SQL Server};SERVER=mssql_db;DATABASE=master;UID=SA;PWD=test_Password123;ENCRYPT=yes;TrustServerCertificate=yes;" \
mageai/mageai \
/app/run_app.sh mage start [project_name]
```
Here are the parameters we're using in that command:
* `DRIVER={ODBC Driver 18 for SQL Server}`: a formatted string with a version number matching that of the SQL server
* `SERVER=database_server`: the name or IP address of the database server
* `DATABASE=database`: the name of the database
* `UID=user`: the user ID for the database
* `PWD=password`: the password for the user
* `ENCRYPT=yes`: whether to use encryption
* `TrustServerCertificate=yes`: whether to trust the server certificate
# Run docker image as non-root user
Source: https://docs.mage.ai/development/docker/non-root-user
Run Mage docker image as non-root user to restrict access to the system
By default, the official Mage docker images in DockerHub are using the root users. For production deployment,
it's recommended to use non-root users in the docker image to restrict access to the system for better security.
Here is the Dockerfile template that you can use to build your custom docker image that runs Mage as non-root user.
```dockerfile theme={"system"}
ARG VERSION
FROM mageai/mageai:$VERSION
# Add non-root user for Mage service
RUN adduser --disabled-password --gecos '' mage && adduser mage mage
# Grant the user permissions to the Mage related directories
RUN mkdir /home/src/mage_data; chown -R mage /home/src/mage_data
RUN mkdir /home/src/default_repo; chown -R mage /home/src/default_repo
RUN pip3 install -r path/to/requirements.txt
# Set the Mage user
USER mage
ENV PYTHONPATH="${PYTHONPATH}:/home/mage/.local/lib/python3.10/site-packages"
WORKDIR /home/src
CMD ["/bin/sh", "-c", "/app/run_app.sh"]
```
# Mage Pro CI/CD overview
Source: https://docs.mage.ai/development/git/pro-overview
Deploy your code in seconds!
Mage Pro provides integrated CI/CD capabilities to streamline your data pipeline development and deployment workflows. This section covers:
* **Version Control Application** - Built-in Git terminal with authentication and real-time collaboration features
* **Deployment Guide** - Step-by-step instructions for connecting repositories and configuring deployment stages
* **Git Operations** - Standard version control commands and troubleshooting guidance
* **Multi-stage Deployments** - Production-ready deployment workflows with rollback capabilities
Supported Git providers include GitHub and GitLab, with OAuth and SSH authentication options available for secure repository integration.
# Get started with Git
Source: https://docs.mage.ai/development/git/setting-up-git
There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.
# Mage Pro
Unlock seamless version control and effortless deployments with Mage Pro!
Check out this [guide](/guides/data-sync/version-control-guide) to experience a more intuitive, user-friendly way to manage and deploy your code.
# Open source
## Choose your adventure
There three different ways to use Git in Mage and *four* ways to version control your filesโ we'll discussing each of them. We note that our methods for using Git are mutually exclusive.
If you're simply looking for version control, you're in luck! It ships out-of-the-box with Mage. That means you can *always* restore previous versions of your files, even if you haven't configured any version control system. Check out [this page](/getting-started/file-versions) for more info.
## Methods of using Git in Mage
| Recommended | Use | Recommendation | Ease of use | Functionality |
| ----------- | ---------------------------------------- | --------------------------------------------------------------- | ----------- | ---------------------- |
| โ | Develop/deploy Mage using GitHub | [Git application](/guides/data-sync/version-control) | Moderate | Full w/ OAuth |
| โ | Deploy Mage using Git with any provider | [Integration w/ Git Sync](/guides/data-sync/git-sync) | Advanced | Full w/ SSH/HTTPS Auth |
| โ ๏ธ | Develop Mage using Git with any provider | [Integration *w/o* Git Sync](/guides/data-sync/git-integration) | Advanced | Full w/ SSH/HTTPS Auth |
Git Sync is particularly useful in production instances, where other Git methods can be used more meaningfully in development instances. Since Git sync is a one-way sync, it's not very useful for collaboration, since changes get overridden. However, this makes it great for a production Mage instance, where verified changes should only be accepted from development environments.
A common pattern is to use development instances to edit and push (Git integration), then have production instances use Git Sync or CI/CD.
# IO Config
Source: https://docs.mage.ai/development/io_config
The `io_config` YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.
Here's a sample `io_config` file. You can learn more about variable interpolation and referencing secrets via our variables overview [here](/development/variables/overview).
> โน๏ธ To learn how to set up your `io_config.yaml` file, check the [setup guide](/development/io_config_setup).
> The `io_config.yaml` file should be placed under your project folder.
```yaml theme={"system"}
version: 0.1.1
default:
# Default profile created for data IO access.
# Add your credentials for the source you use, and delete the rest.
# AWS
AWS_ACCESS_KEY_ID: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
AWS_SECRET_ACCESS_KEY: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
AWS_SESSION_TOKEN: session_token (Used to generate Redshift credentials)
AWS_REGION: region
# Azure
AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
# ClickHouse
CLICKHOUSE_DATABASE: default
CLICKHOUSE_HOST: host.docker.internal
CLICKHOUSE_INTERFACE: http
CLICKHOUSE_PASSWORD: null
CLICKHOUSE_PORT: 8123
CLICKHOUSE_USERNAME: null
# Druid
DRUID_HOST: hostname
DRUID_PASSWORD: password
DRUID_PATH: /druid/v2/sql/
DRUID_PORT: 8082
DRUID_SCHEME: http
DRUID_USER: user
# Google
GOOGLE_SERVICE_ACC_KEY:
type: service_account
project_id: project-id
private_key_id: key-id
private_key: "-----BEGIN PRIVATE KEY-----\nyour_private_key\n-----END_PRIVATE_KEY"
client_email: your_service_account_email
auth_uri: "https://accounts.google.com/o/oauth2/auth"
token_uri: "https://accounts.google.com/o/oauth2/token"
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/your_service_account_email"
GOOGLE_SERVICE_ACC_KEY_FILEPATH: "/path/to/your/service/account/key.json"
GOOGLE_LOCATION: US # Optional
# MongoDB
MONGODB_DATABASE: database
MONGODB_HOST: host
MONGODB_PASSWORD: password
MONGODB_PORT: 27017
MONGODB_COLLECTION: collection
MONGODB_USER: user
# MSSQL
MSSQL_DATABASE: database
MSSQL_SCHEMA: schema
MSSQL_DRIVER: "ODBC Driver 18 for SQL Server"
MSSQL_HOST: host
MSSQL_PASSWORD: password
MSSQL_PORT: 1433
MSSQL_USER: SA
# MySQL
MYSQL_DATABASE: database
MYSQL_HOST: host
MYSQL_PASSWORD: password
MYSQL_PORT: 3306
MYSQL_USER: root
# Pinot
PINOT_HOST: hostname
PINOT_PASSWORD: password
PINOT_PATH: /query/sql
PINOT_PORT: 8000
PINOT_SCHEME: http
PINOT_USER: user
# PostgreSQL
POSTGRES_CONNECT_TIMEOUT: 10
POSTGRES_DBNAME: postgres
POSTGRES_SCHEMA: public # Optional
POSTGRES_USER: username
POSTGRES_PASSWORD: password
POSTGRES_HOST: hostname
POSTGRES_PORT: 5432
# Redshift
REDSHIFT_SCHEMA: public # Optional
REDSHIFT_DBNAME: redshift_db_name
REDSHIFT_HOST: redshift_cluster_id.identifier.region.redshift.amazonaws.com
REDSHIFT_PORT: 5439
REDSHIFT_TEMP_CRED_USER: temp_username
REDSHIFT_TEMP_CRED_PASSWORD: temp_password
REDSHIFT_DBUSER: redshift_db_user
REDSHIFT_CLUSTER_ID: redshift_cluster_id
REDSHIFT_IAM_PROFILE: default
# Snowflake
SNOWFLAKE_USER: username
SNOWFLAKE_PASSWORD: password
SNOWFLAKE_ACCOUNT: account_id.region
SNOWFLAKE_DEFAULT_WH: null # Optional default warehouse
SNOWFLAKE_DEFAULT_DB: null # Optional default database
SNOWFLAKE_DEFAULT_SCHEMA: null # Optional default schema
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null # Optional private key passphrase
SNOWFLAKE_PRIVATE_KEY_PATH: null # Optional private key path
SNOWFLAKE_ROLE: null # Optional role name
SNOWFLAKE_TIMEOUT: null # Optional timeout in seconds
# Trino
trino:
catalog: postgresql # Change this to the catalog of your choice
host: 127.0.0.1
http_headers:
X-Something: 'mage=power'
http_scheme: http
password: mage1337 # Optional
port: 8080
schema: core_data
session_properties: # Optional
acc01.optimize_locality_enabled: false
optimize_hash_generation: true
source: trino-cli # Optional
user: admin
verify: /path/to/your/ca.crt # Optional
data_type_properties:
timestamp_precision: 3 # Optional. Set TIMESTAMP(n) precision
```
# Unified config
Mage Pro provides a unified framework for configuring connections using `io_config.yaml` across different types of pipelines:
* **Batch pipeline** (Python block, SQL block)
* **Data integration pipeline**
* **Streaming pipeline**
Each pipeline reads from the same io\_config.yaml file, which allows you to centralize connection information and switch between pipeline types with minimal configuration changes.
## Batch pipeline
### Example: Python block (Postgres)
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
@data_loader
def load_data_from_postgres(**kwargs):
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
### Example: SQL block
Just select the desired IO Config profile in the UI (under the "Profile" dropdown).
The profile must exist in your io\_config.yaml and contain valid credentials for the source or destination.
## Data integration pipeline
Data integration pipelines use source and destination blocks.
These blocks can reference connection credentials from `io_config.yaml` via the `io_config_profile` key.
The following sources and destinations systems currently support this configuration method:
* BigQuery
* Microsolft SQL Server
* MySQL
* Postgres
* Redshift
* Snowflake
### Example source config (Postgres)
```yaml theme={"system"}
io_config_profile: default # profile in the io_config.yaml
```
### Example destination config (Postgres)
```yaml theme={"system"}
io_config_profile: default # profile in the io_config.yaml
schema: test # Override the schema
```
With this setup, you donโt need to hard-code connection details in your block configsโjust use the profile name.
## Streaming pipeline
Streaming pipelines also support `io_config.yaml` for setting up destinations.
### Example: Destination block config (BigQuery)
```yaml theme={"system"}
connector_type: bigquery
profile: default # profile in the io_config.yaml
config:
table_id: table_id
if_exists: "append"
```
The `profile` field points to the profile in your `io_config.yaml`, and additional block-specific settings can still be defined under config.
# IO Config Setup
Source: https://docs.mage.ai/development/io_config_setup
Learn how to set up and configure the io_config file for your Mage project
# IO Config Setup Guide
The `io_config` file is a crucial configuration file in Mage that stores credentials and connection information for accessing various data sources. This guide will help you understand where to store it and how to create it from scratch.
## File Location
The `io_config` file should be stored in your Mage project's root directory. The default path is:
```
your_mage_project/
โโโ io_config.yaml
```
For detailed information about project structure and file organization, see the [Project Structure documentation](/design/abstractions/project-structure#project-structure).
## Creating a New io\_config File
### 1. Basic Structure
Create a new file named `io_config.yaml` in your project's root directory (e.g. `/home/src/your_mage_project`) with the following basic structure:
```yaml theme={"system"}
version: 0.1.1 # Optional but recommended
default:
# Your default profile configuration here
```
### 2. Configuration Formats
Mage supports two formats for the `io_config` file:
#### Standard Format (Recommended)
```yaml theme={"system"}
version: 0.1.1
default:
# PostgreSQL configuration
POSTGRES_DBNAME: your_database
POSTGRES_USER: your_username
POSTGRES_PASSWORD: "{{ env_var('POSTGRES_PASSWORD') }}"
POSTGRES_HOST: your_host
POSTGRES_PORT: 5432
# Snowflake configuration
SNOWFLAKE_USER: your_username
SNOWFLAKE_PASSWORD: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
SNOWFLAKE_ACCOUNT: your_account
SNOWFLAKE_DEFAULT_WH: your_warehouse
SNOWFLAKE_DEFAULT_DB: your_database
SNOWFLAKE_DEFAULT_SCHEMA: your_schema
# AWS configuration
AWS_ACCESS_KEY_ID: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
AWS_SECRET_ACCESS_KEY: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
AWS_REGION: your_region
```
#### Legacy Format (Verbose)
```yaml theme={"system"}
default:
PostgreSQL:
database: your_database
user: your_username
password: your_password
host: your_host
port: 5432
AWS:
region: your_region
Redshift:
database: your_redshift_database
port: 5439
```
### 3. Using Variables and Secrets
For security, it's recommended to use environment variables and other secret management systems for sensitive information. Mage provides several variable syntax options:
#### Variable Syntax
| Syntax | Description |
| ------------------------------------ | ----------------------------------------------------------- |
| `{{ env_var('secret') }}` | Get secret from environment variables |
| `{{ mage_secret_var('secret') }}` | Get secret from Mage secrets |
| `{{ aws_secret_var('secret') }}` | Get secret from AWS Secrets Manager |
| `{{ azure_secret_var('secret') }}` | Get secret from Azure Key Vault |
| `{{ gcp_secret_var('secret') }}` | Get secret from Google Cloud Secret Manager (Mage Pro only) |
| `{{ doppler_secret_var('secret') }}` | Get secret from Doppler |
| `{{ json_value(json_obj, 'key') }}` | Extract value from a JSON string |
| `{{ file('path/to/file.txt') }}` | Load contents of a local file (Mage Pro only) |
#### Example Configurations
**Environment Variables:**
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_PASSWORD: "{{ env_var('POSTGRES_PASSWORD') }}"
SNOWFLAKE_PASSWORD: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
```
**Mage Secrets:**
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_PASSWORD: "{{ mage_secret_var('postgres_password') }}"
SNOWFLAKE_PASSWORD: "{{ mage_secret_var('snowflake_password') }}"
```
**AWS Secrets Manager:**
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_PASSWORD: "{{ aws_secret_var('prod/postgres/password') }}"
SNOWFLAKE_PASSWORD: "{{ aws_secret_var('prod/snowflake/password') }}"
```
**Azure Key Vault:**
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_PASSWORD: "{{ azure_secret_var('postgres-password') }}"
SNOWFLAKE_PASSWORD: "{{ azure_secret_var('snowflake-password') }}"
```
**Google Cloud Secret Manager (Mage Pro only):**
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_PASSWORD: "{{ gcp_secret_var('postgres-password') }}"
SNOWFLAKE_PASSWORD: "{{ gcp_secret_var('snowflake-password') }}"
```
**Doppler:**
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_PASSWORD: "{{ doppler_secret_var('POSTGRES_PASSWORD') }}"
SNOWFLAKE_PASSWORD: "{{ doppler_secret_var('SNOWFLAKE_PASSWORD') }}"
```
To use Doppler, configure the Mage runtime with `DOPPLER_TOKEN`, `DOPPLER_PROJECT`, and `DOPPLER_CONFIG`. Mage caches Doppler secrets in memory for `DOPPLER_SECRETS_CACHE_TTL_SECONDS` seconds, defaulting to `10`. Set the TTL to `0` to refresh on every lookup. Secret changes are picked up on the next config render after the TTL expires; already-open connections may continue using the previous value until recreated.
**File Content (Mage Pro only):**
```yaml theme={"system"}
version: 0.1.1
default:
# Load from local file
POSTGRES_PASSWORD: "{{ file('secrets/postgres_password.txt') }}"
# Load JSON field from file
SNOWFLAKE_PASSWORD: "{{ json_value(file('config/database.json'), 'snowflake_password') }}"
```
**JSON Value Extraction:**
```yaml theme={"system"}
version: 0.1.1
default:
# Extract specific field from JSON stored in environment variable
POSTGRES_PASSWORD: "{{ json_value(env_var('DB_CONFIG_JSON'), 'postgres_password') }}"
# Extract from JSON stored in Mage secret
SNOWFLAKE_PASSWORD: "{{ json_value(mage_secret_var('database_config'), 'snowflake_password') }}"
```
### 4. Multiple Profiles
You can create multiple profiles for different environments or use cases:
```yaml theme={"system"}
version: 0.1.1
default:
# Default profile configuration
POSTGRES_DBNAME: production_db
development:
# Development environment configuration
POSTGRES_DBNAME: development_db
staging:
# Staging environment configuration
POSTGRES_DBNAME: staging_db
```
## Using io\_config in Your Code
### SQL Block Example
In SQL blocks, you can select the desired IO Config profile from the UI using the "Profile" dropdown menu. For detailed information about configuring SQL blocks, see the [SQL Blocks documentation](/guides/blocks/sql-blocks#configure-sql-block).
### Python Block Examples
You can use `io_config` in your Python code with different profile selection strategies:
**Basic Profile Usage:**
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
@data_loader
def load_data_from_postgres(**kwargs):
query = 'SELECT * FROM your_table'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default' # or any other profile name
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
**Dynamic Profile Selection:**
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
@data_loader
def load_data_from_postgres(**kwargs):
query = 'SELECT * FROM your_table'
config_path = path.join(get_repo_path(), 'io_config.yaml')
# Get profile from pipeline variables or use default
config_profile = kwargs.get('profile', 'default')
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
**Environment-based Profile Selection:**
```python theme={"system"}
import os
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
@data_loader
def load_data(**kwargs):
query = 'SELECT * FROM your_table'
config_path = path.join(get_repo_path(), 'io_config.yaml')
# Automatically select profile based on environment
environment = os.getenv('ENV', 'development')
config_profile = environment # 'development', 'staging', 'production'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
## Databases using io\_config.yaml
The following database and storage integrations use `io_config.yaml` for credentials and connection settings. See each doc for the exact keys and examples:
**Relational & SQL databases**
* [PostgreSQL](/integrations/databases/PostgreSQL)
* [MySQL](/integrations/databases/MySQL)
* [Microsoft SQL Server](/integrations/databases/MicrosoftSQLServer)
* [SQLite](/integrations/databases/SQLite)
* [Redshift](/integrations/databases/Redshift)
* [Trino](/integrations/databases/trino)
* [OracleDB](/integrations/databases/OracleDB)
* [ClickHouse](/integrations/databases/ClickHouse)
* [DuckDB](/integrations/databases/DuckDB)
* [MotherDuck](/integrations/databases/MotherDuck)
**Cloud data warehouses & analytics**
* [BigQuery](/integrations/databases/BigQuery)
* [Snowflake](/integrations/databases/Snowflake)
* [Databricks](/integrations/databases/Databricks)
* [Athena](/integrations/databases/Athena)
* [Microsoft Fabric Warehouse](/integrations/databases/MicrosoftFabricWarehouse)
**Storage & data lakes**
* [S3](/integrations/databases/S3)
* [Google Cloud Storage](/integrations/databases/GoogleCloudStorage)
* [Azure Blob Storage](/integrations/databases/AzureBlobStorage)
* [Azure Data Lake Storage Gen2](/integrations/databases/AzureDataLakeStorage)
* [Iceberg](/integrations/databases/Iceberg)
**Other databases & services**
* [MongoDB](/integrations/databases/MongoDB)
* [Druid](/integrations/databases/Druid)
* [Pinot](/integrations/databases/Pinot)
* [Google Sheets](/integrations/databases/GoogleSheets)
* [Spark](/integrations/databases/Spark)
* [Power BI](/integrations/databases/PowerBI)
**Vector stores & search**
* [Chroma](/integrations/databases/Chroma)
* [Qdrant](/integrations/databases/Qdrant)
* [Weaviate](/integrations/databases/Weaviate)
* [Algolia](/integrations/databases/Algolia)
## Best Practices
1. **Security**:
* Never commit sensitive credentials directly in the `io_config` file
* Use environment variables or secret variables for sensitive information
* Consider using a secrets management service for production environments
2. **Organization**:
* Use meaningful profile names
* Group related configurations together
* Document any non-standard configurations
* Use the standard format for new configurations
3. **Version Control**:
* Add `io_config.yaml` to your `.gitignore` file
* Provide a template file (`io_config.yaml.template`) with dummy values
* Document the required environment variables
## Troubleshooting
### Common Issues & Solutions
**1. "FileNotFoundError" or "No such file or directory"**
* Make sure your `io_config.yaml` file exists in your project root (e.g., `/home/src/your_mage_project/io_config.yaml`).
* If running in a different environment (e.g., Docker), confirm the file is mounted and accessible.
**2. "KeyError" or "Missing profile"**
* Double-check that the profile name you are referencing in your code (e.g., `'default'`, `'staging'`) exists in your `io_config.yaml`.
* Profile names are case-sensitive.
**3. "Missing required field" or "NoneType" errors**
* Ensure all required fields for your data source/destination are present in the selected profile.
**4. "Environment variable not set"**
* If you use `{{ env_var('VAR_NAME') }}` in your config, make sure the environment variable is set in deployment environment.
* You can check this by running `echo $VAR_NAME` in Mage terminal.
**5. YAML syntax errors**
* Use a YAML linter or validator to check for indentation or formatting issues.
* Strings containing special characters (like `:` or `{}`) should be quoted.
**6. "Invalid credentials" or authentication failures**
* Double-check your credentials and permissions.
* For cloud services, ensure your service account or IAM user has the necessary roles.
**7. "Cannot import ConfigFileLoader" or similar import errors**
* Make sure you are importing from `mage_ai.io.config` (not `mage_ai.io.io_config`).
* Example:
```python theme={"system"}
from mage_ai.io.config import ConfigFileLoader
```
**8. "Profile not selected" in SQL blocks**
* In the Mage UI, select the correct profile from the "Profile" dropdown before running your SQL block.
If you continue to have issues, ask for help in the [Mage community Slack](https://mage.ai/chat).
## Additional Resources
* [Variables Overview](/development/variables/overview)
* [Data Integration Guide](/data-integrations/overview)
* [Pipeline Configuration](/design/data-pipeline-management)
# Initial start screen
Source: https://docs.mage.ai/development/project/landing-page
Choose which page opens first when you launch Mage.
By default, when you enter a workspace, Mage takes you to the
**Pipelines page** (`/pipelines`). You can change which page opens
first by picking a different option in Workspace Preferences.
## Available start screens
| Option | Route |
| ------------------- | ---------------- |
| Pipelines (default) | `/pipelines` |
| Monitoring | `/overview` |
| Runs | `/pipeline-runs` |
| Triggers | `/triggers` |
## Steps to change your start screen
1. Open the `Settings` page (with the wrench icon) from the left
vertical navigation panel and select **Workspace** โ **Preferences**.
2. Scroll to the **Workspace** section.
3. In the **Initial start screen** dropdown, pick the page you want to
land on when you next open the workspace.
The change takes effect the next time you open the workspace home
URL (`/`) โ for example by clicking the Mage logo or refreshing.
You can change it back at any time.
This preference is saved in your **browser's local storage**, not in
your project's `metadata.yaml` file. As a result:
* It applies only to the browser where you set it. Other browsers,
other devices, and other team members will keep their own
preference (or the default Pipelines page).
* Clearing your browser data resets it to the default.
* Open tabs do not sync with each other until they are reloaded.
Instance managers always land on the `/manage` page regardless of
this setting.
# Local timezone
Source: https://docs.mage.ai/development/project/local-timezone
Display timestamps throughout the app in your local timezone.
You can display dates on various pages (e.g. the Pipelines list and
Triggers/Pipeline runs/Block runs pages) in your local timezone, as
well as configure the start date for triggers and backfills in local
time. In the Version control terminal, this setting also controls new
terminal session timezone behavior: sessions use UTC by default, and use your
browser's local IANA timezone for prompt timestamps and the `TZ` environment
variable when `Display local timezone` is enabled.
## Steps to enable local timezone setting
1. Go to your project's `Settings` page (with the wrench icon) from the
left vertical navigation panel.
2. Under the "Features" section, click the toggle to turn the `Display local timezone`
setting on.
3. Then click the blue `Save project settings` button to save and persist the
new settings. If you do not click this button, you will lose any changes when
leaving the page.
* Alternatively (starting with v0.9.48), you can simply click on the time in the header
and toggle on the `Display local timezone` setting from the dropdown menu. This may
require a page refresh for the timestamps to change to UTC or local time.
Please note that certain pages (e.g. a pipeline's Monitor page) or components
(e.g. Pipeline run bar charts) may still be in UTC time. These UTC dates may
only display the date (e.g. 2023-09-01) without any time. Dates in local time
will have a timezone offset (e.g. -07:00) in the timestamp (e.g.
2023-09-01T12:30:45-07:00).
# Project metadata.yaml
Source: https://docs.mage.ai/development/project/metadata
Reference for configuring project-level settings in metadata.yaml.
## Overview
Each Mage project has a `metadata.yaml` file at the root of the project
directory. The file controls storage locations, feature flags, compute
configuration, and defaults that apply to all pipelines in the project.
```
your_project/
โโ pipelines/
โโ .../
โโ io_config.yaml
โโ metadata.yaml <-- project metadata lives here
โโ requirements.txt
```
Use the fields below to customize how your project behaves. For
environment-specific overrides, add an `overrides` section (see
`/extensibility/env-config/project`).
## Sample `metadata.yaml`
```yaml theme={"system"}
project_type: standalone
variables_dir: ~/.mage_data
remote_variables_dir: s3://bucket/path_prefix
help_improve_mage: true
notification_config:
alert_on:
- trigger_failure
slack_config:
webhook_url: "{{ env_var('MAGE_SLACK_WEBHOOK_URL') }}"
pipelines:
settings:
triggers:
save_in_code_automatically: true
project_uuid: 123456789abcdefg
```
## Top-level fields
Project layout. Common options: `standalone` (default), `main`, or `sub`.
Optional cluster type used by the project (e.g., `k8s`, `docker`). Used by Mage Pro
when running managed workspaces.
Unique identifier for the project. Generated when the project is created.
Local path for storing pipeline variables and outputs. Paths are relative to
the project root unless an absolute path is provided. Default to `/home/src/mage_data`.
Remote path (e.g., `s3://bucket/prefix`) for storing variables in object
storage instead of the local filesystem.
How long variables are retained (e.g., `90d`).
Default project metadata applied to new workspaces created from this project.
Mage Pro workspaces only.
Default workspace configuration applied across environments (e.g., k8s
defaults when running on Kubernetes). Mage Pro workspaces only.
Allows Mage to collect limited telemetry to improve the product.
Feature flags for the project (e.g., `command_center`, `dbt_v2`,
`automatic_kernel_cleanup`). Keys are booleans.
Whether new or updated triggers are automatically written to code.
Environment-specific overrides for any top-level field. Mage Pro only. See
`/extensibility/env-config/project`.
## Compute and execution
Amazon EMR cluster settings (instance types, security groups, key pair, etc.).
Spark configuration shared across pipelines (e.g., `spark_master`,
`executor_env`, `spark_jars`, `use_custom_session`).
Project-level defaults for AWS ECS execution.
Project-level defaults for GCP Cloud Run execution.
Project-level defaults for Azure Container Instances execution.
Kubernetes executor defaults applied to pipelines and blocks.
Limits and concurrency settings at the project level.
Configuration for queueing pipeline runs.
State store configuration used by pipelines.
## Observability and safety
Project-level alerting configuration (alert types, Slack/Teams webhooks, etc.).
Configure log destinations and formats for the project.
Default retry behavior applied at the project level.
Global AI-related settings (e.g., model providers).
Retrieval-augmented generation settings shared across pipelines.
Project-level OpenAI API key used by AI features when applicable.
Optional LDAP connection settings for authentication.
# Project setup
Source: https://docs.mage.ai/development/project/setup
How to set up a folder containing your Mage project code and other files.
## Parent folder
Create a parent folder that will contain your Mage project code:
```bash theme={"system"}
mkdir data_monorepo
```
Change directory into the parent folder:
```bash theme={"system"}
cd data_monorepo
```
## Create a Dockerfile
Create an empty Dockerfile:
```bash theme={"system"}
echo "" > Dockerfile
```
Paste the contents from this [Dockerfile](https://github.com/mage-ai/docker/blob/master/Dockerfile)
into the Dockerfile you just created.
## Create a .gitignore file
Create an empty .gitignore file:
```bash theme={"system"}
echo "" > .gitignore
```
Paste the following contents into the .gitignore file you just created:
```
.DS_Store
.file_versions
.gitkeep
.log
.logs/
.mage_temp_profiles
.preferences.yaml
.variables/
__pycache__/
docker-compose.override.yml
logs/
mage-ai.db
mage_data/
secrets/
```
## Create Mage project
Follow these [instructions](/getting-started/setup) to create a Mage project in your
current directory. If you named your project `demo_project`, your current folder structure
will look like this:
```
data_monorepo/
|-- demo_project/
|-- .gitignore
|-- Dockerfile
```
# Configuring Mage settings backend
Source: https://docs.mage.ai/development/settings-backend
## Overview
Available in Mage versions >= 0.9.66
Not all Mage settings can be read through a settings backend. You can find a list of settings
that can be read through a settings backend [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/settings/keys/__init__.py).
When any settings backend is enabled, Mage will first attempt to read the settings value
from the settings backend before falling back to environment variables.
By default, many of Mage's configurable settings are read from environment variables. This may make sense
for more settings, but there are some settings that are too sensitive to be stored as environment
variables.
You can configure a settings backend to read settings from a different source. To set this up, you will
need to add a `settings_backend` section to your project's `metadata.yaml` file.
```yaml theme={"system"}
settings_backend:
backend_type: ...
```
Only the `backend_type` field is required for all settings backends. The other fields will be passed
to the backend as keyword arguments when initializing the backend.
See below sections for the different types of settings backends that are available.
## AWS Secrets Manager
When the AWS Secrets Manager settings backend is enabled, Mage will read the value from the
AWS secret name: `{prefix}{settings_name}`. For example if the setting is `OKTA_CLIENT_SECRET`
and the prefix is `default_repo/settings/`, Mage will fetch the value for `default_repo/settings/OKTA_CLIENT_SECRET`.
```yaml theme={"system"}
settings_backend:
backend_type: aws_secrets_manager
prefix: default_repo/settings/ # optional
use_cache: true # optional
cache_config: # optional
max_cache_size: 1024
exception_retry_delay_base: 1
exception_retry_growth_factor: 2
exception_retry_delay_max: 3600
default_version_stage: "AWSCURRENT"
secret_refresh_interval: 3600 # in seconds
```
Optional fields:
* `prefix`: A prefix to add to the settings name when fetching the value from AWS Secrets Manager.
* `use_cache`: Whether to use a cache for the settings backend. Defaults to `false`.
* `cache_config`: configuration options for the cache if `use_cache` is `true`.
* `max_cache_size` (int): The maximum number of secrets to cache.
* `exception_retry_delay_base` (int): The number of seconds to wait after an exception is encountered and before retrying the request.
* `exception_retry_growth_factor` (int): The growth factor to use for calculating the wait time between retries of failed requests.
* `exception_retry_delay_max` (int): The maximum amount of time in seconds to wait between failed requests.
* `default_version_stage` (str): The default version stage to request.
* `secret_refresh_interval` (int): The number of seconds to wait between refreshing cached secret information.
# Troubleshooting
Source: https://docs.mage.ai/development/troubleshooting
Debugging common issues in Mage and other frequently asked questions.
## Pipeline edit page freezes browser
The data output of a block contains too much information for the browser to render.
To temporarily fix this, open the `/terminal` page in Mage.
Then type the following command press enter to run it:
```bash theme={"system"}
rm -rf mage_data/[project_name]/pipelines/[pipeline_name]/.variables
```
Change `[project_name]` to your Mage project name and change `[pipeline_name]` to the name of the pipeline that is freezing the browser.
The above command will remove the data output files for each block in your pipeline.
***
## Canโt connect to local database from within Mage via Docker
If youโre running Mage via Docker and youโre trying to connect to a local database outside of
Docker, you must use the following `host` or `IP` value when trying to connect to it:
`host.docker.internal`.
***
## Database migration fails during startup
If Mage fails to start because an Alembic or SQLAlchemy database migration failed, run `mage db diagnose` to inspect the current migration state and get the next recommended command.
See [Troubleshoot database migration failures](/development/databases/default#troubleshoot-database-migration-failures) for `mage db history`, `mage db schema`, `mage db migrate`, `mage db rollback`, and `mage db stamp` recovery steps.
***
## Unpredictable Columns in an Upstream Dataframe
Sometimes, upstream data is out of our control. If we're expecting a column in a dataframe and writing SQL against it, we might see a `SQL compilation` error: `invalid identifier.`
Thankfully, [Mage Community](https://www.mage.ai/chat) member [Josh Pachner](https://mageai.slack.com/team/U05B9DLK0N7) has a solution.
Say you have an upstream block that looks like:
```SQL theme={"system"}
SELECT
df."col_1",
df."col_2",
df."col_3",
FROM {{ df }} df
```
Where the underlying `df` might all three columns... or just a subset. For example, `col_1` isn't always present.
For cases where expected columns *might* not in our dataframe, we can add a transformer block upstream of an exporter:
```python theme={"system"}
# headers = [cols we need in our dataframe]
# df = our DataFrame()
for head in headers:
if head not in df.columns:
df[head] = None
```
This is especially helpful in databases like Snowflake, which don't have an `iferror()` function to check for missing columns.
***
## `InvalidToken` error when accessing data in S3
Accessing data located in S3, you may see an error like this:
```text theme={"system"}
An error occurred (InvalidToken) when calling the GetObject operation:
The provided token is malformed or otherwise invalid.
```
This [Stack Overflow](https://stackoverflow.com/questions/54837248/an-error-occurred-invalidtoken-when-calling-the-listbuckets-operation-the-pr) article details the issue. Removing the `AWS_SESSION_TOKEN` line from the `io_config.yaml` should resolve the problem.
## Mage doesn't load properly when running locally
When attempting to connect to Mage with a port *other* than `6789`, the browser may not render the page correctly with `localhost:port`.
Instead, use `127.0.0.1:port` to connect to Mage.
## Using credential files in Docker
If you have files containing secret credentials (e.g. GCP credentials JSON files), youโll need to mount them as a volume so that those files appear in your running Docker container.
Use the `-v` command in your `docker run` command to mount the directory containing your files:
```bash theme={"system"}
docker run -it -p 6789:6789
-v /Users/dragon/secrets:/home/secrets \
-v $(pwd):/home/src mageai/mageai \
/app/run_app.sh mage start [project_name]
```
or, using `docker compose`, add a volume mount to your Mage service. The following is an example of mounting the `~/.aws` directory for authenticating via the AWS CLI:
```yaml theme={"system"}
services:
mage:
image: mageai/mageai:latest
...
volumes:
- ~/.aws:/root/.aws
```
## "Run block" button doesn't work when running Mage with Nginx
If Mage terminal or "Run block" button doesn't work, it usually means there's a WebSocket connection issue. If Mage server is deployed with Nginx, you can add the below config to Nginx config to enable WebSocket connection.
```
location / {
proxy_pass http://127.0.0.1:6789;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
}
```
## Block gets stuck in running status
If your block gets stuck in running status in the Mage editor, you can try the following things:
* [Restart the kernel](/getting-started/kernels#restart-kernel) and refresh the page.
* Check whether the CPU and memory usage of the instance is too high. Increase the CPU and memory if the usage is above 50%.
## Server fails to start or keeps restarting due to healthcheck failures
When deploying your Mage server on Cloud (Kubernetes, AWS, GCP, Azure, etc.) with Terraform scripts or Helm chart, there're health checks set up for the
cloud services. If you notice the server keeps restarting, one possible reason is the health check keeps failing due to timeout. You can try increasing the timeout or the initial delay of the health checks to resolve the issue.
# Updating Mage
Source: https://docs.mage.ai/development/updating-mage
How to get the latest and greatest from the Mage team.
```bash theme={"system"}
docker pull mageai/mageai:latest
```
1. Are you a technologist? *Yes, you are from the future*
2. Do you live on the edge of the galaxy? *Yes you do Sith Lord of the Empire*
If you said `Yes`, `No`, or didnโt reply to any of these,
you should download the `alpha` build instead:
```bash theme={"system"}
docker pull mageai/mageai:alpha
```
```bash pip theme={"system"}
pip install -U mage-ai
```
```bash conda theme={"system"}
conda update mage-ai
```
# Mage block variables
Source: https://docs.mage.ai/development/variables/block-variables
Use pipeline-specific variables scoped to an individual block.
Requires version `0.9.23` or greater.
In addition to the global runtime variables that can be used by any block in
a pipeline, there are also variables scoped at the block-level, meaning
they are accessible in a specific block but not other blocks in the same pipeline.
Block variables are not accessible by other pipelines that use the same block.
## Using block variables
Block variables can be accessed via the `kwargs` parameter in your block function.
The variables are stored in a block's configuration, which is accessible through
`kwargs['configuration']`.
```python theme={"system"}
@data_exporter
def export_data(*args, **kwargs):
demo_block_var = kwargs['configuration'].get('demo_block_var')
```
### Supported block types
Block variables are currently only supported for Python blocks with the
following block types:
1. `data_loader`
2. `transformer`
3. `data_exporter`
4. `custom`
5. `sensor`
## Creating block variables
### In Mage UI
You can create new block variables from the Mage UI through the "Block settings" tab
in the Sidekick of the Pipeline Editor page. The block settings can also be quickly accessed
by clicking the settings icon in a block's header.
In the "Block variables" section of the block settings, click the `+ New` button, enter
the variable's name and value in the input fields and then press `Enter` or `Return` on
your keyboard while still focused on one of the input fields (this step can be repeated
multiple times).
This will add the variable(s), but not save it. You need to also click the blue
`Update block settings` button below the "Block variables" section in order to save
and persist these new variables. You can also edit/delete existing block variables by
hovering over a variable and clicking the edit/trash icons. You can press the `Escape`
key to exit a variable's edit mode.
### In code
You can also create or edit block variables by going to a pipeline's `metadata.yaml` file
(located in the `pipelines/[pipeline_uuid]` directory) and adding them to a block's
`configuration` property.
```yaml theme={"system"}
blocks:
- all_upstream_blocks_executed: true
color: null
configuration:
demo_block_var: block_var_value
block_var2: val2
block_var3: val3
downstream_blocks: []
executor_config: null
executor_type: local_python
has_callback: false
language: python
name: billowing snow
retry_config: {}
status: updated
timeout: '60'
type: data_exporter
upstream_blocks:
- muddy_waterfall
uuid: billowing_snow
```
# Mage environment variables
Source: https://docs.mage.ai/development/variables/environment-variables
Configure Mage settings through environment variables.
## Environment variables in Mage Pro
Environment variables are set from the Mage Pro cluster management portal. Check out the steps and video below on how to create environment variables.
### Adding or modifying environment variables:
* Click the "Edit / add variables" button
* On the Environment Variables screen, you can either input a value directly or add a custom environment variable by specifying its key-value pair
* Click "Save changes" to update the settings.
After clicking the "Save changes" button the cluster will automatically restart.
## Environment variables in Mage OSS
Environment variables may be set in your Docker container to configure Mage settings. If youโre running Mage using `pip` or `conda`, your local machineโs environment variables
are accessible within the running Mage app.
Environment variables are different from [runtime variables](/getting-started/runtime-variable) or [secrets](/development/variables/secrets).
If youโre running Mage using Docker, you must add the following command line flags:
```bash theme={"system"}
-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2
```
The command to run Mage using Docker could look like this:
```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2 \
mageai/mageai /app/run_app.sh mage start [project_name]
```
## Setting the prefix for Mage URL
You may need to add a prefix to the Mage url. You can configure the Mage server to start
at a prefix to the url by setting the `MAGE_BASE_PATH` environment variable.
Make sure to set the base path without the leading slash. For example, if you want to start Mage at
`http://localhost:6789/base/url/path`, you would configure your environment
variable like this:
`export MAGE_BASE_PATH=base/url/path`
You can also configure the base path separately for client requests and backend routes by setting
the `MAGE_REQUESTS_BASE_PATH` and `MAGE_ROUTES_BASE_PATH` environment variables respectively.
| | |
| ------------------------- | ----------------------------------------------------------------------------------- |
| `MAGE_BASE_PATH` | Set base path for the entire app. |
| `MAGE_REQUESTS_BASE_PATH` | Set base path for frontend requests and urls, defaults to value of `MAGE_BASE_PATH` |
| `MAGE_ROUTES_BASE_PATH` | Set base path for backend routes, defaults to value of `MAGE_BASE_PATH` |
## Default variables
These are the environment variables used in Mage.
| Variable name | Description | Example values |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ACTIVE_DIRECTORY_DIRECTORY_ID` | Set this value to your organization's tenant id to enable signing in with Microsoft account. | [More info](/production/authentication/microsoft) |
| `ALLOW_DOWNLOADS_OUTSIDE_PROJECT` | Allow signed file downloads for files outside the current project directory. | `0`, `1` |
| `AUTHENTICATION_MODE` | Set the authentication mode in Mage. Omitting the variable will default to basic password authentication. | `ldap` |
| `DATAFRAME_SAMPLE_COUNT_PREVIEW` | Number of lines to display in block sample output preview for dataframes. Not suggested to go over 10 lines (defaults to 10). | 5 |
| `DEBUG` | Turn on "debug" mode. Mage will output more logs that could be useful for debugging issues. | 1 |
| `DEFAULT_EXECUTOR_TYPE` | The default executor type to run blocks | See enum types [here](https://docs.mage.ai/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) |
| `K8S_DEFAULT_CONFIG_PATH` | **(Mage Pro only.)** Path to YAML file for default K8s executor config; merged with block/project config. Prefer configuring via Helm `k8sExecutorConfig`. [K8s executor](/production/executors/k8#default-config-file) ยท [Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) | โ |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_CPU` | **(Mage Pro only.)** Default CPU request for K8s executor jobs (when not specified in config) | `250m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_MEMORY` | **(Mage Pro only.)** Default memory request for K8s executor jobs (when not specified in config) | `512Mi` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_CPU` | **(Mage Pro only.)** Default CPU limit for K8s executor jobs (when not specified in config) | `500m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_MEMORY` | **(Mage Pro only.)** Default memory limit for K8s executor jobs (when not specified in config) | `1Gi` |
| `K8S_EXECUTOR_DEFAULT_SERVICE_ACCOUNT_NAME` | **(Mage Pro only.)** Default service account for K8s executor jobs. See [K8s executor docs](/production/executors/k8#environment-variables-reference) for more details | `default` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_URL` | **(Mage Pro only.)** Prometheus base URL for K8s executor resource usage summaries. See [Monitoring with Prometheus](/observability/external/prometheus#kubernetes-resource-usage-in-mage-pro). | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN` | **(Mage Pro only.)** Bearer token for Prometheus resource usage queries. Takes precedence over basic auth when set. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME` | **(Mage Pro only.)** Username for Prometheus basic authentication. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD` | **(Mage Pro only.)** Password for Prometheus basic authentication. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS` | **(Mage Pro only.)** Timeout for each Prometheus resource usage query. | `5` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` | **(Mage Pro only.)** Query range step for Prometheus CPU and memory samples. | `15` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS` | **(Mage Pro only.)** Delay before retrying Prometheus once for completed K8s executor runs with too few samples. | `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS` | **(Mage Pro only.)** Set to `0`, `false`, `no`, or `off` to skip TLS certificate verification for Prometheus resource usage queries. | `true` |
| `WORKSPACE_DEFAULT_CONFIG_PATH` | **(Mage Pro only.)** Path to YAML file for default workspace config; merged with user config. Prefer configuring via Helm `workspaceConfig`. [Workspaces](/guides/developer-ux/workspaces#default-workspace-config-file) ยท [Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) | โ |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU` | **(Mage Pro only.)** Default CPU request for workspace containers (when not specified in container\_config). See [Workspaces docs](/guides/developer-ux/workspaces#default-resource-configuration) | `500m` |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY` | **(Mage Pro only.)** Default memory request for workspace containers (when not specified in container\_config) | `2Gi` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU` | **(Mage Pro only.)** Default CPU limit for workspace containers (when not specified in container\_config) | `4` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY` | **(Mage Pro only.)** Default memory limit for workspace containers (when not specified in container\_config) | `8Gi` |
| `DISABLE_NOTEBOOK_EDIT_ACCESS` | Setting this variable will disable edit access based on the value. [More information](/production/configuring-production-settings/overview#read-only-access) | 1 |
| `DISABLE_TERMINAL` | Setting this variable will disable access to the terminal. | 1 |
| `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS` | Maximum number of dynamic child block runs a dynamic block may create in one expansion. Leave unset or set to `0` to allow unlimited child runs. | `1000` |
| `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS_BEHAVIOR` | Behavior when a dynamic block exceeds `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS`. Use `fail` to stop the run or `limit` to create only up to the configured maximum. | `fail`, `limit` |
| `DYNAMIC_BLOCKS_MAX_CONCURRENT_CHILD_BLOCKS` | Maximum number of dynamic child block runs that may be queued or running at the same time for one pipeline run. Leave unset or set to `0` to allow unlimited concurrent dynamic child runs. | `25` |
| `ENV` | Set the environment of the Mage app. You can use different code or configurations in different environments. | `dev`, `staging`, `production`/`prod`, `test` |
| `GIT_REPO_LINK` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_REPO_PATH` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_USERNAME` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_EMAIL` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_AUTH_TYPE` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_BRANCH` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SYNC_ON_PIPELINE_RUN` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SYNC_ON_START` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SSH_PUBLIC_KEY` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_SSH_PRIVATE_KEY` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `GIT_ACCESS_TOKEN` | [More information](/production/data-sync/git-sync#git-sync-settings-as-environment-variables) | See link |
| `HIDE_ENV_VAR_VALUES` | Filter out environment variables in notebook output. | 1 |
| `JUPYTER_KERNEL_IDLE_TIMEOUT_SECONDS` | **(Mage Pro only.)** Number of seconds an inactive per-user Jupyter kernel can stay alive after its websocket clients disconnect. Used when `MULTI_USER_JUPYTER_KERNELS=1`. | `1800` |
| `JUPYTER_KERNEL_MAX_PER_SERVER` | **(Mage Pro only.)** Maximum number of per-user Jupyter kernels to keep on one Mage server. When the limit is reached, Mage evicts idle kernels first. Used when `MULTI_USER_JUPYTER_KERNELS=1`. | `50` |
| `JWT_SECRET` | Modify the json web token secret used for encoding and decoding access tokens. Defaults to `materia`. | `your_jwt_secret` |
| `JWT_DOWNLOAD_SECRET` | Modify the json web token secret used for encoding and decoding download tokens. Defaults to randomly generated string value. Use an explicit value in horizontally scalable deployments, e.g.: kubernetes, cloud deployments | `your_jwt_secret` |
| `LDAP_ADMIN_USERNAME` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_SERVER` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_BIND_DN` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_BIND_PASSWORD` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_BASE_DN` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_AUTHENTICATION_FILTER` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_AUTHORIZATION_FILTER` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_GROUP_FIELD` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_ROLES_MAPPING` | [More information](/production/authentication/overview#ldap) | See link |
| `LDAP_DEFAULT_ACCESS` | [More information](/production/authentication/overview#ldap) | See link |
| `LOGS_DIR_PATH` | Override the logs directory path. Default path is the variables\_dir path (e.g. `/home/src/mage_data/`) | `/home/src/logs_dir` |
| `MAGE_ACCESS_TOKEN_EXPIRY_TIME` | [More information](/production/authentication/overview#ldap) | See link |
| `MAGE_BASE_PATH` | The base path or prefix of the Mage server url. [More information](/getting-started/setup#setting-prefix-for-mage-url) | `base/path` |
| `MAGE_REQUESTS_BASE_PATH` | [More information](/getting-started/setup#setting-prefix-for-mage-url) | `base/path` |
| `MAGE_ROUTESBASE_PATH` | [More information](/getting-started/setup#setting-prefix-for-mage-url) | `base/path` |
| `MAGE_CACHE_DIRECTORY` | The directory Mage uses for storing and retrieving temporary data used in the UI. | `/root/.mage_data/default_repo/.cache` |
| `MAGE_CODEX_CHAT_ATTACHMENT_MAX_BYTES` | **(Mage Pro only.)** Maximum size, in bytes, for one Mage Pro AI Chat attachment. | `10485760` |
| `MAGE_CODEX_CHAT_ATTACHMENT_MAX_COUNT` | **(Mage Pro only.)** Maximum number of attachments allowed in one Mage Pro AI Chat session. | `20` |
| `MAGE_CODEX_CHAT_ATTACHMENT_MAX_SESSION_BYTES` | **(Mage Pro only.)** Maximum total attachment size, in bytes, allowed in one Mage Pro AI Chat session. | `52428800` |
| `MAGE_CODEX_CHAT_MAGE_API_KEY` | **(Mage Pro only.)** API key the Mage Pro AI Chat runner passes to Mage agent calls back into the current Mage API. Falls back to `MAGE_AGENT_CODEX_API_KEY` or `MAGE_API_KEY`. | `your_api_key` |
| `MAGE_CODEX_CHAT_MAGE_API_KEY_SECRET` | **(Mage Pro only.)** Optional Mage secret name that stores the API key for AI Chat runner callbacks into the current Mage API. No secret name is used by default. | `MAGE_AGENT_API_KEY` |
| `MAGE_CODEX_CHAT_MAGE_API_URL` | **(Mage Pro only.)** Base Mage API URL the Mage Pro AI Chat runner gives to the Mage agent MCP server. Falls back to `MAGE_CODEX_CHAT_CLUSTER_URL`, local server settings, or the cluster URL. | `https://your-mage-url` |
| `MAGE_CODEX_CHAT_MAX_CONCURRENT_RUNS` | **(Mage Pro only.)** Maximum number of Codex chat turns that can run at the same time in one Mage process. Unset, empty, invalid, or non-positive values mean unlimited. | `0` (unlimited), `4` |
| `MAGE_CODEX_CHAT_MODEL` | **(Mage Pro only.)** Default model used by Mage Pro AI Chat when a chat request does not override it. | `gpt-5.5` |
| `MAGE_CODEX_CHAT_OPENAI_API_KEY` | **(Mage Pro only.)** OpenAI or Codex API key used by Mage Pro AI Chat direct model authentication. `OPENAI_API_KEY` is also accepted as a fallback. | `your_openai_api_key` |
| `MAGE_CODEX_CHAT_OPENAI_API_KEY_SECRET` | **(Mage Pro only.)** Mage secret name that stores the API key for Mage Pro AI Chat direct model authentication. | `OPENAI_API_KEY` |
| `MAGE_CODEX_CHAT_REASONING_EFFORT` | **(Mage Pro only.)** Default reasoning effort used by Mage Pro AI Chat when a chat request does not override it. | `low`, `medium`, `high`, `xhigh` |
| `MAGE_CODEX_CHAT_SERVICE_TIER` | **(Mage Pro only.)** Default service tier used by Mage Pro AI Chat when a chat request does not override it. | `priority` |
| `MAGE_CODEX_MODEL_AUTH_MODE` | **(Mage Pro only.)** Select direct API-key authentication or Mage Pro proxy authentication for the AI Chat Codex model provider. | `direct`, `proxy` |
| `MAGE_CODEX_MODEL_PROXY_BASE_URL` | **(Mage Pro only.)** OpenAI-compatible proxy base URL used when `MAGE_CODEX_MODEL_AUTH_MODE=proxy`. Defaults from the Mage Pro API endpoint when available. | `https://pro-api.mage.ai/api/v1/codex/openai/v1` |
| `MAGE_DATABASE_CONNECTION_URL` | Specify the database connection url for orchestration DB. Defaults to local sqlite db. | [Example](/production/configuring-production-settings/overview#postgres) |
| `MAGE_DATABASE_APPLICATION_NAME` | Override the PostgreSQL `application_name` used by Mage DB connections. Defaults to a generated `mage::` value. | `mage:production:server-1` |
| `MAGE_DATABASE_IDLE_IN_TRANSACTION_SESSION_TIMEOUT_MS` | PostgreSQL session timeout, in milliseconds, for idle transactions opened by Mage. Set to `0` to disable the per-session timeout. | `120000` |
| `MAGE_DATABASE_LOCK_TIMEOUT_MS` | Optional PostgreSQL per-session lock wait timeout, in milliseconds, for Mage DB connections. `0` disables this setting. | `0` |
| `MAGE_DATABASE_MAX_OVERFLOW` | Extra PostgreSQL connections a Mage process may open above `MAGE_DATABASE_POOL_SIZE`. | `2` |
| `MAGE_DATABASE_POOL_RECYCLE` | Number of seconds before SQLAlchemy recycles an idle PostgreSQL pooled connection. | `1800` |
| `MAGE_DATABASE_POOL_SIZE` | Base SQLAlchemy connection pool size per Mage process for PostgreSQL orchestration DB connections. Use small values for shared workspaces. | `5` |
| `MAGE_DATABASE_POOL_TIMEOUT` | Number of seconds a Mage request waits for a PostgreSQL pooled connection before failing with a pool exhaustion error. | `10` |
| `MAGE_DATABASE_STATEMENT_TIMEOUT_MS` | Optional PostgreSQL per-session statement timeout, in milliseconds, for Mage DB connections. `0` disables this setting. | `0` |
| `MAGE_PRO_API_API_KEY` | **(Mage Pro only.)** Mage Pro API key used when Mage Pro proxy mode or other Mage Pro API integrations call the shared Mage Pro API service. | `your_mage_pro_api_key` |
| `MAGE_PRO_API_ENDPOINT` | **(Mage Pro only.)** Mage Pro API host used by Mage Pro proxy mode and other Mage Pro API integrations. Do not include the URL scheme. Hosted production clusters use `pro-api.mage.ai`; staging hosts such as `pro-api-staging.mage.ai` are for non-production environments. | `pro-api.mage.ai` |
| `MAGE_PUBLIC_HOST` | The public host url that can be used to access the Mage app. This value will be used in emails or other notifications. | `http://localhost:6789` |
| `MAGE_RESTART_STREAMING_PIPELINES_ON_REQUIREMENTS_CHANGE` | **(Mage Pro only.)** When true, running streaming pipelines are restarted after the project `requirements.txt` changes. | 1 |
| `MAX_PRINT_OUTPUT_LINES` | The max number of `stdout` lines, such as from `print` statements, included in a message from the websocket server. Defaults to `1000`. | `100` |
| `MULTI_USER_JUPYTER_KERNELS` | **(Mage Pro only.)** Enable a separate Python Jupyter kernel for each authenticated user in the pipeline editor so multiple users can run blocks at the same time. | `0` |
| `REQUIRE_USER_AUTHENTICATION` | Enable user authentication in Mage. [More information](/production/authentication/overview) | 1 |
| `RUN_PIPELINE_IN_ONE_PROCESS` | When True, pipelines will run in one process, unless otherwise specified in the settings for a given pipeline. | 1 |
| `SERVER_VERBOSITY` | [More information](/development/observability/logging#server-logging) | See link |
| `SHELL_COMMAND` | Set shell command to use for the Mage terminal. Default command is `bash` for macOS/Unix and `cmd` for Windows. | `bash`, `cmd`, ... |
| `ULIMIT_NO_FILE` | Override the maximum number of open files allowed in Mage processes. | 8192 |
| `USER_CODE_PATH` | Set this path to be `/home/src/{your_project_name}` so that Mage starts your project and installs the packages in requirements.txt when starting the container. | `/home/src/mage_project` |
# Mage variables
Source: https://docs.mage.ai/development/variables/overview
There are a few ways to set up variables in Mage, here's how you can use them to make the best pipelines the world has ever seen. ๐
There are four types of variable in Mage: Environment variables, runtime variables, block variables, and secrets.
| Variable Type | Scope | Creation | Reference | Encrypted |
| -------------------- | -------- | -------- | ------------------------------- | --------- |
| Environment variable | System | Code | `{{ env_var() }}` | ๐ซ |
| Runtime variable | Pipeline | Code/UI | `kwargs` or `{{ variables() }}` | ๐ซ |
| Block variable | Block | Code/UI | `kwargs['configuration'][key]` | ๐ซ |
| Secret | Project | UI | `{{ mage_secret_var() }}` | โ |
Here's a brief overview of each:
## Environment variables
Environment variables are a type of variable that is used to store configuration settings and other system-level information for operating systems and applications. These variables are accessible to any process running on the system, which allows multiple programs to share the same configuration information. Environment variables exist in virtually all operating systems, including Linux, macOS, and Windows.
When running Mage in Docker, Mage will inherit the environment variables from the host machine. You can also set environment variables in the Dockerfile. Read more about environment variables in Mage [here](/development/variables/environment-variables) and Docker [here](https://docs.docker.com/compose/environment-variables/set-environment-variables/).
## Runtime variables
Runtime Variables are a set of global variables that can be used by every block in a pipeline. These are useful for storing constants shared by multiple blocks or constants whose value is determined at pipeline runtime (hence runtime variables).
If you're familiar with Airflow, runtime variables are similar to the Airflow "context" object. You can use runtime variables to pass information between blocks in a pipeline. Read more [here](/getting-started/runtime-variable).
## Block variables
In addition to the global runtime variables that can be used by any block in a pipeline, there are also variables scoped at the block-level, meaning they are accessible in a specific block but not other blocks in the same pipeline. Block variables are not accessible by other pipelines that use the same block. Read more [here](/development/variables/block-variables).
## Secrets
Secrets are a way to input sensitive information *via the Mage UI* that can be used by blocks in a pipeline. Secrets are stored in the Mage database and are encrypted. Secrets can come in handy when you have multiple users developing in an environment or are otherwise unable to use environment variables. Read more [here](/development/variables/secrets).
# Accessing variables in Mage
Source: https://docs.mage.ai/development/variables/referencing-variables
You've got all of these great variables and secrets, now let's put them to use. Learn how to access them in your code. ๐ฆธ๐ปโโ๏ธ
Mage allows users to interpolate variables specific to your [pipeline](/getting-started/runtime-variable) or [project](/development/variables/environment-variables). The following syntax is specific to `yaml` and `SQL` files.
Here's a list of the different variables and functions you can use in your code:
| Syntax | Description |
| ------------------------------------ | -------------------------------------------------------------------------------------- |
| `{{ env_var('secret') }}` | Get secret from [environment variables](/development/variables/environment-variables). |
| `{{ variables('secret') }}` | Get secret from [runtime variables](/getting-started/runtime-variable). |
| `{{ mage_secret_var('secret') }}` | Get secret from [Mage secrets](/development/variables/secrets). |
| `{{ aws_secret_var('secret') }}` | Get secret from [AWS Secrets Manager](/production/deploying-to-cloud/secrets/AWS). |
| `{{ azure_secret_var('secret') }}` | Get secret from [Azure Key Vault](/production/deploying-to-cloud/secrets/Azure). |
| `{{ doppler_secret_var('secret') }}` | Get secret from Doppler. |
| `{{ json_value(json_obj, 'key') }}` | Extract value from a JSON string. |
| `{{ n_days_ago(N) }}` | Get the date `N` days ago (data integration pipelines only). |
## Mage Pro only variable syntax
| Syntax | Description |
| -------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `{{ file('path/to/file.txt') }}` | Load the contents of a [local file](/development/variables/referencing-variables#file-content-syntax). |
### File content syntax
This is useful for dynamically loading configuration files, secrets, or text assets during pipeline execution.
The path can be either absolute path or relative path to the project.
**Example Usage**
```jinja theme={"system"}
{{ file('secrets.txt') }}`
{{ file('/home/src/default_repo/settings.json') }}`
```
If you want to get a json field from the local file, you can use it together with the `json_value` syntax like
```
{{ json_value(file('path/to/file.json'), 'key') }}`
```
### Doppler secrets syntax
Use `doppler_secret_var` to read a secret directly from Doppler when the template is rendered:
```jinja theme={"system"}
{{ doppler_secret_var('POSTGRES_PASSWORD') }}
```
Mage reads the default Doppler project and config from these environment variables:
| Variable | Description |
| ----------------------------------- | -------------------------------------------------------------------------------------- |
| `DOPPLER_TOKEN` | Doppler token used to read secrets. |
| `DOPPLER_PROJECT` | Doppler project identifier. |
| `DOPPLER_CONFIG` | Doppler config name. |
| `DOPPLER_API_HOST` | Optional API host. Defaults to `https://api.doppler.com`. |
| `DOPPLER_SECRETS_CACHE_TTL_SECONDS` | Optional in-memory cache TTL. Defaults to `10`. Set to `0` to refresh on every lookup. |
You can override the project or config for an individual reference:
```jinja theme={"system"}
{{ doppler_secret_var('POSTGRES_PASSWORD', project='analytics', config='prd') }}
```
When a Doppler secret changes, Mage picks up the new value on the next template render after the cache TTL expires. Already-open database connections or clients may continue using the value they were created with until they are recreated.
# Secrets
Source: https://docs.mage.ai/development/variables/secrets
Storing and using secrets through Mage
In various surfaces in Mage, you may be asked to input config for certain integrations
such as cloud databases or services. In these cases, you may need to input a password
or an api key, but you don't want it to be shown in plain text. To get around this issue,
we created a way to store your secrets in the Mage database.
Your secrets are encrypted before being stored into the Mage database. The encryption key
will be stored in the Mage data folder which by default will be created at `~/.mage_data`. If
you need more secure encryption, we recommend using a secrets manager.
## Secret management
### Creating secrets
To manage your secrets, you can go to the **Secrets** page by clicking on the **Secrets**
icon in the side navigation. This will open the global secrets management page where
you can see all your secrets, create new ones, and delete existing ones.
You can also manage secrets directly from the pipeline edit page. In the sidekick,
you should see a `Secrets` tab. To create a secret, press the `New` button on this tab.
Input a name for the secret and the value of the secret, and press Enter or Return to save.
The secrets can be shared across the project that they are created in.
Pipeline level secrets coming soon...
### Secrets in different environments
Secrets are stored to the Mage database. If you are using the same database across multiple
Mage environments, i.e. development and production, the secrets will not be shared across the environments
unless you shared the same Mage data directory for every environment.
Each secret will be saved to the database with a name and a uuid. The name will be used to identify
the secret, and the uuid is used to identify the environment.
Example:
```
Development environment
project
โโโโmage_data
โ โโโโsecrets
โ โ โ key: abc123
โ โ โ uuid: dev-uuid
Production environment
project
โโโโmage_data
โ โโโโsecrets
โ โ โ key: def456
โ โ โ uuid: prod-uuid
```
Secrets table in database
| Name | Uuid | Value |
| --------- | ----------- | --------------------------- |
| `secret1` | `dev-uuid` | sooper dooper secret string |
| `secret2` | `dev-uuid` | xXxSecreTxXx |
| `secret3` | `dev-uuid` | dev only secret |
| `secret1` | `prod-uuid` | real production secret |
| `secret2` | `prod-uuid` | m4g3r0x! |
| `secret1` | `random` | random secret |
| Command | Development Value | Production Value |
| ----------------------------- | --------------------------- | ---------------------- |
| `get_secret_value('secret1')` | sooper dooper secret string | real production secret |
| `get_secret_value('secret2')` | xXxSecreTxXx | m4g3r0x! |
| `get_secret_value('secret3')` | dev only secret | `None` |
Coming soon:
* secrets scoped to a pipeline
## Using Secrets
You can use the following syntax to have Mage interpolate the secret when reading from the config:
```yaml theme={"system"}
field_name: "{{ mage_secret_var('your_secret_name') }}"
```
You can also fetch the secret value in a Mage code block by importing a helper method:
```python theme={"system"}
from mage_ai.data_preparation.shared.secrets import get_secret_value
get_secret_value('your_secret_name')
```
# Create backfill
Source: https://docs.mage.ai/extensibility/api-reference/backfills/create-backfills
POST /api/pipelines/:pipeline_uuid/backfills
`POST /api/pipelines/:pipeline_uuid/backfills`
Pipeline UUID to create the backfill for.
Body
```curl cURL theme={"system"}
curl --request POST \
--url 'http://localhost:6789/api/pipelines/example_pipeline/backfills' \
--header 'OAUTH-TOKEN=some_really_long_string' \
--header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--data '{
"backfill": {
"name": "fire wind"
}
}'
```
```json Response theme={"system"}
{
"backfill": {
"id": 10,
"block_uuid": null,
"completed_at": null,
"created_at": "2023-04-01 01:02:18.411997+00:00",
"end_datetime": null,
"failed_at": null,
"interval_type": null,
"interval_units": null,
"metrics": null,
"name": "fire wind",
"pipeline_schedule_id": null,
"pipeline_uuid": "example_pipeline",
"start_datetime": null,
"started_at": null,
"status": null,
"updated_at": "2023-04-01 01:02:18.411997+00:00",
"variables": null
}
}
```
# Delete backfill
Source: https://docs.mage.ai/extensibility/api-reference/backfills/delete-backfill
DELETE /api/backfills/:backfill_id
`DELETE /api/backfills/:backfill_id`
```curl cURL theme={"system"}
curl -X DELETE
--url http://localhost:6789/api/backfills/1
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=some_really_long_string'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
```
```python Python theme={"system"}
headers = {
"Content-Type": "application/json",
"X-API-KEY": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
"Cookie": "oauth_token=some_really_long_string"
}
r = requests.delete('http://localhost:6789/api/backfills/1', headers=headers)
```
```json Response theme={"system"}
{
"backfill": {
"id": 1,
"block_uuid": null,
"completed_at": null,
"created_at": "2023-07-17 21:48:42",
"end_datetime": null,
"failed_at": null,
"interval_type": null,
"interval_units": null,
"metrics": null,
"name": "arwen_starlight_backfill",
"pipeline_schedule_id": null,
"pipeline_uuid": "arwen_starlight",
"start_datetime": null,
"started_at": null,
"status": null,
"updated_at": "2023-07-17 21:48:42",
"variables": null
}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/backfills/overview
Programmatically create multiple pipeline runs to backfill a pipeline.
### Backfill object
```json Response theme={"system"}
{
"block_uuid": null,
"completed_at": null,
"created_at": "2023-03-23 18:27:58.848959+00:00",
"end_datetime": "2023-03-05 00:00:00+00:00",
"failed_at": null,
"id": 9,
"interval_type": "hour",
"interval_units": 4,
"metrics": null,
"name": "sparkling glitter",
"pipeline_schedule_id": null,
"pipeline_uuid": "example_pipeline",
"start_datetime": "2023-03-01 00:00:00+00:00",
"started_at": null,
"status": null,
"total_run_count": 25,
"updated_at": "2023-03-23 18:28:22.379725+00:00",
"variables": null
}
```
If the backfill uses custom code to generate the pipeline runs, this block UUID contains the code
that will be used.
Date and time the backfill finished running successfully.
Date and time the record was created.
The date and time the backfill should stop on.
Date and time the backfill failed.
Unique ID for the record.
`minute`, `hour`, `day`, `week`, `month`, `year`
The number of `interval_type` to use in between each pipeline run that the backfill creates.
For example, if `interval_type` is `day` and `interval_units` is 3, then the backfill will create
pipeline runs spanning 3 day increments between the `start_datetime` and `end_datetime`.
Various metrics related to the pipeline run; e.g. number of rows processed in a
data integration pipeline sync.
Name of the record.
Pipeline schedule that this record belongs to.
Pipeline that this record belongs to.
The date and time the backfill should start on.
Date and time the backfill started running.
`initial`, `running`, `completed`, `failed`, `cancelled`
Number of pipeline runs that will be created for this backfill.
Date and time the record was last updated.
Object containing variables for the pipeline that is being overwritten by this backfill.
The property name is user defined.
# Read backfills
Source: https://docs.mage.ai/extensibility/api-reference/backfills/read-backfills
GET /api/backfills
`GET /api/backfills`
Maximum number of logs to be returned in the API response.
Example: `20`
Read logs after N number of logs, where N equals `_offset`.
Example: `10`
The pipeline uuid for which to read backfills.
A boolean which indicates whether to return the run count in the response metadata object.
```curl cURL theme={"system"}
curl --request GET \
--url 'http://localhost:6789/api/backfills?_limit=20&_offset=0&include_run_count=true&pipeline_uuid=example_pipeline&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'OAUTH-TOKEN=some_really_long_string' \
```
```json Response theme={"system"}
{
"backfills": [
{
"block_uuid": null,
"completed_at": null,
"created_at": "2023-03-23 18:27:58.848959+00:00",
"end_datetime": "2023-03-05 00:00:00+00:00",
"failed_at": null,
"id": 9,
"interval_type": "hour",
"interval_units": 4,
"metrics": null,
"name": "sparkling glitter",
"pipeline_schedule_id": null,
"pipeline_uuid": "example_pipeline",
"start_datetime": "2023-03-01 00:00:00+00:00",
"started_at": null,
"status": null,
"total_run_count": 25,
"updated_at": "2023-03-23 18:28:22.379725+00:00",
"variables": null
}
],
"metadata": {
"count": 0,
"next": false
}
}
```
# Update backfill
Source: https://docs.mage.ai/extensibility/api-reference/backfills/update-backfill
PUT /api/pipelines/:pipeline_uuid/backfills/:backfill_id
`PUT /api/pipelines/:pipeline_uuid/backfills/:backfill_id`
All default attributes for a backfill may be updated in addittion to the following:
A datetime formatted string representing the start time of the backfill, e.g. `"2023-09-05 23:56:05"`.
A datetime formatted string representing the end time of the backfill, e.g. `"2023-09-05 23:56:05"`.
The interval type for the backfill. One of: `minute`, `hour`, `day`, `week`, `month`, or `year`.
The number of interval units for the backfill.
A new name for the backfill.
The backfill status. One of `initial`, `running`, `completed`, `failed`, `cancelled`.
An object containing Global Pipeline Variables to be overridden by the backfill. Any Global Variable may be overridden.
```curl cURL theme={"system"}
curl -X PUT
--url http://localhost:6789/api/pipelines/arwen_starlight/backfills/6
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHBpcmVzIjoxNjkyMzE0ODM3LjE1MTkzOSwidG9rZW4iOiJieHlQajlVcHE5N2phTC1nTGtOQTVnM2draWZwRjJwSEZZZkNzLTJ2NmpnIn0.bWqUP7Xg9PmHBW703AGPOE270Cc73F4kb0IKHLppGbM'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
-d '{
"backfill": {
"start_datetime": "2023-07-17 23:56:05",
"end_datetime": "2023-07-18 23:56:05",
"name": "new_name"
}
}'
```
```python Python theme={"system"}
import requests
headers = {
"Content-Type": "application/json",
"X-API-KEY": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
"Cookie": "oauth_token=some_really_long_string"
}
json_data = {
'backfill': {
"start_datetime": '2023-07-17 23:56:05',
"end_datetime": '2023-07-18 23:56:05',
'name':'new_name'
}
}
r = r.put('http://localhost:6789/api/pipelines/arwen_starlight/backfills/6', json=json_data, headers=headers)
```
```json Response theme={"system"}
{
"backfill": {
"id": 6,
"block_uuid": null,
"completed_at": null,
"created_at": "2023-07-18 23:30:26",
"end_datetime": "2023-07-18 23:56:05",
"failed_at": null,
"interval_type": null,
"interval_units": null,
"metrics": null,
"name": "new_name",
"pipeline_schedule_id": null,
"pipeline_uuid": "arwen_starlight",
"start_datetime": "2023-07-17 23:56:05",
"started_at": null,
"status": null,
"updated_at": "2023-07-18 23:31:57",
"variables": null
}
}
```
# Create block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/create-block
POST /api/pipelines/:pipeline_uuid/blocks
`POST /api/pipelines/:pipeline_uuid/blocks`
Create a new block in one or more pipelines.
Human readable block name. This is used to generate the UUID, e.g. `restless dust`. Names will be slugified to a safe-uuid.
Type of block:
`callback`, `chart`, `conditional`, `custom`, `data_exporter`, `data_loader`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`, `sensor`, `transformer`
Block color displayed by the frontend for `custom` blocks. All other block types have predefined colors.
An optional parameter specifying a block configuration file For example, `{data_source: "s3"}` will create an s3 data loader.
Miscellaneous configuration settings for the block.
Database or data warehouse for the SQL block to connect to.
Database name to use when saving the output of the SQL block.
Profile target for the dbt block.
Schema name to use when saving the output of the SQL block.
Table name to use when saving the output of the SQL block.
Whether to `replace` the existing table of the SQL block output, `append`, or
raise an error and `fail`.
Toggle writing raw SQL in the block. Read more [here](/guides/blocks/sql-blocks#using-raw-sql).
The content of block formatted as a string. Use `/n` for newlines. Omitting will create an empty block.
UUID of extensions, currently [Great Expectations](/development/testing/great-expectations) is the only extension supported.
The block language:
`python`, `sql`, `r`, `yaml`, `markdown`
Array of strings containing the pipeline uuids for which the block is a member.
The 0-indexed position of the block in the pipeline, e.g. 0 is the first block location.
An array of strings representing the uuids of upstream blocks (the blocks on which the new block depends). For example, `["a", "b"]` will create a block that depends on blocks `a` and `b`.
The unique id of the block. This is generated from `name`, so it's not necessary to populate this field.
```bash Request theme={"system"}
curl 'http://localhost:6789/api/pipelines/broken_wave/blocks?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Authorization: Bearer 018Cc8ydDUa6YWoLad2oMdw0EKdWWJuyg85D02GN1ww' \
-H 'Content-Type: application/json' \
--data-raw '
{
"block": {
"name": "silent fire",
"priority": 2,
"configuration": {
"data_source": null
},
"language": "python",
"type": "data_exporter"
},
"api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2"
}
' \
--compressed
```
```json Response theme={"system"}
{
"block": {
"content": null,
"name": "silent fire",
"priority": 2,
"config": {
"data_source": null
},
"language": "python",
"type": "data_loader",
"configuration": {},
"upstream_blocks": []
},
}
```
# Delete block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/delete-block
DELETE /api/pipelines/:pipeline_uuid/blocks/:block_uuid
`DELETE /api/pipelines/:pipeline_uuid/blocks/:block_uuid`
Delete a block from a pipeline.
By default, blocks with downstream dependencies may not be deleted. To force delete blocks, set the `force` query parameter to the boolean `True`.
The following example deletes a fictional block `winter_wildflower` from the `broken_wave` pipeline.
```bash Request theme={"system"}
curl 'http://localhost:6789/api/pipelines/broken_wave/blocks/winter_wildflower?block_type=data_loader&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2&force=True' \
-X 'DELETE'
```
```json Response theme={"system"}
{
"block": {
"all_upstream_blocks_executed": true,
"color": null,
"configuration": {},
"downstream_blocks": [],
"executor_config": null,
"executor_type": "local_python",
"has_callback": false,
"name": "winter wildflower",
"language": "python",
"retry_config": null,
"status": "not_executed",
"type": "data_loader",
"upstream_blocks": [],
"uuid": "winter_wildflower"
}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/blocks/overview
The Blocks API is used to create, update, and delete blocks objects from a pipeline.
### Block Object
Whether or not all upstream blocks have been successfully executed.
The color of the block in the notebook UI.
Miscellaneous configuration settings for the block.
Database or data warehouse for the SQL block to connect to.
Database name to use when saving the output of the SQL block.
Profile target for the dbt block.
Schema name to use when saving the output of the SQL block.
Table name to use when saving the output of the SQL block.
Whether to `replace` the existing table of the SQL block output, `append`, or
raise an error and `fail`.
Toggle writing raw SQL in the block. Read more [here](/guides/blocks/sql-blocks#using-raw-sql).
Block file contents.
The block UUIDs that depend on this block.
Used when configuring a Kubernetes executor. See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details
The type of executor to use for the block:
`ecs`, `gcp_cloud_run`, `azure_container_instance`, `k8s`, `local_python`, `pyspark`
See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details.
`python`, `r`, `sql`, `yaml`, `markdown`
Human readable name of block.
Status of block: `executed`, `failed`, `not_executed`, `updated`
Type of block:
`callback`, `chart`, `conditional`, `custom`, `data_exporter`, `data_loader`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`, `sensor`, `transformer`
The block UUIDs that this block depends on.
Unique identifier for the block.
```json Response theme={"system"}
{
"block": {
"all_upstream_blocks_executed": true,
"color": null,
"configuration": {},
"content": "some_long_block_content_string",
"downstream_blocks": [],
"executor_config": null,
"executor_type": "local_python",
"has_callback": false,
"name": "solitary silence",
"language": "python",
"retry_config": null,
"status": "not_executed",
"type": "data_loader",
"upstream_blocks": [],
"uuid": "solitary_silence",
"content": "some_long_block_content_string",
"outputs": []
}
}
```
# Read block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/read-block
GET /api/pipelines/:pipeline_uuid/blocks/:block_uuid
`GET /api/pipelines/:pipeline_uuid/blocks/:block_uuid`
Fetch and return the contents and metadata of a block.
```bash Request theme={"system"}
curl --location 'http://localhost:6789/api/pipelines/broken_wave/blocks/solitary_silence' \
--header 'Cookie: oauth_token=long_token_string' \
--header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
```
```json Response theme={"system"}
{
"block": {
"all_upstream_blocks_executed": true,
"color": null,
"configuration": {},
"downstream_blocks": [],
"executor_config": null,
"executor_type": "local_python",
"has_callback": false,
"name": "solitary silence",
"language": "python",
"retry_config": null,
"status": "not_executed",
"type": "data_loader",
"upstream_blocks": [],
"uuid": "solitary_silence",
"content": "some_long_block_content_string",
"outputs": []
}
}
```
# Update block
Source: https://docs.mage.ai/extensibility/api-reference/blocks/update-block
PUT /api/pipelines/:pipeline_uuid/blocks/:block_uuid
`PUT /api/pipelines/:pipeline_uuid/blocks/:block_uuid`
Human readable block name. This is used to generate the UUID, e.g. `restless dust`. Names will be slugified to a safe-uuid.
Type of block:
`callback`, `chart`, `conditional`, `custom`, `data_exporter`, `data_loader`, `dbt`, `extension`, `global_data_product`, `markdown`, `scratchpad`, `sensor`, `transformer`
Data integration bookmark values as a list of strings.
Array of [callback](/development/blocks/callbacks) block objects. Same shape as `blocks`.
Block color displayed by the frontend for `custom` blocks. All other block types have predefined colors.
Miscellaneous configuration settings for the block.
Database or data warehouse for the SQL block to connect to.
Database name to use when saving the output of the SQL block.
Profile target for the dbt block.
Schema name to use when saving the output of the SQL block.
Table name to use when saving the output of the SQL block.
Whether to `replace` the existing table of the SQL block output, `append`, or
raise an error and `fail`.
Toggle writing raw SQL in the block. Read more [here](/guides/blocks/sql-blocks#using-raw-sql).
The content of block formatted as a string. Use `/n` for newlines. Omitting will create an empty block.
The destination table for a data integration pipeline.
Array of blocks downstream from the created block.
Used when configuring a Kubernetes executor. See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details
Used when configuring a Kubernetes executor. See the [Kubernetes config](/production/configuring-production-settings/compute-resource#2-set-executor-type-and-customize-the-compute-resource-of-the-mage-executor) page for more details
UUID of extensions, currently [Great Expectations](/development/testing/great-expectations) is the only extension supported.
The block language:
`python`, `sql`, `r`, `yaml`, `markdown`
Array of strings containing the pipeline uuids for which the block is a member.
The configuration for block retry logic. See the [pipeline runs](/orchestration/pipeline-runs/retrying-block-runs) docs for retry config examples.
The block tap stream ID for data integration pipelines.
An array of strings representing the uuids of upstream blocks (the blocks on which the new block depends). For example, `["a", "b"]` will create a block that depends on blocks `a` and `b`.
The unique id of the block. This is generated from `name`, so it's not necessary to populate this field.
```bash Request theme={"system"}
curl --location --request PUT 'http://localhost:6789/api/pipelines/broken_wave/blocks/solitary_silence' \
--header 'Cookie: oauth_token=long_token_string' \
--header 'Content-Type: text/plain' \
--header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--data '{
"block": {
"color": "teal"
}
}'
```
```json Response theme={"system"}
{
"block": {
"all_upstream_blocks_executed": true,
"color": "teal",
"configuration": {},
"downstream_blocks": [],
"executor_config": null,
"executor_type": "local_python",
"has_callback": false,
"name": "solitary silence",
"language": "python",
"retry_config": null,
"status": "not_executed",
"type": "data_loader",
"upstream_blocks": [],
"uuid": "solitary_silence",
"content": "some_long_content_string"
}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/logs/overview
Fetch log entries for your Mage instance.
### Log object
```json Response theme={"system"}
{
"block_run_logs": [
{
"name": "export_titanic_clean.log",
"path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/export_titanic_clean.log",
"content": ""
}
],
"pipeline_run_logs": [
{
"name": "pipeline.log",
"path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/55/20230308T045408/pipeline.log",
"content": "2023-03-08T04:54:08 {\"cpu\": 0.21, \"cpu_total\": 8, \"cpu_usage\": 0.02625, \"memory\": 4013, \"memory_total\": 17010, \"memory_usage\": 0.23592004703115815, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251248.712125, \"uuid\": \"161125d3c3944d109ce945fa31f8c8a2\"}\n2023-03-08T04:54:08 {\"block_run_id\": 648, \"block_uuid\": \"load_titanic\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 648: pipeline example_pipeline block load_titanic\", \"timestamp\": 1678251248.796509, \"uuid\": \"2223dba30d2d4217828ce031f5aa0f05\"}\n2023-03-08T04:54:09 {\"block_run_id\": 648, \"block_uuid\": \"load_titanic\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 648 (block_uuid: load_titanic) completes.\", \"timestamp\": 1678251249.079394, \"uuid\": \"8c2a6c5ce8d64bbf96b51de463b0e91a\"}\n2023-03-08T04:54:18 {\"cpu\": 0.21, \"cpu_total\": 8, \"cpu_usage\": 0.02625, \"memory\": 4016, \"memory_total\": 17010, \"memory_usage\": 0.23609641387419164, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251258.908166, \"uuid\": \"fd079d501f33478985bf260b92d567f3\"}\n2023-03-08T04:54:18 {\"block_run_id\": 649, \"block_uuid\": \"fill_in_missing_values\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 649: pipeline example_pipeline block fill_in_missing_values\", \"timestamp\": 1678251258.997002, \"uuid\": \"8c4c4d69560a4083a4ff0bf30722e3fa\"}\n2023-03-08T04:54:19 {\"block_run_id\": 649, \"block_uuid\": \"fill_in_missing_values\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 649 (block_uuid: fill_in_missing_values) completes.\", \"timestamp\": 1678251259.224986, \"uuid\": \"44a35d24ae874e77af98cc1e6ab86278\"}\n2023-03-08T04:54:29 {\"cpu\": 0.2, \"cpu_total\": 8, \"cpu_usage\": 0.025, \"memory\": 4016, \"memory_total\": 17010, \"memory_usage\": 0.23609641387419164, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251269.186595, \"uuid\": \"97affe68017a4c65a69fad3492b80ea5\"}\n2023-03-08T04:54:29 {\"block_run_id\": 650, \"block_uuid\": \"export_titanic_clean\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 650: pipeline example_pipeline block export_titanic_clean\", \"timestamp\": 1678251269.292511, \"uuid\": \"97aa24db4b6e4da1b6f98222e1d0cd06\"}\n2023-03-08T04:54:29 {\"block_run_id\": 650, \"block_uuid\": \"export_titanic_clean\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 650 (block_uuid: export_titanic_clean) completes.\", \"timestamp\": 1678251269.383662, \"uuid\": \"d06e3d9dcb2448a583ae323e25e04045\"}\n2023-03-08T04:54:30 {\"block_run_id\": 651, \"block_uuid\": \"falling_glitter\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Execute PipelineRun 543, BlockRun 651: pipeline example_pipeline block falling_glitter\", \"timestamp\": 1678251270.335112, \"uuid\": \"70b81921cb374e0d9c712ce6379f2148\"}\n2023-03-08T04:54:30 {\"block_run_id\": 651, \"block_uuid\": \"falling_glitter\", \"pipeline_run_id\": 543, \"pipeline_schedule_id\": 55, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"BlockRun 651 (block_uuid: falling_glitter) completes.\", \"timestamp\": 1678251270.571607, \"uuid\": \"f2220d588ca746f49b5fafca1113be16\"}\n2023-03-08T04:54:38 {\"cpu\": 0.2, \"cpu_total\": 8, \"cpu_usage\": 0.025, \"memory\": 4019, \"memory_total\": 17010, \"memory_usage\": 0.23627278071722516, \"pipeline_run_id\": 543, \"pipeline_uuid\": \"example_pipeline\", \"level\": \"INFO\", \"message\": \"Pipeline example_pipeline for run 543 in schedule 55 is alive.\", \"timestamp\": 1678251278.430712, \"uuid\": \"3de507ebca894100b20ae4a1645724d7\"}\n"
}
],
"total_block_run_log_count": 1,
"total_pipeline_run_log_count": 1
}
```
Logs for block runs.
File name of log file.
Full path of log file.
Contents of log file.
Logs for pipeline runs.
File name of log file.
Full path of log file.
Contents of log file.
Number of block run logs in the API response.
Number of pipeline run logs in the API response.
# Read logs
Source: https://docs.mage.ai/extensibility/api-reference/logs/read-logs
GET /api/pipelines/:pipeline_uuid/logs
`GET /api/pipelines/:pipeline_uuid/logs`
Pipeline UUID of the logs.
Example: `example_pipeline`
Maximum number of logs to be returned in the API response.
Example: `20`
Read logs after N number of logs, where N equals `_offset`.
Example: `10`
Read logs on or after a specified timestamp.
Example: `1677648116`
```curl cURL theme={"system"}
curl --request GET \
--url 'http://localhost:6789/api/pipelines/example_pipeline/logs?_limit=20&_offset=0&start_timestamp=1677648116&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'OAUTH-TOKEN=some_really_long_string' \
```
```json Response theme={"system"}
{
"logs": [
{
"block_run_logs": [
{
"name": "solitary_brook.log",
"path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/solitary_brook.log",
"content": "very_long_content"
},
{
"name": "load_titanic.log",
"path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/load_titanic.log",
"content": "very_long_content"
}
],
"pipeline_run_logs": [
{
"name": "pipeline.log",
"path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/55/20230308T045408/pipeline.log",
"content": "very_long_content \"timestamp\": 1678251278.430712, \"uuid\": \"3de507ebca894100b20ae4a1645724d7\"}\n"
},
{
"name": "pipeline.log",
"path": "very_long_content"
}
],
"total_block_run_log_count": 2,
"total_pipeline_run_log_count": 2
}
],
"metadata": {}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/oauth-access-tokens/overview
Fetch OAuth access tokens.
### OAuth access token object
```json Response theme={"system"}
{
"oauth_access_tokens": [
{
"expires": "2023-08-18 23:41:52.480598+00:00",
"token": "JS3Umh826UoZzbuGQUsj36PSBwruVCzQYlRFf3KmJWA",
"user_id": 2
}
],
"metadata": {
"count": 1,
"next": false
}
}
```
Datetime in UTC when the access token expires and is no longer valid to use.
The token used to make authenticated API requests to Mage.
ID of user that the OAuth access token belongs to.
# Read OAuth tokens
Source: https://docs.mage.ai/extensibility/api-reference/oauth-access-tokens/read-oauth-access-tokens
GET /api/oauth_access_tokens
`GET /api/oauth_access_tokens`
Maximum number of logs to be returned in the API response.
Example: `20`
Read logs after N number of logs, where N equals `_offset`.
Example: `10`
If `show_all` isnโt present, then only show active OAuth access tokens for the current user.
If `show_all` is `1`, then show active and expired OAuth access tokens for the current user.
Example: `1`
```bash Request theme={"system"}
curl --request GET \
--url 'http://localhost:6789/api/pipelines/oauth_access_tokens/logs?_limit=20&_offset=0&show_all=1&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'OAUTH-TOKEN=some_really_long_string' \
```
```json Response theme={"system"}
{
"oauth_access_tokens": [
{
"expires": "2023-08-18 23:41:52.480598+00:00",
"token": "JS3Umh826UoZzbuGQUsj36PSBwruVCzQYlRFf3KmJWA",
"user_id": 2
},
{
"expires": "2023-08-18 23:41:51.418942+00:00",
"token": "bScSh2uTtnjMlfOjMqyytvbXj1HHQ-CJJt17qIzaP9g",
"user_id": 2
},
{
"expires": "2023-08-11 18:37:52.832016+00:00",
"token": "ZBVozeqrPgnF-CoaUHEzMW1BbDJS2o66EJ6T2XOk2xE",
"user_id": 2
},
{
"expires": "2023-08-07 20:34:40.608736+00:00",
"token": "dAq4jS5XaPrntFMIcQMNaXpKEUyi2ajAsgqAWXFw1fI",
"user_id": 2
},
{
"expires": "2023-08-04 20:28:16.966716+00:00",
"token": "tQYy-W363hpWK7JRHur8gOFb-3BFe6Tp9afWDGtrQh4",
"user_id": 2
},
{
"expires": "2023-08-01 01:24:37.132478+00:00",
"token": "dUSVnZc_8JirhcB1QHXqyFTIUFwwpoWLdHT4I8Fuxmc",
"user_id": 2
},
{
"expires": "2023-08-01 01:24:36.096940+00:00",
"token": "Uv2C4YklZaKtEPFigzGblFZ8fCx33598ZvZ3JJCyDiw",
"user_id": 2
}
],
"metadata": {
"count": 7,
"next": false
}
}
```
# API overview
Source: https://docs.mage.ai/extensibility/api-reference/overview
Our API powers everything in the appโ our UI makes calls to the backend, which operates the app. You can learn more about how these work and make your own calls in this section.
Want to jump-in and interact with Mage via API? Follow the [quickstart](#quickstart) below to get going! Otherwise, head to the [components](#components) section to learn about how our API is structuredโ this will be helpful if you're interested in adding new API endpoints or developing Mage.
You can read more about the components in the *Design* section on the sidebar to the leftโ it contains pages for [Resources](api-reference/resources), [Policies](api-reference/policies), and [Presenters](api-reference/presenters), which are the three components that make up an API endpoint.
## Quickstart
You'll need your API key to authenticate calls. You can find your API key quickly by navigating to the GUI and checking the network request log from the container.
You can read more about how to do this on Chromium based browsers [here](https://developers.google.com/web/tools/chrome-devtools/network/reference#headers).
Once you have your API key, we recommend storing it as a secret and using a solution like [dotenv](https://pypi.org/project/python-dotenv/) to load it into your environment. This will prevent you from exposing it as plain text.
To authenticate your session, we recommend using the `oauth_token` cookie. This can be done simply in Python requests with the following, where `token` is the value of the `token` key in the response payload:
```python theme={"system"}
token = mage_session.get('token')
s.headers['Cookie'] = f'oauth_token={token}'
```
With your API session configured, you can now make API calls! Here's an example of how to list all pipelines:
```python theme={"system"}
pipelines_url = base_url + '/api/pipelines'
r = s.get(pipelines_url)
list_pipelines = r.json()
print(list_pipelines)
```
## Authentication
Most API endpoints require authentication. The API request must include the following:
1. API key
2. OAuth token
### API key
The API request must include the API key in the query parameter as `api_key` or in the request payload body as `api_key`. You can find your API key quickly by navigating to the GUI and checking the request log from the container or going to the "Network" tab in your browser's developer tools.
### OAuth token
The OAuth token is generated by creating a new [session](/api-reference/sessions). Sessions are only required when `REQUIRE_USER_AUTHENTICATION` is enabled for your projectโ that is, when users must login to access Mage.
The token provided in the session payload is the **raw** value. There are two ways to authenticate with this token:
1. Supply the raw token in the header as:
```json theme={"system"}
{"Cookie": "oauth_token=[RAW-TOKEN-FROM-RESPONSE]"}
```
2. Supply the decoded token in the header as:
```json theme={"system"}
{"OAUTH-TOKEN": "[DECODED-TOKEN-FROM-RESPONSE]"}
```
To decode a token:
```python Python theme={"system"}
from mage_ai.authentication.oauth2 import decode_token
decoded_token = decode_token(token_from_session_response) ['token']
```
### Examples
Decoded token, API key in in URL:
```curl theme={"system"}
curl -X GET "localhost:6789/api/pipelines?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2" \
-H "OAUTH-TOKEN: [DECODED-TOKEN-FROM-RESPONSE]"
```
Decoded token, API key in body:
```curl theme={"system"}
curl -X POST "localhost:6789/api/pipelines" \
-H "OAUTH-TOKEN: [DECODED-TOKEN-FROM-RESPONSE]" \
-d '{
"api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
"pipeline": {}
}'
```
Raw token, API key in body:
```curl theme={"system"}
curl -X POST "localhost:6789/api/pipelines" \
-H "Cookie: oauth_token=[RAW-TOKEN-FROM-RESPONSE]" \
-d '{
"api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2",
"pipeline": {}
}'
```
### Responses
*All* our responses return a `200` status in the main payload. The any error code will be stored in the `error` key. For example, the following is a response from a failed session auth:
```json theme={"system"}
{
"error": {
"code": 404,
"message": "Email/username and/or password invalid.",
"type": "record_not_found"
},
"status": 200
}
```
To check for an error, one might try a solution like the following:
```python Python theme={"system"}
err_code = r.json().get('error').get('code'):
```
## Components
If you're not developing our API, you can jump right in and start querying endpoints using the [quickstart](#quickstart) above. Otherwise, our API is based on three components:
1. [Resources](/api-reference/resources)
2. [Policies](/api-reference/policies)
3. [Presenters](/api-reference/presenters)
If you'd like to contribute a new endpoint, you'll need to create one of each. To update an endpoint, it's important to understand how they work together.
**Resources** are the core of the API. They define the endpoints and the logic behind them. They are the only component that is required to create a new endpoint. They are also the only component that is required to update an existing endpoint.
**Policies** determine what actions can be taken on a resource: what attributes can be read, and which attributes can be written.
**Presenters** determine what attributes of the resource are returned to the client in the response. You can have different sets of attributes be included in the response based on the action (e.g. create, detail, delete, list, update) or a custom format.
To see examples of each, take a look at the `mage-ai/mage_ai/api` directory [here](https://github.com/mage-ai/mage-ai/tree/master/mage_ai/api).
## Testing
Create a test file in `mage_ai/tests/api/operations/test_[resource_name].py`;
replace `resource_name` with the name of your resource.
Paste in the following sample code into your file:
```python theme={"system"}
from mage_ai.orchestration.db.models.oauth import User
from mage_ai.tests.api.operations.base import BaseApiTestCase
from mage_ai.tests.factory import create_user
class SomeResourceOperationTests(BaseApiTestCase):
model_class = User
async def test_execute_delete(self):
user = create_user(email=self.faker.email())
await self.base_test_execute_delete(user.id)
async def test_execute_detail(self):
user = create_user()
await self.base_test_execute_detail(user.id, dict(
email=user.email,
username=user.username,
))
```
This is only a sample test file. Itโs running tests on the `UserResource`.
Youโll need to update the test logic to unit test your custom API endpoint.
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-runs/overview
Status and metrics for an individual run of a pipeline.
### Pipeline run object
```json Response theme={"system"}
{
"backfill_id": null,
"block_runs": [
{
"block_uuid": "load_titanic",
"completed_at": "2023-02-16 06:40:49.769087+00:00",
"created_at": "2023-02-16 06:40:43.575738+00:00",
"id": 33,
"metrics": null,
"pipeline_run_id": 23,
"started_at": "2023-02-16 06:40:48.816197+00:00",
"status": "completed",
"updated_at": "2023-02-16 06:40:48.873526+00:00"
}
],
"block_runs_count": 4,
"completed_at": "2023-02-16 06:41:19.249956+00:00",
"created_at": "2023-02-16 06:40:38.681910+00:00",
"event_variables": null,
"execution_date": "2023-02-16 04:41:03.300124+00:00",
"id": 23,
"metrics": null,
"passed_sla": false,
"pipeline_schedule_id": 1,
"pipeline_schedule_name": "late forest",
"pipeline_schedule_token": "260385a38d62466fb1c88fa8c5ca503f",
"pipeline_schedule_type": "time",
"pipeline_uuid": "example_pipeline",
"status": "completed",
"updated_at": "2023-02-16 06:41:09.154102+00:00",
"variables": null
}
```
Backfill that this run belongs to.
The block runs for this specific pipeline run.
Block UUID this block run belongs to.
Date and time the block run finished running successfully.
Date and time the block run was created.
ID of block run.
Various metrics related to the block run; e.g. error messages.
Pipeline run ID that this block run belongs to.
Date and time the block run started running.
`initial`, `queued`, `running`, `completed`, `failed`, `cancelled`, `upstream_failed`
Date and time the block run record was most recently updated.
Number of block runs for this pipeline run.
Date and time the pipeline run finished running successfully.
Date and time the pipeline run was created.
When pipeline run is triggered from an API request, the API request payload is saved in the
pipeline runโs `event_variables` column.
Date and time the pipeline run started running.
ID of pipeline run.
Various metrics related to the pipeline run; e.g. number of rows processed in a
data integration pipeline sync.
Whether or not the pipeline run has passed the configured SLA.
Pipeline schedule ID that this pipeline run belongs to.
Name of the pipeline schedule this pipeline run belongs to.
Unique token of the pipeline schedule that this pipeline run belongs to.
Pipeline schedule type that this pipeline run belongs to.
Pipeline UUID that this pipeline run belongs to.
`initial`, `running`, `completed`, `failed`, `cancelled`
Date and time the pipeline run record was most recently updated.
Object containing variables for the pipeline that is being overwritten by this pipeline run.
The property name is user defined.
# Read pipeline runs
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-runs/read-pipeline-runs
GET /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs
`GET /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs`
Pipeline schedule ID that pipeline runs should all belong to.
Maximum number of logs to be returned in the API response.
Example: `20`
Read logs after N number of logs, where N equals `_offset`.
Example: `10`
```curl cURL theme={"system"}
curl --request GET \
--url 'http://localhost:6789/api/pipeline_schedules/1/pipeline_runs?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'OAUTH-TOKEN=some_really_long_string' \
```
```json Response theme={"system"}
{
"pipeline_runs": [
...
],
"metadata": {
"count": 22,
"next": false
}
}
```
# Trigger pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-runs/trigger-pipeline
POST /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs/:token
`POST /api/pipeline_schedules/:pipeline_schedule_id/pipeline_runs/:token`
Run a pipeline using an API request. For more information,
read this [guide](/orchestration/triggers/trigger-pipeline-api).
Pipeline schedule ID that pipeline runs should all belong to.
Unique token of the pipeline schedule that you want to trigger.
```curl cURL theme={"system"}
curl --request POST \
--url 'http://localhost:6789/api/pipeline_schedules/71/pipeline_runs/4506463a01944ef492f70005996dac1a' \
--data '{
"env": "dev",
"test": 1
}'
```
```json Response theme={"system"}
{
"pipeline_run": {
"id": 1249,
"created_at": "2023-04-01 00:36:23.507282+00:00",
"updated_at": "2023-04-01 00:36:39.962084+00:00",
"pipeline_schedule_id": 71,
"pipeline_uuid": "example_pipeline",
"execution_date": "2023-04-01 00:36:39.904371+00:00",
"status": "running",
"completed_at": null,
"variables": {},
"passed_sla": false,
"event_variables": {
"env": "dev",
"test": 1
},
"metrics": null,
"backfill_id": null
}
}
```
# Create pipeline schedule
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/create-pipeline-schedule
POST /api/pipelines/:pipeline_uuid/pipeline_schedules
`POST /api/pipelines/:pipeline_uuid/pipeline_schedules`
Pipeline UUID that the pipeline schedule should all belong to.
```bash Request theme={"system"}
curl --request POST
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=some_really_long_string'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
-d '{
"pipeline_schedule": {
"name": "Example Pipeline Schedule Name",
"schedule_type": "time",
"event_matchers": [],
"schedule_interval": "@once",
"start_time": "2023-03-06 04:53:00",
"variables": {
"env": "dev112222",
"test": 11111
},
"sla": 1000,
"settings": {
"allow_blocks_to_fail": true,
"create_initial_pipeline_run": false,
"skip_if_previous_running": true
}
}
}'
--url http://localhost:6789/api/pipelines/example_pipeline/pipeline_schedules
```
```json Response theme={"system"}
{
"pipeline_schedule": {
"id": 55,
"created_at": "2023-03-08 04:52:54.268096+00:00",
"updated_at": "2023-04-01 00:52:11.753497+00:00",
"name": "Example Pipeline Schedule Name",
"pipeline_uuid": "example_pipeline",
"schedule_type": "time",
"start_time": "2023-03-06 04:53:00+00:00",
"schedule_interval": "@once",
"status": "inactive",
"variables": {
"env": "dev112222",
"test": 11111
},
"sla": 1000,
"token": "67d62ed3e66143839f58945bb7d16387",
"settings": {
"allow_blocks_to_fail": true,
"create_initial_pipeline_run": false,
"skip_if_previous_running": true
},
"event_matchers": []
}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/overview
Triggers for a pipeline.
### Pipeline schedule object
```json Response theme={"system"}
{
"created_at": "2023-03-01 20:27:23.645692+00:00",
"event_matchers": [],
"id": 41,
"last_pipeline_run_status": null,
"name": "dark brook",
"pipeline_runs_count": 0,
"pipeline_uuid": "dry_bird",
"schedule_interval": "@once",
"schedule_type": "time",
"settings": {
"allow_blocks_to_fail": true,
"create_initial_pipeline_run": true,
"skip_if_previous_running": false
},
"sla": 950400,
"start_time": "2023-03-01 22:04:00+00:00",
"status": "inactive",
"tags": ["core data"],
"token": "1dca215c01564d48bf0d3db7d5450a20",
"updated_at": "2023-03-01 22:04:29.761238+00:00",
"variables": {
"env": "dev"
}
}
```
Date and time the trigger was created.
Event matching logic for event triggers.
`aws_event`
Event matcher ID.
Name of event matcher.
Trigger ID.
Name of trigger.
Pipeline UUID the trigger belongs to.
`@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`
Trigger type: `api`, `event`, `time`
Additional settings for the trigger.
If `true`, then blocks in the pipeline will continue to run even if unrelated blocks fail.
If `true`, then an initial pipeline run will be created immediately after enabling the
trigger if the trigger's current execution date is after its start date time but before
current time.
For example, if the current time is `1/4/24 12:00 UTC` and you create a daily trigger, the
current execution date would be `2024-01-04 00:00`. If the start date was set to
`1/3/24 12:00 UTC`, which is before the current execution date, and the `create_initial_pipeline_run`
setting was enabled, a pipeline run with execution date `2024-01-04 00:00` would get created
immediately.
If `true`, then new pipeline runs wonโt be triggered until the most recent pipeline run is
no longer running.
If the pipeline run has been running for more than the SLA value, then the pipeline will fail.
Date and time when the trigger should start running.
`active`, `inactive`
An array of tag names.
Unique string used in the URL when triggering pipeline via API request.
Date and time the trigger was last updated.
Object containing variables for the pipeline that is being overwritten by this trigger.
The property name is user defined.
# Read pipeline schedules
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/read-pipeline-schedules
GET /api/pipelines/:pipeline_uuid/pipeline_schedules
`GET /api/pipelines/:pipeline_uuid/pipeline_schedules`
Pipeline UUID that the pipeline schedules should all belong to.
Fetch pipeline schedules that have the specified tag.
If you include 2 or more `tag[]` query parameters,
then the pipeline schedules will be fetched if it has at
least 1 of the tags specified.
```curl cURL theme={"system"}
curl --request GET \
--url 'http://localhost:6789/api/pipelines/:pipeline_uuid/pipeline_schedules?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'Authorization: Bearer ZKXScwLKLZUujbw7vwMeWnhtTTSLqifw8_RnBHYipww' \
```
```json Response theme={"system"}
{
"pipeline_schedules": [
{
"id": 42,
"created_at": "2023-03-01 22:04:33.609624+00:00",
"updated_at": "2023-03-01 22:04:35.163834+00:00",
"name": "dry darkness",
"pipeline_uuid": "dry_bird",
"schedule_type": "time",
"start_time": "2023-03-01 22:04:00+00:00",
"schedule_interval": "@hourly",
"status": "inactive",
"variables": null,
"sla": null,
"tags": ["core data"],
"token": "f176321c4eac425782236acd0eb9886d",
"settings": {},
"event_matchers": [],
"last_pipeline_run_status": null,
"pipeline_runs_count": 0
},
{
"id": 41,
"created_at": "2023-03-01 20:27:23.645692+00:00",
"updated_at": "2023-03-31 05:15:51.171762+00:00",
"name": "dark brook",
"pipeline_uuid": "dry_bird",
"schedule_type": "time",
"start_time": "2023-03-01 22:04:00+00:00",
"schedule_interval": "@once",
"status": "inactive",
"variables": {
"env": "dev2"
},
"sla": 950400,
"tags": [],
"token": "1dca215c01564d48bf0d3db7d5450a20",
"settings": {
"allow_blocks_to_fail": true,
"create_initial_pipeline_run": false,
"skip_if_previous_running": true
},
"event_matchers": [],
"last_pipeline_run_status": null,
"pipeline_runs_count": 0
}
],
"metadata": {
"count": 3,
"next": false
}
}
```
# Update schedule
Source: https://docs.mage.ai/extensibility/api-reference/pipeline-schedules/update-pipeline-schedule
PUT /api/pipeline_schedules/:id
Pipeline schedule ID
Body
```curl cURL theme={"system"}
curl --request PUT \
--url 'http://localhost:6789/api/pipeline_schedules/55?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'OAUTH-TOKEN=some_really_long_string' \
--data '{
"pipeline_schedule": {
"name": "black voice11111",
"schedule_type": "time",
"event_matchers": [],
"schedule_interval": "@once",
"start_time": "2023-03-06 04:53:00",
"variables": {
"env": "dev112222",
"test": 11111
},
"sla": 1000,
"settings": {
"allow_blocks_to_fail": true,
"create_initial_pipeline_run": false,
"skip_if_previous_running": true
}
}
}'
```
```json theme={"system"}
{
"pipeline_schedule": {
"id": 55,
"created_at": "2023-03-08 04:52:54.268096+00:00",
"updated_at": "2023-04-01 00:52:11.753497+00:00",
"name": "black voice11111",
"pipeline_uuid": "example_pipeline",
"schedule_type": "time",
"start_time": "2023-03-06 04:53:00+00:00",
"schedule_interval": "@once",
"status": "inactive",
"variables": {
"env": "dev112222",
"test": 11111
},
"sla": 1000,
"token": "67d62ed3e66143839f58945bb7d16387",
"settings": {
"allow_blocks_to_fail": true,
"create_initial_pipeline_run": false,
"skip_if_previous_running": true
},
"event_matchers": []
}
}
```
# Create pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/create-pipeline
POST /api/pipelines
`POST /api/pipelines`
Human readable name of the pipeline.
The type of the pipeline:
`integration`, `pyspark`, `python`, `streaming`
Note that `python` is a standard (batch) pipeline with a python backend, while `pyspark` is a batch pipeline with a spark backend.
If supplied, the uuid of a pipeline to clone.
Array of [extension](/design/blocks/extension) block objects. Same shape as `blocks`.
Array of [callback](/development/blocks/callbacks) block objects. Same shape as `blocks`.
Array of [conditional](/development/blocks/conditionals/) block objects. Same shape as `blocks`.
```bash Request theme={"system"}
curl --request POST
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=some_really_long_string'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
-d '{"pipeline": {"name": "arwen-starlight", "type": "python"}}'
--url http://localhost:6789/api/pipelines
```
```json Response theme={"system"}
{
"pipeline": {
"data_integration": null,
"description": null,
"executor_config": {},
"executor_count": 1,
"executor_type": null,
"name": "arwen-starlight",
"notification_config": {},
"type": "python",
"updated_at": null,
"uuid": "arwen_starlight",
"spark_config": {},
"blocks": [],
"callbacks": [],
"conditionals": [],
"widgets": []
}
}
```
# Delete pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/delete-pipeline
DELETE /api/pipelines/:uuid
`DELETE /api/pipelines/:uuid`
```bash Request theme={"system"}
curl -X DELETE
--url http://localhost:6789/api/pipelines/arwen_starlight
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=some_really_long_string'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
```
```json Response theme={"system"}
{
"pipeline": {
"data_integration": null,
"description": null,
"executor_config": {},
"executor_count": 1,
"executor_type": null,
"name": "arwen-starlight",
"notification_config": {},
"type": "integration",
"updated_at": null,
"uuid": "arwen_starlight",
"spark_config": {},
"blocks": [],
"callbacks": [],
"conditionals": [],
"widgets": []
}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/overview
Data pipeline containing execution settings, resources, block information, and block dependencies.
## Pipeline object
```json Response theme={"system"}
{
"blocks": [
{
"all_upstream_blocks_executed": true,
"color": null,
"configuration": {},
"downstream_blocks": [],
"executor_config": null,
"executor_type": "local_python",
"has_callback": null,
"name": "load_titanic",
"language": "python",
"status": "executed",
"type": "data_loader",
"upstream_blocks": [],
"uuid": "load_titanic",
"content": "...",
"metadata": {}
}
],
"description": null,
"extensions": {},
"name": "example_pipeline",
"schedules": [
{
"created_at": "2023-03-14T23:24:17.814478+00:00",
"id": 59,
"name": "dry haze",
"pipeline_uuid": "aged_night",
"schedule_interval": null,
"schedule_type": "api",
"start_time": "2023-03-14T23:25:00+00:00",
"status": "inactive",
"updated_at": "2023-03-14T23:25:27.351528+00:00"
}
],
"type": "python",
"uuid": "example_pipeline",
"variables": {
"env": "prod"
},
"widgets": []
}
```
Array of block objects. See the [blocks](/api-reference/blocks/overview) section for more details.
Description for the pipeline.
Array of extension block objects. Same shape as `blocks`.
Human friendly name of the pipeline.
Array of trigger objects.
Date and time the trigger was created.
Trigger ID.
Name of trigger.
Pipeline UUID the trigger belongs to.
`@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`
Trigger type: `api`, `event`, `time`
Date and time when the trigger should start running.
`active`, `inactive`
Date and time the trigger was last updated.
The type of the pipeline:
`integration`, `pyspark`, `python`, `streaming`
Note that `python` is a standard (batch) pipeline with a python backend, while `pyspark` is a batch pipeline with a spark backend.
Unique identifier for the pipeline.
Object containing variables for the pipeline.
The property name is user defined.
Array of widget block objects. Same shape as `blocks`.
# Read pipelines
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/read-pipeline
GET /api/pipelines
To retrieve all pipelines:
`GET /api/pipelines`
To fetch a single pipeline:
`GET /api/pipelines/:uuid`
If `1`, the pipeline object will include associated triggers.
```bash Request theme={"system"}
curl --request GET \
--url 'http://localhost:6789/api/pipelines/arwen_starlight?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'OAUTH-TOKEN=some_really_long_string'
```
```json Response theme={"system"}
{
"pipeline": {
"data_integration": null,
"description": null,
"executor_config": {},
"executor_count": 1,
"executor_type": null,
"name": "arwen-starlight",
"notification_config": {},
"type": "python",
"updated_at": null,
"uuid": "arwen_starlight",
"spark_config": {},
"blocks": [],
"callbacks": [],
"conditionals": [],
"widgets": [],
"extensions": {}
}
```
# Update pipeline
Source: https://docs.mage.ai/extensibility/api-reference/pipelines/update-pipeline
PUT /api/pipelines/:uuid
`PUT /api/pipelines/:uuid`
All default attributes for a pipeline may be updated in addition to the following:
The uuid for a block upstream of the pipeline.
Array of block objects. See the [blocks](/api-reference/blocks/overview) section for more details.
Array of [callback](/development/blocks/callbacks) block objects. Same shape as `blocks`.
Array of [conditional](/development/blocks/conditionals/) block objects. Same shape as `blocks`.
Array of [extension](/design/blocks/extension) block objects. Same shape as `blocks`.
Array of [schedule](/guides/triggering-pipelines/) trigger objects.
```bash Request theme={"system"}
curl --request PUT
--url http://localhost:6789/api/pipelines/arwen_starlight
-H 'Content-Type: application/json'
-H 'Cookie: oauth_token=some_really_long_string'
-H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
-d '{"pipeline": {"type": "integration"}}'
```
```json Response theme={"system"}
{
"pipeline": {
"data_integration": null,
"description": null,
"executor_config": {},
"executor_count": 1,
"executor_type": null,
"name": "arwen-starlight",
"notification_config": {},
"type": "integration",
"updated_at": null,
"uuid": "arwen_starlight",
"spark_config": {},
"blocks": [],
"callbacks": [],
"conditionals": [],
"widgets": [],
"extensions": {}
}
}
```
# API policies
Source: https://docs.mage.ai/extensibility/api-reference/policies
Policies for resources determine what actions can be taken on the resource, what attributes can be read, and which attributes can be written.
Define your policy by creating a file `api/policies/SwordPolicy`:
```python theme={"system"}
from .BasePolicy import BasePolicy
class SwordPolicy(BasePolicy):
pass
```
## API actions
There are 5 actions you can permit:
1. Create: `operations.CREATE` aka `SwordResource.create` method
2. Delete: `operations.DELETE` aka `SwordResource().delete` method
3. Detail: `operations.DETAIL` aka `SwordResource.member` method
4. List: `operations.LIST` aka `SwordResource.collection` method
5. Update: `operations.UPDATE` aka `SwordResource().update` method
## API scopes
There are 2 scopes you can permit:
1. Private: `OauthScope.CLIENT_PRIVATE`; user must be logged in to have access
2. Public: `OauthScope.CLIENT_PUBLIC`; user must be logged out to have access
## Permit CRUD actions
Define which actions a user can perform:
```python theme={"system"}
from api import operations
from api.oauth_scope import OauthScope
class SwordPolicy(BasePolicy):
pass
SwordPolicy.allow_actions([
operations.CREATE,
operations.LIST,
], scopes=[
OauthScope.CLIENT_PRIVATE,
])
```
If you want to permit certain actions for a user only if a condition is met, do the following:
```python theme={"system"}
SwordPolicy.allow_actions([
operations.DELETE,
operations.DETAIL,
operations.UPDATE,
], scopes=[
OauthScope.CLIENT_PRIVATE,
], condition=lambda policy: policy.current_user.is_staff)
```
## Permit query parameters
Define which URL query parameters a user is allowed to use:
```python theme={"system"}
SwordPolicy.allow_query([
'is_powerful',
], scopes=[
OauthScope.CLIENT_PRIVATE,
])
```
If you want to permit a different query parameter for a specific action, do the following:
```python theme={"system"}
SwordPolicy.allow_query([
'is_super_powerful',
], scopes=[
OauthScope.CLIENT_PRIVATE,
], on_action=[
operations.LIST,
])
```
You can also add a condition to these permissions:
```python theme={"system"}
SwordPolicy.allow_query([
'is_super_powerful',
], scopes=[
OauthScope.CLIENT_PRIVATE,
], on_action=[
operations.LIST,
], condition=lambda policy: policy.current_user.is_staff)
```
## Permit read attributes
Define which attributes of the resource the current user can read:
```python theme={"system"}
class SwordPolicy(BasePolicy):
def belongs_to_current_user(self):
return self.current_user.id == self.resource.user_id
SwordPolicy.allow_read([
'id',
'name',
'strength',
], scopes=[
OauthScope.CLIENT_PRIVATE,
], on_action=[
operations.DETAIL,
operations.LIST,
operations.UPDATE,
], condition=lambda policy: policy.belongs_to_current_user())
```
## Permit write attributes
Define which attributes of the resource the current user can write to when creating a new
resource or updating an existing resource:
```python theme={"system"}
# Allow writing the name and strength attribute when creating a new resource
SwordPolicy.allow_write([
'name',
'strength',
], scopes=[
OauthScope.CLIENT_PRIVATE,
], on_action=[
operations.CREATE,
])
# Only allow writing the strength attribute when updating the resource
SwordPolicy.allow_write([
'strength',
], scopes=[
OauthScope.CLIENT_PRIVATE,
], on_action=[
operations.UPDATE,
], condition=lambda policy: policy.belongs_to_current_user())
```
# API presenters
Source: https://docs.mage.ai/extensibility/api-reference/presenters
This will determine what attributes of the resource are returned to the client in the response. You can have different sets of attributes be included in the response based on the action (e.g. create, detail, delete, list, update) or a custom format.
Define your presenter by creating a file `api/presenters/SwordPresenter`:
```python theme={"system"}
from .BasePresenter import BasePresenter
class SwordPresenter(BasePresenter):
default_attributes = [
'id',
'name',
'strength',
]
```
By default, whenever a sword is being returned in an API response, it will return a sword object in
JSON format with the `id`, `name`, and `strength` keys.
## Register different formats for different actions
If you want the resource to return different data, you can register unique formats and the
attributes that format should return:
```python theme={"system"}
from .BasePresenter import BasePresenter
from api import operations
class SwordPresenter(BasePresenter):
default_attributes = [
'id',
'name',
'strength',
]
SwordPresenter.register_formats([
operations.LIST,
], [
'id',
])
```
This will only return the `id` attribute when making an API call to collections endpoint
`/swords` to retrieve a list of sword resources.
## Register custom formats
You may want to define a custom format that can return specific attributes when making an API
call by including the `_format` URL parameter. For example, you can present specific fields by
calling the `GET /swords?_format=all_attributes` endpoint.
```python theme={"system"}
from .BasePresenter import BasePresenter
from api import operations
class SwordPresenter(BasePresenter):
default_attributes = [
'id',
'name',
'strength',
]
SwordPresenter.register_formats([
'all_attributes',
], [
'created_at',
'deleted_at',
'id',
'name',
'strength',
'team',
'user',
])
```
Note: be sure to permit reading these attributes in the `SwordPolicy`.
# API resources
Source: https://docs.mage.ai/extensibility/api-reference/resources
This file handles the CRUD operations.
When you create a new API called `Swords`, the resource
name is `Sword` and the file name is `SwordResource`. The endpoint for CRUD operations are:
| Action | HTTP method | Endpoint |
| ------ | ----------- | ------------- |
| Create | `POST` | `/swords` |
| List | `GET` | `/swords` |
| Detail | `GET` | `/swords/:id` |
| Update | `PUT` | `/swords/:id` |
| Delete | `DELETE` | `/swords/:id` |
If your resource belongs to a parent model, for example a sword belongs to a team, then your
endpoints will be the following (note that swords for the create and list endpoints are nested
under the `teams` resource:
| Action | HTTP method | Endpoint |
| ------ | ----------- | ------------------- |
| Create | `POST` | `/teams/:id/swords` |
| List | `GET` | `/teams/:id/swords` |
| Detail | `GET` | `/swords/:id` |
| Update | `PUT` | `/swords/:id` |
| Delete | `DELETE` | `/swords/:id` |
## Collection
This method is executed when you make a request for a list of a specific resource.
Example: `GET /swords`.
Create a file called `SwordResource` that is a subclass of 1 of the following:
1. `GenericResource`: subclass this class if you want to execute arbitrary code
2. `DatabaseResource`: subclass this class only if your resource is directly associated to a
database model. This subclass has many methods defined by default that handles CRUD operations
automatically.
### Creating a subclass of `GenericResource`
Create a file called `backend/api/resources/SwordResource.py`:
```python theme={"system"}
from .GenericResource import GenericResource
import requests
class SwordResource(GenericResource):
@classmethod
def collection(self, query, meta, user, **kwargs):
"""
query:
a dictionary of URL query parameters
meta:
a dictionary of meta parameters from the URL; such as _limit and _format
user:
a user object that represents the current user who is making the API call
kwargs:
a dictionary of other values that are passed down through the API request lifecycle; such
as parent_model, etc.
"""
# Fetch a list of Swords from the internet
response = requests.get('https://www.dropbox.com/s/mnjsxzdsyc39hr3/IkeaStorageJson.json')
return self.build_result_set(response['swords'], user, **kwargs)
```
### Creating a subclass of `DatabaseResource`
```python theme={"system"}
from .DatabaseResource import DatabaseResource
from examples.models import Sword
class SwordResource(DatabaseResource):
# Set this class variable to the model that the resource is associated with
model_class = Sword
"""
Optionally override collection because the DatabaseResource will handle limits
and pagination automatically. If you donโt override the collection method, DatabaseResource
will automatically fetch a list of models from the database for you.
"""
@classmethod
def collection(self, query, meta, user, **kwargs):
"""
Use collection to handle pre-filtering of the results from the database.
"""
# Filter swords that are weak and belong to the current team
team = kwargs['parent_model']
return Sword.objects.filter(team_id=team.id, weak=False)
```
Here is how we would get a list of swords that are weak and belong to team ID 1:
```python theme={"system"}
user = User.objects.get(id=3)
team = Team.objects.get(id=1)
# You typically donโt have to call this function by itself.
# This function is called by making a request to /teams/1/swords.
swords = SwordResource.collection({}, {}, user, parent_model=team)
```
## Create
This method is executed when you make a request to create a new instance of a resource.
Example: `POST /swords`.
### `GenericResource`
```python theme={"system"}
from services.datadog import increment
class SwordResource(GenericResource):
# code from before
@classmethod
def create(self, payload, user, **kwargs):
"""
payload:
a dictionary of the request body
For example, if your request body is { "sword" : { "weak": false } },
then your payload will be the following dictionary: { 'weak': False }.
NOTE:
Payloads must be nested inside a key named after the singular form of the resource name.
user:
a user object that represents the current user who is making the API call
kwargs:
a dictionary of other values that are passed down through the API request lifecycle;
such as parent_model, etc.
"""
# Increment a create metric in DataDog
increment('swords.create')
if payload.get('weak'):
name = 'Weak sword'
else:
name = 'Strong sword'
# Always return a new instance of the resource
return self({ 'name': name }, user, **kwargs)
```
### `DatabaseResource`
You typically never have to override the `create` class method. The instances where it is overridden
include a scenario when you want to execute a callback function only if the resource was
successfully created.
The `DatabaseResource` `create` method will take the keys and values in the payload argument, assign
those values to a newly instantiated `Sword` object, and then save the model to the database.
```python theme={"system"}
class SwordResource(DatabaseResource):
# code from before
@classmethod
def create(self, payload, user, **kwargs):
def _upload(resource):
# upload something only if the resource was successfully created
self.on_create_callback = _upload
return super().create(payload, user, **kwargs)
```
## Member
This method is executed when you make a fetch a single resource using itโs ID.
Example: `GET /swords/:id`.
### `GenericResource`
```python theme={"system"}
class SwordResource(GenericResource):
# code from before
@classmethod
def member(self, pk, user, **kwargs):
"""
pk:
a string or integer from the URL parameter :id
user:
a user object that represents the current user who is making the API call
kwargs:
a dictionary of other values that are passed down through the API request lifecycle; such
as parent_model, etc.
"""
# Fetch 1 of Sword from the internet
array = requests.get('https://www.dropbox.com/s/mnjsxzdsyc39hr3/IkeaStorageJson.json')
sword = filter(lambda x: x['id'] == pk, array)
return self({ 'name': sword['name'], 'weak': False }, user, **kwargs)
```
### `DatabaseResource`
You almost never have to override the `member` method.
If you do, see `TrainingRunResource` for an example.
## Update
This method is executed when you want to update a single resource using itโs ID.
Example: `PUT /swords/:id`.
This is almost the same as the `create` method except itโs an instance method and the ID of the
resource you want to update must be included in the URL.
```python theme={"system"}
class SwordResource(GenericResource):
# code from before
def update(self, payload, **kwargs):
"""
payload:
a dictionary of the request body
For example, if your request body is { "sword" : { "weak": false } },
then your payload will be the following dictionary: { 'weak': False }.
NOTE:
Payloads must be nested inside a key named after the singular form of the resource name.
kwargs:
a dictionary of other values that are passed down through the API request lifecycle; such
as parent_model, etc.
"""
print('Awesome update!')
# Always return self
return self
```
### `DatabaseResource`
You almost never have to override the `update` method.
If you do, see `ModelVersionResource` for an example.
## Delete
This method is executed when you want to delete a single resource using itโs ID.
Example: `DELETE /swords/:id`.
```python theme={"system"}
class SwordResource(GenericResource):
# code from before
def delete(self, **kwargs):
"""
kwargs:
a dictionary of other values that are passed down through the API request lifecycle; such
as parent_model, etc.
"""
print('Deleted!')
```
### `DatabaseResource`
You almost never have to override the `delete` method.
## Registering parent models
If your resource belongs to a parent model, for example a sword belongs to a team, you must do
the following in order for these endpoints to work properly:
1. Create: `POST /teams/:id/swords`
2. List: `GET /teams/:id/swords`
```python theme={"system"}
import importlib
class SwordResource(DatabaseResource):
# code from before
SwordResource.register_parent_resource(
getattr(importlib.import_module('api.resources.TeamResource'), 'TeamResource'),
)
```
## Pre-load single associated object
The sword model belongs to a user via the `sword.user_id` column. If you are fetching a list of 100
swords and in your code you are accessing the methods of the `user` object via `sword.user`
(e.g. `sword.user.name`), you will inadvertently make 100 extra API calls to the `users`
database table. Do the following to prevent this:
```python theme={"system"}
import importlib
class SwordResource(DatabaseResource):
# code from before
SwordResource.register_collective_loader_find(
getattr(importlib.import_module('api.resources.UserResource'), 'UserResource'),
)
```
## Pre-load multiple associated objects
The sword has many opponents that were defeated by it. You can query all the opponents via the
`swords.opponents.objects.all()` method. This will fetch all opponents from the database where
`opponent.sword_id` equals `sword.id`. If you are accessing the opponents of each sword when
fetching a list of 100 swords, you will be making 100 extra API calls to the `opponents` database.
Do the following to prevent this:
```python theme={"system"}
import importlib
class SwordResource(DatabaseResource):
# code from before
SwordResource.register_collective_loader_select(
getattr(importlib.import_module('api.resources.OpponentResource'), 'OpponentResource'),
)
```
# Create session
Source: https://docs.mage.ai/extensibility/api-reference/sessions/create-session
POST /api/sessions
`POST /api/sessions`
```curl cURL theme={"system"}
curl --request POST \
--url http://localhost:6789/api/sessions \
--header 'Content-Type: application/json' \
--header 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--data '{
"session": {
"email": "admin@admin.com",
"password": "admin"
}
}'
```
```json Response theme={"system"}
{
"session": {
"expires": "2023-04-30 03:02:14.781970+00:00",
"token": "...",
"user": {
"avatar": null,
"first_name": null,
"id": 1,
"last_name": null,
"owner": true,
"roles": null,
"roles_display": "Owner",
"username": "admin"
}
}
}
```
# Overview
Source: https://docs.mage.ai/extensibility/api-reference/sessions/overview
Sign in and retrieve an OAuth token.
### Session object
```json Response theme={"system"}
{
"expires": "2023-04-30 03:02:14.781970+00:00",
"token": "...",
"user": {
"avatar": null,
"first_name": null,
"id": 1,
"last_name": null,
"owner": true,
"roles": null,
"roles_display": "Owner",
"username": "admin"
}
}
```
The date and time the session is no longer valid.
OAuth token used to authenticate each API request.
Object containing variables for the pipeline.
User ID
If true, the current user has owner permissions.
Roles a user has, represented as an integer.
Human-readable role.
Unique username for user.
# Pipeline configuration environment overrides
Source: https://docs.mage.ai/extensibility/env-config/pipeline
Add environment-specific overrides for your pipeline configuration.
### Location of pipeline configuration
Each pipeline in Mage has a `metadata.yaml` configuration file located in the
pipeline uuid's folder under the `pipelines` directory. For example, if you
had a pipeline with uuid `charismatic_inventor`, the folder structure for that
pipeline might look something like this:
```
your_project/
โโ pipelines/
โ โโ charismatic_inventor/
โ โ โโ __init__.py
โ โ โโ interactions.yaml
โ โ โโ metadata.yaml
โ โ โโ triggers.yaml
โ ...
```
### **Override pipeline config based on environment**
In the pipeline's `metadata.yaml` config file, add an `overrides` key at the top-level
(no indentations) with the name of your environment (e.g. `prod`, `dev`, `test`) under
the `overrides` key and indented once. Then under the environment name key, add the
properties of your base pipeline config that you want to override. Make sure the
indentations of the properties match those of the base config. Any environment-specific
overrides will REPLACE the matching property in the base config, so be careful when
overriding properties with nested values.
The environment name should match the environment defined in the `ENV`
[environment variable](https://docs.mage.ai/development/variables/environment-variables).
### Example pipeline config file with environment overrides
```yaml theme={"system"}
# your_project/pipelines/charismatic_inventor/metadata.yaml
blocks:
- all_upstream_blocks_executed: true
color: null
configuration:
disable_output_preview: false
file_source:
path: data_loaders/load_titanic.py
sample_count_preview: 0
downstream_blocks:
- starry_forest
executor_config: null
executor_type: local_python
has_callback: false
language: python
name: load_titanic
retry_config: {}
status: executed
timeout: null
type: data_loader
upstream_blocks: []
uuid: load_titanic
- all_upstream_blocks_executed: true
color: null
configuration:
sample_count_preview: 8
downstream_blocks: []
executor_config: null
executor_type: local_python
has_callback: false
language: python
name: starry forest
retry_config: {}
status: executed
timeout: null
type: custom
upstream_blocks:
- load_titanic
uuid: starry_forest
cache_block_output_in_memory: false
callbacks: []
concurrency_config:
pipeline_run_limit: 5
pipeline_run_limit_all_triggers: 5
conditionals: []
created_at: '2024-09-27 23:18:10.554952+00:00'
data_integration: null
description: null
executor_config: {}
executor_count: 1
executor_type: null
extensions: {}
name: charismatic inventor
notification_config: {}
remote_variables_dir: null
retry_config: {}
run_pipeline_in_one_process: false
settings:
triggers: null
spark_config: {}
state_store_config: {}
tags: []
type: python
uuid: charismatic_inventor
variables_dir: /root/.mage_data/default_repo
widgets: []
overrides:
prod:
concurrency_config:
pipeline_run_limit: 10
pipeline_run_limit_all_triggers: 20
description: Overridden description for charismatic inventor in prod
dev:
concurrency_config:
pipeline_run_limit: 1
pipeline_run_limit_all_triggers: 1
description: Overridden description for charismatic inventor in dev
```
In the example above when in the `dev` environment, the pipeline's configuration property
of `concurrency_config` will have its `pipeline_run_limit` and `pipeline_run_limit_all_triggers`
properties overridden to be `1` (instead of `5` as defined in the base config).
The `description` property will also be overridden to be
`Overridden description for charismatic inventor in dev` instead of `null`.
Similarly for the `prod` environment, `pipeline_run_limit` will be replaced with `10` and
`pipeline_run_limit_all_triggers` with `20`. The `description` property will be overridden
to be `Overridden description for charismatic inventor in prod`. Other environments
(not `dev` or `prod`) will not utilize the `overrides` section.
# Project configuration environment overrides
Source: https://docs.mage.ai/extensibility/env-config/project
Add environment-specific overrides for your project configuration.
### Location of project configuration
Each project in Mage has a `metadata.yaml` configuration file located in the
project's root directory. For example, if you had a project called `your_project`,
the folder structure for that project might look something like this:
```
your_project/
โโ pipelines/
โโ data_loaders/
โโ transformers/
โโ data_exporters/
โโ .../
โโ io_config.yaml
โโ metadata.yaml
โโ requirements.txt
โโ ...
```
### **Override project config based on environment**
In the project's `metadata.yaml` config file, add an `overrides` key at the top-level
(no indentations) with the name of your environment (e.g. `prod`, `dev`, `test`) under
the `overrides` key and indented once. Then under the environment name key, add the
properties of your base project config that you want to override. Make sure the
indentations of the properties match those of the base config. Any environment-specific
overrides will REPLACE the matching property in the base config, so be careful when
overriding properties with nested values.
The environment name should match the environment defined in the `ENV`
[environment variable](https://docs.mage.ai/development/variables/environment-variables).
### Example project config file with environment overrides
```yaml theme={"system"}
# your_project/metadata.yaml
project_type: standalone
variables_dir: ~/.mage_data
remote_variables_dir: s3://bucket/path_prefix
variables_retention_period: '90d'
emr_config:
master_instance_type: 'r5.4xlarge'
slave_instance_type: 'r5.4xlarge'
master_security_group: 'sg-xxxxxxxxxxxx'
slave_security_group: 'sg-yyyyyyyyyyyy'
ec2_key_name: '[ec2_key_pair_name]'
spark_config:
app_name: 'my spark app'
spark_master: 'local'
executor_env: {}
spark_jars: []
spark_home:
others: {}
use_custom_session: false
custom_session_var_name: 'spark'
help_improve_mage: true
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
slack_config:
webhook_url: "{{ env_var('MAGE_SLACK_WEBHOOK_URL') }}"
teams_config:
webhook_url: "{{ env_var('MAGE_TEAMS_WEBHOOK_URL') }}"
project_uuid: 123456781234abcd1234abcde123456
features:
add_new_block_v2: true
code_block_v2: true
command_center: true
custom_design: true
data_integration_in_batch_pipeline: true
dbt_v2: true
interactions: true
display_local_timezone: true
notebook_block_output_split_view: true
operation_history: true
polars: true
automatic_kernel_cleanup: false
pipelines:
settings:
triggers:
save_in_code_automatically: true
overrides:
dev:
help_improve_mage: false
pipelines:
settings:
triggers:
save_in_code_automatically: false
features:
automatic_kernel_cleanup: true
global_hooks: true
```
In the example above when in the `dev` environment, the project's nested configuration
property of `save_in_code_automatically` will be overridden to be `false` (instead of
`true` as defined in the base config). The other properties (i.e. `help_improve_mage`,
`automatic_kernel_cleanup`, and `global_hooks`) will also be overridden.
Other environments (not `dev`) will not utilize the `overrides` section.
# Global data products
Source: https://docs.mage.ai/extensibility/global-data-products/overview
Generate and orchestrate the final output of a pipeline (aka data product). Data products can be globally referenced from any pipeline and its data output can be used in any block.
## Overview
A data product is any piece of data created by 1 or more blocks in a pipeline.
For example, a block can create a data product that is an in-memory DataFrame,
or a JSON serializable data structure, or a table in a database.
A global data product is a data product that can be referenced and used in any pipeline across the
entire project.
A global data product is entered into the global registry (`global_data_products.yaml`)
under a unique ID (UUID) and it references an existing pipeline.
This feature is important because multiple pipelines can depend on a single global data product,
without having to regenerate the global data product.
In addition, the global data product doesnโt have to run unless something needs its data and
its data is outdated. Its data is outdated if it hasnโt ran for a preset amount of time.
This preset time is configurable.
Example
If you have a computationally expensive data pipeline called `users_ltv`
that generates the lifetime value (LTV) of each user, and if you have 2 downstream pipelines
that require the data from the `users_ltv` pipeline, then using a global data product
will make sure that the `users_ltv` pipeline is ran only once as long as the `users_ltv`
data product isnโt outdated.
### Minimize number of runs
A global data product is configured to be outdated after a certain amount of time has passed
since its most recent successful run.
It can also be configured to be outdated starting at a specific time of the
year, month, week, day, hour, minute, etc.
A global data product wonโt regenerate unless its been outdated.
### Lazy triggering
A global data product wonโt run unless something requires its data. For example, if a block in
another pipeline depends on a global data product, then the global data product will be triggered
to run when that block is complete.
If multiple blocks in the same or different pipelines depend on a global data product and they
all attempt to trigger the global data product, it will only run once no matter how many other
blocks depend on it.
### Global referencing
A global data product can be added as a block in any pipeline.
Once added, any other blocks in that pipeline can depend on it or it can depend on other blocks.
***
## Register global data product
You must have at least 1 pipeline already created.
1. Go to the global data products list page (e.g. [http://localhost:6789/global-data-products](http://localhost:6789/global-data-products))
and click the button labeled + New global data product.
You must enter a unique ID (UUID) that is unique across your entire project.
2. Choose the object type `Pipeline`.
The object types that are currently supported are:
* `Pipeline`
**`Block` object type coming soon.**
3. From the Object UUID dropdown, select the pipeline you want to register as a global data product.
4. Click the button labeled Create global data product.
***
## Configure settings
Once youโve created a global data product, you can configure several different settings.
### Outdated after
After the global data product successfully completes running,
how long after that will the global data product be outdated?
You can set how many seconds, weeks, months, or years will have to pass
after the most recent successful run for the global data product to be considered outdated.
### Outdated starting at
If enough time has passed since the last global data product has ran successfully and
the global data product is determined to be outdated,
then you can configure it to be outdated at a specific date or time.
For example, letโs say the global data product is outdated after 12 hours.
The last successful run was yesterday at 18:00.
The global data product will be outdated today at 06:00.
However, if the Outdated starting at has a value of 17 for Hour of day,
then the global data product wonโt run again until today at 17:00.
### Block data to output
The data output from the block(s) you select below will be the data product that is returned
when a downstream entity is requesting data from this global data product.
When requesting data from this global data product,
the selected block(s) will return data from its most recent partition.
You can override this by adding a value in the partitions setting.
For example, if you set the partitions value to 5,
then the selected block will return data from its 5 most recent partitions.
If you set the partitions value to 0, then all the partitions will be returned.
***
## Add to a pipeline
In a pipeline, you can add a block and choose `Global data product`.
After selecting an existing global data product, itโll be added to the current pipeline as
a block.
### Override global settings
The default settings for the global data product will be used when itโs triggered from
another pipeline. However, you can override those default settings by configuring the
global data product block settings.
Select the global data product block you added to your pipeline.
Then, in the top right corner of the block, click the settings icon to open the block settings
in the right sidekick. From there, you can override the
Outdated after, Outdated starting at, and Block data to output settings.
## Guides
# Global Hooks Guide
Source: https://docs.mage.ai/extensibility/global-hooks/guides
Follow this step by step guide on creating a global hook in Mage.
### Step 1: Navigate to the Global Hooks UI
1. Log in to your Mage account.
If you are new to Mage, follow these steps to enable Global Hooks:
1. From the Dashboard UI, click Pipelines from the left navigation pane
2. Once in the Pipelines UI, click click Settings from the left navigation pane
3. Scroll down to view the features settings and toggle on Global Hooks
2. Click on Global Hooks in the left navigation pane
3. Once on the Global Hooks view, click on the "+ Add new global hook" button.
### Step 2: Configure the Hook
1. **Hook UUID**: Enter a unique name for your hook.
2. **Resource Type**: Select the type of resource (API call type) from the resource drop down.
3. **Operation Type**: Select the type of operation needed for your hook.
Once the "Create new global Hook" button is clicked the rest of the configuration settings will populate.
4. **Targeting Conditions**: Enter conditions if you want it to apply the hook to only certain pipelines. Leave this field blank to apply the hook to all pipelines.
5. **Before Operation Starts Toggle**: Toggle to on if you want the hook to run before an operation starts.
6. **After Operation Completes Toggle**: Toggle to on if you want the hook to run after an operation ends.
7. **Run if Operation Succeeds Toggle**: Toggle to on if you want to hook to run only if the operation succeeds.
8. **Run if Operation Fails Toggle**: Toggle to on if you want the hook to run if the operation fails.
9. **Pipeline to Execute**: Select the pipeline you want to execute the hook from.
10. **Valid Code Snapshot**: Click the "Create Snapshot of code" button to validate the pipeline code.
When making a code change to the pipeline, you have to re-validate the global hook by clicking the "Create snapshot" button
11. **Stop Operation if Hook Fails Toggle**: Toggle on if you want to stop the operation from running if the hook fails.
12. **Execute Hook with History and Logging Toggle**: Toggle to on if you want to log the hook's execution, but be aware of potential performance impacts.
Toggling on the history and logging feature will significantly degrade performance of the of the operations resolution time.
13. **Run Hook Asynchronously:** Enable this setting to run hooks asynchronously and not block the current resource operation from resolving
14. **Block Outputs**: To select block output features, Click the "+ Add block outputs" button
15. **Block to Extract Data From**: If needed, select the block data will be merge from into the hook operationโs data.
16. **Object to Merge Block into data**: If needed, select the type of data that will be merged into the hook operationโs data.
17. **Additional Dictionary Keys**: Enter information into this field if you have nested objects in your block.
### **Step 3: Save and Test the Hook**
1. Click the "Save changes" button to update the global hook with the new configuration changes.
2. To execute a global hook, perform the action or operation that matches the hook's configured conditions, and the hook will run the designated code or pipeline based on its defined settings.
# Global Hooks
Source: https://docs.mage.ai/extensibility/global-hooks/overview
Global Hooks are a powerful feature in Mage that allows you to execute custom functions before or after an API request. These hooks can be used to perform additional operations, validate data, or integrate with external systems.
### **How Global Hooks Work**
Global Hooks can be triggered at two different points during the execution of a pipeline:
1. **Pre-completion of a Block**: The hook will run before a specific block in the pipeline is executed.
2. **Post-completion of a Block**: The hook will run after a specific block in the pipeline has completed execution.
### **Configuring Global Hooks**
The Global Hooks UI in Mage provides several configuration options to customize the behavior of your hooks. Here's a breakdown of the available settings:
**Hook UUID:** A unique identifier for the hook. This is used to reference the hook within your pipelines.
**Resource Type:** The type of API resource that the hook is attached to (e.g., Pipeline, Dataset, etc.).
**Operation Type:** The operation that triggers the hook (e.g., Create, Update, Delete).
### **Targeting Conditions**
The Targeting Condition fields in the Global Hooks UI allows you to specify a set of conditions that determine which subset of pipelines the hook should run on. This feature gives you control over the hookโs execution of a subset of features.
### **When to Run a Hook**
**Before/After Operation Toggles:** Toggles to specify whether the hook should run before or after the operation.
**Run if Operation Succeeds/Fails Toggles:** Toggles to specify whether the hook should run only if the operation succeeds or fails.
### Code to Run
When a hook is triggered, it will execute the pipeline that was selected in the pipeline to execute dropdown.
**Pipeline to Execute:** The pipeline that will be executed when the hook is triggered.
**Valid Code Snapshot:** A snapshot of the pipeline code that validates the hook before it runs.
When making a code change to the pipeline, you have to re-validate the global hook by clicking the "Create snapshot" button
### How to Run a Hook
**Stop Operation if Hook Fails Toggle:** If enabled, the operation will be stopped if the hook fails.
**Execute Hook with History and Logging Toggle:** If enabled, the hook's execution will be logged and recorded in the history.
Toggling on the history and logging feature will significantly degrade performance of the of the operations resolution time.
**Run Hook Asynchronously Toggle:** If enabled, the hook will run separately from the main operation, so they don't slow down or block the main task from completing
### Block Outputs
Block outputs are optional features that are used to mutate the hook operationโs input and output features.
**Block to Extract Data From:** The block in the pipeline whose output data will be merged into the resource type operation data.
**Object to Merge Block into Data:** The type of API resource output data (e.g., Payload, Resource, Metadata) into which the block output data will be merged.
**Additional Dictionary Keys:** Additional keys that allow merging the block output data into nested objects or values in the resource type operation data.
With Global Hooks, you can extend the functionality of Mage and integrate it with your existing systems and workflows. By leveraging the various configuration options, you can fine-tune the behavior of your hooks to meet your specific requirements.
# Create Deployment
Source: https://docs.mage.ai/extensibility/pro-api-reference/deployments/overview
POST /deployment_settings/{setting_name}/deployments
Trigger a deployment using a specific deployment setting.
`POST /deployment_settings/{setting_name}/deployments`
Name of the deployment setting to use for this deployment. Find the deployment setting name on your Mage Pro cluster's `/apps/deploy` page.
The name of the GitHub branch where the commit is located (e.g., `main`, `feature/xyz`).
A human-readable description of the commit (e.g., GitHub commit message).
The GitHub commit SHA that will be deployed.
The branch name in the target deployment environment (e.g., `deploy-main`).
The environment where the deployment will be created (e.g., `development`, `staging`, `production`).
Optional. The directory path inside the deployment environment where the code will be deployed.
```bash Request theme={"system"}
curl --request POST \
-H 'Content-Type: application/json' \
-H 'OAUTH-TOKEN: ' \
-d '{
"deployment": {
"source": {
"head": {
"name": "main"
},
"name": "Add feature X",
"version": "abc123def456"
},
"stage": {
"branch": {
"name": "deploy-main"
},
"environment": "production",
"target_directory": "/home/src/project_name"
}
},
"api_key": "zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2"
}' \
--url https://your-mage-instance.com/api/deployment_settings/prod/deployments
```
# Mage Pro API overview
Source: https://docs.mage.ai/extensibility/pro-api-reference/overview
Mage Pro offers two main APIs for managing your workflows and infrastructure:
* **Mage Cloud API**: Used to manage and update your Mage clusters from the Mage Cloud Portal.
* **Mage Pro Cluster API**: Runs on your Mage Pro cluster and is similar to the Mage OSS API, used to manage pipelines, runs, variables, and other resources.
***
# Mage Cloud API
The Mage Cloud API is a private API used internally by the Mage Cloud Portal to manage Mage Pro clusters. It supports operations such as:
* Starting, stopping, and restarting clusters
* Updating cluster environment variables and configurations
## Base URL
`https://cloud.mage.ai/api/v1`
## Authentication
To interact with the Mage Cloud API, you must first authenticate with your Mage Cloud credentials and obtain a session token.
### Example: Initialize session and get token
```Python theme={"system"}
import requests
API_KEY = 'oMqz2m320NfVERFonfytPFVETfPTHyN5N7db9dLK'
BASE_API = 'https://cloud.mage.ai/api/v1'
SESSION_API = f'{BASE_API}/sessions'
def authenticate_session(username: str, password: str) -> requests.Session:
"""
Authenticate with the Mage Cloud API using credentials and return an authorized session.
Args:
username (str): Mage Cloud login email.
password (str): Mage Cloud login password.
Required ENV:
MAGE_CLOUD_API_KEY
Returns:
requests.Session: An authenticated session with token set in cookies.
"""
headers = {
"Content-Type": "application/json",
"X-API-KEY": API_KEY
}
payload = {
"session": {
"email": username,
"password": password
}
}
session = requests.Session()
session.headers = headers
response = session.post(SESSION_API, json=payload)
if response.json().get('error'):
raise ValueError(response.text)
token = response.json()['session']['token']
session.headers['Cookie'] = f'oauth_token={token}'
return session
```
## Start or Stop a Cluster
Once authenticated, you can update the cluster status by modifying the replicaCount and status in the cluster's config.
### Example: Start or Stop a Cluster
```Python theme={"system"}
def update_cluster_status(session: requests.Session, team_id: str, cluster_id: str, start: bool = True):
"""
Start or stop a Mage Pro cluster by updating replicaCount and status.
Args:
session (requests.Session): Authenticated Mage Cloud session.
team_id (str): Mage Cloud team ID.
cluster_id (str): Cluster ID.
start (bool): Set True to start, False to stop the cluster.
"""
cluster_url = f'{BASE_API}/teams/{team_id}/clusters/{cluster_id}'
response = session.get(cluster_url)
response.raise_for_status()
cluster = response.json()['cluster']
config = cluster.get('config', {})
config['replicaCount'] = 1 if start else 0
cluster['status'] = 'Active' if start else 'Stopped'
update_payload = {
"cluster": {
"config": config,
"status": cluster['status']
}
}
put_response = session.put(cluster_url, json=update_payload)
put_response.raise_for_status()
print(f"Cluster has been {'started' if start else 'stopped'}.")
```
**Example usage**
```Python theme={"system"}
# Pass in credentials from your secret manager or CLI prompt
username = 'your_email@example.com'
password = 'your_password_here'
session = authenticate_session(username, password)
# Start the cluster
update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=True)
# Stop the cluster
# update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=False)
```
## Update Cluster Environment Variables
To add or update environment variables in a Mage Pro cluster, modify the `extraEnvs` field in the cluster config.
### Example: Update Environment Variables
```Python theme={"system"}
def update_cluster_env_vars(session, team_id, cluster_id, updates):
"""
Add or update environment variables in a Mage Pro cluster.
Args:
session (requests.Session): Authenticated session with Mage Cloud API.
team_id (str): Mage team ID.
cluster_id (str): Cluster ID.
updates (dict): Dictionary of environment variable key-value pairs to add or update.
"""
cluster_url = f'https://cloud.mage.ai/api/v1/teams/{team_id}/clusters/{cluster_id}'
# Fetch current cluster config
response = session.get(cluster_url)
response.raise_for_status()
cluster = response.json()['cluster']
config = cluster.get('config', {})
# Load existing environment variables
existing_envs = config.get('extraEnvs', [])
env_map = {env['name']: env['value'] for env in existing_envs}
# Apply updates
env_map.update(updates)
# Convert back to list format
config['extraEnvs'] = [{'name': k, 'value': v} for k, v in env_map.items()]
# Build payload and apply update
update_payload = {
"cluster": {
"config": config
}
}
put_response = session.put(cluster_url, json=update_payload)
put_response.raise_for_status()
print("Cluster environment variables updated successfully.")
```
**Example usage**
```Python theme={"system"}
session = authenticate_session(username='your_email@example.com', password='your_password')
update_cluster_env_vars(
session=session,
team_id='your_team_id',
cluster_id='your_cluster_id',
updates={
'AI_SIDEKICK': '1',
'DEFAULT_EXECUTOR_TYPE': 'k8s',
'ENABLE_SOME_FEATURE': 'true'
}
)
```
***
# Mage Pro cluster API
The Mage Pro Cluster API is the set of HTTP APIs available on your running Mage Pro cluster. It extends the Mage OSS API and allows:
* Create/Get/Update/Delete pipelines/blocks
* Trigger pipelines
* CRUD pipeline schedules/triggers
* CRUD pipeline runs
* CRUD workspaces
* Deploy git commits
* More APIs available in Mage Pro UI
## Base URL
`https:///api`
For the full list of cluster API endpoints, see the [Mage Open Source API Reference](/extensibility/api-reference/overview).
> Note: Mage Pro may expose additional admin or authentication APIs depending on your deployment and licensing tier.
# Update Workspace
Source: https://docs.mage.ai/extensibility/pro-api-reference/workspaces/update-workspace
PUT /api/workspaces/{workspace_name}
Update the configuration of an existing workspace in your Mage Pro cluster.
`PUT /api/workspaces/{workspace_name}`
## Path Parameters
Name of the workspace to update. You can find the workspace name on your Mage Pro cluster's `/apps/workspaces` page.
## Request Body Parameters
All request body parameters are nested under the `workspace` object.
The action to perform on the workspace. Supported values: `"stop"`, `"resume"`, `"patch"`, `"add_to_ingress"`.
See the [Supported Actions](#supported-actions) section below for details on each action and which parameters they support.
Optional. YAML configuration string for the container. Used with the `patch` action.\
Allows you to update container-level settings such as environment variables, resource limits, and other container configurations.
For example, updating environment variables:
```
env:
- name: ENV
value: test
- name: API_KEY
value: secret-key
```
Optional. Dictionary/object containing lifecycle configuration. Used with the `patch` action.
Defines lifecycle management policies including termination policies, pre-start scripts, and post-start hooks.
Structure:
```json theme={"system"}
{
"termination_policy": {
"enable_auto_termination": true,
"max_idle_seconds": 1800
},
"pre_start_script_path": "/path/to/script.sh",
"post_start": {
"command": ["echo", "Workspace started"],
"hook_path": "/path/to/hook.sh"
}
}
The action to perform. Use `"patch"` to apply partial updates to the workspace configuration.
Optional. Configuration string for the container.
For example, updating the workspace image tag and environment variables:
```
image\_tag: 2026.06.16
env:
* name: ENV
value: test
````
Optional. Whether to update workspace settings with the current cluster's environment variables. Used with the `patch` action.
Set to `true` to sync the workspace configuration with the cluster-level environment variables.
When enabled, the workspace will inherit environment variables from the cluster configuration.
## Supported Actions
The `action` field determines what operation will be performed on the workspace. Each action supports different parameters and cluster types.
### `stop`
Stops a running workspace. The workspace metadata is preserved, but the cloud instance is stopped or paused.
**Supported cluster types:** K8S, Docker
**Required parameters:**
- `workspace.action`: `"stop"`
**Optional parameters:** None
**Request body:**
```json
{
"workspace": {
"action": "stop"
}
}
````
**Example:**
```bash theme={"system"}
curl --request PUT \
-H 'Content-Type: application/json' \
-H 'OAUTH-TOKEN: ' \
-d '{
"workspace": {
"action": "stop"
}
}' \
--url https://your-mage-instance.com/api/workspaces/my-workspace
```
### `resume`
Resumes a stopped workspace, restarting the cloud instance.
**Supported cluster types:** K8S, Docker
**Required parameters:**
* `workspace.action`: `"resume"`
**Optional parameters:** None
**Request body:**
```json theme={"system"}
{
"workspace": {
"action": "resume"
}
}
```
**Example:**
```bash theme={"system"}
curl --request PUT \
-H 'Content-Type: application/json' \
-H 'OAUTH-TOKEN: ' \
-d '{
"workspace": {
"action": "resume"
}
}' \
--url https://your-mage-instance.com/api/workspaces/my-workspace
```
### `patch`
Applies partial updates to the workspace configuration, such as updating environment variables, container settings, or lifecycle configurations.
**Supported cluster types:** K8S, Docker
**Required parameters:**
* `workspace.action`: `"patch"`
**Optional parameters:**
* `workspace.container_config` (string) - YAML configuration string for the container
* `workspace.lifecycle_config` (object) - Dictionary containing lifecycle configuration (termination policy, pre-start scripts, post-start hooks)
* `workspace.update_workspace_settings` (boolean) - Whether to sync workspace settings with cluster-level environment variables
**Request body:**
```json theme={"system"}
{
"workspace": {
"action": "patch",
"container_config": "env:\n - name: ENV\n value: test",
"lifecycle_config": {
"termination_policy": {
"enable_auto_termination": true,
"max_idle_seconds": 1800
},
"pre_start_script_path": "/path/to/script.sh"
},
"update_workspace_settings": true
}
}
```
**Example:**
```bash theme={"system"}
curl --request PUT \
-H 'Content-Type: application/json' \
-H 'OAUTH-TOKEN: ' \
-d '{
"workspace": {
"action": "patch",
"container_config": "env:\n - name: ENV\n value: test",
"update_workspace_settings": true
}
}' \
--url https://your-mage-instance.com/api/workspaces/my-workspace
```
### `add_to_ingress`
Adds the workspace service to the Kubernetes ingress configuration, making it accessible via the ingress URL.
**Supported cluster types:** Kubernetes (K8S) only
**Required parameters:**
* `workspace.action`: `"add_to_ingress"`
**Optional parameters:** None
**Request body:**
```json theme={"system"}
{
"workspace": {
"action": "add_to_ingress"
}
}
```
**Example:**
```bash theme={"system"}
curl --request PUT \
-H 'Content-Type: application/json' \
-H 'OAUTH-TOKEN: ' \
-d '{
"workspace": {
"action": "add_to_ingress"
}
}' \
--url https://your-mage-instance.com/api/workspaces/my-workspace
```
Optional. Whether to update workspace settings with the current cluster's environment variables.\
Set to `true` to sync the workspace configuration with the cluster-level environment.
```bash Request theme={"system"}
curl --request PUT \
-H 'Content-Type: application/json' \
-H 'OAUTH-TOKEN: ' \
-d '{
"workspace": {
"action": "patch",
"container_config": "env:\n - name: ENV\n value: test",
"container_config": "image_tag: 2026.06.16\nenv:\n - name: ENV\n value: test",
"update_workspace_settings": true
}
}' \
--url https://your-mage-instance.com/api/workspaces/my-workspace
```
# Mage Pro data pipeline
Source: https://docs.mage.ai/getting-started/build-pro-pipeline
Build your first Mage Pro data pipeline
Mage makes building data pipelines intuitive through its block-based architecture. Instead of writing complex scripts, you create blocks that handle specific parts of your data workflow. Mage offers a complete data pipeline workflow architecture so you can extract, transform, and load your data to a target in one tool.
**Pipeline Types:**
Mage supports three main pipeline patterns to match your use case:
* **Batch pipelines** for scheduled data processing jobs
* **Data integration pipelines** for syncing data to and from different systems
* **Streaming pipelines** for real-time data processing
**What are blocks?**
Blocks are modular pieces of code that perform specific functions in your pipeline. Each block has a defined purpose:
* **Data loader blocks** bring data into your pipeline from various sources
* **Transformer blocks** clean, filter, and reshape your data
* **Data exporter blocks** send processed data to its final destination
Let's walk through creating your first batch pipeline step by step.
**Steps to build you first pipeline:**
**Step 1**: From the home page click โPipelinesโ from the left popout navigation menu
**Step 2**: Click the green โNew pipelineโ button
**Step 3**: Click โStart from scratchโ
Youโll notice that you can select from the three main data engineering use cases as described above: batch, data integration, and streaming. For the purposes of this getting started tutorial we will select the batch pipeline option.
After you filled in the Name, and optionally completed the Description and Tags click the blue โCreate new pipelineโ button at the top of the Page. This will take you to the pipeline editor page where we will create our first data\_loader block.
**Extract data from source systems**
The first block weโll add to out data pipeline is a data loader block. the data loader block is meant to extract data from source systems, whether its cloud storage, an API or database, Mage Pro has a connection for nearly anything.
**Steps to create your first block:**
**Step 1**: Click the โBlocksโ button located in the upper left portion of the UI
**Step 2**: A drop down menu will appear, just hover over โLoaderโ then select the โBase template (generic)โ block under Python.
**Step 3**: A popup will appear, give the block a name, something like โfetch dataโ and then click โSave and addโ.
This will add a new generic Python block to your pipeline. Once the block appears in your pipeline editor UI add the code below and click the โRun codeโ button located in the top right portion of the block.
```Python theme={"system"}
import pandas as pd
import requests
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data(*args, **kwargs):
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/medical.csv'
df = pd.read_csv(url)
return df
```
Congratulations on adding your first block to a pipeline. This is the first major step in creating a working data pipeline in Mage Pro. Next weโre going to add a transformation step in data pipeline.
**Transform data**
**Step 1**: Add a generic python transformation block to your pipeline by following the same steps above when you added the data loader block
**Step 2**: Give the block a meaningful name such as โtransformation riskโ
**Step 3**: Add the code below to the transformation block and click the โRun codeโ button to return a sample output.
```Python theme={"system"}
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@transformer
def transform(data, *args, **kwargs):
def classify_health_risk(health_score):
if health_score <= 3:
return "Low"
elif health_score <= 6:
return "Medium"
else:
return "High"
data['heart_health_risk'] = data['health_score'].apply(classify_health_risk)
return data
```
Finally weโll want to export our data to a target destination. Mage has many prebuilt templates for writing data to target destinations. From Snowflake to GCS, even Iceberg, Mage can write data to many different target vendors.
To export data complete the following steps:
**Step 1**: Add a generic python exporter block to your pipeline by following the same steps above when you added the data loader and transformer block.
**Step 2**: Give the code block a meaningful name such as โrisk data export.โ
**Step 3**: Add the code below to the transformation block and click the โRun codeโ button to return a sample output.
```Python theme={"system"}
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(data, *args, **kwargs):
return data
```
You did it, you created your first pipeline in Mage. Wasnโt it fun? Mage was created with developers in mind and we pride ourselves on an experience that is both easy and fun for developers. But, weโre not done yet. Next we need to schedule this pipeline so we can automate our workflow.
# Cluster Management
Source: https://docs.mage.ai/getting-started/cluster-management-portal
Manage all your Mage Pro clusters
### Cluster Management Portal
Mage Pro is split into two environments, the Cluster Manager, which supports keeping your clusters and schedulers turned on at all times, and a development environment. The Cluster Manager is where you can see consumption insights into your cluster, add and edit environment variables, and check out estimated pricing for the cluster.
### Add Cluster
Adding a new cluster to your environment is simple in Mage Pro. Just click the โLanch new clusterโ button, provide it a name, and then click โCreate cluster.โ Your new cluster will take a few minutes to spin up. Once complete you can open the cluster to your development environment or
### Environment variables
Environment variables are key-value pairs stored in the operating system that can be accessed by applications running on that system. They provide a way to configure application behavior without changing code, which is particularly useful for:
* Storing configuration settings
* Managing credentials securely (without hardcoding them)
* Controlling application behavior across different environments (development, testing, production)
**Add or edit an environment variable**
After accessing the cluster management page by clicking the blue link under the Clusters column, you'll see a blue "Edit / Add variables" button. This section explains how to use this feature to configure your environment variables.
* Navigate to the cluster management page
* Locate and click the blue "Edit / Add variables" button
* In the interface that appears, you can:
* Add new variables by specifying a name and value
* Edit existing variables by modifying their values
* Remove variables when they're no longer needed
### Add users to cluster manager
User management is a component of both the Mage Pro cluster manager and development environment. The cluster manager implements a simple permission system with only three distinct roles:
* **Admin**ย - Full control to create and delete clusters, modify environment variables, and manage cluster operations (start/stop)
* **Editor**ย - Limited permissions to modify aspects of the cluster within already created clusters.
* **Viewer**ย - Read-only access to view cluster configuration and technical details
These role-based permissions help teams maintain appropriate access controls while collaborating on data pipeline projects.
### **How to add or edit user roles**:
Team administrators can manage access permissions for cluster members through the cluster manager interface. Here's how to handle common user management tasks:
**Inviting new users**
1. Navigate to the users page and click "Invite user"
2. Enter the person's email address in the popup box
3. Select the appropriate permission level from the dropdown menu
4. Click "Invite user" to send the invitation
**Modifying existing users**
When you need to adjust permissions or remove someone:
1. Find and click on the user's row in the members list
2. Click the "Actions" button to reveal options
3. Choose "Remove user" to revoke access completely, or
4. Select "Change role" to modify their permissions
5. If changing roles, select the new role from the dropdown
6. Confirm by clicking "Change role"
These straightforward controls help maintain proper access management as your team evolves.
### Monitor your costs and usage
Keeping track of your resource usage and expenses is straightforward with Mage Pro's built-in monitoring tools. The insights dashboard gives you visibility into each cluster's financial and performance metrics.
**Accessing Usage Metrics**
1. Open your cluster by clicking its blue link in the "Clusters" column
2. Select the "Insights" tab from the navigation menu
3. The default view shows your estimated costs to date
4. Toggle between views by clicking โRealtime cpu usage" or "Realtime memory usage" to check real-time resource utilization
This visibility helps teams manage their infrastructure spending and identify potential performance bottlenecks before they impact operations. Regularly reviewing these metrics can help optimize your cluster configuration for both cost and performance.
# Install dependencies
Source: https://docs.mage.ai/getting-started/install-dependencies
Install dependencies using requirements.txt with automatic environment management
Mage Pro includes an advanced dependency management system that automatically creates and manages isolated virtual environments for your projects. Adding new packages is simple and doesn't require cluster restarts - you can immediately run your code after installation.
## Quick Start: Installing Dependencies via UI
The easiest way to install dependencies is through the Mage UI:
description: "Install dependencies using requirements.txt"
----------------------------------------------------------
Mage Pro makes dependency management simple through the requirements.txt file in your workspace's root folder. Adding new packages doesn't require cluster restarts, so you can immediately run your code after installation.
Installing dependencies in Mage Pro can be done through the requirements.txt file located in the project folder of your cluster or workspace.
### Steps:
1. **Navigate to the Terminal page** in your Mage workspace
2. **Open `requirements.txt`**: Browse to `/home/src/[project_path]/requirements.txt` and click to open it in the editor
3. **Add your packages**: Edit the file to include your dependencies
```txt theme={"system"}
pandas==2.0.0
numpy>=1.24.0
scikit-learn==1.3.0
```
4. **Click the "Install dependencies" button**: This will automatically:
* Open a terminal at the bottom of the page
* Run `mage deps install` command
* Show real-time installation progress
* Install only the packages that need to be added or updated
Your dependencies are now installed and ready to use - **no cluster restart required**!
## How It Works
Behind the scenes, Mage Pro's dependency management system:
* **Creates isolated virtual environments** for each unique combination of dependencies
* **Caches and reuses environments** based on content fingerprinting for faster installation
* **Intelligently filters packages** to reuse compatible packages from the global environment
* **Uses `uv` for fast installation** with automatic fallback to `pip`
* **Automatically activates environments** when the server starts
When you restart your cluster, the system automatically creates or updates the virtual environment based on your latest `requirements.txt`, ensuring your dependencies are always synchronized.
## Alternative Methods
### Via Mage Terminal
You can also run dependency commands directly in the Mage terminal:
```bash theme={"system"}
# Install dependencies
mage deps install
# Install from custom requirements file
mage deps install -r production_requirements.txt
# List all virtual environments
mage deps list
# Clean up old environments
mage deps cleanup
```
### Automatic on Cluster Restart
When you restart your Mage cluster, dependencies are automatically installed based on your `requirements.txt` - no manual intervention needed.
## Learn More
For advanced dependency management features including environment fingerprinting, cleanup policies, and programmatic usage, see the complete [Dependency Management](/development/dependencies/dependency-management) guide.
Follow these steps to add and install requirements:
1. Click the requirements.text file to open the file in your editor
2. Add your packages to the file and save it
3. Navigate to the terminal page
4. Run `pip install -r requirements.txt`
Following this steps will install your requirements and you will not need to restart the cluster after installation. You can run your code as written
# Python kernels
Source: https://docs.mage.ai/getting-started/kernels
Configure Python kernel execution in the pipeline editor.
1. [Python3](#python3)
2. [Multi-user kernels in Mage Pro](#multi-user-kernels)
3. [Restart kernel](#restart-kernel)
## Python3 kernel
Python3 is the default kernel. You can prototype and transform small to medium
size datasets with this kernel. Pipelines built with this kernel can be executed
in Python environments.
## Multi-user kernels in Mage Pro
Mage Pro can run a separate Python kernel for each authenticated user in the
pipeline editor. This lets multiple users run blocks at the same time without
sharing one notebook kernel, execution queue, or output stream.
Enable the feature by setting the following environment variable on your Mage
Pro cluster:
```bash theme={"system"}
MULTI_USER_JUPYTER_KERNELS=1
```
When this setting is enabled:
* Each authenticated user gets an isolated Python kernel for interactive block
execution from the pipeline editor.
* Kernel output is routed only to websocket clients for the user that started
the block run.
* Interrupt and restart actions apply to the current user's kernel.
* Idle kernels without connected websocket clients are cleaned up automatically.
You can tune resource cleanup with:
| Variable | Description | Default |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------- |
| `JUPYTER_KERNEL_IDLE_TIMEOUT_SECONDS` | Number of seconds an inactive user kernel can stay alive after its websocket clients disconnect. | `1800` |
| `JUPYTER_KERNEL_MAX_PER_SERVER` | Maximum number of user kernels to keep on one Mage server. When the limit is reached, Mage evicts idle kernels first. | `50` |
For debugging, admins can list all active user kernels with:
```bash theme={"system"}
GET /api/kernels?include_all=1
```
By default, `GET /api/kernels` only returns the kernel for the current user.
Multi-user Jupyter kernels are a Mage Pro feature. Mage uses the Python3 kernel
for this mode; the legacy PySpark notebook kernel is not supported.
## Restart kernel
If your block execution is hanging in the notebook and no block output is
printed, you can try interrupting the block execution and restarting the kernel.
You can find the "Restart kernel" button in the "Run" menu at the top of the
notebook.
# Mage Pro API
Source: https://docs.mage.ai/getting-started/mage-pro-api
Deploy your code in seconds!
### **Mage Pro API overview**
Mage Pro offers two main APIs for managing your workflows and infrastructure:
* **Mage Cloud API**: Used to manage and update your Mage clusters from the Mage Cloud Portal.
* **Mage Pro cluster API**: Runs on your Mage Pro cluster and is similar to the Mage OSS API, used to manage pipelines, runs, variables, and other resources.
# **Mage Cloud API**
The Mage Cloud API is a private API used internally by the Mage Cloud Portal to manage Mage Pro clusters. It supports operations such as:
* Starting, stopping, and restarting clusters
* Updating cluster environment variables and configurations
## **Base URL**
**`https://cloud.mage.ai/api/v1`**
## **Authentication**
To interact with the Mage Cloud API, you must first authenticate with your Mage Cloud credentials and obtain a session token.
### **Example: Initialize session and get token**
```Python theme={"system"}
import requests
API_KEY = 'oMqz2m320NfVERFonfytPFVETfPTHyN5N7db9dLK'
BASE_API = 'https://cloud.mage.ai/api/v1'
SESSION_API = f'{BASE_API}/sessions'
def authenticate_session(username: str, password: str) -> requests.Session:
"""
Authenticate with the Mage Cloud API using credentials and return an authorized session.
Args:
username (str): Mage Cloud login email.
password (str): Mage Cloud login password.
Required ENV:
MAGE_CLOUD_API_KEY
Returns:
requests.Session: An authenticated session with token set in cookies.
"""
headers = {
"Content-Type": "application/json",
"X-API-KEY": API_KEY
}
payload = {
"session": {
"email": username,
"password": password
}
}
session = requests.Session()
session.headers = headers
response = session.post(SESSION_API, json=payload)
if response.json().get('error'):
raise ValueError(response.text)
token = response.json()['session']['token']
session.headers['Cookie'] = f'oauth_token={token}'
return session
```
## **Start or stop a cluster**
Once authenticated, you can update the cluster status by modifying the replicaCount and status in the clusterโs config.
### **Example: Start or stop a cluster**
```Python theme={"system"}
def update_cluster_status(session: requests.Session, team_id: str, cluster_id: str, start: bool = True):
"""
Start or stop a Mage Pro cluster by updating replicaCount and status.
Args:
session (requests.Session): Authenticated Mage Cloud session.
team_id (str): Mage Cloud team ID.
cluster_id (str): Cluster ID.
start (bool): Set True to start, False to stop the cluster.
"""
cluster_url = f'{BASE_API}/teams/{team_id}/clusters/{cluster_id}'
response = session.get(cluster_url)
response.raise_for_status()
cluster = response.json()['cluster']
config = cluster.get('config', {})
config['replicaCount'] = 1 if start else 0
cluster['status'] = 'Active' if start else 'Stopped'
update_payload = {
"cluster": {
"config": config,
"status": cluster['status']
}
}
put_response = session.put(cluster_url, json=update_payload)
put_response.raise_for_status()
print(f"Cluster has been {'started' if start else 'stopped'}.")
```
Example usage:
```Python theme={"system"}
# Pass in credentials from your secret manager or CLI prompt
username = 'your_email@example.com'
password = 'your_password_here'
session = authenticate_session(username, password)
# Start the cluster
update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=True)
# Stop the cluster
# update_cluster_status(session, team_id='your_team_id', cluster_id='your_cluster_id', start=False)
```
## **Update cluster environment variables**
To add or update environment variables in a Mage Pro cluster, modify theย **`extraEnvs`**ย field in the cluster config.
### **Example: update environment variables**
```Python theme={"system"}
def update_cluster_env_vars(session, team_id, cluster_id, updates):
"""
Add or update environment variables in a Mage Pro cluster.
Args:
session (requests.Session): Authenticated session with Mage Cloud API.
team_id (str): Mage team ID.
cluster_id (str): Cluster ID.
updates (dict): Dictionary of environment variable key-value pairs to add or update.
"""
cluster_url = f'https://cloud.mage.ai/api/v1/teams/{team_id}/clusters/{cluster_id}'
# Fetch current cluster config
response = session.get(cluster_url)
response.raise_for_status()
cluster = response.json()['cluster']
config = cluster.get('config', {})
# Load existing environment variables
existing_envs = config.get('extraEnvs', [])
env_map = {env['name']: env['value'] for env in existing_envs}
# Apply updates
env_map.update(updates)
# Convert back to list format
config['extraEnvs'] = [{'name': k, 'value': v} for k, v in env_map.items()]
# Build payload and apply update
update_payload = {
"cluster": {
"config": config
}
}
put_response = session.put(cluster_url, json=update_payload)
put_response.raise_for_status()
print("Cluster environment variables updated successfully.")
```
Example usage:
```Python theme={"system"}
session = authenticate_session(username='your_email@example.com', password='your_password')
update_cluster_env_vars(
session=session,
team_id='your_team_id',
cluster_id='your_cluster_id',
updates={
'AI_SIDEKICK': '1',
'DEFAULT_EXECUTOR_TYPE': 'k8s',
'ENABLE_SOME_FEATURE': 'true'
}
)
```
# **Mage Pro cluster API**
The Mage Pro Cluster API is the set of HTTP APIs available on your running Mage Pro cluster. It extends the Mage OSS API and allows:
* Create/Get/Update/Delete pipelines/blocks
* Trigger pipelines
* CRUD pipeline schedules/triggers
* CRUD pipeline runs
* CRUD workspaces
* Deploy git commits
* More APIs available in Mage Pro UI
The `/runs` endpoint executes a single block within a pipeline and returns its output in the same HTTP response. No polling required.
This is the right tool when an external application needs the result of a specific block right now, not after a full pipeline run. Common use cases include real-time scoring, record lookups, and enrichment logic triggered per request.
***
### What It Does
When you call `/runs`, Mage works through the following steps:
1. Validates the token against an API-type pipeline trigger
2. Resolves the target pipeline and block
3. Runs upstream blocks if `run_upstream_blocks` is `true`
4. Executes the target block with any variables injected via `kwargs`
5. Returns the block output in the same HTTP response
6. Writes a `PipelineRun` and `BlockRun` record to the database if `record` is `true`
***
### Endpoint
```
POST https:///api/runs/{token}
```
***
### Authentication
Every call to `/runs` requires a token from an API-type pipeline trigger.
**Option A: Token in the URL**
```
POST /api/runs/your_token_here
```
**Option B: Bearer token in the Authorization header**
```
POST /api/runs
Authorization: Bearer your_token_here
```
To generate a token:
1. Open your pipeline in Mage Pro and go to the **Triggers** tab
2. Click **Add Trigger** and select type **API**
3. Save and copy the token from the trigger detail page
***
### Request Body
```json theme={"system"}
{
"run": {
"pipeline_uuid": "your_pipeline_uuid",
"block_uuid": "your_block_uuid",
"variables": {},
"project": null,
"record": true,
"store_variables": true,
"run_upstream_blocks": false,
"incomplete_only": false
}
}
```
***
### Example: Run a single block
```python theme={"system"}
import requests
MAGE_URL = 'https://'
TOKEN = 'your_trigger_token_here'
response = requests.post(
f'{MAGE_URL}/api/runs/{TOKEN}',
json={
'run': {
'pipeline_uuid': 'my_pipeline',
'block_uuid': 'my_block'
}
}
)
result = response.json()['run']
print(result)
```
***
### Example: Run a block with runtime variables
Pass variables in the request body and they get injected into the block via `kwargs`.
```python theme={"system"}
response = requests.post(
f'{MAGE_URL}/api/runs/{TOKEN}',
json={
'run': {
'pipeline_uuid': 'patient_lookup',
'block_uuid': 'transform_medical_data',
'variables': {
'patient_id': 'PID00087'
}
}
}
)
result = response.json()['run']
print(result)
```
Access the variable inside your block:
```python theme={"system"}
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def transform(df, *args, **kwargs):
patient_id = kwargs.get('patient_id')
# use patient_id in your block logic
return df
```
***
### Response
A successful call returns HTTP 200 with the block output under the `run` key:
```json theme={"system"}
{ "run": { ...block output... } }
```
***
### Request Options Reference
| Option | Default | Description |
| --------------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `pipeline_uuid` | required | The unique identifier of the pipeline containing the target block |
| `block_uuid` | required | The unique identifier of the block to execute |
| `variables` | `{}` | Key-value pairs injected into the block at runtime via `kwargs` |
| `project` | `null` | Scopes the pipeline lookup to a specific sub-project in a multi-project workspace |
| `record` | `true` | Write a PipelineRun and BlockRun record to the Mage database |
| `store_variables` | `true` | Persist the passed variables to disk alongside the block run record |
| `run_upstream_blocks` | `false` | Run all upstream block dependencies before executing the target block |
| `incomplete_only` | `false` | When `run_upstream_blocks` is `true`, skip upstream blocks that already completed successfully |
***
### Error Handling
The endpoint raises exceptions directly rather than wrapping them in an error envelope.
| Condition | Behaviour |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| Missing or invalid token | 401 Unauthenticated. The request is rejected before any pipeline lookup. |
| Token does not match a trigger | 404 Resource Not Found |
| `pipeline_uuid` not found | 404 Resource Not Found. UUIDs are case-sensitive. |
| `block_uuid` not found | 404 Resource Not Found. The block must exist within the specified pipeline. |
| Block throws an exception | The exception propagates to the HTTP response. Database records are marked `FAILED` when `record` is `true`. |
***
### Audit Trail
When `record` is `true`, Mage writes a `PipelineRun` and `BlockRun` record to the database on every call. You can see these in the Mage Pro UI under pipeline run history. Records are marked `COMPLETED` or `FAILED` depending on the outcome.
If you are calling this endpoint at high frequency and do not need the audit log, set `record` to `false` to skip the database writes.
## **Base URL**
**`https:///api`**
For the full list of cluster API endpoints, see theย [**Mage Open Source API Reference**](https://docs.mage.ai/extensibility/api-reference/overview).
> Note: Mage Pro may expose additional admin or authentication APIs depending on your deployment and licensing tier.
# Pipeline variables and keyword arguments
Source: https://docs.mage.ai/getting-started/runtime-variable
Runtime Variables are a set of global variables that can be used by every block.
These are useful for storing constants shared by multiple blocks or constants
whose value is determined at pipeline runtime (hence *runtime* variables).
## Using Runtime Variables
Runtime Variables can be accessed via the `**kwargs` parameter in your block
function.
```python theme={"system"}
from pandas import DataFrame
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def loader_data(**kwargs) -> DataFrame:
filepath = kwargs.get('filepath')
row_limit = kwargs.get('row_limit')
return pd.read_csv(filepath, nrows=row_limit)
```
Currently, runtime variables must be of primitive Python types or some basic
containers:
* integer
* string
* float
* boolean
* list
* dictionary
* set
Runtime variable names must be a valid Python identifier (i.e., no spaces in
name, can't start with a number).
## Creating Runtime Variables
### In Mage editor
You can create new global variables from the Mage UI through the "Variables" tab
in the sidekick. Click the "New" button, and configure your variable's name and
value, and press Enter to save.
To edit global variables, hover over the variable and click on the edit button.
You can edit the variable name and/or value, and press Enter to save.
### In code
You can also create pipeline-level variables in code by editing the pipeline's `metadata.yaml` file.
If you'd like to create environment specific pipeline variables, you can format the `variables` arguments as follows:
```yaml theme={"system"}
variables:
Sharedvar1: value1
sharedvar2: value2
dev:
var: value1
staging:
var: value2
prod:
var: value3
```
This create dictionaries of runtime variables as the blocks are executed.
In your execution environment, you may then set variables for each environment `ENV=dev, ENV=staging, ENV=prod` which can be leveraged to indicate the execution environment.
A simple python function can then be used to access the variables:
```python theme={"system"}
import os
def testfunc(**kwargs):
env = os.environ.get('ENV')
myyvar = kwargs[env][var]
```
A big thanks to community member [Preston Dotsey](https://mageai.slack.com/team/U061CMVAJH1) for their contribution!
### In Python code
You can also save the variable in Python code. One example usage is to save checkpoint data.
```python theme={"system"}
from mage_ai.data_preparation.variable_manager import set_global_variable
set_global_variable(pipeline_uuid, key, value)
```
## Default Runtime Variables
Mage provides some default variables to give context on the pipeline execution.
* `execution_date`: A datetime object that the pipeline is executed at.
* `event`: If the pipeline is triggered by event, the `event` variable contains
the event payload.
## Running Pipeline with Runtime Variables
### Run from command line
You can execute your pipeline with runtime variables from the command line.
First, make sure you installed the package.
```bash theme={"system"}
pip install mage-ai
```
Once the package is installed, you can run your pipeline through the command
line.
```bash theme={"system"}
mage run --runtime-vars '{"name": "value"}'...
# example with 2 runtime variables "name" and "ds":
mage run default_repo default_pipeline --runtime-vars '{"name": "default", "ds": "2022-08-18"}'
```
### Run from Python script
If your pipeline is configured to use runtime variables, you can still execute
your pipeline outside the code editor. Provide the runtime variables as keyword
arguments to `mage_ai.run()`:
```python theme={"system"}
mage_ai.run('sample_pipeline', 'repos/default_repo', filepath = 'path/to/my/file.csv', row_limit=1000)
```
## Example - Aggregating Daily Logs
A common use of ETL pipelines is to process and analyze daily events. In the
case of this example, we will create an ETL pipeline to analyze log messages
from a web application. Suppose the following is an example of a log file that
our web application produces.
| log\_date | type | source |
| ------------------- | ------- | --------- |
| 2022-07-27T20:19:20 | INFO | react-1 |
| 2022-07-27T19:18:45 | WARNING | express-2 |
| 2022-07-27T16:35:28 | DEBUG | react-1 |
| 2022-07-27T10:19:32 | INFO | express-1 |
| 2022-07-27T07:20:26 | ERROR | express-1 |
| 2022-07-27T00:42:37 | ERROR | react-1 |
Suppose we want to know the distribution of log types at the end of every day.
Using Mage's runtime variables this is made a very simple task:
1. Create a data loader to load all log files modified on a specific date. We
will specify the log folder and the date to load logs from using **runtime
variables** which are passed to this block via the `**kwargs` parameter.
```python theme={"system"}
from datetime import datetime, timedelta, date
from pandas import DataFrame, concat, read_csv
from pathlib import Path
import os
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_log_data(**kwargs) -> DataFrame:
start = datetime.fromisoformat(kwargs.get('current_date'))
end = start + timedelta(days=1)
logpath = Path(kwargs.get('log_folder'))
logs = []
for file in logpath.iterdir():
print(file)
modification_time = datetime.fromtimestamp(os.path.getmtime(file))
if modification_time >= start and modification_time < end:
df = read_csv(file)
logs.append(df)
return concat(logs, axis=0)
```
*Note*: This code ignores the edge case of a log file that spills over
between days. Since the modification date is used instead of the creation
date, a log file that is modified between days will only be considered in the
latter day.
2. Calculate the distribution of log types over this date
```python theme={"system"}
from pandas import DataFrame
from os import path
import pandas as pd
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def extract_statistics(df: DataFrame, **kwargs) -> DataFrame:
now = kwargs.get('current_date')
count = df['type'].value_counts()
count = DataFrame({now: count}).T
return count
```
The result of this transformer is a new data frame that looks like below:
| | DEBUG | ERROR | INFO | WARNING |
| ---------- | ----- | ----- | ---- | ------- |
| 2022-07-28 | 816 | 765 | 828 | 871 |
This data can then be ingested into a log statistics database.
**Key:** As the current date and log folder are not hardcoded in the pipeline
but instead provided as a runtime variable, your pipeline code remains reusable
without having to change any code. Every day, the pipeline can be ran using
`mage_ai.run()`, providing the current date and log folder as keyword arguments:
```python theme={"system"}
mage_ai.run('log_stats_ingestion', 'repos/default_repo', current_date='2022-07-29', log_folder='logs/webapp')
```
## Example - Model Rockets
Consider the following sample data tracking the launch angle (in degrees) and
vertical velocity (in meters per second) of model rocket tests:
| launch\_id | angle | vertical\_velocity |
| ---------- | ----- | ------------------ |
| 1 | 28.4 | 61.34 |
| 2 | 14.4 | 32.03 |
| 3 | 61.4 | 113.19 |
| 4 | 44.2 | 89.96 |
| 5 | 39.5 | 82.00 |
| 6 | 79.3 | 126.73 |
| 7 | 74.9 | 124.51 |
| 8 | 38.1 | 79.6 |
Suppose we want to convert the launch angle from degrees to radians in our
pipeline. The example below uses the `pi` runtime variable (passed in through
`**kwargs`) to convert the degrees column of some input data frame to a radians.
This allows us to control the precision with which we want to store `pi`.
```python theme={"system"}
from pandas import DataFrame
from os import path
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def convert_to_radians(df: DataFrame, **kwargs) -> DataFrame:
pi = kwargs.get('pi')
df['angle_radians'] = df['angle'] * pi / 180
return df
```
Suppose also want to convert the velocity from meters per second to kilometers
per hour by multiplying by the constant 3.6. Using the runtime variable
`conversion_factor` (again passed in through `**kwargs`), the vertical velocity
in kilometers per hour can be computed:
```python theme={"system"}
from pandas import DataFrame
from os import path
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def convert_velocity(df: DataFrame, **kwargs) -> DataFrame:
conversion_factor = kwargs.get('conversion_factor')
df['converted_vertical_velocity'] = df['vertical_velocity'] * conversion_factor
return df
```
Now that your pipeline is complete, you can run your pipeline using
`mage_ai.run()`. Specify values for runtime variables as keyword arguments to
this function:
```python theme={"system"}
mage_ai.run('model_rocket_data_ingestion', 'repos/default_repo', pi=3.1415, conversion_factor=3.6)
```
Suppose after creating your pipeline, you instead want to store your velocity in
miles per hour instead of kilometers per hour. As the conversion factor is a
runtime variable, you don't have to edit your pipeline - you just have to change
the conversion factor which your pipeline is run with!
```python theme={"system"}
mage_ai.run('model_rocket_data_ingestion', 'repos/default_repo', pi=3.1415, conversion_factor=2.24)
```
# Quickstart
Source: https://docs.mage.ai/getting-started/setup
Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. ๐ฆธโโ๏ธ
## Mage Pro overview
After creating a Mage Pro account or logging into Mage Pro you are directed to the cluster management portal. Here you can manage
your Mage clusters.
### Cluster Pages
1. **Creating a New Cluster**:
* On the clusters page, you can create a new cluster by clicking the "Launch new cluster" button.
* Fill in the required field **Cluster Name**. Note that environment variables can be customized after the cluster is created.
If you are in a free trial you are limited to a single cluster.
2. **Viewing Cluster List**:
* The cluster list page provides an overview of all your clusters, displaying essential information such as the cluster status, ID, UUID, cloud provider, and the number of replicas.
3. **Cluster Details**:
* Click the "Open cluster" button to view detailed information about the cluster. Here, you can see various metrics and statuses that help you monitor and manage your cluster effectively.
* Follow the [environment variables](/development/variables/environment-variables) documentation to customize your Mage Pro cluster.
### Cluster login
1. **Accessing the cluster**:
* Click the "Open cluster" button to navigate to the actual Mage cluster page. Here, you can interact with your cluster and use the Mage tool as you normally would.
* Currently, we use separate databases for each cluster for enhanced security. The users credentials in the Mage Pro cluster are managed inside the cluster. Use the default owner credentials (email: [admin@admin.com](mailto:admin@admin.com), password: **admin**) to log in. After you login, you should either update the password of the default owner user or create your own owner user.
Credentials may differ from cluster portal. For initial setup use Email or username: [admin@admin.com](mailto:admin@admin.com), Password: admin.
### Run your first pipeline
* **Step 1**: From the pipeline editor page click the "New pipeline" button
* **Step 2**: Select a template for inspiration or start from scratch by clicking the "Start from scratch" button
* **Step 3**: Select your pipeline type and give it an optional new name, description, and tag
* **Step 4**: Click the create new pipeline button
* **Step 5**: Add your blocks by following the Youtube video below
## โต๏ธ Mage OSS overview
We recommend using Docker to get started.
Docker is a tool that allows you to run Mage in a containerized environment: you can run Mage on any operating system that supports Docker, including Windows, Mac, and Linux. Using Docker means that you don't have to worry about installing dependencies or configuring your environment. If you'd like to install Mage without Docker, you can use `pip` or `conda`.
If you're familiar with [Docker Compose](https://docs.docker.com/compose/) or plan on adding or extending images (e.g. Postgres) in your project, we recommend starting from the *Docker compose template*. Otherwise, we recommend *Docker run*.
## ๐ช Get Mage
### Get started
First, be sure [Docker](https://docs.docker.com/engine/install/) is installed. Next, create a new folder for your project, change directory into that folder, and run the following from your terminal:
```bash Mac/Linux theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]
```
```bash Windows Command Line theme={"system"}
docker run -it -p 6789:6789 -v "%cd%:/home/src" mageai/mageai /app/run_app.sh mage start [project_name]
```
```bash PowerShell theme={"system"}
docker run -it -p 6789:6789 -v ${PWD}:/home/src mageai/mageai /app/run_app.sh mage start [project_name]
```
```bash GCP VM Instance theme={"system"}
sudo docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]
```
Once the server is running, open `http://localhost:6789` in your browser and explore! If using any cloud provider use external IP address with the port eg 123.456.78:6789. We recommend one of our [guides](/guides/overview) for inspiration.
### What's happening?
We're executing `docker run`, which runs a Docker container from the Mage image. additionally, we're *mounting a volume* (the `-v` command) to persist our project files on our container. This mounts the current folder to `/home/src` in the mage container. We're executing our docker image `mageai/mageai` and running a script to start up the container.
### Get started
The following is useful if you plan on customizing your image or adding additional services to a Docker network with Mage. Read more about Docker compose [here](https://docs.docker.com/compose/).
Be sure that [Docker](https://docs.docker.com/engine/install/) and [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) are installed and run:
```bash Mac/Linux theme={"system"}
git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart \
&& cd mage-quickstart \
&& cp dev.env .env && rm dev.env \
&& docker compose up
```
```bash Windows theme={"system"}
git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart
cd mage-quickstart
cp dev.env .env
rm dev.env
docker compose up
```
```Windows powershell/cmd theme={"system"}
git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart
cd mage-quickstart
copy dev.env .env
del dev.env
docker compose up
```
Once the server is running, open `http://localhost:6789` in your browser and explore! We recommend one of our [guides](/guides/overview) to for inspiration.
### What's happening?
We're cloning our Docker Compose [quickstart repo](https://github.com/mage-ai/compose-quickstart.git), copying `dev.env` to `.env`, which is ignored by git in our configuration (to avoid exposing secrets), then running `docker compose up` to start a Mage server.
```bash pip theme={"system"}
pip install mage-ai
```
```bash conda theme={"system"}
conda install -c conda-forge mage-ai
```
If you're installing on Linux and need help, check out this [community provided video walkthrough](https://www.youtube.com/watch?v=WVH7LHqND1I).
```bash theme={"system"}
mage start [project_name]
```
Open [http://localhost:6789](http://localhost:6789) in your browser.
[Enable Kubernetes](https://docs.docker.com/desktop/kubernetes/#enable-kubernetes) in Docker Desktop to start a Kubernetes cluster locally. Other options for starting a Kubernetes cluster locally are [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and [Minikube](https://minikube.sigs.k8s.io/docs/start/)
First, download the Mage Kubernetes config file [here](https://github.com/mage-ai/mage-ai/blob/master/kube/app.yaml).
Then, replace the `/path/to/mage_project` in the config yaml with the path that you want to use to store your Mage projects.
Install the command line tool, [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), if you haven't already.
Run the command `kubectl create -f kube/app.yaml` to run Mage in a Kubernetes pod. You can check the pod status with command `kubectl get pods -o wide`.
Set up port forwarding with command `kubectl port-forward mage-server 6789:6789`.
Open [http://localhost:6789](http://localhost:6789) in your browser.
## Default login credentials
Starting with **Mage OSS v0.9.78 and above**, and in **Mage Pro**,
**user authentication is enabled by default**.
When you first start Mage, the default login credentials are:
| Field | Value |
| -------- | ----------------- |
| Email | `admin@admin.com` |
| Password | `admin` |
โ ๏ธ **Important:** After logging in for the first time, we strongly recommend changing the default password or creating new user accounts for your team. See [User authentication](/production/authentication/overview) for more details.
## ๐โโ๏ธ Run your first pipeline
If you haven't already, open a browser to `http://localhost:6789`. From the pipelines page, select `example_pipeline` and open the notebook view by selecting `Edit pipeline` from the left side nav.
Select the first block by clicking it and select the "play" icon in the top right to run the block. You've just ran your first Mage block & loaded data from a dataset!
Do the same for the following cells in the pipeline to transform and export the data. Congrats, you're now a Mage ninja! ๐ฅท
## ๐ง๐ปโโ๏ธ Install Mage dependencies (optional)
Mage also has the following add-on packages:
| Package | Install | Description |
| -------------------- | ------------------------------- | -------------------------------------------------- |
| all | `mage-ai[all]` | install all add-ons |
| azure | `mage-ai[azure]` | install Azure related packages |
| clickhouse | `mage-ai[clickhouse]` | use Clickhouse for data import or export |
| dbt | `mage-ai[dbt]` | install dbt packages |
| google-cloud-storage | `mage-ai[google-cloud-storage]` | use Google Cloud Storage for data import or export |
| hdf5 | `mage-ai[hdf5]` | process data in HDF5 file format |
| mysql | `mage-ai[mysql]` | use MySQL for data import or export |
| postgres | `mage-ai[postgres]` | use PostgreSQL for data import or export |
| redshift | `mage-ai[redshift]` | use Redshift for data import or export |
| s3 | `mage-ai[s3]` | use S3 for data import or export |
| snowflake | `mage-ai[snowflake]` | use Snowflake for data import or export |
| spark | `mage-ai[spark]` | use Spark (EMR) in your Mage pipeline |
| streaming | `mage-ai[streaming]` | use Streaming pipelines |
To install these, run the following command from the Mage terminal:
```bash theme={"system"}
pip install "mage-ai[spark]"
```
or add the following to your `requirements.txt` file:
```
mage-ai[spark]
```
You can access the terminal from the side nav on the right in the pipeline editor page. Read more about installing from `requirements.txt` [here](/development/dependencies/requirements).
## ๐งญ Journey on
Navigate to our [tutorials](/guides/overview) to learn more about Mage and how to build your own pipelines or continue exploring our docs for advanced configuration and deployment options.
If you're interested in connecting a database in Docker, check out our [guide](/guides/docker/connecting-a-database) for more information.
# Mage Pro pipeline trigger
Source: https://docs.mage.ai/getting-started/trigger-pipeline
Trigger your first Mage Pro data pipeline
Pipeline automation is a critical component of production data workflows It ensures consistent and reliable execution without manual intervention. Mage Pro provides comprehensive orchestration capabilities through its trigger UI or YAML conde configuration, which initiates pipeline execution based on predefined conditions or external events.
Mage supports two primary orchestration patterns for pipeline automation:
**1. Schedule-Based Orchestration**
Schedule triggers provide time-based automation capabilities with flexible configuration options:
* **Continuous Execution**: Maintains persistent pipeline runs for real-time data processing scenarios
* **Interval-Based Execution**: Configures pipelines to run at specified intervals (hourly, daily, weekly, etc.)
* **One-Time Execution**: Schedules pipelines for single execution at a designated time
* **Custom Timeline Configuration**: Enables complex scheduling patterns with custom cron expressions
* **Deadline Management**: Sets completion time constraints to ensure pipelines finish within specified windows
This method is optimal for batch processing workflows, ETL operations, and scenarios requiring predictable execution patterns.
**2. API-Driven Orchestration**
API triggers enable event-driven pipeline execution through HTTP endpoints:
* **RESTful Integration**: Exposes pipeline execution through standardized POST requests
* **External System Integration**: Allows third-party applications to trigger pipeline runs
* **Programmatic Control**: Enables dynamic pipeline orchestration through application logic
This approach is ideal for real-time processing scenarios, webhook integrations, and microservice architectures requiring on-demand pipeline execution.
**Letโs create a new schedule based trigger:**
**Step 1**: From the pipeline editor page hover over the left popout navigation menu and select triggers.
**Step 2**: Once on the triggers page click the green โNew triggerโ button
**Step 3**: After clicking โNew Trigger,โ you will be taken to a screen where you can create a trigger. Select the โScheduleโ option and the UI pictured below ill populate.
**Step 4**: Complete the text fields below to configure your trigger based on the workflows requirements. You can also configure Run settings, SLA requirements, and Runtime variables in this UI.
**Step 5**: Click Save changes to save your trigger.
**Step 6**: After saving your changes you will be taken to a new screen where you can enable your trigger. Once enabled, the trigger will begin to run based on your previous configuration.
**Step 7**: Click the โSave trigger in codeโ button to save your trigger to a YAML file.
Once your trigger is enabled, if you set a past date in the โStart date and timeโ field your trigger will automatically run the pipeline. If you chose a date further out it will start to run on the schedule you provided in the Frequency and Start date and time fields.
You have successfully automated your first Mage pipeline. This foundational automation capability enables consistent, reliable data processing without manual intervention. You can now apply these same principles to build additional automated workflows for your data infrastructure.
# Batch pipeline tutorial
Source: https://docs.mage.ai/guides/batch-pipeline
Load, transform, and export restaurant data
# Best Practices
Source: https://docs.mage.ai/guides/best-practices
Learn best practices for building, maintaining, and scaling data pipelines in Mage Pro.
This guide provides comprehensive best practices for building, maintaining, and scaling data pipelines in Mage Pro. Following these practices will help you create reliable, performant, and maintainable data engineering solutions.
## Table of Contents
* [Block Design & Architecture](#block-design-architecture)
* [Pipeline Organization](#pipeline-organization)
* [Performance Optimization](#performance-optimization)
* [Testing & Validation](#testing-validation)
* [Error Handling & Reliability](#error-handling-reliability)
* [Resource Management](#resource-management)
* [Collaboration & Version Control](#collaboration-version-control)
* [Monitoring & Observability](#monitoring-observability)
* [Security & Governance](#security-governance)
* [Deployment & CI/CD](#deployment-cicd)
***
## Block Design & Architecture
### Single Responsibility Principle
Each block should have a single, well-defined purpose. This makes blocks easier to understand, test, and maintain.
**โ Good:**
```python theme={"system"}
# Block: load_customer_data
def load_customer_data():
"""Load customer data from source database."""
return pd.read_sql("SELECT * FROM customers", connection)
# Block: transform_customer_data
def transform_customer_data(df):
"""Clean and standardize customer data."""
df = df.dropna(subset=['email'])
df['email'] = df['email'].str.lower()
return df
```
**โ Avoid:**
```python theme={"system"}
# Block: load_and_transform_customers
def load_and_transform_customers():
"""Loads, transforms, and exports customer data."""
# Loading, transforming, and exporting in one block
df = pd.read_sql("SELECT * FROM customers", connection)
df = df.dropna(subset=['email'])
df.to_parquet('output.parquet')
```
### Modularity & Reusability
Design blocks to be reusable across pipelines and projects. Use clear, descriptive names and avoid hardcoded values.
**โ Good:**
```python theme={"system"}
# Block: load_data_from_source
def load_data_from_source(source_table: str, connection_string: str):
"""Generic data loader that can be reused for different tables."""
return pd.read_sql(f"SELECT * FROM {source_table}", connection_string)
```
**โ Avoid:**
```python theme={"system"}
# Block: load_customers_jan_2024
def load_customers_jan_2024():
"""Hardcoded for specific use case."""
return pd.read_sql("SELECT * FROM customers WHERE month='2024-01'", conn)
```
### Block Naming Conventions
Use clear, descriptive names that indicate the block's purpose:
* **Data loaders**: `load__` (e.g., `load_salesforce_contacts`)
* **Transformers**: `transform__` (e.g., `transform_clean_customer_data`)
* **Data exporters**: `export__to_` (e.g., `export_customers_to_warehouse`)
### Organizing Blocks in Subdirectories
Organize blocks in subdirectories within their respective block type folders to improve project structure and maintainability.
**โ Benefits:**
* Better organization for large projects
* Easier to find related blocks
* Clearer project structure
* Supports team collaboration
**How to Organize:**
1. **Create subdirectories by category:**
* By source type: `api/`, `database/`, `file/`, `cloud/`
* By domain: `sales/`, `marketing/`, `finance/`
* By function: `validation/`, `cleaning/`, `aggregation/`
2. **Using the UI:**
* When creating a block, name it with the subdirectory: `subfolder_name/block_name`
* Example: `api/load_stripe` creates `data_loaders/api/load_stripe.py`
* The subdirectory is automatically created if it doesn't exist
3. **Example structure:**
```
data_loaders/
โโโ api/
โ โโโ load_stripe.py
โ โโโ load_salesforce.py
โ โโโ load_hubspot.py
โโโ database/
โ โโโ load_postgres.py
โ โโโ load_mysql.py
โโโ file/
โโโ load_csv.py
โโโ load_parquet.py
transformers/
โโโ cleaning/
โ โโโ remove_duplicates.py
โ โโโ standardize_format.py
โโโ aggregation/
โโโ calculate_metrics.py
โโโ group_by_category.py
```
4. **Best practices:**
* Use consistent subdirectory naming across block types
* Keep subdirectory depth to 1-2 levels maximum
* Group related blocks together logically
* Document your organization structure for your team
For more details, see the [Project Structure Guide](/design/abstractions/project-structure).
### Block Size Guidelines
Keep blocks focused and reasonably sized:
* **Ideal**: 50-200 lines of code per block
* **Maximum**: 500 lines (consider splitting if larger)
* **Minimum**: Enough code to accomplish one clear task
### Use Block Templates
Create reusable block templates for common patterns:
* Standard ETL patterns
* Data validation patterns
* Error handling templates
* Logging templates
This ensures consistency across your team and reduces duplicate code.
***
## Pipeline Organization
### Choosing the Right Pipeline Type
Mage supports three main pipeline types, each optimized for different use cases. Understanding when to use each type is crucial for building efficient and maintainable data pipelines.
#### Data Integration Pipelines
**Use data integration pipelines when:**
* You need to **sync data between systems** (external to internal, or internal to external)
* You're working with **pre-built connectors** for SaaS tools, databases, or data warehouses
* You want **low-code/no-code** data synchronization
* You need **incremental sync** capabilities with automatic state tracking
* You're doing **reverse ETL** (syncing data from your warehouse to external systems)
**Characteristics:**
* Uses Singer spec-compliant connectors
* Handles schema migrations automatically
* Supports incremental and full sync modes
* Minimal code required - mostly configuration
* Can be combined with batch pipelines for custom transformations
**Example use cases:**
* Syncing Salesforce contacts to your data warehouse
* Loading Stripe transaction data into BigQuery
* Exporting customer segments from Snowflake to HubSpot
* Syncing product data from PostgreSQL to MongoDB
**Learn more:** [Data Integration Overview](/data-integrations/overview)
#### Batch Pipelines
**Use batch pipelines when:**
* You need **scheduled data processing** (hourly, daily, weekly)
* You require **custom transformations** using Python, SQL, or R
* You're building **ETL/ELT workflows** with complex business logic
* You need to **process large volumes** of historical data
* You want to **combine multiple data sources** and apply custom logic
* You're building **analytics pipelines** for reporting and dashboards
**Characteristics:**
* Runs on a schedule, completes, and exits
* Full flexibility with custom code blocks
* Can combine data integration blocks with custom transformers
* Supports backfilling historical data
* Ideal for data warehousing and analytics workloads
**Example use cases:**
* Daily revenue reporting with complex aggregations
* Monthly financial reconciliation
* Customer segmentation analysis
* Data quality checks and validation
* Feature engineering for machine learning
**Learn more:** [Building Batch Pipelines](/getting-started/build-pro-pipeline)
#### Streaming Pipelines
**Use streaming pipelines when:**
* You need **real-time data processing** (sub-second to second latency)
* You're processing **continuous data streams** (Kafka, Kinesis, etc.)
* You require **immediate insights** or alerts
* You're building **real-time analytics** or monitoring systems
* You need **event-driven architectures**
* Low latency is critical (milliseconds to seconds)
**Characteristics:**
* Long-running processes that continuously monitor sources
* Processes data as it arrives (not in batches)
* Event-driven architecture
* Lower latency than batch processing
* Supports stateful processing with window aggregations
**Example use cases:**
* Real-time fraud detection
* Live dashboard updates
* IoT sensor data processing
* Real-time recommendation systems
* Event-driven data transformations
* Real-time data quality monitoring
**Learn more:** [Streaming Pipeline Introduction](/guides/streaming/introduction)
#### Decision Matrix
| Requirement | Data Integration | Batch | Streaming |
| -------------------- | ------------------- | -------------------- | ----------------------- |
| **Latency** | Minutes to hours | Minutes to hours | Milliseconds to seconds |
| **Code Complexity** | Low (mostly config) | High (custom code) | Medium (transformers) |
| **Use Case** | Data sync | Scheduled processing | Real-time processing |
| **Data Volume** | Small to large | Large | Continuous streams |
| **Custom Logic** | Limited | Full flexibility | Transformers only |
| **State Management** | Automatic | Manual | Automatic (optional) |
**Hybrid Approach:**
You can combine pipeline types:
* Use **data integration pipelines** to sync data into your warehouse
* Use **batch pipelines** to transform and aggregate the synced data
* Use **streaming pipelines** for real-time processing of critical events
### Pipeline Structure
Organize pipelines logically:
1. **Data Integration Pipelines**: Extract and load data from sources
2. **Transformation Pipelines**: Transform and clean data
3. **Analytics Pipelines**: Generate reports and metrics
4. **Orchestration Pipelines**: Coordinate multiple pipelines
### Pipeline Naming
Use consistent naming conventions:
* **Format**: `__` (e.g., `sales_daily_revenue_report`)
* **Examples**:
* `marketing_campaign_attribution_hourly`
* `finance_monthly_reconciliation`
* `product_user_behavior_realtime`
### Pipeline Tagging
Use tags to organize and group related pipelines for easier discovery and management.
**โ Best Practices for Tagging:**
* **Use consistent tag names**: Establish a tagging convention across your team
* Domain tags: `sales`, `marketing`, `finance`, `product`
* Type tags: `etl`, `analytics`, `reporting`, `integration`
* Environment tags: `production`, `staging`, `development`
* Priority tags: `critical`, `high`, `low`
* **Apply multiple tags**: Pipelines can have multiple tags for flexible grouping
```python theme={"system"}
# Example: A pipeline might have tags: ['sales', 'etl', 'production', 'critical']
```
* **Group by tags**: Use the pipeline list view to group and filter by tags
* Right-click a pipeline โ "Add/Remove tags"
* Select or type tags (separated by Enter)
* Filter and group pipelines by tag on the Pipeline page
* **Tag naming conventions:**
* Use lowercase with underscores: `data_quality`, `customer_analytics`
* Keep tags short and descriptive
* Avoid special characters and spaces
**Benefits of Tagging:**
* Quickly find related pipelines
* Filter pipelines by domain, type, or environment
* Organize pipelines for different teams or projects
* Group pipelines for monitoring and alerting
For more details, see the [Pipeline Tagging Guide](/pipelines/pipeline-tagging).
### Limit Pipeline Complexity
**โ Recommended:**
* **Block count**: 10-30 blocks per pipeline
* **Depth**: Maximum 5-7 levels of dependencies
* **Branches**: Limit parallel branches to 5-10
**โ ๏ธ If your pipeline exceeds these limits:**
* Split into multiple pipelines
* Use orchestration pipelines to coordinate
* Extract common logic into reusable blocks
### Dependency Management
* **Minimize dependencies**: Only create dependencies where data actually flows
* **Avoid circular dependencies**: Design acyclic graphs
* **Use clear data contracts**: Document expected input/output schemas
***
## Performance Optimization
### Efficient Data Processing
**Batch Processing:**
```python theme={"system"}
# โ Process data in batches
def transform_large_dataset(df):
batch_size = 10000
results = []
for i in range(0, len(df), batch_size):
batch = df.iloc[i:i+batch_size]
transformed = process_batch(batch)
results.append(transformed)
return pd.concat(results)
```
**Use Appropriate Data Types:**
```python theme={"system"}
# โ Optimize data types
df['id'] = df['id'].astype('int32') # Instead of int64
df['category'] = df['category'].astype('category') # Memory efficient
```
### Leverage Spark for Large Datasets
For datasets larger than memory or requiring distributed processing:
```python theme={"system"}
# โ Use Spark for large-scale transformations
from mage_ai.data_preparation.models.block import Block
@block
def transform_with_spark(df):
spark_df = spark.createDataFrame(df)
result = spark_df.groupBy('category').agg({'value': 'sum'})
return result.toPandas()
```
### Optimize API Calls
**Batch API Requests:**
```python theme={"system"}
# โ Batch API calls
def load_data_from_api(ids):
# Process in batches instead of one-by-one
batch_size = 100
results = []
for i in range(0, len(ids), batch_size):
batch = ids[i:i+batch_size]
response = api.get_batch(batch)
results.extend(response)
return results
```
**Use Connection Pooling:**
```python theme={"system"}
# โ Reuse connections
from sqlalchemy import create_engine
# Create connection pool at module level
engine = create_engine('postgresql://...', pool_size=10)
def load_data():
return pd.read_sql("SELECT * FROM table", engine)
```
### Memory Management
**Stream Large Files:**
```python theme={"system"}
# โ Stream processing for large files
def process_large_file(file_path):
chunk_size = 10000
for chunk in pd.read_csv(file_path, chunksize=chunk_size):
process_chunk(chunk)
```
**Clean Up Resources:**
```python theme={"system"}
# โ Explicitly close connections and free memory
def load_data():
conn = create_connection()
try:
df = pd.read_sql("SELECT * FROM table", conn)
return df
finally:
conn.close()
del df # Help garbage collector
```
### Pipeline Performance
**Choose the Right Executor:**
* **`local_python` executor**: Use for faster execution when you have sufficient local resources. Blocks run in the same process or separate processes on the same machine, which reduces overhead and improves performance for smaller to medium-sized pipelines.
* **`k8s` executor**: Use for scalability when you need more resources or want to distribute block execution across multiple pods. Ideal for large-scale pipelines, resource-intensive transformations, or when you need to scale horizontally.
**Optimize Execution Mode:**
* **`run_pipeline_in_one_process: true`**: Speeds up pipeline execution by running all blocks in a single process. This reduces process creation overhead and improves performance for pipelines with many small blocks or when blocks need to share memory efficiently.
* **`run_pipeline_in_one_process: false`**: Runs blocks in separate processes (for `local_python` executor) or separate pods (for `k8s` executor). Use this when:
* Blocks need isolation (e.g., different resource requirements)
* You want to leverage parallel execution across multiple processes/pods
* Memory isolation is important to prevent one block from affecting others
**Optimize Block Execution:**
* Use dynamic blocks for parallel processing
* Leverage conditional execution for optional steps
* Cache intermediate results when appropriate
**Control Concurrency:**
Configure concurrency limits to optimize resource usage and prevent system overload. You can set limits at both the global (project) and pipeline levels:
* **Global concurrency**: Limit the maximum number of concurrent block runs across all pipelines
* **Pipeline-level concurrency**: Control concurrent block runs and pipeline runs per trigger
* **Per-block concurrency**: Set specific concurrency limits for individual blocks
For more information, see the [Concurrency Control Guide](/design/data-pipeline-management#concurrency).
***
## Testing & Validation
### Write Tests in Blocks
Mage's built-in testing framework makes it easy to write tests alongside your code:
```python theme={"system"}
# โ Include tests in your blocks
def transform_data(df):
# Transform logic
df['normalized_value'] = df['value'] / df['max_value']
return df
def test_output(output):
"""Test that transformation produces valid results."""
assert output is not None, "Output should not be None"
assert 'normalized_value' in output.columns, "Missing normalized_value column"
assert (output['normalized_value'] <= 1.0).all(), "Values should be normalized"
assert output['normalized_value'].notna().all(), "No null values allowed"
```
### Data Quality Checks
**Schema Validation:**
```python theme={"system"}
def validate_schema(df, expected_schema):
"""Validate DataFrame schema matches expected structure."""
assert set(df.columns) == set(expected_schema.keys()), "Column mismatch"
for col, dtype in expected_schema.items():
assert df[col].dtype == dtype, f"Type mismatch for {col}"
```
**Data Quality Rules:**
```python theme={"system"}
def validate_data_quality(df):
"""Comprehensive data quality checks."""
# Check for nulls in critical columns
assert df['customer_id'].notna().all(), "No null customer IDs allowed"
# Check value ranges
assert (df['age'] >= 0).all() and (df['age'] <= 150).all(), "Invalid age range"
# Check for duplicates
assert df['email'].is_unique, "Duplicate emails found"
# Check referential integrity
assert df['status'].isin(['active', 'inactive', 'pending']).all(), "Invalid status values"
```
### Test-Driven Development
Write tests before or alongside implementation:
1. **Define expected behavior** in test functions
2. **Implement the logic** to pass tests
3. **Run tests** automatically on each block execution
4. **Refactor** while keeping tests green
***
## Error Handling & Reliability
### Graceful Error Handling
**Handle Expected Errors:**
```python theme={"system"}
# โ Handle errors gracefully
def load_data_from_api():
try:
response = api.get_data()
return response
except APIError as e:
logger.error(f"API error: {e}")
# Return empty DataFrame or cached data
return pd.DataFrame()
except Exception as e:
logger.error(f"Unexpected error: {e}")
raise
```
**Validate Inputs:**
```python theme={"system"}
# โ Validate inputs early
def transform_data(df):
if df is None or df.empty:
raise ValueError("Input DataFrame is empty")
required_columns = ['id', 'name', 'value']
missing = set(required_columns) - set(df.columns)
if missing:
raise ValueError(f"Missing required columns: {missing}")
# Continue with transformation
return df
```
### Retry Logic
**Implement Retry for Transient Failures:**
You can implement retry logic in your code for handling transient failures, or use Mage's built-in automatic retry feature for block runs.
**Code-level retry:**
```python theme={"system"}
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
def load_data_with_retry():
"""Retry on transient failures."""
return api.get_data()
```
**Automatic retry for block runs:**
Mage provides automatic retry functionality for failed block runs. This is especially useful for handling transient network issues, temporary API rate limits, or other recoverable errors. Configure automatic retry settings to automatically retry failed blocks without manual intervention.
For more information, see the [Automatic Retry Guide](/orchestration/pipeline-runs/retrying-block-runs#automatic-retry).
### Idempotency
Design blocks to be idempotent (safe to run multiple times):
```python theme={"system"}
# โ Idempotent block
def export_data(df):
"""Export data, safe to run multiple times."""
# Use upsert instead of insert
df.to_sql('table', engine, if_exists='replace', index=False)
# Or use merge/upsert logic
# merge_data(df, 'table', on='id')
```
### Checkpointing
For long-running pipelines, implement checkpointing:
```python theme={"system"}
# โ Save intermediate state
def process_large_dataset(df):
checkpoint_file = 'checkpoint.pkl'
# Load checkpoint if exists
if os.path.exists(checkpoint_file):
processed = pd.read_pickle(checkpoint_file)
start_idx = len(processed)
else:
processed = []
start_idx = 0
# Process remaining data
for i in range(start_idx, len(df)):
result = process_row(df.iloc[i])
processed.append(result)
# Save checkpoint every 1000 rows
if i % 1000 == 0:
pd.DataFrame(processed).to_pickle(checkpoint_file)
return pd.DataFrame(processed)
```
***
## Resource Management
### Compute Resources
**Right-Size Your Resources:**
* Use appropriate compute clusters for workload size
* Monitor CPU and memory usage in Cluster Manager
* Scale up for large transformations, scale down for simple tasks
**Use Spark for Large Datasets:**
* Enable Spark for datasets > 10GB
* Configure Spark resources appropriately
* Monitor Spark job performance
### Connection Management
**Reuse Connections:**
```python theme={"system"}
# โ Module-level connection (reused across runs)
import os
from sqlalchemy import create_engine
DATABASE_URL = os.getenv('DATABASE_URL')
engine = create_engine(DATABASE_URL, pool_size=5, max_overflow=10)
def load_data():
return pd.read_sql("SELECT * FROM table", engine)
```
**Close Connections Properly:**
```python theme={"system"}
# โ Context manager for connections
def load_data():
with create_connection() as conn:
df = pd.read_sql("SELECT * FROM table", conn)
return df
# Connection automatically closed
```
### Memory Optimization
**Process Data in Chunks:**
```python theme={"system"}
# โ Chunk processing for large datasets
def process_large_file(file_path):
chunk_size = 50000
results = []
for chunk in pd.read_csv(file_path, chunksize=chunk_size):
processed = transform_chunk(chunk)
results.append(processed)
# Clear memory
del chunk, processed
return pd.concat(results)
```
**Use Efficient Data Structures:**
```python theme={"system"}
# โ Use appropriate data types
df['id'] = df['id'].astype('int32') # 50% memory savings vs int64
df['category'] = df['category'].astype('category') # Efficient for repeated values
df['date'] = pd.to_datetime(df['date']) # Proper datetime type
```
***
## Collaboration & Version Control
### Use Workspaces Effectively
**Separate Environments:**
* **Development**: For experimentation and testing
* **Staging**: For pre-production validation
* **Production**: For live data pipelines
**Workspace Best Practices:**
* Use consistent naming across workspaces
* Document workspace purposes
* Limit production workspace access
### Version Control Integration
**Commit Frequently:**
* Commit after completing each block or feature
* Use descriptive commit messages
* Tag releases and important milestones
**Branch Strategy:**
* Use feature branches for new development
* Keep main/master branch stable
* Use pull requests for code review
**Git Best Practices:**
```bash theme={"system"}
# โ Good commit messages
git commit -m "feat: Add customer data transformation block"
git commit -m "fix: Resolve null handling in revenue calculation"
git commit -m "refactor: Extract common validation logic to utility"
```
### Code Review
**Review Checklist:**
* [ ] Code follows team standards
* [ ] Tests are included and passing
* [ ] Error handling is appropriate
* [ ] Documentation is clear
* [ ] Performance considerations addressed
* [ ] Security concerns reviewed
### Documentation
**Document Your Blocks:**
```python theme={"system"}
def transform_customer_data(df):
"""
Transform customer data by standardizing formats and cleaning values.
Args:
df: DataFrame with customer data containing columns:
- customer_id: Unique customer identifier
- email: Customer email address
- name: Customer full name
- created_at: Account creation timestamp
Returns:
DataFrame with transformed customer data:
- customer_id: Same as input
- email: Lowercase, validated email addresses
- first_name: Extracted from name
- last_name: Extracted from name
- created_at: Parsed datetime object
Raises:
ValueError: If required columns are missing
"""
# Implementation...
```
**Pipeline Documentation:**
* Document pipeline purpose and business logic
* Include data flow diagrams
* Note dependencies and requirements
* Document expected run times and resource needs
***
## Monitoring & Observability
### Logging
**Use Structured Logging:**
```python theme={"system"}
import logging
logger = logging.getLogger(__name__)
def transform_data(df):
logger.info(f"Starting transformation for {len(df)} rows")
try:
result = perform_transformation(df)
logger.info(f"Transformation completed successfully. Output: {len(result)} rows")
return result
except Exception as e:
logger.error(f"Transformation failed: {e}", exc_info=True)
raise
```
**Log Key Metrics:**
```python theme={"system"}
def process_data(df):
start_time = time.time()
logger.info(f"Processing {len(df)} records")
result = transform(df)
duration = time.time() - start_time
logger.info(f"Processed {len(result)} records in {duration:.2f}s")
logger.info(f"Processing rate: {len(result)/duration:.0f} records/sec")
return result
```
### Monitoring Pipeline Health
**Set Up Alerts:**
* Pipeline failure alerts
* Performance degradation alerts
* Data quality threshold alerts
* Resource usage alerts
**Monitor Key Metrics:**
* Pipeline run duration
* Block execution times
* Data volume processed
* Error rates
* Resource utilization
### Data Quality Monitoring
**Track Data Quality Metrics:**
```python theme={"system"}
def validate_and_log_quality(df):
"""Validate data and log quality metrics."""
metrics = {
'row_count': len(df),
'null_percentage': df.isnull().sum() / len(df) * 100,
'duplicate_count': df.duplicated().sum(),
}
logger.info(f"Data quality metrics: {metrics}")
# Alert if quality thresholds exceeded
if metrics['null_percentage'].max() > 10:
logger.warning(f"High null percentage detected: {metrics['null_percentage']}")
return df
```
***
## Security & Governance
### Authentication
**User Authentication is Enabled by Default:**
* **Mage Pro**: User authentication is **always enabled by default** (`REQUIRE_USER_AUTHENTICATION=True`)
* **Mage OSS (0.9.78+)**: User authentication is **enabled by default**
* **Mage OSS (0.8.4-0.9.77)**: Can be enabled by setting `REQUIRE_USER_AUTHENTICATION=1`
Authentication ensures that only authorized users can access the Mage UI and APIs. All users must sign in before creating, managing, or running pipelines.
**IP Whitelisting (Recommended for SaaS and Self-Hosted Mage Pro):**
For additional security, especially for SaaS or self-hosted Mage Pro deployments, it's recommended to set up an IP whitelist to restrict access to your Mage instance.
**Implementation options:**
* **Load Balancer/Reverse Proxy**: Configure IP whitelisting at the load balancer or reverse proxy level (e.g., Nginx, AWS ALB, Azure Application Gateway)
* **Firewall Rules**: Use network-level firewall rules to restrict access
* **VPN/Private Network**: Deploy Mage Pro in a private network accessible only via VPN
**Best Practices:**
* Whitelist only necessary IP ranges (office networks, VPN endpoints)
* Regularly review and update the whitelist
* Use CIDR notation for network ranges
* Document all whitelisted IPs and their purposes
* Consider using a VPN for remote access instead of public IPs
For more information, see the [Authentication Guide](/production/authentication/overview).
### Secrets Management
**Use Mage Secrets:**
* Store API keys, passwords, and tokens in Mage's secrets management
* Never hardcode credentials in blocks
* Rotate secrets regularly
```python theme={"system"}
# โ Use secrets
import os
api_key = os.getenv('SALESFORCE_API_KEY')
database_password = os.getenv('DATABASE_PASSWORD')
```
For more information on creating and managing secrets, see the [Secrets Guide](/development/variables/secrets#secrets).
### Access Control
**Follow Principle of Least Privilege:**
* Grant minimum necessary permissions
* Use workspace-level access controls
* Review and audit access regularly
### Data Privacy
**Handle Sensitive Data Appropriately:**
* Mask PII in logs and outputs
* Encrypt sensitive data at rest and in transit
* Comply with data protection regulations (GDPR, CCPA, etc.)
```python theme={"system"}
# โ Mask sensitive data in logs
import hashlib
def log_customer_data(customer_id, email):
# Hash sensitive information
email_hash = hashlib.sha256(email.encode()).hexdigest()[:8]
logger.info(f"Processing customer {customer_id} with email {email_hash}...")
```
### Code Security
**Security Best Practices:**
* Avoid executing user input directly
* Validate and sanitize all inputs
* Use parameterized queries for databases
* Keep dependencies updated
```python theme={"system"}
# โ Parameterized queries
def load_customer_data(customer_id):
query = "SELECT * FROM customers WHERE id = :id"
return pd.read_sql(query, engine, params={'id': customer_id})
# โ Avoid SQL injection
# query = f"SELECT * FROM customers WHERE id = {customer_id}"
```
***
## Deployment & CI/CD
### Use CI/CD Deployments (Mage Pro)
**Deployment Workflow:**
1. **Develop** in development workspace
2. **Test** in staging workspace
3. **Deploy** to production using CI/CD
**Best Practices:**
* Automate deployments through CI/CD
* Use deployment pipelines for consistency
* Test deployments in staging first
* Roll back quickly if issues occur
### Environment Management
**Separate Configurations:**
```python theme={"system"}
# โ Environment-specific configuration
import os
ENV = os.getenv('ENVIRONMENT', 'development')
if ENV == 'production':
database_url = os.getenv('PROD_DATABASE_URL')
api_base_url = 'https://api.production.com'
elif ENV == 'staging':
database_url = os.getenv('STAGING_DATABASE_URL')
api_base_url = 'https://api.staging.com'
else:
database_url = os.getenv('DEV_DATABASE_URL')
api_base_url = 'https://api.dev.com'
```
### Pipeline Scheduling
**Schedule Appropriately:**
* Match schedule to data freshness requirements
* Consider source system load
* Account for pipeline execution time
* Use appropriate time zones
**Error Handling in Scheduled Pipelines:**
* Set up retry policies using [automatic retry](/orchestration/pipeline-runs/retrying-block-runs#automatic-retry) for block runs
* Configure alerting for failures
* Implement dead letter queues for failed runs
### Backfilling
**Backfill Strategy:**
* Use Mage's built-in backfilling for historical data
* Process in chunks to avoid resource exhaustion
* Monitor backfill progress
* Validate backfilled data
***
## Summary
Following these best practices will help you:
โ **Build reliable pipelines** with proper error handling and testing\
โ **Optimize performance** through efficient data processing and resource management\
โ **Maintain code quality** with modular, reusable blocks\
โ **Collaborate effectively** using version control and workspaces\
โ **Monitor and debug** with comprehensive logging and observability\
โ **Deploy safely** using CI/CD and environment management
Remember: Start with these practices and adapt them to your team's specific needs and constraints. The goal is to build maintainable, scalable, and reliable data pipelines.
# Level 99 dynamic abilities
Source: https://docs.mage.ai/guides/blocks/batch-read-write
Granular block settings for controlling read/write data partitions using output size, number of chunks, and item count.
## Batch read settings
On a block-by-block basis, customize how it reads data from its upstream blocks.
### Read specific chunks
Downstream blocks can control the volume of data it loads into memory from its upstream blockโs output.
The downstream block will only load the chunk from indigo\_mountain where power equals 5.
```python theme={"system"}
import random
import polars as pl
from mage_ai.data.tabular.mocks import create_dataframe
@data_loader
def load_data(*args, **kwargs):
dfs = []
for i in range(10):
df = create_dataframe(n_rows=100_000, use_pandas=False)
df = df.with_columns(pl.lit(i).alias('power'))
if i == 5:
df = df.with_columns(pl.lit(i).cast(pl.Float64).alias('col_0'))
dfs.append(df)
return pl.concat(dfs)
```
```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
return data
```
### Input data types
Downstream blocks can control the strategy it implements when loading an upstream blockโs
output data into memory.
#### Batch
Read data in batches.
```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
print('Batch size:', len(data[0]))
print('Chunks:', len(data))
```
```
Batch size: 143
Chunks: 21105
```
#### Generator
Batch generator framework to operate and process 1,000+ gigabytes (GB) of data without running out of memory.
```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
for batch in data:
df = batch.deserialize()
print(df.shape)
```
```
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
(1337, 11)
...
```
```python theme={"system"}
import random
import polars as pl
from mage_ai.data.tabular.mocks import create_dataframe
@data_loader
def load_data(*args, **kwargs):
dfs = []
for i in range(10):
df = create_dataframe(n_rows=100_000, use_pandas=False)
df = df.with_columns(pl.lit(i).alias('power'))
if i == 5:
df = df.with_columns(pl.lit(i).cast(pl.Float64).alias('col_0'))
dfs.append(df)
return pl.concat(dfs)
```
```python theme={"system"}
@transformer
def transform(data, *args, **kwargs):
for batch in data:
df = batch.deserialize()
print(df.shape)
```
#### Reader
Invoke methods on the reader object directly.
```python theme={"system"}
from mage_ai.data.tabular.reader import (
read_metadata,
sample_batch_datasets,
scan_batch_datasets_generator,
)
@transformer
def transform(data, *args, **kwargs):
print(data.chunks)
print(data.number_of_outputs)
print(data.resource_usages)
print(data.variable_path)
print(data.variable_type)
print(data.data_source)
output = data.read_sync()
for batch in output:
df = batch.deserialize()
print(df.shape)
```
***
## Batch write settings
Customize how data is written when outputting data from a block.
```python theme={"system"}
import random
import polars as pl
from mage_ai.data.tabular.mocks import create_dataframe
@data_loader
def load_data(*args, **kwargs):
dfs = []
for i in range(10):
df = create_dataframe(n_rows=100_000, use_pandas=False)
df = df.with_columns(pl.lit(i).alias('power'))
if i == 5:
df = df.with_columns(pl.lit(i).cast(pl.Float64).alias('col_0'))
dfs.append(df)
return pl.concat(dfs)
```
### Strategies
Explicitly set the chunking strategy when creating data partitions or batches of data.
Example: create chunks by the value in column `power`
#### Items per chunk
Example: each chunk can only contain a maximum number of 7,777 items.
#### Number of chunks
Example: each chunk must have at least 1,337 items and the total number of chunks cannot exceed 143.
#### Byte size per check
Example: each chunk cannot exceed 100MB in size on disk.
### Modes
Control how data is written.
1. Append
2. Replace
# Custom blocks
Source: https://docs.mage.ai/guides/blocks/custom-blocks
Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.
## Adding a custom block to the pipeline
1. Create a new pipeline or open an existing pipeline.
2. Add a custom block by clicking on the `Custom` button from the list of block types available.
The `Custom` button is visible at the bottom of the main section or when hovering over the divider between two blocks.
3. Specify the language for your block (Python and SQL currently supported). For custom Python blocks, use the `@custom` decorator.
4. Specify the color for your block. The colors are a way to visually identify or organize your custom blocks (e.g. The block will have this color in the dependency tree.) and do not affect how they are run.
5. After clicking on the color from the dropdown menu, a new custom block should appear in your pipeline.
6. You can rearrange the blocks by clicking on the block header and dragging it to another position in the pipeline.
# Detaching blocks
Source: https://docs.mage.ai/guides/blocks/detach-blocks
Detach other pipeline associations from a block shared by multiple pipelines.
## Overview
If a block is being used by multiple pipelines, changing the content of that block
will change it for all of the pipelines sharing that block. If you want to create an
isolated copy of that block for a specific pipeline so it doesn't affect other
pipelines sharing it, you can "detach" it.
Detaching a block duplicates it so it is no longer shared with any other pipelines,
removing/detaching other pipeline associations. A new block file is created with the
original block's contents, and the original block's dependencies in the current pipeline
are preserved.
### Supported pipeline types
Detaching a block is supported in `standard` and `streaming` pipelines. It is not supported
in `integration` pipelines.
### Supported block types
* Data loader
* Transformer
* Data exporter
* Sensor
* Custom
### Unsupported block types
* DBT
* Callback
* Conditional
* Extension
## How to detach a block
1. If a block is shared by multiple pipelines, there will be a symbol, which looks something
like `<> X pipelines`, with the number of pipelines in the block's header on the Pipeline
Editor page. Refer to the screenshot above for an example. All 3 of the example pipeline's
blocks in the screenshot are shared by multiple pipelines.
2. Click on the shared pipelines symbol (`<>`) or the block settings icon in the block
header to go to the Block Settings panel in the Sidekick.
3. Click on the `Detach ` button at the top right of the block settings panel. Note
that this button will only be visible if the selected block is shared by multiple pipelines.
Blocks that are not shared by any other pipelines do not have the option to detach itself.
4. A popup will appear allowing you to name the new block that will be created to replace
the existing block. This new block will not be shared by any other pipelines.
5. Click `Save and replace block`.
# Dynamic blocks
Source: https://docs.mage.ai/guides/blocks/dynamic-blocks
Create a pipeline with dynamic blocks.
## Build pipeline
1. Add a new data loader block.
2. Paste in the following code:
```python theme={"system"}
from typing import Dict, List
@data_loader
def load_data(*args, **kwargs) -> List[List[Dict]]:
users = []
metadata = []
for i in range(3):
i += 1
users.append(dict(id=i, name=f'user_{i}'))
metadata.append(dict(block_uuid=f'for_user_{i}'))
return [
users,
metadata,
]
```
3. Run the data loader block and the output will be:
```python theme={"system"}
[{'id': 1, 'name': 'user_1'},
{'id': 2, 'name': 'user_2'},
{'id': 3, 'name': 'user_3'}]
```
4. In the top right corner of the block, click on the triple dot button (`...`) and click the
dropdown selection labeled `Set block as dynamic`.
5. Add a new transformer block.
6. Paste in the following code:
```python theme={"system"}
from typing import Dict, List
@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
data['id'] = int(data['id']) * 100
return [data]
```
7. Run the transformer block and the output will be:
```python theme={"system"}
{'id': 100, 'name': 'user_1'}
```
8. Add another transformer block.
9. Paste in the following code:
```python theme={"system"}
from typing import Dict, List
import uuid
@transformer
def transform(data: Dict, *args, **kwargs) -> List[Dict]:
data['token'] = uuid.uuid4().hex
return [data]
```
10. Change the 2nd transformer blockโs upstream parent block to only be the data loader.
This will require you to remove the existing upstream parent block which is currently pointing to
the 1st transformer block.
11. In the top right corner of this 2nd transformer block,
click on the triple dot button (`...`) and click the
dropdown selection labeled `Reduce output`.
12. Run the transformer block and the output should look something like this:
```python theme={"system"}
{'id': 1, 'name': 'user_1', 'token': '0ea0f28c2de647d997e5c341a857c92f'}
```
13. Add a new data exporter block.
14. Paste in the following code:
```python theme={"system"}
from typing import Dict, List
@data_exporter
def export_data(users: List[Dict], **kwargs):
print(users)
```
15. Run the data exporter block and the output should look something like this:
```python theme={"system"}
[{'id': 1, 'name': 'user_1', 'token': 'bc35e36ecfc24d77a2d910f4b1d6611d'}]
```
When executing the entire pipeline, this output will contain 3 items in this list.
The pipelineโs final dependency graph should look like this:
## Run pipeline
1. Click the pipeline name in the top left corner (e.g the breadcrumbs).
This will take you to the pipelineโs trigger page.
2. Click the button `[Run @once]` in the page's subheader,
or create a trigger to run once.
3. Click the name of the newly created trigger.
4. The trigger will create 1 pipeline run with 9 block runs.
The block runs should look like this:
## Limit dynamically created blocks
To prevent a dynamic block from creating too many downstream block runs, set a maximum number
of dynamic child block runs. To throttle execution after those rows are created, set a maximum
number of concurrent dynamic child block runs. Unset values and `0` allow unlimited dynamic child
block creation or concurrent dynamic child execution.
Set workspace-wide defaults with the `DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS` and
`DYNAMIC_BLOCKS_MAX_CONCURRENT_CHILD_BLOCKS` environment variables. Set
`DYNAMIC_BLOCKS_MAX_CHILD_BLOCKS_BEHAVIOR` to `fail` to stop runs that exceed the creation limit,
or `limit` to create only up to the configured maximum. You can also add this to the project
`metadata.yaml`:
```yaml theme={"system"}
dynamic_blocks:
max_child_block_runs: 1000
on_max_child_block_runs_exceeded: fail
max_concurrent_child_block_runs: 25
```
Set a pipeline-specific limit in the pipeline `metadata.yaml`:
```yaml theme={"system"}
dynamic_blocks:
max_child_block_runs: 500
on_max_child_block_runs_exceeded: limit
max_concurrent_child_block_runs: 10
```
Set a stricter limit for one dynamic block in that block's configuration:
```yaml theme={"system"}
configuration:
dynamic:
max_child_block_runs: 100
on_max_child_block_runs_exceeded: limit
max_concurrent_child_block_runs: 5
```
## Dynamic blocks 2.0
In version 1.0 of dynamic blocks,
Mage was loading all the data of an upstream dynamic child block from disk into memory
in order to calculate how many dynamic child blocks should be created.
This can result in using large amounts of memory.
In version 2.0 of dynamic blocks,
Mage leverages the metadata of upstream block outputs
to calculate the number of downstream blocks for dynamic blocks and dynamic child blocks.
The metadata is only several bytes on disk and less than a kilobyte in RAM.
### Stream mode
Here is a sample scenario to describe the previous and current state:
1. Dynamic block A returns a list of 10 items and has 1 direct downstream block B.
2. B is a dynamic child block and has 2 direct downstream block: C and D.
3. C is a dynamic child block and has 1 direct downstream block: D.
```mermaid theme={"system"}
graph TD
A --> B
B --> C
B --> D
C --> D
```
In the past, this is the sequence of events:
1. Block A finishes executing and returns an output of 10 items.
2. Block B spawns 10 dynamic child blocks.
3. Block B executes all 10 dynamic blocks concurrently.
4. Once all 10 block runs from block B finishes, block C starts.
5. Block C spawns 10 child blocks and executes them.
6. Once Block C completes all 10 runs, block D starts.
7. Block D spawns 100 child blocks and executes them.
With dynamic block 2.0, you can enable stream mode which will allow downstream dynamic child blocks to execute as soon as its upstream parent dynamic child blocks finish. Here is the sequence of events in the new world:
1. Block A executes and can be handled in 2 ways:
1. Serial: block A executes its code line by line and at the end, the return statement outputs the entire result of 10 items in a list.
2. Generator: block A executes its code line by line up until a yield block. Once yield is called, the object that is yielded is stored as an output of block A. In this example, block A would yield 10 times, 1 for each item.
2. As soon as 1 output is detected from block A, block B spawns 1 dynamic child that consumes that single output as its input argument.
3. As soon as 1 output is detected from block B, block C spawns 1 dynamic child.
4. As soon as 1 output is detected from both block B and block C, block D spawns 1 dynamic child.
# Level 99 dynamic abilities
Source: https://docs.mage.ai/guides/blocks/dynamic-blocks-2
Execute 100,000+ dynamically created block runs concurrently with upgraded memory management and concurrency.
## Memory optimizations
In version 1.0 of dynamic blocks,
Mage was loading all the data of an upstream dynamic child block from disk into memory
in order to calculate how many dynamic child blocks should be created.
This can result in using large amounts of memory.
In version 2.0 of dynamic blocks,
Mage leverages the metadata of upstream block outputs
to calculate the number of downstream blocks for dynamic blocks and dynamic child blocks.
The metadata is only several bytes on disk and less than a kilobyte in RAM.
## Stream mode
Stream block output data to dynamically generated blocks without waiting for the upstream parent
block to finish executing.
Here is a sample scenario to describe the previous and current state:
1. Dynamic block A returns a list of 10 items and has 1 direct downstream block B.
2. B is a dynamic child block and has 2 direct downstream block: C and D.
3. C is a dynamic child block and has 1 direct downstream block: D.
```mermaid theme={"system"}
graph TD
A --> B
B --> C
B --> D
C --> D
```
In the past, this is the sequence of events:
1. Block A finishes executing and returns an output of 10 items.
2. Block B spawns 10 dynamic child blocks.
3. Block B executes all 10 dynamic blocks concurrently.
4. Once all 10 block runs from block B finishes, block C starts.
5. Block C spawns 10 child blocks and executes them.
6. Once Block C completes all 10 runs, block D starts.
7. Block D spawns 100 child blocks and executes them.
With dynamic block 2.0, you can enable stream mode which will allow downstream dynamic child blocks to execute as soon as its upstream parent dynamic child blocks finish. Here is the sequence of events in the new world:
1. Block A executes and can be handled in 2 ways:
1. Serial: block A executes its code line by line and at the end, the return statement outputs the entire result of 10 items in a list.
2. Generator: block A executes its code line by line up until a yield block. Once yield is called, the object that is yielded is stored as an output of block A. In this example, block A would yield 10 times, 1 for each item.
2. As soon as 1 output is detected from block A, block B spawns 1 dynamic child that consumes that single output as its input argument.
3. As soon as 1 output is detected from block B, block C spawns 1 dynamic child.
4. As soon as 1 output is detected from both block B and block C, block D spawns 1 dynamic child.
## Enabling stream mode
Stream mode allows downstream dynamic child blocks to execute without waiting for every upstream block to finish, improving pipeline performance.
### Accessing block settings
To enable stream mode, access the Dynamic block v2 configuration:
1. Click the **block settings button** next to the "Run code" button
* Alternatively, use the **Block settings** tab in the right side navigation menu
### Configuration
1. Scroll down to the "Dynamic block v2 configuration" section
2. Select **"Stream"** from the Mode dropdown
3. Enter a **poll interval** in seconds - this is how often the system checks for new data or completed upstream blocks. Lower values (like 1 second) provide faster response times but use more system resources
4. **(Optional)** Toggle **"Combine all output into 1"** to consolidate outputs for downstream processing
Stream mode enables parallel processing and better resource utilization by allowing child blocks to start processing data as it becomes available.
## Memory generator
If a dynamic block contains a yield block,
the object yielded will be an output from that block. Once that object is stored,
it is released from memory and the next object in the generator within that block
is processed and loaded into memory.
For example, if you want to loop through a total of 1 billion indexes in 1 million increments
so that you can load 1 million records into memory, you can use a yield block to process a
batch of 1 million, then yield the result as an output, release it from memory automatically,
and then proceed to the next batch.
```python theme={"system"}
@custom
def process(*args, **kwargs):
for i in range(1_000):
yield range(i * 1_000_000, (i + 1) * 1_000_000)
```
# Markdown blocks
Source: https://docs.mage.ai/guides/blocks/markdown-blocks
A markdown block is a standalone text block that supports markdown and can be used to make notes or organize your blocks.
## Adding a markdown block to the pipeline
1. Create a new pipeline or open an existing pipeline.
2. Edit the pipeline.
3. Add a markdown block by clicking on the `Markdown` button from the list of block types available.
The `Markdown` button is visible at the bottom of the main section or when hovering over the divider between two blocks.
4. You can rearrange the blocks by clicking on the block header and dragging it to another position in the pipeline.
### Shortcuts
* Edit mode: Double-click the markdown block's content area to open the content editor. You can
also click the "Edit" icon in the block header to edit the content.
* Exit edit mode: Press `Escape` on your keyboard while the content editor is active.
You may need to press `Escape` twice, once to exit the editor and another time to deselect
the block. When the block is deselected, the editor will automatically close.
### Directory
The files associated with the markdown blocks are located in the `markdowns` directory
in the File Browser.
### Deleting a markdown block
When you delete a markdown block that is not used in any other pipeline,
the file may also be deleted automatically as well.
## Supported syntax
#### Horizontal breaks
***
#### H1-H5 Headlines
Normal text
**bold text**
*italicized text*
> blockquote
#### Ordered list
1. First item
2. Second item
3. Third item
#### Unordered list
* First item
* Second item
* Third item
You can use inline `test_code` in your sentences.
Links: [Mage homepage](https://www.mage.ai)
#### Tables
| Column1 | Column2 | Column3 |
| ------- | ------- | ------- |
| Text1 | TextA | TextI |
| Text2 | TextB | TextII |
| Text3 | TextC | TextIII |
#### Code blocks
```
{
"firstName": "John",
"lastName": "Smith",
"age": 25
}
```
#### Checkboxes
* [x] Todo 1 checked
* [ ] Todo 2 unchecked
* [ ] Todo 3 unchecked
#### Images
Images are supported with the typical image markdown syntax:
```
```
or with the `` HTML tag syntax:
```
```
With the `` syntax, you can create customizable image sizes with `width` and `height` attributes.
# Native support for Polars
Source: https://docs.mage.ai/guides/blocks/polars
Serialize and deserialize data using Polars DataFrames.
Mage supports using Polars DataFrame in addition to Pandas DataFrame to process the data.
```python theme={"system"}
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
import polars as pl
@data_loader
def load_data(*args, **kwargs):
df = pl.DataFrame(
{
'A': [1, 2, 3, 4, 5] * 10,
'fruits': ['banana', 'banana', 'apple', 'apple', 'banana'] * 10,
'ham': ['a', 'b', 'c', 'd', 'e'] * 10,
'dt': ['2023-03-01', '2023-03-02', '2023-03-03', '2023-03-04', '2023-03-05'] * 10,
'arr': [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] * 10,
'obj': [dict(a=[1, 2]), dict(a=[3, 4]), dict(a=[5, 6]), dict(a=[7, 8]), dict(a=[9, 10])] * 10,
}
)
df = df.with_columns(
pl.col('dt').str.to_datetime('%Y-%m-%d')
)
return df
```
If you return a Pandas or Polars DataFrame, itโll be serialized to disk using PyArrow and stored in Parquet format. When a downstream block loads an upstream blockโs output data as an input argument, that data will be deserialized back to the data type before it was serialized.
Leveraging Polars and PyArrow in this process improves the memory usage while serializing data to disk and deserializing data from disk by leveraging Polarโs lazy data frames, PyArrow partitions, and Parquet file format features such as accessing metadata and schema information without loading the raw data into memory.
# R blocks
Source: https://docs.mage.ai/guides/blocks/r-blocks
You can write R language to transform data in blocks.
## Requirements
R blocks are only supported when running Mage using [Docker](/getting-started/setup#using-docker).
## Add R block to pipeline
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `R`.
## Example pipeline
1. Data loader
```R theme={"system"}
load_data <- function() {
# Specify your data loading logic here
# Return value: loaded dataframe
df <- read.csv(url('https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'))
df
}
```
2. Transformer
```R theme={"system"}
library("pacman")
p_load(dplyr)
transform <- function(df_1, ...) {
# Specify your transformation logic here
# Return value: transformed dataframe.
df_1 <- filter(df_1, Pclass < 3)
df_1
}
```
3. Data exporter
```R theme={"system"}
export_data <- function(df_1, ...) {
# Specify your data exporting logic here
# Return value: exported dataframe
write.csv(df_1, "titanic_filtered.csv", row.names = FALSE)
}
```
## Install R packages
Add the following at the start of your code in your R block:
```R theme={"system"}
pacman::p_load(package1, package2, package3)
```
Or
```R theme={"system"}
library("pacman")
p_load(dplyr)
```
> Note
>
> When you run the R block for the 1st time, the package will be installed.
> The 2nd time you run the R block, the package wonโt need to be installed again.
### What is `pacman`?
`pacman` is an R package management tool. You can use `p_library()` to view all the available packages.
Here is the documentation for `pacman` where you can find more useful methods: [https://www.rdocumentation.org/packages/pacman/versions/0.5.1](https://www.rdocumentation.org/packages/pacman/versions/0.5.1)
## Runtime variables
Runtime variables can be accessed via `global_vars` vector, like `global_vars['execution_date']`.
Example code:
```R theme={"system"}
load_data <- function() {
df <- read.csv(file='titanic_clean.csv')
df['date'] <- global_vars['execution_date']
df
}
```
# Reordering blocks
Source: https://docs.mage.ai/guides/blocks/reorder-blocks
Reorder blocks in the notebook when editing a pipeline.
## Overview
When you are in the Pipeline Editor page of a standard or streaming pipeline, you might
want to rearrange the order that the blocks appear in the notebook. This could help with
organizing your pipeline or allow you to group your blocks a certain way.
Keep in mind that reordering the blocks from the UI will change the order of
the blocks in your pipeline's `metadata.yaml` configuration file.
## How to reorder blocks
1. Go to your pipeline's Edit page (`/pipelines/[pipeline_uuid]/edit`).
2. Click on the block header of the block you want to move, and drag it
over the block whose position you want to move it to.
3. Dragging a block from below another block will move it to the position
above that block. Dragging a block from above another block will move it
to the position below that block.
4. Once you have the block in the position you want, drop the block.
5. The updated block order should be reflected in your pipeline's
`metadata.yaml` file.
## Reordering upstream blocks received as positional arguments
If a block has multiple upstream blocks, you can access the upstream blocks' outputs via
the positional arguments of the downstream block's decorated function. You can rearrange
these positional arguments by dragging and dropping the associated blocks using the same
steps as [above](#how-to-reorder-blocks) for reordering blocks.
# Replicating blocks
Source: https://docs.mage.ai/guides/blocks/replicate-blocks
Reuse the same block multiple times within a single pipeline.
## What does it mean to replicate a block
Letโs say you have a pipeline named `fire_pipeline` with the following 3 blocks:
```
[load_source_a] [load_source_b]
โ
[transform_data]
```
The transformer block named `transform_data` contains reusable code. Instead of creating a brand
new block in this pipeline that does the exact same thing as `transform_data`, you can replicate
the `transform_data` block.
Once you replicate the `transform_data` block, you can configure it however you want. For example,
you can name the replica block `transform_source_b` or
set the replica blockโs upstream block to `load_source_b`.
The `fire_pipeline` will now look like this:
```
[load_source_a] [load_source_b]
โ โ
[transform_data] [transform_source_b]
```
A replica blockโs code and execution logic mirrors the block it replicated.
In this example, the `transform_source_b` block will get its code from the `transform_data` block.
### Why is replicating blocks useful
Replicating blocks make it easier to reuse code in a pipeline while maintaining observability of
each branch in the pipeline because each block, even when replicated, will have its own block
run instance when a pipeline is executed.
These block run instances have monitoring, alerting, and logging built-in as well as producing
partitioned data products when successfully completed.
### Terminology
| Term | Definition |
| ---------------- | --------------------------------------------------------------------------------- |
| Replicated block | An original block in a pipeline that has 1 or more replica blocks. |
| Replica block | A block in a pipeline that uses the code from another block in the same pipeline. |
## How to replicate a block
All block types, except dbt blocks, can be replicated. In the top right corner of the block,
click the triple dot icon to display a dropdown menu. Under the dropdown menu, click the option
labeled `Replicate block`.
This will add a new block to your current pipeline. You can optionally name the new replica block.
When the pipeline runs, itโll create a block run for each block in the pipeline. When a block run
is created for a replica block, the block run will reference that block using the following
block UUID convention: `[replica_block_uuid]:[replicated_block_uuid]`.
For example:
1. Original blockโs UUID is `transform_data`.
2. The `transform_data` block is replicated.
3. The newly created replica blockโs UUID is `transform_source_b`.
Itโs replicated from the block `transform_data`.
A block runโs `block_uuid` attribute for the replica block `transform_source_b` will appear in
dashboards and logs as `transform_source_b:transform_data`.
# Reuse blocks
Source: https://docs.mage.ai/guides/blocks/reuse-blocks
Learn how to reuse blocks within pipelines, across pipelines, and across projects in Mage for better code organization and efficiency.
## Overview
Mage allows you to reuse blocks across multiple pipelines within the same project, enabling you to build modular, maintainable data pipelines. When you reuse a block, any changes made to that block will automatically apply to all pipelines that use it.
### Benefits of reusing blocks
* **Code efficiency** - Write once, use many times
* **Consistency** - Ensure the same logic is applied across all pipelines
* **Maintenance** - Update logic in one place and it applies everywhere
* **Modularity** - Build complex pipelines from smaller, reusable components
* **Testing** - Test block logic once and reuse with confidence
## Reuse Scenarios and Methods
| Scenario | Method | Description | Best For |
| -------------------- | ----------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------- |
| **Within Pipeline** | [Replicate blocks](#replicate-blocks-within-a-pipeline) | Right-click to duplicate blocks in the same pipeline | Similar data sources, parallel processing, testing variations |
| **Across Pipelines** | [Drag & drop](#drag-and-drop-from-file-browser) | Drag blocks from file browser to pipeline canvas | Quick reuse of existing blocks |
| **Across Pipelines** | [Search bar](#use-the-search-block-bar) | Search and add blocks from all projects | Finding specific blocks across projects |
| **Code Reuse** | [Utility files](#create-utility-files-and-import-in-blocks) | Create shared Python modules and import them | Small functions, data processing utilities |
| **Code Reuse** | [Block templates](#use-block-templates) | Create reusable templates with placeholders | Common patterns with slight variations |
| **Across Projects** | [Pro templates](#block-templates-across-projects-mage-pro) | Use Mage Pro's template library | Organization-wide standardization |
| **Across Projects** | [Export/import](#cross-project-block-sharing) | Export blocks from one project to another | Sharing between different projects |
## How block sharing works
When you create a block in one pipeline, you can reference that same block in other pipelines. The block file exists in the project's file system and can be imported by any pipeline within the same project.
### Block sharing indicators
When viewing a pipeline, you'll see indicators showing which blocks are shared:
* **Shared blocks** display a symbol like `<> 2 pipelines` in the block header
* **Non-shared blocks** show no additional symbols
* The number indicates how many pipelines are using that block
### Viewing pipelines using a block
To see which pipelines are using a specific block:
1. **Click on a block** in the pipeline editor
2. **Open block settings** - Click the settings icon in the right side panel
3. **View "Pipelines using this block"** - See a list of all pipelines that reference this block
4. **Check project paths** - See which project each pipeline belongs to
This helps you understand the impact of changes to shared blocks across your project.
## Methods to reuse blocks
### Drag and drop from file browser
This is the most intuitive way to reuse blocks across pipelines:
1. **Open the file browser** on the Pipeline Editor page
2. **Navigate to the block** you want to reuse in the file browser
3. **Click and hold** the block file
4. **Drag the block file** onto the center of the page (the Notebook area)
5. **Release the button** and the block will be added to the pipeline
**Supported block types:**
* `callbacks`, `conditionals`, `custom`, `data_exporters`, `data_loaders`
* `dbt`, `dbts`, `markdowns`, `scratchpads`, `sensors`, `transformers`
**Mage OSS**: Version 0.9.64 or greater is required to add callback and conditional blocks from the file browser.
**Mage Pro**: Callback and conditional blocks are supported by default in all versions.
### Use the search block bar
Search for existing blocks across all your projects:
1. **Click in the search bar** at the bottom of the pipeline editor
2. **Type keywords** to search for existing blocks (e.g., "load titanic")
3. **Browse the search results** showing blocks from different projects
4. **Click on a block** from the dropdown to add it to your pipeline
5. **The block will be added** to your current pipeline
**Mage OSS**: The "Add new block v2" feature must be enabled in project settings and is available in standard (batch) pipelines. If recently enabled, Mage must be restarted for blocks to be indexed.
**Mage Pro**: This feature is enabled by default and available in all pipeline types.
### Replicate blocks within a pipeline
To reuse a block within the same pipeline (create a copy):
1. **Click the triple dot icon** in the top right corner of the block you want to replicate
2. **Select "Replicate block"** from the dropdown menu
3. **Name the new replica block** (optional)
4. **Customize the new block** as needed for your specific use case
5. **Connect the blocks** in your pipeline flow as required
**Use cases for block replication:**
* **Similar data sources** - Load from different tables with similar structure
* **Multiple transformations** - Apply similar logic to different data streams
* **Parallel processing** - Process the same data with different parameters
* **Testing variations** - Compare different approaches side by side
Learn more about [replicating blocks](/guides/blocks/replicate-blocks).
## Other ways to reuse code
### Create utility files and import in blocks
For shared functions and logic that don't need to be full blocks:
1. **Create a utility file** in your project (e.g., `default_repo/utils/data_processing.py`)
2. **Write reusable functions** in the utility file
3. **Import the functions** in your blocks using standard Python imports
**Example utility file:**
```python theme={"system"}
# default_repo/utils/data_processing.py
import pandas as pd
def clean_column_names(df):
"""Clean column names by removing spaces and special characters."""
df.columns = df.columns.str.replace(' ', '_').str.lower()
return df
def validate_dataframe(df, required_columns):
"""Validate that dataframe has required columns."""
missing_cols = set(required_columns) - set(df.columns)
if missing_cols:
raise ValueError(f"Missing columns: {missing_cols}")
return True
```
**Using in a block:**
```python theme={"system"}
# In your data loader block
from default_repo.utils.data_processing import clean_column_names, validate_dataframe
@data_loader
def load_data():
df = pd.read_csv('data.csv')
df = clean_column_names(df)
validate_dataframe(df, ['id', 'name', 'email'])
return df
```
Learn more about [importing custom Python code](/development/dependencies/custom-files#importing-custom-python-code).
### Use block templates
Create reusable block templates for common patterns:
1. **Navigate to Templates** - Click "Templates" in the left navigation menu
2. **Create new template** - Click "New block template" button
3. **Define template properties** - Set template UUID, block type, and language
4. **Edit template content** - Write the template code with placeholders
5. **Save template** - Click "Save template" to create the template
Learn more about [creating and using user-defined templates](/guides/blocks/user-defined-templates).
## Reusing blocks across projects
### Block templates across projects (Mage Pro)
**Mage Pro feature**: Mage Pro supports configuring a central template library that shares templates across projects, allowing you to create and share reusable block templates across multiple projects in your organization.
**Creating block templates:**
1. **Navigate to Templates** - Click "Templates" in the left navigation menu
2. **Create new template** - Click "New block template" button
3. **Define template properties** - Set template UUID, block type, and language
4. **Edit template content** - Write the template code with placeholders
5. **Save template** - Click "Save template" to create the template
**Using block templates:**
1. **Open pipeline editor** - Navigate to the pipeline where you want to add the block
2. **Add new block** - Click the "+" button to add a new block
3. **Browse templates** - Select "Browse templates" from the block creation options
4. **Select template** - Choose the template you want to use from the available templates
5. **Name the block** - Enter a specific name for your new block instance
### Cross-project block sharing
For organizations with multiple projects:
1. **Export blocks** by downloading the block file from the source project
2. **Import blocks** by uploading the block file in the file browser of the target project
3. **Maintain version control** across project boundaries
4. **Share common utilities** and configurations
**Best practices for cross-project sharing:**
* **Document dependencies** - Clearly list required packages and configurations
* **Use relative imports** - Make blocks portable across different project structures
* **Test thoroughly** - Ensure blocks work in different project environments
## Best practices for block reuse
### Design reusable blocks
* **Make blocks generic** - Avoid hardcoded values, use runtime variables instead
* **Clear naming** - Use descriptive names that indicate the block's purpose
* **Documentation** - Add comments explaining what the block does and its inputs/outputs
* **Error handling** - Include proper error handling for different use cases
### Runtime variables for flexibility
Use runtime variables to make blocks adaptable across different pipelines:
```python theme={"system"}
# Instead of hardcoded values
database_name = "production_db"
# Use runtime variables
database_name = kwargs.get("database_name", "default_db")
```
Learn more about [using runtime variables](/getting-started/runtime-variable#using-runtime-variables).
### Block dependencies
When reusing blocks, consider their dependencies:
* **Data loaders** - Ensure data sources are accessible from all pipelines
* **Transformers** - Make sure input data format is consistent
* **Exporters** - Verify output destinations are appropriate for each pipeline
## Managing shared blocks
### Viewing shared block usage
1. **Click on a shared block** in any pipeline
2. **Look for the sharing indicator** (`<> X pipelines`)
3. **Click the indicator** to see which pipelines are using the block
4. **Review the usage** to understand the block's impact
### Updating shared blocks
When you modify a shared block:
1. **Changes apply everywhere** - All pipelines using the block will be affected
2. **Test thoroughly** - Ensure changes work for all use cases
3. **Communicate changes** - Notify team members about significant modifications
4. **Version control** - Use Git to track changes and enable rollbacks if needed
### Detaching blocks when needed
If you need to customize a shared block for a specific pipeline:
1. **Use the detach feature** - See [Detaching blocks](/guides/blocks/detach-blocks) guide
2. **Create a copy** - This creates an independent version
3. **Customize freely** - Make changes without affecting other pipelines
## Common use cases
### Data loading blocks
Reuse data loading logic across multiple pipelines:
```python theme={"system"}
# Shared data loader block
@data_loader
def load_customer_data():
# Load customer data from database
# Use runtime variables for database connection
return df
```
### Transformation blocks
Share common data transformations:
```python theme={"system"}
# Shared transformer block
@transformer
def clean_customer_data(df):
# Standard customer data cleaning logic
# Works with any customer dataset
return cleaned_df
```
### Export blocks
Reuse export logic for different destinations:
```python theme={"system"}
# Shared exporter block
@data_exporter
def export_to_warehouse(df):
# Export to data warehouse
# Use runtime variables for destination configuration
pass
```
## Related documentation
* [Block design overview](/design/blocks) - Understanding block architecture
* [Detaching blocks](/guides/blocks/detach-blocks) - Separating shared blocks
* [User-defined templates](/guides/blocks/user-defined-templates) - Creating custom block and pipeline templates
* [Runtime variables](/getting-started/runtime-variable) - Making blocks flexible
# SQL blocks
Source: https://docs.mage.ai/guides/blocks/sql-blocks
Execute SQL commands directly in your database, data warehouse, etc.
## Credentials
Before starting, you need to add credentials so Mage can execute your SQL
commands.
Follow the steps for the database or data warehouse of your choice:
* [BigQuery](/integrations/databases/BigQuery#add-credentials)
* [ClickHouse](/integrations/databases/ClickHouse#add-credentials)
* [Databricks](/integrations/databases/Databricks#add-credentials) (Mage Pro only)
* [Druid](/integrations/databases/Druid#add-credentials)
* [DuckDB](/integrations/databases/DuckDB#add-credentials)
* [MySQL](/integrations/databases/MySQL#add-credentials)
* [Oracle](/design/data-loading#oracledb) (Mage Pro only)
* [PostgreSQL](/integrations/databases/PostgreSQL#add-credentials)
* [Redshift](/integrations/databases/Redshift#add-credentials)
* [Snowflake](/integrations/databases/Snowflake#add-credentials)
* [SQLite](/integrations/databases/SQLite#add-credentials) (Mage Pro only)
* [Microsoft SQL Server](/integrations/databases/MicrosoftSQLServer#add-credentials)
* [Trino](/integrations/databases/Trino#add-credentials)
***
## Add SQL block to pipeline
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
***
## Configure SQL block
There are 4 - 5 fields that must be configured for each SQL block:
| Field | Required | Description |
| -------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Data provider | Yes | The database or data warehouse you want to execute your SQL commands in. |
| Profile | Yes | When you add your credentials to the `io_config.yaml` file, you added them under a key. That key is called the profile. Choose which set of credentials you want this SQL block to use. For detailed information about setting up profiles, see the [IO Config Setup documentation](/development/io_config_setup). |
| Use raw SQL | No | You can write raw SQL and itโll be executed as written in your data provider. |
| Database | Depends on data provider | Some data warehouses require that we explicitly state the name of the database we want to write to. If this is present, itโs required. The name of the table that is created follows this convention: `[database].[schema].[pipeline UUID]_[block UUID]`. |
| Schema to save | Yes | Every SQL block will save data to your data provider. The name of the table that is created follows this convention: `[schema].[pipeline UUID]_[block UUID]`. |
| Table | No | SQL blocks will automatically name your table for you using a naming convention (see the section `Automatic naming of tables` for more information). You can override this automatic naming convention by filling in a value in this field. |
| Write policy | Yes | How do you want to handle existing data with the same database, schema, and table name? See below for more information. |
#### Write policies
| Policy | Description |
| ------- | -------------------------------- |
| Append | Add rows to the existing table. |
| Replace | Delete the existing data. |
| Fail | Raise an error during execution. |
#### YAML configuration
You can also modify block configuration in pipeline's `metadata.yaml` file. Each block has a `configuration` field.
Example configuration
```yaml theme={"system"}
configuration:
data_provider: sqlserver
data_provider_profile: default
data_provider_schema: ''
export_write_policy: append
limit: 1000
limit_in_pipeline_run: 1
use_raw_sql: false
```
In addition to the fields mentioned in the table above. Here are some extra fields that can be included in the configuration:
| Field | Required | Description |
| ------------------------ | -------- | -------------------------------------------------------------------------------- |
| limit | No | The maximum number of rows to return in notebook. |
| limit\_in\_pipeline\_run | No | The maximum number of rows to return when running the block in the pipeline run. |
#### Dynamic `data_provider_profile`
You can configure the `data_provider_profile` in a SQL block to resolve dynamically at runtime using [Mage variable syntax](/development/variables/referencing-variables).
This is done by modifying the pipelineโs `metadata.yaml` directly.
**Example:**
```yaml theme={"system"}
configuration:
data_provider: snowflake
data_provider_profile: '{{ variables("DB_PROFILE") }}'
data_provider_schema: analytics
export_write_policy: append
```
When the pipeline runs, the profile name will be resolved from the referenced variable (environment variable, pipeline variable, or secret variable).
***
***
## Automatically created tables
Each SQL block will create a table in the data provider of your choice.
When you run a block, itโll execute your SQL command, then store the results in
a table created in your database or data warehouse.
### Using raw SQL
If youโre using version `0.9.69` or less and interpolating a SQL command from an upstream block that uses raw SQL:
if the downstream blockโs SQL command is referencing the upstream block using a variable such as `{{ df_1 }}` and ends the statement with a semi-colon (`;`), there is an interpolation error.
To work around the error, remove the semi-colon `;`.
A fix will be available in `0.9.70`.
If you toggle this setting, youโre responsible for writing the `CREATE TABLE` command and the
`INSERT` command.
For example, if a table already exists then you can write the `INSERT` statement:
```sql theme={"system"}
INSERT INTO mage.users
SELECT 1 AS id, 'Urza' AS username;
```
If the table doesnโt exist yet, you can write both the `CREATE TABLE` statement
and the `INSERT` statement:
```sql theme={"system"}
CREATE TABLE IF NOT EXISTS mage.users (
id BIGINT
, username VARCHAR(255)
);
WITH users AS (
SELECT
1 AS id
, 'Urza' AS username
)
INSERT INTO mage.users
SELECT
*
FROM users;
```
This SQL query will create a table named `mage.users` with 2 columns: `id` as a `BIGINT`
and `username` as a `VARCHAR(255)`.
Then, itโll insert a single row into that table.
#### Required SQL statements
When writing raw SQL, you must use at least 1 of the following statements:
* `SELECT`
```sql theme={"system"}
SELECT 1;
```
* `INSERT`
```sql theme={"system"}
INSERT INTO some_table
SELECT 1;
```
* `CREATE TABLE`
```sql theme={"system"}
CREATE TABLE some_table (id BIGINT);
```
* `DROP TABLE`
```sql theme={"system"}
DROP TABLE some_table;
```
* `UPDATE`
```sql theme={"system"}
UPDATE some_table
SET col1 = 'value1'
WHERE col2 = 'value2'
```
#### Multiple SQL statements
You can execute multiple SQL statements in a SQL block. Separate your SQL statements using a semi-colon (`;`).
### Automatic naming of tables
If you donโt choose the setting for using raw SQL,
the name of this automatically created table follows these conventions:
* If `Database` field is configured:
`[database].[schema].[pipeline UUID]_[block UUID]`
* If no `Database` field is configured: `[schema].[pipeline UUID]_[block UUID]`
Where pipeline UUID is the name of the current pipeline youโre editing.
Where block UUID is the name of the SQL block you are running.
### Upstream blocks
If your SQL block depends on upstream blocks that arenโt SQL blocks (e.g. Python
code blocks), then those blocks will also automatically create tables.
The name of those tables follows the same naming convention mentioned above.
***
## Variables
All SQL blocks have the following variables they can access in their query:
### `{{ execution_date }}`
The date and time the block is run.
**Example**
```sql theme={"system"}
SELECT '{{ execution_date }}' AS today
```
**Result**
| `today` |
| :--------------------------- |
| `2022-09-24 23:01:08.376057` |
***
If a SQL block has 1 or more upstream blocks, then they have access to their
parent blocksโ output using the following variable:
### `{{ df_1 }}`
Depending on how many upstream blocks there are, the variable name changes. For
example, if there are 3 upstream blocks then there are 3 variables that can be
accessed:
* `{{ df_1 }}`
* `{{ df_2 }}`
* `{{ df_3 }}`
The SQL block UI will display which variable maps to which upstream block. By
convention, the 1st added upstream block will be `{{ df_1 }}`, and every
upstream block added after that will have an incrementing number in the variable
name after the prefix `df_`.
**Example**
```sql theme={"system"}
SELECT a.id , b.username FROM {{ df_1 }} AS a LEFT JOIN {{ df_2 }} AS b ON
a.id = b.user_id LIMIT 1
```
**Result**
| `id` | `username` |
| :--- | :----------------- |
| `1` | `Sorcerer supreme` |
***
## Variable interpolation
Available in versions >= `0.9.59`
Interpolate values in the blockโs code using data from:
1. Upstream block output
2. Variables
1. Global variables
2. Pipeline variables
3. Runtime variables
3. Environment variables
### Upstream block output
Use the data from 1 or more upstream blockโs output by using the `block_output` function.
#### `block_uuid`
The UUID of the upstream block to get data from.
If argument isnโt present, data from all upstream blocks will be fetched.
| | |
| -------- | --------------------- |
| Optional | โ |
| Type | `str` |
| Example | `'data_loader_block'` |
#### `parse`
A lambda function to parse the data from an upstream blockโs output.
If the `parse` argument isnโt present, then the fetched data from the upstream blockโs output
will be interpolated as is.
| | |
| -------- | -------------------------------------------------------------- |
| Optional | โ |
| Type | `function` |
| Example | `lambda data, variables: data['runtime'] * variables['tries']` |
Arguments
* `data`
If the `block_uuid` argument isnโt present, then the 1st argument in the lambda function is
a list of objects.
The list of objects contain the data from an upstream blockโs output.
The positional order of the data in the list corresponds to the current blockโs upstream blocks
order.
For example, if the current block has the following upstream blocks with the following output:
1. `load_api_data`: `[1, 2, 3]`
2. `load_users_data`: `{ 'mage': 'powerful' }`
Then the 1st argument in the lambda function will be the following list:
```python theme={"system"}
[
[1, 2, 3],
{ 'mage': 'powerful' },
]
```
| | |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| Optional | โ |
| Type | If `block_uuid` argument is present, then the type depends on the output from that block. If `block_uuid` isnโt present, then the type is `list`. |
| Example | `{ 'mage': 'powerful' }` |
* `variables`
A dictionary containing pipeline variables and runtime variables.
| | |
| -------- | ---------------- |
| Optional | โ |
| Type | `dict` |
| Example | `{ 'fire': 40 }` |
#### Example
With `block_uuid`
```sql theme={"system"}
SELECT
'{{ block_output("magic_config", parse=lambda data, _vars: data["power"]) }}' AS power
, '{{ block_output("magic_config", parse=lambda data, _vars: data["spell"]) }}' AS spell
FROM mage.core_spells
```
```sql theme={"system"}
SELECT
'{{ block_output("data_loader_block", parse=lambda data, variables: data["runtime"] * variables["tries"]) }}' AS data
FROM mage.core_spells
```
```sql theme={"system"}
SELECT
'{{ block_output("load_recent_user", parse=lambda user, _variables: user["id"]) }}'
FROM mage.core_spells
```
Without `block_uuid`
```sql theme={"system"}
SELECT
'{{ block_output(parse=lambda data, _vars: data[0]["power"]) }}' AS power
, '{{ block_output(parse=lambda data, _vars: data[0]["spell"]) }}' AS spell
FROM mage.core_spells
```
```sql theme={"system"}
SELECT
'{{ block_output(parse=lambda data, variables: data[0]["runtime"] * variables[0]["tries"]) }}' AS data
FROM mage.core_spells
```
```sql theme={"system"}
SELECT
'{{ block_output(parse=lambda user, _variables: user[0]["id"]) }}'
FROM mage.core_spells
```
### Variables
Interpolate values from a dictionary containing keys and values from:
1. Global variables
2. Pipeline variables
3. Runtime variables
#### Example
```sql theme={"system"}
SELECT
'{{ variables("mana") }}' AS mana
FROM mage.dual_lands
```
### Environment variables
Interpolate values from the environment variables.
#### Example
```sql theme={"system"}
SELECT
'{{ env_var("ALLOW_POWER_9") }}' AS vintage
FROM mage.mox
```
***
## Executing pure SQL
Available in versions >= `0.9.59`
Mage automatically pre-processes your SQL command before sending it to the final destination.
The pre-process extracts and structures the command so that Mage can build upstream tables and
properly display preview data.
However, there are scenarios where the SQL command you write must be executed as is.
You can disable Mageโs pre-processing by opening a SQL block and checking the box labeled
Use raw SQL and Disable query preprocessing.
# Transformer blocks
Source: https://docs.mage.ai/guides/blocks/transformer-blocks
Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.
* [Overview](#overview)
* [Tutorial](#tutorial)
* [Core Concepts](#core-concepts)
* [Building Transformer Actions](#building-transformer-actions)
* [Transformer Action Reference](#transformer-action-reference)
* [Appendix: Filter Syntax](#appendix-filter-syntax)
* [Appendix: Column Types](#appendix-column-types)
## Tutorial
To introduce how the Transformer Actions library works, we will perform some cleaning actions on the Titanic Survival dataset. Open the Mage app and create a new pipeline. Add a data loader with the following code to load the Titanic Survival dataset:
```python theme={"system"}
from mage_ai.io.file import FileIO
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_file(**kwargs) -> DataFrame:
"""
Template for loading data from filesystem.
"""
filepath = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
return FileIO().load(filepath)
```
Run this block to load the Titanic Survival dataset into your pipeline.
You'll notice that there are three columns that have missing values: "Cabin", "Age", "Embarked". Let's perform imputation to fill in these missing values using the Transformer Action Library. Create a new transformer block below, and select the "Fill in missing values" template.
You should see the following code in your new transformer block:
```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis, ImputationStrategy
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
"""
Execute Transformer Action: ActionType.IMPUTE
"""
action = build_transformer_action(
df,
action_type=ActionType.IMPUTE,
arguments=df.columns, # Specify columns to impute
axis=Axis.COLUMN,
options={'strategy': ImputationStrategy.CONSTANT}, # Specify imputation strategy
)
return BaseAction(action).execute(df)
```
Let's break this code down:
* `build_transformer_action()` creates a transformer action **payload**, which is a dictionary that describes settings for the action to complete. The arguments passed in are
* `df` - the data frame to transform, used to infer column types
* `action_type` - specifies the action to perform. In this case, we are performing an `IMPUTE` action.
* `arguments` - specifies the columns to perform the action on. Currently, the action is configured to impute all columns of the data frame. Since we only want to transform "Cabin", "Age", and "Embarked", we will update this entry to only contain these column names.
* `axis` - specifies which axis to perform the action on. Imputing values is done on a per column basis, so this is a column transformation.
* `options` - these are extra settings to specify for this action. For an impute transformation, the main option to specify is `'strategy'`, dictating the imputation strategy to use. See [Fill in Missing Values](#fill-in-missing-values) for details on the valid imputation strategies. For now we will use `MODE`, which specifies to fill empty values with the most frequent value per column.
The modified transformer block is:
```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis, ImputationStrategy
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
"""
Execute Transformer Action: ActionType.IMPUTE
"""
action = build_transformer_action(
df,
action_type=ActionType.IMPUTE,
arguments=['Cabin', 'Age', 'Embarked'], # Specify columns to impute
axis=Axis.COLUMN,
options={'strategy': ImputationStrategy.MODE}, # Specify imputation strategy
)
return BaseAction(action).execute(df)
```
Execute this transformer - you should notice that all null values are now filled with the mode value of each respective column! After the payload is created by `build_transformer_action()`, the payload is used to construct the `BaseAction` object, representing a reusable instance of the transformation. Then the transformation is executed on the input data frame using `execute()`, returning the transformed data frame when done.
Suppose then you're interested in computing the average fare and age by each location of embarkation - Southampton (S), Cherbourg (C), and Queenstown (Q). Transformer Actions can help us here as well - create a new transformer using the "Aggregate/Aggregate by average value" template.
You should see the following code in your new transformer block.
```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
"""
Execute Transformer Action: ActionType.AVERAGE
"""
action = build_transformer_action(
df,
action_type=ActionType.AVERAGE,
action_code='', # Enter filtering condition on rows before aggregation
arguments=[], # Enter the columns to compute aggregate over
axis=Axis.COLUMN,
options={'groupby_columns': []}, # Enter columns to group by
outputs=[
# The number of outputs below must match the number of arguments
{'uuid': 'new_aggregate_column_1', 'column_type': 'number_with_decimals'},
{'uuid': 'new_aggregate_column_2', 'column_type': 'number_with_decimals'},
],
)
return BaseAction(action).execute(df)
```
As before, we can fill in the settings we need for performing this aggregation by average value:
* `action_code` is used by aggregation-type transformer actions to filter rows of the dataset before performing the aggregation. For example, if the "Embarked" column had null values, we could filter them out using the filter `'Embarked != null'`. In this case we won't perform any filtering. See [Filter Syntax](#appendix-filter-syntax) for more details on valid filter syntax.
* Since we want to aggregate over "Fare" and "Age", our arguments are `['Fare', 'Age']`
* In `options`, the only parameter is `groupby_columns` which is the list of the columns to group by when performing the aggregation. In this case, the column we want to use to form groups is "Embarked"
* `outputs` is used to specify the name and type of the new columns created by the aggregation. As there are two columns in `arguments` that we want to perform the aggregation over, there should be two corresponding entries in `outputs`.
The resulting template after these edits are:
```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.base import BaseAction
from mage_ai.data_cleaner.transformer_actions.constants import ActionType, Axis
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
from pandas import DataFrame
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def execute_transformer_action(df: DataFrame, *args, **kwargs) -> DataFrame:
"""
Execute Transformer Action: ActionType.AVERAGE
"""
action = build_transformer_action(
df,
action_type=ActionType.AVERAGE,
action_code='', # Enter filtering condition on rows before aggregation
arguments=['Fare', 'Age'], # Enter the columns to compute aggregate over
axis=Axis.COLUMN,
options={'groupby_columns': ['Embarked']}, # Enter columns to group by
outputs=[
# The number of outputs below must match the number of arguments
{'uuid': 'avg_fare_by_embarked_loc', 'column_type': 'number_with_decimals'},
{'uuid': 'avg_age_by_embarked_loc', 'column_type': 'number_with_decimals'},
],
)
return BaseAction(action).execute(df)
```
Run this transformer, and your output data frame should now include two new columns named "avg\_fare\_by\_embarked\_loc" and "avg\_age\_by\_embarked\_loc" which contain the average fare and average age by embarkation location. Each row gets a new value corresponding to the group which that row falls into.
This workflow can be summarized to describe how Transformer Actions can be used effectively:
1. Identify a transformation that may need to be made to your data
2. Create a transformer using the transformer action that corresponds to your template
3. Fill in the parameters for the transformer action to build your transformer action payload
4. Build a `BaseAction` object using the payload and execute the action to get the transformed data frame
## Core Concepts
The core object of any transformer action is the **payload**, a JSON object that describes the transformation to perform alongside settings for that transformation. A sample transformer action payload is shown below:
```json theme={"system"}
{
"action_type": "impute",
"action_arguments": ["age", "name"],
"action_options": {
"strategy": "constant",
"value": "32"
},
"axis": "column",
"action_variables": {
"age": {
"uuid": "age",
"column_type": "number"
},
"name": {
"uuid": "hourly_salary",
"column_type": "text"
}
}
}
```
The above transformer action imputes missing values in the "age" and "hourly\_salary" columns with the constant value `32`. Here is a list of the keys used in transformer action payloads:
| Key | Description |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| axis | Axis to perform the transformation on. Either 'row' or 'column'. |
| action\_type | Type of action to perform. See [Transformer Action Reference](#transformer-action-reference) for the list of action types. |
| action\_code | Text field for specifying any code such as filters. |
| action\_arguments | The columns to perform the transformation on. |
| action\_options | Other settings to provide to the transformer action. See [Transformer Action Reference](#transformer-action-reference) for specific details on which actions require which options. |
| action\_variables | Variable metadata for each input argument (column). Specifies column name and type. |
| outputs | Variable metadata for each output column. Specifies column name and type. See [Column Types](#appendix-column-types) for information on possible column types to specify. |
This JSON formatted payload is difficult to generate so it is recommended to use the `build_transformer_action` factory to generate the payload. The above snippet generates the same payload as above:
```python theme={"system"}
from mage_ai.data_cleaner.transformer_actions.utils import build_transformer_action
df = pd.DataFrame({...}) # your input data frame
payload = build_transformer_action(
df,
action_type=ActionType.IMPUTE,
arguments=['age'],
axis=Axis.COLUMN,
options={'strategy': 'constants', 'values': 32},
)
```
See [Building Transformer Actions](#building-transformer-actions) for detailed information on using `build_transformer_action`.
The transformer action payload, once built, can be used to construct a `BaseAction` object. This object processes the payload to identify the transformations to apply a data frame. Through this action object, the transformer action can be applied to a data frame using the `execute` function:
```python theme={"system"}
payload = build_transformer_action(...)
transformed_df = BaseAction(payload).execute(df)
```
# Building Transformer Actions
***build\_transformer\_action*** - `build_transformer_action(df: DataFrame, action_type: ActionType | str, arguments: List[str], action_code: str, options: Dict, axis: Axis | str, outputs: List[Dict]) -> Dict:`
Source: `mage_ai.data_cleaner.transformer_actions.utils.build_transformer_action`
Builds transformer action payload from arguments. The output of this function can be passed
as input to `transformer_actions.base.BaseAction` in order to perform the requested transformation.
Note: Action variables are inferred from `arguments` and `df`.
* **Args**:
* `df (DataFrame)`: The data frame to build a transformer action payload for.
* `action_type (ActionType | str)`: Transformer action to perform.
* `arguments (List[str], optional)`: Columns to perform this action on. Defaults to `[]`.
* `action_code (str, optional)`: Special code or query to execute with action. Defaults to `''`.
* `options (Dict, optional)`: Options specifying behavior of action. Defaults to `{}`.
* `axis (Union[Axis, str], optional)`: Axis of the data frame to apply the action to. Defaults to `Axis.COLUMN`.
* `outputs (List[Dict], optional)`: Specifies metadata of newly created columns. Defaults to `[]`.
* **Returns**: (`Dict`) Transformer action payload to use with `transformer_actions.base.BaseAction`
* **Example**:
```python theme={"system"}
action = build_transformer_action(
df,
action_type=ActionType.FIRST,
action_code='retailer != null and retail_price >= 0', # Enter filtering condition on rows before aggregation
arguments=['retail_price', 'review'], # Enter the columns to compute aggregate over
axis=Axis.COLUMN,
options={'groupby_columns': ['retailer']}, # Enter columns to group by
outputs=[
# The number of outputs below must match the number of arguments
{'uuid': 'first_retail_price', 'column_type': 'number_with_decimals'},
{'uuid': 'first_review', 'column_type': 'category'},
],
)
```
# Transformer Action Reference
* [Building Transformer Actions](#building-transformer-actions)
* [Transformer Action Reference](#transformer-action-reference)
* [Column Actions](#column-actions)
* [Aggregation Actions](#aggregation-actions)
* [Formatting Actions](#formatting-actions)
* [Clean Column Names](#clean-column-names)
* [Fix Syntax Errors](#fix-syntax-errors)
* [Reformat Values](#reformat-values)
* [Column Removal Actions](#column-removal-actions)
* [Select Columns](#select-columns)
* [Remove Columns](#remove-columns)
* [Row Shifting Actions](#row-shifting-actions)
* [Shift Up](#shift-up)
* [Shift Down](#shift-down)
* [Feature Scaling Actions](#feature-scaling-actions)
* [Normalize Data](#normalize-data)
* [Standardize Data](#standardize-data)
* [Other Column Actions](#other-column-actions)
* [Difference](#difference)
* [Fill In Missing Values](#fill-in-missing-values)
* [Remove Outliers](#remove-outliers)
* [Row Actions](#row-actions)
* [Drop Duplicates](#drop-duplicates)
* [Filter](#filter)
* [Remove Rows](#remove-rows)
* [Sort](#sort)
* [Appendix: Filter Syntax](#appendix-filter-syntax)
* [Appendix: Column Types](#appendix-column-types)
# Column Actions
## Aggregation Actions
Applies some aggregation function over groups determined by values in some columns. A set of grouping columns are provided, whose unique combinations of values are used to form groups. For example, consider the dataset below:
| Paid Dues | Due Amount |
| --------- | ---------- |
| Yes | \$5.00 |
| No | \$15.20 |
| Yes | \$3.50 |
| Yes | \$1.28 |
| No | \$25.03 |
The data could be grouped by "Paid Dues" to form the following groups:
| Paid Dues | Due Amount |
| --------- | ---------- |
| Yes | \$5.00 |
| Yes | \$3.50 |
| Yes | \$1.28 |
| Paid Dues | Due Amount |
| --------- | ---------- |
| No | \$15.20 |
| No | \$25.03 |
Then an aggregation function is applied to each group over the input columns. In the example, we can compute the total dues by group by aggregating over "Due Amount":
| Paid Dues | Total Due Amount |
| --------- | ---------------- |
| Yes | \$9.78 |
| No | \$40.23 |
Then the output of this aggregation is used to create a new column for each input column, where each row receives the corresponding aggregate of the group it falls into. In the example above, the final output data frame would be:
| Paid Dues | Due Amount | Total Due Amount by Payment Status |
| --------- | ---------- | ---------------------------------- |
| Yes | \$5.00 | \$9.78 |
| No | \$15.20 | \$40.23 |
| Yes | \$3.50 | \$9.78 |
| Yes | \$1.28 | \$9.78 |
| No | \$25.03 | \$40.23 |
All of these transformer actions have the same set of arguments when using `build_transformer_action`, but have different types corresponding to the different aggregation functions that could be applied to the groups.
**Example**
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.AVERAGE,
action_code='brand != null',
arguments=['revenue', 'cost'],
axis=Axis.COLUMN,
options={'groupby_columns': ['brand']},
outputs=[
{'uuid': 'avg_rev_per_brand', 'column_type': 'number_with_decimals'},
{'uuid': 'avg_cost_per_brand', 'column_type': 'number_with_decimals'},
],
)
```
**Args**
* ***action\_code:*** Filter for selecting rows before aggregation is performed. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.
* ***arguments:*** Columns to compute aggregate values over. For each argument a new column will be created to store the aggregates values over the column.
* If no arguments are provided, no aggregation is performed
* ***options*:**
* `groupby_columns` (optional) - list of columns to form groups with when aggregating. A group consists of each unique combination of values chosen from each of the columns specified in this option.
* If this is empty, aggregation is performed over the first column specified in *arguments* (only if *arguments* is nonempty - if empty no aggregation is performed) and only one column is added
```
The following options are used as a shortcut to simplify filtering over time windows.
Let $A$ and $B$ be datetime columns, and let $\text{window}$ be an integer describing the length of
a time window in seconds. If the following options are specified, rows for which the time value for
$B \in [A, A + \text{window}]$ are selected (All rows where value for $B$ must (a) be greater than
or equal to the value for $A$ and (b) less than or equal to the value $A+window$).
```
* `timestamp_feature_a` (optional) - time feature A (a column in the data frame)
* `timestamp_feature_b` (optional) - time feature B (a column in the data frame)
* `window` (optional) - max time window between feature A and feature B in seconds
If some (or none) of these options are specified, then the filtering is not performed. Regardless if this filtering is performed, *action\_code* filtering is still performed.
* ***outputs*:** Specifies the metadata for each output column of the aggregation (corresponding to the aggregation created from input argument in *arguments*). This array must be the same length as *arguments*, and each entry must contain the following metadata:
* `uuid (string)`: Name of the new column
* `column_type (string | ColumnType)`: Type of the new column ( See `mage_ai.data_cleaner.column_types.constants.ColumnType` for options)
**Aggregation Functions**
These are the possible aggregations that can be applied to each group per each input column. To use the aggregation function specify the corresponding action type in `build_transformer_action`.
| Name | Action Type Enum | Description |
| -------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------- |
| ***Average*** | `ActionType.AVERAGE` | Averages input column value over each group |
| ***Count Distinct*** | `ActionType.COUNT_DISTINCT` | Counts the number of distinct rows in each group per input column |
| ***Count*** | `ActionType.COUNT` | Counts the number of rows in each group per input column |
| ***First*** | `ActionType.FIRST` | For each input column, selects first item in each group (as appearing in the order of the original data frame) |
| ***Last*** | `ActionType.LAST` | For each input column, selects last item in each group (as appearing in the order of the original data frame) |
| ***Maximum*** | `ActionType.MAX` | Gets the maximum value per group for each input column |
| ***Median*** | `ActionType.MEDIAN` | Gets the median value per group for each input column |
| ***Minimum*** | `ActionType.MIN` | Gets the maximum value per group for each input column |
| ***Sum*** | `ActionType.SUM` | Sums over all values in each group per input column |
## Formatting Actions
These transformer actions involve reformatting column names or the values in columns
### Clean Column Names
Cleans column names according to the following rules:
1. Names are converted to snake case
2. All wrapping whitespace or underscores are removed
3. All non-alphanumeric characters (except "\_") are removed
4. Numbers are prefixed with "number\_"
5. All Python keywords are postfixed by "\_"
Snake case is chosen as
1. all characters are lower case
2. no whitespace is used (which makes referring to columns in code easier)
| Original Name | Cleaned Name | Notes |
| ----------------------- | -------------------- | ------------------------------------------------- |
| good\_name | good\_name | columns in snake case are left unchanged |
| bad case | bad\_case | |
| %@#342%34@@#342 | number\_34234342 | Symbols are removed first, then number is cleaned |
| PascalCaseColumn | pascal\_case\_column | |
| camelCaseText | camel\_case\_text | |
| \_\_\_snake\_case\_\_\_ | snake\_case | Surrounding underscores are stripped |
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.CLEAN_COLUMN_NAME,
arguments=df.columns,
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** Columns whose name to clean. If empty, no columns names are cleaned.
### Fix Syntax Errors
Marks syntax errors in column values. Syntax errors are defined as values that are improperly formatted or of the incorrect type. For example:
* A number in a text column
* An improperly formatted email, phone number, or zip code
* An improperly formatted date
Values that break these syntax errors are currently marked as:
* `"invalid"` if column is of categorical or string type
* `pandas.NaT` if column is a datetime
* `np.nan` if column is a number type
*WIP*: Intelligently fix syntax errors in these columns automatically
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.FIX_SYNTAX_ERRORS,
arguments=df.columns,
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** Columns to mark syntax errors for. If empty no columns are checked.
### Reformat Values
Reformats values in column based on requested action. The currently supported reformats are:
* Standardize capitalization (`reformat = 'caps_standardization'`): Forces text column to follow a single capitalization strategy (either lowercase or uppercase)
* Convert currencies to a number (`reformat = 'currency_to_num'`): Converts a currency value (by default stored as a string) to a decimal number. If unable to convert, doesn't perform conversion
* Supported currencies: **\$, CAD, ยฃ, โฌ, ยฅ, Rs, ๅ **
| Original String | Parsed Decimal | Notes |
| ---------------- | -------------- | ----------------------------------------------- |
| " \$ 10000" | 10000 | |
| "- ยฅ 22.324523" | -22.324523 | Negation in front of symbol supported |
| " 10000 ๅ " | 10000 | Currencies with postfixed symbols are supported |
| "0.42 โฌ" | 0.42 | |
| "- 3.42032 CAD" | -3.42032 | |
| "Rs - 100000.23" | -100000.23 | Negation after symbol is supported |
* Convert string to datetime (`reformat = 'date_time_conversion'`): Converts a string value representing a datetime to a `pandas.Timestamp` object if correctly formatted, else converts to `None`.
* Trim whitespace (`reformat = 'trim'`): Removes leading and trailing whitespace from text columns.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REFORMAT,
arguments=[],
axis=Axis.COLUMN,
options={'reformat': 'caps_standardization', 'capitalization': 'lowercase'},
)
```
**Args**
* ***arguments:*** Columns to apply the reformat action to
* ***options:***
* `reformat` (optional): The reformat action to apply to the specified columns. Can be:
* `None` - no reformat is applied
* `'caps_standardization'` - standardize capitalization
* `'currency_to_num'` - convert currency string to number
* `'date_format_conversion'` - convert datetime string to `pandas.Timestamp`
* `'trim'` - remove leading and trailing whitespace from text columns
* `capitalization` (optional): Specifies the capitalization strategy to use when standardizing capitalization. This argument is ignored unless `reformat = "caps_standardization"`. Options are `['lowercase', 'uppercase']`.
## Column Removal Actions
### Select Columns
Keeps only the specified columns; removes all other columns from data frame.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SELECT,
arguments=[],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to keep in the data frame. If empty, all columns are removed.
### Remove Columns
Drops the specified columns.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REMOVE,
arguments=[],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to remove in the data frame. If empty, no columns are removed.
## Row Shifting Actions
These actions copy a column with all rows shifted up or down by a certain amount. When rows are shifted, one or more values are deleted and some new values have to be filled in. The fill-in data types are:
| Type | Fill Value |
| ----------- | ------------ |
| Numeric | `np.nan` |
| Datetime | `pandas.NaT` |
| Other Types | `None` |
### Shift Up
Shifts all rows in a column up by one. The first entry is deleted and the last entry is turned to a null type.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SHIFT_UP,
arguments=['Name'],
axis=Axis.COLUMN,
outputs=[{'uuid': 'shifted_name', 'type': 'text'}],
)
```
**Args**
* ***arguments:*** The column to shift up. At most one column can be shifted at once:
* If multiple columns are provided as arguments, only the first is shifted up
* If no columns are provided then no shifting is performed
* ***outputs:*** Metadata for the newly added upshifted column. Must have at most a single entry containing the following information:
* `uuid`: Name of the new upshifted column
* `type`: Data type of the new upshifted column
### Shift Down
Shifts all rows up by *periods* rows. The last *periods* entries are deleted and the first *periods* are converted to a null type.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SHIFT_DOWN,
arguments=['created_time'],
axis=Axis.COLUMN,
options={'periods': 3},
outputs=[{'uuid': 'downshifted_created_time', 'type': 'datetime'}],
)
```
**Args**
* ***arguments:*** The column to shift down. At most one column can be shifted at once:
* If multiple columns are provided as arguments, only the first is shifted down
* If no columns are provided then no shifting is performed
* ***options:***
* `periods` (optional): Specifies how many rows to shift down. Defaults to 1.
* `groupby_columns` (optional): Specifies columns to group by. If specified, shifts each *group* by *periods* rows instead of the entire data frame. The resulting column is then returned. If unspecified no grouping is performed. Multiple columns can be specified to group by.
* ***outputs:*** Metadata for the newly added downshifted column. Must have at most a single entry containing the following information:
* `uuid`: Name of the new downshifted column
* `type`: Data type of the new downshifted column
## Feature Scaling Actions
These actions are used to normalize the range of independent variables or features of data. These actions are generally performed during the data preprocessing step.
| Type | Formula |
| --------------- | ---------------------- |
| Normalization | `(X-Xmin)/(Xmax-Xmin)` |
| Standardization | `(X-Xmean)/Xstd` |
### Normalize Data
Perform Normalization on the desired columns of a dataset.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.NORMALIZE,
arguments=['Column1','Column2'],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to Normalize.
* If no columns are provided then no Normalization is performed
### Standardize Data
Perform Standardization on the desired columns of a dataset.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.STANDARDIZE,
arguments=['Column1','Column2'],
axis=Axis.COLUMN,
)
```
**Args**
* ***arguments:*** The columns to Standardize.
* If no columns are provided then no Standardization is performed
## Other Column Actions
### Difference
Adds a new column where each row is the difference between every consecutive two rows in the original column.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.DIFF,
arguments=['position'],
axis=Axis.COLUMN,
outputs=[{'uuid': 'delta_position', 'column_type': 'number_with_decimals'}],
)
```
**Args**
* ***arguments:*** The column to calculate difference on. At most one column can be specified:
* If multiple columns are provided as arguments, only the first is used in calculated
* If no columns are provided then no calculation
* ***outputs:*** Metadata for the newly added difference column. Must have at most a single entry containing the following information:
* `uuid`: Name of the difference column
* `type`: Data type of the difference column (most often will be a number)
### Fill In Missing Values
Imputes missing values using different strategies:
| Strategy | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Average | Fill missing values with the average per column. |
| Column | Fill missing values with values from another column. |
| Constant | Fill missing values with a placeholder constant. Can fill with a specific value; otherwise a default placeholder is chosen by column type. |
| Median | Fill missing values with the median per column. |
| Mode | Fill missing values with the most frequent value per column. |
| Random | Fill missing values with a randomly sampled non-null value from same column. |
| Sequential | Fill missing values with the previous row. Should only be used with timeseries columns. |
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.IMPUTE,
arguments=df.columns, # Specify columns to impute
axis=Axis.COLUMN,
options={'strategy': ImputationStrategy.CONSTANT}, # Specify imputation strategy
)
```
**Args**
* ***arguments:*** The columns to impute (fill in the missing values).
* ***options:***
* `strategy` (required): Specifies strategy to use. See table above or enum `ImputationStrategy`
* `value` (optional): Value to fill in missing value with. This argument is ignored unless `strategy = 'constant'`, in which case the constant value used to impute is `value`.
* `timeseries_cols = []` (optional): Columns which specify the time series in the data frame. This argument is ignored unless `strategy = 'sequential'`, in which case the data frame is sorted along these columns to sequentially impute data in the order of the time series.
### Remove Outliers
Removes multidimensional outliers in the data frame by analyzing numerical columns. If there are no numerical columns, no check is performed. Null values are ignored when checking for outliers.
The following methods are supported for detecting outliers:
| Method | Description |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `'itree'` | Uses the [Isolation Tree](https://scikit-learn.org/stable/modules/outlier_detection.html#isolation-forest) algorithm. Number of trees is set to 100. |
| `'lof'` | Uses the [Local Outlier Factor](https://scikit-learn.org/stable/modules/outlier_detection.html#local-outlier-factor) algorithm. Number of neighbors for computing local density scales with amount of data and contamination rate. |
| `'auto'` | Automatically determines from above methods based on number of data points and dimensionality of the data. |
Notes:
* The contamination rate for both algorithms is estimated via the per-dimension IQR (which discounts multidimensional interactions but still estimates how many outliers there likely are)
* If the input data has more than 20 dimensions, the dimensionality is reduced using PCA
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REMOVE_OUTLIERS,
arguments=df.columns, # Specify columns to remove outliers from
axis=Axis.COLUMN,
options={'method': 'auto'}, # Specify algorithm to use for outlier removal
)
```
**Args**
* ***arguments:*** The set of columns to consider when checking for outliers. Of these columns, only numerical columns will be used to calculate outliers.
* ***options:***
* `method` (optional): Specifies outlier detection method to use:
* `'itree'`
* `'lof'`
* `'auto'`
Defaults to `'auto'`.
# Row Actions
These are transformer actions that can be applied to each row of a data frame, and so have `axis = Axis.ROW`.
### Drop Duplicates
Drops duplicate rows from the data frame.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.DROP_DUPLICATE,
arguments=[],
axis=Axis.ROW,
options={'keep': 'first'},
)
```
**Args**
* ***arguments:*** the columns along which comparisons will be made to check for duplicate rows. If empty, all columns will be used to perform comparison.
* ***options:***
* `keep` (optional): specifies policy on keeping duplicates:
* `'first'` - keeps the first occurrence of the row
* `'last'` - keeps the last occurrence of the row
* `False` - removes occurrences of duplicates
Defaults to `'last'`.
### Filter
Filters row by boolean condition. Selects all rows that meets the filter condition.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.FILTER,
axis=Axis.ROW,
action_code='column1 != null and column2 <= 5'
)
```
**Args**
* ***action\_code:*** filter to apply to select rows. See [Filter Syntax](#appendix-filter-syntax) for more information on valid filter syntax.
### Remove Rows
Removes rows specified by their index.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.REMOVE,
axis=Axis.ROW,
options={'rows': []},
)
```
**Args**
* ***options:***
* `rows`: List of the indices corresponding to the rows to remove. If empty, no rows are removed.
### Sort
Sorts rows in the specified order.
**Example**:
```python theme={"system"}
build_transformer_action(
df,
action_type=ActionType.SORT,
arguments=[],
axis=Axis.ROW,
options={'ascending': True},
)
```
**Args**
* ***arguments:*** the columns whose value to consider while sorting rows. If empty then sort is not performed.
* ***options:***
* `ascending` (optional): Specifies how to sort rows. If `True`, then rows are sorted in ascending order; if `False` sorted in descending order. Defaults to `True`.
* If sorted in ascending order, null values are placed at the top of the data frame in the same order as they appeared in the original data frame
* If sorted in descending order, null values are placed at the bottom of the data frame in the same order as they appeared in the original data frame
* `ascendings` (optional): A list of booleans specifying if each column specified in *arguments* should be sorted in ascending order (must be the same length). If specified, this list will take precedence over the `ascending` option.
# Appendix: Filter Syntax
Valid filters are composed of clauses of the form:
```
[column_name] [op] [value or column_name]
```
* List of valid operations: `==, !=, <=, <, >, >=, contains, not contains`
* `contains` (and `not contains`) checks if the value is a substring (or not a substring) of values in the column specified.
* The right hand side of each clause can either be a literal value (boolean, number, string) or another column to compare to.
Clauses can be connected using the `and` and `or` operators and grouped with parentheses.
```
(brand != null and value > 60) or (discounted == null)
```
**Rules on Naming Columns**
* If column names contain only alphanumeric characters and the special character '\_', then the raw column name can used to filter code.
Example: `age != null and price_of_item >= 50.23`
* If columns contain whitespace, newlines, and some of the following special characters: `+=-*&^%$! ?~|<>(){}[],.`, then the quoted column name can be used to filter code. Any quote character can be used.
Example: `("+= -*&^%$! ?~ |<>" == False and ' kas22d fe ($)' > 50.23) or ("dis>>> ??cou nted" contains "partial")`
* If columns contain any other special characters apart from those listed above, the column cannot be filtered. This is a fundamental limitation of pandas, see [here](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html?highlight=query#pandas.DataFrame.query:~:text=During%20parsing%20a,not%20be%20escaped) for more information.
# Appendix: Column Types
These are list of column types used in the Mage tool. One of these column types should be specified for every entry in the *options* argument of a transformer action. The enum source is `mage_ai.data_cleaner.column_types.constants.ColumnType`.
| Type | Enum | String Value | Description |
| ------------------------- | -------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| Category | `ColumnType.CATEGORY` | "category" | Categorical variable that takes one of some finite set of values. This set has at most 255 entries. |
| High Cardinality Category | `ColumnType.CATEGORY_HIGH_CARDINALITY` | "category\_high\_cardinality" | Categorical variable that takes one of some finite set of values. This set has more than 255 entries. |
| Datetime | `ColumnType.DATETIME` | "datetime" | Datetime objects or strings representing a datetime. |
| Email | `ColumnType.EMAIL` | "email" | Strings that contains emails. |
| List | `ColumnType.LIST` | "list" | Lists of primitives (integers, strings, booleans, floats) of variable length. |
| Number | `ColumnType.NUMBER` | "number" | Integer numbers. Can be a Python integer, NumPy integer or a stringified integer. |
| Floating Point Number | `ColumnType.NUMBER_WITH_DECIMALS` | "number\_with\_decimals" | Floating point numbers. Can be a Python float, NumPy float, or a stringified float. |
| Phone Number | `ColumnType.PHONE_NUMBER` | "phone\_number" | Phone numbers (either in string form or integer form). |
| Text | `ColumnType.TEXT` | "text" | Text values, such as sentences or paragraphs. |
| Binary | `ColumnType.TRUE_OR_FALSE` | "true\_or\_false" | A variable that takes on two different values (including booleans). |
| Zip Code | `ColumnType.ZIP_CODE` | "zip\_code" | Zip codes (either in string form or integer form). |
# User-defined templates
Source: https://docs.mage.ai/guides/blocks/user-defined-templates
Learn how to create and use custom block templates and pipeline templates in Mage for better code reusability and standardization.
## Overview
User-defined templates allow you to create reusable code patterns for blocks and pipelines, enabling faster development and consistent coding standards across your organization. Templates can include placeholders for customization and can be shared across projects.
## Block Templates
### Creating block templates
Block templates are reusable code patterns that can be customized with variables. They help standardize common patterns across your organization.
**Steps to create a block template:**
1. **Navigate to Block templates** - Click "Templates" in the left navigation menu, then open **Block templates**. You can also go directly to `/apps/templates/blocks`.
2. **Create new template** - Click "New block template" button
3. **Define template properties**:
* **Template UUID** - Enter a unique identifier (e.g., `sql_data_loader`)
* **Block type** - Select the type (Data loader, Transformer, Data exporter, etc.)
* **Language** - Choose the programming language (Python, SQL, R, YAML, etc.)
4. **Edit template content** - Write the template code with placeholders
5. **Save template** - Click "Save template" to create the template
Template UUIDs can include letters, numbers, underscores, hyphens, and folder separators (`/`). They must start with a letter or number, cannot include empty, `.` or `..` folder segments, and must be 255 characters or fewer. The block type and language are locked after the template is created.
### Template File Structure
When you create a block template, Mage automatically creates the following file structure:
```
custom_templates/
โโโ blocks/
โโโ {template_uuid}/
โโโ metadata.yaml # Template metadata and configuration
โโโ {template_uuid}.py # Template code file
```
**Template files:**
* **`metadata.yaml`** - Contains template metadata, block type, language, and other configuration
* **`{template_uuid}.py`** - The actual template code file that users will customize
The code file extension follows the selected language. For example, Python templates use `.py`, SQL templates use `.sql`, R templates use `.r`, and YAML templates use `.yaml`.
**Example structure for `test_template`:**
```
custom_templates/
โโโ blocks/
โโโ test_template/
โโโ metadata.yaml
โโโ test_template.py
```
### Template Library Interface
The template library provides a centralized location to manage all your block and pipeline templates:
**Template Library Features:**
* **Template categories** - Filter templates by type (Blocks or Pipelines)
* **Block template categories** - Filter block templates by block type
* **Pipeline template sources** - Browse all pipeline templates or team-created pipeline templates
* **Template preview** - View template details before using
* **Template management** - Edit, delete, or duplicate existing templates
**Accessing Templates:**
1. **From left navigation** - Click "Templates" to access the template library. The default Templates page redirects to the block template library at `/apps/templates/blocks`.
2. **From pipeline editor** - Use "Browse templates" when adding new blocks
3. **From pipeline creation** - Select "From a template" when creating new pipelines
4. **Template organization** - Templates are organized by type and category for easy discovery
**Viewing Pipeline Templates:**
* **Open Pipeline templates** - Click the Pipelines tab in the template library or go directly to `/apps/templates/pipelines`
* **Filter by source** - Select **All templates** or **Team** in the pipeline template navigation
* **View template details** - See template name, description, and metadata
* **Select template** - Click on a template to use it for creating new pipelines
### Using block templates
**Steps to use a block template:**
1. **Open pipeline editor** - Navigate to the pipeline where you want to add the block
2. **Add new block** - Click the "+" button to add a new block
3. **Browse templates** - Select "Browse templates" from the block creation options
4. **Select template** - Choose the template you want to use from the available templates
5. **Name the block** - Enter a specific name for your new block instance
6. **Customize as needed** - Modify the generated code for your specific use case
## Pipeline Templates
### Creating pipeline templates
Pipeline templates define complete pipeline structures that can be customized for different use cases.
**Steps to create a pipeline template:**
1. **Navigate to Pipeline templates** - Open **Templates**, then **Pipeline templates**, or go directly to `/apps/templates/pipelines`.
2. **Create new template from an existing pipeline** - From the pipeline list, right-click the pipeline you want to reuse and select the action to create a template. This opens `/apps/templates/pipelines/new?pipeline_uuid={pipeline_uuid}`.
3. **Configure template settings** - Set the template UUID, name, description, and other properties.
4. **Save template** - The pipeline structure will be saved as a reusable template.
Pipeline template UUIDs follow the same rules as block template UUIDs: letters, numbers,
underscores, hyphens, and folder separators (`/`) are allowed, the value must start with a
letter or number, `.` and `..` folder segments are rejected, and the value must be 255
characters or fewer.
**Example pipeline template structure:**
```
custom_templates/pipelines/
โโโ example_pipeline_template/
โโโ metadata.yaml
โโโ triggers.yaml
```
### Using pipeline templates
**Steps to use a pipeline template:**
1. **Navigate to the pipelines page** in your project
2. **Click "New"** button to open the pipeline creation menu
3. **Select "From a template"** from the dropdown options
4. **Choose your template** from the available pipeline templates
5. **Customize the pipeline** - Modify blocks, connections, and settings as needed
6. **Save the pipeline** with your specific configuration
**Note**: Pipeline templates are created from existing pipelines using the right-click context menu. This allows you to reuse successful pipeline patterns across your projects.
## Best Practices
### Template design
* **Keep templates focused** - Each template should serve a specific purpose
* **Use clear variable names** - Make variable purposes obvious
* **Include comprehensive documentation** - Explain usage and customization options
* **Test with different configurations** - Ensure templates work with various inputs
* **Follow coding standards** - Maintain consistent style and practices
### Template maintenance
* **Regular updates** - Keep templates current with latest Mage features
* **User feedback** - Collect and incorporate user suggestions
* **Performance optimization** - Monitor and improve template performance
* **Security review** - Ensure templates follow security best practices
# Community Examples
Source: https://docs.mage.ai/guides/community-examples
A collection of Mage projects and examples created by our community.
## Written Tutorials
### [Simplifying Data Pipelines With Mage Data Integrations](https://ethanbrown.substack.com/p/simplifying-data-pipelines-with-mage)
Ethan Brown introduces Mage Data Integration Blocks, bridging no-code simplicity with standard pipeline capabilities for seamless data integration across various sources. From Postgres to Salesforce, Ethan showcases the effortless process of pulling and leveraging data within standard pipelines, promising streamlined development and efficient syncs. Explore the future of data integration with Mage, where complexity meets accessibility.
### [Mage.ai โ Tutorials and Intro](https://janzednicek.cz/en/category/mage-ai-etl-tool/)
Jan provides a series of tutorials from intro and installation to building pipelines and charts.
### [How to Streamline Communication in Data Pipelines Using Mage](https://medium.com/@xiaoxugao/1ab2a885dd73)
In her post, Xiaoxu introduces the concept of a *communication lifecycle* for data pipelines. Proactive feedback to stakeholders is essential for establishing trust and building confidence in data. Xiaoxu demonstrates how Mage's [callbacks](/design/blocks/callbacks) can be used to automate tasks based on block status, reducing the communication lifecycle and establishing a healthy relationship between data owners and consumers.
### [Modern Data Engineering with Mage](https://www.analyticsvidhya.com/blog/2023/06/modern-data-engineering-with-mage-empowering-efficient-data-processing/)
Sri Nikitha walks through a demo of Mage: from installation to a sample pipeline that loads data from a NYC taxi dataset, transforms it, and exports to BigQuery.
### [Postgres to Snowflake Data Transfer](https://sudsy-pail-0c7.notion.site/Data-Migration-Tutorial-using-Mage-46c027b4095d4c9c8216c566eee3c3a4)
Jafar Sharif walks through a tutorial on how to transfer data from Postgres to Snowflake using Mage, including some core ETL functionality. A quick and fun read!
## Video Tutorials
### How To Create Data Pipelines in Mage
Shashank walks through a high-level overview of how to create data pipelines using Mage.
### End-to-end Data Engineering Project with Mage
Using Mage and Python for orchestration and ingestion, Darshil walks through how to take his dataset from source to visualization using a combination of popular tools, including Mage, GCS, BigQuery, and Looker Studio.
### Building an ETL Pipeline with Mage
Arul discusses the process of constructing an ETL (extract-transform-load) pipeline in Mage. Starting from Netflix's top 100 movie dataset, Arul extracts data, transforms it to a useable format, ans loads it into a database for further analysis.
### Apache Airflow Vs. Mage.AI! Apache and Mage.AI Compared for Data Orchestration!
The Data Guy compares and contrasts Apache Airflow and Mage.AI!
# How to build a data integration pipeline
Source: https://docs.mage.ai/guides/data-integration-pipeline
Here are the high level steps to build a data integration pipeline:
## Demo video
***
## Requirements
You must run Mage using [Docker](/getting-started/setup#using-docker) because there are a lot of
dependencies required for data integrations that can be challenging to install using `pip` or `conda`.
***
## Add new data integration pipeline
1. Open Mage in your browser and click the **`[+ New pipeline]`** button.
2. Click the dropdown menu option **Data integration**.
***
## Configure source
1. Click the dropdown menu under **Select source** and choose the option you
want to load data from (e.g. Amplitude).
2. Depending on the chosen source, youโll need to enter credentials and options
into the section labeled **Configuration**. For example, if you chose
Amplitude, youโll need to enter credentials like this:
```yaml theme={"system"}
account: ...
database: ...
password: "{{ env_var('PASSWORD') }}"
schema: ...
table: ...
username: ...
warehouse: ...
```
**Best practices**: you can interpolate values in the configuration using the
following syntax:
1. `"{{ env_var('SECRET_KEY') }}"`: this will extract the value from the
`SECRET_KEY` key in your environment variables.
2. `"{{ variables('SECRET_KEY') }}"`: this will extract the value from the
`SECRET_KEY` key in your
[runtime variables](/getting-started/runtime-variable).
You can also add a prefix to output tables by configuring the following variables in the `source` configuration yaml:
```yaml theme={"system"}
_patterns:
destination_table: MY_PREFIX_{{ variables('stream') }}
```
3. After you enter in all the credentials, click the button
**`[Fetch list of streams]`** under the section labeled **Select stream**.
4. Shortly after clicking the above button, click the new dropdown menu under
the section labeled **Select stream**. Then, choose the stream (aka table)
you want to load data from.
### Configure stream
After selecting a stream (aka table), youโll need to configure the schema.
Configuring the schema informs your pipeline on which fields to synchronize, how
to determine if a record is unique, and what to do if their are conflicts (aka
duplicate records).
Here are the steps you can optionally go through:
1. Selected field(s):
* Check the box next to the field name to include the field in your
synchronization.
* Uncheck the ones you donโt want to sync.
2. Field type(s)
* Each field will have a default field type.
* Add additional field types or remove them if they donโt fit your needs.
3. Unique field(s)
* On the right of the field names, there is a box you can check that will
determine which field(s) need to have unique values.
* If the box is un-checkable, that means you cannot use that field as a
unique field.
4. Bookmark field(s)
* Under the column labeled **Bookmark**, check the box to use the field as a
way to keep track of progress during synchronization.
* Upon every synchronization, these columns are used to pick up from where
the previous synchronization left off. In addition, if a synchronization
fails midway, these bookmark columns are used to track the record that was
most recently successful.
5. Replication method
* `FULL_TABLE`: synchronize the entire set of records from the source.
* `INCREMENTAL`: synchronize the records starting after the most recent
bookmarked record from the previous synchronization run.
6. Unique conflict method: choose how to handle duplicate records
* `IGNORE`: skip the new record if itโs a duplicate of an existing record.
* `UPDATE`: update the existing record with the new recordโs properties.
Once you've configured the schema for the stream, you can optionally choose to
run the stream in parallel. Parallel streams will be attempted to be synchronized
in parallel if there are enough worker resources available. This may speed up the
sync process if you have a large number of streams.
### Editing bookmark property values
* You can edit the bookmark values for your next sync by editing them
in the `Bookmark property values` table under the
**Manually edit bookmark property values** setting in the **Settings** section.
* The **Manually edit bookmark property values** setting will not appear until
at least one column is selected as a bookmark field.
* In order to enable the toggle that displays the `Bookmark property values`
table, you need to select a destination first.
* Click the toggle and enter the new value for your bookmark property, and then
click `Save`. You MUST click the `Save` button in order for the bookmark value
to be updated.
***
## Configure destination
1. Click the dropdown menu under **Select destination** and choose the option
you want to export data to (e.g. Snowflake).
2. Depending on the chosen source, youโll need to enter credentials and options
into the section labeled **Configuration**. For example, if you chose
Snowflake, youโll need to enter credentials like this:
```yaml theme={"system"}
account: ...
database: ...
password: "{{ env_var('PASSWORD') }}"
schema: ...
table: ...
username: ...
warehouse: ...
```
Best practices: you can interpolate values in the configuration using the
following syntax:
1. `"{{ env_var('PASSWORD') }}"`: this will extract the value from the
`PASSWORD` key in your environment variables.
2. `"{{ variables('PASSWORD') }}"`: this will extract the value from the
`PASSWORD` key in your runtime variables.
***
## Run pipeline and start sync
Once youโre done configuring your pipeline, go back to the pipelineโs trigger
page by clicking the name of your pipeline in your header.
The breadcrumbs in your header could look like this:
`Pipelines / pipeline name / Edit`.
Once youโre on the pipeline triggers page, create a
[new scheduled trigger](/design/data-pipeline-management#create-trigger)
and choose the `@once` interval. For more schedules, read the
[other options here](/orchestration/triggers/schedule-pipelines).
### Monitoring pipeline
After you create a scheduled trigger, click the **`[Start trigger]`** button at
the top of the page. Youโll see a new pipeline run appear shortly on the screen.
You can [click the logs](/development/observability/logging) for that pipeline
run to view the progress of your synchronization. Any string config values over
8 characters long in the logs will be redacted for security purposes. You'll see
these values hidden as `********` in the logs.
***
## Support
If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat)
# Mage Pro CI/CD workflow
Source: https://docs.mage.ai/guides/data-sync/cicd-workflow
Deploy and manage your pipeline code safely across environments using Mage Pro's integrated deployment application.
## Introduction
Mage Pro supports two CI/CD approaches to fit different organizational needs. Both follow the same core workflow: develop โ staging โ deploy to production using the Git terminal and the Mage Pro deployment application.
This document clarifies the two primary CI/CD approaches available in Mage Pro:
1. **CI/CD by cluster** - Separate clusters for development, staging, and production
2. **CI/CD by workspace** - Multiple workspaces within a single cluster
Both approaches follow the same core principles: develop in isolation, test thoroughly, and deploy safely to production. The choice between cluster based and workspace based workflows depends on your infrastructure requirements, security policies, and team structure.
## 1. CI/CD by cluster
**Setup:** Separate clusters for each environment
* **Development cluster** - Active development
* **Staging cluster** (optional) - Pre-production validation
* **Production cluster** - Live pipelines
## 2. CI/CD by workspace
**Setup:** Single cluster with multiple workspaces (see our [workspaces guide](https://docs.mage.ai/guides/developer-ux/workspaces) for more details on configuring workspaces)
* **Development workspace** - Active development
* **Staging workspace** (optional) - Pre-production validation
* **Production workspace** - Live pipelines
## Git Branch Strategy & Environment Mapping
### Recommended Branch Structure
```python theme={"system"}
main/master โ Production environment
staging โ Staging environment
development โ Development environment
feature/* โ Development environment (feature branches)
```
### Workflow Best Practices
**Development:**
* Create feature branches from `development`: `feature/new-pipeline`
* Develop pipelines in feature branch and push changes to Github using the Git terminal (for more details on pushing code to Github see Mage Proโs [Git terminal](https://docs.mage.ai/guides/data-sync/git-terminal) doc)
**Staging:**
* Merge `development` โ `staging` when ready for testing
* Staging environment deploys from `staging` branch
* Run full validation and integration tests
**Production:**
* Merge `staging` โ `main` after successful testing
* Production environment deploys from `main` branch only
* Use deployment application for controlled production releases (for more on Mage Proโs deployment application see the [version control guide](https://docs.mage.ai/guides/data-sync/version-control-guide))
### Environment-Specific Configurations
**Same Code, Different Targets:**
The same codebase connects to different systems using environment-specific configurations:
**Environment Variables & Secrets:**
* Each cluster/workspace maintains its own connection settings
* `DATABASE_URL` points to dev/staging/prod databases
* `API_KEYS` use environment-appropriate credentials
**Configuration Examples:**
```python theme={"system"}
Development: DATABASE_URL = dev-warehouse.company.com
Staging: DATABASE_URL = staging-warehouse.company.com
Production: DATABASE_URL = prod-warehouse.company.com
```
**In Mage Pro:**
* Configure different data source connections per environment
* Use environment variables in pipeline io\_config.yaml file: `{{ env_var('DATABASE_URL') }}`
* or python interpolation in a block `os.environ.get('DB_USERNAME')`
Using these strategies will allow you to develop the same pipeline code, but select different source and targets for each cluster / workspace environment.
## Conclusion
Both approaches use the same tools and follow identical workflows. The choice depends on your infrastructure and security requirements. Use [Git terminal](https://docs.mage.ai/guides/data-sync/git-terminal) for code management and the deployment application for production deployments in either setup. Use our [version control guide](https://docs.mage.ai/guides/data-sync/version-control-guide) to implement this in your Mage Pro environment.
# Using GitClient in Mage blocks
Source: https://docs.mage.ai/guides/data-sync/git-client
Programmatically perform Git operations (add, commit, push, pull) directly within your Mage pipeline blocks using the GitClient class.
The `GitClient` class provides a simple, Pythonic interface for performing Git operations within your Mage pipeline blocks. This is useful for automating version control workflows, such as:
* Automatically committing and pushing pipeline changes after successful runs
* Syncing code from remote repositories before execution
* Creating automated deployment pipelines
## Prerequisites
Before using `GitClient`, ensure you have configured Git settings in Mage through one of these methods:
1. **Git Sync Settings** (legacy): Configure via Settings โ Workspace โ Sync data, or see the [Git Sync guide](/development/git/configure)
2. **Deployment Settings** (Mage Pro): Configure via the Deploy app at `/apps/deploy`
## Quick Start
### Basic Usage
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
# Initialize the client (uses legacy git sync settings by default)
git = GitClient()
# Add, commit, and push all changes in one operation
result = git.add_commit_push('Auto-commit from Mage pipeline')
if result.success:
print(f"โ Pushed to {result.remote}/{result.branch}")
print(f" Commit: {result.commit_hash}")
else:
print(f"โ Failed: {result.error}")
```
### Using Deployment Settings (Mage Pro)
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
# Use deployment settings instead of legacy git sync
git = GitClient(use_deployment_settings=True)
result = git.add_commit_push('Deploy changes via Mage Pro')
```
## Using GitClient in Pipeline Blocks
### Data Exporter Block Example
The most common use case is pushing changes at the end of a pipeline:
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(*args, **kwargs):
"""
Push pipeline changes to Git after processing data.
"""
# Initialize GitClient
git = GitClient()
# Push all changes
result = git.add_commit_push(
message='Pipeline run completed - auto commit',
files='.', # Stage all files
add_flags=['-A'], # Include deletions
)
if result.success:
print(f"โ Successfully pushed to {result.remote}/{result.branch}")
else:
print(f"โ Push failed: {result.error}")
return {
'git_result': result.to_dict(),
}
```
### Data Loader Block Example
Pull latest changes before processing:
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data(*args, **kwargs):
"""
Pull latest changes from remote before loading data.
"""
git = GitClient()
# Pull latest changes
result = git.pull()
if result.success:
print(f"โ Pulled latest from {result.remote}/{result.branch}")
else:
print(f"โ Pull failed: {result.error}")
# Your data loading logic here
return {'status': 'loaded'}
```
## GitClient Methods
### Core Operations
Stage files for commit.
**Parameters:**
* `files` (str | List\[str]): File path(s) to stage. Use `'.'` for all files.
* `flags` (List\[str], optional): Git add flags (e.g., `['-A']` to include deletions).
**Returns:** `GitClient` (for method chaining)
```python theme={"system"}
git.add('.') # Stage all files
git.add(['file1.py', 'file2.py']) # Stage specific files
git.add('.', flags=['-A']) # Stage all including deletions
```
Commit staged changes.
**Parameters:**
* `message` (str): The commit message.
* `files` (List\[str], optional): Files to add before committing.
**Returns:** `str` - The commit hash
```python theme={"system"}
commit_hash = git.commit('Update pipeline configuration')
```
Push commits to the remote repository.
**Parameters:**
* `remote_name` (str, optional): Remote name. Defaults to configured remote.
* `branch_name` (str, optional): Local branch name. Defaults to current branch.
* `remote_branch_name` (str, optional): Remote branch name if different from local.
**Returns:** `GitOperationResult`
```python theme={"system"}
# Push to default remote/branch
result = git.push()
# Push to specific remote and branch
result = git.push(remote_name='origin', branch_name='main')
# Push local 'develop' to remote 'staging'
result = git.push(
remote_name='origin',
branch_name='develop',
remote_branch_name='staging',
)
```
Pull changes from the remote repository.
**Parameters:**
* `remote_name` (str, optional): Remote name. Defaults to configured remote.
* `branch_name` (str, optional): Branch name. Defaults to current branch.
**Returns:** `GitOperationResult`
```python theme={"system"}
result = git.pull()
result = git.pull(remote_name='origin', branch_name='main')
```
Fetch changes from the remote without merging.
**Returns:** `GitOperationResult`
```python theme={"system"}
result = git.fetch()
```
Clone the remote repository (replaces local content).
**Parameters:**
* `sync_submodules` (bool): Whether to also sync git submodules.
**Returns:** `GitOperationResult`
```python theme={"system"}
result = git.clone(sync_submodules=True)
```
### Convenience Methods
Add, commit, and push changes in one operation. This is the recommended method for most use cases.
**Parameters:**
* `message` (str): The commit message.
* `files` (str | List\[str]): File path(s) to stage. Default: `'.'`
* `add_flags` (List\[str], optional): Git add flags (e.g., `['-A']`).
* `remote_name` (str, optional): Remote name.
* `branch_name` (str, optional): Local branch name.
* `remote_branch_name` (str, optional): Remote branch name if different.
**Returns:** `GitOperationResult`
```python theme={"system"}
# Simple push of all changes
result = git.add_commit_push('Update from pipeline')
# Push specific files
result = git.add_commit_push(
message='Update config',
files=['pipelines/', 'data_loaders/'],
)
# Push including deletions
result = git.add_commit_push(
message='Clean up',
files='.',
add_flags=['-A'],
)
# Push to a different remote branch
result = git.add_commit_push(
message='Deploy to staging',
branch_name='main',
remote_branch_name='staging',
)
```
Sync local repository with remote (fetch + reset --hard). Discards local changes.
**Parameters:**
* `branch` (str, optional): Branch to sync. Defaults to current branch.
**Returns:** `GitOperationResult`
```python theme={"system"}
result = git.sync() # Reset to match remote
```
### Branch Operations
Switch to a different branch.
```python theme={"system"}
result = git.switch_branch('feature-branch')
```
Merge another branch into the current branch.
```python theme={"system"}
result = git.merge_branch('feature-branch', message='Merge feature')
```
Delete a local branch.
```python theme={"system"}
result = git.delete_branch('old-feature')
```
### Status Properties
Get comprehensive git status information.
```python theme={"system"}
status = git.status
print(status)
# {
# 'branch': 'main',
# 'remote': 'origin',
# 'remote_url': 'git@github.com:user/repo.git',
# 'modified_files': ['file1.py', 'file2.py'],
# 'is_dirty': True,
# 'status': '...'
# }
```
Get the current branch name.
```python theme={"system"}
branch = git.current_branch # 'main'
```
Check if there are uncommitted changes.
```python theme={"system"}
if git.is_dirty:
print("There are uncommitted changes")
```
Get list of modified files.
```python theme={"system"}
files = git.modified_files # ['file1.py', 'file2.py']
```
Get list of local branches.
```python theme={"system"}
branches = git.branches # ['main', 'develop', 'feature-x']
```
Get list of configured remotes.
```python theme={"system"}
remotes = git.remotes
# [{'name': 'origin', 'urls': ['git@github.com:user/repo.git']}]
```
Get number of commits not yet pushed to remote.
```python theme={"system"}
count = git.unpushed_commits_count # 3
```
## GitOperationResult
All Git operations return a `GitOperationResult` object with the following attributes:
| Attribute | Type | Description |
| ------------- | ---- | -------------------------------------------- |
| `success` | bool | Whether the operation succeeded |
| `message` | str | Success or status message |
| `branch` | str | The branch name involved |
| `remote` | str | The remote name involved |
| `commit_hash` | str | The commit hash (for commit/push operations) |
| `error` | str | Error message if operation failed |
```python theme={"system"}
result = git.push()
if result.success:
print(f"Pushed to {result.remote}/{result.branch}")
print(f"Commit: {result.commit_hash}")
else:
print(f"Error: {result.error}")
# Convert to dictionary
result_dict = result.to_dict()
```
## Method Chaining
`GitClient` supports method chaining for fluent API usage:
```python theme={"system"}
git = GitClient()
# Chain add and commit, then push
git.add('.').commit('Update files')
result = git.push()
```
## Advanced Examples
### Conditional Push Based on Changes
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
@data_exporter
def export_data(data, *args, **kwargs):
git = GitClient()
# Only push if there are actual changes
if git.is_dirty:
print(f"Found {len(git.modified_files)} modified files")
result = git.add_commit_push('Auto-commit changes')
return {'pushed': result.success, 'commit': result.commit_hash}
else:
print("No changes to commit")
return {'pushed': False, 'message': 'No changes'}
```
### Push to Multiple Branches
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
@data_exporter
def deploy_to_environments(data, *args, **kwargs):
git = GitClient()
# Push to staging
staging_result = git.add_commit_push(
message='Deploy to staging',
remote_branch_name='staging',
)
# Push to production (same commit)
if staging_result.success:
prod_result = git.push(remote_branch_name='production')
return {
'staging': staging_result.to_dict(),
'production': prod_result.to_dict(),
}
return {'error': staging_result.error}
```
### Pre-run Sync
```python theme={"system"}
from mage_ai.deployments.git_client import GitClient
@data_loader
def load_with_sync(*args, **kwargs):
git = GitClient()
# Fetch latest and check for updates
git.fetch()
if git.unpushed_commits_count < 0: # We're behind
print("Local is behind remote, syncing...")
git.sync()
# Now load your data
return load_data_from_synced_repo()
```
## Using the Built-in Git Push Template
Mage provides a built-in data exporter template for Git push operations. To add it to your pipeline:
1. Click **Blocks** in the top navigation
2. Select **Exporter**
3. Navigate to **Version control** โ **Git push**
The template provides a ready-to-use block with configurable options:
```python theme={"system"}
# Parameters available in the template:
commit_message: str # Commit message
files: str | List[str] # Files to stage (default: '.')
include_deletions: bool # Include deleted files (default: True)
remote_name: str # Remote name (optional)
branch_name: str # Branch name (optional)
use_deployment_settings: bool # Use Mage Pro settings (default: False)
```
## Setting Up Automated Git Push Jobs
You can automate Git push operations by creating a pipeline with a Git push block and configuring a trigger to run it on a schedule or in response to events.
### Step 1: Create a Git Push Pipeline
1. Create a new pipeline (e.g., `auto_git_push`)
2. Add a **Data Exporter** block using the Git push template:
* Click **Blocks** โ **Exporter** โ **Version control** โ **Git push**
3. Configure the block with your desired settings:
```python theme={"system"}
from typing import Dict, List, Optional, Union
from mage_ai.deployments.git_client import GitClient
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_to_git(*args, **kwargs) -> Dict:
"""
Push changes to a git remote repository.
Uses the git settings configured via git settings (/settings/workspace/sync-data)
or deployment settings (/apps/deploy).
Docs: https://docs.mage.ai/guides/data-sync/git-client
"""
# Get configuration from kwargs
commit_message: str = kwargs.get(
'commit_message',
'Auto-commit from Mage pipeline'
)
files: Union[str, List[str]] = kwargs.get('files', '.')
include_deletions: bool = kwargs.get('include_deletions', True)
remote_name: Optional[str] = kwargs.get('remote_name')
branch_name: Optional[str] = kwargs.get('branch_name')
use_deployment_settings: bool = kwargs.get('use_deployment_settings', False)
# Initialize git client
git = GitClient(use_deployment_settings=use_deployment_settings)
# Set add flags
add_flags = ['-A'] if include_deletions else None
# Add, commit, and push
result = git.add_commit_push(
message=commit_message,
files=files,
add_flags=add_flags,
remote_name=remote_name,
branch_name=branch_name,
)
# Print result
if result.success:
print(f"โ Successfully pushed to {result.remote}/{result.branch}")
if result.commit_hash:
print(f" Commit: {result.commit_hash}")
else:
print(f"โ Push failed: {result.error}")
# Return the result and pass through the input data
return {
'result': result.to_dict(),
}
```
### Step 2: Create a Trigger
Navigate to your pipeline's **Triggers** tab and create a new trigger:
For periodic backups (e.g., daily at midnight):
1. Click **+ New trigger**
2. Select **Schedule** as the trigger type
3. Configure the schedule:
* **Name**: `daily_git_backup`
* **Frequency**: Daily (or use Custom with cron: `0 0 * * *`)
* **Start date**: Set your desired start date
4. Click **Save changes**
**Example cron patterns:**
| Pattern | Description |
| ---------------- | ------------------------- |
| `0 * * * *` | Every hour |
| `0 0 * * *` | Daily at midnight |
| `0 9,18 * * 1-5` | Weekdays at 9 AM and 6 PM |
| `*/30 * * * *` | Every 30 minutes |
For on-demand pushes from external systems:
1. Click **+ New trigger**
2. Select **API** as the trigger type
3. Configure:
* **Name**: `git_push_api`
4. Click **Save changes**
5. Copy the API endpoint URL
**Trigger the pipeline via API:**
```bash theme={"system"}
curl -X POST "https://your-mage-instance.com/api/pipeline_schedules/{trigger_id}/pipeline_runs/{token}" \
-H "Content-Type: application/json" \
-d '{
"pipeline_run": {
"variables": {
"commit_message": "Triggered via API"
}
}
}'
```
For more details on trigger configuration, see the [Pipeline Triggers documentation](/orchestration/triggers/triggering-pipelines).
## Troubleshooting
If you encounter authentication errors:
1. Verify your Git settings are configured correctly
2. For SSH: Ensure your SSH keys are properly set up
3. For HTTPS: Verify your access token has the correct permissions
4. Check that the remote URL is accessible from your Mage instance
If you see "Permission denied" errors:
1. Check that your user has write access to the repository
2. Verify the branch is not protected (or you have permission to push to protected branches)
3. Ensure your OAuth token hasn't expired
If `add_commit_push` reports no changes:
1. Check `git.status` to see the current state
2. Verify files are not in `.gitignore`
3. Use `git.modified_files` to see what files Git detects as changed
## Related Resources
* [Git Sync Configuration](/development/git/configure) - Configuring Git sync settings
* [Mage Pro Deployment Settings](/guides/data-sync/version-control-guide#1-deployments-app) - Configure deployment settings for `use_deployment_settings=True`
# Git integration for code version control
Source: https://docs.mage.ai/guides/data-sync/git-integration
Mage's Git integration provides a way for you to push changes to a remote repositoryโ for example collaborating on a Mage project with your teammates.
"Git integration" in Mage is a way to collaborate with teammates using Git. It provides a way to push changes to a branch in an external repository. If you're using GitHub, we recommend Mage's direct [GitHub integration](/production/data-sync/github) rather than this method.
First, you'll need to configure Gitโ see this [guide](/development/git/configure). Once your git settings are configured, you can perform various Git actions in Mage. To start, click the button at the top right with the name of the current branch.
Once the modal is open, you can start performing actions. Currently, the Git integration supports the following actions:
Clone repository will allow you to clone the specified git repository to your local machine.
*This operation will overwrite all local changes you have in the local git directory.*
Create new branch will create a new branch in your local git directory. Specify the name of the branch and click `Create`. You will
automatically be switched to the new branch.
To switch your local branch, you can click the dropdown menu with the current active branch and select the branch you would like to switch to.
First, select the files you would like to commit. You can select all files by clicking the `Include all changes` button.
Then, enter a descriptive commit message and click `Commit`. This will commit your changes to your local git directory.
Once your changes are committed, you can push your commit to the remote repository by clicking `Push`.
Pull your changes from the remote branch to your local branch. This operation is equivalent to `git pull mage-repo `.
Hard reset your local branch with the remote branch. This operation is equivalent to `git reset --hard mage-repo/`.
*This operation will overwrite your local changes.*
For any other operations you may need, you can perform them through the terminal. You can open the terminal by clicking the terminal toggle in the top right of the modal.
# One-way code sync using Git
Source: https://docs.mage.ai/guides/data-sync/git-sync
Enabling one-way Git sync in Mage ensures data will be synced in one direction with a specified git repository.
Git sync is a Git feature that allows you to tie your local project to a remote repo. When Git sync is enabled, you will only be able to pull in changes from the remote repository. Git sync is particularly useful for deploymentโ for example if you'd like a hosted Mage instance to stay up-to-date with a repo.
Running a Git sync will ***overwrite all local changes*** and replace them with the code in the specified remote branch. Git syncs can be run either from the Git settings page, or before every trigger run if you selected the `Sync before each trigger run` field.
To enable Git Sync, first [configure Git in Mage](/development/git/configure), then navigate to the settings page and tick the `Git Sync` box.
### Syncing git submodules
Repositories synced through Mage will only update 1 level of submodules. If your submodule contains other submodules, you will need to manually update them or reformat your submodules.
If your repo contains git submodules, you can enable syncing for them by ticking the `Include submodules` box. Mage will
use the same credentials that are provided for the main repository to update the submodules.
## Git sync settings as environment variables
For Mage versions `>= 0.9.51`
Mage will prioritize git settings from environment variables when they are set. If you want to be able
to change git settings from the UI, you will need to set the `GIT_OVERWRITE_WITH_PROJECT_SETTINGS` environment
variable to `1`. Otherwise, edits to the git settings from the UI will not be applied.
You can choose to pass in Git settings as environment variables. Mage will read from these environment
variables to configure your Git repo.
| Setting | Variable | Description | Required |
| ------------------------------------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| Git settings | `GIT_REPO_LINK` | Corresponds to `Remote repo url` | โ |
| Git settings | `GIT_REPO_PATH` | Corresponds to `Local directory path` | โ |
| Git settings | `GIT_USERNAME` | Corresponds to `Username` | โ |
| Git settings | `GIT_EMAIL` | Corresponds to `Email` | โ |
| Git settings | `GIT_AUTH_TYPE` | The method of authentication. See enum values [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/sync/__init__.py#L13-L16) | โ |
| Git sync settings | `GIT_BRANCH` | Corresponds to `Branch name` | ๐ซ |
| Git sync settings | `GIT_SYNC_ON_PIPELINE_RUN` | Corresponds to `Sync before each trigger run`, options | ๐ซ |
| Git sync settings | `GIT_SYNC_ON_START` | Corresponds to `Sync on server start up`, options | ๐ซ |
| Git sync settings | `GIT_SYNC_ON_EXECUTOR_START` | If set to "1", Mage will attempt to run a git sync every time an executor is started, options | ๐ซ |
| Git sync settings | `GIT_SYNC_SUBMODULES` | Corresponds to `Include submodules`, options | ๐ซ |
| Authentication environment variables | `GIT_SSH_PUBLIC_KEY` | SSH public key encoded in base64, used for ssh auth type | ๐ซ |
| Authentication environment variables | `GIT_SSH_PRIVATE_KEY` | SSH private key encoded in base64, used for ssh auth type | ๐ซ |
| Authentication environment variables | `GIT_ACCESS_TOKEN` | Git access token, used for https auth type | ๐ซ |
| Global git settings | `GIT_ENABLE_GIT_INTEGRATION` | Force git integration module to show in the UI. Primarily useful when the version control app does not work for your repository. | ๐ซ |
| Global git settings | `GIT_OVERWRITE_WITH_PROJECT_SETTINGS` | Allow users to overwrite git settings set by environment variables through the Git Settings page in the UI. | ๐ซ |
# Git terminal with built-in authentication and shortcuts
Source: https://docs.mage.ai/guides/data-sync/git-terminal
Mage Pro supports Git-based version control, making it easy to collaborate, track changes, and manage code across environments.
### Introduction
The Git Terminal is a browser based terminal with built in Git authentication,
directly integrated into Mage Pro. It provides standard Git functionality with
real-time collaboration features, eliminating the need to switch between tools for
version control operations. Multiple team members can work simultaneously in
the same terminal session, sharing commands and coordinating development work.
### Location
From the home page, hover over the left popout navigation menu and click
**Terminal**. You will then be taken to a terminal screen directly integrated
with Git.
### Integration
The Git Terminal connects with your existing development workflow. Key integration points include:
* GitHub, GitLab, and other Git providers
* Maintains authentication across sessions
* Works within existing Mage Pro workflow
### Use Cases
Use the Git Terminal in your development environment to push changes to GitHub for source version control after making modifications to pipelines, blocks, or configurations in the Mage UI. Essential use cases include:
* Before deploying changes to save work to version control
* For regular commits to maintain code history and backup
### Authentication
Before using the Git Terminal, authenticate with your Git provider through the Deploy App. This one-time setup enables Git operations within Mage Pro.
**Quick Setup:**
1. Navigate to **Deployments** โ **Connect repository**
2. Complete OAuth authentication for GitHub, Bitbucket, GitLab, or Azure DevOps
3. Configure your repository connection
*For detailed authentication steps, see the [Version control guide](https://docs.mage.ai/guides/data-sync/version-control-guide).*
Once authenticated, the Git Terminal automatically uses your credentials for all Git operations without requiring additional setup.
For OAuth connections, each terminal session receives a session-scoped Git credential helper for the connected repository's Git provider. HTTPS Git commands such as `git fetch`, `git pull`, and `git push` can use the OAuth token already connected in Mage, and provider CLI commands can read the matching session token environment variable without running a separate CLI login such as `gh auth login`.
If the terminal header shows **Authenticate Git**, Mage has not detected usable Git authentication for the repository, either from the terminal environment such as an environment token, GitHub CLI login, or SSH key, or from a connected Git OAuth token. Select **Authenticate Git** to open the repository connection page for the current project, complete the Deployments Git OAuth flow, and then return to the terminal. If you authenticate Git directly in the terminal, Mage hides the action after it detects that Git authentication is available.
The Mage Pro image includes GitHub CLI (`gh`) for optional GitHub authentication
and status checks. For GitLab, Azure DevOps, and Bitbucket, use standard Git
commands with Mage's credential helper or configured SSH/PAT authentication;
provider-specific CLIs are not required.
### Usage
The Git Terminal supports all standard Git operations you're familiar with. Run standard Git commands such as:
```python theme={"system"}
git status # Check current changes
git add . # Stage all changes
git commit -m "message" # Commit with description
git push origin main # Push to remote repository
```
### Terminal controls
Use terminal tabs when you need more than one shell context. Each tab runs its
own terminal session, keeps its own working directory, and can be reconnected or
closed without interrupting the other tabs.
The prompt bar shows the current folder, Git branch, and working tree change
counts for the active tab. Autocomplete appears while typing and can suggest
shell commands, Git subcommands, files, folders, executable commands on `PATH`,
and recent command history.
Terminal prompt timestamps and shell commands use UTC by default. If your
workspace has **Display local timezone** enabled, new terminal sessions use your
browser's local IANA timezone for the prompt timestamp and the session `TZ`
environment variable.
### Conclusion
The Git Terminal streamlines your development workflow by bringing version control directly into the Mage Pro environment.
With built-in authentication, real-time collaboration, and standard Git functionality, teams can maintain professional development
practices without the complexity of managing separate tools. This integrated approach enables faster development cycles while ensuring
code changes are properly tracked and shared across your team.
# Version control application with Git
Source: https://docs.mage.ai/guides/data-sync/version-control
Authenticate with a Git provider then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.
On the homepage in the left side navigation, click the link labeled Version control.
The URL could look something like this: [http://localhost:6789/version-control](http://localhost:6789/version-control).
# Mage Pro
Unlock seamless version control and effortless deployments with Mage Pro!
Check out this [guide](/guides/data-sync/version-control-guide) to experience a more intuitive, user-friendly way to manage and deploy your code.
# Open source
## Demo
## Setup
### Set Git directory
The Git directory is the local folder that will be linked to your remote repository.
### Authentication
You will be able to authenticate with GitHub by default, but if you need to authenticate with
GitHub Enterprise (GHE) or other providers, you will need to follow the steps below. You can
authenticate with multiple git providers if you need to. Mage will attempt to use the oauth provider
based on the URL of the remote.
Mage authenticates with a default GitHub OAuth application that may not be able to access
your repository depending on the repository or organization settings. If you encounter issues
accessing your repository after authenticating with GitHub, you may need to create your own
OAuth application that has access to the repositories you want to work with. You can follow the
steps in the `Authenticate with GitHub or GHE` section below to connect Mage with your OAuth
application.
When creating the OAuth app for GHE, BitBucket, GitLab, you will need to set the authorized
callback URL to `http(s):///oauth`.
In order for Mage to authenticate with your GitHub host, you will need to add Mage as an
authorized oauth application. You can follow the steps [here](https://docs.github.com/en/enterprise-cloud@latest/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app)
to set it up.
Make sure you copy the client id and client secret after the app is created.
You will need to set the following environment variables in your Mage environment:
1. GHE\_HOSTNAME: the hostname of your GitHub instance. If youโre using non-enterprise GitHub or GitHub Enterprise Cloud, you should set this to `github.com`
2. GHE\_CLIENT\_ID: the client id you retrieved in the previous step
3. GHE\_CLIENT\_SECRET: the client secret you retrieved in the previous step
You should now be able to authenticate with GitHub and follow the rest of the steps in this document.
In order for Mage to authenticate with your BitBucket, you will need to add Mage as an
OAuth consumer. You can follow the steps [here](https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/)
to set it up.
Make sure you copy the key and secret after the consumer is created.
You will need to set the following environment variables in your Mage environment:
1. BITBUCKET\_OAUTH\_KEY: the key you retrieved in the previous step
2. BITBUCKET\_OAUTH\_SECRET: the secret you retrieved in the previous step
You should now be able to authenticate with Bitbucket and follow the rest of the steps in this document.
Mage will request `read_user`, `write_repository`, and `api` scopes from GitLab in order
to perform the necessary git actions.
In order for Mage to authenticate with your GitLab, you will need to add a Mage oauth application.
You can follow the steps [here](https://docs.gitlab.com/ee/integration/oauth_provider.html)
to set it up.
Make sure you copy the Application ID and Secret after the oauth app is created.
You will need to set the following environment variables in your Mage environment:
1. GITLAB\_CLIENT\_ID: the application ID you retrieved in the previous step
2. GITLAB\_CLIENT\_SECRET: the secret you retrieved in the previous step
3. GITLAB\_HOST (optional): hostname for your GitLab instance. Defaults to `https://gitlab.com`
You should now be able to authenticate with GitLab and follow the rest of the steps in this document.
First, you will need to enable the [Active Directory](https://docs.mage.ai/production/authentication/microsoft) integration in order for Mage to authenticate your user.
Once the Active Directory integration is enabled, you simply need to add your DevOps organization as an environment variable.
```bash theme={"system"}
export AZURE_DEVOPS_ORGANIZATION=`
```
You should now be able to authenticate with Azure DevOps and follow the rest of the steps in this document.
Click the Authenticate button next to your Git Provider to start the authentication process.
After authorizing Mage, youโll be redirected back to the initial page and an access token will
be in the URL as a parameter. This access token is saved in your applicationโs database;
it never leaves your applicationโs database.
### Add remote
Enter a name for your new remote. For the remote URL, you must use the `https` version of the
GitHub repository (as opposed to the `SSH` version).
For example, the remote URL for Mage is `https://github.com/mage-ai/mage-ai.git`.
Note that a remote added via HTTPS/SSH authentication *will not* workโ only a remote in the format above is accepted.
Once you fill in both input text fields, click the button labeled Create new remote.
### Actions
#### Clone remote repository
Cloning will copy all the files and folders from the remote repository and put them into the Git directory you specified above.
Cloning wonโt automatically create a folder that is named after the remote repository.
For example, if you have a file named `magic.py` in a remote repository named `project_romeo`
then that file will be cloned here `/magic.py`
as opposed to `/project_romeo/magic.py`.
If you want to clone the content of a remote repository into a folder named after the remote repository, then change the Git init directory above to
`/[remote repository name]`
1. Under the Actions dropdown, select the option labeled Clone.
2. Under the Remote dropdown, select the name of the remote you want to clone from (e.g. origin).
#### Pull remote branch
1. Under the Actions dropdown, select the option labeled Pull.
2. Under the Remote dropdown, select the name of the remote you want to pull from (e.g. origin).
3. Under the Branch dropdown, select the name of the branch you want to pull from the remote you chose from the previous step.
***
## Branches
### Switch branch
1. Under the dropdown labeled Current branch, select the name of the branch you want to change to.
2. After selecting a branch, the current branch will be updated to what you selected.
### Create branch
1. Type in the name of the new branch in the text input field.
2. Click the button labeled + Create new branch.
### Merge branch
1. Under the Base branch dropdown, select the name of the branch you want to merge into your current branch.
2. Under the Action dropdown, select the option labeled Merge.
3. Type a commit message for the merge action.
4. Click the button labeled Merge.
### Rebase branch
1. Under the Base branch dropdown, select the name of the branch you want to rebase into your current branch.
2. Under the Action dropdown, select the option labeled Rebase.
3. Type a commit message for the rebase action.
4. Click the button labeled Rebase.
### Delete branch
1. Under the Base branch dropdown, select the name of the branch you want to delete.
2. Under the Action dropdown, select the option labeled Delete.
3. Click the button labeled Delete.
4. Confirm the popup dialog box.
***
## Files
### Add files
1. On the left side under Not staged, check the boxes next to the files you want to add.
2. Once youโre done selecting the files, click the button labeled Add files.
3. The selected files will be staged.
### Checkout files
1. On the left side under Not staged, check the boxes next to the files you want to checkout.
2. Once youโre done selecting the files, click the button labeled Checkout files.
3. Confirm the popup dialog box.
4. All changes to the selected files will be undone.
### Reset files
1. On the right side under Staged files, check the boxes next to the files you want to reset.
2. Once youโre done selecting the files, click the button labeled Reset files.
3. The selected files will be unstaged.
### Commit
1. Enter a commit message.
2. Click the button labeled Commit N files with message.
***
## Push
### Push local changes
1. Under the Remote dropdown, select the name of the remote you want to push to (e.g. origin).
2. Under the Branch dropdown, select the name of the branch you want to push.
3. Click the button labeled Push \[remote] \[branch].
### Create pull request
1. Under the Repository dropdown, select the name of the repository you want to create a pull request for.
2. Under the Base branch dropdown, select the name of the branch you want your pull request to compare against (e.g. main).
3. Under the Compare branch dropdown, select the name of the branch you want your pull request to be for (e.g. feature branch).
4. Type the title of the pull request in the text input field labeled Title.
5. Type the description of the pull request in the text input field labeled Description.
# Mage Pro Version Control
Source: https://docs.mage.ai/guides/data-sync/version-control-guide
Connect Mage Pro to GitHub, Bitbucket, GitLab, or Azure DevOps using OAuth, HTTPS+PAT, or SSH. Manage data pipelines with Git and deploy commits to your cluster.
Mage Pro's version control workflow has three parts:
1. **Authentication** โ [Connect your Git host](#1-authentication) to Mage (OAuth with GitHub, Bitbucket, GitLab, or Azure DevOps; HTTPS with a personal access token; or SSH). You do this in the **Deployments** app, but it also enables the [Version control terminal](#3-version-control-terminal).
2. **Deployments** โ [Deploy](#2-deployments-app) commits to your Mage Pro cluster or workspace from the visual Deployments app.
3. **Version control terminal** โ [Run Git in the browser](#3-version-control-terminal) (push, pull, branches) once you are connected.
## 1. Authentication
**Authentication is shared** by the [Deployments app](#2-deployments-app) and the [version control terminal](#3-version-control-terminal). Configure it in **Deployments** โ open **Connect repository** or deployment **Setup** โ **Connection** โ **Authentication type**. Until this is set up, you cannot use Git-based deployment or the terminal.
Mage Pro supports three **authentication types**:
| Type | When to use |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **OAuth** | Default path: sign in with GitHub, Bitbucket, GitLab, or Azure DevOps and pick org/workspace, repo, and branch from the UI. |
| **HTTPS + PAT** | Your organization requires a **personal access token** instead of OAuth, or you use a provider / URL where OAuth is not an option. |
| **SSH** | You use **deploy keys** or user SSH keys and `git@โฆ` (or `ssh://`) remotes. |
If you change authentication type, **save authentication settings** before continuing with the rest of the connection configuration. Switching types clears the credentials from the other method in your deployment settings.
### Open the connection UI
From the Home page, select **Deployments** in the left-hand navigation menu to access the Deployments interface.
Click the **Setup connection** button to begin syncing with your code repository.
Mage Pro provides two main applications for version control and deployment:
1. **Deployments App**: Deploy code from Git commits to your Mage Pro cluster
2. **Version Control Terminal**: Push/pull code and manage branches using Git commands
## 1. Deployments App
The Deployments App is your primary tool for deploying code from Git commits to your Mage Pro cluster. It provides a visual interface for managing deployments across different environments.
### 1.1 Authentication
First, authenticate with your Git provider in the Deployments application to enable version control and deployment features.
#### Access Deployments
From the Home page, select **Deployments** in the left-hand navigation menu to access the Deployments interface.
Click the Setup connection button to begin syncing with your code repository.
### Project settings
Fill in the **Project settings** fields at the top of the Connect repository page, then click **Save settings** before configuring authentication below.
* **Configuration name** โ a label for this deployment configuration.
* **Project directory** โ the user code path for your project. The default in Mage Pro is `/home/src/default_repo`. If you changed the `USER_CODE_PATH` environment variable in the Mage Cloud Management Portal, set this field to match.
* **Target directory (optional)** โ the directory code changes are deployed into. If left empty, the project directory is used. For a nested repository you may want to set this to `/home/src`.
#### Configure Repository Settings
Once you are in the connect repository UI, take the following steps:
**Step 1**: Set your directory as the user code path you intend to use for your project.
* In Mage Pro the default directory will be **`/home/src/default_repo`**.
* If you changed the **`USER_CODE_PATH`** environment variable in the Mage Cloud Management Portal, your directory will need to reflect that change.
**Step 2 (optional)**: Set your target directory for your deployment if it will be different from your top level directory.
* If you have a nested GitHub project, you may want to set the target directory to `/home/src`
After clicking **Save settings**, proceed to the **Connection** section below to set your authentication type and connect your provider.
### HTTPS and personal access token (PAT)
1. Under **Connection**, set **Authentication type** to **HTTPS+PAT**.
2. Enter the **remote repository URL** in HTTPS form (for example `https://github.com/your-org/your-repo.git`). The UI derives owner and repository name from this URL.
3. Enter a **username** and **email** for Git commit authorship. These are also used as HTTP credentials: the username is your Git host account name and the PAT is sent in place of a password.
4. In your Git host, create a **personal access token** (or app password) scoped to at least **clone, fetch, and push**. Scope names differ by provider โ for example, GitHub requires the `repo` scope; GitLab requires `read_repository` and `write_repository`; Bitbucket uses the `repository:write` permission.
5. Paste the token into **Personal access token** and click **Save authentication settings**. The token is stored as a **Mage secret** named `mage_git_access_token_{user_id}` (where `{user_id}` is your Mage user ID).
HTTPS+PAT uses your stored credentials for Git in the [version control terminal](#3-version-control-terminal) the same way OAuth does.
### SSH key authentication
1. Under **Connection**, set **Authentication type** to **SSH**.
2. Enter the **remote repository URL** in SSH form (for example `git@github.com:your-org/your-repo.git` or a supported `ssh://` URL from your provider).
3. Enter a **username** and **email** for Git **commit** identity (as shown in `git log`), matching what you use for the repository.
4. Add **SSH public and private key** values. The form expects the key material in **base64** (as labeled in the UI). In your Git host, add the public key as a **deploy key** (repo-scoped) or a user key, with permission to read/write the repo as needed.
5. Click **Save authentication settings**.
With **SSH** authentication, Mage cannot use the providerโs **Git hosting API** for some deployment flows the same way as with OAuth (for example, **listing commits in the Deployments** UI). For **`git push`**, **`git pull`**, fetches, and other Git wire operations, use the **[version control terminal](#access-the-version-control-terminal)**. A warning is shown in the app when this applies.
### GitHub (OAuth)
Mage Pro supports integration with GitHub repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the GitHub **Connect** OAuth flow below.
**Connect to GitHub using OAuth tokens:**
**Step 1**: Click the **Connect** button to the right of the GitHub connection.
* This will take you to the GitHub sign-in authentication.
**Step 2**: Sign in with your GitHub login credentials.
* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to GitHub.
* Click **Save authentication settings**.
**Step 3**: Select your GitHub organization from the list.
* If your organization isn't appearing, enter the name in the **Organization name** field and click **Set organization**.
To deploy from a **private repository**, click the **Install Mage Pro app on GitHub** button that appears below the Organization section. Once installed, you can verify or update the installation at any time using the **Verify installation** or **Configure installation settings** buttons.
**Step 4**: Select the repository from the dropdown.
* You should see all your GitHub repositories if you authenticated correctly.
**Step 5**: Select the production branch from the dropdown.
* The production branch is the primary branch your production code will mirror.
### Bitbucket (OAuth)
Mage Pro supports integration with Bitbucket repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the Bitbucket **Connect** OAuth flow below.
**Connect to Bitbucket using OAuth tokens:**
#### GitHub Authentication
**Step 1**: Click the "Connect" button to the right of the GitHub connection
* This will take you to the GitHub sign-in authentication
**Step 2**: Sign in with your GitHub login credentials
* After authenticating you will return to the Mage-Pro deployment app
* You should now see that you are connected to GitHub
**Step 3**: Select your GitHub organization from the dropdown list.
* If there are no organizations listed, add an organization to the text field below and click the Set organization button
**Step 4**: Select the repo you want to commit code to by clicking the repository drop down box
* You should see all your GitHub repos here if you authenticated correctly
**Step 5**: Select your branch (usually main or master) to commit your code to by clicking the dropdown box and selecting the branch name
**Step 6**: Click the "Deployments" button to see a list of commits from your Git hosting vendor, and deploy from the most recent or any specific commit.
If you need to connect deployments to a private GitHub repo, provide an organization then click the Install Mage Pro app on GitHub button.
#### BitBucket Authentication
Mage Pro supports integration with BitBucket repositories for version control, collaboration, and deployments.
**Connect to BitBucket using OAuth tokens:**
**Step 1**: Set your directory as the user code path you intend to use for your project.
* In Mage Pro the default directory will be `/home/src/default_repo`.
* If you changed the `USER_CODE_PATH` environment variable in the Mage Cloud Management Portal, your directory will need to reflect that change.
**Step 1**: Click the **Connect** button to the right of the Bitbucket connection.
* This will take you to the Bitbucket sign-in authentication.
**Step 2**: Sign in with your Bitbucket login credentials.
* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to Bitbucket.
* Click **Save authentication settings**.
**Step 2**: Click the "Connect" button to the right of the BitBucket connection
* This will take you to the BitBucket sign-in authentication
**Step 3**: Sign in with your BitBucket login credentials
* After authenticating you will return to the Mage-Pro deployment app
* You should now see that you are connected to BitBucket
**Step 3**: Select your Bitbucket workspace from the dropdown list.
* If there are no workspaces listed, enter a workspace slug in the text field below and click **Set workspace**.
**Step 4**: Select the repository from the dropdown menu.
* You should see a list of all the repositories in that workspace.
**Step 5**: Select the production branch from the dropdown list.
* The production branch is the primary branch your production code will mirror.
**Step 4**: Select your BitBucket workspace from the dropdown list.
* If there are no organizations listed, add an organization to the text field below and click the "Set organization" button
**Step 5**: Select the repository from the dropdown menu.
* You should see a list of all the repositories that were created in the workspace.
**Step 6**: Finally, select the production branch from the dropdown list
* The production branch is the primary branch that your production code will mirror.
### GitLab (OAuth)
Mage Pro supports integration with GitLab repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the **Connect** GitLab OAuth flow below.
**Connect to GitLab using OAuth tokens:**
**Step 1**: Click the **Connect** button to the right of the GitLab connection.
* This will take you to the GitLab sign-in authentication.
**Step 2**: Sign in with your GitLab login credentials.
* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to GitLab.
* Click **Save authentication settings**.
**Step 3**: Select your GitLab group from the dropdown list.
* If there are no groups listed, enter a group name in the text field below and click **Set group**.
**Step 4**: Select the repository from the dropdown menu.
* You should see a list of all the repositories in that group.
**Step 5**: Select the production branch from the dropdown list.
* The production branch is the primary branch your production code will mirror.
### Azure DevOps (OAuth)
Mage Pro supports integration with Azure DevOps repositories for version control, collaboration, and deployments. For **HTTPS+PAT** or **SSH**, use [PAT](#https-and-personal-access-token-pat) or [SSH key](#ssh-key-authentication) instead of the Azure DevOps **Connect** OAuth flow below.
**Connect to Azure DevOps using OAuth tokens:**
**Step 1**: Click the **Connect** button to the right of the Azure DevOps connection.
* This will take you to the Azure DevOps sign-in authentication.
**Step 2**: Sign in with your Azure DevOps credentials.
* After authenticating you will return to the Mage Pro deployment app.
* You should now see that you are connected to Azure DevOps.
* Click **Save authentication settings**.
**Step 3**: Select your Azure DevOps **organization** from the dropdown, or type the organization name directly in the text field and click **Save**.
* If your organization isn't appearing in the dropdown, enter the name in the **Organization name** field and click **Set organization**.
**Step 4**: Select your **Azure DevOps project** from the dropdown.
* This is the project that contains the repository you want to deploy changes from.
**Step 5**: Select the **repository** from the dropdown.
* You should see a list of all repositories in that project.
**Step 6**: Select the **production branch** from the dropdown.
* This is the primary branch your production code will mirror.
* If your branch isn't listed, type the branch name into the text field and click **Set branch**.
## 2. Deployments App
The Deployments app is the control surface for [deploying](#merge-pull-request-and-deploy) merged work from Git to your Mage Pro cluster. It assumes you have already completed [authentication](#1-authentication).
### Merge pull request and deploy
Once you push code to your Git branch, open your Git provider and merge the pull request. When the merge is done, the deployment is available in the Deployments app.
Mage then deploys a chosen commit to your environment:
1. In Mage, go to the **Deployments** view and open the deployments list (highlighted in green at the top when updates are available).
2. Click **Deploy** to ship the version you just merged. Use **Rollback** to deploy an earlier commit if you need to revert.
#### GitLab Authentication
Mage Pro supports integration with GitLab repositories for version control, collaboration, and deployments.
**Connect to GitLab using OAuth tokens:**
**Step 1**: Set your directory as the user code path you intend to use for your project.
* In Mage Pro the default directory will be `/home/src/default_repo`.
* If you changed the `USER_CODE_PATH` environment variable in the Mage Cloud Management Portal, your directory will need to reflect that change.
**Step 2**: Click the "Connect" button to the right of the GitLab connection
* This will take you to the GitLab sign-in authentication
**Step 3**: Sign in with your GitLab login credentials
* After authenticating you will return to the Mage-Pro deployment app
* You should now see that you are connected to GitLab
**Step 4**: Select your GitLab group from the dropdown list.
* If there are no groups listed, add a group to the text field below and click the "Set group" button
**Step 5**: Select the repository from the dropdown menu.
* You should see a list of all the repositories that were created in the group.
**Step 6**: Finally, select the production branch from the dropdown list
* The production branch is the primary branch that your production code will mirror.
Currently the deployments application is only available for GitHub, GitLab, and Bitbucket users.
### 1.2 Deploy Code
Deploy code from Git commits to your Mage Pro cluster or workspace using the Deployments application.
#### Merge Pull Request
Once you push code to your Git branch, head to your Git provider and merge the pull request. Once the pull request is merged your deployment will be available in the deployments application interface.
#### Deploy Code
Mage simplifies the deployment of code to production pipelines. Follow these steps to deploy your pipeline's CI/CD Git branches:
1. Once you have merged the pull request in your Git provider, navigate back to the deployments tab and click deployments highlighted in green in the top
2. Click the Deploy button to deploy the current version you just merged
* To revert to a previous version, simply click the "Rollback" button to deploy the desired version.
## 3. Version control terminal
Once you have completed [Authentication](#1-authentication), you can use standard Git in the browser: status, commit, push, pull, and branch operations against your configured remote.
If you have not set up Git access yet, follow [Authentication](#1-authentication) first: **OAuth**, or **HTTPS+PAT** / **SSH** in the **Deployments** connection settings. All three options enable the version control terminal.
With **OAuth**, the terminal automatically attaches a session-scoped Git credential helper for the connected repository's provider. HTTPS Git commands can use the OAuth token that is already connected in Mage, so you do not need to run a separate provider CLI login in the terminal.
Mage Pro images include GitHub CLI (`gh`) for optional GitHub auth helper and status checks. GitLab, Azure DevOps, and Bitbucket Git operations use the Mage credential helper or configured SSH/PAT authentication and do not require provider-specific CLIs in the runtime image.
### Access the Git terminal
* From the home page, open the left navigation and select **Terminal**.
* When the page loads, type Git commands in the terminal and press **Enter** to run them.
* If the terminal header shows **Authenticate Git**, Mage has not detected usable Git authentication for the repository. Select it to open the repository connection page for the current project, complete Git OAuth in Deployments, and then return to the terminal. If you authenticate Git directly in the terminal, Mage hides the action after it detects that Git authentication is available.
* Use tabs to keep multiple shell sessions open. Each tab keeps its own terminal session and working directory, so closing one tab does not interrupt the other tabs.
* The terminal status row shows the active folder, Git branch, and working tree change counts. Autocomplete appears while typing and suggests shell commands, Git subcommands, files, folders, executable commands, and recent command history from the active working directory.
* Terminal prompt timestamps and shell commands use UTC by default. If **Display local timezone** is enabled for the workspace, new terminal sessions use your browser's local IANA timezone for the prompt timestamp and the session `TZ` environment variable.
### Essential Git commands
* `git checkout -b `: create and switch to a new feature branch.
* `git status`: inspect modified, staged, and untracked files.
* `git add .`: stage changes in the current directory.
* `git commit -m ""`: commit the staged changes.
* `git pull`: fetch and merge remote changes into the current branch.
* `git push origin `: push the current branch to the remote repository.
## Best Practices for Environment Management
### Development Environment
* **Primary Tool - Version Control Terminal**:
* Create feature branches, develop code, commit changes, push to Git
* Switch between branches, merge changes
* **Secondary Tool - Deployments App**:
* Deploy to dev environment for testing and validation
* Rollback changes or switch to another feature branch (can also be done via terminal)
* **Workflow**: Develop in Mage Pro โ Commit to feature branch โ Deploy to dev environment for testing
* **Workflow**: Develop locally โ Commit to feature branch โ Deploy to dev environment for testing
### Staging Environment
* **Primary Tool - Deployments App**:
* Pull changes from Git (staging/main/master branch)
* Deploy to staging environment and run testing
* **Secondary Tool - Version Control Terminal**:
* Make hotfixes and push changes to Git
* Merge feature branches and prepare releases
* **Workflow**: Pull from Git โ Deploy to staging โ Run integration tests โ Make hotfixes if needed
### Production Environment
* **Primary Tool - Deployments App**:
* Deploy commits from main/master branch
* Rollback changes if issues occur
* Monitor deployment status
* **Secondary Tool - Version Control Terminal**:
* Emergency patches and hotfixes only
* **Workflow**: Deploy from main/master โ Monitor โ Rollback if needed
# Add an existing dbt project to Mage
Source: https://docs.mage.ai/guides/dbt/add-existing-dbt
If you have an existing dbt project, you can add it to your Mage project.
For example, if your Mage project is named `demo_project` and your existing dbt
project is named `demo`, add your dbt project in the Mage project under the
following path:
```
demo_project/dbt/demo/
```
You can have multiple dbt projects in a single Mage project under the folder
named `dbt/` at the root location of your Mage project.
For example:
```
demo_project/
| dbt/
| | demo/
| | | ...
| | project_2/
| | | ...
```
***
## Required structure
### Connection profiles
For Mage to properly run dbt commands, you must have a `profiles.yml` file at
the root of each of your dbt projects. For example:
```
demo_project/
| dbt/
| | demo/
| | | profiles.yml
| | project_2/
| | | profiles.yml
```
For more information on how to configure dbt connection profiles, read this
[documentation](/dbt/connection-profiles).
# dbt connection profiles
Source: https://docs.mage.ai/guides/dbt/connection-profiles
Mage will look for a file named `profiles.yml` located in your dbt project root
folder.
For example, if your Mage project is named `demo_project` and your dbt project
name is `demo`, then your `profiles.yml` file for that dbt project should be
located in `demo_project/dbt/demo/profiles.yml`.
If you have a 2nd dbt project (e.g. named `example`), the connection profiles
for that project should be located in `demo_project/dbt/example/profiles.yml`.
For more information on how to configure your connection profiles, read
[dbtโs documentation](https://docs.getdbt.com/docs/get-started/connection-profiles).
Using Mage, you may also [interpolate variables](/guides/dbt/variable-interpolation) in your connection profiles.
Using Mage, you may also [interpolate variables](/dbt/variable-interpolation) in your connection profiles.
***
## How connection profiles are used
When you run a [single dbt model](/guides/dbt/run-single-model) or
[selected dbt models](/guides/dbt/run-selected-model), there is an input field labeled
When you run a [single dbt model](/dbt/run-single-model) or
[selected dbt models](/dbt/run-selected-model), there is an input field labeled
`dbt profile target`. The value in that field will be the connection profile
target that Mage uses when running a dbt model block.
The command Mage executes is `dbt run` with the flag `--target` and the value of
that flag is the value from the input field labeled `dbt profile target`.
# Develop dbt in Mage
Source: https://docs.mage.ai/guides/dbt/developing-dbt-in-mage
Learn how to develop your favorite dbt models in Mage.
dbt integration is currently only supported when using Mage in
[Docker](/getting-started/setup#using-docker) or in [Mage Pro](https://cloud.mage.ai/sign-up).
If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat).
Read the [setup guide](/getting-started/setup) to initialize a new project and
start the Mage tool locally.
For the rest of this tutorial, weโll use the project name `demo_project`.
* Open Mage and go to the terminal page: [http://localhost:6789/terminal](http://localhost:6789/terminal)
* Initiate your dbt project using the `init` command (for this tutorial, weโll
use the dbt project name `demo`):
```bash theme={"system"}
cd demo_project/dbt
dbt init -s demo
touch demo/profiles.yml
```
For more information on creating a dbt project, read their
[documentation](https://docs.getdbt.com/reference/commands/init).
* Go to the Mage dashboard and click the button
**`+ New pipeline`** and select the option labeled `Standard (batch)`.
* Click the `Pipeline settings` icon in the left pane, and change its name
to `dbt demo pipeline`, then click the `Save pipeline settings` button.
* On the left side of the page in the file browser, expand the folder
`demo_project/dbt/demo/`.
* Click the file named `profiles.yml`.
* Paste the following credentials in that file:
```yaml theme={"system"}
demo:
target: dev
outputs:
dev:
dbname: xyleviup
host: queenie.db.elephantsql.com
password: edSrMWH7Mww-lTKpp-jPHX9sYSNLy7LG
port: 5432
schema: dbt_demo
type: postgres
user: xyleviup
```
* Save the `profiles.yml` file by pressing `Command (โ)` + `S`.
* Close the file by pressing the **`X`** button on the right side of the file
name `dbt/demo/profiles.yml`.
* Click the **`+ Data loader`** button, select `Python`, then click `API`.
* In the popup dialog `Data loader block name`, change its name to `load data`,
then click the `Save and add block` button.
* Paste the following code in that block:
```python theme={"system"}
import io
import pandas as pd
import requests
from pandas import DataFrame
@data_loader
def load_data_from_api(**kwargs) -> DataFrame:
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
response = requests.get(url)
df = pd.read_csv(io.StringIO(response.text), sep=',')
df.columns = ['_'.join(col.split(' ')) for col in df.columns]
return df
```
* Under the data loader block you just added, click the button **`dbt model`**,
then click the option `Single model`.
* In the file browser that pops up, click the file named
`my_second_dbt_model.sql` under the folders `demo/models/example/`.
* This will add 2 dbt blocks to your pipeline: 1 for the dbt model named
`my_first_dbt_model` and the 2nd for the dbt model named `my_second_dbt_model`.
* The model named `my_first_dbt_model` was added to
the pipeline because `my_second_dbt_model` references it.
* In the dbt block named `my_first_dbt_model`, next to the label
`Target` at the top, choose `dev` in the dropdown list. You can also
check `Manually enter target`, and enter `dev` in the input field.
* Paste the following SQL into the dbt model named `my_first_dbt_model`:
```sql theme={"system"}
WITH source_data AS (
SELECT 1 AS id
UNION ALL
SELECT 2 AS id
)
SELECT *
FROM source_data
```
* Run the dbt model block by pressing the play button on the top right of the
block or by pressing `Command` + `Enter`.
* You should see a preview of the query execution logs. To see the query results,
click the `Expand table` link at the bottom right corner.
* After previewing the results, in the top right corner of the block,
click on the triple dots to reveal a dropdown menu.
* Under the dropdown menu, click the option Run model. This command
will execute the `dbt run` command and create the table in your data source.
* In the dbt block named `my_second_dbt_model`, next to the label
`Target` at the top, choose `dev` in the dropdown list. You can also
check `Manually enter target`, and enter `dev` in the input field.
* Paste the following SQL into the dbt model named `my_second_dbt_model`:
```sql theme={"system"}
SELECT
a.*
, b.*
FROM {{ ref('my_first_dbt_model') }} AS a
LEFT JOIN {{ source('mage_demo', 'dbt_demo_pipeline_load_data') }} AS b
ON 1 = 1
WHERE a.id = 1
```
> [dbt sources](https://docs.getdbt.com/docs/build/sources)
>
> When a dbt model depends on an upstream block that isnโt a dbt model, a
> source for that block is automatically added to the
> `demo_project/dbt/demo/models/mage_sources.yml` file.
>
> Read more about dbt sources in their
> [documentation](https://docs.getdbt.com/docs/build/sources).
* Run the dbt model block by pressing the play button on the top right of the
block or by pressing `Command` + `Enter`.
* You should see a preview of the query execution logs. To see the query results,
click the `Expand table` link at the bottom right corner.
* On the right side of the screen, click the tab labeled **`Terminal`**.
* Create a new dbt test file by running the following command:
```bash theme={"system"}
touch demo_project/dbt/demo/tests/test_my_second_dbt_model.sql
```
* On the left side of the page in the file browser, expand the folder
`demo_project/dbt/demo/tests/` and click the file named
`test_my_second_dbt_model.sql`. If you donโt see it, refresh the page.
* Paste the following SQL in the file:
```sql theme={"system"}
SELECT id
FROM {{ ref('my_second_dbt_model') }}
GROUP BY id
HAVING (id = 0)
```
* Read more about dbt tests in their
[documentation](https://docs.getdbt.com/docs/build/tests).
* Click the name of the pipeline in the header breadcrumbs to go back to the
detail page.
* Create a new trigger with a type `Schedule` and a Frequency `once`.
For more details, follow these
[steps](/design/data-pipeline-management#create-trigger).
* After your trigger is created, click the **`Start trigger`** button at the
top of the page.
* The pipeline will eventually fail because a dbt test failed. This means
everything is working as expected.
* Open the file `demo_project/dbt/demo/models/example/schema.yml` and remove
the tests named `unique` under both models. Your file should look like this:
```yaml theme={"system"}
version: 2
models:
- name: my_first_dbt_model
description: "A starter dbt model"
columns:
- name: id
description: "The primary key for this table"
tests:
- not_null
- name: my_second_dbt_model
description: "A starter dbt model"
columns:
- name: id
description: "The primary key for this table"
tests:
- not_null
```
* Click on the **`Failed`** button next to the pipeline run and click
**`Retry run`**. It should complete running successfully after a few
minutes.
Congratulations! Youโve created a data pipeline that orchestrates your dbt
models.
# Serve dbt docs in production
Source: https://docs.mage.ai/guides/dbt/docs
To serve dbt docs in production, you will need to enable a container to host the dbt docs webserver in the cloud service you are using.
## Mage Pro (auto-generate and serve)
If you're using Mage Pro, dbt docs can be automatically generated and served when the app starts.
* Set the environment variable `ENABLE_DBT_DOCS=true`.
* Mage will discover dbt projects under `/dbt`, generate docs (if missing), and serve them.
* Open a project's docs from the UI:
* In File Browser, right-click the dbt project folder and click "Open dbt docs".
* The docs are available at `/dbt_docs/[dbt_project_name]`.
* The path `/dbt_docs` shows the status for all dbt projects (running port, errors, etc.).
To regenerate docs manually, add a generic dbt block in a pipeline and run `dbt docs generate`.
## Generating docs
Before enabling dbt docs in production, we need to make sure the static files for the docs are generated.
1. Configure the dbt super project.
* Your dbt projects should be in the `/path/to//dbt` directory.
* The `/dbt` directory will serve as a "super-project" for all of your dbt projects in your Mage project.
* Make sure the `/...//dbt` directory has the files `dbt_project.yml`, `profiles.yml`, and `packages.yml`. These files are needed for dbt to create docs for all the projects within this directory.
1. `dbt_project.yml`
```yaml theme={"system"}
name: 'base'
version: '1.0.0'
config-version: 2
profile: 'base'
target-path: "target"
clean-targets:
- "target"
- "dbt_packages"
```
2. `profiles.yml`: The config in this file is not used, but it needs to still be a valid output target in order for the project to compile.
```yaml theme={"system"}
base:
target: dev
outputs:
dev:
```
3. `packages.yml`: Add all projects that you want to be included in the documentation.
```yaml theme={"system"}
packages:
- local: ./project1
- local: ./project2
```
2. Generate the dbt docs
```bash theme={"system"}
dbt deps
dbt docs generate
```
## Terraform
### GCP
In the [mage-ai-terraform-templates](https://github.com/mage-ai/mage-ai-terraform-templates) repository, the terraform templates for GCP have commented out resources at the end of the following files for enabling the dbt docs service.
* [main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf#L184)
* [load\_balancer.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/load_balancer.tf#L81)
Once you uncomment those resources, you should see an output `docs_service_ip` at the end of `terraform apply` with the IP to access the dbt docs in the cloud.
```bash theme={"system"}
Outputs:
docs_service_ip = ""
service_ip = ""
```
### AWS
Coming soon...
# Incremental models
Source: https://docs.mage.ai/guides/dbt/incremental-models
For more information about how to use incremental models, please read dbtโs
[documentation](https://docs.getdbt.com/docs/build/incremental-models).
## How to rebuild incremental models?
You can rebuild 1 or more models using the `--full-refresh` flag
([dbt documentation](https://docs.getdbt.com/docs/build/incremental-models#how-do-i-rebuild-an-incremental-model))
and graph operators ([dbt documentation](https://docs.getdbt.com/reference/node-selection/graph-operators)).
In order to do this in Mage, follow these steps:
1. Create a new [trigger](/guides/triggering-pipelines).
You may want to create a scheduled trigger with an interval of `@once` because
running a full refresh of an incremental model is usually done ad hoc and not
on a regular basis.
2. On the right side of the edit trigger page,
look for the section labeled dbt runtime settings.
3. Under the column labeled Flags, check the box labeled `--full-refresh`.
4. \[Optional] Under the column labeled Prefix, write the graph operators of your choice.
5. \[Optional] Under the column labeled Suffix, write the graph operators of your choice.
# Use dbt with Mage: Simplify Your Data Transformation Pipelines
Source: https://docs.mage.ai/guides/dbt/overview
Learn how Mage integrates with dbt to orchestrate, schedule, and monitor dbt models as part of your data pipelines.
dbt integration is currently only supported when using Mage in
[Docker](/getting-started/setup#using-docker) or in [Mage Pro](https://cloud.mage.ai/sign-up).
## What is Mage's dbt Integration?
Mage is a modern data pipeline tool that lets you **build, schedule, and monitor dbt models** alongside your data workflows โ all in one platform. With Mage, you can fully automate your dbt workflows without using Airflow or dbt Cloud.
## Who Should Use dbt in Mage?
Mageโs dbt integration is designed for both analytics engineers and data engineers who want more control, flexibility, and observability in their data workflows.
### Analytics engineers
If you're transforming raw data into clean, analytics-ready datasets using dbt, Mage helps you:
* A **visual interface** to build and run dbt models
* The ability to **chain dbt steps** with Python or SQL blocks
* Easy scheduling and triggering for model runs
* **Live SQL previews** to speed up development
Mage empowers analytics teams to move faster, test confidently, and collaborate more effectively โ all within a unified platform.
### Data engineers
If you're building and orchestrating production-grade data pipelines, Mage lets you:
* Run dbt models as part of larger **end-to-end workflows**
* Replace complex orchestration tools like **Airflow**
* Run dbt across environments โ including **Docker**, **Kubernetes**, [**Mage Pro**](https://cloud.mage.ai), or in the **cloud (AWS, GCP, Azure)**
* Monitor, debug, and get alerts on dbt runs with built-in **observability tools**
Mage simplifies pipeline orchestration and lets you focus on data logic โ not infrastructure setup or maintenance.
## Key Features
### Schedule dbt model runs
Set dbt models to run:
* On a schedule
* When a file or event triggers
* From an API call or upstream task
This is perfect for productionizing your dbt workflows.
### Run specific dbt models and their dependencies
Use Mageโs visual interface to:
* Select and run individual dbt models
* Automatically include required dependencies
This saves time during development and debugging.
### Exclude Models When Running All
Need to run everything except one model? No problem. Mage lets you exclude specific dbt models in any run.
### Chain dbt Models with Other Tasks
Build workflows where dbt models run after:
* Pulling data from APIs
* Completing a Python or SQL block
* Finishing a different pipeline
You can treat dbt as a native building block in your DAG.
### Preview dbt model results as you write SQL
Mage lets you **preview dbt model output as you write SQL**, helping you test and iterate faster without running full pipelines.
### Build dynamic dbt pipelines using flexible variable interpolation
Take advantage of dynamic variable interpolation like:
* `{{ env_var('...') }}`
* `{{ variables('...') }}`
* `{{ mage_secret_var('...') }}`
### Automatically run dbt tests every time a pipeline runs
Mage will:
* Detect and run your dbt tests during each pipeline run
* Fail the pipeline if any dbt test fails
This enforces quality without extra setup.
### Built-In Monitoring and Alerting
Mage gives you full observability into your dbt runs:
* Track model run history
* See logs and SQL output
* Get alerts when models fail
You donโt need to wire up additional logging systems.
## Supported connectors
1. `dbt-bigquery`
2. `dbt-clickhouse`
3. `dbt-core`
4. `dbt-dremio`
5. `dbt-duckdb`
6. `dbt-mysql`
7. `dbt-postgres`
8. `dbt-redshift`
9. `dbt-snowflake`
10. `dbt-spark`
11. `dbt-sqlserver`
12. `dbt-synapse`
13. `dbt-trino`
## Tutorials
Explore our tutorial to set up and run your first dbt models in Mage:
* [How to Set Up dbt Models & Orchestrate Runs in Mage](/guides/dbt/setup-dbt)
# Run dbt-spark against a pySpark session.
Source: https://docs.mage.ai/guides/dbt/run-dbt-spark
The following procedure demonstrates how to run `dbt-spark` with a pySpark session.
1. Build a Mage docker image with Spark following the instructions given at
[Build Mage docker image with Spark environment](/integrations/spark-pyspark#1-build-mage-docker-image-with-spark-environment).
2. Run the following command in your terminal to start Mage using docker:
```bash theme={"system"}
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
/app/run_app.sh mage start demo_project
```
3. Create a new pipeline with a name `dbt_spark`, and add a `Scratchpad` to
test out the connection with PySpark, with the following code:
```python theme={"system"}
from pyspark.sql import SparkSession
import os
spark = SparkSession.builder.master(os.getenv('SPARK_MASTER_HOST', 'local')).getOrCreate()
spark.sql("show databases;")
```
It should return results similar to the following when running:
```
[Stage 0:>
(0 + 1) / 1]
namespace
0 default
```
4. Click the `Terminal` icon on the right side of the Mage UI, and create a dbt
project `spark_demo`, with the following commands:
```bash theme={"system"}
root@488dc9529cf3:/home/src# cd demo_project/dbt
root@488dc9529cf3:/home/src# dbt init spark_demo -s
root@488dc9529cf3:/home/src# touch spark_demo/profiles.yml
```
5. On the left side of the page in the file browser, expand the folder `demo_project/dbt/spark_demo/`.
Click the file named `profiles.yml`, and add the following settings to this file:
```yaml theme={"system"}
spark_demo:
target: dev
outputs:
dev:
type: spark
method: session
schema: default
host: local
```
6. Save the `profiles.yml` file by pressing `Command (โ) + S`, then close the file
by pressing the X button on the right side of the file name `dbt/spark_demo/profiles.yml`.
7. Click the button `dbt model`, and choose the option `New model`. Enter `model_1`
as the `Model name`, and `spark_demo/models/example` as the folder location.
8. In the dbt block named `model_1`, next to the label `Target` at the top,
choose `dev` in the dropdown list. You can also check `Manually enter target`,
and enter `dev` in the input field.
9. Paste the following SQL into the dbt block `model_1`:
```python theme={"system"}
{{ config(materialized='table') }}
with source_data as (
select 1 as id
union all
select null as id
)
select *
from source_data
```
Click the `Compile & preview` button to execute this new model, which would
generate the results similar to the following:
# Run multiple models
Source: https://docs.mage.ai/guides/dbt/run-selected-model
(and optionally exclude others)
## Run
1. Under the data loader block you just added, click the button
`dbt model`, then click the option `All models`.
2. In the `dbt project name` input field, enter the name of the dbt project
that contains the models you want to build and run (e.g. `demo`).
3. In the `dbt profile target` input field, enter the name of the dbt
connection profile (e.g. `dev`) that you want to use when running the
selected dbt models.
4. In the text area of the code block, write the models you want to select or
exclude using dbtโs `--select` and `--exclude` flags and syntax.
For more information on the `--select` and `--exclude` syntax,
[read dbtโs documentation](https://docs.getdbt.com/reference/node-selection/syntax#examples).
For example:
```yaml --select theme={"system"}
$ dbt run --select my_dbt_project_name # runs all models in your project
$ dbt run --select my_dbt_model # runs a specific model
$ dbt run --select path.to.my.models # runs all models in a specific directory
$ dbt run --select my_package.some_model # run a specific model in a specific package
$ dbt run --select tag:nightly # run models with the "nightly" tag
$ dbt run --select path/to/models # run models contained in path/to/models
$ dbt run --select path/to/my_model.sql # run a specific model by its path
```
```yaml --exclude theme={"system"}
$ dbt run --exclude my_dbt_project_name
$ dbt run --exclude my_dbt_model
$ dbt run --exclude path.to.my.models
$ dbt run --exclude my_package.some_model
$ dbt run --exclude tag:nightly
$ dbt run --exclude path/to/models
$ dbt run --exclude path/to/my_model.sql
```
## Variables
Add additional variables to your `dbt` command by writing the following in your blockโs code:
```yaml theme={"system"}
# other optional dbt command line arguments
--vars '{"key": "value"}'
```
## Interpolate values
Interpolate values in the blockโs code using data from:
1. Upstream block output
2. Variables
1. Global variables
2. Pipeline variables
3. Runtime variables
3. Environment variables
### Upstream block output
Use the data from 1 or more upstream blockโs output by using the `block_output` function.
#### `block_uuid`
The UUID of the upstream block to get data from.
If argument isnโt present, data from all upstream blocks will be fetched.
| | |
| -------- | --------------------- |
| Optional | โ |
| Type | `str` |
| Example | `'data_loader_block'` |
#### `parse`
A lambda function to parse the data from an upstream blockโs output.
If the `parse` argument isnโt present, then the fetched data from the upstream blockโs output
will be interpolated as is.
| | |
| -------- | -------------------------------------------------------------- |
| Optional | โ |
| Type | `function` |
| Example | `lambda data, variables: data['runtime'] * variables['tries']` |
Arguments
* `data`
If the `block_uuid` argument isnโt present, then the 1st argument in the lambda function is
a list of objects.
The list of objects contain the data from an upstream blockโs output.
The positional order of the data in the list corresponds to the current blockโs upstream blocks
order.
For example, if the current block has the following upstream blocks with the following output:
1. `load_api_data`: `[1, 2, 3]`
2. `load_users_data`: `{ 'mage': 'powerful' }`
Then the 1st argument in the lambda function will be the following list:
```python theme={"system"}
[
[1, 2, 3],
{ 'mage': 'powerful' },
]
```
| | |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| Optional | โ |
| Type | If `block_uuid` argument is present, then the type depends on the output from that block. If `block_uuid` isnโt present, then the type is `list`. |
| Example | `{ 'mage': 'powerful' }` |
* `variables`
A dictionary containing pipeline variables and runtime variables.
| | |
| -------- | ---------------- |
| Optional | โ |
| Type | `dict` |
| Example | `{ 'fire': 40 }` |
#### Example
With `block_uuid`
```yaml theme={"system"}
--select models/{{ block_output('data_loader_block') }}.sql
```
```yaml theme={"system"}
--select models/{{ block_output('data_loader_block', parse=lambda data, variables: data['runtime'] * variables['tries']) }}.sql
```
```yaml theme={"system"}
--vars '{"user_id": "{{ block_output('load_recent_user', parse=lambda user, _variables: user['id']) }}"}'
```
Without `block_uuid`
```yaml theme={"system"}
--select models/{{ block_output() }}.sql
```
```yaml theme={"system"}
--select models/{{ block_output(parse=lambda outputs, variables: outputs[1]['runtime'] * variables['tries']) }}.sql
```
```yaml theme={"system"}
--vars '{"user_id": "{{ block_output(parse=lambda outputs, _variables: outputs[0]['id']) }}"}'
```
### Variables
Interpolate values from a dictionary containing keys and values from:
1. Global variables
2. Pipeline variables
3. Runtime variables
#### Example
```yaml theme={"system"}
--select models/{{ variables('some_var_name') }}.sql
```
```yaml theme={"system"}
--vars '{"mage": "{{ variables('power') }}"}'
```
### Environment variables
Interpolate values from the environment variables.
#### Example
```yaml theme={"system"}
--select models/{{ env_var('some_environment_variable_name') }}.sql
```
```yaml theme={"system"}
--vars '{"environment": "{{ env_var('ENV') }}"}'
```
# Run a single model
Source: https://docs.mage.ai/guides/dbt/run-single-model
1. Under the data loader block you just added, click the button **`dbt model`**,
then click the option `Single model`.
2. In the file browser that pops up, click the file of the dbt model you want to
add to your pipeline.
3. In the `dbt profile target` input field, enter the name of the dbt connection
profile (e.g. `dev`) that you want to use when running the selected dbt
models.
***
## Depending on upstream blocks
If your dbt model references other models, those models will also be added to
the current pipeline as upstream blocks.
Once youโve added 1 or more dbt models to your pipeline, you can set its
dependencies on other blocks.
The dbt model wonโt run until all upstream blocks have successfully been
completed.
***
## Preview dbt model results
You can run a dbt model block and see the results of the SQL query.
Under the hood, Mage is running
[`dbt compile`](https://docs.getdbt.com/reference/commands/compile) for that
single model, then executing the compiled SQL query in the data source from your
dbt projectโs
[connection profile](https://docs.getdbt.com/docs/get-started/connection-profiles)
target.
***
## Pipeline execution run
When pipeline is triggered and executes, itโll run each dbt model block using
the [`dbt run`](https://docs.getdbt.com/reference/node-selection/syntax)
command.
***
## Variable interpolation
You can add
[variables](/getting-started/runtime-variable)
specific to your pipeline. These variables are accessible in each block of your
pipeline.
In addition, all the environment variables are accessible within the SQL query
using the `var` syntax.
| Syntax | Description | Example |
| ------------------ | ---------------------------------------------------------------- | ----------------- |
| `{{ var('...') }}` | Get a value from the environment variables or runtime variables. | `{{ var('ds') }}` |
### SQL statement
Within the SQL query in a dbt model block, you can access these variables using
the `var` syntax. To learn more, read
[dbtโs documentation](https://docs.getdbt.com/reference/dbt-jinja-functions/var).
For example:
```SQL theme={"system"}
SELECT *
FROM users_v1
WHERE created_at >= '{{ var("ds") }}'
```
In the above example, the variable named `ds` is defined as part of the runtime
variables for the pipeline.
### Upstream block output
You can also use the output from an upstream Mage block inside a dbt SQL model
with `block_output`. This is useful when a Python, SQL, or dynamic block decides
which value the dbt model should query, filter, or materialize.
For example, an upstream data loader named `select_city` can return a record:
```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
return [
{
'city': 'seattle',
'run_label': 'daily_refresh',
},
]
```
Then a dbt SQL model block can interpolate that output before dbt runs:
```sql theme={"system"}
select
'{{ block_output("select_city", parse=lambda rows: rows[0]["city"]) }}' as city,
'{{ block_output("select_city", parse=lambda rows: rows[0]["run_label"]) }}' as run_label
```
If the upstream block is dynamic, Mage resolves the output for the current
dynamic run and passes that value to the dbt model. For structured outputs, use
`parse` to return a scalar value that is valid SQL.
Mage only hydrates `block_output(...)` before invoking dbt. Other dbt Jinja such
as `{{ config(...) }}`, `{{ ref(...) }}`, and `{{ source(...) }}` is left for dbt
to compile. Hydrated dbt SQL is written to an isolated temporary copy of the dbt
project for each execution, so concurrent runs of the same model do not overwrite
the source model file or read another run's interpolated SQL.
***
## Adding variables when running a YAML dbt block
In addition to interpolating variables described above, you can define variables directly
in the dbt code block if it is a YAML dbt code block (i.e. not a SQL dbt block) by following
these steps:
1. In the Pipeline Editor, add a dbt model by selecting on the `All models (w/option exclusion)`
option from the dropdown when clicking on the `dbt model` button for adding a block.
2. For the newly added code block, you should see that is `YAML` type indicated in the top left
corner of the block.
3. On the same line as your `--select` or `--exclude` syntax, add the `--vars` syntax with the
variables you want to include. Note: Putting the `--vars` syntax on a separate line may cause
issues.
4. Format the variables like a JSON object, so there should be double quotes around the keys and
string values. Single quotes surrounding the variables object are optional.
5. You can interpolate global or environment variables using double brackets around the
variable name. See below for examples.
Examples of YAML dbt code block content (note the syntax for the variables object):
```
--select demo/models --vars '{"demo_key": "demo_value", "date": 20230101}'
--select demo/models --vars {"demo_key":"demo_value","date":20230101}
--select demo/models --vars '{"global_var": {{ test_global_var }}, "env_var": {{ test_env_var }}}'
--select demo/models --vars {"refresh":{{page_refresh}},"env_var":{{env}}}
```
# Running dbt tests
Source: https://docs.mage.ai/guides/dbt/running-dbt-tests
You can write tests for your dbt models by following
[dbtโs documentation](https://docs.getdbt.com/docs/build/tests).
When a pipeline runs, each dbt model block will run and then each test
associated with that dbt model will be ran.
For example, if you add a dbt block to your pipeline for a dbt model named
`my_third_dbt_model` and a test file named `test_my_third_dbt_model.sql` with
the following folder structure:
```
demo_project/
| dbt/
| | demo/
| | | models/
| | | | example/
| | | | | my_third_dbt_model.sql
| | | tests/
| | | | example/
| | | | | test_my_third_dbt_model.sql
```
Then whenever your pipeline runs, the test in `test_my_third_dbt_model.sql`
will also be ran after the dbt run for the model `my_third_dbt_model` is
complete.
Any errors in the dbt test will also fail the pipeline run. The output of the
test will be logged and viewable when inspecting the pipelineโs run and logs.
***
## Folder and file naming conventions
In your tests folder, you must mirror the path of the model you want to test.
For example, if your model is named `my_third_dbt_model` and if its located
under the folder named `example`, then your test file must be located here:
```
models/example/my_third_dbt_model.sql
```
```
tests/example/my_third_dbt_model.sql
```
Note that the model file and test file have the same name (e.g.
`my_third_dbt_model.sql`) and are nested under a folder with identical names
(e.g. `example`).
***
## Example
The content of the dbt model named `my_third_dbt_model` is:
```sql theme={"system"}
SELECT 1 AS n_constant_number
```
### Failed test
The content of the test for `my_third_dbt_model.sql` that will cause a test
failure is:
```sql theme={"system"}
SELECT n_constant_number
FROM {{ ref('my_third_dbt_model') }}
GROUP BY n_constant_number
HAVING (n_constant_number = 1)
```
Here is the sample output of a failed test for the above files:
```
23:57:21 Running with dbt=1.3.0
23:57:21 Found 3 models, 4 tests, 0 snapshots, 0 analyses, 289 macros, 0 operations, 0 seed files, 6 sources, 0 exposures, 0 metrics
23:57:21
23:57:23 Concurrency: 1 threads (target='dev')
23:57:23
23:57:23 1 of 1 START test my_third_dbt_model ........................................... [RUN]
23:57:23 1 of 1 FAIL 1 my_third_dbt_model ............................................... [FAIL 1 in 0.76s]
23:57:24
23:57:24 Finished running 1 test in 0 hours 0 minutes and 3.08 seconds (3.08s).
23:57:24
23:57:24 Completed with 1 error and 0 warnings:
23:57:24
23:57:24 Failure in test my_third_dbt_model (tests/example/my_third_dbt_model.sql)
23:57:24 Got 1 result, configured to fail if != 0
23:57:24
23:57:24 compiled Code at target/compiled/demo/tests/example/my_third_dbt_model.sql
23:57:24
23:57:24 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
Traceback (most recent call last):
File "/home/src/mage_ai/cli/main.py", line 146, in
main()
File "/home/src/mage_ai/cli/main.py", line 116, in main
ExecutorFactory.get_pipeline_executor(pipeline).execute(
File "/home/src/mage_ai/data_preparation/executors/pipeline_executor.py", line 28, in execute
asyncio.run(self.pipeline.execute(
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/src/mage_ai/data_preparation/models/pipeline.py", line 233, in execute
await execution_task
File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 147, in run_blocks
await asyncio.gather(*remaining_tasks)
File "/home/src/mage_ai/data_preparation/models/block/__init__.py", line 94, in execute_and_run_tests
run_dbt_tests(
File "/home/src/mage_ai/data_preparation/models/block/dbt/utils/__init__.py", line 466, in run_dbt_tests
raise Exception('dbt test failed.')
Exception: dbt test failed.
```
### Successful test
The content of the test for `my_third_dbt_model.sql` that will yield a passing
test is:
```sql theme={"system"}
SELECT n_constant_number
FROM {{ ref('my_third_dbt_model') }}
GROUP BY n_constant_number
HAVING (n_constant_number = 0)
```
Here is the sample output of a successful test for the above files:
```
23:59:56 Running with dbt=1.3.0
23:59:56 Found 3 models, 4 tests, 0 snapshots, 0 analyses, 289 macros, 0 operations, 0 seed files, 6 sources, 0 exposures, 0 metrics
23:59:56
23:59:57 Concurrency: 1 threads (target='dev')
23:59:57
23:59:57 1 of 1 START test my_third_dbt_model ........................................... [RUN]
23:59:58 1 of 1 PASS my_third_dbt_model ................................................. [PASS in 0.72s]
23:59:59
23:59:59 Finished running 1 test in 0 hours 0 minutes and 3.00 seconds (3.00s).
23:59:59
23:59:59 Completed successfully
23:59:59
23:59:59 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
```
# Make dbt magical
Source: https://docs.mage.ai/guides/dbt/setup-dbt
โฐ Run dbt models in Mage in under a minute.
## About
Our demo dbt repo currently offers two quickstarts: one that runs a simple dbt pipeline and one that runs a dbt pipeline from an external repo. At this time, running dbt in Mage requires Docker, so both of our demos are `docker compose` based.
## Prerequisites
1. [Docker](https://docs.docker.com/engine/install/)
2. [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
## Configuration
The following command will clone the [demo repo](https://github.com/mage-ai/dbt-quickstart), copy `dev.env` to `.env` and run `docker compose up` to start Mage *and* a Postgres database.
```bash Mac/Linux theme={"system"}
git clone https://github.com/mage-ai/dbt-quickstart mage-dbt-quickstart \
&& cd mage-dbt-quickstart \
&& cp dev.env .env && rm dev.env \
&& docker compose up
```
```bash Windows theme={"system"}
git clone https://github.com/mage-ai/dbt-quickstart mage-dbt-quickstart
cd mage-dbt-quickstart
cp dev.env .env
rm dev.env
docker compose up
```
After running the above command, open the Mage overview page at [http://localhost:6789](http://localhost:6789). Click the pipelines icon on the left to enter the pipelines overview.
## Tutorial
First, select the `simple_dbt_python_pipeline` by double-clicking the row from the list of pipelines this will take you directly to the editor. You can also single click,
then select the "code" icon from the side nav.
Scroll to the bottom-most cell and click *Execute with all upstream blocks*.
You just:
* Seeded a dbt model
* Performed a dbt transformation
* Took the transformed data and performed a Python transformation
* Wrote the data to a Postgres source
To see the output, you can use a querying tool (like [DataGrip](https://www.jetbrains.com/datagrip/) or [psql](https://www.postgresql.org/docs/current/app-psql.html)) to the locally hosted Postgres database and query `public.analytics.cur_customers`.
| customer\_id | first\_name | last\_name | letters\_first\_name | is\_alliterative |
| ------------ | ----------- | ---------- | -------------------- | ---------------- |
| 1 | Michael | P. | 7 | false |
| 2 | Shawn | M. | 5 | false |
| 3 | Kathleen | P. | 8 | false |
| 4 | Jimmy | C. | 5 | false |
| 5 | Katherine | R. | 9 | false |
| 6 | Sarah | R. | 5 | false |
| 7 | Martin | M. | 6 | true |
| 8 | Frank | R. | 5 | false |
| 9 | Jennifer | F. | 8 | false |
| 10 | Henry | W. | 5 | false |
Select the `dynamic_dbt_pipeline` by double-clicking the row from the list of pipelinesโ this will take you directly to the editor. You can also single click, then select the "code" icon from the side nav.
Scroll to the bottom-most cell and click *Execute and run all upstream blocks*.
You just:
* Pulled a GitHub repo and wrote it to your local Mage directory
* Wrote a demo `profiles.yaml` file that interpolated environment variables from your instance
* Compiled `dbt build` to write data to your local postgres database.
To run this pipeline, click the name of the pipeline to open the *Triggers* page and select *Run @once* to trigger a pipeline run. Select the *logs* icon to see the run in realtime!
Now, in addition to pulling the external Jaffle Shop demo and writing the necessary profiles, you'll perform [`dbt build`](https://docs.getdbt.com/reference/commands/build), which will: run models, test tests, snapshot snapshots, and seed seeds. ๐ฅณ
To see the output, you can use a querying tool (like DataGrip or psql) to the locally hosted Postgres database. To learn more about the Jaffle Shop demo, [check out the repo](https://github.com/dbt-labs/jaffle_shop).
# Run dbt snapshot
Source: https://docs.mage.ai/guides/dbt/snapshots
Run multiple individual snapshots or bulk snapshots
## Multiple individual snapshots
You can add 1 or more snapshot SQL files into your pipeline as a dbt block.
Add a dbt block as if you were adding a [dbt model](/dbt/run-single-model).
However, select a snapshot file instead of a model file.
## Bulk snapshots
Instead of adding a snapshot file to your pipeline 1 at a time,
you can run the `dbt snapshot` command on an entire folder or exclude certain folders
by doing the following:
1. Add a dbt block and select the option labeled Generic dbt command.
2. In the text input field labeled command, enter the value `snapshot`.
3. You can optionally add the `--select` or `--exclude` flag in the body of the block.
# dbt sources and upstream dependencies
Source: https://docs.mage.ai/guides/dbt/sources
dbt model blocks can depend on other dbt models and other blocks:
[sensors](/design/blocks/sensor), Python,
[SQL](/guides/blocks/sql-blocks),
[R](/guides/blocks/r-blocks).
***
## dbt models depending on other dbt models
When you add a dbt model block to your pipeline, if it contains any `ref`s,
those dbt models will be added to the current pipeline as upstream dependencies
to the dbt model you initially added.
***
## dbt models depending on other blocks
When a dbt model depends on a block that isnโt another dbt model, a YAML file is
created named `mage_sources.yml`. This file is automatically generated and
updated whenever a new non-dbt model block is an upstream dependency of a dbt
model block.
If you have a Mage project named `demo_project` with a dbt project named `demo`,
here is an example of where the `mage_sources.yml` file could be located:
```
demo_project/
| dbt/
| | demo/
| | | models/
| | | | mage_sources.yml
| | | | example/
| | | | | my_first_dbt_model.sql
| | | | | my_second_dbt_model.sql
```
### Sources
The contents of the `mage_sources.yml` could look like this:
```yaml theme={"system"}
sources:
- loader: mage
name: mage_demo
schema: public
tables:
- identifier: mage_dbt_demo_pipeline_load_data
meta:
block_uuid: load_data
pipeline_uuid: dbt_demo_pipeline
name: dbt_demo_pipeline_load_data
version: 2
```
The source name follows this convention: `mage_[DBT_project_name]`.
The table names under the source follow this convention:
`[pipeline_uuid]_[block_uuid]`.
To reference the preceding block in a dbt block use this convention:
`{{ source('[source_name]', '[pipeline_uuid]_[block_uuid]') }`
e.g.
`{{ source('mage_demo', 'dbt_demo_pipeline_load_data') }`
For more information on sources, read
[dbtโs documentation](https://docs.getdbt.com/docs/build/sources).
# dbt variable interpolation
Source: https://docs.mage.ai/guides/dbt/variable-interpolation
## Overview
Mage allows users to reference variables specific to your
[pipeline](/getting-started/runtime-variable) or
[project](/development/variables/environment-variables).
These variables are accessible throughout your dbt profiles, and pipelines by using the following syntax.
| Syntax | Description | Example |
| ------------------------ | ------------------------------------------ | ----------------------------------- |
| `{{ env_var('...') }}` | Get from environment variables. | `{{ env_var('ENV') }}` |
| `{{ variables('...') }}` | Get from runtime variables (used in YAML). | `{{ variables('execution_date') }}` |
| `{{ var('...') }}` | Get from runtime variables (used in SQL). | `{{ var('execution_date') }}` |
DBT also comes with variable and environment variable support. By default we forward
[pipeline](/getting-started/runtime-variable)
variables as dbt [Project variables](https://docs.getdbt.com/docs/build/project-variables).
## Variable interpolation examples
### dbt models
```sql theme={"system"}
SELECT
*
# get the global environment variable for prod/dev
FROM {{ env_var('ENV') }}_users.accounts
WHERE
# limit query based on the runtime variable, execution_date
created_at BETWEEN '2023-01-01' AND {{ var('execution_date') }}
```
### dbt profiles
```yaml theme={"system"}
jaffle_shop:
target: dev
outputs:
dev:
type: postgres
host: host.docker.internal
user: postgres
password: postgres
port: 5432
dbname: postgres
schema: {{ variables('DBT_SCHEMA') }}
threads: 4
```
### dbt targets
Here, we've used a manually entered target to specify a postgres database dependent on the environment `postgres_{{ env_var('ENV') }}`.
| Environment | Key | Example string | Result |
| ----------- | ----- | ------------------------------- | --------------- |
| `prod` | `ENV` | `postgres_{{ env_var('ENV') }}` | `postgres_prod` |
| `dev` | `ENV` | `postgres_{{ env_var('ENV') }}` | `postgres_dev` |
# Block dependency tree
Source: https://docs.mage.ai/guides/developer-ux/block-dependency-tree
Money does grow on trees... if you plant them with magic beans.
## Drag-and-drop blocks to update dependencies
Click a block in the tree, hold down the mouse, and drag it to another block and release.
This will set the dragged block as a downstream block from the one that you dropped it on.
### Drag an entire group of blocks
Blocks in the tree are visually grouped if every block in the group has the same set of
upstream blocks.
When a group is visually formed, you can drag-and-drop the entire group onto another block
and the entire group will be set as the downstream for the block you dropped it on.
***
## Right click menu
Right click on any block to reveal a dropdown menu full of useful commands.
1. Run block
2. Rename block
3. Show all dependencies
4. Add upstream block
5. Add downstream block
6. Remove upstream dependencies
7. Remove downstream dependencies
8. Add/Edit interactions
9. Delete block
10. Delete block (ignore) dependencies
11. View file versions
***
## Edit the connection between blocks
Click on the line between any block and a menu will appear with these options:
1. Add new block between
2. Remove connection
***
## Drag lines to connect blocks
Hover your mouse over a block to reveal 2 orbs (aka circles, dots, etc.).
Click and hold down on the top orb and drag it to another block then release.
This will set the 1st block as the downstream dependency of the 2nd block.
If you click on the bottom orb, then the 1st block will be set as the upstream dependency of the 2nd block.
***
# Code editor with enhanced productivity
Source: https://docs.mage.ai/guides/developer-ux/code-browser
Code browser and editor with syntax highlighting, code formatting, and version history.
1. Multi-column and multi=row layout
2. Support for rendering files of all types across every Mage and non-Mage projects
3. Automatically format code styling and run linters to fix syntax
4. Customizable project and pipeline settings per environment.
5. Manage multiple Mage projects and any other files from a single application
6. Local file edit tracking and version history for restoring changes made in the past
7. Syntax highlighting
8. Extensions
# Pipeline editor canvas
Source: https://docs.mage.ai/guides/developer-ux/code-canvas
Build, run, and manage complex pipelines with the new canvas mode.
# Writing data pipeline code
Source: https://docs.mage.ai/guides/developer-ux/code-editor
Build the future you want to see.
## Adding blocks made easy
You can quickly add blocks of code to your pipeline with these methods:
**Supported block type directories:** `callbacks`, `conditionals`, `custom`, `data_exporters`, `data_loaders`, `dbt`,
`dbts`, `markdowns`, `scratchpads`, `sensors`, and `transformers`
Version 0.9.64 or greater is required to add `callback` and `conditional`
blocks from the file browser.
1. Open the file browser on the Pipeline Editor page.
2. Click a block file and keep holding down.
3. Drag the block file onto the center of the page (the Notebook area).
4. Release the button and itโll be added to the pipeline.
Find existing code and add it to the current pipeline.
You must first enable the feature named Add new block v2 in your
[project settings](http://localhost:6789/settings/workspace/preferences).
This feature is available in standard (batch) pipelines.
If you just recently enabled the feature Add new block v2, you must restart Mage
so that your blocks can be indexed.
Version 0.9.64 or greater is required to add `callback` and `conditional`
blocks from the block search.
Instead of clicking a button to add a new block, type in the search bar to find an
existing block across all your projects.
โ + / (Command + Forward Slash) keyboard shortcut will instantly focus your typing on the
search bar from anywhere on the page (as long as you arenโt already typing in a
block of code).
Use the arrow up and arrow down key to browse
through the results.
Create a template for a block or a pipeline that can be reused throughout your project.
[Read more here](/custom-templates/overview)
***
## Notes on editing pipelines
Avoid opening and editing the **same pipeline in multiple browser tabs or windows** at the same time.
Changes from one tab may overwrite changes from another.
Tips to stay safe:
* Edit a pipeline in **one tab only**.
* If you need to monitor the pipeline elsewhere, use another tab in **read-only mode**.
* Close extra editing tabs if opened by mistake.
***
### Collaboration best practices
If multiple users need to develop the **same pipeline**:
* Use **[Workspace mode in Mage Pro](/guides/developer-ux/workspaces)**, which provides:
* Isolated environments per user or team.
* Safe collaboration without overwriting each otherโs changes.
* The ability to merge updates back into shared pipelines.
This ensures smooth teamwork and prevents conflicts when more than one person is editing.
***
## Re-arrange and hide or show blocks
Drag a block and move it up or down by dropping it on another block.
When editing a pipeline, open the Files tab and use the selector above the search box
to switch between All Files and This pipeline only. The
This pipeline only filter shows repo-backed files related to the current pipeline,
including pipeline metadata and configuration files, trigger configuration, referenced block
source files, and supporting project configuration such as io\_config.yaml,
metadata.yaml, and requirements.txt. Runtime data and log files
are not included.
Use the Visibility tab to hide blocks in the notebook. Hidden blocks are collapsed,
and you can click a hidden block to expand it again. This can help optimize vertical screen
space while writing code.
***
## Pre-built data integrations for 3rd party sources
Sync data from 3rd party sources without writing boilerplate code by using pre-built
data integrations (Singer Taps and Targets) provided by the data engineering community.
Data integrations to sync your data to 3rd party destinations are also available in the
same feature set.
These sources and destinations can be combined and chained together with
your existing blocks.
[Read more here](/data-integrations/batch-pipelines)
***
## Use AI to write code for you
A line of code you donโt write is a line of code you never have to debug.
Use AI to generate the code for you.
[Read more here](/guides/ai/overview#generate-block-using-ai)
***
## Document and comment code (AI)
โWork smart, not hard.โ
Let the robots document your code so you can focus on the fun and creative side of coding.
[Read more here](/guides/ai/overview#add-documentation-for-a-block)
[Read more here](/guides/ai/overview#add-comments-in-a-block)
[Read more here](/guides/ai/overview#add-documentation-for-a-pipeline-and-all-its-blocks)
***
## Block upgrades
Enhance your code with these quick power-ups:
A very `powerful` ultimate *spell*.
With great power... comes great fun.
[Read more here](/design/blocks/dynamic-blocks)
[Read more here](/guides/blocks/replicate-blocks)
[Read more here](/design/blocks/conditionals)
A callback block is associated to another block.
When the parent block succeeds or fails, the callback block functions are executed.
[Read more here](/design/blocks/callbacks)
***
## Add new block v2
An improved experience for adding new blocks to your pipeline in the
Pipeline Editor (`/pipelines/[pipeline_uuid]/edit`), which allows you to
search for existing block files and use data integration blocks inside of
a standard (batch) pipeline.
***
## Code and output side-by-side view
Run code and see its output right next to where youโre writing.
This makes it so that you donโt have to keep scrolling down every time you make a code change
and want to see its results.
For table outputs, click a column header to sort the preview or use the filter
icon beside the header to filter rows by column value. These controls are
temporary preview tools and donโt change the underlying block output.
To enable this feature:
1. Go to your [project settings](http://localhost:6789/settings/workspace/preferences).
2. Enable the feature named Notebook block output split view.
3. Open the edit view of a pipeline.
4. Under the header menu named View, click the option Show output next to code.
***
# Browse all files
Source: https://docs.mage.ai/guides/developer-ux/file-browser
Add, edit, move, and delete files or folders in your project.
## How to show the file browser
1. When editing a pipeline, expand the left side panel to show the file browser.
2. Go to the Files page [http://localhost:6789/files](http://localhost:6789/files).
3. Launch the Text Editor application.
***
## Right click menu
### File
* Rename file
* Move file
* Download file
* Delete file
### Folder
* Open all files in the immediate directory
* New folder
* Rename folder
* Move folder
* Delete folder
* Upload files
* Expand all subfolders
* Collapse all subfolders
* New Mage project
* New dbt project
***
## File versions
When you save a file after making a change, a new file version is created and stored locally.
You can undo any change to a file by reverting to a previous version (this is different than git).
[Read more here](/getting-started/file-versions).
***
## Hidden files
The file browser will hide hidden folders and files by default. You can show them by toggling
the option Show hidden files under the View menu dropdown options.
***
## Filter files to the current pipeline
When editing a pipeline, open the Files tab and use the selector above the search box to
switch between All Files and This pipeline only. The This pipeline only
filter shows repo-backed files related to the current pipeline, including pipeline metadata and
configuration files, trigger configuration, referenced block source files, and supporting project
configuration such as io\_config.yaml, metadata.yaml, and
requirements.txt. Runtime data and log files are not included.
## Block visibility tab
Use the Visibility tab to hide blocks in the notebook. Hidden blocks are collapsed, and
you can click a hidden block to expand it again. This can help optimize vertical screen space
while writing code.
***
## Block files used/unused by pipelines
Requires version `0.9.63` or greater.
You can tell if a block is being used by a pipeline (or not used by any pipelines)
by looking at the related block file's icon in the file browser. This is a quick way
for users to see which block files are currently being used by *any* pipeline; the specific
pipeline using a block file is not indicated in the file browser.
You need to be viewing the file browser in the Files (`/files`) and Pipeline Editor
(`/pipelines/[pipeline_uuid]/edit`) pages specifically. Other pages, modals, or
components that have a file browser may not have this feature.
**Applicable block directories:** `callbacks`, `conditionals`, `custom`, `data_exporters`,
`data_loaders`, `dbt`, `extensions`, `markdowns`, `scratchpads`, `sensors`, and `transformers`
### Block file states
There are three different states for a block file in the file browser that can affect the
icon in front of the block filename and the filename font color. Note that the square icons
can be different colors depending on which block directory it is.
1. **Used by exactly one pipeline** - Filled square icon. For example in the screenshot
above, `aged_sea.py` and `api_load_data.py` are used by a single pipeline.
2. **Used by more than one pipeline** - Unfilled cyan diamond icon and filename with cyan
font color. In the screenshot above, `amazon_s3.yaml` and `ancient_pine.yaml` are used by
multiple pipelines.
3. **Not used by any pipeline** - Unfilled square icon. For example in the screenshot above,
`bigquery_loader.py` is not used by any pipeline.
### Unused block files
If you specifically want to view the block files that are not being used by any pipeline,
you can apply the "Unused block files" filter found in the file search input at the top of
the file browser. Non-block files may still be visible when this filter is applied.
***
## Display files by type
The feature is currently only available in the Text Editor application
Instead of viewing all files grouped by its parent directory,
you can toggle a different view that shows these files flattened and then grouped
by its file extension type.
This can help browse relevant files faster since you donโt have to dig into each folder
to find the file youโre looking for.
# Personalize your workspace
Source: https://docs.mage.ai/guides/developer-ux/personalize
Customize workspace themes, personalize accounts, and re-configure UI component layouts.
## Themes
***
## Logo
Change the logo that appears in the top left corner.
***
## Layout
### Add block buttons
Customize the buttons that show up by default on the Add new block bar on the notebook page
while building a pipeline.
You can change the order of what appears first as well as which buttons are visible by default
and which are displayed when the dropdown menu is shown after clicking the button All blocks.
# Cross-pipeline dependencies
Source: https://docs.mage.ai/guides/developer-ux/pipeline-dependencies
Manage cross-pipeline dependencies and execution flow across every pipeline within a project.
# Pipeline sidekick
Source: https://docs.mage.ai/guides/developer-ux/sidekick
Have you ever wished you had a sidekick like Batman and Robin? Now you have one.
The Pipeline sidekick appears when you edit a pipeline.
In the top right corner of the page, click the `Chevron` or `Arrow` looking icon to expand
the right panel and reveal the Sidekick.
***
## Visualize block output data
Add 1 or multiple charts to visualize the data produced by a block.
When viewing block output as a table in the Sidekick data view, click a column
header to sort the preview or use the filter icon to narrow rows by column value.
Sorting and filtering only change the current table preview; they donโt modify
the block output saved by the pipeline.
[Read more](/visualizations/dashboards) about how to create entire dashboards
for an individual pipeline or for the entire project.
***
## Configure pipeline variables
Use variables in your code and interpolate them at runtime.
These variables can be defined at the moment the pipeline is triggered or
pre-configured from the Sidekick.
[Read more here](/getting-started/runtime-variable)
***
## Store sensitive secrets
Secrets donโt make friends... but they save companies from oblivion.
[Read more here](/development/variables/secrets)
***
## Customize settings for every individual block
1. Rename a block
2. Set what execution engine the block should use
3. Configure how the block should retry itself in case of failure
4. Set a timeout limit for how long the block can run before it should fail
5. Configure variables specific to a block
6. View which pipelines are currently using a block
***
# Develop Mage Pro Code with VS Code or Cursor
Source: https://docs.mage.ai/guides/developer-ux/visual-studio-code
Learn how to develop Mage Pro pipelines and projects using Visual Studio Code or Cursor. This guide walks you through secure remote access via Tailscale and SSH, environment configuration, and integrating your local IDE with Mage Pro.
## Overview
Mage Pro supports remote development using [Visual Studio Code](https://code.visualstudio.com/) or [Cursor](https://www.cursor.so/), so you can build and manage your pipelines in a full-featured local IDE while connecting securely to your Mage Pro environment. This integration leverages [Tailscale VPN](https://tailscale.com/) for secure networking and SSH for remote access.
> โ ๏ธ This feature is not available by default on Trial clusters.
> To enable access, please contact the Mage team at **[pro@mage.ai](mailto:pro@mage.ai)** to request activation for your Mage Pro account.
## Core components
* **Tailscale VPN**: Secure network connection between local and remote environments
* **Remote SSH Extension**: VS Code's tool for remote development
* **Mage Pro Cluster**: Containerized environment for data pipelines
* **Authentication System**: Secure key-based access control
To integrate with VS Code or Cursor please contact Mage to request an account upgrade. The Mage team will need to enable this feature for your organization.
## Setup guide
### 1. Tailscale VPN setup
Step 1: Navigate to the Tailscale website and download the appropriate version for your operating system.
Step 2: Install Tailscale following the in-app instructions.
Step 3: Access the Tailscale admin console from their main website.
Step 4: Generate an Auth Key by navigating to Settings > Keys and selecting "Generate Auth Key."
### 2. Mage Pro configuration
#### Setting Environment Variables for the Cluster
Step 1: Navigate to your Mage Pro management cluster.
Step 2: Select "Edit/Add variableโ to add environment variables.
Step 3: Add a new custom variable:
* Key: `TS_AUTHKEY`
* Key: `TS_AUTH_KEY`
* Value: `YOUR_AUTH_KEY`
Step 4: Click the โSave changesโ button to save the TS\_AUTHKEY variable.
(This will restart your cluster).
#### Setting Environment Variables for Workspaces
If you are using Mage Pro **workspaces**, you must configure environment variables separately at the workspace level.
> **Note**: Each workspace requires its own **unique** `TS_AUTHKEY`.
> Generate a separate Tailscale auth key for each workspace to ensure proper network isolation and security.
To configure environment variables for a workspace, follow the guide here:
๐ [Workspace Environment Configuration](/guides/developer-ux/workspaces#environment-configuration)
Hereโs an example workspace configuration:
```yaml theme={"system"}
env:
- name: TS_AUTHKEY
value:
- name: TS_EXTRA_ARGS
value: --accept-routes
- name: ENV
value: dev
- name: USER_CODE_PATH
value: /home/src/
```
### 3. Connection verification
Step 1: Return to the Tailscale admin panel after your cluster restarts.
Step 2: Locate your Mage Pro cluster in your Machineโs list (i.e. mageai-1-tutorials).
Step 3: Note the private IP address from the address column.
Step 4: Open a terminal on your local machine and run the following command:
```bash theme={"system"}
curl :6789/api/status
```
Step 5: Verify that you receive a successful connection response.
### 4. VS Code setup
Step 1: Open Visual Studio Code on your local machine.
Step 2: Install the "Remote - SSH" extension from the Extensions marketplace (Ctrl+Shift+X or Cmd+Shift+X on macOS).
Step 3: Open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P on macOS).
Step 4: Type "Remote-SSH: Connect to Host" and select it.
Step 5: Enter or select the correct ip address.
Step 6: Enter the login username and password which will first be provided to you by Mage
### 5. Accessing your code
Step 1: When prompted for the folder to open, enter your user code path.
* Default user code path: `/home/src/default_repo`
Step 3: If you've customized this path, use your specific directory.
Step 4: Click "OK" to open your project files in VS Code.
# Setting up Tailscale on AWS
Connect your AWS instances to your secure network using Tailscale to enable access to your cloud resources. When using Mage Pro in AWS environments, proper route configuration ensures reliable connectivity between your local machine and cloud resources.
## Setting up route acceptance in Tailscale
Route acceptance in Tailscale can be configured in two ways to ensure proper connectivity between your local machine and AWS-hosted Mage Pro environment.
### 1. Using the command line
Run the following command on your local machine to accept routes advertised by Tailscale nodes:
```bash theme={"system"}
tailscale up --accept-routes
```
### 2. Using the admin console
Step 1: Navigate to the Tailscale admin console ([https://login.tailscale.com/admin](https://login.tailscale.com/admin))
Step 2: Go to the "Machines" page
Step 3: Find the machine that's advertising routes (in your case, the "vpn" node)
Step 4: Click on it to view details
Step 5: Under "Subnet routes" section, approve which subnet routes should be advertised to the entire tailnet
### Understanding route configuration
The two configuration methods work together:
* Admin UI settings control which machines can advertise routes (a tailnet-wide setting)
* Command-line option (`-accept-routes`) determines whether your specific machine will accept those advertised routes
### Configuring Mage Pro for route acceptance
To ensure your Mage Pro environment automatically accepts routes when it starts, add the following environment variable:
* Key: `TS_EXTRA_ARGS`
* Value: `-accept-routes`
### Verifying route acceptance
After configuring route acceptance, verify it's working correctly:
```bash theme={"system"}
tailscale status
```
You should no longer see health warnings about routes not being accepted, and you should be able to access the subnet IPs that were being advertised.
# Mage Pro workspaces
Source: https://docs.mage.ai/guides/developer-ux/workspaces
Set up Mage Pro workspaces for organizing your own personal environment.
## Overview
Mage Pro workspaces provide isolated environments for developing and managing data pipelines. Each workspace operates as a separate container instance, allowing for independent resource allocation and configuration management.
## Core Components
* Runtime Environment: Containerized Python environment
* Storage System: Persistent volume for code and data
* Resource Manager: Kubernetes-based resource allocation
* Access Control: Role-based permission system
To utilize the collaborative workspace environment, please contact Mage to request an account upgrade.
The Mage team will need to enable this feature for your organization.
## Setup Guide
* **Step 1**: After logging into the Mage Pro environment you will be in the Workspaces interface (url: [https://cluster.mage.ai/\[cluster-uuid\]/manage](https://cluster.mage.ai/\[cluster-uuid]/manage)). Press the Create new workspace button.
* **Step 2**: Give the workspace a name.
* **Step 3**: Click the dropdown arrow and toggle on the Kubernetes configuration if you need to custom configure your Kubernetes container. (see Kubernetes container configuration section for more details).
* **Step 4**: Click the Lifecycle (optional) dropdown to create lifecycle management policies (see Lifecycle management policies section for more information).
* **Step 5**: Click the create button and wait a few minutes for your workspace to start.
* **Step 6**: Once your workspace is running, click the workspace open button under the open column in the workspace management interface.
## Workspace Configuration Options
Mage Pro workspaces support comprehensive configuration options that allow you to customize your workspace environment, resources, and behavior. Configuration options vary by cluster type (Kubernetes, Docker).
### Basic Workspace Configuration
All workspace types support these basic fields:
* **`name`** (required): The workspace name. Must be unique within your cluster.
* **`lifecycle_config`** (optional): Lifecycle management policies (see [Lifecycle Management Policies](#lifecycle-management-policies-optional) section below).
### Kubernetes Workspace Configuration
Kubernetes workspaces support the following configuration options:
#### Container Configuration (YAML)
The `container_config` field accepts a YAML string that allows you to configure:
1. **Resource Configuration**: Controls memory and CPU allocation for your workspace container
2. **Environment Variables**: Customize environment variables for your workspace
3. **Other Container Settings**: Any other Kubernetes container configuration options
* **Step 4**: Click the create button and wait a few minutes for your workspace to start.
* **Step 5**: Once your workspace is running, click the workspace open button under the open column in the workspace management interface.
## Kubernetes container configuration
Kubernetes provides a robust container orchestration system for managing your workspace resources and environment. The configuration above covers two key aspects:
1. Resource Configuration: Controls memory and CPU allocation for your workspace container, with both minimum guarantees and upper limits to ensure stable performance.
2. Environment Configuration: Enables customization of environment variables, specifically the project code location if you've modified it from the default path.
These settings help maintain workspace stability and ensure proper resource allocation for development tasks.
### Environment configuration
#### Setting `USER_CODE_PATH`
If you've modified the default `USER_CODE_PATH` environment variable in the base project (Mage Pro management portal), update your Kubernetes configuration:
```yaml theme={"system"}
env:
- name: USER_CODE_PATH
value: /home/src/
```
This environment variable tells your workspace where to find your project code.
#### Adding Other Environment Variables
You can define additional environment variables in the same env section to further configure your workspace:
```yaml theme={"system"}
env:
- name: USER_CODE_PATH
value: /home/src/
- name: ENV
value: dev
- name: TS_AUTHKEY
value:
- name: TS_EXTRA_ARGS
value: --accept-routes
```
**Note on ENV variable**: If you set `WORKSPACE_ENV_PATTERN` at the cluster level, the `ENV` variable will be automatically set based on the workspace name. You can still explicitly set `ENV` in the container configuration to override the pattern. See the [Auto-set ENV variable from workspace name](#auto-set-env-variable-from-workspace-name) section for more details.
### Resource configuration (optional)
**Optionally** configure your workspace's computing resources in Kubernetes:
```yaml theme={"system"}
resources:
requests: # Minimum resources guaranteed
memory: "2Gi" # 2 gigabytes of RAM
cpu: "1" # 1 full CPU core
limits: # Maximum resources allowed
memory: "4Gi" # 4 gigabytes RAM ceiling
cpu: "2" # Can use up to 2 CPU cores
```
This configuration ensures your workspace has adequate resources for development while preventing resource overconsumption. The requests represent guaranteed minimums, while limits prevent your workspace from using more than its allocated share.
#### Default Resource Configuration
These environment variables set default resource requests and limits for Kubernetes workspace containers. They are applied when a workspace's `container_config` does not specify resources (or when specific fields like `requests.cpu` or `limits.memory` are omitted). User-specified values in `container_config` always take precedence.
| Variable | Purpose | Example Value |
| -------------------------------------------- | ----------------------------------------------- | ------------- |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU` | Default CPU request for workspace containers | `500m` |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY` | Default memory request for workspace containers | `2Gi` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU` | Default CPU limit for workspace containers | `4` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY` | Default memory limit for workspace containers | `8Gi` |
**Example:**
```bash theme={"system"}
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU="500m"
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY="2Gi"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU="4"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY="8Gi"
```
#### Complete Kubernetes Container Configuration Example
Here's a complete example of a Kubernetes `container_config` YAML:
```yaml theme={"system"}
env:
- name: USER_CODE_PATH
value: /home/src/
- name: ENV
value: dev
- name: TS_AUTHKEY
value:
- name: TS_EXTRA_ARGS
value: --accept-routes
resources:
requests: # Minimum resources guaranteed
memory: "2Gi" # 2 gigabytes of RAM
cpu: "1" # 1 full CPU core
limits: # Maximum resources allowed
memory: "4Gi" # 4 gigabytes RAM ceiling
cpu: "2" # Can use up to 2 CPU cores
```
**Resource Configuration Details:**
* **`requests`**: Minimum resources guaranteed to your workspace. Kubernetes will ensure these resources are available.
* **`limits`**: Maximum resources your workspace can use. Prevents resource overconsumption and ensures fair resource allocation across workspaces.
**Note**: You can include any valid Kubernetes container configuration in the `container_config` YAML, including:
* Container image (`image`)
* Environment variables (`env`)
* Resource requests and limits (`resources`)
* Volume mounts (`volumeMounts`)
* Security context (`securityContext`)
* And other Kubernetes container spec fields
#### Container Image Configuration
When creating or updating a workspace, you can specify the container image in the `container_config` YAML:
```yaml theme={"system"}
image: [image_repo]/mage-pro:20260112-1200
```
**Important behavior when updating workspaces:**
* **If `image` is specified**: The workspace will use the specified image.
* **If `image` is NOT specified**: The system will use the image from the main cluster pod configuration template (`pod_config`), **not** the current workspace's image.
This means that when updating a workspace without specifying an `image` in `container_config`, the workspace image may change to match the main cluster pod configuration, even if the workspace currently uses a different image. To preserve the current image, explicitly include it in your `container_config` when updating.
### Pod Scheduling Configuration
You can control which nodes your workspace pods are scheduled on using `nodeSelector`, `affinity`, and `tolerations`.
#### Node Selector
Use `nodeSelector` to schedule workspace pods on nodes with specific labels. This is the simplest way to target specific nodes.
**Example:**
```yaml theme={"system"}
nodeSelector:
disktype: ssd
instance-type: gpu
```
#### Affinity
Use `affinity` for more advanced scheduling rules, including node affinity, pod affinity, and pod anti-affinity.
**Node Affinity Example:**
```yaml theme={"system"}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-type
operator: In
values:
- compute-optimized
```
**Pod Anti-Affinity Example (spread workspaces across nodes):**
```yaml theme={"system"}
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- workspace
topologyKey: kubernetes.io/hostname
```
#### Tolerations
Use `tolerations` to allow workspace pods to be scheduled on nodes with taints.
**Example:**
```yaml theme={"system"}
tolerations:
- key: "workspace"
operator: "Equal"
value: "true"
effect: "NoSchedule"
- key: "gpu"
operator: "Exists"
effect: "NoSchedule"
```
#### Complete Example with Scheduling Configuration
Here's a complete example combining resource configuration with pod scheduling:
```yaml theme={"system"}
env:
- name: USER_CODE_PATH
value: /home/src/my-project
- name: ENV
value: production
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
nodeSelector:
instance-type: gpu
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: zone
operator: In
values:
- us-west-2a
- us-west-2b
tolerations:
- key: "gpu"
operator: "Equal"
value: "true"
effect: "NoSchedule"
```
**Note**: These pod-level scheduling configurations (`nodeSelector`, `affinity`, `tolerations`) are applied to the workspace pod, not the container. They control where Kubernetes schedules your workspace pods in the cluster.
## Share secrets across workspaces
To share secrets across different workspaces, you need to set the `SHARE_SECRETS_ACROSS_WORKSPACES` environment variable at the **cluster level** (not at the workspace level) and set the value to `1` to enable this feature. This will allow developers to share their secrets across different workspaces within a cluster.
**Note**: The `SHARE_SECRETS_ACROSS_WORKSPACES` environment variable must be set on the **main cluster container**, not on individual workspaces.
## Share secrets across workspaces
To share secrets across different workspaces developers need to create a custom environment variable
called `SHARE_SECRETS_ACROSS_WORKSPACES` then set the value to `1` to enable this feature. This will
allow developers to share their secrets across different workspaces within a cluster.
## Inherit environment variables
Configuring your Mage Pro workspaces to inherit the main clusterโs environment
variables is essential for efficiently configuring your workspace container.
The following options allow you to control how variables are shared within your environment.
To enable environment variable inheritance in workspaces, you must set the `INHERIT_ENV_VARS` environment variable at the **cluster level** (not at the workspace level):
To enable environment variable inheritance in workspaces, you must set:
```bash theme={"system"}
INHERIT_ENV_VARS=1
```
**Note**: The `INHERIT_ENV_VARS` environment variable must be set on the **main cluster container**, not on individual workspaces. This applies to **both Managed Cloud** and **Self-hosted Mage Pro**.
### Managed Cloud
The `USER_DEFINED_VARS` environment variable (set at the **cluster level**) lets you control which environment variables are inherited by workspaces.
This applies to **both Managed Cloud** and **Self-hosted Mage Pro**.
### Managed Cloud
The USER\_DEFINED\_VARS environment variable lets you control which environment variables are inherited by workspaces.
It accepts a comma-separated list of variable names, giving you flexibility to include only the configuration values you need across workspaces.
* If `USER_DEFINED_VARS` is **not set**: all environment variables will be inherited automatically.
* If `USER_DEFINED_VARS` is **set**: only the specified variables will be inherited.
**Example โ inherit specific variables:**
```bash theme={"system"}
USER_DEFINED_VARS=VAR1,VAR2
```
**Example โ inherit by pattern (wildcard/regex):**
```bash theme={"system"}
USER_DEFINED_VARS=DEV_ENV_*
```
This will inherit all environment variables matching the pattern (e.g., any starting with `DEV_ENV_*`).
### Self-hosted Mage Pro (Hybrid Cloud, Private Cloud, On-prem)
You **must** set `USER_DEFINED_VARS` at the **cluster level** to define which variables to inherit:
You **must** set `USER_DEFINED_VARS` to define which variables to inherit:
```bash theme={"system"}
USER_DEFINED_VARS=VAR1,VAR2
```
Wildcard/regex-style patterns are supported here as well.
### Auto-set ENV variable from workspace name
You can automatically set the `ENV` environment variable for each workspace based on its name using the `WORKSPACE_ENV_PATTERN` cluster-level environment variable. This eliminates the need to manually set `ENV` in each workspace's container configuration.
**Configuration:**
Set `WORKSPACE_ENV_PATTERN` at the **cluster level** (not at the workspace level):
```bash theme={"system"}
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
```
* `{workspace_name}`: Placeholder that will be replaced with the actual workspace name
* The pattern supports any string with the `{workspace_name}` placeholder
**How it works:**
* If `ENV` is not explicitly set in a workspace's container configuration, it will be automatically set using the pattern
* If `ENV` is explicitly set in the workspace's container configuration, that value takes precedence over the pattern
* The pattern is only applied when `WORKSPACE_ENV_PATTERN` is defined
**Examples:**
**Example 1: Simple prefix pattern**
```bash theme={"system"}
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
```
For a workspace named `my-workspace`, this sets `ENV=dev_my-workspace`.
**Example 2: Environment prefix with suffix**
```bash theme={"system"}
WORKSPACE_ENV_PATTERN=env-{workspace_name}-prod
```
For a workspace named `analytics`, this sets `ENV=env-analytics-prod`.
**Example 3: Just workspace name**
```bash theme={"system"}
WORKSPACE_ENV_PATTERN={workspace_name}
```
For a workspace named `production`, this sets `ENV=production`.
**Integration with environment-specific variable inheritance:**
When `ENV` is automatically set via `WORKSPACE_ENV_PATTERN`, it enables the environment-specific variable inheritance feature. You can then use:
```bash theme={"system"}
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
USER_DEFINED_VARS__ENV__dev_my-workspace=DEV_ENV_*
```
This automatically sets `ENV=dev_my-workspace` for a workspace named `my-workspace`, and that workspace will inherit all variables matching `DEV_ENV_*` from the cluster.
### Inherit variables by workspace environment
You can define inheritance rules per workspace environment using cluster-level environment variables with the `ENV` variable:
### Inherit variables by workspace environment (new)
You can define inheritance rules per workspace environment using the `ENV` variable:
```bash theme={"system"}
USER_DEFINED_VARS__ENV__{env_value}={pattern}
```
* `{env_value}`: value of the `ENV` variable in the workspace.
* `{pattern}`: wildcard/regex pattern for matching variables.
**Example:**
```bash theme={"system"}
USER_DEFINED_VARS__ENV__dev=DEV_ENV_*
```
Any workspace with `ENV=dev` inherits all variables matching `DEV_ENV_*`.
**Combined example with WORKSPACE\_ENV\_PATTERN:**
```bash theme={"system"}
# Automatically set ENV based on workspace name
WORKSPACE_ENV_PATTERN=dev_{workspace_name}
# Inherit variables based on the auto-set ENV value
USER_DEFINED_VARS__ENV__dev_workspace1=DEV_ENV_*
USER_DEFINED_VARS__ENV__dev_workspace2=PROD_ENV_*
```
In this example:
* A workspace named `workspace1` automatically gets `ENV=dev_workspace1` and inherits variables matching `DEV_ENV_*`
* A workspace named `workspace2` automatically gets `ENV=dev_workspace2` and inherits variables matching `PROD_ENV_*`
### Updating workspaces after environment variable changes
When you add or modify environment variables in your base cluster, existing workspaces will not automatically inherit these changes. To ensure your workspaces reflect the updated environment configuration, you must restart the affected workspaces.
To update a workspace:
1. Open the workspace settings by clicking into the workspace you want to update
2. Click the update button for your workspace
3. Click the dropdown arrow on the Kubernetes configuration section
4. Toggle the "update workspace settings" option to true
5. Click update
The container will take up to several minutes to update and restart with the new environment configuration.
**Note on container image during updates:**
When updating a workspace, if you don't explicitly specify the `image` field in your `container_config` YAML, the system will use the image from the main cluster pod configuration template, not the workspace's current image. This means:
* If your workspace currently uses a custom image (e.g., `my-custom-image:v1`) and you update the workspace without specifying `image` in `container_config`, the image may change to match the main cluster pod configuration (e.g., `mageai/mageai:latest`).
* To preserve your current workspace image during updates, explicitly include it in your `container_config`:
```yaml theme={"system"}
image: my-custom-image:v1
env:
- name: USER_CODE_PATH
value: /home/src/my-project
```
## Lifecycle Management Policies (Optional)
Mage Pro workspaces support various lifecycle management policies that control how your workspace behaves throughout its runtime. These policies include automated termination settings, pre-initialization scripts, and post-initialization procedures, giving you complete control over your workspace's behavior from startup to shutdown.
Lifecycle configuration is specified in the `lifecycle_config` field as a YAML object:
```yaml theme={"system"}
termination_policy:
enable_auto_termination: true
max_idle_seconds: 1800
pre_start_script_path: /path/to/script.sh
post_start:
command: ["echo", "Workspace started"]
hook_path: /path/to/hook.sh
```
### Termination Policy
The termination policy controls how your workspace shuts down when inactive.
**Configuration:**
* **`enable_auto_termination`** (boolean): Enable or disable automatic termination. Defaults to `false`.
* **`max_idle_seconds`** (integer): The number of seconds of inactivity before the workspace is automatically terminated. The idle time is measured in seconds, and you can set this duration based on your needs. For example:
* `1800` seconds = 30 minutes
* `3600` seconds = 1 hour
* `7200` seconds = 2 hours
**Example:**
```yaml theme={"system"}
termination_policy:
enable_auto_termination: true
max_idle_seconds: 1800
```
## Lifecycle management policies (optional)
Mage Pro workspaces support various lifecycle management policies that control how your workspace behaves throughout its runtime. These policies include automated termination settings, pre-initialization scripts, and post-initialization procedures, giving you complete control over your workspace's behavior from startup to shutdown.
### Termination Policy
The termination policy controls how your workspace shuts down when inactive. You can enable auto-termination, which will automatically shut down your workspace after a specified period of inactivity. The idle time is measured in seconds, and you can set this duration based on your needs. For example, you might set it to 1800 seconds (30 minutes) or 3600 seconds (1 hour).
### Pre-Start Scripts
Pre-start scripts run before your workspace container is initialized. These scripts must include a `get_custom_configs` method that returns a dictionary of configurations. This is where you can set up environment variables, modify container configurations, and prepare any necessary resources. The script executes before any workspace services start, making it ideal for initial setup and configuration tasks.
**Configuration:**
* **`pre_start_script_path`** (string): Path to the pre-start script. Can be:
* Absolute path: `/absolute/path/to/script.sh`
* Relative path: `scripts/pre_start.sh` (relative to the root project directory)
**Example:**
```yaml theme={"system"}
pre_start_script_path: /home/src/scripts/setup_workspace.sh
```
### Post-Start Scripts
Post-start scripts execute after requirements installation and before the workspace server starts. These scripts are executed via the `mage run-post-start-hooks` command and run with graceful error handlingโif a post-start hook fails, it logs a warning but does not prevent the container from starting. These scripts are typically used for starting services, performing health checks, or completing any additional setup tasks that depend on installed requirements but must run before the workspace server is fully started.
**Configuration:**
* **`post_start.command`** (string or array of strings): Command to execute. Can be specified as:
* **JSON array of strings**: `["echo", "Workspace started"]` or `["python", "script.py", "--arg", "value"]`
* **Shell command string**: `echo "Hello" && echo "World"` or `pip install playwright && playwright install --with-deps`
* **`post_start.hook_path`** (string): Path to a script file. Can be:
* Absolute path: `/absolute/path/to/hook.sh`
* Relative path: `scripts/post_start.sh` (relative to the root project directory)
**Note**: You can specify either `command` or `hook_path`, or both. If both are specified, the command will execute first, followed by the hook script. Post-start hooks execute after requirements installation but before the Mage server starts, ensuring they run early in the container startup process.
**Example 1: Post-start command as JSON array**
```yaml theme={"system"}
post_start:
command: ["echo", "Workspace started"]
```
**Example 2: Post-start command as shell string**
```yaml theme={"system"}
post_start:
command: echo "Hello" && echo "World"
```
**Example 3: Post-start command with multiple operations (shell string)**
```yaml theme={"system"}
post_start:
command: pip install playwright && playwright install --with-deps
```
**Example 4: Post-start hook script**
```yaml theme={"system"}
post_start:
hook_path: /home/src/scripts/post_start.sh
```
Where `post_start.sh` contains multiple commands:
```bash theme={"system"}
#!/bin/bash
echo "Starting service 1..."
service1 start
echo "Starting service 2..."
service2 start
```
**Example 5: Command and hook script (command runs first, then hook)**
```yaml theme={"system"}
post_start:
command: ["echo", "Workspace started"]
hook_path: /home/src/scripts/start_services.sh
```
**Important Notes:**
* Post-start hooks execute after requirements installation but before the Mage server starts
* If a post-start hook fails, it logs a warning but does not prevent the container from starting
* Shell command strings support operators like `&&`, `||`, etc.
* Hook scripts must be executable and located in a path accessible to the container
## Setting workspaces for self-hosted Mage Pro cluster
This section applies to **self-hosted Mage Pro clusters** only. These configurations are not applicable to Managed Cloud deployments.
### Workspace Configuration Environment Variables
These environment variables are set at the **cluster level** to configure default workspace behavior. They control how workspaces are created, what storage they use, and how they're exposed via ingress.
#### Ingress Configuration
* **`WORKSPACE_INGRESS_NAME`** (optional): Name of the Kubernetes ingress resource to use for workspaces. Defaults to the cluster UUID (`MAGE_CLUSTER_UUID`). If not set, workspaces will be accessible through the service directly rather than via ingress.
* **`WORKSPACE_INGRESS_TYPE`** (optional): Type of ingress controller to use. Supported values:
* `nginx` (default): Use NGINX ingress controller
* `alb`: Use AWS Application Load Balancer ingress
* `istio`: Use Istio ingress gateway
Default: `nginx`
* **`WORKSPACE_INGRESS_PATH_TYPE`** (optional): Path type for ingress paths. Supported values:
* `ImplementationSpecific` (default): Path matching is implementation-specific
* `Prefix`: Path must match as a prefix
Default: `ImplementationSpecific`
#### Storage Configuration
You can configure workspace storage in one of two ways:
##### Option 1: Individual PVCs per Workspace (using WORKSPACE\_STORAGE\_CLASS)
* **`WORKSPACE_STORAGE_CLASS`** (optional): Kubernetes storage class name to use for workspace persistent volumes. When configured, each workspace will create its own PersistentVolumeClaim (PVC) using this storage class.
Default: `efs-sc-us-west-2`
**Use case**: When you want each workspace to have its own isolated storage volume.
##### Option 2: Shared PVC across Workspaces (using WORKSPACE\_PVC\_NAME + WORKSPACE\_PVC\_SUBPATH\_PREFIX)
* **`WORKSPACE_PVC_NAME`** (optional): Name of an existing PersistentVolumeClaim (PVC) to share across all workspaces. When set, all workspaces will use this shared PVC instead of creating individual PVCs.
* **`WORKSPACE_PVC_SUBPATH_PREFIX`** (optional): Prefix for subpaths when using a shared PVC. Each workspace will use a separate subfolder under this prefix to isolate its data. **Required** when `WORKSPACE_PVC_NAME` is set.
**Example**: If `WORKSPACE_PVC_SUBPATH_PREFIX=workspaces`, a workspace named `my-workspace` will use the subpath `workspaces/my-workspace`.
**Use case**: When you want to share storage across multiple workspaces and manage storage more efficiently.
**Important**: Configure either `WORKSPACE_STORAGE_CLASS` (for individual PVCs) or `WORKSPACE_PVC_NAME` + `WORKSPACE_PVC_SUBPATH_PREFIX` (for shared PVC), but not both. If both are configured, the shared PVC option takes precedence.
#### Pod Configuration
* **`INHERIT_POD_ANNOTATIONS`** (optional, boolean): If `true`, workspace pods will inherit annotations from the main cluster pod. This is useful for passing through annotations like Prometheus scraping labels, service mesh configuration, etc.
Default: `false`
#### Default Resource Configuration
These environment variables set default resource requests and limits for Kubernetes workspace containers. They are applied when a workspace's `container_config` does not specify resources (or when specific fields like `requests.cpu` or `limits.memory` are omitted). User-specified values in `container_config` always take precedence.
| Variable | Purpose | Example Value |
| -------------------------------------------- | ----------------------------------------------- | ------------- |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU` | Default CPU request for workspace containers | `500m` |
| `WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY` | Default memory request for workspace containers | `2Gi` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU` | Default CPU limit for workspace containers | `4` |
| `WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY` | Default memory limit for workspace containers | `8Gi` |
**Example:**
```bash theme={"system"}
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_CPU="500m"
export WORKSPACE_DEFAULT_RESOURCE_REQUESTS_MEMORY="2Gi"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_CPU="4"
export WORKSPACE_DEFAULT_RESOURCE_LIMITS_MEMORY="8Gi"
```
#### Default Workspace Config File
* **`WORKSPACE_DEFAULT_CONFIG_PATH`** (optional): Path to a YAML file that defines default workspace configuration applied when creating or updating workspaces. The file is merged with user-defined config based on the `enforce` flag. The file is read on the **control-plane** only; it is not mounted into workspace pods.
**Helm chart (recommended):** When deploying **Mage Pro** with the [Mage Helm chart](https://github.com/mage-ai/helm-charts), configure the default via **`workspaceConfig`** in `values.yaml`. The chart creates a ConfigMap from your config, mounts it on the control-plane, and sets `WORKSPACE_DEFAULT_CONFIG_PATH` automatically. Do not set the env var manually when using the chart. See [Using Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) for details.
**Supported fields** (top-level in the YAML file):
| Field | Type | Description |
| ---------------------- | --------------------- | ------------------------------------------------------------------------------------------ |
| `container_config` | YAML string or object | Default container configuration (image, resources, env, volumeMounts, etc.). |
| `service_account_name` | string | Default Kubernetes service account for workspace pods. |
| `lifecycle_config` | object | Default lifecycle configuration (termination policy, pre-start scripts, post-start hooks). |
**Policy fields:**
| Field | Description |
| -------------- | ---------------------------------------------------------------------------------------------------------------- |
| `enabled` | If `false`, the policy is not applied. |
| `enforce` | Controls who wins when both the default file and the user set the same key. See **Priority when merging** below. |
| *(other keys)* | `container_config`, `service_account_name`, `lifecycle_config` (see table above). |
**Priority when merging default file with user config**
The default file is merged with the workspace config supplied by the user (UI or API). The **effective priority** (who wins for a given key) depends on `enforce`:
| `enforce` | Effective priority | Result |
| ----------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`true`** | **Default file is highest.** Default file โ user config. | Values from the file override user-defined values. Use this to enforce cluster-wide workspace defaults (e.g. resources, service account) that users cannot override. |
| **`false`** | **User is highest.** User config โ default file (fills gaps only). | User config overrides the default file for the same keys. The file only fills in keys the user did not set. Use this to provide optional defaults while allowing per-workspace overrides. |
**Top-level format**:
```yaml theme={"system"}
enabled: true # Set to false to disable this policy entirely
enforce: false # User wins on conflicts; file only fills in missing keys
container_config:
resources:
limits:
cpu: "2"
memory: "4Gi"
requests:
cpu: "1"
memory: "2Gi"
nodeSelector:
node.kubernetes.io/instance-type: "m5.large"
tolerations:
- key: "dedicated"
operator: "Equal"
value: "mage"
effect: "NoSchedule"
lifecycle_config:
post_start:
command: echo "Hello World"
service_account_name: workspace-sa
```
The config file must exist at the specified path on the **control-plane** pod. Use a ConfigMap or volume mount to make the file available.
#### Example Configuration
**Example 1: Individual PVCs per workspace**
```bash theme={"system"}
# Ingress configuration
export WORKSPACE_INGRESS_NAME=my-cluster-ingress
export WORKSPACE_INGRESS_TYPE=nginx
export WORKSPACE_INGRESS_PATH_TYPE=Prefix
# Storage configuration - each workspace gets its own PVC
export WORKSPACE_STORAGE_CLASS=fast-ssd
# Pod configuration
export INHERIT_POD_ANNOTATIONS=true
```
**Example 2: Shared PVC across workspaces**
```bash theme={"system"}
# Ingress configuration
export WORKSPACE_INGRESS_NAME=my-cluster-ingress
export WORKSPACE_INGRESS_TYPE=nginx
export WORKSPACE_INGRESS_PATH_TYPE=Prefix
# Storage configuration - all workspaces share one PVC with separate subfolders
export WORKSPACE_PVC_NAME=shared-workspace-storage
export WORKSPACE_PVC_SUBPATH_PREFIX=workspaces
# Pod configuration
export INHERIT_POD_ANNOTATIONS=true
```
**Note**: These environment variables are set on the **main cluster container**, not on individual workspaces. They control the default behavior when creating new workspaces. Individual workspaces can override some of these settings (like storage class) in their own configuration.
### Docker Workspace Configuration
Docker workspaces support additional configuration options specific to Docker containers:
#### Docker Workspace Environment Variables
These environment variables are set at the **cluster level** to configure default Docker workspace behavior:
* **`DOCKER_HOST`** (optional): Docker daemon host. Defaults to `unix://var/run/docker.sock`.
* **`DOCKER_TLS_VERIFY`** (optional): Enable TLS verification for Docker connection. Set to `1` to enable, `0` to disable. Defaults to `0`.
* **`DOCKER_CERT_PATH`** (optional): Path to TLS certificates for Docker connection.
* **`DOCKER_SWARM_MODE`** (optional): Enable or disable Docker Swarm mode. Set to `1` to enable, `0` to disable. Defaults to `0`.
**Note**: These environment variables are set on the **main cluster container**, not on individual workspaces. They control the default behavior when creating new Docker workspaces. Individual workspaces can override these settings in their own configuration.
#### Docker Container Configuration (YAML)
Similar to Kubernetes, Docker workspaces support a `container_config` YAML string:
**Example Docker Workspace Configuration:**
```yaml theme={"system"}
image: mage-pro:version
volumes:
/host/path:
bind: /home/src
mode: rw
environment:
TEST_ENV_VAR: "1"
public_url: mage.example.com/test1
```
**Note**: For Docker workspaces, the `container_config` field is the primary way to configure image, volumes, and environment. The system will parse this YAML string and extract:
* `image`: Docker image to use
* `volumes`: Volume mounts configuration
* `environment`: Environment variables
* `public_url`: Public URL for the workspace
### Post-Start Scripts
Post-start scripts execute after your workspace is fully initialized and running. You can specify either a direct command or provide a path to a script for more complex procedures. These scripts are typically used for starting services, performing health checks, or completing any additional setup tasks that require the workspace to be fully operational.
## Global Roles
Users can be assigned global roles that apply across all workspaces within your Mage Pro environment. Global role assignment is managed by project administrators through Users tab in left navigation menu located in the main project interface.
## Workspace-Specific Roles
Individual workspaces can have unique role settings. Workspace owners can:
* Assign user roles specific to their workspace
* Modify existing role assignments
* Remove user access to their workspace
Understanding and properly configuring your Mage Pro workspaces is essential for efficient data pipeline development and management. With the right combination of resource allocation, lifecycle policies, and user permissions, you can create a stable and secure environment tailored to your project's needs.
# Configure Git
Source: https://docs.mage.ai/guides/git/configure
This page will walk you through the process of authenticating with SSH or HTTPS and configuring Git for use with Mage Git Sync.
If you're wondering which Git resource to use in Mage, see [this](docs/getting-started/setting-up-git) page for a high-level overview of Git functionality. If you'd like to use Mage's GitHub integration (recommended for GitHub users), there's a different method of configuration. See [this](/production/data-sync/github) page for more information.
If you've decided on a method that uses HTTPS or SSH authentication (Git Sync or Git Integration), this page will walk you through configuring one of those methods.
## Configure Git settings
By using the Git integration through Mage, Mage will automatically set your local repo as a safe directory (i.e. `git config --global --add safe.directory `)
To use the Mage Git integration, you will need to first configure Git settings. The following instructions are tailored to Github, but the Git integration can work with other Git providers as well, such as GitLab, BitBucket, or Azure DevOps:
**SSH keys with passphrase are not supported**
The Git integration currently does not support repositories that require an SSH key with a passphrase.
You can, however, use the Mage terminal and the git CLI to perform your git actions.
You'll still need to generate your SSH keys as normal. You can start a ssh-agent in the terminal so that
you don't have to enter your SSH key passphrase every time you perform an action.
`eval $(ssh-agent -s) && ssh-add ~/.ssh/id_rsa`
Ensure that `.preferences.yaml` is added to your repo's `.gitignore` file so that your Git settings aren't pushed to Github.
Navigate to the Wizard icon in the top right, then select *Settings* to open your Mage settings.
There should be an option for *Git settings* on the left side of the settings modal.
Change the authentication type to SSH. The interface will change for you to input a remote repo url.
If you haven't already, configure your SSH keyโย this will allow your instance to access GitHub. Our [instructions](/development/git/configure#generate-ssh-token#generate-ssh-token) can help if you're unfamiliar with the process.
* `Remote repo url`: Your Github repository's remote url. Use the SSH link instead of HTTPS link.
* `Local directory path`: The directory in your machine that will be synced with the Github repository. If this field is omitted, it will default to the path determined by Python's `os.getcwd()`. You can check what your current working directory is in a Scratchpad block.
Filling out these fields will allow Mage to recreate your git and SSH settings if they are deleted (i.e. if your container restarts):
If you are using a AWS CodeCommit repository, you will need to set the username to the SSH Key ID of the SSH public key added to your AWS IAM user.
* `Username`: Your git username. If you push to your remote repo, this username will show up next to your commits.
* `Email`: Your git email. Used for the same purpose as your username.
* `SSH public key in base64`, `SSH public key in base64`: Enter your SSH public and private key encoded in base64. You can follow the instructions in the field description if you want to Mage to save your SSH keys.
Click "Save repository settings" to complete the setup.
Ensure that `.preferences.yaml` is added to your repo's `.gitignore` file so that your Git settings aren't pushed to Github.
Navigate to the Wizard icon in the top right, then select *Settings* to open your Mage settings.
There should be an option for *Git settings* on the left side of the settings modal.
* [GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
* [GitLab](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token)
* `Remote repo url`: Your Github repository's remote url. Use the SSH link instead of HTTPS link.
* `Local directory path`: The directory in your machine that will be synced with the Github repository. If this field is omitted, it will default to the path determined by Python's `os.getcwd()`. You can check what your current working directory is in a Scratchpad block.
* `Username`: Your git username. If you push to your remote repo, this username will show up next to your commits.
* `Email`: Your git email. Used for the same purpose as your username.
* `Access token`: Enter the Git access token that you created in step 4.
Click "Save repository settings" to complete the setup.
### Generate SSH Token
If you're using a method that requires an SSH token in Mage, this section will walk you through generating one. If you're using a method that requires an HTTPS token, you can skip this section.
Follow GitHub's guide [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/checking-for-existing-ssh-keys) to check for an existing SSH key.
If no key exists, generate a new key using this [guide](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
You may skip this step if you already have a key.
This step allows your Git provider to recognize your SSH key and communicate.
* [GitHub Docs](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
* [GitLab Docs](https://docs.gitlab.com/ee/user/ssh.html#add-an-ssh-key-to-your-gitlab-account)
* [GitHub Docs](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/testing-your-ssh-connection)
* [GitLab Docs](https://docs.gitlab.com/ee/user/ssh.html#verify-that-you-can-connect)
The first time you connect using your Git provider, you'll get a message about verifying authenticity and adding the provider to your known hosts.
For example:
```txt theme={"system"}
The authenticity of host 'gitlab.example.com (35.231.145.151)' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gitlab.example.com' (ECDSA) to the list of known hosts.
```
This is normal and expected. Type `yes` and press enter to continue.
# ETL pipeline tutorial
Source: https://docs.mage.ai/guides/load-api-data
Build a data pipeline that loads restaurant data, transforms it, then exports it to a DuckDB database. ๐ฆ
## Overview
In this tutorial, we'll create a data pipeline with the following steps:
We'll use a Python block to load data from an online endpointโ a CSV file containing restaurant user transactions. We'll also run a test to make sure the data is clean.
We'll use a Python block to transform the data by cleaning the columns and creating a new column, `number of meals`, that counts the number of meals for each user.
Finally, we'll write the transformed data to a local DuckDB table.
If you haven't created a Mage project before, follow the [setup guide](/getting-started/setup) before starting this tutorial.
## Quickstart
Want to dive in? Simply run the following command to clone a pre-built repo:
```bash Mac/Linux theme={"system"}
git clone https://github.com/mage-ai/etl-demo mage-etl-demo \
&& cd mage-etl-demo \
&& cp dev.env .env && rm dev.env \
&& docker compose up
```
```bash Windows theme={"system"}
git clone https://github.com/mage-ai/etl-demo mage-etl-demo
cd mage-etl-demo
cp dev.env .env
rm dev.env
docker compose up
```
Then navigate to `https://localhost:6789` in your browser to see the pipeline in action!
## Tutorial
Open the pipelines list page (`/pipelines`). This is the default page when navigating to Mage in your web browser. If you have Mage running, just click [here](https://localhost:6789/pipelines).
In the top left corner, select the button labeled *+ New*, then select the option labeled *Standard (batch)* to create a new pipeline.
In the left vertical navigation, click the last link labeled *Pipeline settings*. Change the pipeline's name to `ETL demo`. Click the button labeled *Save pipeline settings*.
In the left vertical navigation, click the 1st link labeled *Edit pipeline*. Click the button labeled *+ Data loader* then hover over *Python*, and click the option labeled *API*.
A dialog menu will appear. Change the block name to `load data`. Click the button labeled *Save and add block*. Paste the following code in the data loader block:
```python theme={"system"}
import io
import pandas as pd
import requests
@data_loader
def load_data_from_api(*args, **kwargs):
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
@test
def test_row_count(df, *args) -> None:
assert len(df.index) >= 1000, 'The data does not have enough rows.'
```
Run the block by clicking the play icon button in the top right corner of the data loader block or one of the following keyboard shortcuts:
* `โ + Enter`
* `Control + Enter`
* `Shift + Enter`
After you run the block (โ + Enter), you'll see a sample of the data that was loaded.
Let's transform the data in 2 ways:
* Add a column that counts the number of meals for each user.
* Clean the column names to properly store in a PostgreSQL database.
Follow these steps:
Click the button labeled *+ Transformer*, then hover over *Python*, and click the option labeled *Generic (no template)*. Paste the following code in the transformer block:
```python theme={"system"}
def number_of_rows_per_key(df, key, column_name):
data = df.groupby(key)[key].agg(['count'])
data.columns = [column_name]
return data
def clean_column(column_name):
return column_name.lower().replace(' ', '_')
@transformer
def transform(df, *args, **kwargs):
# Add number of meals for each user
df_new_column = number_of_rows_per_key(df, 'user ID', 'number of meals')
df = df.join(df_new_column, on='user ID')
# Clean column names
df.columns = [clean_column(col) for col in df.columns]
return df.iloc[:100]
@test
def test_number_of_columns(df, *args) -> None:
assert len(df.columns) >= 11, 'There needs to be at least 11 columns.'
```
Run the block by clicking the play icon button in the top right corner of the data loader block or press 1 of the following keyboard shortcuts:
* `โ + Enter`
* `Control + Enter`
* `Shift + Enter`
After you run the block (โ + Enter), you'll see a sample of the data that was transformed.
Let's export the data to a local DuckDB database! Create a new `Data Exporter` SQL block and change the connection type to be `DuckDB`.
Next, change the `Table name` to be `magic_duck.restaurant_user_transactions` and make sure the content of the block is:
```sql theme={"system"}
SELECT * FROM {{ df_1 }}
```
This will take our data from the previous block and export it to a local DuckDB tableโย `magic_duck.restaurant_user_transactions`! ๐
After you run the block (โ + Enter), you'll see a sample of the data that was exported.
You should also see some logs indicating the export was completed, e.g.
```bash theme={"system"}
DuckDB initialized
โโ Opening connection to DuckDB...DONE
Exporting data from upstream block floral_star to restaurant_user_transactions.
Type: integer
Type: integer
Type: string
Type: integer
Type: string
Type: integer
Type: string
Type: integer
Type: integer
Type: integer
```
**๐ Congratulations!**
You've successfully built an end-to-end ETL pipeline that loaded data,
transformed it, and exported it to a database.
Now you're ready to raid the dungeons and find magical treasures with your new powers!
If you have more questions or ideas, get real-time help in our live support
[Slack channel](https://www.mage.ai/chat).
# Mage Guides
Source: https://docs.mage.ai/guides/overview
Mage guides for getting started, building pipelines, and more.
This section of our documentation contains helpful guides and tutorials to getting started with Mage.
We welcome contributions from our community. If you would like to contribute your own guides or content, we encourage that as well! It's written entirely in Markdown and *very* user friendly.
At the bottom of every doc, there's a link to edit the page on GitHub. Click that link and you'll be taken to the Markdown file for that page. You can edit the file directly in your browser and submit a pull request.
# Importing pipelines
Source: https://docs.mage.ai/guides/pipelines/importing-pipelines
Import a pipeline zip file into your project.
Requires version `0.9.63` or greater. This feature is supported in single projects.
It may not work in setups using the multi-project platform.
## Overview
If you have a zip file with a pipeline's `metadata.yaml` configuration file and block files,
you can import that pipeline into your project. This could be useful if you want to import
a pipeline from a different project.
## Supported block file types
1. Custom
2. Data exporter
3. Data loader
4. Markdown
5. Scratchpad
6. Sensor
7. Transformer
## Unsupported block file types
Including these types of block files in your pipeline zip file may cause errors when trying
to import it.
1. Callback
2. Chart
3. Conditional
4. dbt
5. Extension
## Downloading a pipeline
You can download a pipeline from your project as a zip file by going to the Pipelines
Dashboard (`/pipelines`), right-clicking one of the pipeline rows, and selecting
"Download (keep folder structure)" or "Download (without folder structure)". You can use
this pipeline zip file to import the pipeline into another Mage project.
Note that the unsupported block file types listed above may not be included in the downloaded
pipeline zip file.
## Required files and folder structure of imported pipeline zip file
At a minimum, the pipeline zip file (which has a `.zip` file extension) should contain a
root folder and a pipeline configuration file (i.e. `metadata.yaml`). There could be zero block
files if the pipeline has no blocks.
### Single folder structure
* Root folder
* Pipeline config file (`metadata.yaml`)
* Block file 1 (e.g. `block1.py`)
* Block file 2 (e.g. `block2.py`)
* Additional block files...
### Subfolder structure
* Root folder
* Pipelines folder (i.e. `pipelines`)
* Pipeline folder (this is the pipeline UUID)
* Pipeline config file (`metadata.yaml`)
* Block type 1 folder (e.g. `data_loaders`, `transformers`, `custom`, etc.)
* Block file 1 for block type 1 (e.g. `data_loader_block_file1.py`)
* Block file 2 for block type 1 (e.g. `data_loader_block_file2.py`)
* Additional block files for block type 1...
* Block type 2 folder (e.g. `data_exporters`, `sensors`, `scratchpads`, etc.)
* Block file 1 for block type 2 (e.g. `transformer_block_file1.py`)
* Block file 2 for block type 2 (e.g. `transformer_block_file2.py`)
* Additional block files for block type 2...
* Additional block folders and files...
## How to import a pipeline
1. Go to the Pipelines Dashboard (`/pipelines`).
2. Click on the `+ New` button.
3. Select "Import pipeline zip" from the dropdown menu.
4. Toggle the "Overwrite pipeline files" setting on if you want to have any conflicting pipelines
replaced and conflicting block files overwritten (e.g. if the pipeline with the same blocks
already exists in your project). If this setting is left disabled, a new pipeline and new
block files will be created if the pipeline/blocks already exist.
5. Add a pipeline zip file by clicking the file drop area to select a file or by
dragging and dropping a file into the file drop area.
6. The pipeline import should automatically start.
# ActiveMQ
Source: https://docs.mage.ai/guides/streaming/destinations/activemq
## Basic config
```yaml theme={"system"}
connector_type: activemq
connection_host: 'localhost'
connection_port: 61613
queue_name: 'queue_name'
username: 'admin'
password: 'admin'
```
# Amazon S3
Source: https://docs.mage.ai/guides/streaming/destinations/amazon-s3
## Basic config
```yaml theme={"system"}
connector_type: amazon_s3
bucket: bucket
prefix: prefix
file_type: parquet
buffer_size_mb: 5
buffer_timeout_seconds: 300
```
* `bucket`: The S3 bucket you want to use to store the data.
* `prefix`: The S3 path prefix. The whole S3 path will be `s3://{bucket}/{prefix}`.
* `file_type`: `parquet`, `csv`, or `delta` (for Delta Lake tables)
* `file_type`: `parquet` or `csv`
* `buffer_size_mb` and `buffer_timeout_seconds`: Mage puts messages in a buffer before uploading to S3.
You can configure the size and timeout of the buffer to control the file size and the delay.
You may see a `region` parameter in the default config template, it can be ignored. If you specify any valid AWS region names or null, you can still access to S3 buckets in other regions. However, if you do include the parameter value for `region`, it must be a valid region name for AWS or null so it won't error out.
## Configure time-based partition
```yaml theme={"system"}
date_partition_format: "%Y%m%dT%H"
```
If you want to store the S3 data in time-based partition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.
## Delta Lake configuration
**Note:** Delta Lake format (`file_type: delta`) is a Mage Pro only feature.
When using `file_type: delta`, you can write data to Delta Lake tables in Amazon S3.
```yaml theme={"system"}
connector_type: amazon_s3
bucket: my-bucket
prefix: my-prefix
file_type: delta
table_uri: s3://my-bucket/path/to/delta-table # Optional: if not provided, constructed from bucket/prefix
mode: append # 'append' or 'overwrite'
buffer_size_mb: 5
buffer_timeout_seconds: 300
aws_access_key_id: YOUR_ACCESS_KEY # Optional: falls back to environment variables or IAM roles
aws_secret_access_key: YOUR_SECRET_KEY # Optional: falls back to environment variables or IAM roles
aws_region: us-west-2 # Optional: defaults to us-west-2
```
* `table_uri`: (Optional) Full URI for the Delta Lake table (e.g., `s3://bucket/path/to/table`). If not provided, the URI will be automatically constructed from `bucket` and `prefix`.
* `mode`: Write mode for Delta Lake tables. Valid values:
* `append`: Adds new rows to the existing table (default)
* `overwrite`: Replaces all existing rows in the table
* `aws_access_key_id`: (Optional) AWS access key ID. If not provided, will fall back to the `AWS_ACCESS_KEY_ID` environment variable or the IAM role attached to the instance. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
* `aws_secret_access_key`: (Optional) AWS secret access key. If not provided, will fall back to the `AWS_SECRET_ACCESS_KEY` environment variable or the IAM role attached to the instance. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
* `aws_region`: (Optional) AWS region. If not provided, will fall back to the `AWS_REGION` environment variable or default to `us-west-2`.
**Note:** When using Delta Lake format, time-based partitioning (`date_partition_format`) is not applied as Delta Lake manages its own internal file structure.
If you want to store the S3 data in time-based paratition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.
## Authentication
Here are the options to authenticate with the AWS S3 bucket.
1. Add the following keys and values to your environment variables
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`
2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate.
You can grant the ECS task permissions to access other AWS services by attaching IAM policies
to this ECS task execution role.
**Security Note:** Storing credentials in configuration files can be insecure. Prefer using environment variables or IAM roles when possible, especially for production deployments. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
# Azure Data Lake
Source: https://docs.mage.ai/guides/streaming/destinations/azure_data_lake
## Basic config
```yaml theme={"system"}
connector_type: azure_data_lake
table_uri: abfs://container@storagename.dfs.core.windows.net/
account_name: account_name
access_key: access_key
mode: append
file_type: delta
```
* `table_uri`: Azure Data Lake container location, or local path, where the Delta Table will be written.
* `account_name`: Azure Storage Account name
* `access_key`: Azure Storage Access key
* `mode`: write mode. 'append' or 'overwrite' are accepted. default is 'append'
* `file_type`: File type to insert in Azure Data Lake. Only 'delta' is supported.
# BigQuery
Source: https://docs.mage.ai/guides/streaming/destinations/bigquery
Stream data to Google BigQuery using the Storage Write API, with optional dead-letter queue and metadata interpolation for CDC.
The BigQuery streaming destination uses the **Generic IO** sink. Writes go through the **BigQuery Storage Write API** (default stream, Arrow format) for better throughput and lower cost. The target table must already exist.
***
## Basic config
```yaml theme={"system"}
connector_type: bigquery
profile: default # profile in io_config.yaml
config:
table_id: project_id.dataset_id.table_id # or dataset_id.table_id (uses profile project)
if_exists: "append"
# dead_letter is optional and top-level (outside config)
# dead_letter:
# destination_type: file # "file" or "table"
# enabled: true
# table_suffix: "_dlq"
# table_name: null
# file_path: null
```
* **`profile`** selects which credentials and connection settings to use. It is the name of a **profile** (section) in your projectโs `io_config.yaml`. The sink loads that profile via the same [IO config](/development/io_config_setup) used elsewhere in Mage (e.g. `ConfigFileLoader(repo_path/io_config.yaml, profile)`). Use the `default` profile unless you have multiple BigQuery connections (e.g. dev vs prod); then define separate profiles (e.g. `default`, `prod`) in `io_config.yaml` and set `profile: prod` in the sink.
* **`config`** is a key-value map passed to the BigQuery **export** call. It controls *where* and *how* data is written (e.g. `table_id`, `if_exists`, `database`, `overwrite_types`). It is merged with defaults (e.g. `if_exists: append`) and supports [metadata interpolation](#metadata-interpolation) so you can use values from each message (e.g. `table_id: "{schema}.{table}"`). **`config` is separate from credentials**: credentials come from the chosen **profile**; **config** only holds export options like table and column types.
## How it works
* **Storage Write API**: Streaming inserts use the BigQuery Storage Write API (not the older streaming insert API). Requires `google-cloud-bigquery-storage` and PyArrow. If the table does not exist, the pipeline will not create it; create the table beforehand.
* **Schema**: Only DataFrame columns that exist in the BigQuery table schema are written. Extra columns are dropped. Column types are cast to match the table (STRING, INT64, TIMESTAMP, RECORD, etc.).
* **CDC columns**: The sink automatically maps `_mage_created_at`, `_mage_updated_at`, and `_mage_deleted_at` to BigQuery `TIMESTAMP` type when present in the data.
* **Required (NOT NULL) fields**: If the table has REQUIRED columns and any row has nulls in those columns (including nested RECORD fields), the write fails and failed rows can be sent to a dead-letter queue (see below).
## Optional config
```yaml theme={"system"}
config:
table_id: project_id.dataset_id.table_id
if_exists: "append"
database: project_id # override project when table_id is dataset.table
# Override column types (e.g. for _mage_* or custom columns)
overwrite_types:
_mage_created_at: TIMESTAMP
my_column: STRING
# Dead-letter queue (top-level, outside config). Schema: DeadLetterConfig
dead_letter:
destination_type: file # "file" (default) or "table"
enabled: true
table_suffix: "_dlq" # for destination_type=table: append to main table_id
table_name: null # optional: full table id (overrides table_suffix)
file_path: null # for destination_type=file: path for JSONL file
```
## Config parameters
| Parameter | Type | Description | Default |
| ------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `table_id` | string | BigQuery table identifier. Use `project_id.dataset_id.table_id` or `dataset_id.table_id` (project from profile or `database`). Supports [metadata interpolation](#metadata-interpolation), e.g. `"{schema}.{table}"` for CDC. | *(required)* |
| `database` | string | Default project when `table_id` is two-part (`dataset_id.table_id`). Overrides the profile project if set. | profile project |
| `if_exists` | string | Behavior when the table already exists: `append` (add rows), `replace` (truncate/overwrite existing rows; preserves table schema), or `fail` (throw an error). | `append` |
| `overwrite_types` | object | Map of column name โ BigQuery type (e.g. `STRING`, `INT64`, `TIMESTAMP`, `RECORD`). Used to cast columns before write. Useful for `_mage_*` or custom columns. | โ |
| `unique_constraints` | list (or string) | List of column names for upsert/merge. When set with `unique_conflict_method`, exports as MERGE instead of append. Supports interpolation, e.g. `"{key_columns}"` from [PostgreSQL CDC](/guides/streaming/sources/postgres-cdc). | โ |
| `unique_conflict_method` | string | For upsert: e.g. `UPDATE` to update existing rows on conflict. Used only when `unique_constraints` is set. | โ |
The sink automatically sets `streaming_inserts: true` so writes use the BigQuery Storage Write API. The target table must already exist (the sink does not create it).
## Dead-letter queue
When a batch fails (e.g. required field has null, or Storage Write API returns row errors), failed rows can be written to a dead-letter destination. **Dead-letter is top-level** (sibling to `config`), not inside `config`. Schema: **DeadLetterConfig**:
| Field | Description |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `destination_type` | Where to write failed rows: `file` (default) = JSONL file; `table` = BigQuery table (same project/dataset as main table). |
| `enabled` | If `true`, write failed rows to the DLQ (per `destination_type`). If `false`, do not write to DLQ. Default: `true`. |
| `table_suffix` | When `destination_type` is `table`: suffix applied to the main `table_id` to form the DLQ table (e.g. `_dlq` โ `project.dataset.table_dlq`). Ignored if `table_name` is set. Default: `_dlq`. |
| `table_name` | When `destination_type` is `table`: full table identifier for the DLQ table (e.g. `project_id.dataset_id.dlq_table_id`). Overrides `table_suffix` when set. |
| `file_path` | When `destination_type` is `file`: path for the JSONL file. Default: under the pipeline variables dir (e.g. `/streaming_dlq.jsonl`). |
When `dead_letter` is omitted entirely, failed rows are written to the default file path (same as `destination_type: file` with default `file_path`).
**DLQ record format** (each failed row becomes one record):
* `original_record`: the failed row as a JSON object (file) or JSON string column (table).
* `failure_reason`: error message.
* `failure_timestamp`: ISO timestamp (UTC).
* `connector_type`: e.g. `bigquery`.
* `row_errors`: optional list of per-row error details (index, code, message) when the API provides them.
When writing to a **table**, the DLQ table must have columns compatible with these fields (e.g. `original_record` STRING/JSON, `failure_reason` STRING, `failure_timestamp` STRING/TIMESTAMP, `connector_type` STRING, `row_errors` STRING/JSON). When writing to a **file**, each line is one JSON object with the fields above.
If failed rows are successfully written to the dead-letter destination, the pipeline continues. If the DLQ write fails or is disabled, the pipeline raises the error so you can alert or retry.
## Metadata interpolation
You can use message metadata in the config (e.g. schema/table per source). Keys in `config` are interpolated with `{metadata_key}`. Example: `table_id: "{schema}.{table}"` using `metadata.schema` and `metadata.table` from each message.
This pattern works particularly well with **CDC** sources such as [PostgreSQL CDC](/guides/streaming/sources/postgres-cdc), where each event carries `schema`, `table`, and primary key metadata. For example, you can route changes from many source tables into matching BigQuery tables by setting:
```yaml theme={"system"}
config:
table_id: "{schema}.{table}"
```
profile: default # profile in the io\_config.yaml
config:
table\_id: table\_id
if\_exists: "append"
````
## Optional fields
```yaml
database: database # Override the database
````
# ClickHouse
Source: https://docs.mage.ai/guides/streaming/destinations/clickhouse
## Basic config
```yaml theme={"system"}
connector_type: clickhouse
profile: default # profile in the io_config.yaml
config:
table_name: table_name
```
Enter the DB connection credentials in [`io_config.yaml`](/development/io_config) file and specify the profile in the config.
## Optional fields
```yaml theme={"system"}
database: database # Override the database
create_table_statement: true/false # Whether to run create table statement
```
# DuckDB
Source: https://docs.mage.ai/guides/streaming/destinations/duckdb
## Basic config
```yaml theme={"system"}
connector_type: duckdb
profile: default # profile in the io_config.yaml
config:
table_name: table_name
```
## Optional fields
```yaml theme={"system"}
schema_name: schema_name # Override the schema name. Default schema is "main"
```
# Dummy
Source: https://docs.mage.ai/guides/streaming/destinations/dummy
Dummy sink will print the message optionally and discard the message. This dummy sink will be useful when users want to trigger other pipelines or 3rd party services using the ingested data in transformer.
## Config
```yaml theme={"system"}
connector_type: dummy
print_msg: true
```
# Elasticsearch
Source: https://docs.mage.ai/guides/streaming/destinations/elasticsearch
## Basic config
```yaml theme={"system"}
connector_type: elasticsearch
host: http://localhost
index_name: elastic_index
# When using Data streams this must be set to create as they are Append-only indexes
# _op_type: None / index, create or update
# _source: None (Default '_doc') / Specify the source field
# Whether to have update for the documents, need to mention the unique key field in the document as _id
# _id: _key
# # Whether to verify SSL certificates to authenticate.
# verify_certs: true
# ca_cert: "certificate.cert"
```
## Authentication
```yaml theme={"system"}
verify_certs: true
api_key: api_key_for_elastic_search
ca_cert: "certificate.cert"
```
* **verify\_certs**: Whether to verify SSL certificates to authenticate.
# Google Cloud PubSub
Source: https://docs.mage.ai/guides/streaming/destinations/google-cloud-pubsub
## Basic config
```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
pubsub_emulator_host: null # "host.docker.internal:8085"
path_to_credentials_json_file: "./google_credentials.json"
```
The `pubsub_emulator_host` parameter is optional, which could be used when you need to test
the Google Cloud PubSub service locally through an emulator (see more details at the bottom).
## Data format
The Google Cloud PubSub data exporter scalar values
```
message =
```
or dictionaries with a specific format as messages:
```
message = {
'data': ,
'metadata':
}
```
If the data contains no dictionary it will be written to the pubsub as value.
Otherwise, every dictionary message contains at least some data in a `data` field.
The following messages are valid:
```python theme={"system"}
message = {'data': {'bees': 23, 'ants': 30}}
```
The following message shows the configuration of elements data:
```python theme={"system"}
message = {
'data': {
'bees': 23,
'ants': 30,
}
}
```
If not all elements are configured, default values are assumed.
* The project\_id and topic\_id has to be configured in the yaml file.
```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
```
## Set up Google Cloud Pubsub emulator
To test out Google Cloud PubSub source locally, follow the instructions in
[Testing apps locally with the emulator](https://cloud.google.com/pubsub/docs/emulator)
to set up the local Google Cloud PubSub emulator.
## Create a `project_id` for testing
Use the command listed under the `Starting the emulator` section to create a `project_id`, e.g.,
```
gcloud beta emulators pubsub start --project=test-project-id
$(gcloud beta emulators pubsub env-init)
```
## Create a topic and a subscription
Use the `gcloud pubsub topics create` command to create a topic:
```
gcloud pubsub topics create test-topic-id
```
After you create a topic, you can subscribe or publish to it. Use the
`gcloud pubsub subscriptions create` command to create a subscription.
```
gcloud pubsub subscriptions create test-subscription-id-0 --topic test-topic-id
```
Only messages published to the topic after the subscription is created
are available to subscriber applications.
# Google Cloud Storage
Source: https://docs.mage.ai/guides/streaming/destinations/google-cloud-storage
## Basic config
```yaml theme={"system"}
connector_type: google_cloud_storage
bucket: bucket
prefix: prefix
file_type: parquet
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
```
* `bucket`: The Google Cloud Storage bucket you want to use to store the data.
* `prefix`: The Google Cloud Storage path prefix. The whole GCS path will be `gs://{bucket}/{prefix}`.
* `file_type`: `parquet`, `csv`, or `delta` (for Delta Lake tables)
* `buffer_size_mb` and `buffer_timeout_seconds`: Mage puts messages in a buffer before uploading to Google Cloud Storage.
You can configure the size and timeout of the buffer to control the file size and the delay.
* `path_to_credentials_json_file`: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
* `prefix`: The Google Cloud Storage path prefix. The whole S3 path will be `s3://{bucket}/{prefix}`.
* `file_type`: `parquet` or `csv`
* `buffer_size_mb` and `buffer_timeout_seconds`: Mage puts messages in a buffer before uploading to Google Cloud Storage.
You can configure the size and timeout of the buffer to control the file size and the delay.
* `path_to_credentials_json_file`: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate.
## Configure time-based partition
```yaml theme={"system"}
date_partition_format: "%Y%m%dT%H"
```
If you want to store the Google Cloud Storage data in time-based partition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.
## Delta Lake configuration
**Note:** Delta Lake format (`file_type: delta`) is a Mage Pro only feature.
When using `file_type: delta`, you can write data to Delta Lake tables in Google Cloud Storage.
```yaml theme={"system"}
connector_type: google_cloud_storage
bucket: my-bucket
prefix: my-prefix
file_type: delta
table_uri: gs://my-bucket/path/to/delta-table # Optional: if not provided, constructed from bucket/prefix
mode: append # 'append' or 'overwrite'
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
```
* `table_uri`: (Optional) Full URI for the Delta Lake table (e.g., `gs://bucket/path/to/table`). If not provided, the URI will be automatically constructed from `bucket` and `prefix`.
* `mode`: Write mode for Delta Lake tables. Valid values:
* `append`: Adds new rows to the existing table (default)
* `overwrite`: Replaces all existing rows in the table
**Note:** When using Delta Lake format, time-based partitioning (`date_partition_format`) is not applied as Delta Lake manages its own internal file structure.
**Security Note:** Storing credentials in configuration files can be insecure. Prefer using environment variables (e.g., `GOOGLE_APPLICATION_CREDENTIALS`) or secret variables when possible, especially for production deployments. See [Referencing Variables](/development/variables/referencing-variables) for configuring credentials via environment variables or secret variables.
If you want to store the Google Cloud Storage data in time-based paratition folders. You can add `date_partition_format`
to the config. Example values: `%Y%m%d`, `%Y%m%dT%H`.
# InfluxDB
Source: https://docs.mage.ai/guides/streaming/destinations/influxdb
## InfluxDB instance
InfluxDB can be [run locally](https://docs.influxdata.com/influxdb/v2.7/install/) or in the cloud.
## Basic config
Basic configuration includes the server connection details:
```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
```
## Data format
The InfluxDB data exporter accepts only dictionaries as messages following a specific format:
```
message = {
'data':
'metadata':
}
```
Every message contains at least some data in a `data` field. The following messages are valid:
```python theme={"system"}
message_1 = {'data': {'bees': 23, 'ants': 30}}
message_2 = {'data': 23} # implicit for {'_value': 23}
message_3 = {'data': [23, 30]} # implicit for {'_value': 23} and {'_value': 30}
```
InfluxDB supports [data elements](https://docs.influxdata.com/influxdb/v2.7/reference/key-concepts/data-elements/) for structuring your data.
The influxdb data exporter can create these elements if the messages received contain a metadata dictionary:
```python theme={"system"}
message['metadata'] = {
'measurement': str
'time': int
'tags': Dict
}
```
The following message shows the configuration of all possible elements data and metadata:
```python theme={"system"}
message = {
"data": {'bees': 23},
"metadata": {
'measurement': 'census',
'time': 1693472675783, # timestamp with ms precision
'tags': {'location': 'klamath', 'scientist': 'anderson'}
}
}
```
If not all elements are configured, default values are assumed.
* The default measurement name is `default` and can be configured in the yaml file.
```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
measurement: "default"
```
* The default time timestamp is the current time of execution.
* The default tags are none.
The `data` part of the message is taken for the fields elements.
If the data is a scalar value or a list type it will be written to the database with the field name `_value`.
## Advanced config
The influxdb data exporter writes data to the InfluxDB server in batches, if either the batch size or the timeout are reached. Both values can be changed if necessary in the config file:
```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
batch_size: 100
timeout_ms: 500
```
# Kafka
Source: https://docs.mage.ai/guides/streaming/destinations/kafka
## Basic config
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
api_version: "0.10.2"
batch_size: 100
timeout_ms: 500
```
## SSL authentication
```yaml theme={"system"}
security_protocol: "SSL"
ssl_config:
cafile: "CARoot.pem"
certfile: "certificate.pem"
keyfile: "key.pem"
password: password
check_hostname: true
```
## SASL authentication
SASL\_SSL with PLAIN or SCRAM:
```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
mechanism: "PLAIN" # or "SCRAM-SHA-256", "SCRAM-SHA-512"
username: username
password: password
ssl_config:
cafile: "pemfilename.pem" # Optional but recommended
```
SASL\_SSL with OAUTHBEARER (OAuth 2.0):
```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
mechanism: "OAUTHBEARER"
oauth_token_url: "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token"
oauth_client_id: "kafka-client"
oauth_client_secret: "your-client-secret"
oauth_scope: "kafka" # Optional
ssl_config:
cafile: "pemfilename.pem"
check_hostname: true
```
**OAUTHBEARER Authentication:**
* The `oauth_token_url`, `oauth_client_id`, and `oauth_client_secret` are **required** when using `OAUTHBEARER` mechanism.
* The `oauth_scope` parameter is optional.
* OAuth tokens are automatically cached and refreshed before expiry (60 seconds before expiration by default).
* It's recommended to use `SASL_SSL` (not `SASL_PLAINTEXT`) with OAUTHBEARER for secure token transmission.
* Compatible with OAuth 2.0 providers like Keycloak, Okta, and other identity servers that support client credentials flow.
```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
mechanism: "PLAIN"
username: username
password: password
```
## Data format
The Kafka data exporter scalar values
```
message =
```
or dictionaries with a specific format as messages:
```
message = {
'data': ,
'metadata':
}
```
If the data contains no dictionary it will be written to the database as value with the current time value and no key.
Otherwise, every dictionary message contains at least some data in a `data` field.
The following messages are valid:
```python theme={"system"}
message = {'data': {'bees': 23, 'ants': 30}}
```
Kafka supports [structuring and partitioning](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html#kafka.KafkaProducer.send) your data.
The Kafka data exporter can create these elements if the messages received contain a metadata dictionary:
```python theme={"system"}
message['metadata'] = {
'dest_topic': str
'key': str
'time': int
}
```
The following message shows the configuration of all possible elements data and metadata:
```python theme={"system"}
message = {
'data': {
'bees': 23,
'ants': 30,
},
'metadata': {
'dest_topic': 'census',
'key': 'key',
'time': 1693396630163, # timestamp with ms precision
}
}
```
If not all elements are configured, default values are assumed.
* The default topic has to be configured in the yaml file.
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
```
* The default time timestamp is the current time of execution.
* In case you are using newer versions of Kafka brokers, you should consider using corresponding Kafka api\_version.
# Kinesis
Source: https://docs.mage.ai/guides/streaming/destinations/kinesis
## Config
```yaml theme={"system"}
connector_type: kinesis
stream_name: stream_name
partition_key: key
```
* **partition\_key**: A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to.
## Authentication
Here are the options to authenticate with the AWS Kinesis.
1. Add the following keys and values to your environment variables
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`
2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate.
You can grant the ECS task permissions to access other AWS services by attaching IAM policies
to this ECS task execution role.
If you get the `botocore.exceptions.NoRegionError` error, try setting the `AWS_DEFAULT_REGION`
environment variable.
# MongoDB
Source: https://docs.mage.ai/guides/streaming/destinations/mongodb
## Basic config
```yaml theme={"system"}
connection_string: "mongodb://mongodb_container_name:27017/"
database_name: database_name
collection_name: collection_name
```
* `connection_string`: URI defines connection between application and MongoDB instance.
* `database_name`: The database where you want to store the data.
* `collection_name`: The collection where you want to group the documents.
## Optional fields
```yaml theme={"system"}
unique_constraints: ['col1', 'col2'] # Upsert documents based on unique constraints
```
# Microsoft SQL Server
Source: https://docs.mage.ai/guides/streaming/destinations/mssql
## Basic config
```yaml theme={"system"}
connector_type: mssql
profile: default # profile in the io_config.yaml
config:
table_name: table_name
```
## Optional fields
```yaml theme={"system"}
schema_name: schema_name # Override the schema name. Default schema is "dbo"
```
# MySQL
Source: https://docs.mage.ai/guides/streaming/destinations/mysql
## Basic config
```yaml theme={"system"}
connector_type: mysql
profile: default # profile in the io_config.yaml
config:
table_name: table_name
```
# Opensearch
Source: https://docs.mage.ai/guides/streaming/destinations/opensearch
## Basic config
```yaml theme={"system"}
connector_type: opensearch
host: https://[cluster_name].[region].es.amazonaws.com
index_name: test_index
```
## Authentication
```yaml theme={"system"}
verify_certs: true
http_auth: "@awsauth"
```
* **verify\_certs**: Whether to verify SSL certificates to authenticate.
* **http\_auth**: Authentication setting
1. "@awsauth": Authenticate with AWS Signature Version 4. Need to provide
AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY in environment variables.
2. "username:password": Authenticate with username and password.
# Oracle Database
Source: https://docs.mage.ai/guides/streaming/destinations/oracledb
## Basic config
```yaml theme={"system"}
connector_type: oracledb
user: your_user_name
password: your_user_password
host: host_name
port: port_number
table: exported_table_name
service_name: exported_service_name
```
* `user`: The user account in Oracle Database to export data with.
* `password`: password of the user account.
* `host`: Oracle Database host name.
* `port`: Oracle Database port number.
* `table`: Table name to export.
* `service_name`: Oracle Database service name.
## Optional fields
```yaml theme={"system"}
mode: mode_to_export
```
* `mode`: Export mode. Available values: `thin`, `thick`.
# Postgres
Source: https://docs.mage.ai/guides/streaming/destinations/postgres
## Basic config
```yaml theme={"system"}
connector_type: postgres
database: database
host: host
password: password
port: 5432
schema: schema
username: username
table: table
```
## Upsert record
To upsert records to Postgres, you can define `unique_conflict_method` and `unique_constraints` params
in the config.
```yaml theme={"system"}
unique_conflict_method: UPDATE
unique_constraints:
- col1
- col2
allow_reserved_words: False
auto_clean_name: True
case_sensitive: False
```
# RabbitMQ
Source: https://docs.mage.ai/guides/streaming/destinations/rabbitmq
## Basic config
```yaml theme={"system"}
connector_type: rabbitmq
connection_host: 'localhost'
connection_port: 5672
queue_name: 'queue_name'
username: 'guest'
password: 'guest'
amqp_url_virtual_host: '%2f'
url_protocol: 'amqp'
```
# Redshift
Source: https://docs.mage.ai/guides/streaming/destinations/redshift
## Basic config
```yaml theme={"system"}
connector_type: redshift
profile: default # profile in the io_config.yaml
config:
table_name: table_name
schema_name: schema_name # Override the schema name. Default schema is "public"
```
## Alternative
Another option to stream data into Redshift is to stream data to Kinesis.
You can configure streaming ingestion for your Amazon Redshift cluster and create a materialized view using SQL statements.
You can follow [the guide from AWS](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion-getting-started.html) to set it up.
Or you can follow the steps below to set up the materialized view.
### 1. Create and attach IAM role to Redshift cluster
Create an IAM Role which has the policies `AmazonKinesisReadOnlyAccess` and `AmazonRedshiftFullAccess`.
Go to the Redshift cluster's `Properties` tab. In the `Cluster permissions` section, attach the IAM role to the Redshift cluster.
### 2. Create an external schema from Kinesis
Go to Redshift cluster's query editor, execute the command to create the schema:
```sql theme={"system"}
CREATE EXTERNAL SCHEMA redshift_stream_test
FROM KINESIS
IAM_ROLE ;
```
### 3. Create a materialized view which loads data from Kinesis stream
Execute the command in query editor to create the materialized view from the kinesis stream:
```sql theme={"system"}
CREATE MATERIALIZED VIEW mage_kinesis_redshift_stream
AUTO REFRESH YES
AS
SELECT
*,
json_parse(kinesis_data) as payload
FROM redshift_stream_test."kinesis-stream-name"
```
You can change the materialized view name to the name you want to use.
### 4. Query data in the materialized view.
Execute the command to query the data:
```sql theme={"system"}
SELECT * FROM mage_kinesis_redshift_stream;
```
# Snowflake
Source: https://docs.mage.ai/guides/streaming/destinations/snowflake
## Basic config
```yaml theme={"system"}
connector_type: snowflake
profile: default # profile in the io_config.yaml
config:
table_name: table_name
```
## Optional fields
```yaml theme={"system"}
database: database # Override the database
schema: schema # Override the schema
```
# Trino
Source: https://docs.mage.ai/guides/streaming/destinations/trino
## Basic config
```yaml theme={"system"}
connector_type: trino
profile: default # profile in the io_config.yaml
config:
table_name: table_name
```
## Optional fields
```yaml theme={"system"}
schema_name: schema_name # Override the schema name.
```
# Overview
Source: https://docs.mage.ai/guides/streaming/introduction
Streaming pipelines in Mage are designed to process data in real-time as it arrives, rather than in batches at scheduled intervals. This feature is particularly beneficial for use cases where immediate data processing is crucial, such as real-time analytics, monitoring, and decision-making systems.
One of the key benefits of streaming pipelines in Mage is the reduction in latency from data ingestion to actionable insights. For example, a technology solutions company was able to reduce the latency of transforming stream data and writing to MongoDB from 1.5 seconds to 200ms, even for larger size messages, by replacing Spark Structured Streaming with Mage's streaming pipelines. This represents a significant improvement in performance, enabling faster response times and more timely data-driven actions.
Another benefit is the simplification of the data pipeline architecture. Streaming pipelines in Mage allow for a more straightforward setup compared to other tools that may require complex configurations and tuning for performance. This ease of use can lead to increased developer productivity and a lower barrier to entry for teams to implement real-time data processing solutions
## Case study
The use case where streaming pipelines in Mage outperform Spark Structured Streaming involves processing streaming data with high efficiency and low latency. Specifically, the example provided involves a company that needed to transform stream data and write it to MongoDB.
Using Spark Structured Streaming on Databricks for this task, the company experienced an average latency of 1.2 to 1.9 seconds, and for larger size messages, the latency could go up to 3 to 4 seconds. This level of latency was not optimal for their requirements, which necessitated a more efficient solution.
Upon switching to Mage's streaming pipelines, they were able to significantly reduce the latency of their data processing and writing operations to MongoDB. They achieved a latency of less than 200 milliseconds, even for larger size messages. This represents a substantial improvement over their previous setup with Spark Structured Streaming, making Mage's streaming pipelines a better choice for their specific use case.
## Performance difference
Spark Streaming processes data in micro-batches, which inherently introduces a delay as data is collected over a period before being processed. This micro-batching approach, while effective for many use cases, is not truly real-time.
Mageโs performance advantage over Spark Streaming in real-time data processing scenarios include focus on real-time streaming (as opposed to micro-batching), simpler configuration and setup process, and reduced need for performance tuning. Itโs designed with streaming pipelines that can handle data in real-time, reducing latency significantly for applications that require immediate data processing and insights.
## Benefits
The key benefits of using Mage's streaming pipelines in this context include:
1. **Reduced latency**
1. Mageโs streaming pipelines provided a drastic reduction in latency, from seconds to milliseconds, which is crucial for real-time data processing and analytics applications.
2. **Simplicity and efficiency**
1. Mageโs approach to streaming pipelines allows for a more straightforward and efficient data processing workflow, reducing the complexity and overhead associated with managing Spark Structured Streaming jobs.
3. **Scalability and flexibility**
1. Mageโs streaming pipelines are designed to be scalable and flexible, accommodating varying data volumes and processing requirements without significant adjustments to the infrastructure.
This use case demonstrates the effectiveness of Mageโs streaming pipelines in scenarios where low latency and high efficiency are critical, making it a superior alternative to Spark Structured Streaming for specific real-time data processing tasks.
***
## Introduction
Each streaming pipeline has three components:
* Source: The data stream source to consume message from.
* Transformer: Write python code to transform the message before writing to destination.
* Sink (destination): The data stream destination to write message to.
Check out this [tutorial](/guides/streaming-pipeline#create-a-new-pipeline) to set up an example streaming pipeline.
In version `0.8.80` or greater.
You can build a streaming pipeline with one source, multiple transformers, and multiple sinks. Here is an example streaming
pipeline:
## Supported sources
* [Amazon SQS](/streaming/sources/amazon-sqs)
* [Azure Event Hub](/streaming/sources/azure-event-hub)
* [Google Cloud PubSub](/streaming/sources/google-cloud-pubsub)
* [Kafka](/streaming/sources/kafka)
* [Kinesis](/streaming/sources/kinesis)
* [RabbitMQ](/guides/streaming-pipeline-rabbitmq)
## Supported sinks (destinations)
* [Amazon S3](/streaming/destinations/amazon-s3)
* [BigQuery](/streaming/destinations/bigquery)
* [ClickHouse](/streaming/destinations/clickhouse)
* [DuckDB](/streaming/destinations/duckdb)
* [Dummy](/streaming/destinations/dummy)
* [Elasticsearch](/streaming/destinations/elasticsearch)
* [Kafka](/streaming/destinations/kafka)
* [Kinesis](/streaming/destinations/kinesis)
* [MongoDB](/streaming/destinations/mongodb)
* [Microsoft SQL Server](/streaming/destinations/mssql)
* [MySQL](/streaming/destinations/mysql)
* [Opensearch](/streaming/destinations/opensearch)
* [Postgres](/streaming/destinations/postgres)
* [Redshift](/streaming/destinations/redshift)
* [Snowflake](/streaming/destinations/snowflake)
* [Trino](/streaming/destinations/trino)
## Test pipeline execution
After finishing configuring the streaming pipeline, you can click the button `Execution pipeline` to test streaming pipeline execution.
## Run pipeline in production
Create the trigger in [triggers page](/design/data-pipeline-management#create-trigger) to run streaming pipelines in production.
### Executor count
If you want to run multiple executors at the same time to scale the streaming pipeline execution, you can set the `executor_count` variable
in the pipeline's metadata.yaml file. Here is an example:
```yaml theme={"system"}
blocks:
- ...
- ...
executor_count: 10
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline
```
### Executor type
When running Mage on Kubernetes cluster, you can also configure streaming pipeline to be run on separate k8s pods by setting `executor_type`
field in the pipeline's metadata.yaml to `k8s`.
Example config:
```yaml theme={"system"}
blocks:
- ...
- ...
executor_type: k8s
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline
```
## Contributing guide
Follow this [doc](contributing) to add a new source or destination (sink) to Mage streaming pipeline.
# ActiveMQ
Source: https://docs.mage.ai/guides/streaming/sources/activemq
This guide will walk you through the process of setting up and using ActiveMQ with Mage for streaming data pipelines.
### Step 1: Create a new streaming pipeline
1. Open Mage in your browser.
2. Click on "+ New Pipeline" and select "Streaming".
3. Give your pipeline a name and click "Create".
### Step 2: Add an ActiveMQ data loader
1. In your new pipeline, click "+ Add Block" and select "Data loader".
2. Choose "ActiveMQ" as the data source.
### Step 3: Configure your ActiveMQ connection
In the data loader block, use the following YAML configuration as a template:
```yaml theme={"system"}
connector_type: 'activemq'
connection_host: 'localhost'
connection_port: 61613
queue_name: 'default'
username: 'admin'
password: 'admin'
```
Adjust these values according to your ActiveMQ setup:
* `connection_host`: Your ActiveMQ broker's hostname or IP address.
* `connection_port`: The port your ActiveMQ broker is listening on (default STOMP port is 61613).
* `queue_name`: The name of the queue you want to consume messages from.
* `username` and `password`: Your ActiveMQ authentication credentials.
Ensure that the stomp port on ActiveMQ is open, which is 61613 by default.
### Step 4: Additional configuration (optional)
Depending on your specific requirements, you may need to add additional configuration options:
1. For SSL/TLS connections:
```YAML theme={"system"}
use_ssl: true
ssl_cert_path: '/path/to/cert.pem'
ssl_key_path: '/path/to/key.pem'
```
2. To set a connection timeout:
```YAML theme={"system"}
connection_timeout: 30 # in seconds
```
3. To control message prefetching:
```YAML theme={"system"}
prefetch_size: 100
```
### Step 5: Test your connection
1. Save your data loader block.
2. Click the "Test" button to verify your connection to ActiveMQ.
3. If successful, you should see a sample of messages from your queue.
# Amazon SQS
Source: https://docs.mage.ai/guides/streaming/sources/amazon-sqs
## Config
```yaml theme={"system"}
connector_type: amazon_sqs
queue_names:
- producer-events
- controller-events
# Optional. Defaults to false so startup fails if any queue can't be initialized.
skip_failed_queues: false
# Optional. Prefer IAM roles or environment variables for production.
# aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
# aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
# aws_session_token: "{{ env_var('AWS_SESSION_TOKEN') }}" # Optional for temporary credentials
# aws_region: "{{ env_var('AWS_DEFAULT_REGION') }}"
batch_size: 10
wait_time_seconds: 1
message_deletion_method: AFTER_RECEIVED # AFTER_RECEIVED or MANUAL
serde_config:
serialization_method: JSON
```
Use `queue_names` when one streaming source block should consume from multiple SQS queues.
Mage polls every configured queue and sends received messages through the same downstream
streaming pipeline. Existing single-queue configs can continue to use `queue_name`:
```yaml theme={"system"}
connector_type: amazon_sqs
queue_name: producer-events
batch_size: 10
wait_time_seconds: 1
```
By default, startup fails if any configured queue can't be initialized, including when a queue
is missing, configured for the wrong AWS region, or denied by IAM. This avoids silently ignoring
events from one producer. If you want best-effort consumption from the queues that can be
initialized, opt in with:
```yaml theme={"system"}
skip_failed_queues: true
```
When `skip_failed_queues` is enabled, Mage logs the failed queue name and continues polling the
remaining queues. If every configured queue fails to initialize, startup still fails.
## Authentication
Here are the options to authenticate with AWS SQS.
1. Grant your Mage runtime an AWS IAM role with access to the configured queues. This is the
recommended option for Mage deployments running on AWS infrastructure.
2. Add the following keys and values to your runtime environment variables:
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_SESSION_TOKEN` if you're using temporary AWS credentials
* `AWS_DEFAULT_REGION`
3. Configure credentials in the source config. Use environment-variable references instead of
pasting raw secrets into the block YAML:
```yaml theme={"system"}
aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
aws_session_token: "{{ env_var('AWS_SESSION_TOKEN') }}"
aws_region: us-east-1
```
4. If you deploy Mage on an AWS ECS cluster, you can use the ECS task execution role to
authenticate. Grant the ECS task permissions to access SQS by attaching IAM policies to this
role.
If you get the `botocore.exceptions.NoRegionError` error, try setting the `AWS_DEFAULT_REGION`
environment variable.
## Pass raw message to transformer
```yaml theme={"system"}
serde_config:
serialization_method: RAW_VALUE
```
## Message deletion method
We support two ways to delete messages:
1. Delete the message in the data loader automatically after deserializing the message body.
* Set `message_deletion_method: AFTER_RECEIVED` in the source config.
* The input of the transformer is the list of deserialized message body.
2. Manually delete the message in transformer after processing the message.
* Set `message_deletion_method: MANUAL` in the source config.
* The input of the transformer is the list of dictionary with the structure
```python theme={"system"}
{
'parsed_msg_body': {'k1': 'v1', ...},
'queue_name': 'producer-events',
'queue_url': 'https://sqs...',
'raw_message': sqs.Message(...)
}
```
You can use the following example code to process the message and delete it after.
```python theme={"system"}
@transformer
def transform(messages: List[Dict], *args, **kwargs):
processed_msgs = []
for msg in messages:
processed_msg = msg['parsed_msg_body']
# Add your own logic to process the message
processed_msgs.append(processed_msg)
msg['raw_message'].delete()
return processed_msgs
```
# Azure Event Hub
Source: https://docs.mage.ai/guides/streaming/sources/azure-event-hub
This guide provides step-by-step instructions to set up and use Azure Event Hub with Mage. By following these instructions, you can create a new streaming pipeline in Mage, authenticate with Azure Event Hub, and configure your data loader, transformer, and exporter blocks to process and manage your data efficiently.
**Start Mage**
1. Open Mage in your browser and then navigate to the pipelines page.
2. Create a new pipeline by clicking the โ+ Newโ button and then select โStreaming.โ
3. Enter a name for the pipeline and click โCreate.โ
**Authentication**
To authenticate with Azure Event Hub, complete the following actions:
1. Configure your io\_config.yml file with the account variables below. If you need further information about how to store secrets in
Mage check out the [documentation](https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure).
```YAML theme={"system"}
AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
```
2. Add a data loader block to your pipeline and then select "Azure Event Hub" as the source.
3. Complete the data loader block with the information below.
```YAML theme={"system"}
connector_type: azure_event_hub
connection_str: 'Endpoint=sb://[namespace].servicebus.windows.net/;...'
eventhub_name: event_hub_name
consumer_group: '$Default'
```
Connection string: Use the `connection_str` parameter in the config as shown above. You can obtain this from your Azure portal.
4. Add the template transformer block or create a custom block to transform data from Azure Event Hub
* The template will populate with the code below. Make adjustments to your specific requirements.
```Python theme={"system"}
from typing import Dict, List
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def transform(messages: List[Dict], *args, **kwargs):
"""
Template code for a transformer block.
Args:
messages: List of messages from the Event Hub.
Returns:
Transformed messages
"""
# Specify your transformation logic here
print(f'Transformed messages: {messages}')
return messages
```
5. Add a data exporter block and select โDummyโ
6. Use the default setting as seen below.
```YAML theme={"system"}
connector_type: dummy
print_msg: true
```
**Additional Configuration Options**
These additional configuration options are specifically for the data loader block that uses the Azure Event Hub connector. They should be added to the YAML configuration of your data loader block.
1. prefetch: This determines the maximum number of events to retrieve in a single batch when reading from Event Hub.
2. max\_wait\_time: This sets the maximum time to wait for a batch to be filled before returning.
3. starting\_position: This optional parameter specifies where to start reading events from in the Event Hub.
* Common starting positions include:
* -1 or @latest: Start from the end of the stream
* -2 or @earliest: Start from the end of the stream
```YAML theme={"system"}
prefetch: # optional, usually between 100 - 1000
max_wait_time: # optional
starting_position: #optional, example '@latest'
```
By following this guide, you can successfully set up a streaming pipeline in Mage using Azure Event Hub. This allows you to authenticate,
load, transform, and export data efficiently. With these steps, you can leverage Mage and Azure Event Hub to create powerful and scalable
streaming data pipelines.
# Google Cloud PubSub
Source: https://docs.mage.ai/guides/streaming/sources/google-cloud-pubsub
Ingest data from Google Cloud PubSub event streaming sources.
## Config
```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
subscription_id: test-subscription-id-0
timeout: 5
batch_size: 100
pubsub_emulator_host: null # "host.docker.internal:8085"
path_to_credentials_json_file: "./google_credentials.json"
```
The `pubsub_emulator_host` parameter is optional, which could be used when you need to test
the Google Cloud PubSub service locally through an emulator (see more details at the bottom).
## Start Mage
### Create a new pipeline
1. Open Mage in your browser.
2. Click `+ New pipeline`, then select `Streaming`.
3. Add a data loader block, select `Google Cloud PubSub`, and change the settings to those of your PubSub project:
```yaml theme={"system"}
connector_type: google_cloud_pubsub
project_id: test-project-id
topic_id: test-topic-id
subscription_id: test-subscription-id-0
timeout: 5
batch_size: 100
pubsub_emulator_host: null
path_to_credentials_json_file: "./google_credentials.json"
```
4. Add a transformer block and paste the following:
```python theme={"system"}
from typing import Dict, List
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def transform(messages: List[Dict], *args, **kwargs):
"""
Template code for a transformer block.
Args:
messages: List of messages in the stream.
Returns:
Transformed messages
"""
# Specify your transformation logic here
print(f'Transform: {messages}')
return messages
```
5. Add a data exporter block, select `Dummy` and keep the default settings:
```yaml theme={"system"}
connector_type: dummy
print_msg: true
```
##
## Test pipeline
Open the streaming pipeline you just created, and in the right side panel near
the bottom, click the button Execute pipeline to test the pipeline.
### Publish some testing messages
Please use the publisher example given in the [Testing apps locally with the emulator](https://cloud.google.com/pubsub/docs/emulator)
```
python publisher.py test-project-id publish test-topic-id
```
You should see an output similar to the following:
```
[silent_mountain] [GoogleCloudPubSubSource] Subscription already exists: projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub
[silent_mountain] [GoogleCloudPubSubSource] Start consuming batch messages.
[silent_mountain] [GoogleCloudPubSubSource] Number of received messages: 4
[late_wood] Transform: [{'data': 'Google Cloud PubSub Message number 1'}, {'data': 'Google Cloud PubSub Message number 8'}, {'data': 'Google Cloud PubSub Message number 2'}, {'data': 'Google Cloud PubSub Message number 3'}]
[silent_mountain] [DummySink] Batch ingest 4 records, time=1682968557.6466439. Sample: {'data': 'Google Cloud PubSub Message number 1'}
[silent_mountain] [GoogleCloudPubSubSource] Received and acknowledged 4 messages from projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub.
[silent_mountain] [GoogleCloudPubSubSource] Number of received messages: 4
[late_wood] Transform: [{'data': 'Google Cloud PubSub Message number 7'}, {'data': 'Google Cloud PubSub Message number 9'}, {'data': 'Google Cloud PubSub Message number 4'}, {'data': 'Google Cloud PubSub Message number 5'}]
[silent_mountain] [DummySink] Batch ingest 4 records, time=1682968559.027885. Sample: {'data': 'Google Cloud PubSub Message number 7'}
[silent_mountain] [GoogleCloudPubSubSource] Received and acknowledged 4 messages from projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub.
[silent_mountain] [GoogleCloudPubSubSource] Number of received messages: 1
[late_wood] Transform: [{'data': 'Google Cloud PubSub Message number 6'}]
[silent_mountain] [DummySink] Batch ingest 1 records, time=1682968560.894928. Sample: {'data': 'Google Cloud PubSub Message number 6'}
[silent_mountain] [GoogleCloudPubSubSource] Received and acknowledged 1 messages from projects/my-pubsub-project-385206/subscriptions/my-topic-1-sub.
```
## Set up Google Cloud Pubsub emulator
To test out Google Cloud PubSub source locally, follow the instructions in
[Testing apps locally with the emulator](https://cloud.google.com/pubsub/docs/emulator)
to set up the local Google Cloud PubSub emulator.
## Create a `project_id` for testing
Use the command listed under the `Starting the emulator` section to create a `project_id`, e.g.,
```
gcloud beta emulators pubsub start --project=test-project-id
$(gcloud beta emulators pubsub env-init)
```
## Create a topic and a subscription
Use the `gcloud pubsub topics create` command to create a topic:
```
gcloud pubsub topics create test-topic-id
```
After you create a topic, you can subscribe or publish to it. Use the
`gcloud pubsub subscriptions create` command to create a subscription.
```
gcloud pubsub subscriptions create test-subscription-id-0 --topic test-topic-id
```
Only messages published to the topic after the subscription is created
are available to subscriber applications.
# InfluxDB
Source: https://docs.mage.ai/guides/streaming/sources/influxdb
## InfluxDB instance
InfluxDB can be [run locally](https://docs.influxdata.com/influxdb/v2.7/install/) or in the cloud.
## Basic config
Basic configuration includes the server connection details:
```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
```
## Data format
InfluxDB supports [data elements](https://docs.influxdata.com/influxdb/v2.7/reference/key-concepts/data-elements/) for structuring your data.
The influxdb data exporter can create these elements if the messages received contain a metadata dictionary:
```python theme={"system"}
message['metadata'] = {
'measurement': str
'time': int
'tags': Dict
}
```
The following message shows the configuration of all possible elements data and metadata:
```python theme={"system"}
message = {
"data": {'bees': 23},
"metadata": {
'measurement': 'census',
'time': 1693472675783, # timestamp with ms precision
'tags': {'location': 'klamath', 'scientist': 'anderson'}
}
}
```
## Advanced config
The influxdb data reader reads data from InfluxDB server in timed intervals.
Every `timeout_ms` data with time stamps in the interval `(t-time_delay, t-time_delay+timeout_ms)` will be returned, where `t` is the current time.
Note that InfluxDB is not a streaming database, so trying to set `time_delay` too small may result in data loss.
This will occur if data is written to the database after the corresponding interval has been read.
The time delay is given as [InfluxDB duration](https://docs.influxdata.com/flux/v0.x/data-types/basic/duration/).
The `batch_size` limits the size of batches returned by the data reader.
If `print_intervals` is `true` the intervals are printed on every time step.
The `filter_fn` can be used to further specify the data, e.g. `r._measurement == "cpu" and r._field == "usage_system"` (note the usage of single quotation marks outside and double quotation marks inside the filter\_fn string).
Defaults to `true` for no further filtering.
For documentation, see [InfluxDB filter() function](https://docs.influxdata.com/flux/v0.x/stdlib/universe/filter/)
```yaml theme={"system"}
connector_type: influxdb
url: "http://localhost:8086"
token: "my-token"
org: "my-org"
bucket: "data"
time_delay: "5m"
timeout_ms: 1000
batch_size: 100
print_intervals: false
filter_fn: 'true'
```
# Kafka
Source: https://docs.mage.ai/guides/streaming/sources/kafka
## Basic config
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
topics: ['topic_name1', 'topic_name2'] # Optional. Used for specifying multiple topics.
consumer_group: unique_consumer_group
include_metadata: false
api_version: "0.10.2"
batch_size: 100
timeout_ms: 500
auto_offset_reset: latest # Optional
max_partition_fetch_bytes: 5242880 # Optional, default: 5MB (Pro), 1MB (OSS)
max_partition_fetch_bytes: 1048576 # Optional
offset: # Optional
value: null # Optional
type: "int"
partitions: # Optional
- {"topic": "test", "partition": 1}
```
### Configuration Parameters
#### Required Parameters
* **`bootstrap_server`**: Kafka broker address (e.g., `"localhost:9092"` or `"broker1:9092,broker2:9092"`)
* **`consumer_group`**: Unique consumer group identifier for this pipeline
* **`topic`** or **`topics`**: Single topic name (string) or list of topic names to consume from
#### Performance Parameters
* **`batch_size`** (default: `100`): Maximum number of messages to process in a single batch. The consumer will accumulate messages across multiple polls to reach this target when possible.
**Recommendations:**
* Small messages (\< 1 KB): 500-2000
* Medium messages (1-10 KB): 200-1000
* Large messages (> 10 KB): 50-500
* **`max_partition_fetch_bytes`** (default: `5242880` = 5MB for Pro, `1048576` = 1MB for OSS): Maximum amount of data to fetch per partition in a single poll. This is a **per-partition limit** that can constrain your batch size.
**Important:** This value must be:
* Greater than your largest message size
* Aligned with broker settings (`message.max.bytes`, `replica.fetch.max.bytes`)
**Recommendations by message size:**
* Small messages (\< 1 KB): 1-5 MB
* Medium messages (1-10 KB): 5-10 MB
* Large messages (10-100 KB): 10-50 MB
* Very large messages (> 100 KB): 50-100 MB
**Memory consideration:** Total in-flight data = `max_partition_fetch_bytes ร number of partitions`. Ensure your consumer has sufficient heap memory.
* **`timeout_ms`** (default: `500`): Maximum time to wait for messages when polling (in milliseconds). Lower values reduce latency but may increase CPU usage.
#### Other Parameters
* **`auto_offset_reset`** (default: `latest`): What to do when there is no initial offset or the offset is out of range. Options: `earliest`, `latest`, `none`
* **`include_metadata`**: If `true`, includes Kafka metadata (key, partition, offset, timestamp, topic) with each message
* **`api_version`** (default: `"0.10.2"`): Kafka API version. Use a version matching your broker version for best compatibility
### Offset Configuration
Available in versions `>= 0.9.61`
The `offset` config allows you to reset offset when starting the streaming pipeline. If using the `offset` config, the `partitions` config is required.
`offset` config has 4 optional values:
1. `beginning` - Start from the earliest available message
2. `end` - Start from the latest message (skip all existing messages)
3. `int` - Start from a specific numeric offset
4. `timestamp` - Start from messages at or after a specific timestamp
**Examples:**
Start from beginning:
```yaml theme={"system"}
offset:
type: "beginning"
partitions:
- {"topic": "my_topic", "partition": 0}
- {"topic": "my_topic", "partition": 1}
```
Start from specific offset:
```yaml theme={"system"}
offset:
type: "int"
value: 1000
partitions:
- {"topic": "my_topic", "partition": 0}
```
Start from specific timestamp (milliseconds since epoch):
```yaml theme={"system"}
offset:
type: "timestamp"
value: 1693396630163 # Wed Aug 30 2023 11:57:10 UTC
partitions:
- {"topic": "my_topic", "partition": 0}
```
**Note:** Both `beginning` and `end` do not require `value` field.
1. beginning
2. end
3. int
4. timestamp
Both `beginning` and `end` set the consumer to consume data from the beginning and end of the queue, respectively.
Also, they do not require `offset_value`
`int` config set the consumer to consume data from the given offset value inside the queue.
This value correspond to the numeric position inside each partition.
`timestamp` config set the consumer to consume data from the given offset timestamp value inside the queue.
This value correspond to the timestamp of the message (Unit should be milliseconds since beginning of the epoch)
## SSL authentication
```yaml theme={"system"}
security_protocol: "SSL"
ssl_config:
cafile: "CARoot.pem"
certfile: "certificate.pem"
keyfile: "key.pem"
password: password
check_hostname: true
```
## SASL authentication
SASL\_PLAINTEXT config:
```yaml theme={"system"}
security_protocol: "SASL_PLAINTEXT"
sasl_config:
mechanism: "PLAIN"
username: username
password: password
```
SASL\_SSL config:
```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
mechanism: "SCRAM-SHA-512"
username: username
password: password
ssl_config:
cafile: "pemfilename.pem"
```
SASL\_SSL with OAUTHBEARER (OAuth 2.0):
```yaml theme={"system"}
security_protocol: "SASL_SSL"
sasl_config:
mechanism: "OAUTHBEARER"
oauth_token_url: "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token"
oauth_client_id: "kafka-client"
oauth_client_secret: "your-client-secret"
oauth_scope: "kafka" # Optional
ssl_config:
cafile: "pemfilename.pem"
check_hostname: true
```
**OAUTHBEARER Authentication:**
* The `oauth_token_url`, `oauth_client_id`, and `oauth_client_secret` are **required** when using `OAUTHBEARER` mechanism.
* The `oauth_scope` parameter is optional.
* OAuth tokens are automatically cached and refreshed before expiry (60 seconds before expiration by default).
* It's recommended to use `SASL_SSL` (not `SASL_PLAINTEXT`) with OAUTHBEARER for secure token transmission.
* Compatible with OAuth 2.0 providers like Keycloak, Okta, and other identity servers that support client credentials flow.
## Data format
By default, if `include_metadata` is false, the kafka data loader returns data from value field, e.g.
```python theme={"system"}
{"location": "klamath", "scientist": "anderson", "count": 23}
```
Kafka supports [structuring and partitioning](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html#kafka.KafkaProducer.send) your data.
If `include_metadata` is set to true, the kafka data loader returns these elements as messages with `data = {key: value}` and `metadata` with (topic, partition, offset and time), e.g.
```python theme={"system"}
message = {
"data": {"location": "klamath", "scientist": "anderson", "count": 23},
"metadata": {
"key": "bees",
"partition": 0,
"offset": 6,
"time": 1693396630163, # timestamp with ms precision (Wed Aug 30 2023 11:57:10 GMT+0)
"topic": "census",
}}
```
## Deserialize message with protobuf schema
```yaml theme={"system"}
serde_config:
serialization_method: PROTOBUF
schema_classpath: "path.to.schema.SchemaClass"
```
* Specify the `serialization_method` to `PROTOBUF`.
* Set the `schema_classpath` to the path to the Python schema class. Test whether you have access the the schema with the code
in a scratchpad.
```python theme={"system"}
from path.to.schema import SchemaClass
```
## Pass raw message to transformer
```yaml theme={"system"}
serde_config:
serialization_method: RAW_VALUE
```
## Deserialize message with Avro schema in Confluent schema registry
```yaml theme={"system"}
serde_config:
serialization_method: AVRO
schema_registry_url: https://schema_registry_url
schema_registry_username: username
schema_registry_password: password
```
## API version
In case you are using newer versions of Kafka brokers, you should consider using corresponding Kafka api\_version.
## Performance Tuning
### Understanding Batch Size vs. Fetch Bytes
The `batch_size` and `max_partition_fetch_bytes` work together:
* **`batch_size`**: Target number of messages to process together
* **`max_partition_fetch_bytes`**: Maximum data (bytes) fetched per partition per poll
**Example:** If you set `batch_size: 500` but `max_partition_fetch_bytes: 1048576` (1MB) and your messages are 5KB each:
* Each partition can fetch \~200 messages (1MB รท 5KB)
* With 1 partition, you'll get \~200 messages per batch
* With multiple partitions, you may get closer to 500, but each partition is still limited to 1MB
**Solution:** Increase `max_partition_fetch_bytes` to allow more messages per partition:
```yaml theme={"system"}
batch_size: 500
max_partition_fetch_bytes: 5242880 # 5MB allows ~1000 messages per partition
```
### High-Throughput Configuration
For high-volume scenarios with consumer lag:
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "broker1:9092,broker2:9092"
topic: high_volume_topic
consumer_group: high_throughput_consumer
batch_size: 1000
max_partition_fetch_bytes: 10485760 # 10MB
timeout_ms: 1000
auto_offset_reset: earliest
```
### Low-Latency Configuration
For real-time processing with minimal delay:
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: realtime_topic
consumer_group: low_latency_consumer
batch_size: 10
max_partition_fetch_bytes: 1048576 # 1MB
timeout_ms: 100
auto_offset_reset: latest
```
### Memory Planning
Calculate memory requirements:
```
Estimated Memory = max_partition_fetch_bytes ร number_of_partitions ร 1.5
```
The 1.5x multiplier accounts for overhead. Ensure your consumer process has at least this much memory available.
## Troubleshooting
### Issue: Not Receiving Expected Batch Size
**Symptom:** Set `batch_size: 500` but only getting \~200-250 messages per batch.
**Cause:** `max_partition_fetch_bytes` is too low for your message size.
**Solution:**
1. Calculate: `max_partition_fetch_bytes รท average_message_size = max_messages_per_partition`
2. Increase `max_partition_fetch_bytes`:
```yaml theme={"system"}
max_partition_fetch_bytes: 5242880 # 5MB (or higher)
```
3. Verify broker settings allow this value:
* `message.max.bytes >= max_partition_fetch_bytes`
* `replica.fetch.max.bytes >= max_partition_fetch_bytes`
### Issue: Consumer Lag Growing
**Symptom:** Consumer falling behind producer, lag increasing over time.
**Solutions:**
1. Increase `batch_size` and `max_partition_fetch_bytes`
2. Increase `timeout_ms` to allow more time for message accumulation
3. Scale horizontally: Set higher value for [executor\_count](/guides/streaming/introduction#executor-count) to run multiple consumer instances in the same consumer group so Kafka can distribute partitions among them.
4. Optimize downstream processing speed
### Issue: Out of Memory Errors
**Symptom:** Consumer crashes with memory errors.
**Cause:** `max_partition_fetch_bytes ร number_of_partitions` exceeds available memory.
**Solution:**
1. Reduce `max_partition_fetch_bytes`
2. Reduce number of partitions consumed (use topic filtering)
3. Increase consumer heap memory
4. Reduce `batch_size` to process smaller batches
### Issue: Configuration Changes Not Taking Effect
**Symptom:** Changed `batch_size` or `max_partition_fetch_bytes` but behavior unchanged.
**Cause:** Kafka consumer is initialized at pipeline start. Configuration is read during initialization.
**Solution:** Restart the streaming pipeline after changing configuration parameters.
### Issue: Messages Not Being Consumed
**Symptom:** No messages received despite messages in topic.
**Checklist:**
1. Verify `consumer_group` is unique and not being used by another consumer
2. Check `auto_offset_reset` setting (use `earliest` to read from beginning)
3. Verify topic name spelling matches exactly
4. Check network connectivity to `bootstrap_server`
5. Verify consumer has read permissions for the topic
6. Check if using `offset` config - it may be seeking to a specific position
## Complete Example Configurations
### Example 1: Basic Setup with Small Messages
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: user_events
consumer_group: user_events_processor
batch_size: 500
max_partition_fetch_bytes: 2097152 # 2MB
timeout_ms: 500
auto_offset_reset: latest
include_metadata: false
```
### Example 2: High-Volume Processing with Large Messages
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "kafka-broker1:9092,kafka-broker2:9092"
topic: large_payload_topic
consumer_group: large_payload_consumer
batch_size: 100
max_partition_fetch_bytes: 52428800 # 50MB for large messages
timeout_ms: 2000
auto_offset_reset: earliest
include_metadata: true
```
### Example 3: Multiple Topics with Authentication
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "secure-kafka:9093"
topics: ['topic1', 'topic2', 'topic3']
consumer_group: multi_topic_consumer
batch_size: 200
max_partition_fetch_bytes: 10485760 # 10MB
timeout_ms: 1000
security_protocol: SASL_SSL
sasl_config:
mechanism: "SCRAM-SHA-512"
username: kafka_user
password: kafka_password
ssl_config:
cafile: "/path/to/ca-cert.pem"
```
### Example 4: Backfill from Specific Timestamp
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: historical_data
consumer_group: backfill_consumer
batch_size: 1000
max_partition_fetch_bytes: 10485760 # 10MB
timeout_ms: 1000
offset:
type: "timestamp"
value: 1609459200000 # 2021-01-01 00:00:00 UTC
partitions:
- {"topic": "historical_data", "partition": 0}
- {"topic": "historical_data", "partition": 1}
- {"topic": "historical_data", "partition": 2}
```
## Quick Reference
### Configuration Quick Reference Table
| Parameter | Default | Recommended Range | Notes |
| --------------------------- | -------------------- | ----------------- | ------------------------------------------- |
| `batch_size` | 100 | 50-2000 | Higher for small messages, lower for large |
| `max_partition_fetch_bytes` | 5MB (Pro), 1MB (OSS) | 1MB-100MB | Must exceed max message size |
| `timeout_ms` | 500 | 100-5000 | Lower = lower latency, higher CPU |
| `auto_offset_reset` | latest | earliest/latest | earliest for backfill, latest for real-time |
### Message Size to Configuration Mapping
| Message Size | Recommended `batch_size` | Recommended `max_partition_fetch_bytes` |
| ------------ | ------------------------ | --------------------------------------- |
| \< 1 KB | 500-2000 | 1-5 MB |
| 1-10 KB | 200-1000 | 5-10 MB |
| 10-100 KB | 50-500 | 10-50 MB |
| > 100 KB | 10-100 | 50-100 MB |
## Best Practices
1. **Consumer Groups:** Use unique consumer group names per pipeline/environment (e.g., `pipeline_prod`, `pipeline_staging`)
2. **Batch Sizing:** Start with defaults and tune based on message size and throughput needs
3. **Monitoring:** Monitor consumer lag, batch sizes, and memory usage regularly
4. **Broker Alignment:** Ensure `max_partition_fetch_bytes` aligns with broker configuration (`message.max.bytes`, `replica.fetch.max.bytes`)
5. **Error Handling:** Implement proper error handling in your transformation logic to handle malformed messages
6. **Offset Management:**
* Use `auto_offset_reset: earliest` for initial processing or backfills
* Use `auto_offset_reset: latest` for real-time streaming
* Use explicit `offset` config for precise control
7. **Multiple Topics:** When consuming from multiple topics, ensure all topics exist and are accessible
8. **Restart After Config Changes:** Always restart the pipeline after changing `batch_size` or `max_partition_fetch_bytes`
9. **Memory Planning:** Calculate memory needs: `max_partition_fetch_bytes ร partitions ร 1.5`
10. **Testing:** Test configuration changes in a staging environment before production
# Kinesis
Source: https://docs.mage.ai/guides/streaming/sources/kinesis
## Config
```yaml theme={"system"}
connector_type: kinesis
stream_name: stream_name
batch_size: 100
```
## Authentication
Here are the options to authenticate with the AWS Kinesis.
1. Add the following keys and values to your environment variables
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_REGION`
2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate.
You can grant the ECS task permissions to access other AWS services by attaching IAM policies
to this ECS task execution role.
If you get the `botocore.exceptions.NoRegionError` error, try setting the `AWS_DEFAULT_REGION`
environment variable.
# MongoDB
Source: https://docs.mage.ai/guides/streaming/sources/mongodb
## Basic Config
```yaml theme={"system"}
connector_type: mongodb
connection_str: "mongodb://localhost:27017/" #Must be replica set and MongoVersion is higher than Mongo5.1
database: "test"
collection: "test"
```
The following additional configuration options can be included: batch\_size, pipeline, starting\_after, full\_document. Instructions on how to use these options can be found in the [pymongo documentation](https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.watch).
# Microsoft SQL Server CDC
Source: https://docs.mage.ai/guides/streaming/sources/mssql-cdc
Real-time Change Data Capture from SQL Server CDC change tables with automatic schema tracking
## Overview
The Microsoft SQL Server CDC streaming source captures real-time changes from SQL Server Change Data Capture (CDC) change tables and processes them into structured records. It uses SQL Server's built-in CDC feature to poll for changes and provides comprehensive CDC capabilities with automatic schema tracking and primary key detection.
Captures changes as they happen in SQL Server
Uses SQL Server's built-in Change Data Capture feature
Automatically identifies and extracts primary key columns
Tracks Log Sequence Numbers for reliable resume capabilities
## Configuration
### Required Parameters
| Parameter | Type | Description |
| ---------- | ------ | ----------------------------------- |
| `host` | string | SQL Server hostname or IP address |
| `port` | number | SQL Server port (default: 1433) |
| `database` | string | Database name |
| `user` | string | SQL Server username with CDC access |
| `password` | string | SQL Server password |
### Optional Parameters
| Parameter | Type | Default | Description |
| ------------------------ | ------- | ------------------------------- | ---------------------------------------------------------------------------- |
| `connect_timeout` | number | 10 | Connection timeout in seconds |
| `driver` | string | "ODBC Driver 18 for SQL Server" | ODBC driver name |
| `authentication` | string | - | Authentication method (e.g., "ActiveDirectoryPassword") |
| `schema` | string | dbo | Default schema name |
| `schemas` | array | - | Include only these schemas |
| `tables` | array | - | Include only these tables |
| `ignore_schemas` | array | - | Exclude these schemas (supports wildcards like "sys\*") |
| `ignore_tables` | array | - | Exclude these tables (supports wildcards like "temp\_\*") |
| `start_lsn` | string | - | Start LSN to begin reading from (hex string, e.g., '0x00000000000000000000') |
| `capture_instance` | string | - | Capture instance name (auto-detected if not provided) |
| `max_batch_size` | number | 100 | Maximum events per batch |
| `flush_interval_seconds` | number | 1.0 | Maximum time between batch flushes |
| `poll_interval_seconds` | number | 1.0 | How often to poll for new changes |
| `return_db_records_only` | boolean | true | Return only DB records with Mage timestamp columns |
## Record Formats
### DB Records Only Format
When `return_db_records_only: true` (default):
```json theme={"system"}
{
"data": {
"id": 1,
"name": "John",
"email": "john@example.com",
"_mage_created_at": "2021-12-31T23:00:00+00:00",
"_mage_updated_at": null,
"_mage_deleted_at": null
},
"metadata": {
"schema": "dbo",
"table": "users",
"key_columns": ["id"],
"operation": "INSERT",
"lsn": "0x00000000000000000000"
}
}
```
**Note**: The `_mage_*` timestamp columns are returned as datetime objects (ISO 8601 format strings when serialized) in UTC timezone, not Unix timestamps. This ensures proper type handling when writing to databases.
## Event Types
* **Insert** (`operation: "INSERT"`): INSERT operations (CDC operation code 2)
* **Update** (`operation: "UPDATE"`): UPDATE operations (CDC operation code 4, after image only)
* **Delete** (`operation: "DELETE"`): DELETE operations (CDC operation code 1)
These events contain the actual data changes. UPDATE operations only include the after image (the new values), not the before image.
## Primary Key Detection
The source automatically detects and extracts primary key columns:
1. **Schema Discovery**: Queries `INFORMATION_SCHEMA` for table structure
2. **Primary Key Detection**: Identifies actual primary key columns from database constraints
3. **Caching**: Stores schema and primary key information for performance
4. **Key Extraction**: Extracts primary key column names from row events
```json Example Output theme={"system"}
{
"metadata": {
"key_columns": ["user_id", "tenant_id"]
}
}
```
**Note**: `key_columns` is a list of column names (not values) that can be used for deduplication or upsert operations in downstream sinks.
## Examples
```yaml Basic Setup theme={"system"}
connector_type: mssql
host: localhost
port: 1433
database: mydb
user: cdc_user
password: password
schemas: ["dbo"]
return_db_records_only: true
```
```yaml Production Setup theme={"system"}
connector_type: mssql
host: sqlserver-prod.company.com
port: 1433
database: production_db
user: cdc_user
password: secure_password
driver: "ODBC Driver 18 for SQL Server"
schemas: ["dbo", "sales"]
ignore_tables: ["temp_*", "log_*"]
max_batch_size: 500
flush_interval_seconds: 2.0
poll_interval_seconds: 0.5
return_db_records_only: true
```
```yaml Resume from LSN theme={"system"}
connector_type: mssql
host: localhost
port: 1433
database: mydb
user: cdc_user
password: password
start_lsn: "0x00000000000000000000"
schemas: ["dbo"]
return_db_records_only: true
```
```yaml With Azure AD Authentication theme={"system"}
connector_type: mssql
host: sqlserver.database.windows.net
port: 1433
database: mydb
user: user@domain.com
password: password
authentication: "ActiveDirectoryPassword"
driver: "ODBC Driver 18 for SQL Server"
schemas: ["dbo"]
return_db_records_only: true
```
```yaml Filter Specific Tables theme={"system"}
connector_type: mssql
host: localhost
port: 1433
database: mydb
user: cdc_user
password: password
tables: ["users", "orders", "products"]
return_db_records_only: true
```
## Prerequisites
### Enable CDC on Database
CDC must be enabled at the database level before it can be used:
```sql theme={"system"}
USE your_database_name;
GO
EXEC sys.sp_cdc_enable_db;
GO
```
Verify CDC is enabled:
```sql theme={"system"}
SELECT is_cdc_enabled FROM sys.databases WHERE name = 'your_database_name';
-- Should return: 1
```
### Enable CDC on Tables
After enabling CDC on the database, enable it for each table you want to monitor:
```sql theme={"system"}
USE your_database_name;
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'YourTableName',
@role_name = NULL,
@supports_net_changes = 0;
GO
```
**Parameters:**
* `@source_schema`: Schema name (e.g., 'dbo')
* `@source_name`: Table name
* `@role_name`: Role name for CDC access control (NULL = no role restriction)
* `@supports_net_changes`: 0 = all changes, 1 = net changes only
### Verify CDC on Tables
Check which tables have CDC enabled:
```sql theme={"system"}
SELECT
s.name AS schema_name,
t.name AS table_name,
c.capture_instance
FROM cdc.change_tables c
INNER JOIN sys.tables t ON c.source_object_id = t.object_id
INNER JOIN sys.schemas s ON t.schema_id = s.schema_id
ORDER BY s.name, t.name;
```
### SQL Server Agent
CDC requires SQL Server Agent to be running. Verify it's running:
```sql theme={"system"}
EXEC xp_servicecontrol 'QueryState', N'SQLServerAGENT';
```
If not running, start the SQL Server Agent service.
### User Permissions
The user needs the following permissions:
```sql theme={"system"}
-- Grant database access
USE your_database_name;
GO
-- Grant SELECT on tables
GRANT SELECT ON SCHEMA::dbo TO cdc_user;
-- Grant SELECT on CDC tables
GRANT SELECT ON SCHEMA::cdc TO cdc_user;
-- Grant SELECT on system tables for metadata queries
GRANT SELECT ON sys.tables TO cdc_user;
GRANT SELECT ON sys.schemas TO cdc_user;
GRANT SELECT ON sys.databases TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.TABLES TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.COLUMNS TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.TABLE_CONSTRAINTS TO cdc_user;
GRANT SELECT ON INFORMATION_SCHEMA.KEY_COLUMN_USAGE TO cdc_user;
```
## Troubleshooting
1. **CDC Not Enabled on Database**: Run `EXEC sys.sp_cdc_enable_db;` on the database
2. **CDC Not Enabled on Table**: Run `sys.sp_cdc_enable_table` for each table
3. **SQL Server Agent Not Running**: Start SQL Server Agent service
4. **Permission Denied**: Ensure user has SELECT permissions on tables and CDC schema
5. **Connection Timeout**: Increase `connect_timeout` value
6. **LSN Format Error**: Ensure LSN is in hex format (e.g., '0x00000000000000000000')
7. **ODBC Driver Not Found**: Install appropriate ODBC driver (e.g., "ODBC Driver 18 for SQL Server")
8. **Max LSN Returns NULL**: This usually means CDC is not enabled on the database
Check CDC status on database:
```sql theme={"system"}
SELECT is_cdc_enabled FROM sys.databases WHERE name = 'your_database_name';
```
Check CDC enabled tables:
```sql theme={"system"}
SELECT
s.name AS schema_name,
t.name AS table_name,
c.capture_instance
FROM cdc.change_tables c
INNER JOIN sys.tables t ON c.source_object_id = t.object_id
INNER JOIN sys.schemas s ON t.schema_id = s.schema_id;
```
Check current max LSN:
```sql theme={"system"}
SELECT sys.fn_cdc_get_max_lsn() AS max_lsn;
```
Check min LSN for a capture instance:
```sql theme={"system"}
SELECT sys.fn_cdc_get_min_lsn('dbo_YourTableName_CT') AS min_lsn;
```
View recent CDC changes:
```sql theme={"system"}
DECLARE @from_lsn binary(10) = sys.fn_cdc_get_min_lsn('dbo_YourTableName_CT');
DECLARE @to_lsn binary(10) = sys.fn_cdc_get_max_lsn();
SELECT * FROM cdc.fn_cdc_get_all_changes_dbo_YourTableName_CT(
@from_lsn, @to_lsn, N'all'
);
```
Monitor key metrics:
* **Batch Size**: Average events per batch
* **Flush Rate**: How often batches are flushed
* **Error Rate**: Failed events or connections
* **Lag**: Time between event occurrence and processing
* **LSN Progress**: Track Log Sequence Number progress
Check CDC capture instance status:
```sql theme={"system"}
SELECT
capture_instance,
source_schema,
source_table,
capture_job_id,
supports_net_changes
FROM cdc.change_tables;
```
Monitor CDC job status:
```sql theme={"system"}
SELECT
job_id,
name,
enabled,
date_modified
FROM msdb.dbo.sysjobs
WHERE name LIKE 'cdc.%';
```
## Integration with Generic IO Sink
The SQL Server CDC source works seamlessly with the Generic IO sink, which provides:
* **Automatic Column Type Mapping**: Automatically maps `_mage_*` timestamp columns to appropriate database types (TIMESTAMP, DATETIME2, DateTime64, etc.) based on the target database
* **Metadata Interpolation**: Use metadata values from SQL Server CDC events in sink configurations for dynamic routing and table naming
### Supported Databases
Generic IO Sink supports the following databases:
* **BigQuery**
* **ClickHouse**
* **DuckDB**
* **MySQL**
* **MSSQL**
* **Postgres**
### Metadata Interpolation
You can use metadata values from SQL Server CDC events in your sink configuration using Python string formatting syntax:
* **`{schema}`**: Schema name from the event
* **`{table}`**: Table name from the event
* **`{key_columns}`**: List of primary key column names (e.g., `["id"]` or `["user_id", "tenant_id"]`)
When using `{key_columns}` in `unique_constraints`, it will be automatically converted from a string representation to a list. The format supports both Python-style (`"['id']"`) and JSON-style (`'["id"]'`) array strings.
### Example Configurations
```yaml BigQuery Example theme={"system"}
connector_type: bigquery
profile: default
config:
table_id: "{schema}.{table}" # Dynamic table names from metadata
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}" # Uses primary keys from metadata
```
```yaml Postgres Example theme={"system"}
connector_type: postgres
profile: default
config:
schema_name: "{schema}" # Dynamic schema from metadata
table_name: "{table}" # Dynamic table name from metadata
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}" # Uses primary keys from metadata
if_exists: append
```
```yaml MSSQL Example theme={"system"}
connector_type: mssql
profile: default
config:
schema_name: "{schema}"
table_name: "{table}"
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}"
if_exists: append
```
```yaml Static Configuration theme={"system"}
connector_type: postgres
profile: default
config:
schema_name: public
table_name: cdc_events # Static table name
unique_constraints: ['id', 'event_id'] # Static list (string format also works: "['id', 'event_id']")
unique_conflict_method: UPDATE
if_exists: append
```
### How Metadata Interpolation Works
1. **Message Grouping**: Messages are automatically grouped by their interpolated config values. For example, if you use `table_name: "{schema}_{table}"`, messages from `dbo.users` will be grouped together and written to `dbo_users` table.
2. **Key Columns Interpolation**: When using `{key_columns}` in `unique_constraints`, the sink automatically:
* Converts the list to a string representation during interpolation
* Parses it back to a list (supports both `"['id']"` and `'["id"]'` formats)
* Uses it for upsert operations based on `unique_conflict_method`
**Example**: If you have a table `users` with primary key `id`, and you configure `unique_constraints: "{key_columns}"`, it will automatically use `["id"]` for upsert operations.
## Best Practices
1. **Enable CDC on Database First**: Always enable CDC at the database level before enabling on tables
2. **Monitor SQL Server Agent**: Ensure SQL Server Agent is running for CDC to work
3. **Filter Events**: Use schema/table filters to reduce processing overhead
4. **Monitor Resources**: Watch memory and CPU usage, especially with high-volume tables
5. **Test Resume**: Verify checkpoint functionality with LSN tracking
6. **Secure Connections**: Use encrypted connections in production
7. **Regular Backups**: Backup checkpoint files
8. **Schema Validation**: Test with schema changes
9. **Use Generic IO Sink**: Leverage automatic type mapping and metadata interpolation for flexible data routing
10. **Metadata Interpolation**: Use `{schema}`, `{table}`, and `{key_columns}` for dynamic table routing and upsert configuration
11. **Poll Interval**: Adjust `poll_interval_seconds` based on your latency requirements (lower = more frequent polling, higher CPU usage)
12. **Batch Size**: Adjust `max_batch_size` based on your throughput needs (larger = fewer handler calls, more memory usage)
## Timestamp Handling
The SQL Server CDC source automatically adds Mage timestamp columns:
* **`_mage_created_at`**: Set to event timestamp (datetime) for INSERT operations
* **`_mage_updated_at`**: Set to event timestamp (datetime) for UPDATE operations
* **`_mage_deleted_at`**: Set to event timestamp (datetime) for DELETE operations
All timestamps are in UTC timezone and returned as datetime objects, which ensures:
* Proper type handling in downstream databases
* Automatic type conversion in Generic IO sink
* Consistent timezone handling across systems
## Limitations
* **SQL Server 2008+**: Requires SQL Server 2008 or later (Enterprise, Developer, or Standard edition)
* **CDC Feature**: Requires CDC to be enabled on database and tables
* **SQL Server Agent**: Requires SQL Server Agent to be running
* **Network Dependency**: Requires stable network connection
* **Memory Usage**: Schema caching uses memory
* **CDC Retention**: Depends on SQL Server CDC retention settings (default: 3 days)
* **Timezone**: All timestamps are in UTC timezone
* **Polling**: Uses polling mechanism (not push-based like logical replication)
* **UPDATE Events**: Only includes after image, not before image
* **ODBC Driver**: Requires appropriate ODBC driver to be installed
## LSN Format
SQL Server CDC uses Log Sequence Numbers (LSN) in binary format, which are converted to hex strings for storage and checkpointing:
* **Format**: `0x` followed by hexadecimal digits (e.g., `0x00000000000000000000`)
* **Length**: Typically 10 bytes (20 hex characters)
* **Comparison**: LSNs can be compared as integers for range queries
* **Checkpoint**: Last processed LSN is saved in checkpoint file for resume capability
## Capture Instance Names
When CDC is enabled on a table, SQL Server automatically creates a capture instance. The naming convention is:
* **Format**: `{schema}_{table}_CT`
* **Example**: `dbo_users_CT` for table `users` in schema `dbo`
The source automatically discovers capture instances, but you can also specify `capture_instance` in the configuration if needed.
# MySQL CDC
Source: https://docs.mage.ai/guides/streaming/sources/mysql-cdc
Real-time Change Data Capture from MySQL binary logs with schema evolution tracking
## Overview
The MySQL CDC streaming source captures real-time changes from MySQL binary logs and processes them into structured records. It provides comprehensive CDC capabilities with automatic schema evolution tracking and primary key detection.
Captures changes as they happen in MySQL
Tracks table structure changes automatically
Automatically identifies and extracts primary key columns
Handles server ID conflicts and binlog file issues
## Configuration
### Required Parameters
| Parameter | Type | Description |
| ---------- | ------ | ------------------------------------------ |
| `host` | string | MySQL server hostname or IP address |
| `port` | number | MySQL server port (default: 3306) |
| `user` | string | MySQL username with replication privileges |
| `password` | string | MySQL password |
### Optional Parameters
| Parameter | Type | Default | Description |
| ---------------------------- | ------- | ------- | ------------------------------------------------------ |
| `connect_timeout` | number | 10 | Connection timeout in seconds |
| `charset` | string | utf8mb4 | Character set for connection |
| `databases` | array | - | Include only these databases |
| `tables` | array | - | Include only these tables |
| `ignore_databases` | array | - | Exclude these databases |
| `ignore_tables` | array | - | Exclude these tables |
| `server_id` | number | 54001 | Unique server ID for replication |
| `auto_generate_server_id` | boolean | true | Auto-generate unique server\_id if conflict |
| `resume_from_gtids` | string | - | Resume from specific GTID set |
| `start_log_file` | string | - | Resume from specific binlog file |
| `start_log_pos` | number | - | Resume from specific position in binlog file |
| `start_timestamp` | number | - | Resume from specific timestamp (Unix timestamp) |
| `include_ddl` | boolean | false | Include DDL events (CREATE, ALTER, DROP, etc.) |
| `include_heartbeat` | boolean | true | Include heartbeat events |
| `include_transaction_events` | boolean | true | Include transaction commit events |
| `include_gtid_events` | boolean | true | Include GTID events |
| `include_rotate_events` | boolean | false | Include binlog rotation events |
| `include_intvar_events` | boolean | true | Include integer variable events |
| `include_load_query_events` | boolean | true | Include LOAD DATA INFILE events |
| `include_table_map_events` | boolean | true | Include table map events (required for column mapping) |
| `heartbeat_seconds` | number | 1.0 | Heartbeat interval in seconds |
| `blocking` | boolean | true | Use blocking mode for event reading |
| `max_batch_size` | number | 100 | Maximum events per batch |
| `flush_interval_seconds` | number | 1.0 | Maximum time between batch flushes |
| `return_db_records_only` | boolean | true | Return only DB records with Mage timestamp columns |
### SSL Configuration
| Parameter | Type | Description |
| ---------- | ------ | ------------------------------- |
| `ssl.ca` | string | Certificate Authority file path |
| `ssl.cert` | string | Client certificate file path |
| `ssl.key` | string | Client private key file path |
## Record Formats
### Standard Event Format
When `return_db_records_only: false`:
```json theme={"system"}
{
"op": "c",
"schema": "mydb",
"table": "users",
"ts": 1640995200,
"tx": {
"gtid": "uuid:1-100",
"xid": 12345
},
"before": {
"id": 1,
"name": "John",
"email": "john@example.com"
},
"after": {
"id": 1,
"name": "John",
"email": "john@example.com"
},
"offset": {
"file": "mysql-bin.000001",
"pos": 1234,
"gtid": "uuid:1-100"
}
}
```
### DB Records Only Format
When `return_db_records_only: true`:
```json theme={"system"}
{
"data": {
"id": 1,
"name": "John",
"email": "john@example.com",
"_mage_created_at": "2021-12-31T23:00:00+00:00",
"_mage_updated_at": null,
"_mage_deleted_at": null
},
"metadata": {
"schema": "mydb",
"table": "users",
"key_columns": ["id"],
"operation": "INSERT"
}
}
```
**Note**: The `_mage_*` timestamp columns are returned as datetime objects (ISO 8601 format strings when serialized) in UTC timezone, not Unix timestamps. This ensures proper type handling when writing to databases.
## Event Types
* **WriteRowsEvent** (`op: "c"`): INSERT operations
* **UpdateRowsEvent** (`op: "u"`): UPDATE operations with before/after data
* **DeleteRowsEvent** (`op: "d"`): DELETE operations
* **QueryEvent** (`op: "ddl"`): DDL statements (CREATE, ALTER, DROP)
* Automatically clears table schema cache when DDL occurs
* Extracts table name from SQL for targeted cache clearing
* **XidEvent** (`op: "transaction"`): Transaction commit events
* Useful for maintaining transaction boundaries
* **GtidEvent** (`op: "gtid"`): Global Transaction Identifier events
* **RotateEvent** (`op: "rotate"`): Binlog file rotation events
* **HeartbeatLogEvent** (`op: "heartbeat"`): Replication heartbeat events
* **TableMapEvent** (`op: "table_map"`): Table structure metadata
* **IntvarEvent** (`op: "intvar"`): Integer variable changes
* **LoadQueryEvent** (`op: "begin_load_query"`/`op: "execute_load_query"`): LOAD DATA INFILE events
## Primary Key Detection
The source automatically detects and extracts primary key columns:
1. **Schema Discovery**: Queries `INFORMATION_SCHEMA` for table structure
2. **Primary Key Detection**: Identifies actual primary key columns from database constraints
3. **Caching**: Stores schema and primary key information for performance
4. **Key Extraction**: Extracts primary key column names from row events
```json Example Output theme={"system"}
{
"metadata": {
"key_columns": ["user_id", "tenant_id"]
}
}
```
**Note**: `key_columns` is a list of column names (not values) that can be used for deduplication or upsert operations in downstream sinks.
## Examples
```yaml Basic Setup theme={"system"}
connector_type: mysql
host: localhost
port: 3306
user: repl_user
password: password
databases: ["mydb"]
return_db_records_only: true
```
```yaml Production Setup theme={"system"}
connector_type: mysql
host: mysql-prod.company.com
port: 3306
user: cdc_user
password: secure_password
ssl:
ca: "/etc/ssl/mysql/ca.pem"
cert: "/etc/ssl/mysql/client-cert.pem"
key: "/etc/ssl/mysql/client-key.pem"
databases: ["production_db"]
ignore_tables: ["temp_*", "log_*"]
server_id: 1001
auto_generate_server_id: false
max_batch_size: 500
flush_interval_seconds: 2.0
include_ddl: true
return_db_records_only: true
```
```yaml Resume from Position theme={"system"}
connector_type: mysql
host: localhost
port: 3306
user: repl_user
password: password
start_log_file: "mysql-bin.000123"
start_log_pos: 456789
databases: ["mydb"]
return_db_records_only: true
```
## Prerequisites
### MySQL Server Configuration
```sql theme={"system"}
-- Enable binary logging
SET GLOBAL log_bin = ON;
-- Set binlog format to ROW
SET GLOBAL binlog_format = 'ROW';
-- Enable GTID (recommended)
SET GLOBAL gtid_mode = ON;
SET GLOBAL enforce_gtid_consistency = ON;
```
### User Permissions
```sql theme={"system"}
-- Create replication user
CREATE USER 'repl_user'@'%' IDENTIFIED BY 'password';
-- Grant replication privileges
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl_user'@'%';
-- Grant SELECT privileges for schema queries
GRANT SELECT ON information_schema.* TO 'repl_user'@'%';
GRANT SELECT ON mydb.* TO 'repl_user'@'%';
```
## Troubleshooting
1. **Server ID Conflict**: Enable `auto_generate_server_id: true`
2. **Binlog File Not Found**: Check if binlog file exists and is accessible
3. **Permission Denied**: Ensure user has replication privileges
4. **Connection Timeout**: Increase `connect_timeout` value
5. **Schema Not Found**: Ensure `include_table_map_events: true`
Enable debug logging to see detailed event processing:
```python theme={"system"}
import logging
logging.basicConfig(level=logging.DEBUG)
```
Monitor key metrics:
* **Batch Size**: Average events per batch
* **Flush Rate**: How often batches are flushed
* **Error Rate**: Failed events or connections
* **Lag**: Time between event occurrence and processing
## Integration with Generic IO Sink
The MySQL CDC source works seamlessly with the Generic IO sink, which provides:
* **Automatic Column Type Mapping**: Automatically maps `_mage_*` timestamp columns to appropriate database types (TIMESTAMP, DATETIME2, DateTime64, etc.) based on the target database
* **Metadata Interpolation**: Use metadata values from MySQL CDC events in sink configurations for dynamic routing and table naming
### Supported Databases
Generic IO Sink supports the following databases:
* **BigQuery**
* **ClickHouse**
* **DuckDB**
* **MySQL**
* **MSSQL**
* **Postgres**
### Metadata Interpolation
You can use metadata values from MySQL CDC events in your sink configuration using Python string formatting syntax:
* **`{schema}`**: Database/schema name from the event
* **`{table}`**: Table name from the event
* **`{key_columns}`**: List of primary key column names (e.g., `["id"]` or `["user_id", "tenant_id"]`)
When using `{key_columns}` in `unique_constraints`, it will be automatically converted from a string representation to a list. The format supports both Python-style (`"['id']"`) and JSON-style (`'["id"]'`) array strings.
### Example Configurations
```yaml BigQuery Example theme={"system"}
connector_type: bigquery
profile: default
config:
table_id: "{schema}.{table}" # Dynamic table names from metadata
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}" # Uses primary keys from metadata
```
```yaml Postgres Example theme={"system"}
connector_type: postgres
profile: default
config:
schema_name: "{schema}" # Dynamic schema from metadata
table_name: "{table}" # Dynamic table name from metadata
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}" # Uses primary keys from metadata
if_exists: append
```
```yaml MySQL Example theme={"system"}
connector_type: mysql
profile: default
config:
schema_name: "{schema}"
table_name: "{table}"
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}"
if_exists: append
```
```yaml Static Configuration theme={"system"}
connector_type: postgres
profile: default
config:
schema_name: public
table_name: cdc_events # Static table name
unique_constraints: ['id', 'event_id'] # Static list (string format also works: "['id', 'event_id']")
unique_conflict_method: UPDATE
if_exists: append
```
### How Metadata Interpolation Works
1. **Message Grouping**: Messages are automatically grouped by their interpolated config values. For example, if you use `table_name: "{schema}_{table}"`, messages from `mydb.users` will be grouped together and written to `mydb_users` table.
2. **Key Columns Interpolation**: When using `{key_columns}` in `unique_constraints`, the sink automatically:
* Converts the list to a string representation during interpolation
* Parses it back to a list (supports both `"['id']"` and `'["id"]'` formats)
* Uses it for upsert operations based on `unique_conflict_method`
**Example**: If you have a table `users` with primary key `id`, and you configure `unique_constraints: "{key_columns}"`, it will automatically use `["id"]` for upsert operations.
## Best Practices
1. **Use GTID**: Enable GTID for reliable resume capabilities
2. **Filter Events**: Disable unnecessary event types for performance
3. **Monitor Resources**: Watch memory and CPU usage
4. **Test Resume**: Verify checkpoint functionality
5. **Secure Connections**: Use SSL in production
6. **Regular Backups**: Backup checkpoint files
7. **Schema Validation**: Test with schema changes
8. **Use Generic IO Sink**: Leverage automatic type mapping and metadata interpolation for flexible data routing
9. **Metadata Interpolation**: Use `{schema}`, `{table}`, and `{key_columns}` for dynamic table routing and upsert configuration
## Timestamp Handling
The MySQL CDC source automatically converts Unix timestamps to datetime objects:
* **`_mage_created_at`**: Set to event timestamp (datetime) for INSERT operations
* **`_mage_updated_at`**: Set to event timestamp (datetime) for UPDATE operations
* **`_mage_deleted_at`**: Set to event timestamp (datetime) for DELETE operations
All timestamps are in UTC timezone and returned as datetime objects, which ensures:
* Proper type handling in downstream databases
* Automatic type conversion in Generic IO sink
* Consistent timezone handling across systems
## Limitations
* **MySQL 5.7+**: Requires MySQL 5.7 or later
* **ROW Format**: Requires `binlog_format = ROW`
* **Network Dependency**: Requires stable network connection
* **Memory Usage**: Schema caching uses memory
* **Binlog Retention**: Depends on MySQL binlog retention settings
* **Timezone**: All timestamps are in UTC timezone
# NATS.io - JetStream
Source: https://docs.mage.ai/guides/streaming/sources/nats
## Basic config
```yaml theme={"system"}
connector_type: 'nats'
server_url: 'nats://localhost:4222'
subject: 'test_subject'
```
# OracleDB
Source: https://docs.mage.ai/guides/streaming/sources/oracledb
Stream changes from OracleDB with OracleDB CDC.
Mage Pro streaming pipeline supports OracleDB CDC as the source. This document explains how to set up OracleDB CDC and integrate it into Mage's streaming pipeline using LogMiner.
## Basic Config
```yaml theme={"system"}
connector_type: oracledb
host: "oracledb.example.com"
port: 1521
service: "xepdb1"
password: "password"
user: "user"
mode: "thin" # Value can be `thin` or `thick`
table_names: [] # Filter table names
```
## Setting Up OracleDB for CDC
### **Step 1: Enable Supplemental Logging**
Run the following SQL commands to enable CDC:
```sql theme={"system"}
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
```
### **Step 2: Create a User for CDC**
```sql theme={"system"}
CREATE USER cdc_user IDENTIFIED BY YourPassword;
GRANT CONNECT, RESOURCE TO cdc_user;
GRANT EXECUTE ON DBMS_LOGMNR TO cdc_user;
GRANT EXECUTE ON DBMS_LOGMNR_D TO cdc_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO cdc_user;
GRANT SELECT ON V_$DATABASE TO cdc_user;
GRANT SELECT ON V_$ARCHIVED_LOG TO cdc_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO cdc_user;
GRANT SELECT ON V_$LOGMNR_PARAMETERS TO cdc_user;
GRANT SELECT ANY TABLE TO cdc_user;
```
### **Step 3: Start LogMiner Session**
```sql theme={"system"}
BEGIN
DBMS_LOGMNR.START_LOGMNR(
OPTIONS => DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG + DBMS_LOGMNR.NEW
);
END;
/
```
***
## Test using LogMiner for CDC
LogMiner reads redo logs to track **INSERT, UPDATE, DELETE** operations.
### **Mage Data Pipeline for LogMiner-Based CDC**
Create a data loader in Mage to pull real-time CDC data:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.oracledb import OracleDB
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_oracledb(*args, **kwargs):
query = """
SELECT SCN, OPERATION, TIMESTAMP, SQL_REDO
FROM V$LOGMNR_CONTENTS
WHERE SEG_NAME = 'YOUR_TABLE'
AND OPERATION IN ('INSERT', 'UPDATE', 'DELETE')
"""
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with OracleDB.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
# PostgreSQL CDC
Source: https://docs.mage.ai/guides/streaming/sources/postgres-cdc
Real-time Change Data Capture from PostgreSQL WAL with logical replication
## Overview
The PostgreSQL CDC streaming source captures real-time changes from PostgreSQL Write-Ahead Log (WAL) using logical replication (pgoutput) and processes them into structured records. It provides comprehensive CDC capabilities with automatic schema evolution tracking and primary key detection.
Captures changes as they happen in PostgreSQL
Uses PostgreSQL's native logical replication protocol
Automatically identifies and extracts primary key columns
Tracks Log Sequence Numbers for reliable resume capabilities
## Configuration
### Required Parameters
| Parameter | Type | Description |
| ---------- | ------ | ----------------------------------------------- |
| `host` | string | PostgreSQL server hostname or IP address |
| `port` | number | PostgreSQL server port (default: 5432) |
| `database` | string | Database name (default: postgres) |
| `user` | string | PostgreSQL username with replication privileges |
| `password` | string | PostgreSQL password |
### Optional Parameters
| Parameter | Type | Default | Description |
| ---------------------------- | ------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connect_timeout` | number | 10 | Connection timeout in seconds |
| `schemas` | array | - | Include only these schemas |
| `tables` | array | - | Include only these tables |
| `ignore_schemas` | array | - | Exclude these schemas |
| `ignore_tables` | array | - | Exclude these tables |
| `replication_slot` | string | mage\_slot | Logical replication slot name |
| `publication_name` | string | mage\_pub | Publication name for logical replication |
| `start_lsn` | string | - | Log Sequence Number to start from (e.g., '0/1234567') |
| `max_batch_size` | number | 100 | Maximum events per batch |
| `flush_interval_seconds` | number | 1.0 | Maximum time between batch flushes |
| `keep_alive_time` | number | 10.0 | Time between keep-alive messages in seconds |
| `heartbeat_table_schema` | string | - | Schema name for heartbeat table (e.g., 'public') |
| `heartbeat_table_name` | string | - | Table name for heartbeat (e.g., 'cdc\_status') |
| `heartbeat_interval_seconds` | number | 60.0 | How often to write heartbeat in seconds |
| `heartbeat_update_query` | string | - | Custom SQL query to update heartbeat (overrides default). **WARNING**: Query is executed directly without sanitization. Only use queries from trusted sources to prevent SQL injection vulnerabilities. |
| `return_db_records_only` | boolean | true | Return only DB records with Mage timestamp columns |
### SSL Configuration
| Parameter | Type | Description |
| ----------------- | ------ | ---------------------------------------------------------------- |
| `ssl.sslmode` | string | SSL mode (e.g., 'require', 'prefer', 'verify-ca', 'verify-full') |
| `ssl.sslcert` | string | Client certificate file path |
| `ssl.sslkey` | string | Client private key file path |
| `ssl.sslrootcert` | string | Root certificate file path |
## Record Formats
### DB Records Only Format
When `return_db_records_only: true` (default):
```json theme={"system"}
{
"data": {
"id": 1,
"name": "John",
"email": "john@example.com",
"_mage_created_at": "2021-12-31T23:00:00+00:00",
"_mage_updated_at": null,
"_mage_deleted_at": null
},
"metadata": {
"schema": "public",
"table": "users",
"key_columns": ["id"],
"operation": "INSERT",
"lsn": "0/1234567"
}
}
```
**Note**: The `_mage_*` timestamp columns are returned as datetime objects (ISO 8601 format strings when serialized) in UTC timezone, not Unix timestamps. This ensures proper type handling when writing to databases.
## Event Types
* **Insert** (`operation: "INSERT"`): INSERT operations
* **Update** (`operation: "UPDATE"`): UPDATE operations with before/after data
* **Delete** (`operation: "DELETE"`): DELETE operations
These events contain the actual data changes.
* **Relation** (`operation: "RELATION"`): Table structure metadata
* **Truncate** (`operation: "TRUNCATE"`): TRUNCATE TABLE operations
Automatically tracks table schema changes for column name mapping.
* **Begin** (`operation: "BEGIN"`): Transaction begin events
* **Commit** (`operation: "COMMIT"`): Transaction commit events
Useful for maintaining transaction boundaries and ensuring data consistency.
## Primary Key Detection
The source automatically detects and extracts primary key columns:
1. **Schema Discovery**: Queries `INFORMATION_SCHEMA` for table structure
2. **Primary Key Detection**: Identifies actual primary key columns from database constraints
3. **Caching**: Stores schema and primary key information for performance
4. **Key Extraction**: Extracts primary key column names from row events
```json Example Output theme={"system"}
{
"metadata": {
"key_columns": ["user_id", "tenant_id"]
}
}
```
**Note**: `key_columns` is a list of column names (not values) that can be used for deduplication or upsert operations in downstream sinks.
## Examples
```yaml Basic Setup theme={"system"}
connector_type: postgres
host: localhost
port: 5432
database: mydb
user: repl_user
password: password
schemas: ["public"]
return_db_records_only: true
```
```yaml Production Setup theme={"system"}
connector_type: postgres
host: postgres-prod.company.com
port: 5432
database: production_db
user: cdc_user
password: secure_password
ssl:
sslmode: require
sslcert: "/etc/ssl/postgres/client-cert.pem"
sslkey: "/etc/ssl/postgres/client-key.pem"
sslrootcert: "/etc/ssl/postgres/ca.pem"
schemas: ["public", "analytics"]
ignore_tables: ["temp_*", "log_*"]
replication_slot: prod_cdc_slot
publication_name: prod_cdc_pub
max_batch_size: 500
flush_interval_seconds: 2.0
return_db_records_only: true
```
```yaml Resume from LSN theme={"system"}
connector_type: postgres
host: localhost
port: 5432
database: mydb
user: repl_user
password: password
start_lsn: "0/1234567"
schemas: ["public"]
return_db_records_only: true
```
```yaml With Heartbeat Table theme={"system"}
connector_type: postgres
host: aiven-postgres.company.com
port: 5432
database: mydb
user: cdc_user
password: password
heartbeat_table_schema: public
heartbeat_table_name: cdc_status
heartbeat_interval_seconds: 60.0
schemas: ["public"]
return_db_records_only: true
```
```yaml With Custom Heartbeat Query theme={"system"}
connector_type: postgres
host: aiven-postgres.company.com
port: 5432
database: mydb
user: cdc_user
password: password
heartbeat_table_schema: public
heartbeat_table_name: cdc_status
heartbeat_interval_seconds: 60.0
heartbeat_update_query: "UPDATE public.cdc_status SET timestamp = NOW(), last_updated_by = 'mage_cdc' WHERE id = 1"
schemas: ["public"]
return_db_records_only: true
```
## Heartbeat Table
The heartbeat table is used to prevent WAL logs from filling up in managed PostgreSQL services (like Aiven). When configured, the source periodically writes to this table to keep the replication slot active.
### Schema
The heartbeat table is automatically created if it doesn't exist. It has the following schema:
```sql theme={"system"}
CREATE TABLE {schema}.{table} (
id INTEGER PRIMARY KEY DEFAULT 1,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
**Columns:**
* `id` (INTEGER PRIMARY KEY): Primary key column with default value of 1
* `timestamp` (TIMESTAMP): Timestamp column that gets updated with the current timestamp on each heartbeat write
### How It Works
1. **Automatic Creation**: If the table doesn't exist, it's automatically created when the source initializes
2. **Periodic Updates**: The source updates the `timestamp` column every `heartbeat_interval_seconds` (default: 60 seconds)
3. **Default UPSERT Pattern**: Uses `INSERT ... ON CONFLICT DO UPDATE` to update the single row with `id=1`, or falls back to `UPDATE`/`INSERT` if the database doesn't support `ON CONFLICT`
4. **Custom Query**: You can override the default heartbeat update query by providing `heartbeat_update_query` in the configuration
### Security Warning for Custom Heartbeat Query
**Important Security Notice**: The `heartbeat_update_query` parameter executes SQL queries directly without sanitization or parameterization. This could introduce SQL injection vulnerabilities if the query is sourced from untrusted input.
**Best Practices:**
* Only use queries from trusted, validated sources
* Never construct queries dynamically from user input or external APIs
* Validate that the query is not empty or whitespace-only before use
* Consider using the default heartbeat mechanism unless you have specific requirements
* Review and test custom queries thoroughly before deploying to production
The source performs basic validation to ensure the query is not empty or whitespace-only, but does not perform SQL injection protection. Always ensure your configuration management system properly secures and validates this parameter.
### Manual Creation
If you prefer to create the table manually, you can use:
```sql theme={"system"}
CREATE TABLE public.cdc_status (
id INTEGER PRIMARY KEY DEFAULT 1,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Insert initial row
INSERT INTO public.cdc_status (id, timestamp)
VALUES (1, CURRENT_TIMESTAMP);
```
**Note**: The table must have a primary key or unique constraint on the `id` column for the UPSERT pattern to work correctly.
## Prerequisites
### PostgreSQL Server Configuration
```sql theme={"system"}
-- Enable logical replication
ALTER SYSTEM SET wal_level = logical;
-- Restart PostgreSQL for changes to take effect
-- Verify WAL level
SHOW wal_level;
-- Should return: logical
```
### Create Replication Slot
```sql theme={"system"}
-- Create logical replication slot
SELECT pg_create_logical_replication_slot('mage_slot', 'pgoutput');
```
### Create Publication
```sql theme={"system"}
-- Create publication for all tables in a schema
CREATE PUBLICATION mage_pub FOR ALL TABLES;
-- Or create publication for specific tables
CREATE PUBLICATION mage_pub FOR TABLE users, orders, products;
-- Or create publication for specific schemas
CREATE PUBLICATION mage_pub FOR TABLES IN SCHEMA public, analytics;
```
### User Permissions
```sql theme={"system"}
-- Create replication user
CREATE USER repl_user WITH PASSWORD 'password';
-- Grant replication privileges
ALTER USER repl_user WITH REPLICATION;
-- Grant connect privileges
GRANT CONNECT ON DATABASE mydb TO repl_user;
-- Grant usage on schemas
GRANT USAGE ON SCHEMA public TO repl_user;
GRANT USAGE ON SCHEMA analytics TO repl_user;
-- Grant SELECT privileges for schema queries
GRANT SELECT ON ALL TABLES IN SCHEMA public TO repl_user;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO repl_user;
-- Grant SELECT on information_schema for metadata queries
GRANT SELECT ON ALL TABLES IN SCHEMA information_schema TO repl_user;
```
## Troubleshooting
1. **Replication Slot Not Found**: Create the replication slot using `pg_create_logical_replication_slot()`
2. **Publication Not Found**: Create the publication using `CREATE PUBLICATION`
3. **Permission Denied**: Ensure user has `REPLICATION` privilege and `SELECT` on tables
4. **WAL Level Not Logical**: Set `wal_level = logical` and restart PostgreSQL
5. **Connection Timeout**: Increase `connect_timeout` value
6. **LSN Format Error**: Ensure LSN is in format 'X/Y' (e.g., '0/1234567')
Check replication slot status:
```sql theme={"system"}
SELECT * FROM pg_replication_slots WHERE slot_name = 'mage_slot';
```
Check publication status:
```sql theme={"system"}
SELECT * FROM pg_publication WHERE pubname = 'mage_pub';
SELECT * FROM pg_publication_tables WHERE pubname = 'mage_pub';
```
Monitor key metrics:
* **Batch Size**: Average events per batch
* **Flush Rate**: How often batches are flushed
* **Error Rate**: Failed events or connections
* **Lag**: Time between event occurrence and processing
* **LSN Progress**: Track Log Sequence Number progress
Check replication lag:
```sql theme={"system"}
SELECT
slot_name,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS replication_lag
FROM pg_replication_slots
WHERE slot_name = 'mage_slot';
```
## Integration with Generic IO Sink
The PostgreSQL CDC source works seamlessly with the Generic IO sink, which provides:
* **Automatic Column Type Mapping**: Automatically maps `_mage_*` timestamp columns to appropriate database types (TIMESTAMP, DATETIME2, DateTime64, etc.) based on the target database
* **Metadata Interpolation**: Use metadata values from PostgreSQL CDC events in sink configurations for dynamic routing and table naming
### Supported Databases
Generic IO Sink supports the following databases:
* **BigQuery**
* **ClickHouse**
* **DuckDB**
* **MySQL**
* **MSSQL**
* **Postgres**
### Metadata Interpolation
You can use metadata values from PostgreSQL CDC events in your sink configuration using Python string formatting syntax:
* **`{schema}`**: Schema name from the event
* **`{table}`**: Table name from the event
* **`{key_columns}`**: List of primary key column names (e.g., `["id"]` or `["user_id", "tenant_id"]`)
When using `{key_columns}` in `unique_constraints`, it will be automatically converted from a string representation to a list. The format supports both Python-style (`"['id']"`) and JSON-style (`'["id"]'`) array strings.
### Example Configurations
```yaml BigQuery Example theme={"system"}
connector_type: bigquery
profile: default
config:
table_id: "{schema}.{table}" # Dynamic table names from metadata
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}" # Uses primary keys from metadata
```
```yaml Postgres Example theme={"system"}
connector_type: postgres
profile: default
config:
schema_name: "{schema}" # Dynamic schema from metadata
table_name: "{table}" # Dynamic table name from metadata
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}" # Uses primary keys from metadata
if_exists: append
```
```yaml MySQL Example theme={"system"}
connector_type: mysql
profile: default
config:
schema_name: "{schema}"
table_name: "{table}"
unique_conflict_method: UPDATE
unique_constraints: "{key_columns}"
if_exists: append
```
```yaml Static Configuration theme={"system"}
connector_type: postgres
profile: default
config:
schema_name: public
table_name: cdc_events # Static table name
unique_constraints: ['id', 'event_id'] # Static list (string format also works: "['id', 'event_id']")
unique_conflict_method: UPDATE
if_exists: append
```
### How Metadata Interpolation Works
1. **Message Grouping**: Messages are automatically grouped by their interpolated config values. For example, if you use `table_name: "{schema}_{table}"`, messages from `public.users` will be grouped together and written to `public_users` table.
2. **Key Columns Interpolation**: When using `{key_columns}` in `unique_constraints`, the sink automatically:
* Converts the list to a string representation during interpolation
* Parses it back to a list (supports both `"['id']"` and `'["id"]'` formats)
* Uses it for upsert operations based on `unique_conflict_method`
**Example**: If you have a table `users` with primary key `id`, and you configure `unique_constraints: "{key_columns}"`, it will automatically use `["id"]` for upsert operations.
## Best Practices
1. **Use Replication Slots**: Always use replication slots for reliable CDC tracking
2. **Monitor WAL Growth**: Monitor WAL size and ensure proper cleanup
3. **Filter Events**: Use schema/table filters to reduce processing overhead
4. **Monitor Resources**: Watch memory and CPU usage
5. **Test Resume**: Verify checkpoint functionality with LSN tracking
6. **Secure Connections**: Use SSL in production
7. **Regular Backups**: Backup checkpoint files
8. **Schema Validation**: Test with schema changes
9. **Use Generic IO Sink**: Leverage automatic type mapping and metadata interpolation for flexible data routing
10. **Metadata Interpolation**: Use `{schema}`, `{table}`, and `{key_columns}` for dynamic table routing and upsert configuration
11. **Heartbeat Tables**: Use heartbeat tables for managed services (like Aiven) to prevent WAL log filling
## Timestamp Handling
The PostgreSQL CDC source automatically adds Mage timestamp columns:
* **`_mage_created_at`**: Set to event timestamp (datetime) for INSERT operations
* **`_mage_updated_at`**: Set to event timestamp (datetime) for UPDATE operations
* **`_mage_deleted_at`**: Set to event timestamp (datetime) for DELETE operations
All timestamps are in UTC timezone and returned as datetime objects, which ensures:
* Proper type handling in downstream databases
* Automatic type conversion in Generic IO sink
* Consistent timezone handling across systems
## Limitations
* **PostgreSQL 10+**: Requires PostgreSQL 10 or later for logical replication
* **WAL Level**: Requires `wal_level = logical`
* **Network Dependency**: Requires stable network connection
* **Memory Usage**: Schema caching uses memory
* **WAL Retention**: Depends on PostgreSQL WAL retention settings
* **Timezone**: All timestamps are in UTC timezone
* **Replication Slot**: Requires a dedicated replication slot per consumer
* **Publication**: Tables must be added to a publication to be replicated
# RabbitMQ
Source: https://docs.mage.ai/guides/streaming/sources/rabbitmq
## Basic config
```yaml theme={"system"}
connector_type: 'rabbitmq'
connection_host: 'localhost'
connection_port: 5672
queue_name: 'default'
username: 'guest'
password: 'guest'
amqp_url_virtual_host: '%2f'
url_protocol: 'amqp'
```
You can even control some of the RabbitMQ consumer configurations using the following fields.
```yaml theme={"system"}
configure_consume: True
consume_config:
auto_ack: False
exclusive: False
inactivity_timeout: 10.0
```
# ๐ Local stream to dbt transform
Source: https://docs.mage.ai/guides/streaming/tutorials/magic-devcontainer
This project uses sample data from the Google NYC Taxi Pubsub to create a streaming pipeline in Mage that reads and transforms a sample stream, ultimately writing output to a SCD Type-2 table in Postgres.
## ๐ Intro
This project uses sample data fromย [the Google NYC Taxi Pubsub](https://github.com/googlecodelabs/cloud-dataflow-nyc-taxi-tycoon#public-pubsub-data-stream) to create a streaming pipeline in Mage that:
* Reads and transforms the stream
* Writes the stream as parquet to S3
Then, it creates a batch pipeline to:
* Read parquet files from S3.
* Execute a dbt model to store the raw data in a local Postgres instance.
* Execute a dbt model to transform the raw data to an SCD Type-2 table, logging the status updates for each ride.
Read more or make updates to the repo [here](https://github.com/mage-ai/magic-devcontainer/tree/main).
## ๐ ๏ธ Configuration
**Prerequisites**
* Docker
* VSCode
* No VSCode? That's ok: you can use `docker compose` too
* An AWS account (or other cloud storage provider) to store the stream
Let's get started ๐
1. `git clone https://github.com/mage-ai/pubsub-devcontainer-demo`
2. Open the cloned file in VSCode.
3. Create a `.env` file with the requisite variables. See the `.env.example` file for guidance.
4. Select the prompt to `reopen in Container` to start the devcontainer *or* open the command prompt and select `Devcontainers: Rebuild and Reopen in Container`. This will build a devcontainer (a Docker container) and install all dependencies to your project. This may take a few minutes.
1. If you don't have VSCode installed, simply run `docker compose up` from the command line.
5. Navigate to `localhost:6789` when the container is ready. You should see the Mage UI!
## ๐โโ๏ธ Wait, what's happening?
By performing the above, VSCode is creating an isolated environment, installing a few extensions + building and running Docker as defined in `docker-compose.yaml`.
We're using [Devcontainers](https://containers.dev/) to create a consistent development environment for our project. This means that we can be sure that our code will run in the same environment as our teammates and that we can easily share our code with others.
As for the stream itself, you might notice `kafka` and `zookeeper` in the `docker-compose.yaml` file. These are the tools we use to manage the stream.
We're using `kafka` to manage the stream and `zookeeper` to manage `kafka`. You can read more about these tools [here](https://kafka.apache.org/).
The *actual* stream is managed by the `stream` container, which spins up a python container and executes `./stream/bin/send_stream.py`โ this is the script that reads from the pubsub topic data (`./stream/data/taxi-stream.json`).
Data will begin streaming 30 seconds after the container starts.
## ๐ค Stream data from a dummy source to S3
1. Navigate to the "Pipelines" tab in the Mage UI.
2. Two pipelines existโ `kafka_demo` and `dbt_demo`โ click `kafka_demo` then the "code" icon in the top left to open the pipeline code.
3. Double check the configโ these should pull from your `.env` file.
4. Click `Execute pipeline` in the bottom right to run your stream. You should see data flowing!
Nice! We've got a working stream. You can click `Cancel pipeline` after some sample data has been loaded or let the stream run.
Behind the scenes, we're kicking off a Kafka + Zookeeper instance, creating a topic, and writing data to that topic. Mage is then pulling in this "simulated" stream, transforming it, and writing to S3!
## ๐งฑ Run dbt transformations
This container builds a postgres database alongside our Mage app for us to run transformations in dbt. Navigate to the `pipelines` page and select `dbt demo`. Select the code icon.
* The first cell reads in the data from our stream (stored in s3 or your cloud provider of choice). Click the play icon or hit CMD + Enter to run.
* You might notice there are some tests in this cellโ Mage incorporates *runtime tests* to check the quality of your data. What tests are we running here?
* The next cell writes our data to a postgres table using `dbt`. Note how we can read from our mage data *directly*, without the need for an intermediate table.
* Finally, we apply a transform to build an [SCD Type-2](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) table from our dataset. This is also known as a "change-log" format.
* The final cell pulls a sample ride! Do the pickup, enroute, & dropoff times lineup?
# Kafka streaming pipeline
Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-pipeline
Build pipelines that ingest data from event streaming sources like Kafka.
## Set up Kafka
If you donโt have Kafka already setup, here is a quick guide on how to run and
use Kafka locally:
### Using Kafka locally
1. In your terminal, clone this repository:
`git clone https://github.com/wurstmeister/kafka-docker.git`.
2. Change directory into that repository: `cd kafka-docker`.
3. Edit the `docker-compose.yml` file to match this:
```yaml theme={"system"}
version: "2"
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
container_name: docker_kafka
ports:
- "9092:9092"
expose:
- "9093"
environment:
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
```
4. Start Docker:
```bash theme={"system"}
docker-compose up
```
If you encounter the error `command not found: docker-compose`,
try running the following command instead:
```bash theme={"system"}
docker compose up
```
5. Start a terminal session in the running container:
```bash theme={"system"}
docker exec -i -t -u root $(docker ps | grep docker_kafka | cut -d' ' -f1) /bin/bash
```
6. Create a topic:
```bash theme={"system"}
$KAFKA_HOME/bin/kafka-topics.sh --create --partitions 4 --bootstrap-server kafka:9092 --topic test
```
7. List all available topics in Kafka instance:
```bash theme={"system"}
$KAFKA_HOME/bin/kafka-topics.sh --bootstrap-server kafka:9092 --list
```
8. Start a producer on topic named `test`:
```bash theme={"system"}
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic=test
```
9. Send messages to the topic named `test` by typing the following JSON strings in the
terminal (Note that Kafka messages in Mage are assumed to be in JSON format):
```bash theme={"system"}
>{ "hello": 1 }
>{ "this is a test": 1 }
>{ "test": 1 }
>{ "test": 2 }
>{ "test": 3 }
```
10. Open another terminal and start a consumer on the topic named `test`:
```bash theme={"system"}
$KAFKA_HOME/bin/kafka-console-consumer.sh --from-beginning --bootstrap-server kafka:9092 --topic=test
```
11. The output should look something like this:
```bash theme={"system"}
{ "hello": 1 }
{ "this is a test": 1 }
{ "test": 1 }
{ "test": 2 }
{ "test": 3 }
```
Original
[source](https://towardsdatascience.com/introduction-to-kafka-stream-processing-in-python-e30d34bf3a12)
of instructions.
***
## Build streaming pipeline
### Start Mage
#### Using Kafka locally in a Docker container
Start Mage using Docker. Run the following command to run Docker in network
mode:
```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
--env AWS_ACCESS_KEY_ID=your_access_key_id \
--env AWS_SECRET_ACCESS_KEY=your_secret_access_key \
--env AWS_REGION=your_region \
--network kafka-docker_default \
mageai/mageai /app/run_app.sh mage start default_repo
```
Change the environment variables argument depending on your cloud provider.
If the network named `kafka-docker_default` doesnโt exist, create a new network:
```bash theme={"system"}
docker network create -d bridge kafka-docker_default
```
Check that it exists:
```bash theme={"system"}
docker network ls
```
***
If you canโt connect to Kafka locally in a Docker container using Mage in a
Docker container, do the following:
1. Clone Mage: `git clone https://github.com/mage-ai/mage-ai.git`.
2. Change directory into Mage: `cd mage-ai`.
3. Edit the `docker-compose.yml` file to match this:
```yaml theme={"system"}
version: '3'
services:
server:
... (original config)
networks:
- kafka
app:
... (original config)
networks:
kafka:
name: kafka-docker_default
external: true
```
4. Run the following script in your terminal: `./scripts/dev.sh`.
This will run Mage in development mode; which runs it in a Docker container
using `docker compose` instead of `docker run`.
#### Using Kafka without a Docker container
Start Mage using Docker. If you havenโt done this before, refer to the
[setup guide](/getting-started/setup).
### Create a new pipeline
1. Open Mage in your browser.
2. Click `+ New pipeline`, then select `Streaming`.
3. Add a data loader block, select `Kafka`, and paste the following:
```yaml theme={"system"}
connector_type: kafka
bootstrap_server: "localhost:9092"
topic: test
consumer_group: unique_consumer_group
include_metadata: false
api_version: 0.10.2
batch_size: 100
```
1. By default, the `bootstrap_server` is set to `localhost:9092`. If youโre
running Mage in a docker container, the `bootstrap_server` should be
`kafka:9093`.
2. Messages are consumed from source in micro batch mode for better efficiency.
The default batch size is 100. You can adjust the batch size in the source config.
4. Add a transformer block and paste the following:
```python theme={"system"}
from typing import Dict, List
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def transform(messages: List[Dict], *args, **kwargs):
for msg in messages:
print(msg)
return messages
```
5. Add a data exporter block, select `OpenSearch` and paste the following:
```yaml theme={"system"}
connector_type: opensearch
host: https://search-something-abcdefg123456.us-west-1.es.amazonaws.com/
index_name: python-test-index
```
1. Change the `host` to match your OpenSearch domainโs endpoint.
2. Change the `index_name` to match the index you want to export data into.
***
## Test pipeline
Open the streaming pipeline you just created, and in the right side panel near
the bottom, click the button `Execute pipeline` to test the pipeline.
You should see an output like this:
```
[streaming_pipeline_test] [KafkaSource] Start initializing consumer.
[streaming_pipeline_test] [KafkaSource] Finish initializing consumer.
[streaming_pipeline_test] [KafkaSource] Test connection successfully.
[streaming_pipeline_test] [KafkaSource] Start consuming messages in batches.
```
### Publish messages using Python
1. Open a terminal on your local workstation.
2. Install `kafka-python`:
```bash theme={"system"}
pip install kafka-python
```
3. Open a Python shell and write the following code to publish messages:
```python theme={"system"}
from kafka import KafkaProducer
from random import random
import json
topic = 'test'
producer = KafkaProducer(
bootstrap_servers='kafka:9093',
)
def publish_messages(limit):
for i in range(limit):
data = {
'title': 'test_title',
'director': 'Bennett Miller',
'year': '2011',
'rating': random(),
}
producer.send(topic, json.dumps(data).encode('utf-8'))
publish_messages(5)
```
Once you run the code snippet above, go back to your streaming pipeline in Mage
and the output should look like this:
```
[streaming_pipeline_test] Start initializing kafka consumer.
[streaming_pipeline_test] Finish initializing kafka consumer.
[streaming_pipeline_test] Start consuming messages from kafka.
[streaming_pipeline_test] [Kafka] Receive message 2:16: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.7010424523477785}', time=1665618592.226788
[streaming_pipeline_test] [Kafka] Receive message 0:16: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.7886308380991354}', time=1665618592.2268753
[streaming_pipeline_test] [Kafka] Receive message 0:17: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.0673276352704153}', time=1665618592.2268832
[streaming_pipeline_test] [Kafka] Receive message 3:10: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.37935417366095525}', time=1665618592.2268872
[streaming_pipeline_test] [Kafka] Receive message 3:11: v=b'{"title": "test_title", "director": "Bennett Miller", "year": "2011", "rating": 0.21110511524126563}', time=1665618592.2268918
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7010424523477785}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7886308380991354}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.0673276352704153}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.37935417366095525}
[streaming_pipeline_test] {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.21110511524126563}
[streaming_pipeline_test] [Opensearch] Batch ingest data [{'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7010424523477785}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.7886308380991354}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.0673276352704153}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.37935417366095525}, {'title': 'test_title', 'director': 'Bennett Miller', 'year': '2011', 'rating': 0.21110511524126563}], time=1665618592.2294626
```
### Consume messages using Python
If you want to programmatically consume messages from a Kafka topic, here is a
code snippet:
```python theme={"system"}
from kafka import KafkaConsumer
import time
topic = 'test'
consumer = KafkaConsumer(
topic,
group_id='test',
bootstrap_servers='kafka:9093',
)
for message in consumer:
print(f"{message.partition}:{message.offset}: v={message.value}, time={time.time()}")
```
***
## Run in production
1. [Create a trigger](/design/data-pipeline-management#create-trigger).
2. Once trigger is created, click the `Start trigger` button at the top
of the page to make the streaming pipeline active.
# RabbitMQ streaming pipeline
Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-pipeline-rabbitmq
Build pipelines that ingest data from event streaming sources like RabbitMQ.
## Set up RabbitMQ
If you don't have RabbitMQ already setup, here is a quick guide on how to run and
use RabbitMQ locally
### Using RabbitMQ locally
1. In your terminal, execute this command:
`docker run --rm -it -p 15672:15672 -p 5672:5672 rabbitmq:3-management`.
(This should start the RabbitMQ server on port `5672` and a management UI on port `15672`)
### Create a Queue (via RabbitMQ management)
1. Log into your RabbitMQ management UI ( if you just created one, default user/password is guest/guest)
2. Go to "Queues" tab
3. Click on "Add a new queue", define your settings and click on "Add Queue"
4. Your queue should be ready to be used!
### Create a Queue (via python Pika)
1. Install pika in your favorite dependency manager:
`pipenv install pika`
2. Execute this snippet:
```Python theme={"system"}
import pika
connection = pika.BlockingConnection(
pika.ConnectionParameters(host=localhost,port=5672))
# in case of failure connection, uncomment this code
# username = guest
# password = guest
# connection_host = localhost
# connection_port = 5672
# vt_host = r'%2f'
# generated_url = f"amqp://{username}:{password}@" \
# f"{connection_host}:{connection_port}/{vt_host}"
# create_connection = pika.BlockingConnection(
# pika.URLParameters(
# generated_url
# )
# )
channel = connection.channel()
channel.queue_declare(queue='magic_queue', durable=True) # durable is optional
```
3. Your queue should be ready to be used!
## Start Mage
### Create a new pipeline
1. Open Mage in your browser.
2. Click `+ New pipeline`, then select `Streaming`.
3. Add a data loader block, select `RabbitMQ`, and paste the following:
```yaml theme={"system"}
connector_type: 'rabbitmq'
connection_host: localhost
connection_port: 5672
queue_name: default
configure_consume: False
username: 'guest'
password: 'guest'
amqp_url_virtual_host: '%2f'
```
4. Add a transformer block and paste the following:
```python theme={"system"}
from typing import Dict, List
import logging
import json
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
@transformer
def transform(messages: List[Dict], *args, **kwargs):
kwargs['channel'].basic_ack(messages[0].delivery_tag)
print(messages.body.decode())
return json.dumps({'message':messages.body.decode()})
```
5. Add a data exporter block, select `OpenSearch` and paste the following:
```yaml theme={"system"}
connector_type: opensearch
host: https://search-something-abcdefg123456.us-west-1.es.amazonaws.com/
index_name: python-test-index
```
1. Change the `host` to match your OpenSearch domainโs endpoint.
2. Change the `index_name` to match the index you want to export data into.
***
## Test pipeline
Open the streaming pipeline you just created, and in the right side panel near
the bottom, click the button Execute pipeline to test the pipeline.
You should see an output like this:
```
[RabbitMQSource] Trying to connect on {your url}
[RabbitMQSource] Connected on broker
[RabbitMQSource] Start consuming messages.
```
### Publish messages using RabbitMQ Management UI
1. Log into your RabbitMQ management UI ( if you just created one, default user/password is guest/guest)
2. Go to "Queues" tab, and click on you desired Queue
3. Open "Publish Message" dropdown and write "Hello World!" on the payload block
4. Click on the black button called "Publish Message". This message should be seen on both mage-ai and your opensearch index!
### Publish messages using Python Pika
1. Execute this snippet:
```python theme={"system"}
import json
import pika
connection = pika.BlockingConnection(
pika.ConnectionParameters(host='insert_host',port=5672))
# in case of failure connection, uncomment this code
# username = guest
# password = guest
# connection_host = localhost
# connection_port = 5672
# vt_host = r'%2f'
# generated_url = f"amqp://{username}:{password}@" \
# f"{connection_host}:{connection_port}/{vt_host}"
# create_connection = pika.BlockingConnection(
# pika.URLParameters(
# generated_url
# )
# )
channel = connection.channel()
message = "Hello World!"
channel.basic_publish(
exchange='',
routing_key='magic_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode=pika.spec.PERSISTENT_DELIVERY_MODE
)
)
print(" [x] Sent ")
```
2. This message should be seen on both mage-ai and your OpenSearch index!
# Stateful Streaming Pipeline
Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-stateful-store
Create and use stateful store in a streaming pipeline.
Stateful store now is supported in the streaming pipeline. You can access it through the
`state_store` object in the `kwargs` parameter. Redis is the only supported state store.
## Configuration
State store configuration is required to set up in metadata.yml in order to enable stateful store in a streaming pipeline.
```yaml theme={"system"}
state_store_config:
type: 'REDIS'
redis:
redis_url: 'redis_url'
```
This configuration ensures that the pipeline can access and utilize Redis as its state store.
## Usage
Code example to access state store in your streaming pipeline:
```python theme={"system"}
state_store = kwargs['state_store']
state_store.set_state('string_key', 'Transformer Hello, Redis!')
state_store.set_state('int_key', 123)
state_store.set_state('list_key', [1, 2, 3,4, 5])
state_store.set_state('dict_key', {'key1': 'value1', 'key2': 'value2'})
```
```python theme={"system"}
state_store = kwargs['state_store']
state_store.get_state('string_key')
state_store.get_state('list_key')
state_store.get_state('int_key')
state_store.get_state('dict_key')
```
## Window Aggregation
The Window Aggregation Block is now supported in streaming pipeline, allowing for the aggregation of input data over a specified window of time and send it to down stream blocks.
### Usage
When you create a transformer and select the Window Aggregation type, it creates a window aggregation block with the following configuration:
```yaml theme={"system"}
transformer_type: window_aggregation
method: time
type: tumbling
interval_window: 60 # seconds
advance_window: 10
```
You can update `interval_window` with the desired interval.
In the meantime, you can add the following blocks after the window aggregation block.
These blocks will be triggered periodically based on the configuration.
For example, you can add a transformer block to report those data into monitoring system.
It can be used as alerts to your pipeline.
### How It Works
The window block is registered during the streaming pipeline execution with the specified configuration. An independent thread is created to manage each window block and trigger it periodically based on the configuration. When triggered, the downstream blocks are also executed accordingly.
***
๐ **With the stateful store and window aggregation support, your streaming pipeline can now efficiently manage and process stateful data.**
# Mage tips & tricks
Source: https://docs.mage.ai/guides/tips-and-tricks
Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.
Mage has native version control for your files, so you'll never lose work again!
Check it out in our docs [here](/getting-started/file-versions) or see the video below for a tutorial.
Mage integrates with DuckDB! Watch the video below to learn more or read on [here](/integrations/databases/DuckDB).
Mage has an integration with the OpenAI API. Watch the video to learn about how you can generate pipelines using the power of AI.
See our full docs [here](/development/ai/overview) for more info.
Mage supports streaming data to any of our destinations. Watch the video below to learn about configuring a streaming pipeline in under a minute!
See our full docs [here](/streaming/overview) for more info.
Watch this video to learn how you can use Conditional Blocks to add conditional execution and branching logic to your next data pipeline!
See our full docs [here](/design/blocks/conditionals) for more info.
Watch this video to learn how you can use Replica Blocks to add conditional execution and branching logic to your next data pipeline!
See our full docs [here](/guides/blocks/replicate-blocks) for more info.
The Mage pipeline editor can be used in a single, notebook-style interface *or* in a split-pane view, with block output displayed on the right and blocks themselves on the left.
Split pane view can be enabled by selecting the "split" icon in the pipeline editor, as shown below:
Can't wait to try it out? Check out the video below for a demo!
This week's Mage tip is all about organization!
Did you know you can tag, group, and filter your pipelines from the overview page? Watch the video below to learn more ๐
Read more in our docs [here](/pipelines/pipeline-tagging)!
We've received a ton of feedback from our users, and we've been hard at work making updates to the tree view based on your suggestions. So, what's new? Let me give you the lowdown. ๐
๐น Enhanced Functionality: The tree view now comes with a whole right-click menu that allows you to easily run, rename, insert, or delete blocks. You have more control than ever before!
๐น Seamless Connections: You can now remove and add connections with a simple click. Plus, you can insert new blocks between existing ones effortlessly.
๐น Enhanced Visualizations: With the new tree view, you get grouping and better lineage for certain block types ๐
Not only does the new tree view offer improved functionality, but it also makes your pipelines look absolutely stunning! After all, our goal is to make data beautiful. ๐ซ
What are dynamic blocks? Think of them as the magical way to create tasks dynamically in our tool. If you're familiar with Airflow and the concept of Dynamic Task Mapping, this is pretty similar.
If you have a bunch of small tasks that you want to execute in parallel, Dynamic Blocks are here to save the day!
Dynamic Blocks in Mage give you the power to parallelize your workflow and reduce output effortlessly. It's a pattern that has been around in data engineering, but we've made it incredibly easy and efficient.
Learn more [here](/design/blocks/dynamic-blocks).
Did you know you can trigger pipelines directly with API calls in Mage? ๐งโโ๏ธ๐ป
With API Triggers, you can kick-off your ETL runs in just about any way possibleโ a Python request, a cURL command, or a well-timed carrier pigeon (hopefully not). ๐ฆโโฌ
API triggers carry the added benefit of supporting runtime parameters, meaning you can trigger your pipelines and supply external variables to customize functionality/output.
You can read more in our docs [here](/orchestration/triggers/trigger-pipeline-api) & get started today.
Managing pipelines can be tricky, especially when it comes to backfilling missing data or re-running pipelines. But with Mage, backfills are a breeze! No need for custom scripts or complex setups. ๐
You can easily define a backfill for a specific date range. Whether you're missing data or want to add new columns, Mage has got you covered. Simply specify the start and end dates, and let Mage do the rest.
Backfills simulate multiple runs with different execution dates, fetching the data you need and transforming it as required. ๐ ๐
To learn more about backfills using Mage, check out our documentation [here](/orchestration/backfills/overview).
## Run asynchronous code in blocks
```python theme={"system"}
import asyncio
from typing import Any, Dict
import aiohttp
@data_loader
def load_data_sync(*args, **kwargs) -> Dict[str, Any]:
"""
Synchronously load data from a remote API endpoint using aiohttp and asyncio.
Uses a configurable URL with a fallback to a working test endpoint.
Includes error handling to gracefully manage connection issues.
Returns:
Dict[str, Any]: The JSON response from the API as a dictionary, or error info.
"""
url: str = kwargs.get('url', 'https://jsonplaceholder.typicode.com/todos/1')
async def fetch_data(url: str) -> Dict[str, Any]:
# Create an aiohttp session and fetch the data
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
response.raise_for_status()
data: Dict[str, Any] = await response.json()
return data
# Check if there's already a running event loop
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop and loop.is_running():
# If running in an existing event loop (e.g., Jupyter), use create_task and await
result: Dict[str, Any] = loop.run_until_complete(fetch_data(url))
else:
# If no running event loop, use asyncio.run
result: Dict[str, Any] = asyncio.run(fetch_data(url))
return result
```
# Use a completed pipeline
Source: https://docs.mage.ai/guides/train/complete-project
## 1. Clone project
In your terminal, clone this [repository](https://github.com/mage-ai/demo_project)
by running the following command:
```bash theme={"system"}
git clone https://github.com/mage-ai/demo_project.git
```
## 2. Start the tool
Docker
```bash theme={"system"}
./scripts/start.sh demo_project
```
pip
```bash theme={"system"}
mage start demo_project
```
Open [http://localhost:6789](http://localhost:6789) in your browser.
On the left side in the file browser underneath the folder `demo_project/pipelines`,
click the pipeline named `titanic_survivors`.
## 3. Follow the quick start tutorial
Go back to the [quick start tutorial](/guides/train-model),
start from step 3. Play around with scratchpad,
and use the existing code to follow along.
# File versioning and history
Source: https://docs.mage.ai/guides/version-control/file-versions
Local file edit tracking and version history for restoring changes made in the past.
Mage Pro includes built-in file versioning, eliminating the need to configure Git for rolling back changes or recovering previous file versions.
### How It Works
Every update to a file's contents automatically creates a new version that can be restored at any time. This applies to all files, including pipeline configurations.
### Restoring a Previous File Version
1. **Open the file** โ In the pipeline editor, select the file you want to recover from the left hand file browser
2. **Access version history** โ Click the "Version history" button in the top-left portion of the file browser
3. **Select a version** โ Choose the version you want to restore from the available options
4. **Restore** โ Click the "Restore this version" button to apply the changes
Once restored, the selected version becomes the current active version of your file.
Version history gives you the confidence to experiment and iterate freely without fear of losing work. If you make a mistake or need to reference previous logic, you can always roll back to any saved version.
# Airbyte in Mage
Source: https://docs.mage.ai/integrations/airbyte
Trigger a connection sync in Airbyte.
# Airbyte cloud
## Configuration
Mage Pro supports triggering **Airbyte Cloud** connection syncs directly from your data pipelines. To connect to Airbyte Cloud, you need to configure the following keyword arguments.
| Keyword argument | Description | Default value |
| ---------------- | ---------------------------------------- | ------------- |
| `use_cloud` | Whether to use Airbyte Cloud | `False` |
| `client_id` | Airbyte cloud application client\_id | `None` |
| `client_secret` | Airbyte cloud application client\_secret | `None` |
Refer to the [Airbyte documentation](https://docs.airbyte.com/using-airbyte/configuring-api-access#step-1-create-an-application) to retrieve your `client_id` and `client_secret`.
## Example data loader
You can either explicitly hard code the `connection_id` in the data loader block
or you can add the value of the `connection_id`
as a [runtime variable](/getting-started/runtime-variable#creating-runtime-variables-in-editor).
from mage\_ai.services.airbyte import Airbyte
```python theme={"system"}
from mage_ai.services.airbyte import Airbyte
@data_loader
def load_data(*args, **kwargs):
connection_id = kwargs['connection_id']
client = Airbyte(
use_cloud=True,
client_id='client_id',
client_secret='client_secret',
)
job = client.run_sync(connection_id, poll_interval=2)
return job
```
# Open source
## Configuration
Here are the following keyword arguments that can be used to configure `Airbyte`:
| Keyword argument | Description | Default value |
| ---------------- | -------------------------------------- | ---------------------- |
| `api_version` | API version | `v1` |
| `host` | Airbyte server host | `host.docker.internal` |
| `password` | Password to log into Airbyte | `password` |
| `port` | Airbyte server port | `8000` |
| `use_ssl` | If `True`, then service will use HTTPS | `False` |
| `username` | Username to log into Airbyte | `airbyte` |
## Example code
```python theme={"system"}
from mage_ai.services.airbyte import Airbyte
client = Airbyte(
api_version='v1',
host='host.docker.internal',
password='password',
port=8000,
use_ssl=False,
username='airbyte',
)
job = client.run_sync('7a749f2f-74b4-492e-9d13-30a3f390d111', poll_interval=2)
```
Sample result:
```json theme={"system"}
{
"connection_id": "7a749f2f-74b4-492e-9d13-30a3f390d111",
"connection_status": "active",
"job": {
"id": 9,
"configType": "sync",
"configId": "7a749f2f-74b4-492e-9d13-30a3f390d111",
"createdAt": 1671909838,
"updatedAt": 1671909843,
"status": "succeeded"
}
}
```
## Example data loader
You can either explicitly hard code the `connection_id` in the data loader block
or you can add the value of the `connection_id`
as a [runtime variable](/getting-started/runtime-variable#creating-runtime-variables-in-editor).
```python theme={"system"}
from mage_ai.services.airbyte import Airbyte
@data_loader
def load_data(*args, **kwargs):
connection_id = kwargs['connection_id']
client = Airbyte(
host='host.docker.internal',
password='password',
username='airbyte',
)
job = client.run_sync(connection_id, poll_interval=2)
return job
```
# Run Mage pipelines in Airflow
Source: https://docs.mage.ai/integrations/airflow/index
Need help integrating Mage into an existing Airflow project?
Check out this [tutorial](/guides/integrate-mage-airflow) or
get instant help from us in [Slack](https://www.mage.ai/chat).
***
We support running the pipeline in Airflow DAGs.
* You need to firstly install `mage_ai` library by adding `mage_ai` to your requirements.txt file.
* Then you need to download the mage pipeline code into your Airflow directory. You can achieve it by using a git submodule in your Airflow directory.
* In your Mage project's metadata.yaml file, please specify a variables\_dir that you want to store the output of each block execution (You need to have write permission to the variables\_dir). Example: `variables_dir: /tmp`.
We provide multiple ways to run mage pipelines in Airflow.
* [Create DAGs for all the pipelines in Mage project](#create-dags-for-all-the-pipelines-in-mage-project)
* [Run pipeline in a BashOperator](#run-pipeline-in-a-bashoperator)
* [Run pipeline in a PythonOperator](#run-pipeline-in-a-pythonoperator)
* [Run pipeline as an Airflow DAG](#run-pipeline-as-an-airflow-dag)
## Create DAGs for all the pipelines in Mage project
```python theme={"system"}
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from mage_ai.orchestration.airflow import create_dags
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
create_dags(
project_path,
DAG,
PythonOperator,
blacklist_pipelines=[], # Blacklisted pipeline uuids
dag_settings=dict(
start_date=datetime(2022, 8, 2),
),
globals_dict=globals(),
)
```
## Run pipeline in a BashOperator
Example code:
```python theme={"system"}
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
dag = DAG(
'test_mage_pipeline',
start_date=datetime(2022, 7, 14),
description='Test running mage pipeline.',
schedule_interval='@once',
catchup=False,
)
task = BashOperator(
dag=dag,
task_id='run_pipeline',
bash_command=f'mage run {project_path} {pipeline_name}',
)
```
## Run pipeline in a PythonOperator
Example code:
```python theme={"system"}
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
import mage_ai
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
dag = DAG(
'test_mage_pipeline',
start_date=datetime(2022, 7, 14),
description='Test running mage pipeline.',
schedule_interval='@once',
catchup=False,
)
def build_execute_pipeline(project_path, pipeline_name):
def _callable(ds, **kwargs):
mage_ai.run(pipeline_name, project_path)
return _callable
task = PythonOperator(
dag=dag,
task_id='run_pipeline',
python_callable=build_execute_pipeline(project_path, pipeline_name),
)
```
## Run pipeline as an Airflow DAG
Example code:
```python theme={"system"}
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
from mage_ai.data_preparation.models.pipeline import Pipeline
from mage_ai.shared.hash import ignore_keys
import mage_ai
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
dag = DAG(
'test_mage_pipeline',
start_date=datetime(2022, 7, 14),
description='Test running mage pipeline.',
schedule_interval='@once',
catchup=False,
)
pipeline = Pipeline(pipeline_name, repo_path=project_path)
tasks = []
def build_execute_block(block):
def _callable(ds, **kwargs):
mage_ai.run(
pipeline_name,
project_path=project_path,
block_uuid=block.uuid,
analyze_outputs=False,
update_status=False,
)
return _callable
for uuid, b in pipeline.blocks_by_uuid.items():
if b.type == 'scratchpad':
continue
tasks.append(dict(
task_id=uuid,
upstream_task_ids=b.upstream_block_uuids,
python_callable=build_execute_block(b),
))
def initialize_tasks(dag, tasks):
operators = {}
for task_dict in tasks:
task_operator = PythonOperator(dag=dag, **ignore_keys(task_dict, [
'upstream_task_ids',
]))
operators[task_operator.task_id] = task_operator
for task_dict in tasks:
for task_id in task_dict.get('upstream_task_ids', []):
dag.set_dependency(task_id, task_dict['task_id'])
return operators
initialize_tasks(dag, tasks)
```
# Integrate Mage into an existing Airflow project
Source: https://docs.mage.ai/integrations/airflow/integrate-mage-airflow
In this tutorial, weโll create a DAG in Airflow for scheduling and running a data pipeline; all from the Mage UI.
This tutorial requires that you already have Airflow setup and running
locally.
Open the `requirements.txt` file in the root directory of your Airflow project,
and add the `mage-ai` library:
```
mage-ai
```
You can install and run Mage using Docker or using pip.
### Using Docker
```bash theme={"system"}
docker pull mageai/mageai:latest
```
### Using pip
```bash theme={"system"}
pip install mage-ai
```
Change directory into your Airflowโs DAGs folder. This is typically in the
folder `dags/`.
```bash theme={"system"}
cd dags
```
Then, initialize a new Mage project in the `dags/` folder.
If youโre using Docker, run the following command in the `dags/` folder:
```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
mageai/mageai /app/run_app.sh mage init demo_project
```
If you used pip to install Mage, run the following command in the `dags/`
folder:
```bash theme={"system"}
mage init demo_project
```
Once finished, you should have a folder named `demo_project` inside your `dags/`
folder.
Your current folder structure should look like this:
```
airflow_root_directory/
| -- dags/
| -- | -- demo_project/
```
In the `dags/` folder, create a new file named `create_mage_pipelines.py`.
Then, add the following code:
```python theme={"system"}
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from mage_ai.orchestration.airflow import create_dags
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'demo_project')
create_dags(
project_path,
DAG,
PythonOperator,
dag_settings=dict(
start_date=datetime(2022, 8, 5), # Change this to any start date you want
),
globals_dict=globals(),
)
```
### Start Mage
In the `dags/` folder, start the Mage tool.
If youโre using Docker, run the following command in the `dags/` folder:
```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src \
mageai/mageai /app/run_app.sh mage start demo_project
```
If you used pip to install Mage, run the following command in the `dags/`
folder:
```bash theme={"system"}
mage start demo_project
```
Open [http://localhost:6789](http://localhost:6789) in your browser.
Follow steps 1, 2, and 4 in this [tutorial](/guides/load-api-data) to create
a new pipeline, add 1 data loader block, and add 1 transformer block.
1. Open the Airflow webserver UI at
[http://localhost:8080](http://localhost:8080) in your browser.
2. If you named your pipeline `etl demo` based on the tutorial from the previous
step, then find a DAG named `mage_pipeline_etl_demo`. If you named it
something else, find a DAG with the prefix `mage_pipeline_`.
3. Click on the DAG to view the detail page. The URL could typically be this:
[http://localhost:8080/admin/airflow/tree?dag\_id=mage\_pipeline\_etl\_demo](http://localhost:8080/admin/airflow/tree?dag_id=mage_pipeline_etl_demo).
4. Turn that DAG on if its currently off.
5. Trigger a new DAG run.
6. Watch the DAG as it runs each task according to the pipeline you created in
Mage.
# Databricks
Source: https://docs.mage.ai/integrations/compute/databricks
This is a guide for using Databricks Spark cluster with Mage.
Besides supporting running Spark pipelines in AWS EMR cluster and standalone Spark cluster, Mage also supports
running Spark pipelines in Databricks cluster.
## Set up
Here is an overview of the steps required to use Mage with Databricks Cluster:
1. [Set up Databricks cluster](#1-set-up-databricks-cluster)
2. [Use Mage databricks docker image](#2-use-mage-databricks-docker-image)
3. [Configure environment variables](#3-configure-environment-variables)
4. [Sample pipeline with PySpark code](#4-sample-pipeline-with-pyspark-code)
5. [Verify everything worked](#5-verify-everything-worked)
If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat)
### 1. Set up Databricks cluster
Set up a Databricks workspace and cluster following the docs:
* [Create a workspace using the account console](https://docs.databricks.com/administration-guide/account-settings-e2/workspaces.html)
* [Create a cluster](https://docs.databricks.com/clusters/create-cluster.html)
### 2. Use Mage databricks docker image
Contact Mage team to update your Mage Pro cluster to use Mage databricks docker image.
### 3. Configure environment variables
Set the following environment variables in your Mage Pro cluster to enable connectivity with your Databricks workspace:
* **`DATABRICKS_HOST`**\
The base URL of your Databricks workspace.\
Example: `https://.cloud.databricks.com`
* **`DATABRICKS_TOKEN`**\
A personal access token (PAT) used for authenticating with Databricks.\
You can generate this token in the Databricks UI by navigating to:\
**Settings** > **Developer** > **Access Tokens**.
* **`DATABRICKS_CLUSTER_ID`**\
The unique identifier for the Databricks cluster where queries will be executed.\
You can find this in your Databricks workspace under: **Compute** > **Clusters**.\
Refer to [Databricks documentation](https://docs.databricks.com/aws/en/workspace/workspace-details#cluster-url-and-id) for detailed steps on retrieving the cluster ID.
### 4. Sample pipeline with PySpark code
1. Create a new pipeline by clicking `New pipeline` in the `/pipelines` page.
2. Open the pipeline's metadata.yaml file and set the config.
```
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```
3. Click `+ Data loader`, then `Base template (generic)` to add a new data loader
block.
4. Paste the following sample code in the new data loader block:
5. Click "Run code" button to run the block.
```python theme={"system"}
from databricks.connect import DatabricksSession
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data(*args, **kwargs):
spark = DatabricksSession.builder.remote().getOrCreate()
data = [("John", 28), ("Anna", 23), ("Mike", 35)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
return df
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
1. Click `+ Data exporter`, then `Base template (generic)` to add a new data
exporter block.
2. Paste the following sample code in the new data exporter block:
3. Click "Run code" button to run the block.
```python theme={"system"}
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
df.write.format("delta").mode("append").saveAsTable("user_table")
return df
```
### 5. Verify everything worked
Check the table in your Unity Catalog to verify whether the data is written to it correctly.
# Compute management for Spark
Source: https://docs.mage.ai/integrations/compute/management
Manage your Spark compute resources and track Spark pipeline execution metrics.
***
A fully managed service, where we maintain the infrastructure, guarantee uptime,
automatically scale your workloads to handle any volume of pipeline runs,
automatically upgrade new versions of Mage Pro only features,
monitor your production pipelines, and provide enterprise level support.
# PySpark in Mage Pro
Source: https://docs.mage.ai/integrations/compute/spark-pro
This guide explains how to use PySpark in Mage Pro, including how to run PySpark blocks in batch pipelines, connect to Spark on Kubernetes, manage dependencies like Apache Iceberg, and access cloud storage securely. blocks in batch pipelines, manage dependencies like Apache Iceberg, and access cloud storage securely.
Mage Pro provides built-in support for running PySpark code inside your pipelines. You can run distributed Spark jobs directly within a Mage Pro batch pipeline, with automatic resource management, cloud integration, and observability.
Use PySpark in Mage to transform large datasets, connect to cloud object storage like **GCS or S3**, and integrate with modern data lake formats like **Apache Iceberg**.
## How to Use PySpark in Mage Pro
Follow these steps to run PySpark code in Mage Pro:
1. Create a **batch pipeline** in the Mage UI.
2. Add a **block** of type: `Data Loader`, `Transformer`, `Data Exporter`, or `Custom`.
3. In your block, write PySpark code using the provided `SparkSession`.
4. Install or mount any required Spark JARs, such as those for Iceberg or cloud storage access.
## Example Pipeline
Create a standard **batch pipeline** and configure the following settings in the pipeline's `metadata.yaml` file to ensure PySpark works properly:
```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```
These settings ensure data is passed in-memory between PySpark blocks and the pipeline runs in a single JVM process for Spark compatibility.
### Data Loader Block (PySpark)
```python theme={"system"}
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
from pyspark.sql import Row, SparkSession
@data_loader
def load_data(*args, **kwargs):
spark = SparkSession.builder.getOrCreate()
data = [
Row(id=1, name="Alice", age=29, salary=50000),
Row(id=2, name="Bob", age=35, salary=60000),
Row(id=3, name="Charlie", age=40, salary=70000),
Row(id=4, name="David", age=45, salary=None) # Null value example
]
df = spark.createDataFrame(data)
df.show()
return df
```
### Data Exporter Block
```python theme={"system"}
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
df.write.csv("spark_output", header=True, mode="overwrite")
```
This example pipeline loads structured data into a Spark DataFrame and writes it to disk in CSV format.
## Connecting to Spark on Kubernetes
Mage Pro can connect to an external Spark cluster running on Kubernetes. This allows you to leverage existing Spark infrastructure or run Spark jobs on a dedicated Kubernetes cluster.
### Option 1: Configure via metadata.yaml (Recommended)
Configure Spark connection settings in your project's `metadata.yaml` file:
```yaml theme={"system"}
spark_config:
# Application name
app_name: 'my spark app'
# Master URL to connect to Spark on Kubernetes
# For Kubernetes native mode:
spark_master: 'k8s://https://kubernetes.default.svc:443'
# Or for standalone Spark cluster in Kubernetes:
# spark_master: 'spark://spark-master-service:7077'
# Executor environment variables
executor_env: {}
# Jar files to be uploaded to the cluster and added to the classpath
spark_jars: []
# Path where Spark is installed on worker nodes (if applicable)
spark_home: null
# Additional Spark configuration options
others:
spark.executor.memory: '4g'
spark.executor.cores: '2'
spark.kubernetes.namespace: 'default'
spark.kubernetes.container.image: 'your-spark-image:tag'
```
### Option 2: Configure via Environment Variable
Alternatively, you can set the `SPARK_MASTER_HOST` environment variable in your Mage Pro workspace configuration:
* For Kubernetes native mode: `SPARK_MASTER_HOST=k8s://https://kubernetes.default.svc:443`
* For standalone Spark cluster: `SPARK_MASTER_HOST=spark://spark-master-service:7077`
### Setting Up Spark on Kubernetes
If you need to deploy Spark on Kubernetes first, you can use Helm:
```bash theme={"system"}
# Add the Bitnami Helm repository
helm repo add bitnami https://charts.bitnami.com/bitnami
# Install Spark cluster
helm install my-spark bitnami/spark
# Get the Spark master service URL
kubectl get svc my-spark-spark-master-svc
```
Then use the service URL in your `spark_master` configuration. For example:
* Service name: `my-spark-spark-master-svc`
* Port: `7077`
* Configuration: `spark_master: 'spark://my-spark-spark-master-svc:7077'`
### Using Spark in Your Code
Once configured, you can use Spark in your pipeline blocks:
```python theme={"system"}
from pyspark.sql import SparkSession
# SparkSession will automatically use the configuration from metadata.yaml
spark = SparkSession.builder.getOrCreate()
# Your Spark operations here
df = spark.read.csv("your-data.csv")
```
The Spark session will automatically connect to your configured Kubernetes Spark cluster.
## Benefits of Running PySpark in Mage Pro
Mage Pro handles all the infrastructure so you can focus on your PySpark code:
* โ๏ธ Distributed execution with automatic pod scheduling and resource allocation
* โ๏ธ Seamless cloud integration with GCS, S3, and service account/IAM-based authentication
* ๐งฉ Support for Spark JARs and connectors like Apache Iceberg, GCS connectors, Delta Lake, and more
* ๐ Built-in observability, with access to logs, resource usage, and block-level monitoring in the Mage UI
## Notes
* You can customize the `SparkSession` in any block using `.builder.config(...)` to tune performance or integrate external tools.
* Cloud storage credentials (e.g., a GCP service account key or AWS credentials) must be mounted and accessible inside the Mage Pro cluster.
* For advanced use cases (e.g., Apache Iceberg), see the [Iceberg + PySpark guide](/integrations/databases/Iceberg#using-iceberg-with-pyspark).
# Spark and PySpark
Source: https://docs.mage.ai/integrations/compute/spark-pyspark
This is a guide for using Spark (PySpark) with Mage in different cloud providers or Kubernetes cluster.
Want to become a
[Sparkmage](https://c1.scryfall.com/file/scryfall-cards/large/front/0/3/030f4058-54e5-4333-bd6c-2789c334bf12.jpg)?
# Mage Pro: Effortless PySpark Integration
Mage Pro makes it easy to run PySpark pipelines with a streamlined, configuration-free setup. Get started immediately without worrying about complex configurationsโjust write your PySpark code and execute it seamlessly within your Mage Pro cluster.
## Example: Simple PySpark Pipeline in Mage Pro
### Create the pipeline
Create a standard batch pipeline and configure the below settings in pipeline `metadata.yaml` file:
```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```
### Data loader block code:
```python theme={"system"}
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
from pyspark.sql import Row, SparkSession
@data_loader
def load_data(*args, **kwargs):
spark = SparkSession.builder.getOrCreate()
data = [
Row(id=1, name="Alice", age=29, salary=50000),
Row(id=2, name="Bob", age=35, salary=60000),
Row(id=3, name="Charlie", age=40, salary=70000),
Row(id=4, name="David", age=45, salary=None) # Null value example
]
df = spark.createDataFrame(data)
df.show()
return df
```
### Data exporter block code:
```python theme={"system"}
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df, *args, **kwargs):
df.write.csv("spark_output", header=True, mode="overwrite")
```
***
# Open source
## Kubernetes
When you run Mage and Spark in the same Kubernetes cluster, you can set the environment variable `SPARK_MASTER_HOST`
to the url of the master node of the Spark cluster in Mage container. Then you'll be able to connect Mage to your Spark
cluster and execute PySpark code in Mage.
Here is an overview of the steps required to use Mage with Spark in Kubernetes cluster.
### 1. Run Spark cluster in Kubernetes cluster
You can use Helm command to run Spark cluster in the Kubernetes cluster:
```bash theme={"system"}
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-release bitnami/spark
```
Then follow the instructions from the output to get the Spark master WebUI URL:
```bash theme={"system"}
kubectl port-forward --namespace default svc/my-release-spark-master-svc 80:80
echo "Visit http://127.0.0.1:80 to use your application"
```
Find the Spark Master URL at the top of the web page.
### 2. Build Mage docker image with Spark environment
* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Change the current directory to the directory that contains the `Dockerfile`. Build the docker image with the command `docker build -t mage_spark .`
### 3. Run Mage in Kubernetes cluster with the `SPARK_MASTER_HOST` environment variable
* Use the docker image built from step 2 as the container image in your Kubernetes yaml file.
* Add `SPARK_MASTER_HOST` environment variable with the Spark Master URL from step 1 to your the container spec
in your Kubernetes yaml file.
* Then deploy Mage to Kubernetes with the updated yaml file.
Here is the example Pod config:
```yaml theme={"system"}
apiVersion: v1
kind: Pod
metadata:
name: mage-server
spec:
containers:
- name: mage-server
image: mage_spark
imagePullPolicy: Never
ports:
- containerPort: 6789
volumeMounts:
- name: mage-fs
mountPath: /home/src
env:
- name: KUBE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SPARK_MASTER_HOST
value: "spark://spark_master_url:7077"
volumes:
- name: mage-fs
hostPath:
path: /path/to/mage_project
serviceAccountName: mage-user
```
Updates
* In Mage version `0.8.83` and above, you don't need to specify the environment variable anymore.
* You can specify the spark config in the project's `metadata.yaml` file as stated in [Custom Spark Session](#custom-spark-session-at-the-project-level) section.
### 4. Run PySpark code in Mage
When Mage pod is running, you can create a "Standard (batch)" pipeline (`python` kernel) and then write PySpark code in any blocks.
In `Scratchpad` block, you'll need to manually create the Spark session with the code:
```python theme={"system"}
import os
from pyspark.sql import SparkSession
spark = SparkSession.builder.master(os.getenv('SPARK_MASTER_HOST', 'local')).getOrCreate()
```
In other blocks, you can access the Spark session from the `kwargs` via `kwargs['spark']`.
***
## Standalone Spark cluster
### 1. Build Mage docker image with Spark environment
* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Change the current directory to the directory that contains the `Dockerfile`. Build the docker image with the command `docker build -t mage_spark .`
### 2. Start Mage with `SPARK_MASTER_HOST` environment variable
Type this command in your terminal to start Mage using docker
```bash theme={"system"}
docker run -it --name mage_spark -e SPARK_MASTER_HOST='local' -p 6789:6789 -v $(pwd):/home/src mage_spark \
/app/run_app.sh mage start demo_project
```
Notes
* `demo_project` is the name of your project, you can change it to anything you want
* Set the Spark cluster's master node url as the value for environment variable `SPARK_MASTER_HOST`. If you
use local Spark, you can set the value of `SPARK_MASTER_HOST` to `local` or not set the environment variable.
Updates
* In Mage version `0.8.83` and above, you don't need to specify the environment variable anymore.
* You can specify the spark config in the project's `metadata.yaml` file as stated in [Custom Spark Session](#custom-spark-session-at-the-project-level) section.
### 3. Run PySpark code in Mage
When Mage is running, you can create a "Standard (batch)" pipeline (`python` kernel) and then write PySpark code in any blocks.
In `Scratchpad` block, you'll need to manually create the Spark session with the code:
```python theme={"system"}
import os
from pyspark.sql import SparkSession
spark = SparkSession.builder.master(os.getenv('SPARK_MASTER_HOST', 'local')).getOrCreate()
```
In other blocks, you can access the Spark session from the `kwargs` via `kwargs['spark']`.
***
## Custom Spark Session at the Project Level
### 1. Build Mage docker image with Spark environment
* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Build the docker image with the command `docker build -t mage_spark .`
### 2. Start Mage using docker with the following command in your terminal
```bash theme={"system"}
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
/app/run_app.sh mage start demo_project
```
Notes
* `demo_project` is the name of your project, you can change it to anything you want
### 3. Review the `spark_config` section in `metadata.yaml` under the project folder, and make necessary adjustments
```yaml theme={"system"}
spark_config:
# Application name
app_name: 'my spark app'
# Master URL to connect to
# e.g., spark_master: 'spark://host:port', or spark_master: 'yarn'
spark_master: 'local'
# Executor environment variables
# e.g., executor_env: {'PYTHONPATH': '/home/path'}
executor_env: {}
# Jar files to be uploaded to the cluster and added to the classpath
# e.g., spark_jars: ['/home/path/example1.jar']
spark_jars: []
# Path where Spark is installed on worker nodes,
# e.g. spark_home: '/usr/lib/spark'
spark_home: null
# List of key-value pairs to be set in SparkConf
# e.g., others: {'spark.executor.memory': '4g', 'spark.executor.cores': '2'}
others: {}
```
### 4. Run PySpark code in Mage
When Mage is running, you can create a "Standard (batch)" pipeline (`python` kernel), and add a block,
then write PySpark code using the Spark session via `kwargs['spark']`, e.g.,
```python theme={"system"}
spark = kwargs.get('spark')
print(spark.sql('select 1'))
```
In `Scratchpad` block, you'll need to manually create the Spark session with the code:
```python theme={"system"}
from mage_ai.data_preparation.repo_manager import RepoConfig, get_repo_path
from mage_ai.services.spark.config import SparkConfig
from mage_ai.services.spark.spark import get_spark_session
repo_config = RepoConfig(repo_path=get_repo_path())
spark_config = SparkConfig.load(
config=repo_config.spark_config)
spark = get_spark_session(spark_config)
print(spark.sql('select 1'))
```
## Custom Spark Session at the Pipeline Level
It is also possible to use different Spark Sessions in different pipelines under the same project.
### 1. Build Mage docker image with Spark environment
* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/spark/Dockerfile) for Mage with Spark environment.
* Build the docker image with the command `docker build -t mage_spark .`
### 2. Start Mage using docker with the following command in your terminal
```bash theme={"system"}
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
/app/run_app.sh mage start demo_project
```
Notes
* `demo_project` is the name of your project, you can change it to anything you want
### 3. Create a "Standard (batch)" pipeline (`python` kernel), and update the `spark_config` section in `metadata.yaml` under the pipeline folder:
```yaml theme={"system"}
spark_config:
# Application name
app_name: 'my spark app at pipeline level'
# Master URL to connect to
# e.g., spark_master: 'spark://host:port', or spark_master: 'yarn'
spark_master: 'local'
# Executor environment variables
# e.g., executor_env: {'PYTHONPATH': '/home/path'}
executor_env: {}
# Jar files to be uploaded to the cluster and added to the classpath
# e.g., spark_jars: ['/home/path/example1.jar']
spark_jars: []
# Path where Spark is installed on worker nodes,
# e.g. spark_home: '/usr/lib/spark'
spark_home: null
# List of key-value pairs to be set in SparkConf
# e.g., others: {'spark.executor.memory': '4g', 'spark.executor.cores': '2'}
others: {}
```
### 4. Run PySpark code in Mage
Add a block to the pipeline, then write PySpark code using the Spark session via `kwargs['spark']`, e.g.,
```python theme={"system"}
spark = kwargs.get('spark')
print(spark.sql('select 1'))
```
## Custom Spark session via code
It is also possible to create custom Spark ssesion using Python code. Here are the instructions:
1. Set the `run_pipeline_in_one_process: true` in your pipeline's metadata.yaml
2. In the `spark_config` of your project's metadata.yaml or pipeline's metadata.yaml, set `use_custom_session` to `true`. Example config:
```yaml theme={"system"}
spark_config:
use_custom_session: true
custom_session_var_name: 'spark'
```
3. In the first block in your pipeline, write Python code to create the Spark session and set it in the `kwargs['context']['spark']`. Example code:
```python theme={"system"}
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data(*args, **kwargs):
spark = (
SparkSession
.builder
.appName('Test spark')
.getOrCreate()
)
kwargs['context']['spark'] = spark
...
```
4. Then, you can access the Spark session in the subsequent blocks via `kwargs['spark']`.
***
## Hadoop and Yarn cluster for Spark
### 1. Build Mage docker image with a Hadoop Yarn environment
* Download the [Dockerfile template](https://github.com/mage-ai/mage-ai/blob/master/integrations/hadoop/Dockerfile) for Mage with Hadoop environment.
* Build the docker image with the command `docker build -t mage_hadoop .`
### 2. Start Mage with the following command in your terminal using docker
```bash theme={"system"}
docker run -it --name mage_hadoop -p 6789:6789 -v $(pwd):/home/src mage_hadoop \
/app/run_app_with_hadoop.sh mage start demo_project
```
Notes
* `demo_project` is the name of your project, you can change it to anything you want
### 3. Run PySpark code in Mage using Hadoop Yarn
* Change the `metadata.yaml` file in the main pipeline folder to include the following Spark settings:
```yaml theme={"system"}
spark_config:
# Application name
app_name: 'my spark yarn app'
# Master URL to connect to
# e.g., spark_master: 'spark://host:port', or spark_master: 'yarn'
spark_master: 'yarn'
# Executor environment variables
# e.g., executor_env: {'PYTHONPATH': '/home/path'}
executor_env: {}
# Jar files to be uploaded to the cluster and added to the classpath
# e.g., spark_jars: ['/home/path/example1.jar']
spark_jars: []
# Path where Spark is installed on worker nodes,
# e.g. spark_home: '/usr/lib/spark'
spark_home: null
# List of key-value pairs to be set in SparkConf
# e.g., others: {'spark.executor.memory': '4g', 'spark.executor.cores': '2'}
others: {}
```
* Create a new `Standalone (batch)` pipeline and add a `Data loader`, then run it with the following code:
```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
"""
Template code for loading data from any source.
Returns:
Anything (e.g. data frame, dictionary, array, int, str, etc.)
"""
# Specify your data loading logic here
spark = kwargs.get('spark')
spark_conf = spark.sparkContext.getConf()
print('#' * 40)
print(spark_conf.toDebugString())
print('#' * 40)
print(spark.sql('select 1'))
return {}
```
* Verify that the added Spark code is running through a Yarn master.
***
## AWS
Here is an overview of the steps required to use Mage locally with Spark in AWS:
1. [Create an S3 bucket for Spark](#1-create-an-s3-bucket-for-spark)
2. [Start Mage](#2-start-mage)
3. [Configure projectโs metadata settings](#3-configure-projects-metadata-settings)
4. [Launch EMR cluster](#4-launch-emr-cluster)
5. [Allow EMR connection permissions](#5-allow-emr-connection-permissions)
6. [Sample pipeline with PySpark code](#6-sample-pipeline-with-pyspark-code)
7. [Debugging](#7-debugging)
8. [Clean up](#8-clean-up)
If you get stuck, run into problems, or just want someone to walk you through
these steps, please join our [Slack](https://www.mage.ai/chat)
### 1. Create an S3 bucket for Spark
Using Spark on AWS EMR requires an AWS S3 bucket to store logs, scripts, etc.
Follow
[AWSโs guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)
to create an S3 bucket. You donโt need to add any special permissions or
policies to this bucket.
Once you created an S3 bucket, note the name of the bucket (weโll need it
later).
### 2. Start Mage
Using Mage with Spark is much easier if you use Docker.
Type this command in your terminal to start Mage using docker (Note:
`demo_project` is the name of your project, you can change it to anything you
want):
```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
/app/run_app.sh mage start demo_project
```
### 3. Configure projectโs metadata settings
Open your projectโs `metadata.yaml` file located at the root of your projectโs
directory: `demo_project/metadata.yaml` (presuming your project is named
`demo_project`).
Change the values for the keys mentioned in the following steps.
* Set `remote_variables_dir`.
Change the value for key `remote_variables_dir` to equal the S3 bucket you
created in an earlier step.
For example, if your S3 bucket is named `my-awesome-bucket`, then the value for
the key `remote_variables_dir` should be `s3://my-awesome-bucket`.
* Remove optional settings
You can remove the following 2 keys:
* `master_security_group`
* `slave_security_group`
Your final `metadata.yaml` file could look like this:
```yaml theme={"system"}
variables_dir: ./
remote_variables_dir: s3://my-awesome-bucket
emr_config:
master_instance_type: "r5.4xlarge"
slave_instance_type: "r5.4xlarge"
```
Additionally, you can customize the instance count and Spark properties in your EMR cluster. Here is
an example config:
```yaml theme={"system"}
emr_config:
bootstrap_script_path: /path/to/emr_bootstrap.sh # Use a custom bootstrap script for EMR cluster
master_instance_type: 'r5.4xlarge'
master_spark_properties:
spark.driver.memory: 400M
spark.executor.cores: '8'
slave_instance_count: 2
slave_instance_type: 'r5.4xlarge'
slave_spark_properties:
spark.driver.memory: 400M
spark.executor.cores: '8'
scaling_policy: # Set up auto scaling of the EMR cluster
unit_type: 'Instances' # 'InstanceFleetUnits'|'Instances'|'VCPU'
minimum_capacity_units: 1
maximum_capacity_units: 4
maximum_on_demand_capacity_units: 4
maximum_core_capacity_units: 3
spark_jars: # External Jar libraries
- s3://bucket/mysql-connector-java-8.0.28.jar
```
You may need to request an increase in quota limits for using those instance
types.
For more information on how to view your quota limits and request an increase,
check out this
[AWS document](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html).
### 4. Launch EMR cluster
Youโll need an AWS Access Key ID and an AWS Secret Access Key. This is provided
from AWSโs IAM Management console.
Once youโve acquired those credentials, you can switch your kernel in your Mage notebook
to `pyspark` kernel. Mage will automatically creates the EMR cluster when you switch to `pyspark`
kernel. The cluster is usually created and initialized within 8\~10 minutes. Then you can select
your EMR cluster in the dropdown of your `pyspark` kernel selector.
### 5. Allow EMR connection permissions
You need to whitelist your IP in the security group of your EMR cluster to access it.
1. Open the EC2 dashboard in AWS
2. Click on "Security Groups" on the left side panel under the section "Network
& Security"
3. Find the security group named "ElasticMapReduce-master"
4. Add a new inbound rule with the following values:
| Type | Protocol | Port range | Source |
| ---------- | -------- | ---------- | ------ |
| Custom TCP | TCP | 8998 | My IP |
This will allow your locally running Mage to remotely access AWS EMR.
### 6. Sample pipeline with PySpark code
If you arenโt using Docker and you installed Mage using `pip`, you must run the
following commands in your terminal to use the `pyspark` kernel:
```bash theme={"system"}
pip install sparkmagic
mkdir ~/.sparkmagic
wget https://raw.githubusercontent.com/jupyter-incubator/sparkmagic/master/sparkmagic/example_config.json
mv example_config.json ~/.sparkmagic/config.json
sed -i 's/localhost:8998/host.docker.internal:9999/g' ~/.sparkmagic/config.json
```
***
1. Create a new pipeline by going to `File` in the top left corner of the page
and then clicking `New pipeline`.
2. Change the pipelineโs kernel from `python` to `pyspark`. Click the button
with the green dot and the word `python` next to it. This is located at the
top of the page on the right side of your header.
3. Click `+ Data loader`, then `Generic (no template)` to add a new data loader
block.
4. Paste the following sample code in the new data loader block:
```python theme={"system"}
from pandas import DataFrame
import io
import pandas as pd
import requests
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
def data_from_internet():
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
@data_loader
def load_data(**kwargs) -> DataFrame:
df_spark = kwargs['spark'].createDataFrame(data_from_internet())
return df_spark
```
1. Click `+ Data exporter`, then `Generic (no template)` to add a new data
exporter block.
2. Paste the following sample code in the new data exporter block (change the
`s3://bucket-name` to the bucket you created from a previous step):
```python theme={"system"}
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data(df: DataFrame, **kwargs) -> None:
(
df.write
.option('delimiter', ',')
.option('header', 'True')
.mode('overwrite')
.csv('s3://mage-spark-cluster/demo_project/demo_pipeline/')
)
```
#### Verify everything worked
Letโs load the data from S3 that we just created using Spark:
1. Click `+ Data loader`, then `Generic (no template)` to add a new data loader
block.
2. Paste the following sample code in the new data loader block (change the
`s3://bucket-name` to the bucket you created from a previous step):
```python theme={"system"}
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data(**kwargs) -> DataFrame:
df = (
kwargs['spark'].read
.format('csv')
.option('header', 'true')
.option('inferSchema', 'true')
.option('delimiter', ',')
.load('s3://mage-spark-cluster/demo_project/demo_pipeline/*')
)
return df
```
### 7. Debugging
If you run into any problems, 1st thing to try is restarting the kernel: `Run` >
`Restart kernel`.
If the block is hanging when running the block, it could be due to network connection issue.
Make sure the EMR cluster is accessible from the Mage server. You can verify whether Mage server's
security group or IP is whitelisted in EMR cluster's security group following section [Allow EMR connection permissions](#5-allow-emr-connection-permissions)
If that doesnโt work, restart the app by stopping the docker container and
starting it again.
### 8. Clean up
Please make sure to terminate your EMR cluster when youโre done using it so you
can save money.
***
# Algolia Integration in Mage
Source: https://docs.mage.ai/integrations/databases/Algolia
## Credentials Configuration
Open the file named `io_config.yaml` at the root of your Mage project and enter required fields:
```yaml theme={"system"}
version: 0.1.1
default:
ALGOLIA_APP_ID: app_id
ALGOLIA_API_KEY: api_key
ALGOLIA_INDEX_NAME: index_name
```
## Required Algolia Libraries
Add `algoliasearch==3.0.0` into `requirements.txt` to install the required weaviate library.
## Integration using Python Blocks
Follow these steps to integrate Algolia into your workflow using Python blocks:
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter with Template. Under "Databases" category you can
find the "Algolia" template.
* Data loader arguments:
Args:
query\_texts (str): Texts to query.
index\_name (str): Name of the index. Defaults to the name defined in `io_config.yaml`.
column\_names (List): columns to fetch.
Returns:
DataFrame: Data frame loaded.
* Data exporter arguments:
Write data into Algolia.
Args:
df (DataFrame): dataframes to write.
index\_name (str): Name of the index. Defaults to the name defined in io\_config.yaml.
3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.
# Athena
Source: https://docs.mage.ai/integrations/databases/Athena
## Prerequisites
Before using Athena in Mage, make sure you have the following:
* โ An **S3 bucket** to store Athena query results (this is required by Athena)
* โ A **DynamoDB table** if using Athenaโs table versioning (optional but recommended for Glue-backed catalogs)
* โ A **database and table** defined in Athena (via AWS Glue, or manually in the Athena console)
* โ An **IAM user or role** with the following permissions:
* `athena:StartQueryExecution`
* `athena:GetQueryExecution`
* `athena:GetQueryResults`
* `s3:PutObject`, `s3:GetObject`, `s3:ListBucket` (for the staging S3 bucket)
* โ The [`pyathena`](https://pypi.org/project/pyathena/) Python package installed
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data loader).
3. Select `Generic (no template)`.
4. Enter the following code snippet using `pyathena`:
```python theme={"system"}
from pyathena import connect
import pandas as pd
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_athena(**kwargs) -> pd.DataFrame:
conn = connect(
aws_access_key_id='your-access-key',
aws_secret_access_key='your-secret-key',
region_name='us-west-2',
s3_staging_dir='s3://your-athena-query-results/'
)
query = 'SELECT * FROM your_database.your_table LIMIT 100'
df = pd.read_sql(query, conn)
return df
```
5. Run the block.
## Install dependencies
Before running the code, install the required Python package:
```bash theme={"system"}
pip install pyathena
```
## Common errors
### โ Missing or invalid staging directory
Athena requires a staging location in S3 to store temporary query results.
Make sure `s3_staging_dir` is:
* A valid S3 URI (e.g. `s3://your-bucket/path/`)
* Located in the **same region** as your Athena workgroup
* Writable by the credentials you're using
### โ Access Denied
If you see access-related errors:
* Verify your AWS access key and secret are correct
* Confirm that your IAM user or role has permissions for both Athena and S3
* Check the S3 bucket policy to ensure it allows access from your IAM identity
### โ Region mismatch
Make sure all resources are in the same region:
* Athena workgroup
* S3 staging bucket
* AWS region specified in your connection config (e.g. `region_name='us-west-2'`)
Using mismatched regions can result in silent failures or confusing errors.
# Azure Blob Storage
Source: https://docs.mage.ai/integrations/databases/AzureBlobStorage
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
**Required:** storage account name. For authentication you can use either Azure AD (service principal) or default credentials (e.g. Azure CLI, managed identity).
```yaml theme={"system"}
version: 0.1.1
default:
AZURE_STORAGE_ACCOUNT_NAME: your_storage_account_name
# Optional: Service principal (if not using DefaultAzureCredential)
AZURE_CLIENT_ID: ...
AZURE_CLIENT_SECRET: ...
AZURE_TENANT_ID: ...
```
If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are omitted, the client uses `DefaultAzureCredential` (Azure CLI, environment variables, or managed identity when running in Azure).
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data
loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_azure_blob(**kwargs) -> DataFrame:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = '...' # Your container name
blob_path = '...' # Blob path (e.g. folder/data.parquet)
return AzureBlobStorage.with_config(
ConfigFileLoader(config_path, config_profile)
).load(container_name, blob_path)
```
5. Run the block.
### Export data to Azure Blob Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_blob_storage import AzureBlobStorage
from os import path
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_azure_blob(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = '...'
blob_path = '...'
AzureBlobStorage.with_config(
ConfigFileLoader(config_path, config_profile)
).export(df, container_name, blob_path)
```
## Supported formats
Azure Blob Storage supports loading and exporting:
* `.csv`
* `.json`
* `.parquet`
## Permissions
Ensure your Azure AD app or managed identity has the appropriate role on the storage account, for example:
* **Storage Blob Data Contributor** โ read and write blobs
* **Storage Blob Data Reader** โ read-only
# Azure Data Lake Storage Gen2
Source: https://docs.mage.ai/integrations/databases/AzureDataLakeStorage
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
**Required:** storage account name. For authentication you can use either Azure AD (service principal) or default credentials.
```yaml theme={"system"}
version: 0.1.1
default:
AZURE_STORAGE_ACCOUNT_NAME: your_storage_account_name
# Optional: Service principal (if not using DefaultAzureCredential)
AZURE_CLIENT_ID: ...
AZURE_CLIENT_SECRET: ...
AZURE_TENANT_ID: ...
```
If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are omitted, the client uses `DefaultAzureCredential` (Azure CLI, environment variables, or managed identity when running in Azure).
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data
loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_data_lake_storage import AzureDataLakeStorage
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_azure_data_lake(**kwargs) -> DataFrame:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = '...' # File system / container name
file_path = '...' # Path to file (e.g. folder/data.parquet)
return AzureDataLakeStorage.with_config(
ConfigFileLoader(config_path, config_profile)
).load(container_name, file_path)
```
5. Run the block.
### Export data to Azure Data Lake Storage Gen2
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.azure_data_lake_storage import AzureDataLakeStorage
from os import path
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_azure_data_lake(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
container_name = '...'
file_path = '...'
AzureDataLakeStorage.with_config(
ConfigFileLoader(config_path, config_profile)
).export(df, container_name, file_path)
```
## Supported formats
Azure Data Lake Storage Gen2 supports loading and exporting:
* `.csv`
* `.json`
* `.parquet`
## Permissions
Ensure your Azure AD app or managed identity has the appropriate role on the storage account with Data Lake Storage Gen2 (e.g. hierarchical namespace enabled), for example:
* **Storage Blob Data Contributor** โ read and write
* **Storage Blob Data Reader** โ read-only
# Backblaze B2
Source: https://docs.mage.ai/integrations/databases/BackblazeB2
## Prerequisites
Backblaze B2 uses Mage's S3-compatible client, which depends on `boto3`. Docker
and full dev installs already include it. If you installed Mage with a plain
`pip install mage-ai`, install the `s3` extra so the dependency is present:
```bash theme={"system"}
pip install "mage-ai[s3]"
```
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
B2_APPLICATION_KEY_ID: ...
B2_APPLICATION_KEY: ...
# Optional. Defaults to us-west-004. If your bucket is in another region,
# uncomment and set this to that region's endpoint (shown on the B2 bucket
# page), e.g. us-west-001, us-east-005, eu-central-003:
# B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com
```
For backwards compatibility, the loader also falls back to
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_ENDPOINT` (for the
endpoint URL) when the corresponding `B2_*` keys are not set.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data
loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.backblaze_b2 import BackblazeB2
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_backblaze_b2(**kwargs) -> DataFrame:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = '...' # Change to your B2 bucket name
object_key = '...' # Change to your object key
return BackblazeB2.with_config(ConfigFileLoader(config_path, config_profile)).load(
bucket_name,
object_key,
)
```
5. Run the block.
### Endpoint override
The default endpoint is `https://s3.us-west-004.backblazeb2.com`. If your
bucket lives in another B2 region, add:
```yaml theme={"system"}
default:
B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com
```
under your profile in the `io_config.yaml` file. `AWS_ENDPOINT` is honored as a fallback.
### Errors
B2 connection endpoint URL error
Open the `io_config.yaml` file at the root of your project (e.g.
`default_repo/io_config.yaml`) and confirm that `B2_ENDPOINT_URL` (if set)
matches the region of your B2 bucket.
## Permissions
Ensure the [application key](https://www.backblaze.com/docs/cloud-storage-create-and-manage-app-keys)
you create in the Backblaze B2 console grants the following capabilities on
your bucket:
* `readFiles`
* `writeFiles`
* `listFiles`
These permissions are required to:
* Read data from B2 (e.g. `.csv`, `.parquet`, `.json`)
* Write query results or transformed data to B2
* List contents of a bucket when needed
Two common Backblaze B2 gotchas:
* The **master application key** is not S3-compatible. Create a standard
(non-master) application key for use with Mage.
* App keys restricted to a single bucket may also need the
`listAllBucketNames` capability for S3 SDK/integration compatibility.
# BigQuery
Source: https://docs.mage.ai/integrations/databases/BigQuery
## Add credentials
## Required permissions
```
bigquery.datasets.create
bigquery.datasets.get
bigquery.jobs.create
bigquery.readsessions.create
bigquery.readsessions.getData
bigquery.tables.create
```
If you're running queries in the existing BigQuery dataset, make sure your account also have "BigQuery Data Editor"
role for the BigQuery dataset.
## Using SQL blocks
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `BigQuery`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Next to the `Database` label, enter the database name you want this block to
save data to.
7. Next to the `Save to schema` label, enter the schema name you want this block
to save data to.
8. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks)
for more information on write policies).
9. Enter in this test query: `SELECT 1`.
10. Run the block.
## Using Python blocks
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_big_query(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
return BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).load(query)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to BigQuery:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_big_query(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a BigQuery warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#bigquery
"""
table_id = 'your-project.your_dataset.your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
table_id,
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types=None, # Specify the column types to overwrite in a dictionary
)
```
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config.
You can find the supported types in this [doc](https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema.FIELDS.type). Here is the example code:
```python theme={"system"}
BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
table_id,
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types={'col_name': 'STRING'}, # Specify the column types to overwrite
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types. See [BigQuery data types](https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema.FIELDS.type) | `{'column1': 'STRING', 'column2': 'INTEGER'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |
**Example: Using UPSERT (UPDATE or INSERT)**
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.bigquery import BigQuery
from mage_ai.io.config import ConfigFileLoader
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_big_query(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to BigQuery with UPSERT support.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#bigquery
"""
table_id = 'your-project.your_dataset.your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
BigQuery.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
table_id,
if_exists='append', # Use 'append' with UPSERT
unique_constraints=['user_id', 'email'], # Columns that form unique constraint
unique_conflict_method='UPDATE', # Update existing rows, insert new ones
)
```
# Chroma
Source: https://docs.mage.ai/integrations/databases/Chroma
## Credentials
Open the file named `io_config.yaml` at the root of your Mage project and enter chroma required fields:
```yaml theme={"system"}
version: 0.1.1
default:
CHROMA_COLLECTION: collection_name
CHROMA_PATH: path of the chroma persisitant storage
```
## Dependencies
The dependency libraries are not installed in the docker image by default. You'll need to add the libraries to
project `requirements.txt` file manually and install them.
```
chromadb>=0.4.17
```
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter with Template. Under "Databases" category you can
find the "Chroma" template.
* Chroma data loader arguments:
* n\_results: Number of results to match.
* collection: collection to use. Otherwise the collection defined in io\_config.yml will be used.
* query\_texts: the texts or documents used to query.
* query\_embeddings: the embeddings used to query.
Either `query_texts` or `query_embeddings` should be available.
* Chroma data exporter arguments:
* df: Dataframe contains the actual data.
* collection: collection to use. Otherwise the collection defined in io\_config.yml will be used.
* document\_column: specify the column in df dataframe that contains documents to be stored in Chroma.
* id\_column: specify the column in df dataframe that contains the id of the document
* metadata\_column: specify the column in df dataframe that contains the metadata of the document (metadata should be dictionary format)
3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.
# ClickHouse
Source: https://docs.mage.ai/integrations/databases/ClickHouse
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
# ClickHouse
CLICKHOUSE_DATABASE: ...
CLICKHOUSE_HOST: ...
CLICKHOUSE_INTERFACE: http
CLICKHOUSE_PASSWORD: ...
CLICKHOUSE_PORT: 8123
CLICKHOUSE_USERNAME: ...
```
***
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `ClickHouse`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Enter the optional table name of the table to write to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks) for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.
***
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.clickhouse import ClickHouse
from mage_ai.io.config import ConfigFileLoader
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_clickhouse(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
return ClickHouse.with_config(ConfigFileLoader(config_path, config_profile)).load(query)
```
5. Run the block.
***
## Destination table in Data Exporter
If the destination table does not exist and the `Write policy` is set
to `Replace`, `data exporter` will automatically create a table in `ClickHouse`
with `Engine = Memory` and a default schema inferred from the data.
However, this may not be optimal for various use cases. Since [table creation](https://clickhouse.com/docs/en/sql-reference/statements/create/table)
in `ClickHouse` can involve numerous details, it is strongly advised to
create the destination table manually before loading data to ensure it
meets specific requirements.
If generating the destination table via Mage export method, one can overwrite specific column types
by using the `overwrite_types` dict setting.
Example:
```python theme={"system"}
overwrite_types = {'column_name': 'VARCHAR(255)'}
```
Here is the ClickHouse data exporter code snippet that you can use:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.clickhouse import ClickHouse
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_clickhouse(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Clickhouse database.
Specify your configuration settings in 'io_config.yaml'.
"""
database = 'database_name' # Specify the name of the schema to export data to
table_name = 'table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
ClickHouse.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
table_name=table_name,
database=database,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types={'column_name': 'VARCHAR(255)'}, # Overwrite column types
)
```
### Method arguments
| Field name | Description | Example values |
| ---------------- | ------------------------------------------------------ | --------------------------------------------------- |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'VARCHAR(255)', 'column2': 'INTEGER'}` |
# Databricks
Source: https://docs.mage.ai/integrations/databases/Databricks
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
DATABRICKS_ACCESS_TOKEN: access_token
DATABRICKS_DATABASE: database # Databricks Catalog
DATABRICKS_HOST: dbc-123456-abcd.cloud.databricks.com
DATABRICKS_HTTP_PATH: "/sql/1.0/warehouses/abcd1234"
DATABRICKS_SCHEMA: schema
```
## Using SQL Block with Databricks
Follow these steps to use a **SQL block** connected to **Databricks** in your pipeline.
1. **Create or open a pipeline**\
Create a new pipeline or open an existing one.
2. **Add a block**\
Add a **Data loader**, **Transformer**, **Data exporter**, or **Custom** block.
3. **Select block type**\
Set the block type to `SQL`.
4. **Configure the data provider**\
In the `Data provider` dropdown, select `Databricks`.
5. **Choose a profile**\
In the `Profile` dropdown, select `default` (or the profile you configured your Databricks credentials under).
6. **Configure output saving (optional)**\
By default, the SQL block saves query results to a table in your Databricks database.
* **Schema** (optional): Enter the schema where the blockโs output should be saved, default to the schema in `io_config.yaml`.
* **Table** (optional): Enter the table where the blockโs output should be saved, default table is generated based on block uuid.
* **Write policy**: Choose either `Replace` or `Append`. See the [SQL blocks guide](/guides/blocks/sql-blocks) for more details on write policies.
* If you **donโt want to save results** to an intermediate table, enable the **Use raw SQL** option. See the [Raw SQL documentation](/guides/blocks/sql-blocks#using-raw-sql) for more information.
7. **Enter a test query**\
Add a simple test query to confirm the connection works:
```
SELECT 1
```
8. **Run the block**\
Click **Run** to execute the query and verify the block works as expected.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select the `Data lakes -> Databricks SQL` template or `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.databricks_sql import DatabricksSQL
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_databricks(*args, **kwargs):
"""
Template for loading data from Databricks with SQL.
Specify your configuration settings in 'io_config.yaml'.
"""
query = 'SELECT 1' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with DatabricksSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to Databricks:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.databricks_sql import DatabricksSQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_databricks(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to Databricks.
Specify your configuration settings in 'io_config.yaml'.
"""
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with DatabricksSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name=schema_name,
table_name=table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
# Druid
Source: https://docs.mage.ai/integrations/databases/Druid
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
DRUID_HOST: ...
DRUID_PASSWORD: ...
DRUID_PATH: /druid/v2/sql/
DRUID_PORT: 8082
DRUID_SCHEME: http
DRUID_USER: ...
```
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Druid`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Select `Use raw SQL`.
7. Enter in this test query: `SELECT 1`.
8. Run the block.
# DuckDB
Source: https://docs.mage.ai/integrations/databases/DuckDB
Execute SQL commands in DuckDB.
## Credentials
Open the file named `io_config.yaml` at the root of your Mage project and enter DuckDB database name:
```yaml theme={"system"}
version: 0.1.1
default:
DUCKDB_DATABASE: database
DUCKDB_SCHEMA: main
```
## How to use
1. Create a new pipeline or edit an existing pipeline.
2. Add a data loader or transformer SQL block.
3. In the blockโs header, click the dropdown menu labeled Connection and select the option
labeled DuckDB.
4. In the blockโs header, click the dropdown menu labeled Profile and select the option
labeled default.
If the name of your profile in the `io_config.yaml` file is different,
then select the profile name that matches your settings.
5. You can optionally check the box labeled Use raw SQL.
If you do, read more about [how to use SQL blocks with raw SQL](/guides/blocks/sql-blocks#using-raw-sql).
6. Write your SQL statements in the SQL block.
7. Once youโre finished writing your SQL statements,
click the play button in the top right corner of the block to execute the SQL block and preview
the results.
# Google Cloud Storage
Source: https://docs.mage.ai/integrations/databases/GoogleCloudStorage
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
Use **either** a path to your service account key file **or** the key contents as a mapping (e.g. from a secret):
```yaml theme={"system"}
version: 0.1.1
default:
# Option 1: Path to service account JSON file
GOOGLE_SERVICE_ACC_KEY_FILEPATH: /path/to/service_account_credentials.json
# Option 2: Inline credentials (e.g. from env or secrets)
# GOOGLE_SERVICE_ACC_KEY: "{{ json_value(env_var('GCP_CREDENTIALS_JSON'), 'private_key') }}"
# Or use mage_secret_var / aws_secret_var etc. for the full credentials object
```
If the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is set (e.g. to the path of your service account JSON), you can also use the IO class without adding keys to `io_config.yaml`; otherwise use `with_config` as shown below.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data
loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_gcs(**kwargs) -> DataFrame:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = '...' # Change to your bucket name
object_key = '...' # Change to your object key (e.g. path/to/file.parquet)
return GoogleCloudStorage.with_config(
ConfigFileLoader(config_path, config_profile)
).load(bucket_name, object_key)
```
5. Run the block.
### Export data to Google Cloud Storage
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.google_cloud_storage import GoogleCloudStorage
from os import path
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_gcs(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = '...'
object_key = '...'
GoogleCloudStorage.with_config(
ConfigFileLoader(config_path, config_profile)
).export(df, bucket_name, object_key)
```
## Supported formats
Google Cloud Storage supports loading and exporting:
* `.csv`
* `.json`
* `.parquet`
## Permissions
Ensure your Google Cloud service account has the appropriate roles, for example:
* **Storage Object Viewer** โ to read objects
* **Storage Object Creator** โ to write objects
* **Storage Object Admin** โ for full read/write/delete
# Google Sheets
Source: https://docs.mage.ai/integrations/databases/GoogleSheets
Mage supports writing/reading from Google Sheets using the Google Sheets API via the gspread library. Read on to learn about reading/writing data to Sheets.
## Configuration
To configure Google Sheets with Mage, you'll need to enable several APIs in your GCP project and add credentials to Mage.
### Enable relevant APIs
Using Google Sheets in Mage requires the following to be enabled via GCP:
* Google Drive API
* Google Sheets API
Additionally, the Google Sheet must be shared with the service account used in Mage.
You can follow [this guide](https://docs.gspread.org/en/latest/oauth2.html) to enable relevant services.
### Add credentials
Adding credentials for Sheets is the same as adding credentials for other Google services, i.e. BigQuery, GCP.
## Usage
We have two templates for using Google Sheets with Mage. They can be found in Python "Data Loader" and "Data Exporter" templates.
You can see the source code for the templates [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/templates/data_exporters/google_sheets.py) and [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/data_preparation/templates/data_loaders/google_sheets.py).
# Iceberg
Source: https://docs.mage.ai/integrations/databases/Iceberg
Connect to Apache Iceberg tables stored in Amazon S3 using various catalog types
## Add credentials
To access **Apache Iceberg tables stored in Amazon S3**, you'll need to configure your AWS credentials.
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (or the profile you are using):
```yaml theme={"system"}
version: 0.1.1
default:
AWS_ACCESS_KEY_ID: ...
AWS_SECRET_ACCESS_KEY: ...
AWS_REGION: us-west-2 # or the region where your S3 bucket is located
```
> These credentials must have read/write access to the S3 bucket that contains your Iceberg tables.
## Using Python block
You can use Mage to load data from Iceberg tables stored in S3 or export data to Iceberg tables using a configurable Python block.
### Steps
1. **Create or open a pipeline** in your Mage Pro cluster.
2. **Add a block** of type **Data Loader** or **Data Exporter**.
3. From the block template list, choose:\
**Data lakes โ Apache Iceberg**
4. In the generated code block, update the following configuration parameters:
* `base_uri`: Base S3 URI for the Iceberg warehouse (e.g., `s3://your-bucket-name/warehouse/`) - required for SQL catalog
* `namespace`: Namespace for the Iceberg catalog (default: `'default'`)
* `catalog_type`: Type of catalog to use - `'sql'` (default) or `'glue'`
* `table_name`: Name of the Iceberg table
* `bucket_name`: Name of your S3 bucket (for exports)
* `mode`: Write mode for exports - `'append'` (default) or `'overwrite'`
* `metadata_file`: Optional - used to directly access S3 metadata file when table is not in catalog
5. If you're using a non-default profile, update the `config_profile` field accordingly.
6. **Run the block** to load or export data from your Iceberg table stored on S3.
### Configuration Options
#### Catalog Types
Mage supports multiple catalog types for Iceberg, including:
* **SQL Catalog (default)**: Uses a Postgres-backed catalog to store table metadata
* Requires `base_uri` to specify the warehouse location
* Tables can be registered in the catalog or accessed directly via metadata files
* **AWS Glue Catalog**: Uses AWS Glue as the catalog
* No `base_uri` required
* Tables must be registered in AWS Glue
Additional catalog types are also supported. Set `catalog_type` to the appropriate value for your catalog implementation.
#### Loading Data
When loading data, you can:
* Load from catalog: If the table is registered in the catalog, just provide `table_name`
* Load from metadata file: If the table is not in the catalog, provide `metadata_file` to access the S3 metadata file directly
* The metadata file path will be: `{base_uri}{table_name}/metadata/{metadata_file}`
Additional kwargs (e.g., `row_filter`, `selected_fields`, `case_sensitive`, `snapshot_id`) can be passed into the `load()` method to customize the data retrieval. These parameters are passed through to pyiceberg's scan method.
#### Scan Parameters
The following parameters can be passed to the `load()` method to customize data retrieval. These parameters are passed through to pyiceberg's `scan()` method. For the complete method signature, see the [pyiceberg source code](https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L1133-L1135).
| Parameter | Type | Default | Description | Example |
| ----------------- | ---------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| `row_filter` | `str` or `BooleanExpression` | `AlwaysTrue()` | A string or BooleanExpression that describes the desired rows | `'id > 100'` or `'status == "active"'` |
| `selected_fields` | `tuple[str]` | `("*",)` | A tuple of strings representing the column names to return in the output dataframe | `('id', 'name', 'created_at')` |
| `case_sensitive` | `bool` | `True` | If `True`, column matching is case sensitive | `True` |
| `snapshot_id` | `int` or `None` | `None` | Optional Snapshot ID to time travel to. If `None`, scans the table as of the current snapshot ID | `12345` |
| `limit` | `int` or `None` | `None` | An integer representing the number of rows to return in the scan result. If `None`, fetches all matching rows | `1000` |
#### Exporting Data
When exporting data, you can:
* **Append mode** (default): Adds new data to the existing table
* **Overwrite mode**: Replaces all data in the table with the new data
### Example: Loading Data
```python SQL Catalog theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader
# Using SQL catalog
iceberg = Iceberg.with_config(
ConfigFileLoader(config_path, config_profile),
base_uri='s3://my-bucket/warehouse/',
namespace='analytics',
catalog_type='sql',
)
# Load from catalog
df = iceberg.load('my_table')
# Or load from metadata file (if not in catalog)
df = iceberg.load('my_table', metadata_file='00000-abc123.metadata.json')
```
```python AWS Glue Catalog theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader
# Using AWS Glue catalog
iceberg = Iceberg.with_config(
ConfigFileLoader(config_path, config_profile),
namespace='analytics',
catalog_type='glue',
)
df = iceberg.load('my_table')
```
```python With Scan Parameters theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader
iceberg = Iceberg.with_config(
ConfigFileLoader(config_path, config_profile),
base_uri='s3://my-bucket/warehouse/',
namespace='analytics',
catalog_type='sql',
)
# Loading with additional scan parameters
df = iceberg.load(
'my_table',
row_filter='id > 100', # Filter rows
selected_fields=['id', 'name', 'created_at'], # Select specific fields
case_sensitive=True, # Case-sensitive field matching
snapshot_id=12345, # Read from specific snapshot
limit=1000, # Limit number of rows
)
```
### Example: Exporting Data
```python Append Mode theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader
iceberg = Iceberg.with_config(
ConfigFileLoader(config_path, config_profile),
namespace='analytics',
catalog_type='sql',
)
# Append data to table
iceberg.export(
df,
bucket_name='my-bucket',
table_name='my_table',
mode='append',
)
```
```python Overwrite Mode theme={"system"}
from mage_ai.io.iceberg_s3 import Iceberg
from mage_ai.io.config import ConfigFileLoader
iceberg = Iceberg.with_config(
ConfigFileLoader(config_path, config_profile),
namespace='analytics',
catalog_type='sql',
)
# Overwrite table with new data
iceberg.export(
df,
bucket_name='my-bucket',
table_name='my_table',
mode='overwrite',
)
```
### Additional Methods
The Iceberg integration also provides methods for managing tables and namespaces:
* `list_namespaces()`: List all namespaces in the catalog
* `list_tables(namespace)`: List all tables in a namespace
* `get_table_schema(table_name, namespace)`: Get the schema of a table
* `drop_table(table_name, namespace)`: Drop (delete) a table
* `drop_namespace(namespace)`: Drop (delete) a namespace
### Notes
* The Iceberg integration supports direct access to tables through metadata stored in S3 when tables are not registered in the catalog.
* For SQL catalog, `base_uri` should point to your warehouse location and include a trailing slash (e.g., `s3://bucket/warehouse/`).
* For AWS Glue catalog, tables must be registered in AWS Glue.
* When using `metadata_file`, the path is constructed as: `{base_uri}{table_name}/metadata/{metadata_file}`.
* Additional scan parameters (e.g., `row_filter`, `selected_fields`, `case_sensitive`, `snapshot_id`) can be passed to the `load()` method.
## Using Iceberg with PySpark
Mage Pro supports reading from and writing to [Apache Iceberg](https://iceberg.apache.org/) tables using PySpark, enabling scalable and efficient data lake operations.
### Example code
Using Iceberg with Google Cloud Storage (GCS)
```python theme={"system"}
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("IcebergWithGCS") \
.config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.1") \
.config("spark.jars", "/opt/spark/jars/gcs-connector-hadoop3-2.2.5-shaded.jar") \
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.iceberg.type", "hadoop") \
.config("spark.sql.catalog.iceberg.warehouse", "gs://mage_icerberg_test/test") \
.config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") \
.config("spark.hadoop.google.cloud.auth.service.account.enable", "true") \
.config("spark.hadoop.google.cloud.auth.service.account.json.keyfile", "/home/src/default_repo/your_google_service_account_key.json") \
.getOrCreate()
```
Create and Write an Iceberg Table
```python theme={"system"}
# Sample DataFrame
df = spark.createDataFrame([
(1, "apple"),
(2, "banana"),
(3, "banana"),
], ["id", "fruit"])
# Create the database if it doesn't exist
spark.sql("CREATE DATABASE IF NOT EXISTS iceberg.db_name")
# Write data to an Iceberg table
df.writeTo("iceberg.db_name.iceberg_table") \
.using("iceberg") \
.createOrReplace()
```
### Notes
* `iceberg.db_name.iceberg_table` uses the Hadoop catalog type and stores metadata in the specified GCS path.
* You can modify the catalog configs to use **Hive**, **Glue**, or **Nessie** depending on your architecture.
* For AWS S3, update the warehouse path and authentication configurations accordingly.
* The Google service account key file path must be accessible inside the Mage Pro cluster.
* You can run this code inside a block in Mage Pro batch pipeline.
# Microsoft Fabric Warehouse
Source: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
## Overview
Microsoft Fabric Warehouse is a fully managed, cloud-native data warehouse built on Azure Synapse SQL. It provides enterprise-grade performance, scalability, and security for your data analytics workloads.
This integration allows you to:
* Connect to Microsoft Fabric Data Warehouse using Azure AD Service Principal authentication (with optional token-based auth via `AZURE_TENANT_ID`)
* Run SQL in pipelines via **SQL blocks** (transformer) or Python blocks
* Export data using batched row-based inserts (standard export)
This integration allows you to connect to Microsoft Fabric Data Warehouse using Azure AD Service Principal authentication and perform SQL operations directly from Mage pipelines.
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
AZURE_CLIENT_ID: '12345678-1234-1234-1234-123456789012'
AZURE_CLIENT_SECRET: 'your-azure-ad-service-principal-secret'
MICROSOFT_FABRIC_WAREHOUSE_NAME: 'your_database_name'
MICROSOFT_FABRIC_WAREHOUSE_ENDPOINT: 'your-endpoint-here'
MICROSOFT_FABRIC_WAREHOUSE_SCHEMA: 'dbo'
# Optional: use token-based auth (recommended for Fabric). Must be a valid GUID.
# AZURE_TENANT_ID: '87654321-4321-4321-4321-210987654321'
```
### Configuration Parameters
| Parameter | Description | Required | Example |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------- |
| `AZURE_CLIENT_ID` | Azure AD Service Principal Application ID (GUID format) | โ | `12345678-1234-1234-1234-123456789012` |
| `AZURE_CLIENT_SECRET` | Azure AD Service Principal Secret | โ | `your-secret-here` |
| `MICROSOFT_FABRIC_WAREHOUSE_NAME` | Name of your Fabric Data Warehouse database | โ | `my_database` |
| `MICROSOFT_FABRIC_WAREHOUSE_ENDPOINT` | Fabric workspace endpoint URL | โ | `abcde12345.datawarehouse.fabric.microsoft.com` |
| `MICROSOFT_FABRIC_WAREHOUSE_SCHEMA` | Default schema for operations | โ | `dbo` (default) |
| `AZURE_TENANT_ID` | Azure AD tenant ID (GUID format). When set, connection uses **token-based auth** (OAuth2 token injected via ODBC); recommended for Fabric. When omitted, connection uses UID/PWD with ActiveDirectoryServicePrincipal. | โ | `87654321-4321-4321-4321-210987654321` |
## Service Principal Setup
To use service principals with Microsoft Fabric Data Warehouse, you need to:
1. **Create a service principal** in Microsoft Entra ID
2. **Enable service principal access** in Microsoft Fabric
3. **Assign appropriate permissions** to the service principal
For detailed setup instructions, see:
* [Register a Microsoft Entra app and create a service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal)
* [Service principal in Fabric Data Warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/service-principals#prerequisites)
***
## Using SQL blocks
Run SQL directly against Microsoft Fabric Warehouse from a **SQL block** (transformer):
1. Add a **Transformer** block and set its language to **SQL**.
2. In the blockโs connection settings, set **Connection** to **Microsoft Fabric** and **Profile** to your `io_config.yaml` profile (e.g. `default`).
3. Optionally set **Schema** (default `dbo`) and **Use raw SQL** to run statements as-is.
4. Write your SQL. Reference upstream block output with **`{{ df_1 }}`**, **`{{ df_2 }}`**, etc. The block uses the profileโs credentials and schema.
A SQL block writes data to the warehouse in two ways:
1. **Materialized output (no โUse raw SQLโ)**
When โUse raw SQLโ is **off**, the block runs your query and **exports the result** to a table (creates or appends) by executing the query string in the warehouse. The warehouse runs the SQL directly.
2. **Upstream data referenced in SQL (`{{ df_1 }}`, `{{ df_2 }}`, โฆ)**
When your SQL references **`{{ df_1 }}`** (or `{{ df_2 }}`, etc.), the upstream blockโs output must be available in the warehouse. If not already there, Mage **uploads** that DataFrame using batched inserts (see [Export limits](#export-limits)).
***
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.microsoft_fabric_warehouse import MicrosoftFabricWarehouse
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_fabric_warehouse(*args, **kwargs):
"""
Template for loading data from Microsoft Fabric Warehouse.
Specify your configuration settings in 'io_config.yaml'.
Set the following in your io_config:
Docs: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
"""
query = 'SELECT * FROM your_table LIMIT 1000' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MicrosoftFabricWarehouse.with_config(ConfigFileLoader(config_path, config_profile)) as warehouse:
return warehouse.load(query)
```
5. Run the block.
### Export a dataframe
Use a data exporter block and call `warehouse.export()`. Data is inserted in batches (see [Export limits](#export-limits)).
Here is an example code snippet to export a dataframe to Microsoft Fabric Warehouse:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.microsoft_fabric_warehouse import MicrosoftFabricWarehouse
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_fabric_warehouse(df: DataFrame, **kwargs) -> None:
"""
Export to Microsoft Fabric Warehouse.
Set credentials (and optional AZURE_TENANT_ID for token auth) in io_config.yaml.
"""
schema_name = 'dbo'
table_name = 'your_table_name'
Template for exporting data to Microsoft Fabric Warehouse.
Specify your configuration settings in 'io_config.yaml'.
Set the following in your io_config:
Docs: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
"""
schema_name = 'dbo' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MicrosoftFabricWarehouse.with_config(ConfigFileLoader(config_path, config_profile)) as warehouse:
warehouse.export(
df,
schema_name,
table_name,
if_exists='replace',
index=False,
if_exists='replace', # Specify resolution policy if table name already exists
index=False, # Specifies whether to include index in exported table
)
```
### Standard export
Export uses batched `INSERT` statements. Use the same `io_config.yaml` profile as for loading. Optionally set `AZURE_TENANT_ID` (GUID) for token-based auth (recommended for Fabric).
**Configuration:** See [Add credentials](#add-credentials). Optionally add `AZURE_TENANT_ID` for token auth.
**Code:** Call `warehouse.export(df, schema_name, table_name, ...)` as in the example above. See [Export limits](#export-limits) for the column limit.
***
### Export limits
* **Column count:** SQL Server limits the number of parameters per statement (2,000). If your DataFrame has **more than 2,000 columns**, export raises a clear error. Reduce the number of columns or use a different upload method (e.g. export a subset of columns, or stage data elsewhere and load via another path).
***
### Custom types
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.microsoft_fabric_warehouse import MicrosoftFabricWarehouse
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_fabric_warehouse(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to Microsoft Fabric Warehouse.
Specify your configuration settings in 'io_config.yaml'.
Set the following in your io_config:
Docs: https://docs.mage.ai/integrations/databases/MicrosoftFabricWarehouse
Docs: /integrations/databases/MicrosoftFabricWarehouse
"""
schema_name = 'dbo' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with MicrosoftFabricWarehouse.with_config(ConfigFileLoader(config_path, config_profile)) as warehouse:
warehouse.export(
df,
schema_name,
table_name,
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types=overwrite_types,
index=False, # Specifies whether to include index in exported table
)
```
## Troubleshooting errors
### Connection Timeout Errors
**Error**: `Login timeout expired` or `Connection timeout`
*"Verify your Service Principal credentials and network connectivity"*
**Solutions**:
* Ensure `AZURE_CLIENT_ID` is a valid GUID format
* Verify `AZURE_CLIENT_SECRET` is correct and not expired
* Check that the Service Principal has appropriate permissions on your Fabric workspace
* Ensure the Service Principal is not disabled in Azure AD
* Verify network connectivity to the endpoint
* Check if corporate firewall or proxy is blocking the connection
* Try increasing connection timeout settings
### TCP Provider Errors
**Error**: `TCP Provider: Error code 0x2746 (10054)` or `Connection reset by peer`
*"Verify your endpoint URL and network connectivity"*
**Solutions**:
* Ensure `MICROSOFT_FABRIC_WAREHOUSE_ENDPOINT` is correct and accessible
* Check network connectivity to the endpoint (ping, telnet, etc.)
* Verify the endpoint is accessible from your current location
* Ensure the endpoint is not blocked by firewall rules
* Check if Azure region is experiencing connectivity issues
* Verify the Microsoft Fabric service is operational
* Try connecting from a different network to isolate the issue
### Database Object Errors
**Error**: `Invalid object name 'table_name'` or `Database 'database_name' does not exist`
*"Verify your database name, schema, and table existence"*
**Solutions**:
* Ensure `MICROSOFT_FABRIC_WAREHOUSE_NAME` matches exactly (case-sensitive)
* Check that the Service Principal has access to the database
* Verify the database exists in your Fabric workspace
* Ensure the database is not paused or offline
* Verify the table exists in the specified schema
* Check if you're using the correct schema name (default is 'dbo')
* Ensure table names are properly quoted if they contain special characters
### Authentication Errors
**Error**: `Failed to authenticate with Azure AD` or `Login failed for user` or `Could not login because the authentication failed.`
**Error**: `Failed to authenticate with Azure AD` or `Login failed for user`
*"Verify your Service Principal credentials and permissions"*
**Solutions**:
* Ensure `AZURE_CLIENT_ID` is a valid GUID format
* Verify `AZURE_CLIENT_SECRET` is correct and not expired
* Check that the Service Principal has appropriate permissions on your Fabric workspace
* Ensure the Service Principal is not disabled in Azure AD
* Verify the Service Principal has the correct role assignments
* Check if the Service Principal's permissions have been revoked
### Export column limit
**Error**: `DataFrame has X columns, which exceeds SQL Server parameter limit (2000)`
*"Reduce the number of columns or use a different upload method"*
**Solutions**:
* Export a subset of columns (e.g. drop or select only the columns you need)
* Split the DataFrame into multiple tables
* Use a different load path if you have very wide tables (e.g. stage to storage and load via another tool)
### Schema Errors
**Error**: `Schema 'schema_name' does not exist`
*"Verify your schema name and permissions"*
**Solutions**:
* Ensure `MICROSOFT_FABRIC_WAREHOUSE_SCHEMA` is correct
* Check that the Service Principal has access to the schema
* Verify the schema exists in your database
* Use 'dbo' as the default schema if unsure
* Check if the schema name is case-sensitive
* Verify the Service Principal has CREATE/ALTER permissions on the schema
## Best Practices
1. **Service Principal Management**:
* Use dedicated Service Principals for different environments
* Rotate secrets regularly
* Grant minimum required permissions
2. **Connection Management**:
* Always use context managers (`with` statements) when possible
* Close connections explicitly when not using context managers
* Test connections before running large operations
3. **Performance Optimization**:
* Export uses batched inserts; keep tables under the [export column limit](#export-limits) (2,000 columns). For very wide DataFrames, reduce columns or use another load path.
* Use appropriate `if_exists` policies ('replace', 'append', 'fail')
* Consider using `overwrite_types` for optimal column types
* Use LIMIT clauses in queries when testing
4. **Security**:
* Never hardcode credentials in your code
* Use `io_config.yaml` for all configuration
* Ensure `io_config.yaml` is not committed to version control
* Use environment variables for sensitive values in production
## Related Documentation
* [Microsoft Fabric Documentation](https://learn.microsoft.com/en-us/fabric/)
* [Microsoft Fabric Warehouse Documentation](https://learn.microsoft.com/en-us/fabric/data-warehouse/)
* [Azure AD Service Principal Setup](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
* [ODBC Driver Installation](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server)
# Microsoft SQL Server
Source: https://docs.mage.ai/integrations/databases/MicrosoftSQLServer
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
MSSQL_DATABASE: database
MSSQL_SCHEMA: schema
MSSQL_DRIVER: "ODBC Driver 18 for SQL Server"
MSSQL_HOST: host
MSSQL_PASSWORD: password
MSSQL_PORT: 1433
MSSQL_USER: SA
# (Optional) SSH tunnel config
MSSQL_CONNECTION_METHOD: direct
MSSQL_SSH_HOST:
MSSQL_SSH_PORT: 22
MSSQL_SSH_USERNAME:
MSSQL_SSH_PASSWORD:
MSSQL_SSH_PKEY:
```
### SSH tunneling
SSH tunneling is a Mage Pro only feature.
If you want to connect to SQL Server with SSH tunnel, update the value of `MSSQL_CONNECTION_METHOD` to `ssh_tunnel`, and enter the values for keys with prefix `MSSQL_SSH`.
| Key | Description | Sample value |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
| `MSSQL_SSH_HOST` | The host of the intermediate bastion server. | `123.45.67.89` |
| `MSSQL_SSH_PORT` | The port of the intermediate bastion server. Default value: 22 | `22` |
| `MSSQL_SSH_USERNAME` | The username used to connect to the bastion server. | `username` |
| `MSSQL_SSH_PASSWORD` | (Optional) The password used to connect to the bastion server. It should be set if you authenticate with the bastion server with password. | `password` |
| `MSSQL_SSH_PKEY` | (Optional) The path to the private key used to connect to the bastion server. It should be set if you authenticate with the bastion server with private key. | `/path/to/private/key` |
When using SSH tunnel, the `fast_execute` option will automatically be disabled to ensure reliable connections through the tunnel.
## Dependencies
To connect to the Microsoft SQL Server, you'll need to make sure the driver is installed.
By default, ODBC Driver 18 is installed in the docker image.
If you want to use other ODBC Driver versions, you'll need to build a custom docker image (use Mage image as the base image) and install the drivers.
Here is the doc for installing ODBC drivers for SQL Server: [https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server)
***
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider`/`Connection` dropdown, select `Microsoft SQL Server`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Enter the schema and optional table name of the table to write to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks) for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mssql import MSSQL
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_mssql(*args, **kwargs):
"""
Template for loading data from a MSSQL database.
Specify your configuration settings in 'io_config.yaml'.
Set the following in your io_config:
Docs: /integrations/databases/MicrosoftSQLServer
"""
query = 'Your MSSQL query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MSSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to MSSQL:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mssql import MSSQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mssql(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a MSSQL database.
Specify your configuration settings in 'io_config.yaml'.
Set the following in your io_config:
Docs: /integrations/databases/MicrosoftSQLServer
"""
schema_name = 'dbo' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MSSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
fast_execute=True, # Use fast_executemany option to speed up bulk inserting rows
unique_conflict_method='UPDATE', # Specify method to resolve row conflicts
unique_constraints=['col'],
)
```
6. Custom types
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config
Here is an example code snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mssql import MSSQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mssql(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a MSSQL database.
Specify your configuration settings in 'io_config.yaml'.
Set the following in your io_config:
Docs: /integrations/databases/MicrosoftSQLServer
"""
schema_name = 'dbo' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with MSSQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types=overwrite_types,
)
```
## Troubleshooting errors
### `error: ODBC SQL type -155 is not yet supported.`
*โI changed the datetime with timezone data type to a datetime and it starting workingโ*
* [Windy Martin](https://mageai.slack.com/archives/C068SHY6YG6/p1705874712639659?thread_ts=1705668895.141169\&cid=C068SHY6YG6)
# MongoDB
Source: https://docs.mage.ai/integrations/databases/MongoDB
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
# Specify either the connection string or the (host, password, user, port) to connect to MongoDB.
MONGODB_CONNECTION_STRING: "mongodb://{username}:{password}@{host}:{port}/"
MONGODB_HOST: host
MONGODB_PORT: 27017
MONGODB_USER: user
MONGODB_PASSWORD: password
MONGODB_DATABASE: database
MONGODB_COLLECTION: collection
```
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mongodb import MongoDB
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_from_mongodb(*args, **kwargs):
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
query = {}
return MongoDB.with_config(ConfigFileLoader(config_path, config_profile)).load(
query=query,
collection='collection_name',
)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to MongoDB:
```python theme={"system"}
from os import path
from pandas import DataFrame
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mongodb import MongoDB
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mongodb(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
MongoDB.with_config(ConfigFileLoader(config_path, config_profile)).export(
df,
collection='collection_name',
)
```
# MotherDuck
Source: https://docs.mage.ai/integrations/databases/MotherDuck
Execute SQL commands against a MotherDuck database.
The MotherDuck integration is supported in Mage versions `>=0.9.63`
## Credentials
Open the file named `io_config.yaml` at the root of your Mage project and enter DuckDB database name & schema.
To connect to a MotherDuck database, you'll need to add a `MOTHERDUCK_TOKEN` and a `md:` to the database name.
```yaml theme={"system"}
version: 0.1.1
default:
DUCKDB_DATABASE: md:database
DUCKDB_SCHEMA: main
MOTHERDUCK_TOKEN: token
```
## How to use
1. Create a new pipeline or edit an existing pipeline.
2. Add a data loader or transformer SQL block.
3. In the blockโs header, click the dropdown menu labeled Connection and select the option
labeled DuckDB.
4. In the blockโs header, click the dropdown menu labeled Profile and select the option
labeled default.
If the name of your profile in the `io_config.yaml` file is different,
then select the profile name that matches your settings.
5. You can optionally check the box labeled Use raw SQL.
If you do, read more about [how to use SQL blocks with raw SQL](/guides/blocks/sql-blocks#using-raw-sql).
6. Write your SQL statements in the SQL block.
7. Once youโre finished writing your SQL statements,
click the play button in the top right corner of the block to execute the SQL block and preview
the results.
# MySQL
Source: https://docs.mage.ai/integrations/databases/MySQL
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
### Standard connection
```yaml theme={"system"}
version: 0.1.1
default:
MYSQL_DATABASE: ...
MYSQL_HOST: ...
MYSQL_PASSWORD: ...
MYSQL_PORT: 3306
MYSQL_USER: root
```
### SSH tunnel connection
If your MySQL database is in a private network and you need to connect via an SSH bastion host, Mage Pro supports configuring an SSH tunnel directly in io\_config.yaml:
```yaml theme={"system"}
version: 0.1.1
default:
MYSQL_DATABASE: ...
MYSQL_HOST: ...
MYSQL_PASSWORD: ...
MYSQL_PORT: 3306
MYSQL_USER: root
# Pro only: SSH tunnel settings
MYSQL_SSH_HOST: bastion.example.com
MYSQL_SSH_PORT: 22
MYSQL_SSH_USERNAME: ec2-user
# Use one of the following for authentication:
MYSQL_SSH_PKEY: /path/to/private/key.pem # path to SSH private key
MYSQL_SSH_PASSWORD: your_ssh_password # only if using password auth
```
* `MYSQL_SSH_HOST`: Hostname or IP of the SSH bastion.
* `MYSQL_SSH_PORT`: SSH port (default 22).
* `MYSQL_SSH_USERNAME`: SSH user to log in as.
* `MYSQL_SSH_PKEY`: Path to private key file (preferred).
* `MYSQL_SSH_PASSWORD`: Password for SSH user (alternative to key).
โ ๏ธ You only need either `MYSQL_SSH_PKEY` or `MYSQL_SSH_PASSWORD`, not both.
***
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider`/`Connection` dropdown, select `Mysql`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Enter the optional table name of the table to write to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks) for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.
***
## Using Python block
### Using MySQL connector
```python theme={"system"}
import mysql.connector
conn = mysql.connector.connect(
host='...',
password='...',
port=3306,
user='root',
database='...',
)
sql = """
SELECT
id
, email
FROM auth_user
"""
cursor = conn.cursor()
cursor.execute(sql)
rows = cursor.fetchall()
import pandas as pd
df = pd.DataFrame(rows, columns=['id', 'email'])
df.to_csv('default_repo/users.csv', index=False)
```
***
### Using Mage's MySQL python blocks
1. Create a Data Loader block from the MySQL Template
Example snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from os import path
@data_loader
def load_data_from_mysql(*args, **kwargs):
"""
Template for loading data from a MySQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#mysql
"""
query = 'Your MySQL query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
2. Create a Data Exporter block from the MySQL Template
Example snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a MySQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#mysql
"""
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
None,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
)
```
3. Custom types
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config
Here is an example code snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a MySQL database.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#mysql
"""
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
None,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types=overwrite_types,
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'VARCHAR(255)'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |
**Example: Using UPSERT (UPDATE or INSERT)**
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.mysql import MySQL
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_mysql(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a MySQL database with UPSERT support.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#mysql
"""
table_name = 'your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
None,
table_name,
index=False,
if_exists='append', # Use 'append' with UPSERT
unique_constraints=['user_id', 'email'], # Columns that form unique constraint
unique_conflict_method='UPDATE', # Update existing rows, insert new ones
)
```
# OracleDB
Source: https://docs.mage.ai/integrations/databases/OracleDB
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
ORACLEDB_USER: ...
ORACLEDB_PASSWORD: ...
ORACLEDB_HOST: ...
ORACLEDB_PORT: 1521
ORACLEDB_SERVICE: ... # Service name (e.g. ORCL, XEPDB1)
ORACLEDB_MODE: thin # Optional: 'thin' (default) or 'thick'
```
Use `ORACLEDB_MODE: thick` only if you need the Oracle thick mode (e.g. for advanced features); otherwise `thin` is recommended.
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Oracle`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Enter your query (e.g. `SELECT 1 FROM DUAL`) and run the block.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.oracledb import OracleDB
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_oracledb(**kwargs) -> DataFrame:
query = 'SELECT 1 FROM DUAL'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with OracleDB.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
### Export a dataframe to OracleDB
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.oracledb import OracleDB
from os import path
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_oracledb(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
table_name = 'your_table_name'
with OracleDB.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(df, table_name=table_name, if_exists='replace')
```
## Connection modes
* **thin** (default): Pure Python driver; no Oracle Client installation required.
* **thick**: Uses Oracle Client libraries; required for some advanced scenarios. Ensure Oracle Instant Client or full client is installed and configured when using `ORACLEDB_MODE: thick`.
# Pinot
Source: https://docs.mage.ai/integrations/databases/Pinot
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
PINOT_HOST: ...
PINOT_PASSWORD: ...
PINOT_PATH: /query/sql
PINOT_PORT: 8000
PINOT_SCHEME: http
PINOT_USER: ...
```
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader.
3. Select `Pinot`.
4. You will get the below code snippet:
```python theme={"system"}
from mage_ai.data_preparation.repo_manager import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.pinot import Pinot
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_pinot(*args, **kwargs):
"""
Template for loading data from a Pinot warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#pinot
"""
query = 'your Pinot query' # Specify your SQL query here
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Pinot.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
@test
def test_output(output, *args) -> None:
"""
Template code for testing the output of the block.
"""
assert output is not None, 'The output is undefined'
```
5. Enter the SQL query in place of 'your Pinot query'.
6. Run the block.
# PostgreSQL
Source: https://docs.mage.ai/integrations/databases/PostgreSQL
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
POSTGRES_DBNAME: ...
POSTGRES_HOST: ...
POSTGRES_PASSWORD: ...
POSTGRES_PORT: ...
POSTGRES_USER: ...
POSTGRES_CONNECTION_METHOD: direct
POSTGRES_SSH_HOST:
POSTGRES_SSH_PORT: 22
POSTGRES_SSH_USERNAME:
POSTGRES_SSH_PASSWORD:
POSTGRES_SSH_PKEY:
```
### Pro-only fields
```yaml theme={"system"}
POSTGRES_SSL_MODE:
POSTGRES_SSL_ROOTCERT:
POSTGRES_SSL_CERT:
POSTGRES_SSL_KEY:
```
**Notes:**
* `POSTGRES_SSL_MODE` corresponds to the `sslmode` parameter in psycopg2 (e.g., `require`, `verify-full`).
* `POSTGRES_SSL_ROOTCERT`, `POSTGRES_SSL_CERT`, and `POSTGRES_SSL_KEY` should be absolute paths to your SSL files on disk.
### SSH tunneling
If you want to connect to Postgres cluster with ssh tunnel, update the value of `POSTGRES_CONNECTION_METHOD` to `ssh_tunnel`, and enter the values for keys with prefix `POSTGRES_SSH`.
| Key | Description | Sample value |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
| `POSTGRES_SSH_HOST` | The host of the intermediate bastion server. | `123.45.67.89` |
| `POSTGRES_SSH_PORT` | The port of the intermediate bastion server. Default value: 22 | `22` |
| `POSTGRES_SSH_USERNAME` | The username used to connect to the bastion server. | `username` |
| `POSTGRES_SSH_PASSWORD` | (Optional) The password used to connect to the bastion server. It should be set if you authenticate with the bastion server with password. | `password` |
| `POSTGRES_SSH_PKEY` | (Optional) The path to the private key used to connect to the bastion server. It should be set if you authenticate with the bastion server with private key. | `/path/to/private/key` |
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `PostgreSQL`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Next to the `Save to schema` label, enter the schema name you want this block
to save data to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks) for more
information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_postgres(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to Postgres:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
drop_table_on_replace=False, # Whether to drop the table when "if_exists" param is set to "replace"
allow_reserved_words=True,
unique_conflict_method='UPDATE',
unique_constraints=['col'],
)
```
6. Custom types
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config
Here is an example code snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_postgres(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
allow_reserved_words=True,
unique_conflict_method='UPDATE',
unique_constraints=['col'],
overwrite_types=overwrite_types,
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| cascade\_on\_drop | Whether to cascade the table drop. | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'VARCHAR'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. | 'UPDATE' (default: None) |
| unique\_constraints | The unique constraints of the table. | \['col1', 'col2'] (default: None) |
#### Pro-only fields
| Field name | Description | Example values |
| ------------ | ----------------------------------------------------------------- | --------------------------- |
| batch\_write | Use `COPY` command to write to PostgreSQL to improve performance. | True/False (default: False) |
# Power BI
Source: https://docs.mage.ai/integrations/databases/PowerBI
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
**Required:** Azure subscription ID and Power BI workspace ID. For authentication you can use either Azure AD (service principal) or default credentials.
```yaml theme={"system"}
version: 0.1.1
default:
AZURE_SUBSCRIPTION_ID: your_azure_subscription_id
POWER_BI_WORKSPACE_ID: your_power_bi_workspace_id
# Optional: Service principal (if not using DefaultAzureCredential)
AZURE_CLIENT_ID: ...
AZURE_CLIENT_SECRET: ...
AZURE_TENANT_ID: ...
```
If `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` are omitted, the client uses `DefaultAzureCredential` (e.g. Azure CLI login or managed identity). The service principal or user must have access to the Power BI workspace and the **Power BI Service** API.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
### Trigger a dataset refresh
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.power_bi import PowerBI
from os import path
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def trigger_power_bi_refresh(**kwargs):
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
dataset_id = '...' # Your Power BI dataset ID
client = PowerBI.with_config(ConfigFileLoader(config_path, config_profile))
client.load(dataset_id)
```
### Export data to a Power BI dataset
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.power_bi import PowerBI
from os import path
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_power_bi(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
dataset_id = '...'
client = PowerBI.with_config(ConfigFileLoader(config_path, config_profile))
client.export(df, dataset_id=dataset_id)
```
### Check if a dataset exists
```python theme={"system"}
client = PowerBI.with_config(ConfigFileLoader(config_path, config_profile))
exists = client.exists('your-dataset-id')
```
## Permissions
* Your Azure AD app or user needs the **Power BI Service** scope (e.g. `https://analysis.windows.net/powerbi/api/.default`).
* The app or user must have access to the specified Power BI workspace and dataset (e.g. Admin, Member, or Contributor on the workspace).
# Qdrant
Source: https://docs.mage.ai/integrations/databases/Qdrant
## Credentials
Open the file named `io_config.yaml` at the root of your Mage project and enter qdrant required fields:
```yaml theme={"system"}
version: 0.1.1
default:
QDRANT_COLLECTION: collection_name
QDRANT_PATH: path of the qdrant persisitant storage
```
## Dependencies
The dependency libraries are not installed in the docker image by default. You'll need to add the libraries to
project `requirements.txt` file manually and install them.
```
qdrant-client==1.6.9
sentence-transformers==2.2.2
```
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter using the Qdrant template under the "Databases" category.
Both the data loader and exporter use SentenceTransformer 'all-MiniLM-L6-v2' as the default embedding function.
3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.
## Available functions
* Qdrant data loader arguments:
* limit\_results (int): Number of results to return.
* query\_vector (List): vector lit used for query.
* collection\_name (str): name of the collection. Default to use the name defined in io\_config.yaml.
* Qdrant data exporter arguments:
* df (DataFrame): Data to export.
* document\_column (str): Column name containing documents to export.
* id\_column (str): Column name of the id. Default will use index in df.
* vector\_column (str): Column name of the vector. Will use default encoder to auto generate query vector.
* collection\_name (str): name of the collection. Default to use the name defined in io\_config.yaml.
* vector\_size (int): dimension size of vector.
* distance (models.Distance): distance metric to use.
At the same time there is `create_collection` function can be used in your block to create new collection.
# Redshift
Source: https://docs.mage.ai/integrations/databases/Redshift
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
REDSHIFT_DBNAME: ...
REDSHIFT_HOST: ...
REDSHIFT_PORT: ...
REDSHIFT_TEMP_CRED_USER: ...
REDSHIFT_TEMP_CRED_PASSWORD: ...
```
When connecting to **Redshift Serverless**, you can use `workgroup-name.account-number.aws-region.redshift-serverless.amazonaws.com`
as the `REDSHIFT_HOST` value.
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Redshift`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Next to the `Save to schema` label, enter the schema name you want this block
to save data to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks)
for more information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_redshift(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
6. Custom types.
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config
Here is an example code snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.redshift import Redshift
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_redshift(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Redshift cluster.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#redshift
"""
table_name = 'your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with Redshift.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
if_exists='replace', # Specify resolution policy if table already exists
overwrite_types=overwrite_types,
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| allow\_reserved\_words | Whether to allow using reserved words as column names. | True/False (default: False) |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'VARCHAR(255)'}` |
# S3
Source: https://docs.mage.ai/integrations/databases/S3
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
AWS_ACCESS_KEY_ID: ...
AWS_SECRET_ACCESS_KEY: ...
```
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or transformer block (the code snippet below is for a data
loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_s3_bucket(**kwargs) -> DataFrame:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = '...' # Change to your bucket name
object_key = '...' # Change to your object key
return S3.with_config(ConfigFileLoader(config_path, config_profile)).load(
bucket_name,
object_key,
)
```
5. Run the block.
### MinIO Support
To execute requests on MinIO databases, simply add:
```yaml theme={"system"}
AWS_ENDPOINT: your_endpoint:port #example: "https://play.min.io:9000"
```
to the `io_config.yaml` file
### Errors
S3 connection endpoint URL error
Open the `io_config.yaml` file at the root of your project (e.g.
`default_repo/io_config.yaml`).
Change the value for the `AWS_REGION` key to the region of your S3 bucket (e.g.
`us-west-2`).
## Permissions
Ensure your AWS IAM user or role has the following permissions:
* `s3:GetObject`
* `s3:PutObject`
* `s3:ListBucket`
These permissions are required to:
* Read data from S3 (e.g. `.csv`, `.parquet`, `.json`)
* Write query results or transformed data to S3
* List contents of a bucket when needed
# SQLite
Source: https://docs.mage.ai/integrations/databases/SQLite
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
sqlite:
database: /path/to/your/sqlite.db
```
**Notes:**
* `database` should be the absolute path to your SQLite database file
* The database file will be created automatically if it doesn't exist
* Make sure the directory containing the database file has proper write permissions
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `SQLite`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. Next to the `Save to schema` label, enter the schema name you want this block
to save data to.
7. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks) for more
information on write policies).
8. Enter in this test query: `SELECT 1`.
9. Run the block.
### Using raw SQL
You can also use raw SQL with SQLite by enabling the "Use raw SQL" option in the SQL block configuration. This allows you to write complete SQL statements including `CREATE TABLE`, `INSERT`, `UPDATE`, and `DELETE` statements.
**Example raw SQL:**
```sql theme={"system"}
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE
);
INSERT INTO users (name, email) VALUES
('John Doe', 'john@example.com'),
('Jane Smith', 'jane@example.com');
SELECT * FROM users;
```
**Note:** When using raw SQL, you're responsible for writing the complete SQL statements. Mage won't automatically create tables or handle data insertion.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_sqlite(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to SQLite:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_sqlite(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
drop_table_on_replace=False, # Whether to drop the table when "if_exists" param is set to "replace"
)
```
### Custom types
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config
Here is an example code snippet:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_sqlite(df: DataFrame, **kwargs) -> None:
schema_name = 'your_schema_name' # Specify the name of the schema to export data to
table_name = 'your_table_name' # Specify the name of the table to export data to
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'TEXT'}
with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False, # Specifies whether to include index in exported table
if_exists='replace', # Specify resolution policy if table name already exists
overwrite_types=overwrite_types,
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| auto\_clean\_name | Whether to automatically clean the column name (replace the empty space with underscore, avoid using number as the prefix of the column name) | True/False (default: True) |
| case\_sensitive | Whether to support case sensitive columns | True/False (default: False) |
| drop\_table\_on\_replace | Whether to drop the table when "if\_exists" param is set to "replace". | True/False (default: False) |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "replace") |
| overwrite\_types | Overwrite the column types | `{'column1': 'INTEGER', 'column2': 'TEXT'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |
**Example: Using UPSERT (UPDATE or INSERT)**
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.sqlite import SQLite
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_sqlite(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to SQLite with UPSERT support.
Specify your configuration settings in 'io_config.yaml'.
"""
schema_name = 'your_schema_name'
table_name = 'your_table_name'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with SQLite.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
schema_name,
table_name,
index=False,
if_exists='append', # Use 'append' with UPSERT
unique_constraints=['user_id', 'email'], # Columns that form unique constraint
unique_conflict_method='UPDATE', # Update existing rows, insert new ones
)
```
## SQLite-specific considerations
### Database file location
* SQLite databases are stored as single files on disk
* Ensure the file path is accessible and has proper read/write permissions
* Use absolute paths to avoid issues with working directory changes
### Schema support
* SQLite doesn't have true schema support like PostgreSQL
* The "schema" parameter in Mage is used for table naming conventions
* Tables are created in the main database without schema prefixes
### Data types
* SQLite uses dynamic typing (type affinity)
* Common types: `INTEGER`, `REAL`, `TEXT`, `BLOB`, `NULL`
* Mage will map pandas data types to appropriate SQLite types
### Performance tips
* SQLite performs best with smaller datasets (\< 1GB)
* Consider using `PRAGMA` statements for performance tuning
* Use transactions for bulk operations
# Snowflake
Source: https://docs.mage.ai/integrations/databases/Snowflake
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
```yaml theme={"system"}
version: 0.1.1
default:
SNOWFLAKE_USER: ...
SNOWFLAKE_PASSWORD: ...
SNOWFLAKE_ACCOUNT: ...
SNOWFLAKE_DEFAULT_WH: ...
SNOWFLAKE_DEFAULT_DB: ...
SNOWFLAKE_DEFAULT_SCHEMA: ...
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: ...
SNOWFLAKE_PRIVATE_KEY_PATH: ...
SNOWFLAKE_ROLE: ...
SNOWFLAKE_TIMEOUT: ...
```
## Using SQL block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block.
3. Select `SQL`.
4. Under the `Data provider` dropdown, select `Snowflake`.
5. Under the `Profile` dropdown, select `default` (or the profile you added
credentials underneath).
6. In the `Database` input in the block header, enter the database name you want this block to
save data to.
7. In the `Schema` input in the block header, enter the schema name you want this block
to save data to.
8. Under the `Write policy` dropdown, select `Replace` or `Append` (please see
[SQL blocks guide](/guides/blocks/sql-blocks)
for more information on write policies).
9. Enter in this test query: `SELECT 1`.
10. Run the block.
## Methods for configuring database and schema
You only need to include the database and schema config in one of these 3 places, in
order of priority (so if all 3 are included, #1 takes priority, then #2, then #3):
1. Include the db and schema directly in the query (e.g.
`select * from [database_name].[schema_name].[table_name];`). This is supported
when NOT using the "raw sql" query option.
2. Include the db and schema in the SQL code block header inputs, as mentioned in Steps
6 and 7 of the "Using SQL block" section above.
3. Include the default db and schema in the `io_config.yaml` file using these fields:
```
SNOWFLAKE_DEFAULT_DB: YOUR_DB_NAME
SNOWFLAKE_DEFAULT_SCHEMA: YOUR_SCHEMA_NAME
```
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_snowflake(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
```
5. Run the block.
### Export a dataframe
Here is an example code snippet to export a dataframe to Snowflake:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Snowflake warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#snowflake
"""
table_name = 'your_table_name'
database = 'your_database'
schema = 'your_schema'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
database=database,
schema=schema,
if_exists='append', # Specify resolution policy if table name already exists
)
```
### Custom types
To overwrite a column type when running a python export block, simply specify the column name and type in the `overwrite_types` dict in data exporter config:
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to a Snowflake warehouse.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#snowflake
"""
table_name = 'your_table_name'
database = 'your_database'
schema = 'your_schema'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
overwrite_types = {'column_name': 'VARCHAR(255)'}
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
database=database,
schema=schema,
if_exists='replace',
overwrite_types=overwrite_types,
)
```
### Method arguments
| Field name | Description | Example values |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| if\_exists | Specify resolution policy if table name already exists | "fail"/"replace"/"append" (default: "append") |
| overwrite\_types | Overwrite the column types | `{'column1': 'VARCHAR', 'column2': 'NUMBER'}` |
| unique\_conflict\_method | How to handle the conflict on unique constraints. Use 'UPDATE' for UPSERT (update existing rows, insert new ones) or 'IGNORE' to skip duplicates. | 'UPDATE' or 'IGNORE' (default: None) |
| unique\_constraints | The unique constraints of the table. Used with unique\_conflict\_method for UPSERT operations. | \['col1', 'col2'] (default: None) |
**Example: Using UPSERT (UPDATE or INSERT)**
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.snowflake import Snowflake
from pandas import DataFrame
from os import path
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_snowflake(df: DataFrame, **kwargs) -> None:
"""
Template for exporting data to Snowflake with UPSERT support.
Specify your configuration settings in 'io_config.yaml'.
Docs: https://docs.mage.ai/design/data-loading#snowflake
"""
table_name = 'your_table_name'
database = 'your_database'
schema = 'your_schema'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Snowflake.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
loader.export(
df,
table_name,
database=database,
schema=schema,
if_exists='append', # Use 'append' with UPSERT
unique_constraints=['user_id', 'email'], # Columns that form unique constraint
unique_conflict_method='UPDATE', # Update existing rows, insert new ones
)
```
# Spark
Source: https://docs.mage.ai/integrations/databases/Spark
## Add credentials
1. Create a new pipeline or open an existing pipeline.
2. Expand the left side of your screen to view the file browser.
3. Scroll down and click on a file named `io_config.yaml`.
4. Enter the following keys and values under the key named `default` (you can
have multiple profiles, add it under whichever is relevant to you)
For a **local Spark session**:
```yaml theme={"system"}
version: 0.1.1
default:
SPARK_HOST: local
SPARK_METHOD: session
SPARK_SCHEMA: default
```
For a **remote Spark cluster**, set `SPARK_HOST` to the Spark master URL (for example, `spark://host:7077`, `local[*]`, or another valid `SparkSession.builder.master(...)` value). You can also set `SPARK_METHOD` (e.g. `session`) and `SPARK_SCHEMA` to control how the session is created and which default database/schema is used.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader, transformer, or data exporter block (the code snippet
below is for a data loader).
3. Select `Generic (no template)`.
4. Enter this code snippet (note: change the `config_profile` from `default` if
you have a different profile):
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.spark import Spark
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_data_from_spark(**kwargs) -> DataFrame:
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
loader = Spark.with_config(ConfigFileLoader(config_path, config_profile))
return loader.load(query)
```
5. Run the block.
### Export a dataframe to Spark
```python theme={"system"}
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.spark import Spark
from os import path
from pandas import DataFrame
if 'data_exporter' not in globals():
from mage_ai.data_preparation.decorators import data_exporter
@data_exporter
def export_data_to_spark(df: DataFrame, **kwargs) -> None:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
table_name = 'your_table_name'
database = 'default'
loader = Spark.with_config(ConfigFileLoader(config_path, config_profile))
loader.export(df, table_name=table_name, database=database, if_exists='replace')
```
## Notes
* Spark runs in-process; ensure PySpark and any required cluster dependencies are installed in your Mage environment.
* For local development, `SPARK_HOST: local` typically creates a session with `SparkSession.builder.master('local').getOrCreate()`.
* Use `SPARK_SCHEMA` to set the default database/schema for queries and exports.
# Weaviate
Source: https://docs.mage.ai/integrations/databases/Weaviate
## Credentials
Open the file named `io_config.yaml` at the root of your Mage project and enter weaviate required fields:
```yaml theme={"system"}
version: 0.1.1
default:
WEAVIATE_ENDPOINT: https://some-endpoint.weaviate.network
WEAVIATE_INSTANCE_API_KEY: YOUR-WEAVIATE-API-KEY
WEAVIATE_INFERENCE_API_KEY: YOUR-OPENAI-API-KEY
WEAVIATE_COLLECTION: collectionn_name
```
## Required Weaviate Libraries
Add `weaviate-client==3.25.3` into `requirements.txt` to install the required weaviate library.
## Using Python block
1. Create a new pipeline or open an existing pipeline.
2. Add a data loader or data exporter with Template. Under "Databases" category you can
find the "Weaviate" template. Then use corresponding load and export functions.
* data loader arguments:
* properties (List): list of columns to load.
* collection (str): name of the collection. Defaults to the name defined in io\_config.yaml.
* with\_limit (int): limit number of results returned.
* with\_text (str): text to query.
* data exporter arguments:
* df (DataFrame): data frame to write.
* collection (str): name of the collection.
3. Add your customized code into the loader, exporter or add extra transformer blocks.
4. Run the block.
# Trino SQL blocks
Source: https://docs.mage.ai/integrations/databases/trino
Execute SQL commands in Trino.
## Credentials
Open the file named `io_config.yaml` at the root of your Mage project and enter the following:
```yaml theme={"system"}
version: 0.1.1
default:
trino:
catalog: postgresql # Change this to the catalog of your choice
host: 127.0.0.1
http_headers:
X-Something: 'mage=power'
http_scheme: http
password: mage1337 # Optional
port: 8080
schema: core_data
session_properties: # Optional
acc01.optimize_locality_enabled: false
optimize_hash_generation: true
source: trino-cli # Optional
user: admin
verify: /path/to/your/ca.crt # Optional
data_type_properties:
timestamp_precision: 3 # Optional. Set TIMESTAMP(n) precision
overwrite_types: path_to_types_json # Optional
```
Change the values according to your Trino settings.
## How to use
1. Trino must be running on a remote server or on your local workstation.
Read these [instructions](https://github.com/mage-ai/mage-ai/blob/master/mage_integrations/mage_integrations/destinations/trino/README.md#run-trino-locally)
to run Trino locally.
2. Create a new pipeline or edit an existing pipeline.
3. Add a data loader or transformer SQL block.
4. In the blockโs header, click the dropdown menu labeled Connection and select the option
labeled Trino.
5. In the blockโs header, click the dropdown menu labeled Profile and select the option
labeled default.
If the name of your profile in the `io_config.yaml` file is different,
then select the profile name that matches your settings.
6. You can optionally check the box labeled Use raw SQL.
If you do, read more about [how to use SQL blocks with raw SQL](/guides/blocks/sql-blocks#using-raw-sql).
7. You can optionally overwrite your table columns types by adding a JSON to `overwrite_types` config, the format needs to be `{"column_name":"VARCHAR"}`
8. Write your SQL statements in the SQL block.
9. Once youโre finished writing your SQL statements,
click the play button in the top right corner of the block to execute the SQL block and preview
the results.
# dbt Cloud in Mage
Source: https://docs.mage.ai/integrations/dbt-cloud
Trigger model runs in dbt Cloud.
## Configuration
Here are the following keyword arguments that can be used for configuration:
| Keyword argument | Description | Sample value |
| ---------------- | ------------------------------------------------------------------------------------ | ------------ |
| `account_id` | dbt Cloud account ID | `12345` |
| `api_token` | [Service account tokens](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) | `abc123` |
***
## Example code
Trigger job run with `job_id`.
```python theme={"system"}
from mage_ai.services.dbt.dbt import DbtCloudClient
client = DbtCloudClient(dict(account_id=12345, api_token='abcdefg'))
# Set poll_status=True to wait for job completion.
# Set poll_status=False to not wait for job completion.
client.trigger_job_run(12345, poll_status=True)
```
Sample output:
```
Polling dbt Cloud run 12345. Current status: Starting.
Polling dbt Cloud run 12345. Current status: Running.
Polling dbt Cloud run 12345. Current status: Success.
Job run status for run 12345: Success. Polling complete
```
***
## Methods
Common [APIs in dbt Cloud](https://docs.getdbt.com/dbt-cloud/api-v2) implemented in the
[`DbtCloudClient`](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/services/dbt/dbt.py).
1. `get_run`
2. `list_jobs`
3. `list_runs`
4. `trigger_job_run`
# Hightouch in Mage
Source: https://docs.mage.ai/integrations/hightouch
Trigger syncs in Hightouch.
## Configuration
Here are the following keyword arguments that can be used for configuration:
| Keyword argument | Description | Sample value |
| ---------------- | ------------------------------------------------------------------------- | ------------ |
| `api_key` | [Hightouch API key](https://hightouch.com/docs/developer-tools/api-guide) | `abc123` |
***
## Example code
Trigger sync with `sync_id`.
```python theme={"system"}
from mage_ai.services.hightouch.hightouch import HightouchClient
client = HightouchClient(config=dict(api_key='api_key'))
syncs = client.list_syncs() # Get all the syncs
sync_id = syncs['data'][0]['id'] # Get sync id
client.sync_and_poll(sync_id) # Trigger sync and poll status
```
Sample output:
```
Polling Hightouch Sync [id]. Current status: querying. 0% completed.
Polling Hightouch Sync [id]. Current status: success. 100% completed.
Sync request status: success. Polling complete
```
# Run Mage pipelines in Prefect
Source: https://docs.mage.ai/integrations/prefect
We support running the pipeline in Prefect flows.
You need to firstly install `mage_ai` library by adding `mage_ai` to your
requirements.txt file. Then you need to download the mage pipeline code into
your Prefect directory. You can achieve it by using a git submodule in your
Prefect directory.
We provide two ways to run mage pipelines in Prefect.
1. Run pipeline in as a task or ShellTask.
2. Run pipeline as a Prefect Flow.
## Run pipeline as a Prefect task
```python Example code theme={"system"}
from prefect import task, Flow
import mage_ai
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
@task
def run_pipeline():
mage_ai.run(pipeline_name, project_path)
with Flow('My Mage Flow') as flow:
run_pipeline()
flow.run()
```
## Run pipeline in a ShellTask
```python Example code theme={"system"}
from prefect import task, Flow
from prefect.tasks.shell import ShellTask
from prefect.triggers import all_finished
import mage_ai
import os
import prefect
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'project_path')
pipeline_name = 'pipeline_name'
mage_task = ShellTask(return_all=True)
@task(trigger=all_finished)
def output_print(output):
logger = prefect.context.get("logger")
for o in output:
logger.info(o)
with Flow('My Mage Flow') as flow:
run_pipeline = mage_task(command=f'mage run {project_path} {pipeline_name}')
output = output_print(run_pipeline)
flow.run()
```
## Run pipeline as a Prefect flow
```python Example code theme={"system"}
from prefect import Task, Flow
from mage_ai.data_preparation.models.pipeline import Pipeline
import asyncio
import os
ABSOLUTE_PATH = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.join(ABSOLUTE_PATH, 'prefect_repo')
pipeline_name = 'default_pipeline'
pipeline = Pipeline(pipeline_name, repo_path=project_path)
tasks = {}
class ExecuteBlock(Task):
def run(self, block):
asyncio.run(
block.execute(
analyze_outputs=False,
update_status=False,
)
)
flow = Flow('My Mage Flow')
blocks_by_uuid = pipeline.blocks_by_uuid
for uuid, b in blocks_by_uuid.items():
if b.type == 'scratchpad':
continue
task = ExecuteBlock()
tasks[uuid] = task
for uuid, task in tasks.items():
flow.set_dependencies(
task=task,
upstream_tasks=[
tasks[block_uuid] for block_uuid in blocks_by_uuid[uuid].upstream_block_uuids
],
keyword_tasks=dict(block=blocks_by_uuid[uuid])
)
flow.run()
```
# Stitch in Mage
Source: https://docs.mage.ai/integrations/stitch
Trigger syncs in Stitch.
## Configuration
Here are the following keyword arguments that can be used for configuration:
| Keyword argument | Description | Sample value |
| ---------------- | ------------------------------------------------------------------------------------------------------------ | ------------ |
| `access_token` | [Stitch API access token](https://www.stitchdata.com/docs/developers/stitch-connect/api#obtain-access-token) | `abc123` |
***
## Example code
Trigger sync with `source_id`.
```python theme={"system"}
from mage_ai.services.stitch.stitch import StitchClient
client = StitchClient(config=dict(access_token='access_token'))
sources = client.list_sources()
source_id = sources[0]['id'] # Get source id
client.start_replication_job(source_id) # Start replication job
```
Sample output:
```
Start replication job for source 11111. Job name: 111111.222222.sync.abcd1234-a96e-4b48-a10c-9de147a72d64.
Polling Stitch extraction status for source 11111. Current status: running.
Polling Stitch extraction status for source 11111. Current status: running.
Polling Stitch extraction status for source 11111. Current status: running.
Extraction for source 11111 completed.
Finish loading data for all streams: ['stream_1', 'stream_2'].
```
# StrongDM
Source: https://docs.mage.ai/integrations/strongdm
## Setup Docker environment
To effectively connect on a StrongDM Resource inside Mage's docker, you may need to add:
If using docker run command:
`--add-host=host.docker.internal:host-gateway`
If using docker-compose setup:
```yaml theme={"system"}
services:
server:
.....
extra_hosts:
- "host.docker.internal:host-gateway"
```
## Install StrongDM CLI tool
To install StrongDM CLI tool, first access Mage's docker bash and execute the installation script:
```bash theme={"system"}
./scripts/strongdm/install.sh
```
(In case you receive a "Permission denied" error, executing `chmod u+x scripts/strongdm/install.sh` should fix this issue)
Now, login in your StrongDM account using:
```bash theme={"system"}
sdm login
```
And connect to the desired Resources using:
```bash theme={"system"}
sdm connect RESOURCE-NAME
```
## Accessing StrongDM Resources
Now, to connect to a Resource, simply add the required host parameter as "localhost" and the respective port number
MongoDB Source example:
```yaml theme={"system"}
database: admin
host: localhost
password: example
port: 10006 # Specified in StrongDM Resource Generation
user: root
```
# Using an external IDE
Source: https://docs.mage.ai/integrations/text-editor
We love our UI, but here's how to develop in your favorite IDE, too.
## Create a new pipeline
Each pipeline is represented by a YAML file in a folder named `pipelines/` under the Mage
project directory.
For example, if your project is named `demo_project` and your pipeline is named `etl_demo` then
youโll have a folder structure that looks like this:
```
demo_project/
| pipelines/
| | etl_demo/
| | | __init__.py
| | | metadata.yaml
```
Create a new folder in the `demo_project/pipelines/` directory. Name this new folder after the name
of your pipeline.
Add 2 files in this new folder:
1. `__init__.py`
2. `metadata.yaml`
In the `metadata.yaml` file, add the following content:
```yaml theme={"system"}
blocks: []
name: etl_demo
type: python
uuid: etl_demo
```
Change `etl_demo` to whatever name youโre using for your new pipeline.
## Sample pipeline metadata content
This sample pipeline `metadata.yaml` will produce the following block dependencies:
```yaml theme={"system"}
blocks:
- downstream_blocks:
- select_columns
executor_type: local_python
language: python
name: load_data_from_file
type: data_loader
upstream_blocks: []
uuid: load_data_from_file
- downstream_blocks:
- export_to_file
executor_type: local_python
language: python
name: select_columns
type: transformer
upstream_blocks:
- load_data_from_file
uuid: select_columns
- downstream_blocks: []
executor_type: local_python
language: python
name: export_to_file
type: data_exporter
upstream_blocks:
- select_columns
uuid: export_to_file
name: etl_demo
type: python
uuid: etl_demo
```
## `metadata.yaml` sections
### Pipeline attributes
An array of blocks that are in the pipeline.
Unique name of the pipeline.
The type of pipeline. Currently available options are:
* `databricks`
* `integration`
* `pyspark`
* `python` (most common)
* `streaming`
Unique identifier of the pipeline. This UUID must be unique across all pipelines.
Optional description of what the pipeline does.
Pipeline level executor type. Supported values:
* `ecs`
* `gcp_cloud_run`
* `azure_container_instance`
* `k8s`
* `local_python` (most common)
* `pyspark`
Number of concurrent executors to run the pipeline. Used in streaming pipeline.
Optional configuration specific to the selected executor type.
Refer to the following documentation for executor-specific options:
* [`ecs`](/production/executors/aws)
* [`gcp_cloud_run`](/production/executors/gcp)
* [`azure_container_instance`](/production/executors/azure)
* [`k8s`](/production/executors/k8)
Spark-specific configuration for PySpark pipelines. Mirrors the keys you
would normally pass to `SparkConf` (e.g., `spark_master`, `executor_env`,
`spark_jars`).
Retry configuration at the pipeline level.
See [documentation](/orchestration/pipeline-runs/retrying-block-runs#3-pipeline-level-retry-config) for details.
Configuration for pipeline notification messages (e.g., on failure or success).
See [documentation](/observability/alerting/alerting-slack) for details.
Concurrency settings for block execution within the pipeline.
See [documentation](/design/data-pipeline-management#pipeline-level-concurrency) for details.
* `block_run_limit`: Maximum number of blocks that can run in parallel.
* `pipeline_run_limit`
* `pipeline_run_limit_all_triggers`
* `on_pipeline_run_limit_reached`
Whether to cache block output in memory during execution.
If true, runs all blocks in a single process or k8s pod.
Blocks that run after your main graph finishes (e.g., notifications, cleanup).
Same shape as items in `blocks`.
Conditional blocks that can short-circuit or branch execution. Same shape as
items in `blocks`.
Extension blocks (e.g., Great Expectations). Same shape as items in `blocks`.
Chart or reporting blocks shown in the pipeline UI. Same shape as items in
`blocks`.
Pipeline-level settings; currently supports `triggers.save_in_code_automatically`
to control whether new/updated triggers are persisted to YAML.
Key/value pairs available to the pipeline for templating. Useful for
environment-specific values that are not secrets.
Configure an external state store for pipeline runs.
Free-form labels to organize and search for pipelines.
Identifier for the user or process that created the pipeline.
Environment-specific overrides for any top-level pipeline field. Mage Pro only.
See [environment overrides](/extensibility/env-config/pipeline) for guidance.
### Block attributes
An array of block UUIDs that depend on this current block.
These downstream blocks will have access to this current blockโs data output.
The method for running this block of code. Currently available options are:
* `ecs`
* `gcp_cloud_run`
* `azure_container_instance`
* `k8s`
* `local_python` (most common)
* `pyspark`
Optional configuration specific to the selected executor type.
Refer to the following documentation for executor-specific options:
* [`ecs`](/production/executors/aws)
* [`gcp_cloud_run`](/production/executors/gcp)
* [`azure_container_instance`](/production/executors/azure)
* [`k8s`](/production/executors/k8)
Programming language used by the block. Supported values:
* `python` (most common)
* `r`
* `sql`
* `yaml`
Unique name of the block.
The type of block. Currently available options are:
* `chart`
* `custom` (most common)
* `data_exporter`
* `data_loader`
* `dbt`
* `scratchpad`
* `sensor`
* `transformer`
The type of block will determine which folder it needs to be in. For example, if the block `type`
is `data_loader`, then the file must be in the `[project_name]/data_loaders/` folder. It can be
nested in any number of subfolders.
An array of block UUIDs that this current block depends on.
These upstream blocks will pass its data output to this current block.
Unique identifier of the block. This UUID must be unique within the current pipeline. The UUID
corresponds to the name of the file for this block.
For example, if the UUID is `load_data` and the `language` is `python`, then the file name
will be `load_data.py`.
Optional HEX color used to visually organize blocks in the UI.
Block-specific configuration. For integrations or dbt blocks this mirrors the
configuration visible in the block settings panel.
Markdown content rendered in the blockโs documentation tab.
Optional priority for scheduling blocks when multiple are runnable.
Retry configuration at the block level.
See [documentation](/orchestration/pipeline-runs/retrying-block-runs#2-block-level-retry-config) for details.
Labels applied to the block for organization and filtering.
Maximum execution time for the block in seconds.
# Itโs magic.
Source: https://docs.mage.ai/introduction/about-mage-pro
Give your data team magical powers.
Mage Pro is a modern data pipeline tool designed to address data engineering challenges at scale. Itโs a unified platform for data extraction, transformation, loading, and orchestration. It bridges the gap between raw data sources and analytics ready systems. Mage Pro is more performant, reliable, and cost effective than Mage OSS.
Mage Pro can be deployed in 4 ways:
* **Mage Pro SaaS** - Is fully managed and your Mage code executes in our environment.
* **Mage Pro hybrid cloud** - Where Mage Pro infrastructure maintains the control plane, but the data plane is deployed in your cloud.
* **Mage Pro private cloud** - Both control and data plane are deployed in your cloud.
* **Mage Pro on-prem** - Where Mage Pro infrastructure is deployed on your on-prem servers.
Additionally, Mage Pro adds some core capabilities that are not found in the Mage OSS environment. True to Mageโs philosophy on developer implementation and collaboration, many of the additional features make Mage Pro even more developer friendly. Some of those core capabilities are:
* **Cluster health insights**, check how your clusters are performing in Mage Proโs cluster manager insights application. Get pricing estimates and real time CPU and Memory usage.
* **Workspaces** - Segment one or more of your Mage Pro clusters into multiple development environments
* **CICD Deployments** - A powerful tool to easily deploy new versions of your code at the click of a button.
* **Pipeline dependency views** - Have pipelines that orchestrate or depend on other pipelines? Now you can have a view of this in this Mage Pro only application.
Mageโs SaaS offering can be deployed from any cloud region in the world. Currently Mage Pro is deployed in the following regions US West, US East, Canada, Europe, Asia, and Australia.
Letโs get you started on Mage Pro now!
# Itโs magic.
Source: https://docs.mage.ai/introduction/overview
Give your data team magical powers.
Mage is a hybrid framework for transforming and integrating data. It combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.
* Extract and synchronize data from 3rd party sources.
* Transform data with real-time and batch pipelines using Python, SQL, and R.
* Load data into your data warehouse or data lake using our pre-built connectors.
* Run, monitor, and orchestrate thousands of pipelines without losing sleep.
Plus hundreds of enterprise-class features, infrastructure innovations, and magical surprises.
***
**Available in two spellbinding versions**
For teams. Fully managed platform
for integrating and transforming data.
Self-hosted. System to build, run, and
manage data pipelines.
***
## Getting started
Get quickly started on building and deploying magical data pipelines
by watching a short product video and trying out the interactive demo playground.
Get a 5 minute overview
Open our interactive demo
## ๐ฆ Make data magical
We put the *fun* in fun-ctional programming. Mage is a hybrid framework that combines the best of both worlds: the flexibility of notebooks with the rigor of modular code.
* **Extract** and synchronize data from 3rd party sources.
* **Transform** data with real-time and batch pipelines using Python, SQL, and R.
* **Load** data into your data warehouse or data lake using our pre-built connectors.
* Run, monitor, and **orchestrate** thousands of pipelines without losing sleep.
Have you met anyone who loves developing in Airflow? That's why we designed an easy developer experience that you'll enjoy.
* **Simple developer experience**: Start developing locally with a single command or launch a dev environment in your cloud using Terraform.
* **Language of choice**: Write code in Python, SQL, or R in the same pipeline for *ultimate* flexibility.
* **Engineering best practices built-in**: Every step in your pipeline is a standalone file with modular code that's reusable and testable. No more DAGs with spaghetti code ๐.
* **Bring your favorite tools:** Write [dbt](/guides/dbt/overview) models, use your favorite [IDE](/guides/developer-ux/code-editor), [track changes with Git](/guides/data-sync/version-control), and much much more.
Don't waste time waiting for your DAGs to finish testing. Get instant feedback from your code *every* time you run it.
* **Interactive code**: Immediately see results from your code's output with an interactive notebook UI.
* **Data is a first-class citizen**: Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.
* **Collaborate on cloud**: Develop collaboratively on [cloud resources](/production/deploying-to-cloud/architecture), version control with [Git](/development/git/setting-up-git), and [test](/observability/testing/data-validation) pipelines without waiting for an available shared staging environment.
Don't have a large team dedicated to Airflow? Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.
* **Fast deploy**: Deploy Mage to [AWS](/production/deploying-to-cloud/aws/setup), [GCP](/production/deploying-to-cloud/gcp/setup), or [Azure](/production/deploying-to-cloud/azure/setup) with only 2 commands using maintained Terraform templates.
* **Scaling made simple**: Transform very large datasets directly in your data warehouse or through a native integration with [Spark](/integrations/spark-pyspark).
* **Observability**: Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.
## ๐ฎ [Features](/about/features)
| | | |
| --- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| ๐ถ | [Orchestration](/design/data-pipeline-management) | Schedule and manage data pipelines with observability. |
| ๐ | [Notebook editor](/about/features#notebook-for-building-data-pipelines) | Interactive Python, SQL, & R editor for coding data pipelines. |
| ๐๏ธ | [Data integration](/data-integrations) | Synchronize data from 3rd party sources to your internal destinations. |
| ๐ฐ | [Streaming](/guides/streaming/introduction) | Ingest and transform real-time data. |
| ๐งฑ | [dbt](/guides/dbt) | Build, run, and manage your dbt models with Mage. |
## ๐๏ธ [Core design principles](/design/core-design-principles)
Every user experience and technical design decision adheres to these principles.
| | | |
| -- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| ๐ป | [Easy developer experience](/design/core-design-principles#easy-developer-experience) | Open-source engine that comes with a custom notebook UI for building data pipelines. |
| ๐ข | [Engineering best practices](/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
| ๐ณ | [Data as a first-class citizen](/design/core-design-principles#data-is-a-first-class-citizen) | Designed from the ground up specifically for running data-intensive workflows. |
| ๐ช | [Scaling made simple](/design/core-design-principles#scaling-is-made-simple) | Analyze and process large data quickly for rapid iteration. |
## ๐ธ [Core abstractions](/design/core-abstractions)
These are the fundamental concepts that Mage uses to operate.
| | | |
| ----- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ๐ข | [Project](/design/core-abstractions#project) | Like a repository on GitHub; this is where you write all your code. |
| ๐ช | [Pipeline](/design/core-abstractions#pipeline) | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
| ๐งฑ | [Block](/design/core-abstractions#block) | A file with code that can be executed independently or within a pipeline. |
| ๐ค | [Data product](/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage. |
| โฐ | [Trigger](/design/core-abstractions#trigger) | A set of instructions that determine when or how a pipeline should run. |
| ๐โโ๏ธ | [Run](/design/core-abstractions#run) | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
## ๐จโ๐ฉโ๐งโ๐ฆ Community
Individually, we're a mage.
> ๐ง Mage
>
> Magic is indistinguishable from advanced technology. A mage is someone who
> uses magic (aka advanced technology).
Together, we're Magers!
> ๐งโโ๏ธ๐ง Magers (`/หmฤjษr/`)
>
> A group of mages who help each other realize their full potential! Join us on [Slack](https://www.mage.ai/chat).
โจ This documentation & project are brought to you by the following *magical* individuals (learn more about contributing [here](/contributing/overview)):
***
# Migrate from Airbyte to Mage Pro
Source: https://docs.mage.ai/migrations/airbyte-to-mage-pro
Many teams start with Airbyte for data integrationโbut soon encounter challenges with flexibility, orchestration, and observability. Mage Pro offers a unified alternative: a single platform for data integration, transformation, and orchestration with **AI-assisted pipeline development** and **enterprise-ready scalability**.
## Why migrate from Airbyte to Mage Pro?
Airbyte is popular for ELT pipelines, but it wasn't designed for:
* Native orchestration of transformations
* Visual, modular pipeline building
* AI-powered debugging and code generation
* Git-native CI/CD and multi-environment deployments
* Streaming pipelines and event triggers
Mage Pro replaces Airbyteโs connector-based model with **end-to-end data integration pipelines** that seamlessly combine **extract, load, and transform steps**โall while giving teams full code flexibility and better observability.
***
## โจ Mage Pro vs Airbyte: Benefits Overview
Mage Pro combines [**data integration**](/data-integrations/overview) and [**data orchestration**](/orchestration/triggers/triggering-pipelines) in a single, developer-friendly platform.
| Capability | Airbyte | Mage Pro |
| ------------------------------ | ------------------------------------------------ | ---------------------------------------------------------- |
| **Data integration** | โ Pre-built connectors | โ Native connectors for APIs, databases, files |
| **Orchestration** | โ ๏ธ Basic scheduling only | โ Full DAG-level orchestration |
| **Visual pipeline UI** | โ ๏ธ Connector-based only | โ Drag-and-drop pipeline builder |
| **Custom transformations** | โ ๏ธ Limited (requires dbt or custom code) | โ Python, SQL, or AI-generated transformations |
| **AI assistance** | โ | โ Generate, debug, and optimize pipelines with AI Sidekick |
| **Incremental loads** | โ Supported via config | โ Supported natively in pipeline settings |
| **Schema discovery & testing** | โ ๏ธ Limited | โ Data previews, schema validation, tests |
| **Streaming & CDC** | โ ๏ธ Limited | โ Native support for Kafka, CDC, and real-time ingestion |
| **Git integration** | โ ๏ธ Manual export/import only | โ Git-backed pipelines and CI/CD |
| **Secrets management** | โ ๏ธ Env-only | โ Built-in secrets & workspace isolation |
| **Multi-environment support** | โ Manual, not native | โ Dev, staging, and production workspaces |
| **Observability** | โ ๏ธ Basic logs | โ Detailed logs, lineage, and monitoring |
| **Scalability** | โ ๏ธ Self-managed in OSS, limited control in Cloud | โ Auto-scaled executors on Kubernetes or ECS |
***
## ๐ ๏ธ Step-by-Step Migration Instructions
1. **Inventory your Airbyte connectors**
* List all sources and destinations in your Airbyte setup
* Note incremental or full-load settings
* Export transformation logic (e.g., dbt, SQL scripts)
2. **Create a Mage Pro workspace**
* Sign up at [Mage Pro](https://cloud.mage.ai)
* (Optional) Connect your Git repository for version control
3. **Recreate connectors as Mage data integration blocks**
* Each Airbyte source โ **Extract block** in Mage
* Each Airbyte destination โ **Load block** in Mage
* Any dbt or SQL logic โ **SQL block** (dbt-like features built-in)
* Use **Python blocks** for custom transformations
4. **Configure pipelines**
* Combine extract โ transform โ load in a single pipeline
* Use the visual editor or YAML pipeline config
5. **Set up scheduling and triggers**
* Replace Airbyte sync schedules with Mage cron or event-based triggers
* Optionally add webhooks or file-based triggers
6. **Run and validate pipelines**
* Preview data outputs at every step
* Compare final tables and schema to ensure parity
* Validate incremental load and deduplication logic
7. **Monitor and optimize**
* Use Mageโs built-in logs and data lineage
* Configure alerts and retry rules
* Leverage auto-scaling for parallel extracts and loads
***
## ๐ค Convert Airbyte Configs with AI Sidekick
Mage Proโs **AI Sidekick** can automatically help convert Airbyte connector configurations and dbt logic into Mage pipelines.
### ๐ง How to Use It
1. Open your Mage Pro workspace and click **โAsk AIโ**
2. Paste your Airbyte connector settings or dbt SQL models
3. Ask: **โConvert this Airbyte source-destination flow into a Mage pipeline.โ**
4. Sidekick will:
* Generate Extract/Load blocks for your sources and targets
* Translate transformation steps into SQL or Python blocks
* Create a complete pipeline with triggers and dependencies
5. Insert the generated pipeline directly into your workspace.
***
## ๐ง Tips for Migrating Complex Airbyte Flows
* Merge multiple Airbyte connectors into a **single end-to-end pipeline**
* Use **pipeline variables** for shared config (e.g., table names, batch sizes)
* Replace dbt transformations with **SQL blocks** (supports `ref()`, tests, and incremental models)
* For large tables, enable **batching and streaming modes**
***
## โ After Migration: What You Gain
* Unified **integration + transformation + orchestration**
* Visual pipelines and instant debugging
* AI-powered pipeline creation and troubleshooting
* Git-based CI/CD and environment isolation
* Real-time streaming support (Kafka, CDC, webhooks)
* Enterprise-grade RBAC, audit logs, and observability
> No more juggling Airbyte, Airflow, and dbt. With Mage Pro, you get a single platform for all data workflows.
๐ [Start Migrating to Mage Pro](https://cloud.mage.ai)
# Migrate from Airflow to Mage Pro
Source: https://docs.mage.ai/migrations/airflow-to-mage-pro
Many teams start with Apache Airflow for orchestrationโbut soon encounter challenges with scalability, debugging, and developer productivity. Mage Pro offers a modern alternative: visual pipelines, built-in observability, and AI-assisted development, with zero DevOps overhead.
## Why migrate from Airflow to Mage Pro?
Airflow is a powerful scheduler, but it wasn't designed for:
* Debugging pipelines visually
* Mixing SQL and Python naturally
* Streaming or event-driven workflows
* Managing secrets, Git, or multi-tenant workspaces
* AI-powered pipeline development
Mage Pro is built to solve these challenges out-of-the-box, with a modern developer experience and enterprise-grade scalability.
***
## โจ Mage Pro vs Airflow: Benefits Overview
Mage Pro goes far beyond orchestration. Itโs a unified platform for **data integration**, **SQL modeling (dbt-like blocks)**, **AI-powered transformation**, and **streaming pipelines** โ all within a collaborative, Git-native workspace.
| Capability | Airflow | Mage Pro |
| ------------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------- |
| **Visual pipeline UI** | โ Code-only (Jinja, Python) | โ Hybrid UI: drag-and-drop DAG builder + notebook-style coding experience |
| **Visual pipeline UI** | โ Code-only (Jinja, Python) | โ Drag-and-drop editor |
| **Built-in lineage** | โ ๏ธ Plugin-based (e.g., OpenLineage) | โ Native, auto-generated |
| **AI assistance** | โ | โ Generate, fix, and explain code with AI Sidekick |
| **Multi-language support** | โ ๏ธ Python only by default | โ SQL, Python, streaming, APIs |
| **Data integration pipelines** | โ ๏ธ Custom Python or 3rd-party plugins | โ Native connectors for databases, files, APIs |
| **SQL block support** | โ ๏ธ SQLOperator runs raw strings only | โ dbt-style SQL blocks with `ref()`, preview, and test support |
| **Incremental modeling** | โ (requires custom logic) | โ Native in SQL block config |
| **Environment isolation** | โ ๏ธ Requires Airflow deployments per team | โ Per-workspace configs, secrets, variables |
| **Git integration** | โ ๏ธ Manual DAG sync | โ Git-backed version control and CI/CD |
| **Scheduling & triggers** | โ Cron + sensors | โ Cron, events, file triggers, webhooks |
| **Secrets management** | โ ๏ธ Requires Vault or plugins | โ Built-in across workspaces and pipelines |
| **RBAC & SSO** | โ ๏ธ Manual plugins / external auth | โ Built-in, enterprise-ready |
| **Observability & logging** | โ ๏ธ Requires Prometheus, Grafana, etc. | โ Native UI for logs, metrics, traces |
| **Scalability** | โ ๏ธ Manual tuning of workers and queues | โ Auto-scaled executors on K8s or ECS |
| **Streaming support** | โ | โ Kafka, CDC, real-time ingestion |
| **Data previews & interactive runs** | โ | โ Preview data at every step |
| **Team collaboration** | โ ๏ธ Airflow not multi-tenant | โ Workspaces, permissions, activity logs |
***
## ๐ ๏ธ Step-by-Step Migration Instructions
1. **Inventory your Airflow DAGs**
* List your active DAGs and their task types:
* `PythonOperator`, `BashOperator`, `SQLOperator`, etc.
* Identify dependencies and scheduling logic
2. **Create a cluster in Mage Pro**
* Visit [Mage Pro](https://cloud.mage.ai) and sign in
* (Optional) Connect your Git repo for version control
3. **Recreate your DAGs as Mage pipelines**
* Each Airflow DAG โ 1 Mage pipeline
* Each Airflow task โ 1 Mage block:
* `PythonOperator` โ Python block
* `SQLOperator` โ SQL block
* `BranchOperator` โ Dynamic block
* `Sensor` โ Trigger block
* Define dependencies visually using the UI
4. **Configure scheduling and triggers**
* Use cron, interval, or event-based triggers
* Configure schedules via UI or YAML
5. **Run and validate pipelines**
* Use Mage Proโs UI to test individual blocks
* Preview dataframes, logs, and outputs
* Compare results to your Airflow jobs
6. **Monitor, scale, and automate**
* Monitor pipeline runs and resource usage
* Set alerts and retry rules
* Mage Pro autoscaling handles executor resources automatically
***
## ๐ค Convert Airflow DAGs with AI Sidekick
Skip the manual rewrites โ Mage Proโs **AI Sidekick** can automatically convert your Airflow DAG code into a Mage pipeline with just one prompt.
### ๐ง How to Use It
1. Click the **โAsk AIโ** button in the top-right corner of the Mage Pro UI.
2. Paste your Airflow DAG code โ including `@dag`, `@task`, or `PythonOperator`-based workflows.
3. Ask: **โConvert this Airflow DAG to a Mage pipeline.โ**
4. Sidekick will:
* Parse the DAG structure and task relationships
* Generate the corresponding Mage blocks (Python, SQL, dynamic, trigger)
* Define block dependencies and scheduling logic
5. Review and insert the generated pipeline directly into your project.
### ๐ก Why Use Sidekick?
* **Faster migration**: turn entire DAGs into Mage pipelines in seconds
* **Less error-prone**: Sidekick understands scheduling, dependencies, and operator types
* **Context-aware**: uses your project setup and prior block structure
* **Fully editable**: review, tweak, and insert blocks before saving
๐ Learn more in the [AI Sidekick docs](/ai/sidekick).
***
## ๐ง Tips for Migrating Complex DAGs
* Split large DAGs into smaller, modular pipelines
* Use **global variables** or **shared outputs** to pass data between blocks
* Replace Airflow SubDAGs with **block groups**
* Use **dynamic blocks** to support branching and conditional logic
***
## โ After Migration: What You Get with Mage Pro
* AI-powered block generation and debugging (via AI Sidekick)
* Visual pipeline builder and real-time logs
* Git-backed version control and CI/CD
* Built-in access controls, audit logs, and workspace isolation
* Support for SQL, Python, streaming, dbt, Delta Lake, Iceberg, and more
> Your pipelines, your logicโaugmented by AI, automated for scale.
๐ [Migrate to Mage Pro Today](https://cloud.mage.ai)
# Migrate from Dagster to Mage Pro
Source: https://docs.mage.ai/migrations/dagster-to-mage-pro
Many teams start with Dagster for data orchestrationโbut soon encounter challenges with data engineering workflows, visual debugging, and developer productivity. Mage Pro offers a modern alternative: visual pipelines, built-in data integration, AI-assisted development, and UI-based deployment managementโall within a unified platform for your data workflows.
## Why migrate from Dagster to Mage Pro?
Dagster is a powerful orchestration tool focused on data assets, but it wasn't designed for:
* Visual pipeline development and debugging
* Native data integration (ETL/ELT workflows)
* Mixing SQL and Python naturally in the same pipeline
* Streaming or event-driven data processing
* Built-in data previews and interactive development
* AI-powered pipeline development
Mage Pro is built to solve these challenges out-of-the-box, combining orchestration with data engineering capabilities in a modern, developer-friendly platform.
***
## Mage Pro vs Dagster: Benefits Overview
Mage Pro goes far beyond orchestration. It's a unified platform for **data integration**, **SQL modeling (native dbt blocks and Mage SQL blocks)**, **AI-powered transformation**, and **streaming pipelines** โ all within a collaborative, Git-native workspace with UI-based deployment management.
| Capability | Dagster | Mage Pro |
| ------------------------------------ | -------------------------------------------- | ------------------------------------------------------------------------- |
| **Visual pipeline UI** | โ ๏ธ Code-only (Python decorators) | โ Hybrid UI: drag-and-drop DAG builder + notebook-style coding experience |
| **Built-in lineage** | โ Asset lineage | โ Native, auto-generated data lineage |
| **AI assistance** | โ | โ Generate, optimize, fix, and explain code with AI Sidekick |
| **Multi-language support** | โ ๏ธ Python only | โ SQL, Python, R, streaming, APIs |
| **Data integration pipelines** | โ ๏ธ Custom Python or 3rd-party tools | โ 200+ native connectors for databases, files, APIs |
| **SQL block support** | โ ๏ธ Requires custom ops | โ Native dbt blocks, or Mage SQL block, preview, and test support |
| **Incremental modeling** | โ ๏ธ Requires custom logic | โ Native in SQL block config and Data integration pipeline |
| **Data previews & interactive runs** | โ ๏ธ Limited | โ Preview data at every step, run blocks independently |
| **Environment isolation** | โ ๏ธ Requires separate deployments | โ Per-workspace configs, secrets, variables |
| **Git integration** | โ ๏ธ Manual sync | โ Git-backed version control and CI/CD, UI based Deployment App |
| **Scheduling & triggers** | โ Cron + sensors | โ Cron, events, file triggers, webhooks, API |
| **Secrets management** | โ ๏ธ Requires Dagster Cloud or external config | โ Support both built-in secret manager, or external secret manager |
| **RBAC & SSO** | โ ๏ธ Dagster Cloud only | โ Built-in, enterprise-ready |
| **Observability & logging** | โ Dagster UI | โ Native UI for logs, metrics, traces, lineage |
| **Scalability** | โ ๏ธ Manual worker configuration | โ Auto-scaled executors on K8s or Docker |
| **Streaming support** | โ | โ Kafka, CDC, real-time ingestion |
| **Team collaboration** | โ ๏ธ Limited multi-tenancy | โ Workspaces, permissions, activity logs |
***
## Step-by-Step Migration Instructions
1. **Inventory your Dagster assets and jobs**
* List your active assets, ops, and jobs:
* `@asset` and `@op` decorators
* Dagster resources (storage, infrastructure)
* Dependencies and scheduling logic
* Partitions and materializations
* Identify data sources, transformations, and destinations
2. **Create a workspace in Mage Pro**
* Visit [Mage Pro](https://cloud.mage.ai) and sign in
* (Optional) Connect your Git repo for version control
* Set up your deployment configuration using the **Deployment App** in the UI
3. **Recreate your Dagster assets as Mage pipelines**
* Each Dagster **job** โ 1 Mage pipeline
* Each Dagster **asset/op** โ 1 Mage block:
* `@asset` or `@op` with data loading โ **Data loader block**
* `@asset` or `@op` with transformations โ **Transformer block** (Python or SQL)
* `@asset` or `@op` with data writing โ **Data exporter block**
* Dagster resources โ **Data loader/exporter blocks** or pipeline configuration
* Define dependencies visually using the UI (no decorators needed)
4. **Configure scheduling and triggers**
* Dagster schedules โ Mage **triggers** (cron, interval, or event-based)
* Dagster sensors โ Mage **event triggers** or **file triggers**
* Configure schedules via UI or YAML
5. **Run and validate pipelines**
* Use Mage Pro's UI to test individual blocks
* Preview dataframes, logs, and outputs at each step
* Compare results to your Dagster jobs
6. **Monitor, scale, and automate**
* Monitor pipeline runs and resource usage
* Set alerts and retry rules
* Mage Pro autoscaling handles executor resources automatically on Kubernetes or Docker
* Use the **Deployment App** to manage deployments and rollbacks
***
## Mapping Dagster Concepts to Mage Pro
### Assets & Jobs โ Pipelines
**Dagster:**
```python theme={"system"}
from dagster import asset, job, op
@asset
def raw_data():
return pd.read_csv("data.csv")
@asset
def transformed_data(raw_data):
return raw_data.dropna()
@job
def my_etl_job():
transformed_data(raw_data())
```
**Mage Pro:**
* Create a new **pipeline** in the UI
* Each asset/op becomes a **block** (data loader, transformer, or data exporter)
* Dependencies are defined by connecting blocks in the UI
* No need for explicit job definitionsโpipeline structure defines execution
### Assets & Ops โ Blocks
**Dagster:**
```python theme={"system"}
@asset
def extract_data():
return pd.read_csv("data.csv")
@op
def transform_data(context, df):
return df.dropna()
@asset
def load_data(transformed_data):
transformed_data.to_sql("table", engine)
```
**Mage Pro:**
* `extract_data` โ **Data loader block** (Python or connector)
* `transform_data` โ **Transformer block** (Python, SQL, or dbt block)
* `load_data` โ **Data exporter block** (Python or connector)
### Dagster Resources โ Mage Blocks
| Dagster Resource Type | Mage Pro Equivalent |
| --------------------------------------- | ------------------------------------------------------------- |
| `S3Resource`, `GCSResource` | **Data loader/exporter blocks** with native S3/GCS connectors |
| `PostgresResource`, `SnowflakeResource` | **Data loader/exporter blocks** with database connectors |
| `DockerResource`, `K8sResource` | **Executor configuration** in pipeline settings |
| Config resources | **Workspace variables** or **pipeline variables** |
### Partitions & Materializations
**Dagster:**
```python theme={"system"}
from dagster import DailyPartitionsDefinition, asset
daily_partitions = DailyPartitionsDefinition(start_date="2024-01-01")
@asset(partitions_def=daily_partitions)
def partitioned_asset(context):
partition_date = context.partition_key
# Process data for specific partition
...
```
**Mage Pro:**
* Use **pipeline variables** or **dynamic blocks** for partition logic
* Configure **incremental processing** in SQL blocks or Python blocks
* Use **backfill triggers** for historical data processing
* Schedule pipelines with cron expressions for partition-based runs
### Scheduling & Sensors
**Dagster:**
```python theme={"system"}
from dagster import schedule, sensor, RunRequest
@schedule(cron_schedule="0 0 * * *", job=my_etl_job)
def daily_schedule(context):
return RunRequest()
@sensor(job=my_etl_job)
def file_sensor(context):
if file_exists("new_data.csv"):
return RunRequest()
```
**Mage Pro:**
* Create a **schedule trigger** in the pipeline UI
* Configure cron expression, interval, or event-based triggers
* Use **file triggers** for file-based sensors
* Use **event triggers** for webhook-based sensors
* Use the **Deployment App** to manage deployments across environments (dev, staging, prod)
### Secrets & Configuration
**Dagster:**
```python theme={"system"}
from dagster import ConfigurableResource
class MyResource(ConfigurableResource):
api_key: str
@asset
def use_resource(my_resource: MyResource):
key = my_resource.api_key
```
**Mage Pro:**
* Use **workspace variables** or **pipeline variables**
* Access via `variables` dictionary in blocks
* **Built-in secret manager** with encryption, or integrate with **external secret managers** (AWS Secrets Manager, HashiCorp Vault, etc.)
* Secrets are isolated per workspace and pipeline
***
## Convert Dagster Code with AI Sidekick
Skip the manual rewrites โ Mage Pro's **AI Sidekick** can automatically convert your Dagster asset and job code into a Mage pipeline with just one prompt.
### How to Use It
1. Click the **"Ask AI"** button in the top-right corner of the Mage Pro UI.
2. Paste your Dagster code โ including `@asset`, `@op`, `@job`, or asset-based workflows.
3. Ask: **"Convert this Dagster job to a Mage pipeline."**
4. Sidekick will:
* Parse the asset/op structure and dependencies
* Generate the corresponding Mage blocks (data loader, transformer, data exporter)
* Define block dependencies and scheduling logic
* Convert Dagster resources to Mage connectors where applicable
* **Optimize** the generated code for performance and best practices
5. Review and insert the generated pipeline directly into your project.
### Why Use Sidekick?
* **Faster migration**: turn entire jobs into Mage pipelines in seconds
* **Less error-prone**: Sidekick understands asset dependencies, partitions, and op types
* **Code optimization**: automatically optimizes generated code for performance
* **Context-aware**: uses your project setup and prior block structure
* **Fully editable**: review, tweak, and insert blocks before saving
๐ Learn more in the [AI Sidekick docs](/ai/sidekick).
***
## Tips for Migrating Complex Jobs
### Handling Dagster Asset Dependencies
**Dagster:**
```python theme={"system"}
@asset
def parent_asset():
...
@asset
def child_asset(parent_asset):
...
```
**Mage Pro:**
* Dependencies are automatically inferred from block connections in the UI
* Use **upstream blocks** to define data flow
* Blocks with no dependencies run in parallel automatically
### Dagster Partitions
**Dagster:**
```python theme={"system"}
from dagster import DailyPartitionsDefinition
daily_partitions = DailyPartitionsDefinition(start_date="2024-01-01")
@asset(partitions_def=daily_partitions)
def partitioned_asset(context):
partition_date = context.partition_key
...
```
**Mage Pro:**
* Use **pipeline variables** to pass partition information
* Configure **incremental processing** in SQL blocks
* Use **backfill triggers** to process historical partitions
* Schedule pipelines with appropriate cron expressions
### Dagster Resources & Configuration
**Dagster:**
```python theme={"system"}
from dagster import ConfigurableResource, asset
class DatabaseResource(ConfigurableResource):
connection_string: str
@asset
def use_database(database: DatabaseResource):
conn = database.connection_string
...
```
**Mage Pro:**
* Use **workspace variables** or **pipeline variables** for configuration
* Access via `variables` dictionary in blocks
* Use **data loader/exporter blocks** with native connectors for databases
* Secrets are managed through built-in or external secret managers
### Dagster Retries & Error Handling
**Dagster:**
```python theme={"system"}
@asset(retry_policy=RetryPolicy(max_retries=3, delay=60))
def risky_asset():
...
```
**Mage Pro:**
* Configure **retry settings** in block or pipeline configuration
* Set retry count, delay, and backoff strategy
* Built-in error handling and alerting
### Dagster Caching & Materializations
**Dagster:**
```python theme={"system"}
@asset(
compute_kind="pandas",
metadata={"cache": True}
)
def cached_asset():
...
```
**Mage Pro:**
* Use **block output caching** in pipeline settings
* Configure cache expiration and invalidation rules
* Preview cached outputs in the UI
***
## After Migration: What You Get with Mage Pro
* **Unified data platform**: orchestration + data integration + transformation + streaming in one tool
* **Visual pipeline builder** with real-time data previews
* **AI-powered block generation, optimization, and debugging** (via AI Sidekick)
* **Git-backed version control** and CI/CD with **UI-based Deployment App**
* **Built-in access controls**, audit logs, and workspace isolation
* **Support for SQL, Python, R, streaming, dbt blocks, Delta Lake, Iceberg**, and more
* **200+ native connectors** for databases, APIs, and cloud storage
* **Streaming pipelines** for real-time data processing
* **Flexible secrets management**: built-in secret manager or integrate with external systems (AWS Secrets Manager, Vault, etc.)
* **Auto-scaled executors** on Kubernetes or Docker
* **Transparent pricing**: usage-based pricing (SaaS) or cluster/workspace license-based pricing (self-hosted)
> Dagster excels at asset-based orchestration. But when you need **data engineering, visual development, AI assistance, and native data integration** โ **Mage Pro is built for modern data teams**.
๐ [Migrate to Mage Pro Today](https://cloud.mage.ai)
# Migrate from dbt to Mage Pro
Source: https://docs.mage.ai/migrations/dbt-to-mage-pro
## **Overview**
Migrating your dbt Cloud project to Mage Pro enables your team to unify analytics and data engineering workflows, combining dbtโs data transformation power with Mage Proโs orchestration, monitoring, and modular pipeline capabilities. This guide provides a step-by-step process for a seamless migration, allowing you to run, schedule, and monitor dbt models directly inside Mage Proโalongside SQL, Python, and R blocks.
**โจ Mage Pro vs dbt Cloud: Benefits Overview**
Mage Pro goes far beyond SQL modeling. It's a unified platform for **data integration**, **transformation (including native dbt support)**, **AI-powered development**, and **streaming pipelines** โ all within a collaborative, Git-native workspace.
| **Capability** | **dbt Cloud** | **Mage Pro** |
| -------------------------- | ----------------------------------------------------- | ------------------------------------------------------ |
| **Visual pipeline UI** | โ ๏ธ Basic lineage view only | โ Full drag-and-drop pipeline editor |
| **dbt model execution** | โ Native | โ Native with enhanced orchestration |
| **Multi-language support** | โ SQL only | โ SQL, Python, R, and dbt blocks |
| **Data ingestion** | โ Transformation only | โ 200+ built-in connectors |
| **End-to-end pipelines** | โ Requires external orchestration | โ Complete ETL/ELT workflows |
| **Real-time streaming** | โ Batch only | โ Kafka, CDC, event-driven processing |
| **AI assistance** | โ ๏ธ Limited dbt Assist (private beta) | โ AI Sidekick for all pipeline development |
| **Pricing model** | โ ๏ธ Per-seat + model builds (unpredictable) | โ Transparent per-seat pricing |
| **Cost predictability** | โ Consumption charges can spike unexpectedly | โ Fixed monthly costs with no usage surprises |
| **IDE experience** | โ Browser-based SQL IDE | โ Advanced notebook-style blocks + VS Code integration |
| **Git integration** | โ ๏ธ Basic Git functionality, complex workflows limited | โ Full Git workflows with CI/CD |
| **Orchestration** | โ ๏ธ Requires Airflow/external tools | โ Native scheduling and event triggers |
| **Environment isolation** | โ ๏ธ Limited workspace management | โ Per-workspace configs and secrets |
| **Data previews** | โ ๏ธ Limited preview capabilities | โ Interactive data preview at every step |
| **Scalability** | โ ๏ธ Limited concurrency (1-5 jobs) | โ Auto-scaled execution on K8s/ECS |
| **Security concerns** | โ ๏ธ Data passes through dbt infrastructure | โ Direct warehouse connections |
| **Team collaboration** | โ ๏ธ Limited to SQL transformations | โ Full-stack data team collaboration |
## **Migration Process Outline:**
Developers should create a Git repo and push the base Mage Pro project into their chosen vendor. This could be Github, GitLab, GitBucket, etc. For the sake of this migration tutorial we will be using Github.
## Step 1: Connect Mage Pro to Gituhub
Before migrating your dbt project, you must connect Mage Pro to your GitHub repository to enable version control and deployment capabilities.
### Quick Setup
1. **Navigate to Deployments** in the left-hand navigation menu
2. **Click "Setup connection"** to begin GitHub integration
3. **Authenticate with GitHub** using OAuth tokens
4. **Select your organization and repository** from the dropdown menus
5. **Choose your target branch** (typically `main` or `master`)
6. **Complete the connection** by clicking "Next step to deployment stage"
*For detailed authentication steps, see the* [Version Control Guide](https://docs.mage.ai/guides/data-sync/version-control-guide)*.*
Once authenticated, the Git Terminal automatically uses your credentials for all Git operations without requiring additional setup.
## Step 2: Push base Mage Pro project to Github
After connecting to GitHub, you'll need to push your initial Mage Pro project structure to your repository to establish the foundation for your dbt migration.
### **Quick Setup**
1. **Access Git Terminal**ย by hovering over the left navigation menu and clicking "Terminal"
2. **Stage your project files**ย using standard Git commands:
```bash theme={"system"}
git add .
git commit -m "Initial Mage Pro project setup"
git push origin main
```
3. **Verify the push**ย by checking your GitHub repository for the Mage Pro project structure
NOTE - you may need to force push the initial commit to Github
### **What Gets Pushed**
Your initial commit will include:
* Mage Pro project structure (pipelines, data\_loaders, transformers, etc.)
* Configuration files (`io_config.yaml`,ย `metadata.yaml`)
* Environment setup files
### **Authentication**
The Git Terminal uses the authentication you configured in Step 1, so no additional setup is required. All standard Git operations work seamlessly within the Mage Pro interface.
*For detailed Git Terminal usage and advanced features, see theย [Git Terminal Guide](https://docs.mage.ai/guides/data-sync/git-terminal).*
Once your base project is pushed to GitHub, you can proceed with importing and integrating your existing dbt models into the Mage Pro environment.
## Step 3: Clone your dbt project
After pushing your base Mage Pro project, you'll need to clone your existing dbt project into the Mage environment to begin the migration process.
### Quick Setup
1. **Access Git Terminal** by hovering over the left navigation menu and clicking "Terminal"
2. **Navigate to the dbt directory** that Mage provides:
`cd dbt`
3. **Clone your dbt project** into the dbt folder:
`git clone https://github.com/your-org/your-dbt-project.git
cd your-dbt-project`
4. **Verify the clone** by checking that your dbt models, tests, and configurations are present
### Integration Notes
* The dbt folder is specifically designed for dbt project integration
* Your cloned project will work seamlessly with Mage Pro's native dbt support
* Authentication uses the same credentials from your GitHub connection
*For detailed Git Terminal usage and advanced features, see the [Git Terminal Guide](https://docs.mage.ai/guides/data-sync/git-terminal).*
Once your dbt project is cloned into Mage Pro, you can begin creating pipelines that incorporate your existing dbt models alongside new data processing workflows.
## Step 4: Add Mage profiles.yaml file
You will need to custom configure your profiles.yaml file in
After cloning your dbt project, you need to configure the `profiles.yml` file to connect your dbt models to your data warehouse within the Mage Pro environment.
### Quick Setup
1. **Navigate to your dbt project folder** in the Git Terminal:
`cd dbt/your-dbt-project`
2. **Create or edit the profiles.yml file**:
`touch profiles.yml`
3. **Add your warehouse configuration** using the appropriate connection details
NOTE - for this step you should use the Mage terminal not the Git terminal
### BigQuery Configuration Example
```yaml theme={"system"}
mage_dbt_demos:
outputs:
dev:
dataset: BigQuery dataset
job_execution_timeout_seconds: 300
job_retries: 1
keyfile: /home/src//
location: US
method: service-account
priority: interactive
project: CGP_PROJECT_ID
threads: 4
type: bigquery
target: dev
```
### Key Configuration Notes
* **Profile name** should match your `dbt_project.yml` profile setting
* **Keyfile path** must point to your service account credentials within Mage Pro
* **Dataset and project** should match your BigQuery setup
* **Target environment** determines which output configuration dbt uses
### Environment-Specific Configurations
You can add multiple environments (dev, staging, prod) within the same profile to support different deployment targets as your project grows.
Once configured, your dbt models will be able to connect to your data warehouse and execute transformations within the Mage Pro environment.
## **Step 5: Testing the dbt connection**
After migrating your dbt project to Mage Pro and configuring your `profiles.yml`, youโll want to verify that Mage Pro can successfully connect to your data warehouse and run dbt commands. This ensures that your credentials, connection details, and environment setup are correct before integrating further workflows.
### **How to Test Your dbt Connection in Mage Pro**
1. **Open Mage Pro and locate your dbt block** within the relevant pipeline.
2. **Click into the dbt block** to access its configuration and terminal interface.
3. **Run a basic dbt command** to test connectivity. For example:
`dbt debug`
This command will check the connection parameters defined in your `profiles.yml` and confirm dbt is able to reach your data warehouse.
4. **Verify the output** in the terminal panel:
* If the last line isย `All checks passed!`, your connection is successful.
* If you see errors, review the credentials inย `profiles.yml`, check access permissions, and validate that your warehouse is network accessible from Mage Pro.
## **Step 6: Loading dbt Dependencies After Migrating From GitHub**
When bringing a dbt project from GitHub, you may have dependencies declared in your `packages.yml` file (e.g. dbt-utils). Mage Pro supports full dependency resolution for dbt projects cloned or migrated from GitHub.
### **How to Load dbt Dependencies in Mage Pro**
1. **Open Mage Pro and navigate to your cloned dbt project directory** using the built-in terminal or via the dbt block settings.
2. **Locate your `packages.yml`** file (usually in the root of your dbt project).
3. **Install dependencies** by running:
`dbt deps`
This command will download and install all packages specified in your `packages.yml`.
4. **Verify successful installation**:
* Check the console output for confirmation that packages were installed (look for messages like โDependencies installed!โ).
* Confirm that new folders such asย `dbt_packages`ย are present in your dbt project directory.
5. **After installing dependencies, you can now compile and run your dbt models** as usual in Mage Pro.
## Step 7: Connect dbt models to Mage blocks
When you connect a dbt block in Mage Pro to an upstream block (Python, SQL, or R), Mage automatically generates and manages a special `mage_sources.yml` file in your dbt project's model directory. This file describes upstream Mage blocks as dbt sources, making their outputs available to your dbt models.
Below is a sample `mage_sources.yml` that gets created when you connect a data loader or transformation block (e.g., Python, SQL, or R) to a downstream dbt model block:
```yaml theme={"system"}
version: 2
sources:
- description: Dataframes Mage upstream blocks
loader: mage
name: mage_dbt_tutorial
schema: public
tables:
- description: Dataframe for block `get_data` of the `dbt_tutorial` mage pipeline.
identifier: mage_dbt_tutorial_get_data
meta:
block_uuid: get_data
pipeline_uuid: dbt_tutorial
name: dbt_tutorial_get_data
```
### **Explanation**
* **version**: dbt sources file version (alwaysย `2`ย for dbt v1+).
* **sources**: List of source definitions.
* **description**: Human-readable note (optional, here noting that sources come from Mage blocks).
* **loader**: Indicates Mage created the source.
* **name**: The dbt source name, usuallyย `mage_`ย or similar.
* **schema**: Target schema for the seed tables (commonlyย `public`).
* **tables**: List of upstream Mage blocks exposed as tables to dbt.
* **description**: Helps you identify the block in Mage.
* **identifier**: Table identifier in your warehouse, typicallyย `mage__`.
* **meta**: Traceability information, recording Mage pipeline and block UUIDs.
* **name**: dbt-level source table name (used as the second argument forย `source()`ย in dbt SQL).
## Step 8: Make changes to your sources in dbt blocks
Mage Pro manages the mapping between upstream pipeline outputs and your dbt models. Any time you update upstream logicโsuch as renaming a block, changing a pipeline, or adding new data sourcesโMage Pro ensures your `mage_sources.yml` file reflects the latest available sources.
To use an upstream Mage block as a source in your dbt model:
1. In Mage Pro, open your dbt model block from the pipeline.
2. Check that your most recent changes to Python, SQL, or R blocks upstream are reflected in the generated `mage_sources.yml` file. For example, if you add a new transformation block called `get_data`, the following entry should appear:
```yaml theme={"system"}
sources:
- name: mage_dbt_tutorial
tables:
- name: dbt_tutorial_get_data
identifier: mage_dbt_tutorial_get_data
```
3. Reference the Mage source in your dbt SQL model using dbt's `source()` macro:
`select *
from {{ source('mage_dbt_tutorial', 'dbt_tutorial_get_data') }}`
4. Run and test your dbt model as usual. Model compilation and execution automatically reference the latest outputs from your upstream Mage pipeline logic.
## Conclusion
After completing these steps, you've successfully migrated your dbt project to Mage Pro, creating a unified platform that combines dbt transformations with Mage Pro's orchestration and multi-language support. Your dbt models now run seamlessly alongside Python, SQL, and R blocks, while maintaining GitHub version control. This migration enables more comprehensive data workflows and enhanced collaboration while preserving your existing dbt investments.
# Migrate from Estuary to Mage Pro
Source: https://docs.mage.ai/migrations/estuary-to-mage-pro
Estuary is great for real-time data captureโbut teams often hit limits around transformation, orchestration, and control. **Mage Pro** offers an end-to-end solution for batch and streaming pipelines with **Python & SQL transformations**, full **orchestration**, and **AI-powered pipeline generation**.
## Why migrate from Estuary to Mage Pro?
Estuary specializes in real-time syncs with pre-built connectors. But it lacks flexibility for:
* Transforming data beyond the destination database
* Managing complex orchestration and dependencies
* Debugging and testing pipelines visually
* Git-native workflows and team collaboration
* Full control over schema evolution and failure handling
Mage Pro fills these gaps with **streaming + batch pipelines**, modular blocks for ETL, and full visibility into each step of your data flow.
***
## โจ Mage Pro vs Estuary: Benefits Overview
| Capability | Estuary | Mage Pro |
| -------------------------- | ----------------------- | ----------------------------------------- |
| **Streaming support** | โ Native | โ Native (Kafka, CDC, webhooks) |
| **Batch processing** | โ ๏ธ Limited | โ Full batch support |
| **Transformations** | โ ๏ธ Destination-only | โ Python, SQL, AI-generated |
| **Orchestration** | โ None | โ Full DAG-level orchestration |
| **Visual pipeline UI** | โ ๏ธ Schema-level only | โ Drag-and-drop builder |
| **AI assistance** | โ | โ Sidekick for code & pipeline generation |
| **Custom sources/targets** | โ ๏ธ Plugin SDK only | โ Python-based SDK for APIs, files, DBs |
| **Incremental loads** | โ Supported | โ Native with customizable logic |
| **Git integration** | โ | โ Git-backed projects & CI/CD |
| **Environments** | โ ๏ธ Single workspace | โ Dev, staging, and production |
| **Observability** | โ ๏ธ Basic logs | โ Detailed logs, retries, lineage |
| **Hosting** | โ Cloud only | โ Cloud or self-hosted |
| **Cost** | โ ๏ธ Event-volume pricing | โ Transparent usage-based plans |
***
## ๐ ๏ธ Step-by-Step Migration Instructions
1. **Audit your Estuary flows**
* Export list of **flows** and their source โ destination pairs
* Note any field mappings or transformations
* Identify real-time vs batch sync requirements
2. **Set up a Mage Pro workspace**
* Sign up at [Mage Pro](https://cloud.mage.ai)
* Connect a Git repo if needed for versioning and CI/CD
3. **Recreate flows as Mage pipelines**
* Estuary source โ **Source block** in Mage Streaming Pipeline
* Estuary destination โ **Destination block** in Mage Streaming Pipeline
* Custom transforms โ **SQL** or **Python blocks**
* Choose between **streaming** (CDC, Kafka, webhook) or **batch**
4. **Configure scheduling or triggers**
* Use streaming triggers (Kafka, webhooks) or batch schedulers (cron)
* Add event-based triggers for more control
5. **Validate pipeline output**
* Run pipelines in staging
* Preview outputs, compare with Estuaryโs destination tables
* Verify schema integrity, incremental updates, and deduplication
6. **Go live and monitor**
* Enable alerts, retry logic, and audit logging
* Use observability tools to track pipeline health and performance
* Scale pipeline workers automatically via Kubernetes or ECS
***
## ๐ค Use AI Sidekick to Migrate Faster
Estuary flows are often schema-mapped and repetitiveโperfect for AI Sidekick to help convert into Mage pipelines.
### ๐ง How to Use It
1. Open a Mage Pro pipeline and click **"Ask AI"**
2. Paste in an Estuary flow JSON or describe your flow
3. Ask: **"Convert this Estuary flow into a Mage pipeline."**
4. The AI will:
* Create Extract/Load blocks
* Generate SQL or Python for transforms
* Add scheduling and triggers
5. Insert and run the pipeline directly in your workspace
***
## ๐ง Tips for Migrating Streaming Workflows
* Use **Kafka Extract blocks** or **webhook listeners** for streaming sources
* Combine **streaming + batch** in hybrid pipelines
* Use **block-level caching** to test transforms on historical data
* Handle schema evolution with **data validation and pipeline tests**
***
## โ After Migration: What You Gain
* One platform for **streaming + batch pipelines**
* Visual editor with modular, testable blocks
* AI-powered pipeline creation and debugging
* Git-native development & multi-environment deployments
* Better cost control and full observability
* Built-in orchestration and transformation logic
> Estuary handles streams. Mage Pro handles **everything**โwith visibility, scalability, and flexibility built-in.
๐ [Start Migrating to Mage Pro](https://cloud.mage.ai)
# Migrate from Fivetran to Mage Pro
Source: https://docs.mage.ai/migrations/fivetran-to-mage-pro
Many teams start with Fivetran for ELT pipelinesโbut face limitations in flexibility, cost, and transparency. Mage Pro offers a powerful alternative: a unified platform for **data integration, transformation, and orchestration** with full code control, **AI-assisted development**, and **cloud-native scalability**.
## Why migrate from Fivetran to Mage Pro?
Fivetran is convenient for basic replicationโbut it wasnโt built for:
* Custom transformations or logic
* Managing orchestration and dependencies
* Running in your own cloud for cost control and security
* Debugging and optimizing individual steps
* Streaming data or event-driven triggers
Mage Pro replaces Fivetranโs โblack-boxโ connector model with **modular, open pipelines** that give you full control over data extraction, transformation, and loadingโplus visibility into every step.
***
## โจ Mage Pro vs Fivetran: Benefits Overview
Mage Pro offers more flexibility, transparency, and scalability than Fivetranโat a fraction of the cost.
| Capability | Fivetran | Mage Pro |
| ------------------------------ | ---------------------------------------- | -------------------------------------------- |
| **Data integration** | โ Managed connectors | โ Native connectors for APIs, DBs, files |
| **Transformation support** | โ ๏ธ External only (via dbt or dest SQL) | โ Built-in SQL & Python support |
| **Orchestration** | โ None | โ Full DAG-level orchestration |
| **Visual pipeline UI** | โ | โ Drag-and-drop builder |
| **AI assistance** | โ | โ AI Sidekick for code & pipeline generation |
| **Incremental loads** | โ Supported | โ Native with customizable logic |
| **Custom logic** | โ ๏ธ Limited (no in-flow transforms) | โ Full Python & SQL control |
| **Streaming & event triggers** | โ | โ Real-time CDC, Kafka, webhooks |
| **Data preview & validation** | โ ๏ธ Destination-only | โ Block-level previews & schema validation |
| **Git integration & CI/CD** | โ | โ Git-native pipelines & deployments |
| **Secrets management** | โ ๏ธ Env-only | โ Built-in secrets & workspace isolation |
| **Multi-environment support** | โ ๏ธ Manual setup | โ Dev, staging, and production workspaces |
| **Observability & debugging** | โ ๏ธ Basic logs, no pipeline introspection | โ Logs, retries, lineage, and alerting |
| **Scalability** | โ ๏ธ Controlled by Fivetran, usage-priced | โ Auto-scaled executors on your infra |
| **Deployment options** | โ Cloud, Hybrid, Self-Hosted (HVR) | โ Cloud, Hybrid, or Fully On-Prem |
| **Data stays in your VPC** | โ With Hybrid deployment | โ Always (runs in your infra) |
| **Control plane ownership** | โ ๏ธ Fivetran-managed (except self-hosted) | โ Fully under your control |
***
## ๐ ๏ธ Step-by-Step Migration Instructions
1. **Review your Fivetran connectors**
* List all source โ destination syncs
* Identify transformation steps (dbt, SQL, external pipelines)
* Export any sync configuration (e.g., schedule, sync mode)
2. **Create your Mage Pro workspace**
* Sign up at [Mage Pro](https://cloud.mage.ai)
* (Optional) Connect GitHub or GitLab for version control
3. \*\*Rebuild Fivetran sync using Mage pipeline \*\*
* Each Fivetran **source** โ Mage Data Integration **Source block**
* Each Fivetran **destination** โ Mage Data Integration **Destination block**
* dbt or SQL logic โ Mage **SQL block**
* Add **Python blocks** for advanced transforms or conditionals
4. **Design pipelines**
* Combine extract โ transform โ load in one visual pipeline
* Use UI builder or YAML configs for setup
5. **Configure schedules and triggers**
* Replace Fivetran sync schedules with Mage **cron, event, or file triggers**
* Add webhooks or file watchers if needed
6. **Validate pipelines**
* Preview outputs at each block
* Compare target tables and schema to Fivetran outputs
* Test incremental sync and backfill behavior
7. **Monitor and scale**
* View logs, lineage, and metrics in Mage
* Configure **alerts, retries, and concurrency**
* Enable **auto-scaling** for high-volume loads
***
## ๐ค Use AI Sidekick to Convert Fivetran Configs
Mage Proโs **AI Sidekick** helps migrate Fivetran configurations, dbt models, and logic into Mage pipelines.
### ๐ง How to Use It
1. Open your Mage Pro workspace
2. Click the **โAsk AIโ** button
3. Paste in your Fivetran connector settings, sync SQL, or dbt model
4. Ask: **โConvert this Fivetran sync into a Mage pipeline.โ**
5. The AI will:
* Generate Extract and Load blocks
* Create SQL/Python transforms
* Add schedule and retry logic
6. Insert the generated pipeline into your workspace
***
## ๐ง Tips for Migrating Complex Pipelines
* Use **multiple pipelines** for large source groups (e.g., one per schema)
* Parameterize **table names and filters** with pipeline variables
* Replace dbt logic with **modular SQL blocks** using `ref()` and tests
* Use **batching or streaming** modes for large datasets
***
## โ After Migration: What You Gain
* Complete **ELT pipelines** with built-in orchestration
* Visual and modular pipeline builder
* AI-powered development & debugging
* Git-native deployments across environments
* Real-time ingestion with event-based triggers
* Transparent cost, better performance, full control
> Fivetran is great for getting started. But when you're ready for control, observability, and flexibilityโ**Mage Pro is built for scale**.
๐ [Start Migrating to Mage Pro](https://cloud.mage.ai)
# Migrate from Mage OSS to Mage Pro
Source: https://docs.mage.ai/migrations/mage-oss-to-mage-pro
Many teams start with Mage OSS for building data pipelinesโbut as they scale, they need enterprise-grade features like multi-environment workspaces, CI/CD deployments, cluster management, and enhanced observability. **Mage Pro** offers a seamless upgrade path with zero code changes, plus exclusive features that make data engineering more productive and reliable.
## Why migrate from Mage OSS to Mage Pro?
Mage OSS is excellent for getting started, but teams often need:
* **Multi-environment isolation** (dev, staging, production)
* **CI/CD workflows** for safe deployments
* **Cluster health monitoring** and resource optimization
* **Pipeline dependency visualization** across projects
* **AI-powered development** assistance
* **Enhanced enterprise security** (granular RBAC, SSO with group mapping, audit logs)
* **Managed infrastructure** with autoscaling
* **Dedicated support** and SLAs
Mage Pro provides all of this while maintaining **100% code compatibility** with Mage OSSโyour existing pipelines work without modification.
***
## Common Features: OSS vs Pro
Both Mage OSS and Mage Pro share a strong foundation of core features, ensuring a consistent development experience regardless of which version you choose. Most importantly, both provide a **unified platform** that consolidates all data engineering capabilities in one place.
### Unified Platform Capabilities
Both OSS and Pro offer a single, unified platform that combines:
* **Batch pipelines**: Traditional ETL/ELT workflows for scheduled data processing
* **No-code data integration pipelines**: Pre-built connectors for seamless data ingestion and loading
* **Streaming pipelines**: Real-time data processing for event-driven architectures (Pro includes stateful streaming with checkpointing)
* **Unified ingestion, transformation, and orchestration**: All pipeline types share the same interface, configuration, and execution framework
* **Integrated observability**: Built-in monitoring, logging, and data previews across all pipeline types
This unification means you can build end-to-end data workflowsโfrom ingestion to transformation to orchestrationโall within a single platform, without needing separate tools for different pipeline types.
### Features Available in Both OSS and Pro
* **Visual pipeline builder**: Drag-and-drop interface for building data pipelines
* **Multi-language support**: Python, SQL, and R blocks for data transformation
* **Data connectors**: Access to core data sources and destinations (Pro includes 200+ connectors with exclusive options)
* **Built-in secret manager**: Secure storage and management of credentials
* **External vault integration**: Support for AWS Secrets Manager, GCP Secret Manager, Azure Key Vault
* **SSO authentication**: SAML/OAuth/Okta integration for single sign-on
* **TLS encryption**: Secure data in transit
* **Read-only mode**: Protect production notebooks and assets from accidental edits
* **Git integration**: Version control for pipeline code
* **Pipeline scheduling**: Cron-based and event-driven triggers
* **Data previews**: Preview data at each pipeline step
* **Block library**: Reusable code blocks across projects
* **Testing framework**: Built-in testing capabilities for data quality
* **dbt integration**: Run dbt models within pipelines
### When to Use Mage OSS
Mage OSS is ideal for:
* **Small teams or individual developers** getting started with data pipelines
* **Proof of concept projects** or experimental pipelines
* **Organizations with strong DevOps capabilities** who prefer self-managed infrastructure
* **Budget-conscious teams** who can manage their own infrastructure
* **Simple use cases** that don't require multi-environment isolation
* **Teams comfortable with manual Git workflows** and self-service upgrades
* **Organizations with strict data governance** requiring full control over infrastructure
**Best for**: Teams that want the core Mage functionality with full infrastructure control and are comfortable managing deployments, upgrades, and scaling manually.
### When to Use Mage Pro
Mage Pro is recommended for:
* **Growing teams** that need multi-environment workspaces (dev, staging, production)
* **Enterprise organizations** requiring granular RBAC, audit logs, and compliance certifications (SOC 2, ISO 27001)
* **Teams needing CI/CD workflows** with automated deployments and rollback capabilities
* **Organizations requiring managed infrastructure** with autoscaling and high availability
* **Teams building stateful streaming pipelines** or using database CDC (MySQL, PostgreSQL, MSSQL, Oracle)
* **Organizations needing dedicated support** with SLAs and proactive monitoring
* **Teams requiring advanced observability** with cluster health insights and pipeline dependency views
* **Organizations with compliance requirements** that need per-workspace secret scoping and audit trails
* **Teams wanting AI-powered development** assistance with AI Sidekick
* **Organizations needing multi-region deployment** options
**Best for**: Teams that need enterprise-grade features, managed infrastructure, advanced orchestration capabilities, and dedicated support to scale their data engineering operations efficiently.
### Migration Path
Since Mage Pro maintains **100% code compatibility** with Mage OSS, you can start with OSS and seamlessly migrate to Pro when you need additional features. Your pipelines, blocks, and configurations will work without any code changes.
***
## Mage Pro Deployment Options
Mage Pro supports **4 flexible deployment options** to meet your infrastructure and compliance needs:
1. **Mage Pro SaaS** - Fully managed service where Mage maintains the infrastructure and your code executes in our environment. Zero DevOps overhead with automatic scaling and updates.
2. **Mage Pro Hybrid Cloud** - Mage Pro infrastructure maintains the control plane, while the data plane is deployed in your cloud. Best of both worlds: reduced overhead with full data privacy.
3. **Mage Pro Private Cloud** - Both control and data plane are deployed in your cloud infrastructure. Ideal for teams with data residency or compliance requirements who want full control.
4. **Mage Pro On-Prem** - Mage Pro infrastructure is deployed on your on-premises servers. Perfect for organizations with strict data governance or air-gapped environments.
All deployment options provide the same enterprise features, including workspaces, CI/CD deployments, cluster management, and AI Sidekick. Choose the option that best fits your security, compliance, and infrastructure requirements.
***
## Mage Pro vs Mage OSS: Benefits Overview
Mage Pro is more performant, reliable, and cost-effective than Mage OSS, with enterprise-grade features built-in.
| Capability | Mage OSS | Mage Pro |
| -------------------------------------- | ----------------------------- | ---------------------------------------------------------------------- |
| **Code compatibility** | โ | โ 100% compatible (no code changes needed) |
| **Workspaces & Multi-tenancy** | โ Single project environment | โ Workspace-based tenant isolation with hard separation |
| **CI/CD Deployments** | โ Manual Git sync | โ Git-based CI/CD with environment promotion, approvals, and rollback |
| **Managed upgrades & rollbacks** | โ Manual (via images/Helm) | โ Coordinated releases and managed rollbacks |
| **Cluster management** | โ Self-managed infrastructure | โ Cluster health insights, monitoring, and managed infrastructure |
| **Pipeline dependency views** | โ Manual tracking | โ Visual dependency graph across pipelines |
| **AI assistance** | โ | โ AI Sidekick for code generation, debugging, and fixes |
| **Enhanced code editor** | โ ๏ธ Basic editor | โ Smart autocomplete, live error checking, documentation hints |
| **Stateful streaming** | โ Not available | โ Native stateful streaming with checkpointing and operators |
| **Database CDC** | โ Not available | โ Native Change Data Capture for MySQL, PostgreSQL, MSSQL, and Oracle |
| **Unified IO Config** | โ ๏ธ Separate configs | โ Single config framework across all pipeline types |
| **Data connectors** | โ Core connectors | โ 200+ connectors + exclusive Pro connectors (e.g., YouTube Analytics) |
| **Spark/Databricks** | โ ๏ธ Manual setup | โ Simplified integration with less configuration |
| **Iceberg support** | โ | โ Native Iceberg table format support |
| **SSO (SAML/OAuth/Okta)** | โ Supported | โ Enterprise SSO integration |
| **RBAC** | โ ๏ธ Basic auth | โ Granular roles and permissions at workspace/project levels |
| **Audit logs** | โ | โ User activity tracking and audit logs for compliance |
| **SCIM provisioning** | โ | โ Automated user/group lifecycle from your Identity Provider (IdP) |
| **Per-workspace secret scoping** | โ | โ Workspace-level secret isolation |
| **Static CP egress IPs** | โ | โ IP allowlisting for Hybrid/Cloud deployments |
| **Autoscaling pipeline orchestration** | โ Manual scaling | โ Auto-scales pipelines vertically/horizontally |
| **Per-tenant quotas/limits** | โ | โ Compute and concurrency limits per workspace/tenant |
| **Observability** | โ ๏ธ Basic logs | โ Full dashboards (CPU, memory, errors, alerts) with org-level views |
| **Infrastructure management** | โ ๏ธ Self-hosted | โ 4 deployment options: SaaS, Hybrid Cloud, Private Cloud, On-Prem |
| **SOC 2 / ISO 27001** | โ | โ Vendor certifications for Pro service/software |
| **Support** | โ ๏ธ Community | โ 24/7 expert support with SLA-backed incident response |
| **Proactive monitoring** | โ | โ Health checks and post-mortems under support plans |
| **Migration assistance** | โ | โ Implementation/migration credits and onboarding assistance |
***
## Step-by-Step Migration Instructions
1. **Set up Mage Pro**
* **For Mage Pro SaaS**:
* Sign up at [https://cloud.mage.ai](https://cloud.mage.ai) and create an account
* Create your cluster directly in the cloud UI
* Proceed to step 2 to create workspaces
* **For other deployment options** (Hybrid Cloud, Private Cloud, On-Prem):
* Purchase a license from the Mage team by contacting [pro@mage.ai](mailto:pro@mage.ai)
* The Mage team will provide detailed deployment instructions for your chosen option
* Follow the provided instructions to deploy Mage Pro in your cloud infrastructure or on-premises servers
* Once deployed, proceed to step 2
2. **Enable workspaces (optional)**
* **For Mage Pro SaaS**: Contact the Mage team at [pro@mage.ai](mailto:pro@mage.ai) to enable the workspaces feature
* **For other deployment options**: Follow the instructions provided by the Mage team to enable workspaces
* Once workspaces are enabled, you can create workspaces for different environments:
* Development workspace for active development
* Staging workspace (optional) for pre-production validation
* Production workspace for live pipelines
* Configure resource limits and Kubernetes settings per workspace
3. **Import your Mage OSS project and set up CI/CD**
* **Connect your Git repository**:
* In Mage Pro, navigate to the Deployment settings
* Connect your Git repository (GitHub, GitLab, Bitbucket, or Azure DevOps)
* Authenticate using OAuth or SSH keys
* **Import your project**:
* Select the branch containing your Mage OSS pipelines
* Mage Pro will automatically detect and import:
* All pipelines and blocks
* Configuration files (including IO configs)
* Variables and secrets (if stored in Git)
* Project structure
* Trigger files (triggers will be created automatically)
* No code changes requiredโMage Pro is 100% compatible with Mage OSS code
* **Migrate triggers**:
* **Before migration**: In Mage OSS, save your triggers in code by following the [configure triggers in code guide](/orchestration/triggers/configure-triggers-in-code#configure-triggers-in-code)
* **After pulling code from GitHub**: When you pull code from GitHub in Mage Pro, the trigger files will be automatically imported and triggers will be created in Mage Pro
* All your scheduled triggers, event triggers, and API triggers will be automatically set up
* **Deploy commits**:
* The deployments page will show a list of commits from your GitHub branch
* Click the "Deploy" button next to any commit to deploy that version
* Your pipelines will be updated with the code from the selected commit
* **Update USER\_CODE\_PATH environment variable**:
* After pulling code from GitHub, update the `USER_CODE_PATH` environment variable (default value is `/home/src/default_repo`) to point to your Mage project path
* **For SaaS or Hybrid Cloud**: Configure in the [cloud.mage.ai](https://cloud.mage.ai) portal for the cluster, or in the workspace UI
* **For Private Cloud or On-Prem**: Configure via Helm chart values for the cluster, or in the workspace UI
4. **Configure environment variables and credentials**
* **Set up environment variables**:
* **For SaaS or Hybrid Cloud**:
* Configure environment variables in the [cloud.mage.ai](https://cloud.mage.ai) portal for each cluster, OR
* Configure workspace environment variables in the workspace UI (see [workspace environment configuration guide](/guides/developer-ux/workspaces#environment-configuration))
* **For Private Cloud or On-Prem**:
* Configure environment variables for the cluster via Helm chart values, OR
* Configure workspace environment variables in the workspace UI (see [workspace environment configuration guide](/guides/developer-ux/workspaces#environment-configuration))
* Add environment-specific variables (e.g., `ENV`, `API_BASE_URL`)
* Use variable interpolation syntax for dynamic values
* **Upload necessary credentials**:
* Store API keys, database passwords, and service account credentials in Mage Pro's secrets vault
* Upload certificate files, SSH keys, or other authentication files as needed
* Configure secrets at workspace level for environment isolation
* **Enable database connectivity**:
* **For Mage Pro SaaS or Hybrid Cloud**: If connecting to databases in private networks:
* **Static CP egress IPs / IP allowlisting**: Mage Pro provides static outbound IP addresses for the control plane. Get these IP addresses from the cluster settings and whitelist them in your database firewall/security groups
* **PrivateLink / VPC Peering**: Set up PrivateLink or VPC peering between Mage Pro's network and your private database network for secure, direct connectivity without exposing databases to the internet
* Contact Mage Pro support to obtain the required IP ranges and VPC peering/PrivateLink configuration details
* **For Private/Hybrid Cloud**: Configure network connectivity:
* Ensure proper network routing between Mage Pro workspaces and your databases
* Set up security groups, firewall rules, or network policies as needed
* Configure VPC peering or VPN connections if databases are in separate networks
* **For On-Prem**: Configure network access:
* Ensure Mage Pro can reach your databases through your internal network
* Configure firewall rules to allow connections from Mage Pro to database servers
* **Test database connections**:
* Use Mage Pro's connection testing feature to verify connectivity
* Test from each workspace (dev, staging, prod) to ensure proper isolation
5. **Verify data sources and test connections**
* IO configs are automatically imported from your GitHub repository
* Update connection strings to use environment variables and secrets (if needed)
* Verify all data connectors are properly configured
* Test connections to external systems (databases, APIs, cloud storage) using the connection testing feature
6. **Set up authentication and access control**
* Configure RBAC (Role-Based Access Control) for team members
* Set up SSO (Single Sign-On) if needed for enterprise authentication
* Define workspace-level permissions
* Configure resource limits and autoscaling policies per workspace
7. **Test and validate pipelines**
* Run your pipelines in the development workspace
* Use Mage Pro's data preview feature to validate outputs
* Check logs and metrics in the observability dashboard
* Compare results with your Mage OSS environment
* Verify all database connections and external integrations work correctly
8. **Deploy to production**
* Navigate to the deployments page in Mage Pro
* The page will display a list of commits from your GitHub branch
* Click the "Deploy" button next to the commit you want to deploy to production
* Monitor pipeline runs and resource usage in real-time
* Set up alerts for failures or performance issues
* Leverage pipeline dependency views to understand cross-pipeline relationships
***
## Alternative Migration Method: Manual Export/Import
If you prefer not to use Git-based import, you can manually migrate your project:
1. **Export from Mage OSS**
* Copy your entire project directory
* Ensure all pipelines, blocks, and configs are included
2. **Upload to Mage Pro**
* Use the Git terminal in Mage Pro to push your code
* Or use the file browser to upload project files directly
3. **Verify and configure**
* Check that all pipelines are recognized
* Update any environment-specific configurations
* Set up workspace-specific variables and secrets
* Upload credentials and configure database connectivity (see step 5 in migration instructions)
***
## Tips for a Smooth Migration
### Code Compatibility
* **No code changes needed**: All Mage OSS pipelines work in Mage Pro without modification
* **Block types**: All block types (data loaders, transformers, data exporters, sensors) are fully supported
* **Dependencies**: Python dependencies defined in `requirements.txt` are automatically installed
* **IO Configs**: Existing IO configs are automatically imported from your GitHub repository and work as-is, with optional Pro enhancements available
* **Triggers**: Save triggers in code in Mage OSS (see [configure triggers in code guide](/orchestration/triggers/configure-triggers-in-code#configure-triggers-in-code)). When you pull code from GitHub in Mage Pro, trigger files are automatically imported and triggers are created automatically
### Workspace Organization
* **Start with one workspace**: Begin with a single workspace, then create additional ones as needed
* **Use naming conventions**: Name workspaces clearly (e.g., `dev`, `staging`, `prod`)
* **Isolate resources**: Configure different resource limits per workspace based on workload
* **Environment variables**: Use workspace-specific variables for environment differences
### Environment Variables and Credentials
* **Centralize secrets**: Store all credentials in Mage Pro's secrets vault, not in code
* **Workspace isolation**: Use workspace-specific secrets to ensure environment isolation
* **Variable interpolation**: Leverage Mage Pro's enhanced variable interpolation for dynamic configuration
* **Credential migration**: Export credentials from Mage OSS and import them into Mage Pro's secrets vault
### Database Connectivity
* **SaaS deployments**: Contact Mage Pro support early to get IP ranges for whitelisting and VPC peering options
* **Network planning**: Plan network connectivity before migration to avoid delays
* **Test connections**: Always test database connections from each workspace before running pipelines
* **Security groups**: Update security groups and firewall rules to allow Mage Pro access
* **Connection pooling**: Consider connection pooling settings for high-volume pipelines
### CI/CD Best Practices
* **Branch strategy**: Use clear branch naming (main, staging, development)
* **Rollback capability**: Test rollback procedures before going live
* **Approval workflows**: Use manual approvals for production deployments
### Monitoring and Observability
* **Set up alerts**: Configure alerts for pipeline failures and performance issues
* **Monitor resources**: Use cluster health insights to optimize resource allocation
* **Track dependencies**: Use pipeline dependency views to understand relationships
* **Review logs**: Leverage enhanced logging and observability dashboards
***
## After Migration: What You Get with Mage Pro
### Enhanced Developer Experience
* **AI Sidekick**: Get AI-powered code suggestions, explanations, and debugging
* **Enhanced code editor**: Smart autocomplete, live error checking, and documentation hints
* **VSCode & Cursor integration**: Edit and sync directly with your local IDE
* **Personalized UI themes**: Customize your workspace appearance
### Enterprise Features
* **Multi-environment workspaces**: Hard-separated tenant isolation with workspace-based environments
* **CI/CD deployments**: Built-in Git-based deployment with environment promotion, approvals, and rollback
* **Managed upgrades & rollbacks**: Coordinated releases and managed rollback capabilities
* **Cluster health insights**: Real-time CPU and memory monitoring with cost estimates
* **Pipeline dependency views**: Visualize how pipelines orchestrate and depend on each other
* **Enterprise SSO**: Enhanced SAML/OAuth/Okta integration with group mapping per tenant
* **Granular RBAC**: Fine-grained roles and permissions at workspace and project levels
* **SCIM provisioning**: Automated user and group lifecycle management from your Identity Provider (IdP) such as Okta, Azure AD, or Google Workspace
* **Audit logs**: Comprehensive user activity tracking and audit logs for compliance
* **Per-workspace secret scoping**: Workspace-level secret isolation for enhanced security
* **Per-tenant quotas/limits**: Compute and concurrency limits per workspace/tenant
* **Database CDC**: Native Change Data Capture support for MySQL, PostgreSQL, MSSQL, and Oracle
### Advanced Pipeline Capabilities
* **Stateful streaming pipelines**: Run real-time pipelines with built-in state management
* **Database CDC**: Native Change Data Capture for MySQL, PostgreSQL, MSSQL, and Oracle databases
* **Unified IO Config Framework**: Single config structure across all pipeline types
* **Exclusive connectors**: Access to Pro-only data sources and destinations
* **Enhanced integrations**: Optimized Spark, Databricks, and Iceberg support
* **More variable interpolation**: Support for env vars, secrets, and config vars
### Production-Grade Infrastructure
* **Autoscaling pipeline orchestration**: Automatically scale pipelines vertically and horizontally to handle workload spikes
* **High availability**: Built-in redundancy and failover capabilities
* **Multi-region deployment**: Deploy in US, Canada, Europe, Asia, or Australia
* **Static CP egress IPs**: IP allowlisting support for Hybrid and Cloud deployments
* **PrivateLink / VPC peering**: Direct private connectivity to Mage-hosted control plane (Hybrid/Cloud)
* **SOC 2 / ISO 27001**: Vendor certifications for Pro service and software
* **24/7 expert support**: SLA-backed incident response and expert guidance
* **Proactive monitoring**: Health checks and post-mortems under support plans
* **Migration assistance**: Implementation and migration credits with onboarding assistance
* **Managed infrastructure**: Zero DevOps overhead with fully managed deployments and coordinated upgrades
***
## Getting Help
* **Documentation**: Explore the [Mage Pro documentation](/production/mage/pro)
* **Support**: Contact Mage Pro support for migration assistance
* **Community**: Join the [Mage community](https://www.mage.ai/chat) for tips and best practices
> Your pipelines, your logicโnow with enterprise-grade infrastructure, AI assistance, and zero DevOps overhead.
๐ [Start Your Migration to Mage Pro Today](https://cloud.mage.ai)
# Migrate from Prefect to Mage Pro
Source: https://docs.mage.ai/migrations/prefect-to-mage-pro
Many teams start with Prefect for workflow orchestrationโbut soon encounter challenges with data engineering workflows, visual debugging, and developer productivity. Mage Pro offers a modern alternative: visual pipelines, built-in data integration, AI-assisted development, and UI-based deployment managementโall within a unified platform for your data workflows.
## Why migrate from Prefect to Mage Pro?
Prefect is a powerful orchestration tool, but it wasn't designed for:
* Visual pipeline development and debugging
* Native data integration (ETL/ELT workflows)
* Mixing SQL and Python naturally in the same pipeline
* Streaming or event-driven data processing
* Built-in data previews and interactive development
* AI-powered pipeline development
Mage Pro is built to solve these challenges out-of-the-box, combining orchestration with data engineering capabilities in a modern, developer-friendly platform.
***
## Mage Pro vs Prefect: Benefits Overview
Mage Pro goes far beyond orchestration. It's a unified platform for **data integration**, **SQL modeling (native dbt blocks and Mage SQL blocks)**, **AI-powered transformation**, and **streaming pipelines** โ all within a collaborative, Git-native workspace with UI-based deployment management.
| Capability | Prefect | Mage Pro |
| ------------------------------------ | ------------------------------------------- | ------------------------------------------------------------------------- |
| **Visual pipeline UI** | โ ๏ธ Code-only (Python decorators) | โ Hybrid UI: drag-and-drop DAG builder + notebook-style coding experience |
| **Built-in lineage** | โ ๏ธ Requires external tools | โ Native, auto-generated data lineage |
| **AI assistance** | โ | โ Generate, optimize, fix, and explain code with AI Sidekick |
| **Multi-language support** | โ ๏ธ Python only | โ SQL, Python, R, streaming, APIs |
| **Data integration pipelines** | โ ๏ธ Custom Python or 3rd-party tools | โ 200+ native connectors for databases, files, APIs |
| **SQL block support** | โ ๏ธ Requires custom tasks | โ Native dbt blocks, or Mage SQL block, preview, and test support |
| **Incremental modeling** | โ (requires custom logic) | โ Native in SQL block config and Data integration pipeline |
| **Data previews & interactive runs** | โ | โ Preview data at every step, run blocks independently |
| **Environment isolation** | โ ๏ธ Requires separate deployments | โ Per-workspace configs, secrets, variables |
| **Git integration** | โ ๏ธ Manual sync | โ Git-backed version control and CI/CD, UI based Deployment App |
| **Scheduling & triggers** | โ Cron + event triggers | โ Cron, events, file triggers, webhooks, API |
| **Secrets management** | โ ๏ธ Requires Prefect Cloud or external Vault | โ Support both built-in secret manager, or external secret manager |
| **RBAC & SSO** | โ ๏ธ Prefect Cloud only | โ Built-in, enterprise-ready |
| **Observability & logging** | โ Prefect Cloud UI | โ Native UI for logs, metrics, traces, lineage |
| **Scalability** | โ ๏ธ Manual worker configuration | โ Auto-scaled executors on K8s or Docker |
| **Streaming support** | โ | โ Kafka, CDC, real-time ingestion |
| **Team collaboration** | โ ๏ธ Limited multi-tenancy | โ Workspaces, permissions, activity logs |
***
## Step-by-Step Migration Instructions
1. **Inventory your Prefect flows**
* List your active flows and their task types:
* `@flow` and `@task` decorators
* Prefect blocks (storage, infrastructure)
* Dependencies and scheduling logic
* Identify data sources, transformations, and destinations
2. **Create a workspace in Mage Pro**
* Visit [Mage Pro](https://cloud.mage.ai) and sign in
* (Optional) Connect your Git repo for version control
* Set up your deployment configuration using the **Deployment App** in the UI
3. **Recreate your Prefect flows as Mage pipelines**
* Each Prefect **flow** โ 1 Mage pipeline
* Each Prefect **task** โ 1 Mage block:
* `@task` with data loading โ **Data loader block**
* `@task` with transformations โ **Transformer block** (Python or SQL)
* `@task` with data writing โ **Data exporter block**
* Prefect blocks (storage) โ **Data loader/exporter blocks**
* Define dependencies visually using the UI (no decorators needed)
4. **Configure scheduling and triggers**
* Prefect deployments โ Mage **triggers** (cron, interval, or event-based)
* Prefect schedules โ Mage **schedule triggers**
* Configure schedules via UI or YAML
5. **Run and validate pipelines**
* Use Mage Pro's UI to test individual blocks
* Preview dataframes, logs, and outputs at each step
* Compare results to your Prefect flows
6. **Monitor, scale, and automate**
* Monitor pipeline runs and resource usage
* Set alerts and retry rules
* Mage Pro autoscaling handles executor resources automatically on Kubernetes or Docker
* Use the **Deployment App** to manage deployments and rollbacks
***
## Mapping Prefect Concepts to Mage Pro
### Flows โ Pipelines
**Prefect:**
```python theme={"system"}
from prefect import flow, task
@flow(name="my-etl-flow")
def my_etl_flow():
data = extract_data()
transformed = transform_data(data)
load_data(transformed)
```
**Mage Pro:**
* Create a new **pipeline** in the UI
* Add blocks visually or via code
* Dependencies are defined by connecting blocks in the UI
### Tasks โ Blocks
**Prefect:**
```python theme={"system"}
@task
def extract_data():
return pd.read_csv("data.csv")
@task
def transform_data(df):
return df.dropna()
@task
def load_data(df):
df.to_sql("table", engine)
```
**Mage Pro:**
* `extract_data` โ **Data loader block** (Python or connector)
* `transform_data` โ **Transformer block** (Python, SQL, or dbt block)
* `load_data` โ **Data exporter block** (Python or connector)
### Prefect Blocks โ Mage Blocks
| Prefect Block Type | Mage Pro Equivalent |
| --------------------------------- | ------------------------------------------------------------- |
| `S3Block`, `GCSBlock` | **Data loader/exporter blocks** with native S3/GCS connectors |
| `PostgresBlock`, `SnowflakeBlock` | **Data loader/exporter blocks** with database connectors |
| `DockerBlock`, `KubernetesJob` | **Executor configuration** in pipeline settings |
### Scheduling & Deployments
**Prefect:**
```python theme={"system"}
from prefect import serve
from prefect.schedules import CronSchedule
@flow
def scheduled_flow():
...
if __name__ == "__main__":
scheduled_flow.serve(
cron=CronSchedule(cron="0 0 * * *")
)
```
**Mage Pro:**
* Create a **schedule trigger** in the pipeline UI
* Configure cron expression, interval, or event-based triggers
* Use the **Deployment App** to manage deployments across environments (dev, staging, prod)
* No complex deployment setup neededโtriggers and deployments are configured via UI
### Secrets & Configuration
**Prefect:**
```python theme={"system"}
from prefect.blocks.system import Secret
@task
def use_secret():
api_key = Secret.load("my-api-key")
```
**Mage Pro:**
* Use **workspace variables** or **pipeline variables**
* Access via `variables` dictionary in blocks
* **Built-in secret manager** with encryption, or integrate with **external secret managers** (AWS Secrets Manager, HashiCorp Vault, etc.)
* Secrets are isolated per workspace and pipeline
***
## Convert Prefect Flows with AI Sidekick
Skip the manual rewrites โ Mage Pro's **AI Sidekick** can automatically convert your Prefect flow code into a Mage pipeline with just one prompt.
### How to Use It
1. Click the **"Ask AI"** button in the top-right corner of the Mage Pro UI.
2. Paste your Prefect flow code โ including `@flow`, `@task`, or flow-based workflows.
3. Ask: **"Convert this Prefect flow to a Mage pipeline."**
4. Sidekick will:
* Parse the flow structure and task relationships
* Generate the corresponding Mage blocks (data loader, transformer, data exporter)
* Define block dependencies and scheduling logic
* Convert Prefect blocks to Mage connectors where applicable
* **Optimize** the generated code for performance and best practices
5. Review and insert the generated pipeline directly into your project.
### Why Use Sidekick?
* **Faster migration**: turn entire flows into Mage pipelines in seconds
* **Less error-prone**: Sidekick understands scheduling, dependencies, and task types
* **Code optimization**: automatically optimizes generated code for performance
* **Context-aware**: uses your project setup and prior block structure
* **Fully editable**: review, tweak, and insert blocks before saving
๐ Learn more in the [AI Sidekick docs](/ai/sidekick).
***
## Tips for Migrating Complex Flows
### Handling Prefect Subflows
**Prefect:**
```python theme={"system"}
@flow
def parent_flow():
result = subflow()
@flow
def subflow():
...
```
**Mage Pro:**
* Split into **separate pipelines** and trigger them via API or use **dynamic blocks** for conditional logic
* Use **block groups** to organize related blocks within a pipeline
### Prefect Task Runners & Concurrency
**Prefect:**
```python theme={"system"}
from prefect.task_runners import ConcurrentTaskRunner
@flow(task_runner=ConcurrentTaskRunner())
def parallel_flow():
...
```
**Mage Pro:**
* Configure **concurrency settings** in pipeline configuration
* Blocks with no dependencies run in parallel automatically
* Use **executor configuration** for custom parallelism on Kubernetes or Docker
### Prefect Retries & Error Handling
**Prefect:**
```python theme={"system"}
@task(retries=3, retry_delay_seconds=60)
def risky_task():
...
```
**Mage Pro:**
* Configure **retry settings** in block or pipeline configuration
* Set retry count, delay, and backoff strategy
* Built-in error handling and alerting
### Prefect Caching
**Prefect:**
```python theme={"system"}
@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=1))
def cached_task():
...
```
**Mage Pro:**
* Use **block output caching** in pipeline settings
* Configure cache expiration and invalidation rules
* Preview cached outputs in the UI
***
## After Migration: What You Get with Mage Pro
* **Unified data platform**: orchestration + data integration + transformation + streaming in one tool
* **Visual pipeline builder** with real-time data previews
* **AI-powered block generation, optimization, and debugging** (via AI Sidekick)
* **Git-backed version control** and CI/CD with **UI-based Deployment App**
* **Built-in access controls**, audit logs, and workspace isolation
* **Support for SQL, Python, R, streaming, dbt blocks, Delta Lake, Iceberg**, and more
* **200+ native connectors** for databases, APIs, and cloud storage
* **Streaming pipelines** for real-time data processing
* **Flexible secrets management**: built-in secret manager or integrate with external systems (AWS Secrets Manager, Vault, etc.)
* **Auto-scaled executors** on Kubernetes or Docker
* **Transparent pricing**: usage-based pricing (SaaS) or cluster/workspace license-based pricing (self-hosted)
> Prefect excels at orchestration. But when you need **data engineering, visual development, and AI assistance** โ **Mage Pro is built for modern data teams**.
๐ [Migrate to Mage Pro Today](https://cloud.mage.ai)
# Alerting status updates in Discord
Source: https://docs.mage.ai/observability/alerting/alerting-discord
Get status updates in your Discord channel.
## Overview of steps
1. Set up webhooks for Discord
2. Update Mage project settings
***
## Set up webhooks for Discord
Follow these
[instructions on Discord](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks)
to setup incoming webhooks for your workspace.
***
## Update Mage project settings
Once youโve set up webhooks for Discord, you should have a webhook URL that Discord
provides.
Here is an example webhook (yours may vary):
```
https://discordapp.com/api/webhooks/XXX3514XXXXX/MHhahaduwa183-ccgfxcgv88kladJkjBJKB78CHCfds
```
Follow these steps to add that webhook URL to your project settings:
1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).
2. Open a pipeline and start editing it (e.g.
[http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).
3. In your left sidebar in the file browser, scroll all the way down and click
on a file named `metadata.yaml`.
4. In the `metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
```
If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.
Options:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a SLA is missed.
* In this context, SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
* If the pipeline runs *longer* than the SLA, an alert will be sent.
5. In the `metadata.yaml` file, add the following values:
```yaml theme={"system"}
notification_config:
alert_on:
...
discord_config:
webhook_url: ...
```
Change the `webhook_url` value to be the webhook URL you created from Discord.
## Customize message templates
You can customize message templates in `notification_config`. Here is an example config:
```yaml theme={"system"}
notification_config:
alert_on:
...
discord_config:
...
message_templates:
failure:
title: 'Failed to execute pipeline {pipeline_uuid} at {execution_time}'
summary: >
Failed to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message.
success:
title: ...
summary: ...
passed_sla:
title: ...
summary: ...
```
You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify the sentence on `summary` and the `title`.
To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:
1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
* available only for the `failure` message template
8. `stacktrace`
* available only for the `failure` message template
> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.
### ๐ Pro Only: Interpolate Mage Variables in Message Templates
If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Discord message templates using the familiar Jinja-like syntax:
```yaml theme={"system"}
notification_config:
message_templates:
failure:
details: >
Failed at {{ env_var('ENVIRONMENT') }} environment.
Pipeline run: {pipeline_run_url}.
```
In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:
1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)
This makes your Discord notifications more dynamic and context-aware โ ideal for multi-environment deployments and advanced workflows.
***
## What next?
Whenever a pipeline run is successfully completed or fails, a Discord Channel message will
appear in the channel you configured the webhook URL for.
Here is an example of what that message could look like:
| |
| -------------------------------------------------------------------------------------- |
|  |
# Alerting status updates in Email
Source: https://docs.mage.ai/observability/alerting/alerting-email
Get status updates sent to your email inbox.
## Create notification\_config
In the root of your projectโs folder (e.g. `default_repo/`), open the file
`metadata.yaml`.
> Project folder name
>
> If you initialized Mage using a different project name, then your root folder
> will be named differently. `default_repo` is the default project name if you
> didnโt customize it.
In the `default_repo/metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
```
If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.
Options:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a run of a trigger passes sla
## Configure email settings
In the `default_repo/metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
...
email_config:
smtp_host: ...
smtp_mail_from: ...
smtp_user: ...
smtp_password: ...
to_emails:
- someone_lucky@mage.ai
- eng@mage.ai
```
Change the values for each key under the `email_config` section.
## Customize message templates
You can customize message templates in `notification_config`. Here is an example config:
```yaml theme={"system"}
notification_config:
alert_on:
...
email_config:
...
message_templates:
failure:
title: Failed to execute pipeline {pipeline_uuid} at {execution_time}
details: >
Failed to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message.
success:
title: ...
details: ...
passed_sla:
title: ...
details: ...
```
You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify `title` and either `summary` or `details`.
* `title` is used in email subject
* If you specify the `summary`, the email body will be your `summary` + the url of the pipeline run page
* If you specify the `details`, the `details` will be used as your email body
To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:
1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
* available only for the `failure` message template
8. `stacktrace`
* available only for the `failure` message template
> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.
### ๐ Pro Only: Interpolate Mage Variables in Message Templates
If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Email message templates using the familiar Jinja-like syntax:
```yaml theme={"system"}
notification_config:
message_templates:
failure:
details: >
Failed at {{ env_var('ENVIRONMENT') }} environment.
Pipeline run: {pipeline_run_url}.
```
In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:
1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)
This makes your Email notifications more dynamic and context-aware โ ideal for multi-environment deployments and advanced workflows.
***
## What next?
Whenever a pipeline run is successfully completed or fails, an email will be
delivered to all the inboxes listed under `to_emails`.
Here is an example of what an email could look like:
| |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Successfully ran Pipeline `example_pipeline` with Trigger 79 `hourly_trigger` at execution time `2022-09-28 19:00:00`. Open [http://localhost:6789/pipelines/example\_pipeline/triggers/79](http://localhost:6789/pipelines/example_pipeline/triggers/79) to check pipeline run results and logs. |
# Alerting status updates in Opsgenie
Source: https://docs.mage.ai/observability/alerting/alerting-opsgenie
Get status updates in Opsgenie.
## Overview of steps
1. Set up Api Integration for Opsgenie
2. Update Mage project settings
***
## Set up Api Integration for Opsgenie
Follow these
[instructions on Opsgenie](https://support.atlassian.com/opsgenie/docs/create-a-default-api-integration/)
to setup an API Integration in your account.
***
## Update Mage project settings
Once youโve set up API Integration for Opsgenie, you should have an API Key that Opsgenie
provides.
Follow these steps to add that Opsgenie Configuration to your project settings:
1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).
2. Open a pipeline and start editing it (e.g.
[http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).
3. In your left sidebar in the file browser, scroll all the way down and click
on a file named `metadata.yaml`.
4. In the `metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
```
If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.
Options:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a run of a trigger passes sla
5. In the `metadata.yaml` file, add the following values:
```yaml theme={"system"}
notification_config:
alert_on:
...
opsgenie_config:
url: "https://api.eu.opsgenie.com/v2/alerts"
api_key: ""
priority: "P2" # Default P3
tags: # Default empty
- production
details: # Default empty
environment: production
responders: # Default empty
- id: "4513b7ea-3b91-438f-b7e4-e3e54af9147c"
type: "team"
```
Change the `your_api_key` value to be the API Key you created from Opsgenie.
***
## What next?
Whenever a pipeline run is successfully completed or fails, an Opsgenie alert be
created.
Here is an example of what the alert could look like:
| |
| ------------------------------------------------------------------------- |
|  |
# Alerting status updates in Slack
Source: https://docs.mage.ai/observability/alerting/alerting-slack
Get real-time status updates from Mage pipelines directly in your Slack channels.
## Overview of steps
1. Set up webhooks for Slack
2. Update Mage project settings
3. Customize message templates with dynamic variables (Pro only)
***
## Set up webhooks for Slack
To send notifications to Slack, you need to configure an incoming webhook:
* Follow the official [Slack webhook setup guide](https://api.slack.com/messaging/webhooks).
* Create a webhook URL for your desired Slack channel.
***
## Update Mage project settings
Once youโve set up webhooks for Slack, you should have a webhook URL that Slack
provides.
Here is an example webhook (yours may vary):
```
https://hooks.slack.com/services/T01XXXXRKMJ/B04XXXXGY67/1UgUkao8pXXXXPmsuXXXX2lk
```
Follow these steps to add that webhook URL to your project settings:
1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).
2. Open a pipeline and start editing it (e.g.
[http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).
3. In your left sidebar in the file browser, scroll all the way down and click
on a file named `metadata.yaml`.
4. In the `metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
```
If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.
Options:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a SLA is missed.
* In this context, SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
* If the pipeline runs *longer* than the SLA, an alert will be sent.
5. In the `metadata.yaml` file, add the following values:
```yaml theme={"system"}
notification_config:
alert_on:
...
slack_config:
webhook_url: ...
```
Change the `webhook_url` value to be the webhook URL you created from Slack.
## Customize message templates
You can customize message templates in `notification_config`. Here is an example config:
```yaml theme={"system"}
notification_config:
alert_on:
...
slack_config:
...
message_templates:
failure:
details: >
Failed to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message.
success:
details: ...
passed_sla:
details: ...
```
You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify either `summary` or `details`.
* If you specify the `summary`, the slack message will be your `summary` + the url of the pipeline run page
* If you specify the `details`, the `details` will be used as your slack message directly
To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:
1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
* available only for the `failure` message template
8. `stacktrace`
* available only for the `failure` message template
> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.
### ๐ Pro Only: Interpolate Mage Variables in Message Templates
If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Slack message templates using the familiar Jinja-like syntax:
```yaml theme={"system"}
notification_config:
message_templates:
failure:
details: >
Failed at {{ env_var('ENVIRONMENT') }} environment.
Pipeline run: {pipeline_run_url}.
```
In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:
1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)
This makes your Slack notifications more dynamic and context-aware โ ideal for multi-environment deployments and advanced workflows.
***
## What next?
Whenever a pipeline run is successfully completed or fails, a Slack message will
appear in the channel you configured the webhook URL for.
Here is an example of what that message could look like:
| |
| ---------------------------------------------------------------------------------- |
|  |
# Alerting status updates in Teams
Source: https://docs.mage.ai/observability/alerting/alerting-teams
Get status updates in your Teams channel.
## Overview of steps
1. Set up webhooks for Teams
2. Update Mage project settings
***
## Set up webhooks for Teams
Follow these
[instructions on Teams](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook)
to setup incoming webhooks for your workspace.
***
## Update Mage project settings
Once youโve set up webhooks for Teams, you should have a webhook URL that Teams
provides.
Here is an example webhook (yours may vary):
```
https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lk
```
Follow these steps to add that webhook URL to your project settings:
1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).
2. Open a pipeline and start editing it (e.g.
[http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).
3. In your left sidebar in the file browser, scroll all the way down and click
on a file named `metadata.yaml`.
4. In the `metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
```
If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.
Options:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a run of a trigger passes sla
5. In the `metadata.yaml` file, add the following values:
```yaml theme={"system"}
notification_config:
alert_on:
...
teams_config:
webhook_url:
- 'https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lk'
- 'https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lx'
```
Change the `webhook_url` values to be the webhook URLs you created from Teams.
The `webhook_url` must be a string if you are using version 0.9.73 or older and it only accepts one endpoint.
```
teams_config:
webhook_url: 'https://teams.webhook.office.com/webhookb2/B04XXXXGY67/IncomingWebhook/1UgUkao8pXXXXPmsuXXXX2lk'
```
## Customize message templates
You can customize message templates in `notification_config`. Here is an example config:
```yaml theme={"system"}
notification_config:
alert_on:
...
teams_config:
...
message_templates:
failure:
details: >
Failed to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message.
success:
details: ...
passed_sla:
details: ...
```
You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify either `summary` or `details`.
* If you specify the `summary`, the Teams message will be your `summary` + the url of the pipeline run page
* If you specify the `details`, the `details` will be used as your Teams message directly
To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:
1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
* available only for the `failure` message template
8. `stacktrace`
* available only for the `failure` message template
> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.
### ๐ Pro Only: Interpolate Mage Variables in Message Templates
If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Teams message templates using the familiar Jinja-like syntax:
```yaml theme={"system"}
notification_config:
message_templates:
failure:
details: >
Failed at {{ env_var('ENVIRONMENT') }} environment.
Pipeline run: {pipeline_run_url}.
```
In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:
1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)
This makes your Teams notifications more dynamic and context-aware โ ideal for multi-environment deployments and advanced workflows.
***
## What next?
Whenever a pipeline run is successfully completed or fails, a Teams message will
appear in the channel you configured the webhook URL for.
Here is an example of what that message could look like:
| |
| ---------------------------------------------------------------------------------- |
|  |
# Alerting status updates in Telegram
Source: https://docs.mage.ai/observability/alerting/alerting-telegram
Get status updates in your Telegram Group.
## Overview of steps
1. Set Bot with **BotFather** for Telegram
2. Get the chat\_id of your Telegram Group
3. Update Mage project settings
***
## Set Bot with BotFather for Telegram
Follow these
[Tutorial](https://www.directual.com/lesson-library/how-to-create-a-telegram-bot)
to setup Bot with BotFather and **Don't Forget to add your Bot to the Group**.
***
## Get the chat\_id of your Telegram Group
1. Open [https://web.telegram.org/](https://web.telegram.org/) and Follow the instructions
2. Click your Telegram Group and your Group chat\_id is on your url like `https://web.telegram.org/a/#-7222xxxxxx`
## Update Mage project settings
Once youโve set up the **Bot** and get the `chat_id` of your group, you need to set the webhook like example below:
Here is an example webhook (yours may vary):
```
https://api.telegram.org/bot{your_token:from_botfather}/sendMessage?chat_id=-7222xxxxxx
```
Follow these steps to add that webhook URL to your project settings:
1. Open the Mage tool in your browser (e.g. [http://localhost:6789/](http://localhost:6789/)).
2. Open a pipeline and start editing it (e.g.
[http://localhost:6789/pipelines/example\_pipeline/edit](http://localhost:6789/pipelines/example_pipeline/edit)).
3. In your left sidebar in the file browser, scroll all the way down and click
on a file named `metadata.yaml`.
4. In the `metadata.yaml` file, add a section with the following
configuration or update the existing `notification_config` section:
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
```
If you omit the `alert_on` section it will default to `trigger_failure` and `trigger_passed_sla`.
Options:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a SLA is missed.
* In this context, SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
* If the pipeline runs *longer* than the SLA, an alert will be sent.
5. In the `metadata.yaml` file, add the following values:
```yaml theme={"system"}
notification_config:
alert_on:
...
telegram_config:
webhook_url: ...
```
Change the `webhook_url` value to be the webhook URL like mentioned above.
## Customize message templates
You can customize message templates in `notification_config`. Here is an example config:
```yaml theme={"system"}
notification_config:
alert_on:
...
telegram_config:
...
message_templates:
failure:
title: 'Failed to execute pipeline {pipeline_uuid} at {execution_time}'
summary: >
Failed to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message.
success:
title: ...
summary: ...
passed_sla:
title: ...
summary: ...
```
You can customize the message template for `success`, `failure`, `passed_sla` scenarios. For each message template,
you can specify the sentence on `summary` and the `title`.
To interpolate variables in the message template, you can use `{variable_name}` syntax.
Here are the supported variables:
1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error`
* available only for the `failure` message template
8. `stacktrace`
* available only for the `failure` message template
> **Note**: For `pipeline_run_url`, the default host is `http://localhost:6789`. You can specify `MAGE_PUBLIC_HOST` to be
> the host url you want to use in the notification messages.
### ๐ Pro Only: Interpolate Mage Variables in Message Templates
If you're using **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) directly in your Telegram message templates using the familiar Jinja-like syntax:
```yaml theme={"system"}
notification_config:
message_templates:
failure:
title: 'Failed to execute pipeline {pipeline_uuid} at {{ env_var('ENVIRONMENT') }} environment'
summary: >
Failed to execute pipeline {pipeline_run_url}.
Pipeline uuid: {pipeline_uuid}. Trigger name: {pipeline_schedule_name}.
Test custom message.
```
In addition to the variables mentioned above (e.g. `execution_time`, `pipeline_uuid`, etc), you can also use the
following in your message templates in Mage Pro:
1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (this gets the value stored in the `ENV` environment variable)
This makes your Telegram notifications more dynamic and context-aware โ ideal for multi-environment deployments and advanced workflows.
***
## What next?
Whenever a pipeline run is successfully completed or fails, a Telegram Group message will
appear in the group you configured the webhook URL for.
Here is an example of what that message could look like:
| |
| ---------------------------------------------------------------------------------------- |
|  |
# Alerting status updates in Zendesk
Source: https://docs.mage.ai/observability/alerting/alerting-zendesk
Create Zendesk tickets from Mage pipeline events to track incidents and resolution.
## Overview of steps
1. Create an API token in Zendesk
2. Update Mage project settings with Zendesk config
3. Customize ticket templates with dynamic variables (Pro only)
***
## Create an API token in Zendesk
To allow Mage to create tickets in Zendesk, you need to set up an API token:
1. Log in to Zendesk as an **admin**.
2. Go to **Admin Center** โ **Apps and integrations** โ **Zendesk API**.
3. Enable **Token Access**.
4. Click **Add API token**.
5. Enter a description (e.g., "Mage Alerts") and copy the generated token.
> **Note:** This token will only be shown once. Store it securely.
6. Identify your Zendesk **subdomain** (e.g., for `https://acme.zendesk.com` the subdomain is `acme`).
7. Ensure the account you use (email) has **agent** or **admin** permissions.
***
## Update Mage project settings
Once you have your subdomain, email, and API token, add them to your project settings.
Hereโs an example `zendesk_config` (add under `notification_config` in `metadata.yaml`):
```yaml theme={"system"}
notification_config:
alert_on:
- trigger_failure
- trigger_passed_sla
zendesk_config:
subdomain: your-domain
email: bot@example.com
api_token: YOUR_ZENDESK_API_TOKEN
group_id: "1234567890" # optional: numeric group ID
assignee_id: "9876543210" # optional: numeric agent ID
priority: high # optional: low | normal | high | urgent
status: new # optional: new | open | pending | hold
tags: mage,alert,pipeline # optional: comma-separated tags
```
If you omit the `alert_on` section, it will default to `trigger_failure` and `trigger_passed_sla`.
Options for `alert_on`:
* `trigger_failure`: alert when a run of a trigger fails
* `trigger_success`: alert when a run of a trigger succeeds
* `trigger_passed_sla`: alert when a SLA is missed.
* SLA (Service Level Agreement) is an expected amount of time for the pipeline to complete.
* If the pipeline runs *longer* than the SLA, an alert will be sent.
***
## Customize ticket templates
You can customize ticket templates in `notification_config` just like Slack message templates:
```yaml theme={"system"}
notification_config:
alert_on:
...
zendesk_config:
...
message_templates:
failure:
details: >
Pipeline {pipeline_uuid} failed.
See logs: {pipeline_run_url}.
success:
details: ...
passed_sla:
details: ...
```
You can customize templates for `success`, `failure`, and `passed_sla` scenarios.\
For each template:
* If you specify `summary`, the ticket subject will be your summary, and `details` will be used as the body.
* If you specify only `details`, it will be used as the ticket body directly.
**Supported variables:**
1. `execution_time`
2. `pipeline_run_url`
3. `pipeline_schedule_id`
4. `pipeline_schedule_name`
5. `pipeline_schedule_description`
6. `pipeline_uuid`
7. `error` (failure only)
8. `stacktrace` (failure only)
***
### ๐ Pro Only: Interpolate Mage Variables in Ticket Templates
In **Mage Pro**, you can interpolate environment variables and other [Mage variables](/development/variables/referencing-variables#accessing-variables-in-mage) in your Zendesk ticket templates using Jinja-like syntax:
```yaml theme={"system"}
notification_config:
message_templates:
failure:
details: >
Failed at {{ env_var('ENVIRONMENT') }} environment.
Run details: {pipeline_run_url}.
```
Additional variables available in Mage Pro:
1. `start_time`
2. `end_time`
3. `duration` (in seconds)
4. `env` (value of the `ENV` environment variable)
***
## What next?
Whenever a pipeline run completes or fails, Mage will create a Zendesk ticket based on your configuration.\
You can route these tickets to specific groups or agents, tag them, and track resolution in Zendesk.
Example ticket subject/body:
| |
| --------------------------------------------------------------------------------------------------------------------------------- |
| **Subject:** Mage Notification: Failed to run Mage pipeline example\_pipeline |
| **Body:** Pipeline `example_pipeline` failed at 2025-08-13. View logs at: https\://mage\_url/pipelines/example\_pipeline/runs/123 |
# Writing metrics to Datadog
Source: https://docs.mage.ai/observability/external/datadog
## Provide your Datadog credentials
You will need to add the following environment variables in order to use
Datadog in mage:
* DD\_API\_KEY: your datadog api key
* DD\_APP\_KEY: your datadog app key
## Using Datadog in Mage
You can now call Datadog helper functions in your Mage blocks to create or update metrics.
You can see the existing helper functions [here](https://github.com/mage-ai/mage-ai/blob/master/mage_ai/services/datadog/__init__.py).
Example:
```python theme={"system"}
from mage_ai.services import datadog as dd
dd.increment('', tags=dict(pipeline=''))
```
# Metaplane in Mage
Source: https://docs.mage.ai/observability/external/metaplane
Run monitors in Metaplane.
## Configuration
Here are the following keyword arguments that can be used for configuration:
| Keyword argument | Description | Sample value |
| ---------------- | -------------------- | ------------ |
| `api_token` | Metaplane API token. | `abc123...` |
***
## Get all connections
```python theme={"system"}
from mage_ai.services.metaplane.metaplane import Metaplane
client = Metaplane(config=dict(api_token='abc123...'))
client.connections()
```
## Example code
Trigger sync with `source_id`.
```python theme={"system"}
from mage_ai.services.metaplane.metaplane import Metaplane
client = Metaplane(config=dict(api_token='abc123...'))
connection_id = client.connections()[0]['id']
# Run all monitors for a connection, and keep polling until they are all complete.
client.process(
connection_id=connection_id,
poll_interval=60, # optional: check the status of a monitor every N seconds
raise_on_failure=False, # optional: if a monitor test fails, raise an error
)
```
Sample output:
```
Metaplane process started.
Running 1 monitor(s).
Running monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 ran with status 200.
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: False; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:35:52.125462+00:00).
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: False; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:35:52.125462+00:00).
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: False; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:35:52.125462+00:00).
Checking status for monitor ID ab501e82-8412-444c-74a3-fb12f15ad572.
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 completed: True; started 2023-01-24 11:39:41.161332, last created 2023-01-24 11:39:50.791930+00:00).
Monitor ID ab501e82-8412-444c-74a3-fb12f15ad572 passed: True.
Metaplane process completed.
```
# Monitoring with New Relic
Source: https://docs.mage.ai/observability/external/newrelic
## Enable New Relic
You will need to add `ENABLE_NEW_RELIC` environment variable as `true` in order to use
New Relic in Mage.
## New Relic Configuration File
Following [New Relic Python integration guide](https://docs.newrelic.com/install/python/?python-non-web=non-web-yes),
run command: `newrelic-admin generate-config YOUR_LICENSE_KEY newrelic.ini`
to generate the `newrelic.ini` configuration file and edit your `newrelic.ini` and insert values for the following:
...
license\_key = INSERT\_YOUR\_LICENSE\_KEY
...
app\_name = INSERT\_YOUR\_APP\_NAME
...
An example `example_newrelic.ini` file is attacehd in Mage repo under folder `integrations/newrelic`.
Add path of your newrelic configuration into environment variable `NEW_RELIC_CONFIG_PATH`.
## Using New Relic in Mage
Monitoring with New Relic will be enabled the next time you start the Mage server.
The following logs and errors will be reported to New Relic:
* Trigger runs.
* Mage scheduler.
* `mage run` CLI command.
# Monitoring with OpenTelemetry
Source: https://docs.mage.ai/observability/external/opentelemetry
## Overview
This guide details how to monitor your `mage_ai` application using OpenTelemetry. OpenTelemetry provides a unified approach to collecting telemetry data, such as traces and metrics, which are essential for understanding your application's performance and behavior.
## Prerequisites
Ensure the following prerequisites are in place before proceeding:
* An OpenTelemetry Collector is set up and configured to receive telemetry data.
* The `mage_ai` application is ready for instrumentation with OpenTelemetry.
## Configuring OpenTelemetry in `mage_ai`
### Setting Environment Variables
Configure the OpenTelemetry Exporters in your application by setting the following environment variables:
1. **OTLP HTTP Exporter**:
Use `OTEL_EXPORTER_OTLP_HTTP_ENDPOINT` to specify the HTTP endpoint of your OpenTelemetry Collector.
Example:
```bash theme={"system"}
export OTEL_EXPORTER_OTLP_HTTP_ENDPOINT="http://192.168.1.56:3418/v1/traces"
```
2. **OTLP Exporter**:
The OTEL\_EXPORTER\_OTLP\_ENDPOINT environment variable sets a general collector endpoint, which can be used for both gRPC and HTTP connections.
Example:
```bash theme={"system"}
export OTEL_EXPORTER_OTLP_ENDPOINT="192.168.1.56:3417"
```
### Instrumentation in the Application
1. **SQLAlchemy Instrumentation**:
The application uses the SQLAlchemyInstrumentor from OpenTelemetry to instrument database operations. This generates telemetry data like traces for database interactions.
2. **Tornado Instrumentation**:
To monitor HTTP server operations, the application integrates OpenTelemetry's Tornado instrumentation. This collects valuable data related to HTTP requests and server performance.
## Telemetry Data Collection and Analysis
With the environment variables set and the application running, OpenTelemetry will start collecting telemetry data based on the defined instrumentation.
1. **Data Collection**:
The application will send the collected telemetry data, including traces and metrics, to the specified OpenTelemetry Collector endpoint.
2. **Analyzing Telemetry Data**:
For effective visualization and analysis, you can connect your OpenTelemetry Collector to backend tools that support OpenTelemetry data (like Grafana or Jaeger).
# Monitoring with Prometheus
Source: https://docs.mage.ai/observability/external/prometheus
## Enable Prometheus
You will need to add `ENABLE_PROMETHEUS` environment variable as `true` in order to
enable Prometheus style metrics on the \/metrics route.
## Prometheus metrics
When enabled, Mage will publish Metrics for the HTTP Server (Tornado) as well as for
the Python Runtime. The /metrics route will be updated on-demand to respond to scrape requests.
## Kubernetes resource usage in Mage Pro
In Mage Pro, Kubernetes executor resource usage summaries can optionally read
historical CPU and memory samples from Prometheus. Set
`K8S_RESOURCE_USAGE_PROMETHEUS_URL` to your Prometheus base URL to make
Prometheus the preferred source for K8s resource usage metrics. If Prometheus is
not configured or does not return samples for the run pod, Mage falls back to the
Kubernetes Metrics API when it is available.
Mage queries Prometheus for the completed run pod by namespace and pod name. CPU
usage is read from `container_cpu_usage_seconds_total`, and memory usage is read
from `container_memory_working_set_bytes`. The stored samples power the
**Resources** tab on pipeline run pages and the K8s resource usage panel in
pipeline monitoring.
### Set up Prometheus for resource usage
For self-hosted or private-cloud Mage Pro clusters, install Prometheus in the
same Kubernetes cluster as the Mage deployment and expose it to the Mage server
pod through an internal Kubernetes service. Keep Prometheus as a `ClusterIP`
service unless your platform already provides an authenticated internal gateway.
Install Prometheus separately from the Mage Pro Helm chart:
```bash theme={"system"}
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace
```
After installation, confirm the Prometheus service name in your cluster:
```bash theme={"system"}
kubectl get svc -n monitoring | grep prometheus
```
The default service name from `kube-prometheus-stack` is commonly
`prometheus-kube-prometheus-prometheus`, but it can differ based on the Helm
release name and chart values.
### Verify required metrics
Mage Pro needs cAdvisor container CPU and memory metrics. Port-forward
Prometheus, then check that the metrics exist for pods in the namespace where
K8s executor jobs run:
```bash theme={"system"}
kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
```
```bash theme={"system"}
curl -G http://localhost:9090/api/v1/query \
--data-urlencode 'query=count(container_cpu_usage_seconds_total{namespace="mage",pod!=""})'
curl -G http://localhost:9090/api/v1/query \
--data-urlencode 'query=count(container_memory_working_set_bytes{namespace="mage",pod!=""})'
```
Replace `mage` with the namespace used by your K8s executor pods. If either
query returns `0`, confirm that Prometheus is scraping kubelet/cAdvisor metrics
for that namespace before enabling Mage Pro resource usage collection.
### Configure Mage Pro
Set the Prometheus URL on the Mage Pro web/server deployment. Use the in-cluster
service DNS name so traffic stays inside the cluster:
```yaml theme={"system"}
K8S_RESOURCE_USAGE_PROMETHEUS_URL: http://prometheus-kube-prometheus-prometheus.monitoring.svc:9090
K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS: "15"
K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS: "15"
K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS: "5"
K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS: "true"
```
If Prometheus is behind an authenticated proxy, configure either bearer token
auth:
```yaml theme={"system"}
K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN: ""
```
or basic auth:
```yaml theme={"system"}
K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME: ""
K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD: ""
```
Bearer token auth takes precedence over basic auth when both are set.
### Managed cloud deployments
For managed cloud deployments, do not configure customer Mage Pro clusters to
query a shared Prometheus service directly. Route resource usage requests
through a tenant-aware gateway, such as `mage-pro-api`, so authentication,
authorization, and tenant label scoping happen before Prometheus is queried.
### Configuration reference
| Variable | Description | Default |
| -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `K8S_RESOURCE_USAGE_PROMETHEUS_URL` | Prometheus base URL used for Mage Pro K8s resource usage summaries. Leave unset to use only the Kubernetes Metrics API fallback. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN` | Bearer token for Prometheus requests. Takes precedence over basic auth when set. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME` | Username for Prometheus basic authentication. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD` | Password for Prometheus basic authentication. | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS` | Request timeout for each Prometheus query. | `5` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` | Query range step used when sampling each completed run pod. | `15` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS` | Delay before retrying Prometheus once for completed runs that initially return too few samples. | `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS` | Set to `0`, `false`, `no`, or `off` to skip TLS certificate verification for Prometheus requests. | `true` |
# Monitoring with Sentry
Source: https://docs.mage.ai/observability/external/sentry
## Provide your Sentry credentials
You will need to add the following environment variables in order to use
Sentry in Mage:
* `SENTRY_DSN`: sentry dsn url
* `SENTRY_TRACES_SAMPLE_RATE`: (optional) value between 0 and 1
* `SENTRY_SERVER_NAME`: (optional) Name of the Mage server
## Using Sentry in Mage
Once you set the `SENTRY_DSN` environment variable, the Sentry integration will be
enabled the next time you start the Mage server.
The following errors will be reported to Sentry:
* Exceptions from trigger runs.
* Exceptions from the Mage scheduler.
* Exceptions from runs started by the `mage run` CLI command.
# OpenSearch log search
Source: https://docs.mage.ai/observability/log-search-opensearch
Use OpenSearch-backed log search when you want Mage to search pipeline, block,
trigger, and run logs across log files instead of only reading the latest files
from disk.
For Mage Pro SaaS and Hybrid Cloud deployments, OpenSearch log search is enabled
automatically. No Helm chart changes or Kubernetes objects are required.
For other deployment options, including private cloud and on-prem deployments,
use this guide to update Helm chart values and create any required Kubernetes
Secrets for secured OpenSearch. In Kubernetes deployments, the Mage Helm chart
can wire the Mage containers, Fluent Bit sidecar, OpenSearch index setup job,
and shared log volume from the `logSearch.*` values.
## Quick start
For a private cloud or on-prem quick start, enable the bundled OpenSearch
subchart and log search in the same Helm upgrade. This is the shortest path
because the chart can create the OpenSearch resources, Fluent Bit config, and
index setup job for you. Leave `logSearch.opensearch.host` empty; the chart
derives the in-cluster OpenSearch service name.
### Requirements
* A Mage deployment managed by the
[Mage Helm chart](https://github.com/mage-ai/helm-charts/tree/master/charts/mageai).
* A Kubernetes Secret with a strong `OPENSEARCH_INITIAL_ADMIN_PASSWORD` value
for bundled OpenSearch.
* A `ReadWriteOnce` storage class for the bundled OpenSearch data volume when
your cluster does not have a suitable default StorageClass. For example, an
EKS cluster that uses EFS for shared Mage project files may still need an
EBS-backed class such as `gp2` or `gp3` for OpenSearch.
* A Kubernetes PVC that contains the Mage project files and log files. The
quick start reuses this PVC through `logSearch.persistence.existingClaim`.
If the OpenSearch PVC stays pending, the setup job cannot connect to OpenSearch
and Helm may wait on the hook.
### Minimum values
Create a Secret for the bundled OpenSearch admin bootstrap password.
```bash theme={"system"}
kubectl -n mage create secret generic opensearch-initial-admin \
--from-literal=OPENSEARCH_INITIAL_ADMIN_PASSWORD=
```
Then reference the Secret from the OpenSearch subchart values.
```yaml theme={"system"}
opensearch:
enabled: true
extraEnvs:
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: opensearch-initial-admin
key: OPENSEARCH_INITIAL_ADMIN_PASSWORD
logSearch:
enabled: true
persistence:
existingClaim:
setupJob:
enabled: true
```
`logSearch.opensearch.host`, `logSearch.opensearch.port`, and
`logSearch.opensearch.index` can be omitted in bundled mode. The chart derives
the bundled OpenSearch host and defaults to port `9200` and index `mage_logs`.
Add `logSearch.persistence.subPath` only when your project/log PVC requires it.
`logSearch.setupJob.enabled` defaults to `false`, so keep it set to `true` when
you want Helm to create the `mage_logs` index during the upgrade.
Do not commit the real `OPENSEARCH_INITIAL_ADMIN_PASSWORD` value into
`values.yaml`.
If your cluster does not have a suitable default StorageClass for OpenSearch,
add one explicitly:
```yaml theme={"system"}
opensearch:
persistence:
storageClass:
```
### Render and upgrade
Render the exact chart source and values you plan to deploy.
```bash theme={"system"}
helm template mageai /charts/mageai \
--namespace mage \
--values values-log-search.yaml
```
Confirm the rendered manifest includes OpenSearch resources,
`USE_OPENSEARCH_FOR_LOGS=true`, the Fluent Bit sidecar, the chart-rendered
Fluent Bit ConfigMap, and the chart-rendered index setup Job. Then upgrade the
release.
```bash theme={"system"}
helm dependency build /charts/mageai
helm upgrade --install mageai /charts/mageai \
--namespace mage --create-namespace \
--values values-log-search.yaml \
--reuse-values
```
The setup job is idempotent. After it creates the index successfully, you can
set `logSearch.setupJob.enabled=false` in later upgrades if you do not want Helm
to run it again.
## Customize the quick start
Use the quick start as the base, then customize only the pieces your deployment
needs.
### External OpenSearch
To use an external or managed OpenSearch cluster, leave `opensearch.enabled`
unset or `false`, and set `logSearch.opensearch.host`.
```yaml theme={"system"}
logSearch:
enabled: true
opensearch:
host: opensearch-cluster-master.mage-search.svc.cluster.local
port: "9200"
index: mage_logs
persistence:
existingClaim:
subPath:
setupJob:
enabled: true
```
When `opensearch.enabled=false`, `logSearch.opensearch.host` is required. The
chart fails rendering if log search is enabled for an external OpenSearch
cluster without a host.
### Auth and TLS
For OpenSearch auth, create a Secret with `OPENSEARCH_USERNAME` and
`OPENSEARCH_PASSWORD`, then reference it from `logSearch.opensearch.auth`.
```bash theme={"system"}
kubectl -n mage create secret generic opensearch-auth \
--from-literal=OPENSEARCH_USERNAME= \
--from-literal=OPENSEARCH_PASSWORD=
```
```yaml theme={"system"}
logSearch:
opensearch:
auth:
enabled: true
existingSecret: opensearch-auth
```
For TLS, create a Secret with `ca.crt`, then reference it from
`logSearch.opensearch.tls`.
```yaml theme={"system"}
logSearch:
opensearch:
tls:
enabled: true
existingSecret: opensearch-tls-ca
verify: true
```
When TLS is enabled, the chart sets `OPENSEARCH_HTTP_SCHEME=https` for Mage,
uses HTTPS in the index setup hook Job, and mounts the CA certificate at
`/etc/ssl/opensearch/ca.crt`. Fluent Bit receives `OPENSEARCH_TLS=On`.
Securing a bundled OpenSearch cluster also requires OpenSearch security
configuration, not only Mage-side auth variables. Make sure OpenSearch has valid
node/admin certificates, `internal_users.yml`, and role mappings, then run the
OpenSearch security bootstrap for the cluster if your OpenSearch distribution
requires it. Avoid mounting a custom `roles.yml` unless you intentionally need
to replace roles; overriding built-in static roles can prevent security
initialization. If Helm keeps reusing an earlier failed nested security value,
rerender with your intended values and consider an upgrade with `--reset-values`
using a complete values file.
### Project mounts and generated PVCs
If an existing Mage release already has important `extraVolumes` or
`extraVolumeMounts`, check the rendered Deployment carefully. The chart
preserves existing mounts and avoids injecting a duplicate `/home/src` project
mount when `extraVolumeMounts` already contains the log-search mount path. If
your release already mounts the project path, set
`logSearch.persistence.mountInMageContainer=false` and let Fluent Bit reuse that
mount read-only.
For generated log-search PVCs, use `logSearch.persistence.accessModes` and
`storageClassName` that match your topology. When `standaloneScheduler=true` or
`replicaCount > 1`, generated log-search PVCs must include `ReadWriteMany`; use
an existing RWX claim or set `logSearch.persistence.accessModes` accordingly.
### Custom Fluent Bit config
The default path uses a chart-rendered Fluent Bit ConfigMap. Use
`logSearch.fluentBit.existingConfigMap` or
`logSearch.fluentBit.existingParsersConfigMap` only when you need to provide
custom Fluent Bit ConfigMaps yourself.
If you override `logSearch.fluentBit.config`, keep the OpenSearch output
environment variables wired to the chart values:
```text theme={"system"}
Host ${OPENSEARCH_HOST}
Port ${OPENSEARCH_PORT}
Index ${OPENSEARCH_LOG_INDEX}
tls ${OPENSEARCH_TLS}
tls.verify ${OPENSEARCH_VERIFY_CERTS}
```
## Verify the deployment
Check that the Mage pod has both the Mage container and the Fluent Bit sidecar.
```bash theme={"system"}
kubectl -n mage get pods -l app.kubernetes.io/name=mageai
```
Confirm Fluent Bit is sending records.
```bash theme={"system"}
kubectl -n mage logs -l app.kubernetes.io/name=mageai -c fluent-bit --tail=50
```
Confirm the OpenSearch index exists and has documents.
```bash theme={"system"}
kubectl -n mage-search exec -it -- \
curl -s localhost:9200/_cat/indices/mage_logs?v
```
If OpenSearch uses auth or TLS, include the matching `curl` flags for your
cluster, for example `-u ':'`, `https://localhost:9200`, and
`-k` or `--cacert`.
Finally, open a pipeline's **Logs** page in Mage and search for text that appears
in a recent pipeline or block run.
## Troubleshooting
If searches return no results, check these items first:
* `USE_OPENSEARCH_FOR_LOGS` is set to `true` in the Mage web and scheduler pods.
* `OPENSEARCH_HOST`, `OPENSEARCH_PORT`, and `OPENSEARCH_LOG_INDEX` match the
OpenSearch service and index.
* The setup job completed, or the `mage_logs` index already exists.
* The Fluent Bit sidecar can read the mounted project/log path.
* The Fluent Bit sidecar logs show successful flushes instead of connection,
parser, or path errors.
* Auth and TLS settings are consistent across Mage, Fluent Bit, the setup job,
and the OpenSearch cluster.
* The `logSearch.persistence.existingClaim` and `subPath` point at the same
project files that Mage writes logs into.
* If `opensearch.enabled=false`, `logSearch.opensearch.host` is set. The chart
fails rendering when log search is enabled for an external OpenSearch cluster
without a host.
When troubleshooting Helm rendering, compare your values against the examples in
the Helm chart repository under `examples/aws-eks/values-log-search.yaml` and
`examples/aws-eks/log-search-deployment.md`.
## Reference
### What the chart creates
The Helm chart owns the non-secret log-search resources. In the default path,
you do not need to create Fluent Bit ConfigMaps or an `opensearch_setup.py`
ConfigMap before running `helm upgrade`.
When `logSearch.enabled=true`, the chart can:
* set `USE_OPENSEARCH_FOR_LOGS=true` in the Mage web and scheduler containers;
* set `OPENSEARCH_HOST`, `OPENSEARCH_PORT`, and `OPENSEARCH_LOG_INDEX`;
* optionally set OpenSearch auth and TLS environment variables from Kubernetes
Secrets;
* mount the project/log PVC into the Mage container and the Fluent Bit sidecar;
* render the `mageai-fluent-bit` ConfigMap from `logSearch.fluentBit.config`
and `logSearch.fluentBit.parsers`, unless you provide existing ConfigMaps;
* run Fluent Bit to tail Mage log files and ship them to OpenSearch;
* render the `mageai-log-search-index` ConfigMap from
`logSearch.setupJob.mapping`;
* create a log-search PVC when `logSearch.persistence.enabled=true` and no
`existingClaim` is set;
* create a Helm hook Job that applies `logSearch.setupJob.mapping` to create
the OpenSearch log index.
The default index name is `mage_logs`.
If you use a packaged chart from a Helm repository, render that packaged chart
too. If the rendered manifest does not include the `logSearch` environment
variables, OpenSearch subchart resources, Fluent Bit sidecar, chart-rendered
Fluent Bit ConfigMap, and chart-rendered index setup Job, use a newer chart
package or deploy from a checked-out chart source that includes the `logSearch`
templates.
### Helm values
| Helm value | Runtime effect |
| -------------------------------------------- | ------------------------------------------------------------------------------- |
| `logSearch.enabled` | Enables OpenSearch-backed log search and sets `USE_OPENSEARCH_FOR_LOGS=true`. |
| `logSearch.opensearch.host` | Sets `OPENSEARCH_HOST`. Leave empty only when `opensearch.enabled=true`. |
| `logSearch.opensearch.port` | Sets `OPENSEARCH_PORT`. |
| `logSearch.opensearch.index` | Sets `OPENSEARCH_LOG_INDEX`; defaults to `mage_logs`. |
| `logSearch.opensearch.auth.existingSecret` | Reads `OPENSEARCH_USERNAME` and `OPENSEARCH_PASSWORD` from a Kubernetes Secret. |
| `logSearch.opensearch.tls.enabled` | Uses HTTPS for Mage and the setup job, and TLS for Fluent Bit. |
| `logSearch.opensearch.tls.existingSecret` | Mounts `ca.crt` at `/etc/ssl/opensearch/ca.crt`. |
| `logSearch.persistence.enabled` | Creates a log-search PVC when `existingClaim` is empty. |
| `logSearch.persistence.existingClaim` | Reuses a shared Mage project/log PVC for Mage and Fluent Bit. |
| `logSearch.persistence.mountInMageContainer` | Controls whether the chart mounts the log-search PVC into the Mage container. |
| `logSearch.fluentBit.enabled` | Adds the Fluent Bit sidecar that ships log files to OpenSearch. |
| `logSearch.fluentBit.config` | Renders the chart-managed Fluent Bit config. |
| `logSearch.fluentBit.existingConfigMap` | Uses an existing Fluent Bit config ConfigMap instead of rendering one. |
| `logSearch.setupJob.enabled` | Runs the index creation job after Helm install or upgrade. |
| `logSearch.setupJob.mapping` | Renders the index mapping used by the setup job. |
# Logging
Source: https://docs.mage.ai/observability/logging
## Server Logging
The server logger can be configured to adjust the amount of detail that is included in the server logs. This can be helpful for debugging or troubleshooting server issues. You can change the verbosity level of the logger by setting the `SERVER_VERBOSITY` environment variable.
The `SERVER_VERBOSITY` variable accepts the following values: `DEBUG`, `INFO`, `WARNING`, `ERROR` and `CRITICAL`.
## Server logging format
You can customize the format by setting the `SERVER_LOGGING_FORMAT` environment variable. By default it is set as
`SERVER_LOGGING_FORMAT=plaintext`. If it is set to unknown format it will use default value instead.
### Plaintext format
The plaintext logging format will be used by default. Or it can be set explicitly by setting the
`SERVER_LOGGING_FORMAT=plaintext` environment variable.
By default, the server will log messages in the following format:
```
%(levelname)s:%(name)s:%(message)s
```
It can be customized by setting the `SERVER_LOGGING_TEMPLATE` environment variable.
Example:
```
SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'
```
**Important**
Format string with special characters like `\t`, `\n`, etc. can't be passed through `.env` file as they are getting
escaped.
To pass such string you can
1. Use `export SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t[%(name)25.25s]\t%(levelname)5s: %(message)s'` syntax for local
python deployment or docker compose with exported env variables
2. Use combination of `.env` file and `-e` flag for plain docker deployment
`docker run -d --env-file .env -e SERVER_LOGGING_TEMPLATE=$'%(asctime)s\t\t[%(name)25.25s]\t%(levelname)5s: %(message)s' mageai/mageai:latest`
### JSON format
To use JSON logging format, set `SERVER_LOGGING_FORMAT=json`.
JSON formatted logs will have the following fields:
```json theme={"system"}
{
"timestamp": "...",
"level": "...",
"message": "...",
"logger": "...",
"module": "...",
"function": "...",
"line_number": "..."
}
```
More logging formats/customization to come...
## Pipelines logging
By default, logs for each pipeline run will be stored in the
`/pipelines//.logs` folder.
Learn more about logs [here](/design/core-abstractions#log).
### Set logging level
The default logging level is `INFO`. To customize the logging level for logs of block runs and pipeline runs,
you can set the `level` of `logging_config` in your project's `metadata.yaml` file.
Example logging config to only log errors:
```yaml theme={"system"}
logging_config:
level: ERROR
```
### Set log retention period
To delete old logs in your persistent volume, you can specify the `retention_period` in the `logging_config`. Valid period units
are 'h', 'd', and 'w'.
Example logging config:
```yaml theme={"system"}
logging_config:
retention_period: '15d'
```
Then you can run command `mage clean-old-logs [project_path]` to clean up old logs
### Block logging
A logger is available in the block's context. You can use it to log messages to the block's log file. The logger is passed into
each block as a keyword argument. You can retrieve the logger by calling `kwargs.get('logger')` within the block.
Example:
```python theme={"system"}
@data_loader
def load_data(*args, **kwargs):
kwarg_logger = kwargs.get('logger')
kwarg_logger.info('Test logger info')
kwarg_logger.warning('Test logger warning')
kwarg_logger.error('Test logger error')
...
```
## Logging to external destination
### S3
To store logs in S3, you need to set the `logging` config in your project's `metadata.yaml` file.
Example S3 logging config:
```yaml theme={"system"}
logging_config:
type: s3
level: INFO
destination_config:
bucket:
prefix:
aws_access_key_id: <(optional) AWS access key ID>
aws_secret_access_key: <(optional) AWS secret access key>
endpoint_url: <(optional) custom endpoint url>
```
To authenticate with S3, the credentials need to be configured in one of the following ways:
* Configure credentials in the `logging_config`.
* Configure credentials in environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
* Authenticate with IAM role. Grant S3 access to the IAM role of the cloud instance.
The `endpoint_url` can be configured to connect to S3 compatible services (e.g. MinIO).
### Google Cloud Storage
To store logs in GCS, you need to set the `logging` config in your project's `metadata.yaml` file.
Example GCS logging config:
```yaml theme={"system"}
logging_config:
type: gcs
level: INFO
destination_config:
path_to_credentials: <(optional) path to gcp credentials json file>
bucket:
prefix:
```
*More destinations coming...*
***
## Edit pipeline logging
When youโre editing a pipeline (e.g. `/pipelines/[uuid]/edit`), you can execute the code for an
individual block and see the output. Any `print` statements in the block of code is displayed
in the blockโs output.
However, you can redirect those `print` statements to output to logs.
To toggle this feature, go to the pipeline settings page (e.g. `/pipelines/[uuid]/settings`)
and check the box labeled
When running a block while editing a pipeline, output the block messages to the logs.
# Monitoring
Source: https://docs.mage.ai/observability/monitoring
You can monitor many metrics for each of your pipelines and blocks.
Soon, you'll be able to monitor aggregate metrics across all pipelines and
blocks.
## Metrics
Here are the following metrics that are monitored for each pipeline and block:
* Pipeline runs completed successfully
* Pipeline runs failed
* Block runs completed successfully
* Block runs failed
* How long each block ran for
* CPU usage for Kubernetes block and pipeline runs in Mage Pro
* Memory usage for Kubernetes block and pipeline runs in Mage Pro
* *Data validation success for each block (coming soon)*
* *Data validation failure for each block (coming soon)*
For pipelines that run with the Kubernetes executor in Mage Pro, the pipeline
monitoring dashboard can show historical peak CPU and memory usage, sample
counts, and copied resource request and limit recommendations based on stored
run metrics. See [Monitoring with Prometheus](/observability/external/prometheus#kubernetes-resource-usage-in-mage-pro)
to set up Prometheus as the preferred resource usage source.
# Data validation
Source: https://docs.mage.ai/observability/testing/data-validation
Every data loader and transformer block has data validation capabilities built-in.
You can define 1 or more test functions in a single block. Each test function
accepts a data object as an argument.
Within the body of the function, you can write any type of test you want to
validate the input data.
After the blockโs main code is executed, the output data is passed into each
test function for validation. If any tests fail, then the block run will also
fail.
***
### Example
Here is an example of a transformer block with 2 tests:
```python theme={"system"}
from pandas import DataFrame
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
COLUMNS_TO_USE = ['name']
@transformer
def transform_df(df: DataFrame, *args, **kwargs) -> DataFrame:
return df.iloc[:1][COLUMNS_TO_USE]
@test
def test_number_of_rows(df) -> None:
assert len(df.index) >= 2, 'The output has more than 1 row.'
@test
def test_columns(df) -> None:
assert df.columns[0] != COLUMNS_TO_USE[0], 'The output columns donโt match.'
```
You can combine all your data validations into 1 test function or you can
split them up into multiple test functions. The benefit of splitting them up
is that they can run in parallel, speeding up the data validation.
***
### Log output
Each test run is recorded and can be viewed in the logs. Here is an example:
```
Start executing block.
--------------------------------------------------------------
2/2 tests passed.
Finish executing block.
```
***
# Data Quality with Great Expectations
## Setup
1. Before adding expectations to your pipeline, please make sure you have at least 1
data loader, transformer, or data exporter block.
They must be Python blocks (SQL block support coming soon).
If you donโt have any blocks, add a data loader block and paste the following code:
```python theme={"system"}
import io
import pandas as pd
import requests
@data_loader
def load_data_from_api(*args, **kwargs):
url = 'https://raw.githubusercontent.com/mage-ai/datasets/master/restaurant_user_transactions.csv'
response = requests.get(url)
return pd.read_csv(io.StringIO(response.text), sep=',')
```
For more expectations, read Great Expectationโs [documentation](https://greatexpectations.io/expectations/).
2. [Add Great Expectations power up to your pipeline](/design/blocks/extension#how-to-add-extensions-to-your-pipeline).
3. In the list of available power ups, click the option for Great Expectations.
***
## Adding expectations to your pipeline
1. Once youโre on the Great Expectations power up detail page, you can add extension blocks to
the current pipeline by clicking the button labeled \[+ Extension block].
2. In dropdown menu, click the template option labeled Empty template.
3. A popup dialog may appear asking you to enter a name for the new extension block.
If it appears, fill in `test number of rows` as the name and click the button labeled \[Save and add block].
4. Paste the following code in the extension block named `test number of rows`:
```python theme={"system"}
@extension('great_expectations')
def validate(validator, *args, **kwargs):
validator.expect_table_row_count_to_be_between(
min_value=1000,
max_value=10000,
)
```
You can add expectations using code or from a JSON object.
See section [Defining expectations](#defining-expectations) below for more details.
5. In the extension block near the bottom, click the input field that says
โSelect blocks to run expectations onโ.
6. Once you click that input field, a list of blocks from your pipeline will appear.
7. Check the checkbox on the right side of the dropdown to associate that block to this extension block.
8. Click the button labeled Save selected blocks.
9. After you save, a button that is labeled after the name of the block you just selected will appear.
For example, if your block is named `load_api_data_demo`,
then a button labeled load\_api\_data\_demo will appear.
10. Click that button to run your extension block for the block `load_api_data_demo`.
11. The output should look something like this:
```
Calculating Metrics: 0%| | 0/1 [00:00
## Pages with customizable dashboards
1. Project overview page (`/overview?tab=Dashboard`)
2. Pipeline dashboard page (`/pipelines/[:id]/dashboard`)
***
## Adding charts
1. In the top right corner of the dashboard, click the button labeled
+ Add content.
2. Click the dropdown option labeled Create new chart.
***
## Configuring chart
### Chart name
Human readable name for your chart.
### Chart type
Choose from a variety of visualizations.
#### Bar chart
#### Histogram
#### Line chart
#### Pie chart
#### Table
#### Time series bar chart
#### Time series line chart
### Data source
A chart has at least 1 data source.
In the near future, a chart can have more than 1 data source.
#### Block data output (type)
Get the output data from a block in a specific pipeline.
| Field | Description | Required | Example |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------ |
| Pipeline UUID | The pipeline UUID the block belongs to. | โ | `example_pipeline` |
| Block UUID | The block UUID you want data from. | โ | `load_api_data` |
| Partitions | Enter a positive or a negative number. If positive, then data from the block will be the most recent N partitions. If negative, then data from the block will be the oldest N partitions. | | `3` |
#### Block runs (type)
Get data from all block runs or
all block runs from a specific pipeline.
#### Custom code (type)
Define a function using the `@data_source` decorator that returns the data
for your chart.
```python theme={"system"}
@data_source
def d(*args, **kwargs):
return [[dict(id=1), dict(id=2)]]
```
#### Pipelines (type)
Get data from all pipelines.
#### Pipeline runs (type)
Get data from all pipeline runs or
all pipeline runs from a specific pipeline or
all pipeline runs from a specific trigger.
#### Triggers (type)
Get data from all triggers or
all triggers from a specific pipeline.
#### Refresh interval
How frequent do you want this chart to automatically fetch new data from its data source?
Enter a number in milliseconds (e.g. 1000ms is 1 second).
### Chart display settings
#### Group by columns
Select the column to group your data by.
#### Metrics
| Field | Description | Required | Example |
| ----------- | ----------------------------------------------------------------------------------------------------- | -------- | --------- |
| Aggregation | After grouping the data by 1 or more columns, perform the following aggregation function on a column. | โ | Average |
| Column | Select the column to perform the aggregation function on. | โ | `country` |
#### Chart style
Some charts have multiple styles of the same visualization.
#### Sort direction
Some charts can sort the visualizations in ascending or descending order.
#### Number of buckets or slices
You can limit the number of buckets or slices that are displayed for some charts.
#### Time column
For the time series charts, you must choose which column contains the date or time
value to group the data by.
#### Time interval
For the time series charts, you can choose how granular the interval is between each
date or time values in the time column. For example, you can display your time series
data on a daily basis or annual basis.
***
## Custom code for chart
The following decorators can optionally be used to further customize your chart.
### `@data_source`
If the data source type is `Custom code`, then this functionโs return value is the
data that the chart will display.
However, if the data source type is anything else besides `Custom code`,
then this function can be used for post processing of the data from the selected
data source.
For example, letโs say the data source type is `Block data output` and
the number of partitions is `2`. The chart will retrieve the 2 most recent partitions of data from
that block.
In the `@data_source` decorated function, you can write code to operate on 2 of
those block output data:
```python theme={"system"}
from typing import List
@data_source
def filter_data(block_data_output: List, **kwargs) -> List:
df_partition_1, df_partition_2 = block_data_output
df_1 = df_partition_1[df_partition_1['Age'] >= 21]
df_2 = df_partition_2[df_partition_2['Age'] >= 21]
return [df_1, df_2]
```
### `@x`
Write custom code to return the `x` values for your chart.
X values typically contain the column that you group your data on.
For example, group pipelines by their type.
```python theme={"system"}
from typing import List
@x
def some_columns(data_from_data_source, **kwargs) -> List:
df_partition_1 = block_data_output[0]
return df_partition_1.columns[:3]
```
### `@y`
Write custom code to return the `y` values for your chart.
Y values typically contain the value for a group of rows.
For example, the number of pipelines that are data integration pipelines.
```python theme={"system"}
from typing import List
@y
def number_of_data_integration_pipelines(data_from_data_source, **kwargs) -> List:
df_partition_1 = block_data_output[0]
count = len(df_partition_1[df_partition_1['type'] == 'data_integration'].index)
return [count]
```
### `@xy`
This function combines the purpose of `@x` and `@y` into 1 function.
Use this function if you want to custom both the `x` and `y` values and you want to share the
same variables defined within the function.
The difference is that you must return a tuple where
the `x` values are 1st and the `y` values are 2nd.
```python theme={"system"}
from typing import List, Tuple
@xy
def columns_and_numbers(data_from_data_source, **kwargs) -> Tuple[List]:
df_partition_1 = block_data_output[0]
count = len(df_partition_1[df_partition_1['type'] == 'data_integration'].index)
return df_partition_1.columns[:1], [count]
```
### `@configuration`
Each chart has a configuration dictionary that stores the chart settings.
These chart settings are stored in a YAML file.
However, if you want to encode the configurations into the chart file and reuse the configurations
in other dashboards, you can write a function that returns a dictionary that will be used
to configure the chart.
```python theme={"system"}
from typing import Dict
@configuration
def chart_settings(*args, **kwargs) -> Dict:
return dict(
chart_type='time series line chart',
group_by=['execution_date'],
metrics=[dict(
aggregation='count_distinct',
column='id',
)],
time_interval='day',
)
```
### `@columns`
In the chart settings, when choosing the group by columns, number columns, time column, metrics,
etc., the chart must know about what available columns are selectable.
Typically, the chart will attempt to infer the columns from the data source.
However, you can customize the logic of how the columns are inferred or
which columns are available to select.
```python theme={"system"}
from typing import List
@columns
def some_columns(data_from_data_source, **kwargs) -> List:
df_partition_1 = block_data_output[0]
return df_partition_1.columns[:3]
```
### `@render`
Define a function that returns a Base64 string representation of a JPEG/JPG image,
PNG image, or a complete HTML string (with html and body tags).
#### Render JPEG or JPG
```python theme={"system"}
import base64
import io
import matplotlib.pyplot as plt
# render_type can be 'jpeg' or 'jpg'
@render(render_type='jpeg')
def r(*args, **kwargs):
# creating the dataset
data = {'C':20, 'C++':15, 'Java':30,
'Python':35}
courses = list(data.keys())
values = list(data.values())
fig = plt.figure(figsize = (10, 5))
# creating the bar plot
plt.bar(courses, values, color ='maroon',
width = 0.4)
plt.xlabel("Courses offered")
plt.ylabel("No. of students enrolled")
plt.title("Students enrolled in different courses")
plt.show()
my_stringIObytes = io.BytesIO()
plt.savefig(my_stringIObytes, format='jpg')
my_stringIObytes.seek(0)
my_base64_jpgData = base64.b64encode(my_stringIObytes.read()).decode()
plt.close()
return my_base64_jpgData
```
#### Render PNG
```python theme={"system"}
import base64
import io
import matplotlib.pyplot as plt
import numpy as np
@render(render_type='png')
def r(*args, **kwargs):
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t, s)
plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('About as simple as it gets, folks')
plt.grid(True)
plt.show()
my_stringIObytes = io.BytesIO()
plt.savefig(my_stringIObytes, format='png')
my_stringIObytes.seek(0)
my_base64_jpgData = base64.b64encode(my_stringIObytes.read()).decode()
plt.close()
return my_base64_jpgData
```
#### Render HTML
```python theme={"system"}
import plotly.graph_objects as go
@render(render_type='html')
def r(*args, **kwargs):
fig = go.Figure(
data=[go.Bar(y=[2, 1, 3])],
layout_title_text="A Figure Displaying Itself"
)
return fig.to_html(full_html=False)
```
***
## Recommended charts
Depending on which page youโre on, adding recommended charts will automatically configure
and add a few charts to your dashboard with little effort.
The added charts will show visualizations with different chart types and various data sources.
***
## Resize chart
When you resize a chart on the dashboard, you can change the width and height.
### Width
Set the width to a number.
This number is divided by the sum of the width numbers from all the charts in the same row.
The resulting number is the percentage of the row the chart will expand horizontally to.
For example, if you have a row with 5 charts and the 1st chart has a width of 6 and the other
4 charts have a width of 1, then the 1st chart will take up 60% of the entire rowโs width.
### Max width percentage
Even if a chart is to take up a large percentage of the rowโs entire width,
you can set a limit as to how wide the chart should be.
For example, in the above example the 1st chart should be 60% of the entire rowโs width.
However, you can set that chartโs max width percentage to 50 and the 1st chart will only
expand to 50% of the rowโs entire width.
### Height
Set this value to the number of pixels you want your chart to have in height.
***
## Move chart
Click and hold down on a chart to begin dragging it. Then, drag the chart to move it and
then release the click while on top of another chart to insert the initial chart after the
chart that you dropped it on.
***
## Remove chart
This will remove the chart from the current dashboard.
# Backfill guide
Source: https://docs.mage.ai/orchestration/backfills/guides
How to create a backfill.
1. From the pipelines list page, click a pipeline to go to the pipeline detail page.
2. In the vertical navigation bar on the left side of the screen, click the link for Backfills.
3. Click the button labeled + Create new backfill.
4. Under the section labeled Backfill type, select Date and time window card.
5. Enter a name for your backfill.
6. Select a start date and time.
7. Select an end date and time.
8. Select an interval type; this value will help determine how many pipeline runs to create.
9. Enter an interval unit; this value is multiplied by the interval type to determine how many pipeline runs to create.
10. Click the button labeled Save changes.
11. Once the backfill is successful saved, click the button labeled Start backfill.
Check out this [blog article](https://www.mage.ai/blog/backfilling-historical-data-using-mage) for more details!
# Backfilling pipelines
Source: https://docs.mage.ai/orchestration/backfills/overview
Run a pipeline multiple times.
A backfill creates 1 or more pipeline runs for a pipeline. There are 2 types of backfills:
1. Date and time window
2. Custom code
## Date and time window
Create 1 or more pipeline runs between 2 datetime values.
The datetime of each instance is used as the `execution_date` for the pipeline run.
For example, if the backfill has the following attributes:
| Attribute | Value |
| -------------- | --------------------- |
| Start datetime | `2023-01-01T03:00:00` |
| End datetime | `2023-01-05T03:00:00` |
| Interval type | `day` |
| Interval units | `2` |
Then the following pipeline runs will be created:
| `id` | `execution_date` | `ds` | `hr` |
| ---- | --------------------- | ------------ | ---- |
| 1 | `2023-01-01T03:00:00` | `2023-01-01` | `03` |
| 2 | `2023-01-03T03:00:00` | `2023-01-03` | `03` |
***
## Custom code
The output of a backfill code will be used to generate the pipeline runs.
For example, if the backfill code has the following content:
```python theme={"system"}
backfill_data = []
for index, _ in range(3):
backfill_data.append(dict(
partition=index,
power=5,
))
backfill_data
```
Then the following pipeline runs will be created:
| `id` | `execution_date` | `ds` | `hr` | `partition` | `power` |
| ---- | --------------------- | ------------ | ---- | ----------- | ------- |
| 1 | `2023-01-01T00:00:00` | `2023-01-01` | `00` | `0` | `5` |
| 2 | `2023-01-01T00:00:00` | `2023-01-01` | `00` | `1` | `5` |
| 3 | `2023-01-01T00:00:00` | `2023-01-01` | `00` | `2` | `5` |
***
## Extra runtime variables from pipeline run
If your pipeline run belongs to a trigger that is scheduled, then the following extra
variables are available in your Python blockโs keyword arguments (e.g. `kwargs`).
| Key | Description | Example |
| ---------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------- |
| `interval_start_datetime` | The `datetime` when the pipeline run is scheduled for. | `datetime.datetime(2023, 7, 23, 7, 0, 0, 0)` |
| `interval_end_datetime` | The `datetime` when the next pipeline run is scheduled for. | `datetime.datetime(2023, 7, 24, 7, 0, 0, 0)` |
| `interval_seconds` | The number of seconds between the current pipeline run and the next pipeline run. | `86400` |
| `interval_start_datetime_previous` | The `datetime` when the previous pipeline run was scheduled for. | `datetime.datetime(2023, 7, 22, 7, 0, 0, 0)` |
### SQL block
If youโre using a SQL block, here is an example of how you can access these variables:
```sql theme={"system"}
SELECT
'{{ interval_start_datetime }}' AS interval_start_datetime
, '{{ interval_end_datetime }}' AS interval_end_datetime
, '{{ interval_seconds }}' AS interval_seconds
, '{{ interval_start_datetime_previous }}' AS interval_start_datetime_previous
```
# Query pipeline run metadata from database
Source: https://docs.mage.ai/orchestration/pipeline-runs/database-query
Pipeline runs and Block runs data are stored in `pipeline_run` and `block_run` tables in the database.
If you want to query the `PipelineRun` and `BlockRun` metadata, you can either directly query the data
from database or use Python code to fetch `PipelineRun` and `BlockRun` models.
## Query from DB
Here is the query from Postgres. You can adapt it to the syntax of other databases when needed.
```sql theme={"system"}
SELECT
*
FROM pipeline_run
WHERE pipeline_uuid = 'example_pipeline' -- Filter by pipeline_uuid
AND status = 'COMPLETED' -- Filter by status
```
## Query DB models via Python
```python theme={"system"}
from mage_ai.orchestration.db import db_connection
from mage_ai.orchestration.db.models.schedules import PipelineRun, BlockRun
db_connection.start_session()
# Fetch running pipeline runs of a pipeline
pipeline_runs = PipelineRun.query.filter(
# Filter by pipeline uuid
PipelineRun.pipeline_uuid == 'example_pipeline',
# Filter by status
PipelineRun.status == PipelineRun.PipelineRunStatus.RUNNING,
# Filter by execution_date
PipelineRun.execution_date >= '2024-01-01',
PipelineRun.execution_date < '2024-02-01',
)
# Get the block runs of a pipeline run
block_runs = pipeline_runs.all()[0].block_runs
print(f'Block runs: {[b.to_dict() for b in block_runs]}')
# Get pipeline run count
print(f'Number of pipeline runs {pipeline_runs.count()}')
# Batch update pipeline run status
PipelineRun.batch_update_status([p.id for p in pipeline_runs], PipelineRun.PipelineRunStatus.CANCELLED)
```
# Pipeline Run
Source: https://docs.mage.ai/orchestration/pipeline-runs/overview
Learn how to monitor, manage, and troubleshoot pipeline runs in Mage for optimal data pipeline performance.
## Overview
Pipeline runs represent **individual executions of your data pipelines** in Mage. Each time a pipeline is triggered (manually or automatically), a pipeline run record is created that tracks the entire execution process, including all block runs, status updates, timing information, and any errors that occur.
### Key Benefits of Pipeline Run Management:
* **Execution tracking** - Monitor the status and progress of every pipeline execution
* **Performance insights** - Analyze execution times, resource usage, and bottlenecks
* **Error debugging** - Detailed logs and error information for troubleshooting
* **Historical analysis** - Review past executions to identify patterns and issues
* **Retry capabilities** - Restart failed or incomplete pipeline runs
* **Audit trail** - Complete record of when and how pipelines were executed
**Pipeline Run** and **Pipeline Execution** are the same thing in Mage. You may see either term used throughout the product.
## Pipeline Run Lifecycle
### Run Statuses
Pipeline runs progress through several statuses during their lifecycle:
* **`initial`** - Pipeline run has been created but not yet started
* **`running`** - Pipeline is currently executing
* **`completed`** - Pipeline finished successfully
* **`failed`** - Pipeline encountered an error and stopped
* **`cancelled`** - Pipeline run was manually cancelled or timed out
### Block Run Statuses
Within each pipeline run, individual blocks have their own statuses:
* **`initial`** - Block run created but not started
* **`queued`** - Block run is in the queue and waiting to be executed
* **`running`** - Block run is currently executing
* **`completed`** - Block run finished successfully
* **`failed`** - Block run encountered an error
* **`cancelled`** - Block run was cancelled
* **`upstream_failed`** - Block cannot run because upstream blocks failed
* **`condition_failed`** - The block was skipped because its own or an upstream block's condition was not satisfied.
## Viewing Pipeline Runs
### Project-Wide Pipeline Runs Dashboard
View all pipeline runs across your entire project:
1. **Navigate to the homepage** and click **"Pipeline runs"** in the left navigation
2. **Or go directly to** `/pipeline-runs` URL
3. **Filter** by pipeline, status, or date range, and **search** by pipeline or trigger
4. **Monitor key metrics** like success rate, average duration, and recent activity
#### View Types
**Current and Past Runs:**
* View all completed, failed, cancelled, and currently running pipeline runs
* Available in both Mage OSS and Mage Pro
* Shows historical execution data and current status
**Upcoming Runs (Mage Pro only):**
* View scheduled pipeline runs that are yet to execute
* See future run dates and trigger information
* Monitor upcoming pipeline executions and scheduling
#### Pipeline Run List Filters
The pipeline run list page provides comprehensive filtering options:
**Available Filters:**
* **Pipeline UUID** - Filter by specific pipeline identifiers
* **Status** - Filter by run status (Cancelled, Done, Failed, Last run failed, Ready, Running)
* **Pipeline Tag** - Filter by pipeline tags
* **Date range** - Filter current and past runs by bounded UTC preset ranges (Last hour, Last day, Last week, Last 30 days, Last year), all time, or a custom UTC start and end date
* **Search** - Text search by pipeline or trigger (Mage Pro only)
**Filter Actions:**
* **Apply filters** - Apply selected filter criteria
* **Reset filters** - Clear all filters and return to defaults
**Mage OSS vs Pro**:
* **Mage OSS**: Supports viewing current and past runs with basic filtering (Pipeline UUID, Status, Pipeline Tag)
* **Mage Pro**: Supports both current/past runs AND upcoming runs, plus advanced filtering including text search and monitoring features
The project-wide dashboard shows runs from all pipelines, making it easy to monitor your entire data infrastructure at a glance.
### Pipeline-Specific Run History
View runs for a specific pipeline:
1. **Click on a pipeline** from the main Pipelines Dashboard
2. **Navigate to the "Runs" section** or click on a trigger from the "Triggers" section
3. **View detailed run information** including block runs, logs, and metrics
4. **Access individual run details** by clicking on specific runs
**Direct URL**: `/pipelines/[pipeline_uuid]/runs`
#### Batch Operations
The pipeline-specific run page supports batch operations for managing multiple runs:
**Available Batch Actions:**
* **Retry selected** - Retry specific selected pipeline runs
* **Retry all incomplete block runs** - Retry all incomplete block runs across the pipeline
* **Cancel selected running** - Cancel specific selected running pipeline runs
* **Cancel all running** - Cancel all currently running pipeline runs
**How to Use Batch Operations:**
1. **Select pipeline runs** by checking the boxes next to the runs you want to manage
2. **Click the "Actions" dropdown** to access batch operation options
3. **Choose your desired action** from the dropdown menu
4. **Confirm the operation** when prompted
Batch operations are particularly useful for managing multiple failed runs or stopping multiple running pipelines at once, improving operational efficiency.
### Individual Pipeline Run Details
For detailed analysis of a specific run:
1. **Click on a pipeline run** from the runs list
2. **View the individual run page** at `/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`
3. **Examine block run details** including execution order, timing, and status
4. **Access logs and error information** for each block
5. **Review runtime variables** and configuration used
6. **Review Kubernetes resource usage** in the **Resources** tab when stored metrics are available for Mage Pro runs that use the Kubernetes executor
## Pipeline Run Retry
### Retrying Failed Runs
When a pipeline run fails, you can retry it in several ways:
For detailed information about retry configuration and management, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).
#### Retry Incomplete Blocks
Retry from the failed block instead of restarting the entire pipeline. For detailed instructions, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).
#### Retry from Selected Block
Retry starting from any selected block regardless of its current status. For detailed instructions, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).
### Automatic Retry Configuration
Configure automatic retries to handle transient failures. For detailed retry configuration examples and advanced settings, see [Retrying block runs from a pipeline run](/orchestration/pipeline-runs/retrying-block-runs).
## Pipeline Run Data and Monitoring
### Key Metrics and Information
Each pipeline run contains valuable information:
* **Execution timing** - Start time, completion time, and duration
* **Status tracking** - Current status and any status changes
* **Block run details** - Individual block execution information
* **Runtime variables** - Variables used during execution
* **Error information** - Detailed error messages and stack traces
* **Resource usage** - CPU, memory, and other resource metrics
* **SLA compliance** - Whether the run met configured SLAs
### Database Storage
Pipeline run data is stored in database tables:
* **`pipeline_run`** - Main pipeline run records
* **`block_run`** - Individual block run records within each pipeline run
You can query pipeline run data directly from the database or use Mage's Python API to access run information programmatically. For detailed database querying examples, see [Query pipeline run metadata from database](/orchestration/pipeline-runs/database-query). For API reference documentation, see [Pipeline runs API reference](/extensibility/api-reference/pipeline-runs/overview).
### Querying Pipeline Run Data
Query pipeline run data directly from the database or use Mage's Python API. For comprehensive database querying examples and Python API usage, see [Query pipeline run metadata from database](/orchestration/pipeline-runs/database-query).
## Advanced Pipeline Run Features
### Runtime Variables
Override pipeline variables for specific runs:
* **Trigger-level variables** - Set variables when creating or editing triggers (see [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines))
* **API-triggered variables** - Pass variables via API requests (see [Trigger pipeline via API](/orchestration/triggers/trigger-pipeline-api))
* **Manual run variables** - Override variables when manually triggering runs
Runtime variables allow you to customize pipeline behavior for different environments, data sources, or processing requirements without modifying the pipeline code. Learn more about [using runtime variables](/getting-started/runtime-variable).
### SLA Monitoring
Configure and monitor Service Level Agreements. For detailed SLA configuration, see [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines#sla-configuration).
* **Expected completion time** - Set target completion times for pipeline runs
* **Time units** - Configure in minutes, hours, or days
* **SLA tracking** - Monitor whether runs meet their SLA targets
* **Alerting** - Get notified when SLAs are missed
### Timeout Configuration
Control pipeline run execution time. For detailed timeout configuration, see [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines#timeout-configuration).
* **Timeout duration** - Set maximum execution time in seconds
* **Timeout status** - Choose how to handle runs that exceed timeout (failed, cancelled, warning)
* **Resource protection** - Prevent runaway pipelines from consuming excessive resources
## Debugging Pipeline Run Failures
When pipeline runs fail, use these debugging techniques to identify and resolve issues:
### Accessing Run Information
**Individual Run Details:**
1. **Navigate to the failed run** in the pipeline runs list
2. **Click on the run** to view detailed information
3. **Examine the execution timeline** to see where the failure occurred
4. **Review block run statuses** to identify which blocks failed
**Block-Level Debugging:**
1. **Click on individual block runs** to see detailed information
2. **Access block logs** for error messages and stack traces
3. **Open the log source** from the selected error log's source-location link, or click the block name in the Block column, to open the failing block in the pipeline editor. When line metadata is available, Mage focuses that line in the editor.
4. **Review block output** to understand data state at failure
5. **Check upstream dependencies** to ensure data flow is correct
### Common Failure Patterns
**Upstream Block Failures:**
* **Issue**: Downstream blocks show `upstream_failed` status
* **Solution**: Fix the upstream block that failed first, then retry
**Data Validation Errors:**
* **Issue**: Blocks fail due to unexpected data format or missing columns
* **Solution**: Add data validation and error handling in your blocks
**Resource Exhaustion:**
* **Issue**: Pipeline runs timeout or consume excessive resources
* **Solution**: Optimize block code, increase timeout settings, or break into smaller pipelines
**External Dependency Failures:**
* **Issue**: API calls, database connections, or file access failures
* **Solution**: Implement retry logic and proper error handling
**Pipeline Runs Stuck in Initial/Queued Status:**
* **Issue**: Pipeline runs remain in `initial` or `queued` status with no visible errors in logs
* **Solution**: Follow these troubleshooting steps:
1. **Check concurrency limits** - Verify that block run concurrency or pipeline run concurrency limit is not set to 0
2. **Check resource usage** - Monitor CPU and RAM usage to ensure sufficient resources are allocated
3. **Check server error logs** - Review server-side logs for underlying issues
4. **Restart the server** - Try restarting the Mage server to clear any stuck processes
5. **Contact Mage team** - If the issue persists, reach out to the Mage team for debugging assistance at [mage.ai/chat](https://mage.ai/chat)
### Debugging Tools
* **Detailed logs** - Access comprehensive execution logs for each block
* **Error tracking** - View error messages, stack traces, and debugging information
* **Execution timeline** - See the order and timing of block executions
* **Variable inspection** - Review runtime variables and their values
* **Kubernetes resource summary** - In Mage Pro, review stored CPU and memory usage plus copied block-level executor recommendations for runs that use the Kubernetes executor
* **Block output export** - Save block output as CSV for analysis (see [Saving block output as CSV](/orchestration/pipeline-runs/saving-block-output-as-csv))
## Best Practices for Pipeline Run Management
### Monitoring and Alerting
* **Set up monitoring** for key pipeline runs and their success rates
* **Configure alerts** for failed runs, SLA violations, and performance issues
* **Regular review** of run history to identify patterns and improvements
### Error Handling
* **Implement retry logic** for transient failures
* **Use proper error handling** in your block code
* **Validate data** at each step to catch issues early
* **Test thoroughly** before deploying to production
### Performance Optimization
* **Monitor execution times** and optimize slow blocks
* **Use appropriate resources** for your workload
* **Implement caching** where appropriate
* **Consider parallel processing** for independent operations
* **Configure concurrency limits** to control resource usage (see [Concurrency](/design/data-pipeline-management#concurrency))
### Documentation and Maintenance
* **Document pipeline purposes** and expected behavior
* **Maintain run history** for audit and compliance purposes
* **Regular cleanup** of old run data to manage storage
* **Version control** your pipeline code and configurations
## Frequently Asked Questions
You can retrieve the output of a block run by calling the block run output API. For example:
```
{MAGE_HOST}/api/block_runs//outputs?sample_count=100
```
**Steps to get block run ID:**
1. Trigger your pipeline via API or UI
2. Navigate to the pipeline run details page
3. Find the specific block run ID from the block runs list
4. Use the API endpoint above with the block run ID
You can find the relevant API calls in your browser console when using the Mage UI. More details are available in the [API reference guide](/extensibility/api-reference/pipeline-runs/overview).
Yes, you can monitor pipeline run status programmatically using multiple methods:
**API Methods:**
* Use the Mage API to query pipeline run status and outputs
* Access individual pipeline run details via `/api/pipeline_runs/`
* Monitor block run statuses within a pipeline run
**Database Query Methods:**
* Query the `pipeline_run` and `block_run` tables directly
* Use Python scripts to filter by status, execution date, or pipeline UUID
* Batch update pipeline run statuses programmatically
See [Query pipeline run metadata from database](/orchestration/pipeline-runs/database-query) for detailed examples and [API reference guide](/extensibility/api-reference/pipeline-runs/overview) for API documentation.
You can cancel all running pipeline runs using either method:
**UI Method:**
* Navigate to `/pipelines/[pipeline_uuid]/runs` and click **"Cancel all running"** button
* This cancels all running pipeline runs for a specific pipeline
**Programmatic Method:**
* Write a Python script to cancel all ready or running pipeline runs
* Example code and guidance are available in the [database query documentation](/orchestration/pipeline-runs/database-query)
Runtime variables are managed at multiple levels in pipeline runs:
**Variable Storage:**
* Run records store information about runtime variables used during execution
* Variables are typically specified at the time of pipeline trigger
* Variables can be overridden at the trigger level for different environments
**Variable Sources:**
* **Trigger-level variables** - Set when creating or editing triggers
* **API-triggered variables** - Passed via API requests in the payload
* **Manual run variables** - Overridden when manually triggering runs
**Variable Usage:**
* Access variables in blocks using `kwargs.get("variable_name", "default_value")`
* Variables allow customization for different environments without code changes
* Runtime variables are stored in the `variables` field of pipeline run records
Learn more about [using runtime variables](/getting-started/runtime-variable) and [trigger configuration](/orchestration/triggers/triggering-pipelines).
Pipeline runs can get stuck in `initial` or `queued` status due to several common issues:
**Common Causes:**
* **Concurrency limits** - Block run concurrency or pipeline run concurrency limit set to 0
* **Resource constraints** - Insufficient CPU or RAM allocation
* **Executor issues** - Problems with k8s\_executor, volume mounting, or resource constraints
* **Server issues** - Mage server experiencing problems or stuck processes
**Troubleshooting Steps:**
1. **Check concurrency limits** - Verify concurrency settings are not set to 0
2. **Monitor resource usage** - Ensure sufficient CPU and RAM are allocated
3. **Check server error logs** - Review server-side logs for underlying issues
4. **Restart the server** - Try restarting Mage to clear stuck processes
5. **Switch executors** - Try local\_python executor for testing to isolate issues
**Additional Debugging:**
* If logs are not visible in the UI, check log files directly in the log path
* Use the debugging tools available in the pipeline run details page
* Contact the Mage team at [mage.ai/chat](https://mage.ai/chat) for persistent issues
You can set up automated pipeline runs using scheduled triggers in Mage:
**Creating Scheduled Triggers:**
1. **Navigate to pipeline** - Go to your pipeline's trigger page
2. **Create new trigger** - Click "New trigger" and select "Schedule"
3. **Configure frequency** - Choose from hourly, daily, weekly, monthly, or custom cron expressions
4. **Set additional options** - Configure SLA, timeout, runtime variables, and other settings
5. **Save trigger** - Enable the trigger to start automated runs
**Available Frequencies:**
* **Run exactly once** - Execute once and disable
* **Hourly/Daily/Weekly/Monthly** - Standard recurring schedules
* **Always on** - Immediately create new runs when previous completes
* **Custom cron** - Advanced scheduling with cron expressions
**Requirements:**
* **Docker deployments** - Container must remain running for scheduled triggers to work
* **SaaS version** - Mage offers a managed service that handles scheduling automatically
**Advanced Configuration:**
* Set runtime variables for different environments
* Configure SLA monitoring and timeout settings
* Override global variables at trigger level
* Use "Run once" button for testing triggers
See [Pipeline Trigger Overview](/orchestration/triggers/triggering-pipelines) for detailed configuration and [Trigger pipeline via API](/orchestration/triggers/trigger-pipeline-api) for API-based triggering.
**Need more help?** If your question isn't answered here, join the Mage community on [Slack](https://mage.ai/chat) to get help from other users and the Mage team.
# Retrying block runs from a pipeline run
Source: https://docs.mage.ai/orchestration/pipeline-runs/retrying-block-runs
Requires version `0.8.76` or greater.
## Automatic retry
Mage supports configuring automatic retry for block runs with the following ways
### 1. Global retry config
Add `retry_config` to project's `metadata.yaml`. This `retry_config` will be applied to all block runs.
Here is an example:
```yaml theme={"system"}
retry_config:
# Number of retry times
retries: 3
# Initial delay (in seconds) before retry. If exponential_backoff is true,
# the delay time is multiplied by 2 for the next retry
delay: 5
# Maximum time between the first attempt and the last retry
max_delay: 60
# Whether to use exponential backoff retry
exponential_backoff: true
```
### 2. Block level retry config
Add `retry_config` to the block config in pipeline's `metadata.yaml`. The block level `retry_config` will
override the global `retry_config`. Here is an example
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
retry_config:
retries: 2
delay: 5
max_delay: 10
exponential_backoff: false
...
```
### 3. Pipeline level `retry_config`
Add `retry_config` to the pipeline config in pipeline's `metadata.yaml`. The pipeline level `retry_config` will
override the global `retry_config`. Here is an example
```yaml theme={"system"}
blocks:
- ...
- ...
- ...
retry_config:
retries: 2
delay: 5
max_delay: 10
exponential_backoff: false
...
```
The retry logs show in the same block run log page.
## Retry incomplete blocks
* Supported pipeline types: Integration and Standard pipelines
* If a pipeline run fails, you can retry from the failed block
run (instead of retrying the entire pipeline run) by going to the individual
pipeline runs page (`/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`).
* At the top of the individual pipeline runs page, there should be a red
`Retry incomplete blocks` button. Click on it, and the block runs should
retry from the failed block.
## Retry from selected block
* Supported pipeline types: Integration and Standard pipelines
* If you want to retry block runs for a pipeline run starting from a specific
block (regardless of the block run's status), go to the same page for the individual
pipeline run.
* Select a block run row, and a blue `Retry from selected block` button should appear.
* Note that the `Retry from selected block` button will only appear if the pipeline
run is not currently running.
* If there is a failed block run and you try to retry from a selected block that is
downstream from that failed block, you may get `upstream_failed` block run errors.
### Accessing the individual pipeline runs page
1. Click on a pipeline from the main Pipelines Dashboard.
2. Click on the `Runs` section or on an individual trigger from the `Triggers` section.
3. Underneath the `Block runs` column of the pipeline runs table, there is a link that
says something like `See block runs (x)`. Click on that link to redirect to the
individual pipeline runs page, which lists the blocks runs specific to a pipeline run.
# Saving block output as CSV file
Source: https://docs.mage.ai/orchestration/pipeline-runs/saving-block-output-as-csv
Requires version `0.8.92` or greater.
For standard pipelines (not currently supported in integration or streaming
pipelines), you can save the output of a block that has been successfully
run and saved to disk as a CSV file.
Note that there may be limitations for saving very
large block outputs (e.g. greater than 200 mb). Downloading block output
saved to the cloud (e.g. S3) is not currently supported.
There are two places to save the block output as a CSV file:
1. In the Pipeline Editor page after successfully running a block
* After a block has been successfully run, click the `Save` icon
in the block output footer.
2. In the Block Runs table after a block run has successfully completed
* The Block Runs table can be found by going to a pipeline's `Runs`
section (found in left navigation sidebar) of the Pipeline Detail page
and clicking the `Block runs` tab or clicking the `X / X`
link under the `Block runs` column of the Pipeline Runs table.
* Go to the far right of the Block Runs table and click the `Save` button
under the `Output` column.
# Configure triggers in code
Source: https://docs.mage.ai/orchestration/triggers/configure-triggers-in-code
Configure triggers in triggers.yaml under pipeline folder.
In addition to configuring triggers via UI, you can also configure triggers in code.
You can either manually define the triggers in code or you can go to the triggers detail page in the
UI and click the button labeled Save trigger in code on the left side panel to store the
current triggerโs settings to code.
## How triggers in code are synced to the database and UI
* **Source of truth:** The `triggers.yaml` file is the source of truth for trigger definitions.
* **Sync flow:** A background scheduler process periodically syncs triggers from `triggers.yaml` into the database. Only triggers that pass the **envs filter** (see [Envs filter](#envs-filter)) are created or updated on that instance. After a trigger is synced to the database, it appears in the Triggers UI.
* **New/updates flow both ways:** New or updated triggers in the YAML are pushed to the database when the scheduler runs. Edits you make in the UI to a trigger that's defined in code are written back to `triggers.yaml` so UI and code stay in sync.
* **Deletions are not synced:** Removing a trigger from `triggers.yaml` does **not** remove it from the database. You must delete the trigger in the UI (or via API) to remove it from the database. See [Delete triggers](#delete-triggers).
## Create and configure triggers
Here are the steps to create and configure triggers via code:
1. Create a `triggers.yaml` file under your pipeline folder. The file path should be `pipelines/[pipeline_uuid]/triggers.yaml`.
2. Enter your trigger configs into the `triggers.yaml` file.
1. Content structure
```yaml theme={"system"}
triggers:
- name: test_trigger
schedule_type: time
start_time: 2023-01-01
status: active
- name: xxx
...
- name: xxx
...
```
2. **Required trigger fields**
* `name`: Unique identifier of the trigger (per pipeline).
* `schedule_type`: `time`, `api`, or `event`. Determines how the trigger runs (schedule vs API vs event).
* For **time (schedule)** triggers: `schedule_interval` and `start_time` are required.
* `schedule_interval`: `@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`, `@always_on`, or a Cron expression (e.g. `* * * * *`).
* `start_time`: When the schedule starts (e.g. `2023-01-01` or `2023-01-01 00:00:00`).
* For **api** and **event** triggers: `schedule_interval` and `start_time` are typically not used for execution but may still be present in the config.
3. **Optional trigger fields**
* `status`: `active` or `inactive`. Defaults to `inactive` if omitted.
* `variables`: A dictionary of [runtime variables](/getting-started/runtime-variable) for this trigger.
* `sla`: SLA in seconds for this trigger.
* `settings`: A dictionary of [advanced settings](/orchestration/triggers/triggering-pipelines#additional-trigger-settings):
* `skip_if_previous_running`: `true`/`false` โ do not start a new run if the previous run is still in progress.
* `allow_blocks_to_fail`: `true`/`false` โ allow the pipeline to continue running non-dependent blocks if some blocks fail.
* `create_initial_pipeline_run`: `true`/`false` โ create an initial run when the start time is in the past (schedule triggers).
* `envs`: List of environments where this trigger should be synced and run. See [Envs filter](#envs-filter) below.
3. Save the `triggers.yaml` file. On the next scheduler run, trigger configs are synced to the database (subject to the envs filter) and then show in the Triggers UI.
1. Required trigger fields
1. `name`: Unique identifier of the trigger
2. `schedule_type`: `time`, `api`, or `event`
3. `schedule_interval`: `@once`, `@hourly`, `@daily`, `@weekly`, `@monthly`, `@always_on`, or Cron syntax(`* * * * *`)
4. `start_time`: The start time of the trigger (e.g. `2023-01-01`)
2. Optional trigger fields
1. `status`: `active` or `inactive`
2. `variables`: A dictionary of variables that's associated with the trigger
3. `sla`: SLA in seconds
4. `settings`: a dictionary of [advanced settings](/orchestration/triggers/triggering-pipelines#additional-trigger-settings)
1. `skip_if_previous_running`: `true`/`false`
2. `allow_blocks_to_fail`: `true`/`false`
3. `create_initial_pipeline_run`: `true`/`false`
5. `envs`: The environments that the trigger runs in.
* If `envs` is not set or empty, the trigger will run in all environments.
* The environment of the Mage app is set via the `ENV` [environment variable](/development/variables/environment-variables).
* **OSS:** Only supports `dev`, `staging`, `prod`, `test`.
* **Mage Pro:** Supports any value, which must match the `ENV` environment variable.
4. Save the `triggers.yaml` file. The trigger configs will be synced to the triggers UI shortly.
### Example triggers config:
```yaml theme={"system"}
triggers:
- name: test_example_trigger
schedule_type: time
schedule_interval: "@daily"
start_time: 2023-01-01
status: active
- name: test_example_trigger_2
schedule_type: time
schedule_interval: "@hourly"
start_time: 2023-03-01
status: inactive
- name: test_example_trigger_with_extra_settings
schedule_type: time
schedule_interval: "@hourly"
start_time: 2023-03-01
status: inactive
settings:
skip_if_previous_running: true
allow_blocks_to_fail: true
envs:
- prod
- dev
```
## Envs filter
The optional `envs` field limits which instances or workspaces sync and run the trigger. Each instance/workspace has an environment set by the `ENV` [environment variable](/development/variables/environment-variables).
* **When `envs` is not set or empty:** The trigger is synced to **all** instances/workspaces. Every environment will get the trigger in the database and can run it.
* **When `envs` is set:** The trigger is synced **only** to instances/workspaces whose `ENV` value is in the `envs` list. Other instances will not create or update this trigger in their database, so they will not run it.
* **OSS:** Allowed values are `dev`, `staging`, `prod`, and `test`.
* **Mage Pro:** Any value is allowed; it must match the `ENV` set for that instance or workspace.
**If you add an `envs` filter to a trigger that already exists everywhere:** Triggers that were previously synced (e.g. when `envs` was empty) remain in the database on instances that no longer match the new `envs` list. The sync process only creates or updates triggers; it does not delete them. To stop such a trigger from existing (and running) on those instances, you must **delete the trigger manually** in the UI on each affected instance or workspace.
## Modify triggers
If an existing trigger is defined in the YAML file, changes flow both ways: updates in the UI are written back to `triggers.yaml`, and updates in the YAML are synced to the database (and reflected in the UI when you reload). Only triggers that pass the [envs filter](#envs-filter) are synced to the database on each instance.
## Delete triggers
The `triggers.yaml` file is synced into the database automatically, but **deletions work differently**:
* **Deleting triggers completely:**
Removing a trigger from the YAML file does not delete it from the database. You must delete the trigger in the UI on each instance where it exists.
* In the Pipeline triggers table (`/pipeline/[pipeline_uuid]/triggers`), click the trash can icon.
* In **Mage Pro**, right-click on the trigger row and click Delete.
This removes the trigger from both the database and the pipelineโs `triggers.yaml` config file. If you added an [envs filter](#envs-filter) and the trigger was already created on other instances, delete it manually on those instances as well; sync does not remove triggers from the database.
* **Alternative to deletion:**
Instead of deleting, you can update the trigger status in the YAML file (e.g. set `status: inactive`).
Removing a trigger from the yaml file wonโt delete it from the database. You will need to delete the trigger in the UI.
* In the Pipeline triggers table (`/pipeline/[pipeline_uuid]/triggers`), click the trash can icon.
* In **Mage Pro**, right-click on the trigger row and click Delete.
This ensures the trigger is removed both from the database and from the pipelineโs `triggers.yaml` config file.
* **Alternative to deletion:**
Instead of deleting, you can update the trigger status in the yaml file (e.g. set `status: inactive`).
Status changes are automatically synced to the database and UI.
# Query trigger metadata from database
Source: https://docs.mage.ai/orchestration/triggers/database-query
Pipeline triggres are stored in `pipeline_schedule` table in the database. If you want to query
the trigger metadata, you can either directly query the data from database or use Python code
to fetch `PipelineSchedule` models.
## Query from DB
Here is the query from Postgres. You can adapt it to the syntax of other databases when needed.
```sql theme={"system"}
SELECT
*
FROM pipeline_schedule
WHERE pipeline_uuid = 'example_pipeline' -- Filter by pipeline_uuid
AND status = 'ACTIVE' -- Filter by status
AND repo_path = '/home/src/default_repo' -- Filter by repo path
```
## Query DB models via Python
```python theme={"system"}
from mage_ai.orchestration.db import db_connection
from mage_ai.orchestration.db.models.schedules import PipelineSchedule, PipelineRun
from mage_ai.data_preparation.models.triggers import ScheduleStatus
from mage_ai.orchestration.pipeline_scheduler import PipelineScheduler
db_connection.start_session()
pipeline_schedules = PipelineSchedule.query.filter(
# Filter by pipeline uuid
PipelineSchedule.pipeline_uuid == 'example_pipeline',
# Filter by trigger name
PipelineSchedule.name == 'trigger_name',
)
# Get pipeline schedule count
print(f'Number of triggers: {pipeline_schedules.count()}')
# Print the detailed info
print([p.to_dict() for p in pipeline_schedules.all()])
# Update trigger status to ACTIVE
for p in pipeline_schedules:
p.update(status=ScheduleStatus.ACTIVE)
# Update trigger status to INACTIVE
for p in pipeline_schedules:
p.update(status=ScheduleStatus.INACTIVE)
# Cancel pipeline runs associated with the triggers
pipeline_schedule_ids = [s.id for s in pipeline_schedules]
pipeline_runs_to_cancel = PipelineRun.query.filter(
PipelineRun.pipeline_schedule_id.in_(pipeline_schedule_ids)
).filter(
PipelineRun.status.in_([
PipelineRun.PipelineRunStatus.INITIAL,
PipelineRun.PipelineRunStatus.RUNNING,
])
)
for pipeline_run in pipeline_runs_to_cancel:
PipelineScheduler(pipeline_run).stop()
# Delete triggers
for p in pipeline_schedules:
p.delete()
```
# Schedules
Source: https://docs.mage.ai/orchestration/triggers/schedule-pipelines
Schedule pipelines to run periodically
You can schedule your pipeline to run on these intervals:
* Once: The pipeline will run only once
* Hourly: The pipeline will run at the beginning of every hour (i.e 0100, 0200, 0300, etc)
* Daily: The pipeline will run at 0000 UTC / GMT every day
* Weekly: The pipeline will run at 0000 UTC / GMT the first day (Monday) of every week
* Monthly: The pipeline will run at 0000 UTC / GMT on the first day of every month
* Always on: The pipeline will be run as soon as the previous pipeline run is finished
* Custom schedule using
[CRON Expression](https://en.wikipedia.org/wiki/Cron#CRON_expression)
* Every minute using CRON expression: `* * * * *`
* *Streaming pipeline*: coming soon
Learn more about how to
[schedule pipelines](/design/data-pipeline-management#create-trigger).
## Extra runtime variables from pipeline run
Default runtime variables from pipeline run:
| Key | Description | Example |
| --------------------- | ------------------------------------------------------------------------------------------------ | -------------------- |
| `env` | The value is always `prod` in pipeline runs. The value is `dev` when running blocks in notebook. | `prod` |
| `event` | If the pipeline is triggered by event, the event variable contains the event payload. | `{"key1": "value1"}` |
| `execution_date` | A datetime object that the pipeline is executed at. | `86400` |
| `execution_partition` | An automatically formatted partition of the pipeline run using the execution date. | `10/20231225T122520` |
| `pipeline_run_id` | The id of the pipeline run. | `10` |
| `trigger_name` | The trigger name of the pipeline run. | `test run 1` |
If your pipeline run belongs to a trigger that is scheduled, then the following extra
variables are available in your Python blockโs keyword arguments (e.g. `kwargs`).
| Key | Description | Example |
| ---------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------- |
| `interval_start_datetime` | The `datetime` when the pipeline run is scheduled for. | `datetime.datetime(2023, 7, 23, 7, 0, 0, 0)` |
| `interval_end_datetime` | The `datetime` when the next pipeline run is scheduled for. | `datetime.datetime(2023, 7, 24, 7, 0, 0, 0)` |
| `interval_seconds` | The number of seconds between the current pipeline run and the next pipeline run. | `86400` |
| `interval_start_datetime_previous` | The `datetime` when the previous pipeline run was scheduled for. | `datetime.datetime(2023, 7, 22, 7, 0, 0, 0)` |
### SQL block
If youโre using a SQL block, here is an example of how you can access these variables:
```sql theme={"system"}
SELECT
'{{ interval_start_datetime }}' AS interval_start_datetime
, '{{ interval_end_datetime }}' AS interval_end_datetime
, '{{ interval_seconds }}' AS interval_seconds
, '{{ interval_start_datetime_previous }}' AS interval_start_datetime_previous
```
# Trigger pipeline from a block
Source: https://docs.mage.ai/orchestration/triggers/trigger-pipeline
You can trigger another pipeline from a block within a different pipeline.
## How to trigger a pipeline
There are 2 ways you can trigger a pipeline from code:
### Block template
Add a [data loader](/development/blocks/data_loaders/templates#trigger-pipeline)
or [data exporter](/development/blocks/data_exporters/templates#trigger-pipeline)
block template named `Orchestration โ Trigger pipeline`.
### Custom code
Add a block and use the following code example within the blockโs function:
```python theme={"system"}
from mage_ai.orchestration.triggers.api import trigger_pipeline
trigger_pipeline(
'pipeline_uuid',
variables={},
check_status=False,
error_on_failure=False,
poll_interval=60,
poll_timeout=None,
schedule_name=None, # Enter a unique name to create a new trigger each time
verbose=True,
)
```
## Parameters
| Parameter name | Description | Default | Sample | Required |
| ------------------ | -------------------------------------------------------------------------------------------------------------- | ------- | ------------------ | -------- |
| `pipeline_uuid` | The UUID of the pipeline to trigger. | | `example_pipeline` | โ |
| `variables` | Runtime variables for the pipeline. | `{}` | `{ 'env': 'dev' }` | |
| `check_status` | Poll and check the status of the triggered pipeline. | `False` | `False` or `True` | |
| `error_on_failure` | Raise an exception if the triggered pipeline fails. | `False` | `False` or `True` | |
| `poll_interval` | Check the status of triggered pipeline every N seconds. | `60` | `300` | |
| `poll_timeout` | After N seconds have elapsed, raise an exception if the triggered pipeline hasnโt finished running successful. | `None` | `3600` | |
| `schedule_name` | Specify the trigger name. If the trigger doesn't exist, it'll get created with the trigger name | `None` | `trigger_name` | |
| `verbose` | Print status of triggered pipeline run. | `True` | `False` or `True` | |
# Triggering pipeline via API request
Source: https://docs.mage.ai/orchestration/triggers/trigger-pipeline-api
You can trigger a pipeline by making an API request.
## Prerequisites
1. [Create a pipeline](/about/features#data-pipeline-management)
2. [Create a trigger](/design/data-pipeline-management#create-trigger)
***
## Configuring trigger type
When you create a new trigger or edit an existing one, choose the option `API`
under trigger type.
***
## API endpoint
The endpoint to use when making an API request will depend on the domain youโre
running Mage and the triggerโs unique ID.
For example, if youโre running Mage on localhost with port 6789 and the trigger
you just created has an ID of 3, then the API endpoint youโll use is:
```
POST http://localhost:6789/api/pipeline_schedules/3/pipeline_runs/abc123
```
***
## Request payload
You can optionally include runtime variables in your request payload. These
runtime variables are accessible from within each pipeline block.
### Example
```json theme={"system"}
{
"pipeline_run": {
"variables": {
"env": "staging",
"schema": "public"
}
}
}
```
Or
```json theme={"system"}
{
"env": "staging",
"schema": "public"
}
```
When the pipeline is triggered by this API request, each block in the pipeline
can access all the values defined in the variables object (e.g. `env`, `schema`)
within the request payload (in addition to the global variables defined within
the pipeline itself).
To use those runtime variables, use the variable name as a key when accessing
the keyword arguments dictionary (`kwargs`) in the decorated function of your
block. For example:
```python theme={"system"}
@data_loader
def load_data(**kwargs) -> DataFrame:
env = kwargs['env']
schema = kwargs['schema']
# ...other code
```
You can include as many key value pairs within the variables object in the
request payload as you want.
***
## Sample cURL command
Here is a sample cURL command using the examples listed above:
```bash theme={"system"}
curl -X POST http://localhost:6789/api/pipeline_schedules/3/pipeline_runs/abc123 \
--header 'Content-Type: application/json' \
--data '
{
"pipeline_run": {
"variables": {
"env": "staging",
"schema": "public"
}
}
}'
```
# Pipeline Trigger Overview
Source: https://docs.mage.ai/orchestration/triggers/triggering-pipelines
Learn how to configure and manage pipeline triggers in Mage
## Overview
Pipeline triggers are automated scheduling mechanisms that determine **when and how data pipelines should run** in Mage. A single pipeline can have multiple triggers, each with different configurations for various use cases like production runs, testing, or data backfills.
### Key Benefits of Pipeline Triggers:
* **Automated execution** - Run pipelines without manual intervention
* **Flexible scheduling** - Support for cron expressions, API calls, and event-based triggers
* **Environment management** - Different trigger configurations for dev, staging, and production
* **Resource optimization** - Control when pipelines consume compute resources
* **Monitoring and SLA** - Built-in performance tracking and alerting
**Pipeline Schedule** and **Trigger** are the same thing in Mage. You may see either term used throughout the product.
## Trigger Types
### Schedule triggers
Schedule triggers are the most common type, designed for **recurring data pipeline execution** at predetermined intervals. They're perfect for ETL processes, data updates, and regular analytics jobs.
Available scheduling frequencies include:
* **Run exactly once** - Execute the pipeline once and then disable the trigger
* **Hourly** - Run every hour
* **Daily** - Run once per day
* **Weekly** - Run once per week
* **Monthly** - Run once per month
* **Always on** - Immediately create new pipeline run when previous pipeline run completes (either succeeds or fails)
* **Custom (cron syntax)** - Use cron expressions for advanced scheduling patterns
**Custom frequency** allows you to use cron syntax for advanced scheduling patterns. This gives you complete control over when your pipeline runs.
#### Common cron examples
Here are some common cron patterns you can use with custom frequency:
```bash theme={"system"}
# Every 15 minutes
*/15 * * * *
# Every weekday at 9 AM
0 9 * * 1-5
# Every day at 2:30 PM
30 14 * * *
# Every Sunday at midnight
0 0 * * 0
# Every 1st of the month at 6 AM
0 6 1 * *
# Every 5 minutes during business hours (9 AM - 5 PM, weekdays)
*/5 9-17 * * 1-5
```
**Cron expression format:** `[minute] [hour] [day(month)] [month] [day(week)]`
**Timezone behavior:** If you have the `display_local_timezone` setting enabled, the local cron expression will match your local timezone only if the minute and hour values are single values without any special characters (comma, hyphen, or slash). You can still use cron expressions with special characters for the minute/hour values, but they will be based in UTC time.
Use online cron expression generators or cron documentation to create more complex scheduling patterns that fit your specific needs.
### API triggers
API triggers enable **on-demand pipeline execution** through HTTP requests. They're ideal for webhook integrations, external system triggers, and manual pipeline execution from external applications. For more details, refer to the [API Trigger documentation](/orchestration/triggers/trigger-pipeline-api).
**Use cases for API triggers:**
* Webhook integrations with external services
* Manual pipeline execution from external applications
* Event-driven data processing
* Integration with CI/CD pipelines
You can make a POST request to an endpoint provided in the UI when creating or
editing a trigger. The endpoint depends on the Mage domain and the trigger's unique ID.
**API endpoint:**
```bash theme={"system"}
POST {MAGE_HOST}/api/pipeline_schedules/3/pipeline_runs/abc123
```
**Request payload:**
Runtime variables can optionally be included in the request payload and are accessible within each pipeline block.
```json theme={"system"}
{
"pipeline_run": {
"variables": {
"env": "staging",
"schema": "public"
}
}
}
```
### Event triggers
An event-type trigger instructs the pipeline to run whenever a specific
event occurs.
For example, you can configure a pipeline to run when a database query completes
or when a new object is created in Amazon S3 or Google Cloud Storage.
You can also trigger a pipeline using a custom event by making a `POST`
request to the `/api/events` endpoint with your event payload:
```bash theme={"system"}
POST /api/events
Content-Type: application/json
{
"event_type": "custom_event",
"payload": {
"key": "value"
}
}
```
Check out this [tutorial](/design/data-pipeline-management#create-trigger) on how to create an event trigger.
## How to Create and Manage Pipeline Triggers
### Creating a New Trigger
To create a new trigger for your pipeline:
1. Navigate to your pipeline in the Mage UI
2. Click the **Triggers** tab in the pipeline view
* Direct URL: `/pipelines/[pipeline_uuid]/triggers`
3. Click **+ New trigger** button
4. Select the trigger type (Schedule or API)
5. Configure the trigger settings based on the type you selected
6. Click **Save changes** to create the trigger
**Trigger storage**: Triggers created via the UI are stored in the database. For environment synchronization and version control, you can [save triggers in code](/orchestration/triggers/configure-triggers-in-code) to sync them across different environments.
#### Trigger configuration fields
When creating or editing a trigger, you'll see the following configuration fields:
**Basic settings:**
* **Name** - A descriptive name for your trigger
* **Description** - Optional description to explain the trigger's purpose
* **Tags** - Add or search tags to organize and categorize your triggers
**Schedule-specific fields:**
* **Frequency** - Choose from predefined options (Hourly, Daily, Weekly, Monthly) or select Custom for CRON syntax
* **Start date and time** - When the trigger should begin running
* **Enable landing time** - Instead of starting at a specific time, schedule runs to finish by a specified time
**Additional options:**
* **Runtime variables** - Override pipeline variables for this specific trigger
* **Run settings** - Configure timeouts, SLA, and failure handling
* **Advanced settings** - Additional trigger-specific configurations
### Viewing and Monitoring Triggers
#### Pipeline-Specific Trigger Management
To view all triggers for a specific pipeline:
1. Navigate to your pipeline in the Mage UI
2. Click the **Triggers** tab
* Direct URL: `/pipelines/[pipeline_uuid]/triggers`
3. You'll see a comprehensive table listing all configured triggers with the following information:
#### Project-Wide Trigger Dashboard
To view all triggers across your entire project:
1. Go to the Mage homepage
2. Click **Triggers** in the left navigation menu
* Direct URL: `/triggers`
3. You'll see all triggers from all pipelines in your project
**Trigger list columns:**
* **Active** - Toggle switch showing if the trigger is enabled/disabled
* **Logs** - Icon to access trigger execution logs
* **Name** - The trigger name (clickable link with arrow icon for code-defined triggers)
* **Type** - Trigger type (schedule, API, etc.)
* **Frequency** - How often the trigger runs (e.g., @once, @daily)
* **Status** - Current status of the trigger
* **Next** - Scheduled time for the next execution (with info icon for details)
* **Runs** - Total number of pipeline runs initiated by this trigger
* **Tags** - Associated tags for organization
* **Created** - Timestamp when the trigger was created (with info icon for details)
You can click on the logs icon to view detailed execution history and logs for each trigger.
**Enabling/Disabling triggers:**
* **From trigger list**: Use the toggle switch in the "Active" column to enable or disable a trigger
* **From trigger detail page**: Navigate to `/pipelines/[pipeline_uuid]/triggers/[trigger_id]` and use the "Disable trigger" button
* **Disabled triggers**: Will not run automatically but remain configured for future use
* **Enabled triggers**: Will run according to their configured schedule or trigger conditions
**Testing triggers:**
* **Run once button**: Click the "Run\@once" button in the trigger list or detail page to manually trigger a pipeline run
* **Testing purpose**: Use this to test your trigger configuration before enabling it for automatic execution
* **Immediate execution**: The pipeline will start running immediately when you click the button
**Backfill triggers:**
* **Historical data processing**: Create [backfill triggers](/orchestration/backfills/overview) to process historical data for a specific date range
* **One-time execution**: Backfill triggers run once and then disable themselves after completion
* **Data gap filling**: Use backfills to fill in missing data or reprocess data with updated logic
**Deleting triggers:**
* **Mage OSS**: Click the delete (trash) icon in the Actions column to delete a trigger
* **Mage Pro**: Right-click on a trigger row and select **Delete** from the context menu
## Advanced Trigger Configuration
### Run Settings and Performance Optimization
#### Timeout configuration
* **Set a timeout for each run of this trigger (optional)**
* Configure a timeout in seconds for pipeline runs initiated by this trigger
* If a run exceeds the timeout, it will be marked with the specified timeout status
* **Status for runs that exceed the timeout (default: failed)**
* Choose how to handle runs that exceed the configured timeout
* Options typically include: failed, cancelled, or warning
#### SLA configuration
* **Configure trigger SLA**
* Set up Service Level Agreement monitoring for your trigger
* Define expected completion time with configurable time units (minutes, hours, days)
* Helps monitor trigger performance against business requirements
### Additional trigger settings
* **Override runtime variables**
* You can override the [runtime variables](/getting-started/runtime-variable) defined in the pipeline to use different values for different triggers.
* This option only appears if you have runtime variables defined in the pipeline.
* **Benefits**: Allows you to run the same pipeline with different configurations for different use cases (e.g., different data sources, environments, or processing parameters) without creating separate pipelines.
* **Keep running pipeline even if blocks fail**
* When enabled, the pipeline continues running even if individual blocks fail
* The pipeline will continue running blocks that are not dependent on the failed blocks until completion
* **(Schedule triggers only) Skip run if previous run still in progress**
* When enabled, the scheduler will not start a new run if a previous run for this trigger is still in progress.
* **(Schedule triggers only)** Available starting in v0.9.62 **Create initial pipeline run if start date is before current execution period**
* When enabled, this creates an initial pipeline run if the schedule's execution date is after its start datetime but before the current time.
* **Example**: If current time is `1/4/24 12:00 UTC` and you create a daily trigger with start date `1/3/24 12:00 UTC`, a pipeline run with execution date `2024-01-04 00:00` is immediately created.
## Frequently Asked Questions
### What is a trigger in Mage?
A trigger is a set of instructions that determine when or how a pipeline should run. There are three types of triggers in Mage: Schedule, Event, and API.
* **Schedule**: Runs pipelines on a set interval (e.g., hourly, daily, weekly, monthly, or custom cron)
* **Event**: Runs pipelines in response to specific events (e.g., file arrival, database update)
* **API**: Runs pipelines when a specific API endpoint is called, optionally with runtime variables
### How do I create a trigger for my pipeline?
You can create triggers via the Mage UI or by defining them in a `triggers.yaml` file within your pipeline folder. In the UI, go to the pipeline editor, select "Triggers," and click "New trigger." In code, add your trigger configuration to `pipelines/[pipeline_uuid]/triggers.yaml`.
### Can I configure triggers in code and in the UI? How are changes synced?
Yes, you can configure triggers in both the UI and code. Changes made in the UI are synced to the YAML file, and vice versa. However, deleting a trigger from the YAML file does not remove it from the database; you must delete it in the UI.
### Why does my daily trigger run at 2AM instead of my specified time?
By default, @daily triggers run at midnight UTC. If you want your pipeline to run at a specific time, use the "custom" frequency option and enter a cron expression that matches your desired schedule.
### What happens if I delete a pipelineโare its triggers deleted too?
No, deleting a pipeline does not automatically delete its associated triggers. You need to manually remove triggers in the UI to avoid orphaned triggers.
### How do I prevent a pipeline from running if a previous run is still in progress?
There is a trigger setting called "Skip run if previous run still in progress." Enabling this will prevent the scheduler from starting a new run if the previous one is still active.
### Can I allow a pipeline to continue even if some blocks fail?
Yes, you can enable the "Allow blocks to fail" setting in your trigger configuration. This allows the pipeline to continue running blocks that are not dependent on failed blocks.
### How do I trigger a pipeline via API?
You can set up an API-type trigger, which allows you to start a pipeline by making a POST request to a specific endpoint. You can also pass runtime variables in your request payload.
# Pipeline filtering and tagging
Source: https://docs.mage.ai/pipelines/pipeline-tagging
How to organize, filter, group, and tag pipelines in Mage.
A [pipeline](/design/core-abstractions#pipeline) contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.
In the Mage GUI, pipelines can be filtered and grouped to make it easier to find the pipeline you want to run. You can also tag pipelines to make it easier to find pipelines that are related to each other.
To apply a tag, first right click the desired pipeline and select "Add/Remove tags," then, type or select the desired tags, separated by enter.
Mage will populate a list of existing tags for consistency. Once tags have been applied, you can group/filter your pipelines by tag on the "Pipeline" page.
Filters are sticky, so grouping/filtering and navigating away will preserve your filter settings.
# Deployment architecture overview
Source: https://docs.mage.ai/pro-architecture/architecture-overview
Deploy Mage Pro in any environment
This document provides technical details on the **four deployment architectures** available for Mage Pro. These options help technical teams identify which deployment best suits their use case, data governance policies, and operational constraints.
***
### Deployment Architecture Overview
Mage Proโs platform architecture consists of two primary components:
1. **Control Plane**
* Handles deployment of Mage Pro clusters
* Manages cluster lifecycle: version upgrades, security patches, monitoring of cluster health
* Provides centralized governance and operational visibility
2. **Data Plane (Mage Pro Cluster)**
* The **actual Mage Pro cluster** running inside Mage cloud or your environment
* Handles pipeline management and orchestration
* Executes pipelines and manages compute resources
* Provides secret management and metadata services
* Hosts the Mage Pro UI and APIs for day-to-day operations
***
### Deployment Options
#### 1. **SaaS (Cloud-Hosted by Mage)**
* **Control Plane & Data Plane**: Fully managed by Mage in a secure multi-tenant cloud environment.
* **Best for**: Teams who want the fastest time-to-value and minimal operational overhead.
* **Key benefits**: Automatic upgrades, lifecycle management, managed infrastructure, secure isolation, and 24/7 monitoring.
* **Considerations**: Data leaves your environment (not suitable for strict data residency requirements).
***
#### 2. **Hybrid Cloud**
* **Control Plane**: Hosted by Mage (SaaS).
* **Data Plane**: Runs inside your private cloud or on-prem infrastructure.
* **Best for**: Organizations that want Mage to manage cluster lifecycle while ensuring pipelines, secrets, and workloads stay in their own environment.
* **Key benefits**: Compliance with data residency/security policies, reduced operational burden.
* **Considerations**: Requires secure network connectivity (e.g., VPC peering, VPN, or private link).
***
#### 3. **Private Cloud (Customer-Managed)**
* **Control Plane & Data Plane**: Fully deployed in your own cloud account (AWS, Azure, GCP, etc.).
* **Best for**: Enterprises requiring strict compliance, tenant isolation, and cloud sovereignty.
* **Key benefits**: Data never leaves your environment, full Mage Pro cluster (orchestration, secrets, pipelines, UI) runs inside your VPC, integrates with your cloud security controls (IAM, Key Vaults, etc.).
* **Considerations**: Your team is responsible for lifecycle management, upgrades, and monitoring.
***
#### 4. **On-Premises**
* **Control Plane & Data Plane**: Installed in your on-prem infrastructure (e.g., Kubernetes, Docker, bare-metal).
* **Best for**: Highly regulated industries or air-gapped environments where no external connectivity is allowed.
* **Key benefits**: Maximum control, full Mage Pro cluster deployed in your data center, compliance with strictest regulations, supports OpenShift and other enterprise Kubernetes distributions.
* **Considerations**: Highest operational overhead (infrastructure, lifecycle management, monitoring, scaling, DR handled by your team).
***
### Choosing the Right Architecture
* **Speed of deployment** โ SaaS or Hybrid Cloud
* **Strict data residency/compliance** โ Private Cloud or On-Prem
* **Operational simplicity** โ SaaS
* **Air-gapped or high-security environments** โ On-Prem
# Hybrid cloud deployment
Source: https://docs.mage.ai/pro-architecture/hybrid-cloud
Deploy the data plane in your cloud environment
Mage Proโs hybrid deployment architecture splits the control plane and the data place components across environments. The control plane is maintained in Mageโs cloud environment while the data processed in the data plane never leaves your cloud infrastructure.
### Technical Specifications
* **Control Plane**: Managed by Mage in secure cloud infrastructure
* **Data Plane**: Deployed in your AWS, GCP, or Azure environment
* **Connectivity**: Secure agent-based communication between planes
* **Resource Management**: Configurable compute profiles with auto-scaling
* **Pipeline Execution**: All data processing occurs within your network boundary
### Integration Requirements
* VPC/VNET connectivity configuration between control and data planes
* IAM roles or service accounts for resource provisioning
* Network security groups configured to allow necessary communication
* Container registry access for execution images
# On-prem deployment
Source: https://docs.mage.ai/pro-architecture/on-prem
Deploy securely from your own on-prem servers
Deploy the control plane and the data plane on your own servers for the most secure environment.
## Deployment options
Mage Pro supports two modes of on-prem deployment. The Mage team will provide all necessary deliverables and installation support.
### 1. Kubernetes cluster
Run Mage Pro in your own on-prem Kubernetes cluster for production-grade orchestration and scalability. This includes Kubernetes distributions such as **Red Hat OpenShift**, Rancher, or other enterprise Kubernetes platforms.
**Deliverables from Mage**:
* Helm chart(s) pre-configured for Mage Pro (control plane, scheduler, and executor pods)
* Docker image (if your environment does not have internet access) **or** secure registry credentials to pull the image (if your cluster has network access)
* Configuration templates for networking, secrets, monitoring, and scaling
* Documentation and best practices for integrating with your Kubernetes environment
**Highlights**:
* Pod-level isolation for workspaces and jobs
* Leverage Kubernetes-native scaling, security, and monitoring
* Recommended for enterprise-grade, multi-team deployments
***
### 2. Docker
For environments without Kubernetes, Mage Pro can be deployed directly on Docker. This simplifies setup while still providing workspace/job isolation using containers.
**Deliverables from Mage**:
* Docker images for Mage Pro services
* Air-gapped environments (no internet): Mage team provides image files directly
* Connected environments: Mage team provides secure registry credentials to pull images
* Docker Compose file with all required configuration
**Highlights**:
* Quick and easy installation with minimal infrastructure requirements
* Run the Mage Pro control plane in a long-running Docker container
* Option to run pipeline jobs in separate, short-lived containers (for isolation) or locally within the same container process (Pro-only feature)
* Workspaces can also run in their own dedicated containers (Pro-only feature)
* Ideal for smaller teams or environments without Kubernetes
***
## License key
A valid **Mage Pro license key** is required for all on-prem deployments.
* The license key is provided when you purchase Mage Pro.
* It must be included in your deployment configuration (Kubernetes or Docker) before the server will start.
* License keys can be rotated at any time without downtime.
***
## Benefits
* **Maximum security**: All services and data stay within your on-prem environment.
* **Flexibility**: Choose between Kubernetes (e.g., OpenShift) or Docker depending on your infra maturity.
* **Consistency**: Same Mage Pro features and workspace management UI across deployment modes.
# Private cloud deployment
Source: https://docs.mage.ai/pro-architecture/private-cloud
Deploy securely in your private cloud environment
Private cloud deployments allow developers to have complete architectural control over the control plane and the data plane. The entire Mage Pro platform will run on your cloud environment.
### Technical Specifications
* **Deployment Platform**: Kubernetes (AKS, EKS, GKE) or OpenShift
* **Installation Method**: Helm charts or operator-based deployment
* **Resource Requirements**:
* Minimum 2 vCPUs and 8GB RAM for control plane services
* Elastic resources for data plane based on workload
* Persistent storage for metadata and logs
* **Networking**: Internal service mesh for component communication
* **Monitoring**: Prometheus-compatible metrics endpoints
* **Backup**: Automated database backup with configurable retention
### Integration Requirements
* Kubernetes cluster with admin access for service deployment
* Persistent volume provisioning capability
* Load balancer or ingress controller for web UI access
* Container registry access for system images
# Managed cloud
Source: https://docs.mage.ai/pro-architecture/saas
Fully managed Mage Pro
Mage Pro SaaS is fully managed and deployed in the Mage cloud infrastructure. In this configuration, both the control and data planes are hosted and managed within Mage's cloud infrastructure:
### Technical Specifications
* **Deployment Method**: Zero touch deployment with automatic provisioning
* **Infrastructure**: Containerized environment with auto scaling in Kubernetes
* **Networking**: Secure TLS connections to data sources with IP Whitelisting
* **Authentication**: SAML/OAuth2 support for SSO integration
* **Updates**: Automated version management with zero-downtime updates
* **Scalability**: Dynamic resource allocation based on workload patterns
* **Monitoring**: Internal monitoring tools, capable of integrating with external monitoring tools
### IP Whitelisting
Enhance your network security by configuring firewall rules or security groups to allow traffic only from Mage Pro's static IP addresses.
This prevents unauthorized access by restricting connections to trusted sources only.
**Mage Pro IP ranges:**
```
100.21.110.88
34.213.159.59
35.155.191.84
35.80.236.114
44.231.28.12
52.34.99.83
52.37.240.164
52.38.195.57
52.42.154.61
54.185.105.221
54.190.141.196
```
### Integration Requirements
* Ensure network connectivity between your data sources and Mage's cloud environment
* Configure firewall rules to allow connection from Mage's IP ranges
* Set up authentication credentials for data source access
* Establish monitoring integrations if desired
### Regional deployment
Mage Proโs SaaS is currently deployed in the following regions U.S. West, U.S. East, Canada, Europe, Asia, and Australia.
Please reach out to Mage if you would like to deploy Mage Pro from a specific region other than whatโs listed above.
# Sign in with Google
Source: https://docs.mage.ai/production/authentication/google
Enable signing in with a Google account in Mage.
### Pre-requisites
First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).
### Enable Google sign in
Mage will request access for the following scopes:
```
https://www.googleapis.com/auth/userinfo.email
https://www.googleapis.com/auth/userinfo.profile
```
By default, users will not see the "Sign in with Google" option in their sign in page.
You will need to add Mage as an OAuth client in your Google Cloud Platform project.
For more instructions on how to do this, please see the [Google Cloud Platform documentation](https://developers.google.com/identity/protocols/oauth2/openid-connect#appsetup).
Once you have created the credentials, you will need to get the application's client ID and client secret.
You will also need to add `http(s):///oauth` as an authorized redirect URI.
Set the following environment variables in your Mage setup.
| Name | Value |
| ---------------------- | ----------------- |
| `GOOGLE_CLIENT_ID` | `` |
| `GOOGLE_CLIENT_SECRET` | `` |
You should now see the "Sign in with Google" button when you start Mage and attempt to sign in.
# Lightweight Directory Access Protocol
Source: https://docs.mage.ai/production/authentication/ldap
Sign in and authenticate users using LDAP
If you want to enable LDAP authentication mode, Set the following environment variable name and value:
| Variable name | Value |
| --------------------- | ------ |
| `AUTHENTICATION_MODE` | `ldap` |
In this case, you will also need to configure the following variables:
| Variable name | Description | Example value |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| `LDAP_ADMIN_USERNAME` | The username of the owner user created on first run | `admin` |
| `LDAP_SERVER` | The host of the LDAP server | `ldap.example.com` |
| `LDAP_BIND_DN` | The `bind_dn` for the LDAP account that will be used to perform search operations | `cn=admin,dc=example,dc=com` |
| `LDAP_BIND_PASSWORD` | The password for the LDAP account that will be used to perform search operations | `password` |
| `LDAP_BASE_DN` | The base DN of the server | `dc=example,dc=com` |
| `LDAP_AUTHENTICATION_FILTER` | The authentication filter to use in template format using the variable `username` | `(&(\|(objectClass=inetOrgPerson)(objectClass=groupOfNames))(cn={username}))` |
| `LDAP_AUTHORIZATION_FILTER` | The authorization filter to use in template format using the variable `user_dn` | `(&(objectClass=groupOfNames)(cn=group)(member={user_dn}))` |
| `LDAP_GROUP_FIELD` | Name of user attribute to use to determine a user's LDAP group. | `memberOf` |
| `LDAP_ROLES_MAPPING` | Mapping of user LDAP group to Mage roles. Must be in JSON string format. | `'{"Admin": ["Admin"]}'` |
| `LDAP_DEFAULT_ACCESS` | Default access for a new user authenticated through LDAP (default user will have no role) | `Viewer`, `Editor`, `Admin` |
| `UPDATE_ROLES_ON_LOGIN` | Update roles on every login. Otherwise, the roles will only be set when the user logs in for the first time. | `1` |
Make sure to adjust the values of the variables to match your LDAP configuration.
When LDAP authentication is enabled, users will need to provide their LDAP credentials to log in to the system. Once authenticated, Mage will use the authorization filter to determine the user's permissions based on their LDAP group membership.
# Multi-factor authentication
Source: https://docs.mage.ai/production/authentication/mfa
Configure multi-factor authentication for Mage user accounts.
## Overview
Multi-factor authentication (MFA) adds a second verification step when a user signs in.
After MFA is enabled for an account, Mage requires the user's password or SSO sign in
and then a 6-digit code from an authenticator app before issuing a session.
Mage supports time-based one-time passwords (TOTP), which work with authenticator apps
such as Google Authenticator, Microsoft Authenticator, Authy, and 1Password.
MFA is configured per user account. Each user must enable MFA from their own account settings.
## Pre-requisites
Before configuring MFA, make sure:
* User authentication is enabled. See [User authentication](/production/authentication/overview).
* Your Mage deployment has run the latest database migrations.
* The Mage server has the MFA dependencies installed from `requirements.txt`, including `pyotp`
and `qrcode`.
* The user has access to an authenticator app that can scan QR codes.
## Enable MFA
Sign in to Mage and go to **Settings** โ **Workspace** โ **MFA**.
Click **Set up Multi-Factor Authentication**.
Open an authenticator app that supports TOTP codes. Mage supports apps such as:
* Google Authenticator
* Microsoft Authenticator
* Authy
* 1Password
Click **Next**, then scan the QR code with your authenticator app.
The QR code adds a new Mage entry to the authenticator app. The app will start showing
rotating 6-digit verification codes for that Mage account.
Enter the current 6-digit code from the authenticator app and click **Next**.
After the code is accepted, Mage shows a confirmation step.
Click **Complete Setup** to close the setup flow and return to the MFA settings page.
The status changes to **Active** after MFA is enabled for your account.
## Sign in with MFA
After MFA is enabled, signing in has two steps:
1. Enter your email or username and password, or complete your configured SSO sign in.
2. Enter the 6-digit code from your authenticator app on the MFA verification page.
If the MFA verification session expires, return to the sign in page and start again.
After too many invalid MFA codes, Mage temporarily locks MFA verification for that user.
Wait 5 minutes, then sign in again and enter a new code from the authenticator app.
## Disable MFA
To disable MFA for your account:
Sign in to Mage and go to **Settings** โ **Workspace** โ **MFA**.
Click **Disable MFA**.
Enter the current 6-digit code from your authenticator app and confirm.
Mage deactivates the active MFA devices for your account after the code is verified.
## Troubleshooting
| Issue | What to check |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The MFA page does not appear | Confirm that your Mage version includes MFA and that user authentication is enabled. |
| QR code generation fails | Confirm that `pyotp` and `qrcode` are installed in the Mage server environment and restart the server after installing dependencies. |
| The verification code is rejected | Enter the newest 6-digit code from the authenticator app. If codes continue to fail, check that the device running the authenticator app has the correct time. |
| The verification page says the session expired | Sign in again to create a new MFA challenge session. |
| Disable MFA fails | Enter the current 6-digit authenticator code for the account that is signed in. |
# Sign in with Microsoft Entra ID aka Active Directory
Source: https://docs.mage.ai/production/authentication/microsoft
Enable signing in with a Microsoft account in Mage.
### Pre-requisites
First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).
### Enable Microsoft sign in
There are two ways to enable Microsoft sign in. You can either register an application in your organization's
own Microsoft Entra ID, or use the default Mage application. We recommend registering your own application unless
there is a specific reason you cannot do so.
You will need to register an application in your organization's Microsoft Entra ID.
Click the "New registration" button at the top of the page, and fill in the details for your application.
Make sure you add the following web redirect URI: `http(s):///oauth` (for Mage open-source)
or `https://cluster.mage.ai//oauth?provider=active_directory` (for Mage Pro)
or `https:///oauth?provider=active_directory` (for Mage self-hosted).
Click into the application you created, and click "Certificates & secrets" in the left sidebar.
Copy the "Application (client) ID" and the "Directory (tenant) ID" values.
Create a new client secret and copy the value.
Once you have registered your application, you will need to set the following environment variables
in your Mage setup.
| Name | Value |
| -------------------------------- | ----------------------------- |
| `ACTIVE_DIRECTORY_DIRECTORY_ID` | `` |
| `ACTIVE_DIRECTORY_CLIENT_ID` | `` |
| `ACTIVE_DIRECTORY_CLIENT_SECRET` | `` |
You should now see the "Sign in with Microsoft" button when you start Mage and attempt to sign in.
You may need to restart your Mage server for the changes to take effect.
You will need to get your organization's Tenant ID. You can find this in the
Azure Portal:
Once you have found your tenant ID, you will need to set the following environment variable
in your Mage setup.
| | |
| ----- | ------------------------------- |
| Name | `ACTIVE_DIRECTORY_DIRECTORY_ID` |
| Value | `` |
You should now see the "Sign in with Microsoft" button when you start Mage and attempt to sign in.
You may need to restart your Mage server for the changes to take effect.
### Configurable environment variables
To configure the environment for Microsoft Azure Cloud and Microsoft Azure Gov Cloud sign-in,
you need to set the following environment variables:
1. You can set the `AZURE_HOST` environment variable to one of the following based on your environment:
```
Public cloud host url: "https://login.microsoftonline.com"
China host url: "https://login.chinacloudapi.cn"
Germany host url: "https://login.microsoftonline.de"
Government host url: "https://login.microsoftonline.us"
```
2. You can set the `MICROSOFT_GRAPH_API_HOST` environment variable to one of the following basd on your environment:
```
Microsoft Graph global service: "https://graph.microsoft.com"
Microsoft Graph for US Government L4: "https://graph.microsoft.us"
Microsoft Graph for US Government L5 (DOD): "https://dod-graph.microsoft.us"
Microsoft Graph China operated by 21Vianet: "https://microsoftgraph.chinacloudapi.cn"
```
### Map AD role to Mage role
You will need to have created your own application and have app roles in order for this mapping to work.
You can map Active Directory roles to Mage roles by setting the `ACTIVE_DIRECTORY_ROLES_MAPPING` environment variable.
The value should be a JSON object with the AD role as the key and the name of the Mage role as the value. For example:
`export ACTIVE_DIRECTORY_ROLES_MAPPING='{"Mage.Edit": "Editor", "Mage.Admin": "Admin"}'`
You can find the AD role value in the App roles section of your application in the Azure Portal.
When a user signs in for the first time with their Microsoft account, Mage will use this mapping to automatically add
the appropriate roles to the user.
# Sign in with OIDC
Source: https://docs.mage.ai/production/authentication/oidc
Enable signing in with OIDC in Mage.
### Pre-requisites
First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).
### Enable OIDC sign in
By default, users will not see the "Sign in with OIDC" option in their sign in page.
You will need to add Mage as a client for your OIDC provider. The way to do this will
be different depending on your OIDC provider. The redirect URL should be `https:///oauth`.
Once you have registered Mage, you will need to retrieve the client ID, client secret,
and the OIDC discovery URL for your provider.
Set the following environment variables in your Mage setup.
| Name | Value |
| -------------------- | ------------------- |
| `OIDC_DISCOVERY_URL` | `` |
| `OIDC_CLIENT_ID` | `` |
| `OIDC_CLIENT_SECRET` | `` |
You should now see a "Sign in with OIDC" button when you start Mage and attempt to sign in.
Mage will fetch the following OIDC endpoints from the discovery URL:
* authorization\_endpoint
* token\_endpoint
* userinfo\_endpoint
# Sign in with Okta
Source: https://docs.mage.ai/production/authentication/okta
Enable signing in with Okta in Mage.
### Pre-requisites
First, you'll need to enable user authentication within Mage. You can find out more about
this in the [Overview page](/production/authentication/overview).
### Enable Okta sign in
By default, users will not see the "Sign in with Okta" option in their sign in page.
You will need to add Mage as an OAuth client in your Okta organization admin dashboard.
For more instructions on how to do this, please see the [Okta developer documentation](https://developer.okta.com/docs/guides/implement-oauth-for-okta/main/#create-an-oauth-2-0-app-in-okta).
Make sure you select the "Authorization Code" grant flow, and add `http(s):///oauth` as a Sign-in redirect URI.
Once you have created the credentials, you will need to get the application's client ID and client secret.
Make sure you also have your Okta domain URL.
Set the following environment variables in your Mage setup.
| Name | Value |
| -------------------- | ------------------- |
| `OKTA_DOMAIN_URL` | `` |
| `OKTA_CLIENT_ID` | `` |
| `OKTA_CLIENT_SECRET` | `` |
You should now see the "Sign in with Okta" button when you start Mage and attempt to sign in.
# User authentication
Source: https://docs.mage.ai/production/authentication/overview
Create users, manage users, and require sign in to authenticate and use Mage.
***
## Overview
Mage supports user authentication to secure access to the UI and APIs.
Authentication is required before creating, managing, or running pipelines.
* **Mage OSS**:
* Versions **0.9.78 and above** โ user authentication is **enabled by default**.
* Versions **0.8.4 โ 0.9.77** โ user authentication can be turned on optionally.
* **Mage Pro**:
* User authentication is **always enabled by default**.
## Configure user authentication
For Mage OSS versions **0.8.4 โ 0.9.77**, you can enable authentication manually by setting the following environment variable:
| | |
| ----- | ----------------------------- |
| Name | `REQUIRE_USER_AUTHENTICATION` |
| Value | `1` |
For Mage OSS **0.9.78+** and **Mage Pro**, this setting is no longer required because authentication is enabled by default.
### Set access token expiration time
You can set the `MAGE_ACCESS_TOKEN_EXPIRY_TIME` environment variable to customize the expiration time
of a Mage access token. The value should be the length of time in seconds.
Defaults to 2592000 which is 30 days.
| | |
| ----- | ------------------------------- |
| Name | `MAGE_ACCESS_TOKEN_EXPIRY_TIME` |
| Value | `2592000` |
### Running Mage in Docker
If youโre running Mage using Docker, you can run Mage and set the environment variable
in the `docker run` command. Follow these [instructions](/getting-started/setup#environment-variables)
to learn how.
Set the environment variable using `-e REQUIRE_USER_AUTHENTICATION=1`.
For example:
```bash theme={"system"}
docker run -it -p 6789:6789 \
-v $(pwd):/home/src \
-e REQUIRE_USER_AUTHENTICATION=1 \
mageai/mageai /app/run_app.sh mage start demo_project
```
### Running Mage without Docker
If you installed Mage using `pip`, `conda`, `poetry`, etc. and are running it using the `mage start`
command, then you need to set the environment variable on your workstation.
#### macOS
Run the following command to set environment variables:
```bash theme={"system"}
export REQUIRE_USER_AUTHENTICATION=1
```
#### Windows
Read Microsoftโs [instructions](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1)
on how to do this.
## Default owner user
When Mage starts, if there is no existing owner user, then a new user will be created with owner
permissions (e.g. all permissions).
The owner user can create, edit, and delete other users.
Here are the default credentials for the owner user to sign in with:
| Field | Value |
| -------- | ----------------- |
| Email | `admin@admin.com` |
| Password | `admin` |
Available in versions >= `0.9.72`
To customize the email, password and username of the default owner user, you can configure these environment variables:
* `DEFAULT_OWNER_EMAIL`
* `DEFAULT_OWNER_PASSWORD`
* `DEFAULT_OWNER_USERNAME`
## Sign in page
Before using Mage, users will have to sign in at `/sign-in`. If you go to a page that requires
authentication and youโre not signed in, youโll be redirected to the sign in page.
## User management page
If youโre signed in as the owner user, youโll have access to the user management page at
`/settings/workspace/users`.
On this page, you can:
1. Add new users
2. Edit existing users
3. Reset user passwords
4. Delete users
## Roles and permissions
| Role | Permissions |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Viewer | Can read. |
| Editor | Can do everything a viewer can, create, update, and delete. |
| Admin | Can do everything a viewer can, an editor can, and manage users who are viewers/editors (update usernames, emails, passwords, and make viewers editors or vice versa. |
| Owner | In addition to what admins can do, add and delete users. Manage all users, including admins/owners. Make other users any role, including admins/owners. |
## Authenticated requests
If you require user authentication by turning it on via the environment variable mentioned above,
then each API request the front-end client makes to the backend server will include an API key
and an OAuth2 token.
The API key and OAuth2 token is used to authenticate the incoming request. If the requester
doesnโt have the proper permissions, the backend server will respond with a `4XX` error code.
### API errors
If a page loads and no data is visible, it could either mean there is no data or the user
doesnโt have the proper permissions.
If there is no visible error message on the page, you can view the browserโs network tab and
inspect the API request response to view the full error details
(weโre working on showing better errors and making it easier to display them).
Learn how to view the network requests in Chrome by following these
[instructions](https://developer.chrome.com/docs/devtools/network/).
| Error code | Description |
| ---------- | -------------------- |
| `401` | Expired OAuth token. |
| `401` | Invalid OAuth token. |
| `402` | Record is invalid. |
| `403` | Invalid API key. |
| `403` | Unauthorized access. |
| `404` | Record not found. |
| `500` | API resource error. |
# Role-based access controls (RBAC)
Source: https://docs.mage.ai/production/authentication/permissions/overview
Create user defined permissions for CRUD operations on any action or user experience. Add permissions to roles and add roles to users.
## Overview
Define roles with 1 or more permissions.
Each permission can grant or deny read and write operations on
specific resources (e.g. API endpoints).
One or more roles can be assigned to 1 or multiple users.
### Why is this important?
Control which user can perform specific actions at a granular level.
## Features
* Define roles.
* Define permissions.
* Add permissions to 1 or more roles.
* Add roles to 1 or more users.
***
## How to use
### Create roles
You must be the owner or have a role that grants permissions to read and write on the following
entities:
* `Permission`
* `Role`
* `RolePermission`
* `UserRole`
* `User`
1. Go the the Mage project settings and click on the navigation row labeled Roles.
2. On the roles list page, there is a button labeled Create default roles and permissions.
If you click this button, 6 roles and a few hundred permissions will be created.
The roles are:
* Admin default permissions
* Editor default permissions
* Editor with notebook edit access
* Editor with pipeline edit access
* Owner default permissions
* Viewer default permissions
Each role will have a set of permissions that match the Mage API policies that are normally
used to authenticate operations when the user authentication feature is turned on but
the user permissions feature is turned off.
3. Alternatively, click the button Add new role to create a single new role.
4. Enter in a unique name for the role.
5. Click the button Create new role to save the role.
6. Once you create the role, youโll be taken to the role detail page.
7. From here, you can add existing permissions to the role.
8. From here, you can add existing users to the role. This will grant permissions or
deny operations for user.
### Create permissions
1. Go the the Mage project settings and click on the navigation row labeled Permissions.
2. Click the button Add new permission to create a new permission.
3. Select an entity that these permissions will be applied to.
An entity refers to a defined API endpoint that currently exists in the Mage application.
For example, the entity `Pipeline` applies to the API endpoints `/pipelines`.
4. Optionally enter an entity UUID that these permissions are applied to.
For example, you can grant or deny permissions to a specific pipeline by selecting the entity
`Pipeline` and entering the entity `UUID` `example_pipeline`.
This permission will only be applied to the pipeline with the UUID `example_pipeline`.
5. Under the section labeled Access, toggle 1 or more accesses to grant.
For more information on what each access grants or denies,
read the [permissions documentation](/authentication/permissions/permissions).
6. Click the button Create new permission to save the permission.
7. Once you create the role, youโll be taken to the role detail page.
8. From here, you can edit the permission.
9. From here, you can add this permission to existing roles.
### Assign roles to users
1. Go the the Mage project settings and click on the navigation row labeled Users.
2. Click on a row for an existing user.
3. From here, you can assign existing roles to the user.
4. Alternatively, you can add a user to a role by following these steps:
1. Go to the [roles list page](http://localhost:6789/settings/workspace/roles).
2. Click on a row for an existing role.
3. Click the button labeled +Add user.
4. Select 1 or more users to assign this role to.
# Permissions
Source: https://docs.mage.ai/production/authentication/permissions/permissions
Create granular permissions for CRUD operations on any API endpoint.
## Overview
* A permission grants or denies read and write operations and access on a specific entity.
* An entity maps to an existing API endpoint in the Mage application.
* A permission can grant or deny access to a specific entity with a specific UUID.
* Each operation and attribute operation has an access level that denies access
to a specific entity for that particular operation or attribute operation.
* When denying an attribute operation,
define the set of attributes that the permission forbids the user from
querying, reading, or writing.
## Operations
This access level grants permission to perform a `GET` request
to the collections API endpoint for a specific entity.
This access level grants permission to perform a `POST` request
to an API endpoint for a specific entity.
This access level grants permission to perform a `GET` request
to the details API endpoint for a specific entity.
This access level grants permission to perform a `PUT` request
to an API endpoint for a specific entity.
This access level grants permission to perform a `DELETE` request
to an API endpoint for a specific entity.
This access level grants allows the following operations to be
performed for a specific entity:
* List
* Create
* Detail
* Update
* Delete
## Attribute operations
This access level grants permission to use a specific set of query parameters
when making an API request for a given entity.
This access level grants permission to read a specific set of attributes
from an API response for a specific entity.
This access level grants permission to write a specific set of attributes
when submitting a payload in the API request body for a specific entity.
### Query attributes
When granting `Query` access, you must define the set of query parameters this permission allows.
### Read attributes
When granting `Read` access, you must define the set of attributes this permission
allows the user to read.
### Write attributes
When granting `Write` access, you must define the set of attributes this permission
allows the user to write.
***
## Groups
The following access levels contain logic that grants access to multiple operations
and attribute operations.
The attribute operation that the group access grants still requires you
to define the specific set of attributes that the user is permitted to
query, read, or write.
This access level grants the following for a specific entity:
* Operations
* List
* Detail
* Attribute operations
* Read
This access level grants the following for a specific entity:
* Operations
* *Everything from Viewer*
* Create
* Update
* Delete
* Attribute operations
* *Everything from Viewer*
* Query
* Write
This access level grants the following for a specific entity:
* Operations
* *Everything from Viewer*
* *Everything from Editor*
* Attribute operations
* *Everything from Viewer*
* *Everything from Editor*
This access level grants the following for a specific entity:
* Operations
* *Everything from Viewer*
* *Everything from Editor*
* *Everything from Admin*
* Attribute operations
* *Everything from Viewer*
* *Everything from Editor*
* *Everything from Admin*
This access level grants every operation, attribute operation,
all query attributes, all read attributes, and all write attributes
for a specific entity.
***
## Special conditions
This access level will deny the user from performing an operation or attribute operation
on a specific entity
unless the user has notebook edit access.
This access level will deny the user from performing an operation or attribute operation
on a specific entity
unless the user has pipeline edit access.
This access level will deny the user from performing an operation or attribute operation
on a specific entity
unless the user owns the current entity they are attempting to perform an action on.
The only entity this access level supports currently is the `User` entity.
***
## Entity names
1. `ALL`
2. `ALL_EXCEPT_RESERVED`
3. `AutocompleteItem`
4. `Backfill`
5. `Block`
6. `BlockLayoutItem`
7. `BlockOutput`
8. `BlockRun`
9. `BlockTemplate`
10. `Chart`
11. `ClientPage`
12. `Cluster`
13. `CustomTemplate`
14. `DataProvider`
15. `Database`
16. `EventMatcher`
17. `EventRule`
18. `ExtensionOption`
19. `Feature`
20. `File`
21. `FileContent`
22. `FileVersion`
23. `Folder`
24. `GitBranch`
25. `GitCustomBranch`
26. `GitFile`
27. `GlobalDataProduct`
28. `IntegrationDestination`
29. `IntegrationSource`
30. `IntegrationSourceStream`
31. `Interaction`
32. `Kernel`
33. `Llm`
34. `Log`
35. `MonitorStat`
36. `Oauth`
37. `OauthAccessToken`
38. `OauthApplication`
39. `Output`
40. `PageBlockLayout`
41. `PageComponent`
42. `Permission`
43. `Pipeline`
44. `PipelineInteraction`
45. `PipelineRun`
46. `PipelineSchedule`
47. `PipelineTrigger`
48. `Project`
49. `PullRequest`
50. `Role`
51. `RolePermission`
52. `Scheduler`
53. `SearchResult`
54. `Secret`
55. `Session`
56. `SparkApplication`
57. `SparkEnvironment`
58. `SparkExecutor`
59. `SparkJob`
60. `SparkSql`
61. `SparkStage`
62. `SparkStageAttempt`
63. `SparkStageAttemptTask`
64. `SparkStageAttemptTaskSummary`
65. `SparkThread`
66. `Status`
67. `Sync`
68. `Tag`
69. `User`
70. `UserRole`
71. `Variable`
72. `Widget`
73. `Workspace`
### `ALL`
Using this entity for a permission will grant the operation or attribute operation
for every entity listed above.
### `ALL_EXCEPT_RESERVED`
Using this entity for a permission will grant the operation or attribute operation
for every entity listed except the following entities:
1. `Oauth`
2. `OauthAccessToken`
3. `OauthApplication`
4. `Workspace`
# Roles
Source: https://docs.mage.ai/production/authentication/permissions/roles
Add permissions to roles and add roles to users.
## Overview
* A role contains a set of 1 or more permissions.
* A user can have 1 or more roles.
* Each role must have a unique name across the entire project.
# Cloud cloud
Source: https://docs.mage.ai/production/ci-cd/cloud-cloud
# Buildkite deployments
Source: https://docs.mage.ai/production/ci-cd/local-cloud/buildkite
CI/CD with Buildkite.
## Mage project setup
Follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).
***
## Buildkite setup for deployment to AWS
Use these templates for Buildkite:
1. [`deploy-sc.sh`](https://github.com/mage-ai/mage-ai/tree/master/templates/cicd/buildkite/deploy-ecs.sh)
2. [`pipeline.yml`](https://github.com/mage-ai/mage-ai/tree/master/templates/cicd/buildkite/pipeline.yml)
# GitHub Actions
Source: https://docs.mage.ai/production/ci-cd/local-cloud/github-actions
Development (local) and production (cloud) using GitHub Actions.
Mage supports GitHub Actions for CI/CD workflows. We have templates for deploying to [AWS ECS, GCP Cloud Run, and Azure](https://github.com/mage-ai/mage-ai/tree/master/templates/github_actions).
Before you configure your action, follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).
***
## GitHub Actions setup
1. Create a new repository on GitHub.
2. Open your repository on GitHub, then click the tab labeled **Settings**.
3. Click the section labeled **Security** on the left hand side to expand it.
4. Click the link labeled **Actions**.
5. Click the button labeled **New repository secret** in the top right corner.
6. Follow the instructions below for your specific cloud provider:
***
## AWS
1. If you havenโt already, create a new AWS ECR repository.
2. Youโll need AWS credentials with the following policy permissions:
```json theme={"system"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:RegisterTaskDefinition",
"ecs:UpdateService",
"iam:PassRole"
],
"Resource": "*"
}
]
}
```
3. In the field labeled **Name**, enter the value `AWS_ACCESS_KEY_ID`.
4. In the field labeled **Secret**, enter your AWS Access Key ID.
5. Click the button labeled **Add secret** to save.
6. Add a 2nd secret by clicking the button labeled **New repository secret** in
the top right corner.
7. In the field labeled **Name**, enter the value `AWS_SECRET_ACCESS_KEY`.
8. In the field labeled **Secret**, enter your AWS Secret Access Key.
9. Click the button labeled **Add secret** to save.
10. Click on the tab labeled **Actions**.
11. On the left side, click the button labeled **New workflow**.
12. Find the link labeled **`set up a workflow yourself`** and click it.
13. Copy the contents from the GitHub Action YAML file for AWS at
[templates/github\_actions/build\_and\_deploy\_to\_aws\_ecs.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_aws_ecs.yml),
and paste it into the textarea.
14. Change the following values under the key labeled `env`:
```yaml theme={"system"}
env:
AWS_REGION: ...
ECR_REPOSITORY: ...
ECS_CLUSTER: ...
ECS_SERVICE: ...
ECS_TASK_DEFINITION: ...
CONTAINER_NAME: ...
```
| Key | Description | Sample value |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `AWS_REGION` | Region of your AWS ECS cluster. | `us-west-2` |
| `CONTAINER_NAME` | Set this to the name of the container in the containerDefinitions section of your task definition. | `mage-data-production-container` |
| `ECR_REPOSITORY` | The name of the AWS ECR repository you created to store your Docker images. | `mage-data` |
| `ECS_CLUSTER` | The name of your AWS ECS cluster. | `mage-production-cluster` |
| `ECS_SERVICE` | The name of your AWS ECS service. | `mage-production-ecs-service` |
| `ECS_TASK_DEFINITION` | Go to your AWS ECS task definition for the above service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
1. Click the button labeled **Start commit** in the top right corner.
2. Click the button labeled **Commit new file**.
3. Every time you merge a pull request into the master branch, this GitHub
Action will run, building a Docker image using your GitHub code, then
updating AWS ECS to use the new image with the updated code.
## GCP
1. Youโll need to add these roles to the service account that youโll be using to
deploy Mage from GitHub:
* Artifact Registry Read
* Artifact Registry Writer
* Cloud Run Admin
* Service Account Token Creator
* Service Account User
2. In the field labeled **Name**, enter the value `GCP_CREDENTIALS`.
3. In the field labeled **Secret**, enter the JSON string containing your GCP
service account credentials. It should look something like this:
`json { "type": "service_account", "project_id": "mage-123456", "private_key_id": "...", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", "client_email": "mage@mage-123456.iam.gserviceaccount.com", "client_id": "...", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mage%40mage-123456.iam.gserviceaccount.com" } `
4. Click the button labeled **Add secret** to save.
5. Click on the tab labeled **Actions**.
6. On the left side, click the button labeled **New workflow**.
7. Find the link labeled **`set up a workflow yourself`** and click it.
8. Copy the contents from the GitHub Action YAML file for GCP at
[templates/github\_actions/build\_and\_deploy\_to\_gcp\_cloud\_run.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_gcp_cloud_run.yml),
and paste it into the textarea.
9. Change the following values under the key labeled `env`:
```yaml theme={"system"}
env:
GAR_LOCATION: ...
GOOGLE_CLOUD_RUN_SERVICE_NAME: ...
IMAGE: ...
PROJECT_ID: ...
REPOSITORY: ...
```
| Key | Description | Sample value |
| ------------------------------- | ------------------------------------------------------------------------------------------- | ------------- |
| `GAR_LOCATION` | Region that Mage is already deployed in. | `us-east1` |
| `GOOGLE_CLOUD_RUN_SERVICE_NAME` | The name of your Google Cloud Run service. Go to the Google Cloud Run dashboard to find it. | `mage-data` |
| `IMAGE` | The name of the Docker image you pushed to the above GCP Artifact Registry. | `mageai` |
| `PROJECT_ID` | Project ID of where you launched Mage using Terraform. | `mage-123456` |
| `REPOSITORY` | The name of your GCP Artifact Registry that is storing your Docker image. | `mageprod` |
10. Click the button labeled **Start commit** in the top right corner.
11. Click the button labeled **Commit new file**.
12. Every time you merge a pull request into the master branch, this GitHub
Action will run, building a Docker image using your GitHub code, then
updating Google Cloud Run to use the new image with the updated code.
## Azure
Mage can be deployed to Azure using GitHub Actions.
Our template for deploying to an Azure Container instance is located [here](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_azure_container_instance.yml) and based off the "GitHub Workflow" option in Azure's docs.
To configure this workflow, you will need to add the following secrets to your GitHub repo:
| Key | Description | | | |
| ----------------------- | -------------------------------------------------------------------------------------- | - | - | - |
| `AZURE_CREDENTIALS` | The entire JSON output from the service principal creation step | | | |
| `REGISTRY_LOGIN_SERVER` | The login server name of your registry (all lowercase). Example: myregistry.azurecr.io | | | |
| `REGISTRY_USERNAME` | The clientId from the JSON output from the service principal creation | | | |
| `REGISTRY_PASSWORD` | The clientSecret from the JSON output from the service principal creation | | | |
| `RESOURCE_GROUP` | The name of the resource group you used to scope the service principal | | | |
We recommend [Microsoft's Guide](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-github-action?tabs=userlevel#configure-github-workflow) for obtaining these credentials via the Azure CLI.
Microsoft also has templates that are similar to oursโ we use the `Service ID Principal` method for authentication, but you can check Microsoft's docs for the [`OpenID Connect`](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-github-action?tabs=openid#save-credentials-to-github-repo) method.
Alternatively, you can use the `az container app up` command in the [Deploy to Azure](https://github.com/Azure/deploy-to-azure-cli-extension) extension in the Azure CLI. This command streamlines creation of the GitHub workflow and deployment steps.
# GitLab deployments
Source: https://docs.mage.ai/production/ci-cd/local-cloud/gitlab-ci-cd
Development (local) and production (cloud) using GitLab CI/CD.
## Mage project setup
Follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).
***
## GitLab CI/CD setup
1. Create a new repository on GitLab.
2. Open your repository on GitLab, then click the tab labeled **Settings**.
3. Click the section labeled **CI/CD**.
4. Under the section labeled **Variables**, click the button labeled **Expand**.
5. Click the button labeled **Add variable**.
6. Follow the instructions below for your specific cloud provider:
***
## AWS
1. If you havenโt already, create a new AWS ECR repository.
2. Youโll need AWS credentials with the following policy permissions:
```json theme={"system"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:RegisterTaskDefinition",
"ecs:UpdateService",
"iam:PassRole"
],
"Resource": "*"
}
]
}
```
3. When adding a variable, in the field labeled **Key**, enter the value `AWS_ACCESS_KEY_ID`.
4. When adding a variable, in the field labeled **Value**, enter your AWS Access Key ID.
5. Check the box labeled **Mask variable**..
6. Click the button labeled **Add variable** to save.
7. Add a 2nd secret by clicking the button labeled **Add variable** in
the top right corner.
8. In the field labeled **Key**, enter the value `AWS_SECRET_ACCESS_KEY`.
9. In the field labeled **Value**, enter your AWS Secret Access Key.
10. Check the box labeled **Mask variable**..
11. Click the button labeled **Add variable** to save.
12. In the field labeled **Key**, enter the value `AWS_DEFAULT_REGION`.
13. In the field labeled **Value**, enter your AWS default region.
14. Click the button labeled **Add variable** to save.
15. Create a new file in your repository and name it `.gitlab-ci.yml`.
If you followed the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup),
then your current folder structure could look like this:
```
my_team/
| demo_project/
| .gitlab-ci.yml
| Dockerfile
```
16. Copy the contents of this [GitLab CI/CD template file](https://github.com/mage-ai/mage-ai/blob/master/templates/gitlab_cicd/aws.gitlab-ci.yml)
and paste it into the file named `.gitlab-ci.yml`.
17. Change the following values in the file named `.gitlab-ci.yml` under the key labeled `variables`:
```yaml theme={"system"}
variables:
REPOSITORY_URL: ...
CLUSTER_NAME: ...
SERVICE_NAME: ...
TASK_DEFINITION_NAME: ...
```
| Key | Description | Sample value |
| ---------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `REPOSITORY_URL` | The URL of the AWS ECR repository you created to store your Docker images. | `679812356111.dkr.ecr.us-west-2.amazonaws.com/mage-data` |
| `CLUSTER_NAME` | The name of your AWS ECS cluster. | `mage-production-cluster` |
| `SERVICE_NAME` | The name of your AWS ECS service. | `mage-production-ecs-service` |
| `TASK_DEFINITION_NAME` | Go to your AWS ECS task definition for the above service. Use the name of that task definition. | `mage-production-task` |
18. If you use a different branch than `main`, for example `master`,
you must edit the value `only` under the `build` and `deploy` key in the file `.gitlab-ci.yml`.
For example:
```yaml theme={"system"}
build:
# other settings
only:
- master
deploy:
# other settings
only:
- master
```
19. Commit your changes with the new file `.gitlab-ci.yml`.
20. Push your commits to GitLab.
21. Every time you merge a pull request into the main or master branch, this GitLab
CI/CD pipeline will run, building a Docker image using your GitLab code, then
updating AWS ECS to use the new image with the updated code.
***
## GCP
*Coming soon...*
***
## Azure
*Coming soon...*
# Repository setup
Source: https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup
Setup your Mage project repository.
1. Create a parent folder for your Mage project (e.g. `my_team`).
2. Change directory (e.g. `cd my_team`) into the parent folder and start Mage
locally: For example:
```bash theme={"system"}
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
/app/run_app.sh mage start demo_project
```
For more examples, read the [setup guide](/getting-started/setup).
3. Once youโre done developing, copy the contents of the this
[Dockerfile template](https://github.com/mage-ai/docker/blob/master/Dockerfile)
and paste it into a new `Dockerfile` located in the parent folder of your
Mage project (e.g. `my_team/Dockerfile`). Replace all instances of the
string `[project_name]` with your project name. For example, if your project
name is `demo_project`, then your Dockerfile will look like this: The
contents of your `Dockerfile` are:
```dockerfile theme={"system"}
FROM mageai/mageai:latest
ARG PROJECT_NAME=[project_name]
ARG MAGE_CODE_PATH=/home/mage_code
ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}
WORKDIR ${MAGE_CODE_PATH}
# Replace [project_name] with the name of your project (e.g. demo_project)
COPY ${PROJECT_NAME} ${PROJECT_NAME}
# Set the USER_CODE_PATH variable to the path of user project.
# The project path needs to contain project name.
# Replace [project_name] with the name of your project (e.g. demo_project)
ENV USER_CODE_PATH=${USER_CODE_PATH}
# Install custom Python libraries
RUN pip3 install -r ${USER_CODE_PATH}/requirements.txt
# Install custom libraries within 3rd party libraries (e.g. dbt packages)
RUN python3 /app/install_other_dependencies.py --path ${USER_CODE_PATH}
ENV PYTHONPATH="${PYTHONPATH}:/home/mage_code"
CMD ["/bin/sh", "-c", "/app/run_app.sh"]
```
4. Your current folder structure should look like this:
```
my_team/
| demo_project/
| Dockerfile
```
5. Build a custom Docker image using `mageai/mageai:latest` as the base and
using the newly created Dockerfile as the additional set of instructions:
```
docker build --platform linux/amd64 --tag mageprod:latest .
```
Change `mageprod` to any other name. Youโll need this tag name when
deploying to production in the cloud.
6. To test the new image works, run the following command:
```bash theme={"system"}
docker run -it -p 6789:6789 mageprod:latest /app/run_app.sh mage start demo_project
```
7. Open your browser and go to [http://localhost:6789/](http://localhost:6789/)
1. You should see all your pipelines there.
2. Changing the contents of files wonโt change the contents on your local
file system because all your code was packaged together within the Docker
image (this is intentional).
8. The next steps depends on your deployment method. If you are using
[Terraform](/production/deploying-to-cloud/using-terraform), then youโll need to use the previously
tag name (e.g. `mageprod`) when pushing a Docker image to a remote Docker
registry.
# Continuous deployment with Mage Pro
Source: https://docs.mage.ai/production/ci-cd/mage/pro
Manage, deploy, and rollback new code changes, from a remote repository to any environment, all from within the Mage platform.
Automatically deploy new code changes and data pipelines to different environments.
***
A fully managed service, where we maintain the infrastructure, guarantee uptime,
automatically scale your workloads to handle any volume of pipeline runs,
automatically upgrade new versions of Mage Pro only features,
monitor your production pipelines, and provide enterprise level support.
# Overview
Source: https://docs.mage.ai/production/ci-cd/overview
There are 4 development and deployment workflows.
## Single environment
### Development and production (local)
Develop locally and run pipelines locally in a single environment.
| Resources | Development | Production |
| --------- | ----------- | ---------- |
| Local | `X` | `X` |
| Cloud | | |
### Development and production (cloud)
Develop on cloud and run pipelines on cloud in a single environment.
| Resources | Development | Production |
| --------- | ----------- | ---------- |
| Local | | |
| Cloud | `X` | `X` |
***
## Multiple environments
### Development (local) and production (cloud)
Develop locally (development environment) and
run pipelines on cloud (production environment) in a separate environments.
| Resources | Development | Production |
| --------- | ----------- | ---------- |
| Local | `X` | |
| Cloud | | `X` |
### Development (cloud) and production (cloud)
Develop on cloud (development environment) and
run pipelines on cloud (production environment) in a separate environments.
| Resources | Development | Production |
| --------- | ----------- | ---------- |
| Local | | |
| Cloud | `X` | |
| Cloud | | `X` |
# GitHub Actions
Source: https://docs.mage.ai/production/ci-cd/staging-production/github-actions
Development (local), staging (cloud), and production (cloud) using GitHub Actions
Mage supports GitHub Actions for CI/CD workflows. We have templates for deploying to [AWS ECS, GCP Cloud Run, and Azure](https://github.com/mage-ai/mage-ai/tree/master/templates/github_actions).
## GitHub Actions setup
1. Create a new repository on GitHub.
2. Open your repository on GitHub, then click the tab labeled **Settings**.
3. Click the section labeled **Secrets and variables** on the left hand side to expand it.
4. Create separate staging and production GitHub environments in the **Environments** section.
1. You can also choose to require approval before running jobs in your production environment.
More information [here](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers).
2. You may need to change the `environment` variable in the jobs section of the GitHub Actions below
based on the name of your GitHub environments.
5. Click the link labeled **Actions**.
6. Click the button labeled **New repository secret** in the top right corner.
7. Follow the instructions below for your specific cloud provider:
***
## AWS
1. If you havenโt already, create a new AWS ECR repository.
2. Youโll need AWS credentials with the following policy permissions:
```json theme={"system"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:RegisterTaskDefinition",
"ecs:UpdateService",
"iam:PassRole"
],
"Resource": "*"
}
]
}
```
3. In the field labeled **Name**, enter the value `AWS_ACCESS_KEY_ID`.
4. In the field labeled **Secret**, enter your AWS Access Key ID.
5. Click the button labeled **Add secret** to save.
6. Add a 2nd secret by clicking the button labeled **New repository secret** in
the top right corner.
7. In the field labeled **Name**, enter the value `AWS_SECRET_ACCESS_KEY`.
8. In the field labeled **Secret**, enter your AWS Secret Access Key.
9. Click the button labeled **Add secret** to save.
10. Click on the tab labeled **Actions**.
11. On the left side, click the button labeled **New workflow**.
12. Find the link labeled **`set up a workflow yourself`** and click it.
13. Copy the contents from the GitHub Action YAML file for AWS at
[templates/github\_actions/build\_and\_deploy\_to\_aws\_ecs\_staging\_production.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml),
and paste it into the textarea.
14. Change the following values under the key labeled `env`:
```yaml theme={"system"}
env:
AWS_REGION: ...
CONTAINER_NAME: ...
ECR_REPOSITORY: ...
ECS_CLUSTER: ...
ECS_STAGING_SERVICE: ...
ECS_PRODUCTION_SERVICE: ...
ECS_STAGING_TASK_DEFINITION: ...
ECS_PRODUCTION_TASK_DEFINITION: ...
```
| Key | Description | Sample value |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `AWS_REGION` | Region of your AWS ECS cluster. | `us-west-2` |
| `CONTAINER_NAME` | Set this to the name of the container in the containerDefinitions section of your task definition. | `mage-data-production-container` |
| `ECR_REPOSITORY` | The name of the AWS ECR repository you created to store your Docker images. | `mage-data` |
| `ECS_CLUSTER` | The name of your AWS ECS cluster. | `mage-production-cluster` |
| `ECS_STAGING_SERVICE` | The name of your AWS ECS staging service. | `mage-production-cluster` |
| `ECS_PRODUCTION_SERVICE` | The name of your AWS ECS production service. | `mage-production-ecs-service` |
| `ECS_STAGING_TASK_DEFINITION` | Go to your AWS ECS task definition for the staging service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
| `ECS_PRODUCTION_TASK_DEFINITION` | Go to your AWS ECS task definition for the production service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
1. Click the button labeled **Start commit** in the top right corner.
2. Click the button labeled **Commit new file**.
3. Every time you merge a pull request into the master branch, this GitHub
Action will run, building a Docker image using your GitHub code, then
updating AWS ECS to use the new image with the updated code.
# Compute resources
Source: https://docs.mage.ai/production/configuring-production-settings/compute-resource
Follow the instructions in this [document](/production/deploying-to-cloud/using-terraform) to deploy
Mage in a production environment. When running Mage in production,
you can customize the compute resource for the following services:
1. Front-end application
2. Pipeline executor
3. Block executor
## Front-end application
Customize the compute resource of the Mage web service.
The Mage web service is responsible for running Mage web backend, scheduler service
and local block executions.
You can customize the CPU and memory of the Mage web
service by updating the Terraform variables and then running `terraform apply`.
### Amazon Web Services (AWS)
Update the `ecs_task_cpu` and `ecs_task_memory` variables in the
[`mage-ai-terraform-templates/aws/variables.tf`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf)
file.
### Google Cloud Platform (GCP)
Update the `container_cpu` and `container_memory` variables in the
[`mage-ai-terraform-templates/gcp/variables.tf`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
file.
***
## Pipeline executor
Set the pipelineโs executor type to customize the compute resources for pipeline runs.
Here are the available executor types:
* `local_python`
* `k8s`
* `ecs`
### Default for all blocks in a pipeline
You can set the executor type for all the blocks in a pipeline by specifying the `executor_type` at pipeline level.
***
## Block executor
Mage provides multiple executors to execute blocks.
Here are the available executor types:
* `local_python`
* `k8s`
* `ecs`
* `azure_container_instance`
* `gcp_cloud_run`
### Default for all blocks
Mage uses `local_python` executor type by default. If you want to specify another executor\_type as the default executor type for blocks,
you can set the environment variable `DEFAULT_EXECUTOR_TYPE` to one executor type mentioned above.
# Production Configuration
Source: https://docs.mage.ai/production/configuring-production-settings/overview
## Settings
### Read only access
You can configure Mage to not allow any edits to pipelines or blocks and/or not allow edit or execute
access for the notebook. Users will only be able to create triggers and view the existing pipelines.
They will have access to features such as syncing with GitHub and future remote repository integrations.
To enable this feature set the following environment variable:
`export DISABLE_NOTEBOOK_EDIT_ACCESS=`
The available values are
* 0: this is the same as omitting the variable
* 1 (most restrictive): no edit/execute access to the notebook within Mage. Users will not be able to use the notebook to edit pipeline content, execute blocks,
create secrets, or create variables.
* 2 (less restrictive): only disables edits for pipelines and blocks. Users will not be able to edit pipeline/block metadata or content.
### Databases
By default, Mage uses sqlite to store orchestration data (trigger, pipeline run, and block run). To use a different database, you can set the `MAGE_DATABASE_CONNECTION_URL` environment variable.
In production, you can set the environment variable in the corresponding Terraform script.
#### PostgreSQL
`export MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://user:password@host:port/dbname`
To use a specific `schema_name` in PostgreSQL:
`export MAGE_DATABASE_CONNECTION_URL="postgresql+psycopg2://user:password@host:port/dbname?options=-c%%20search_path%%3Dschema_name"`
#### MSSQL
`export MAGE_DATABASE_CONNECTION_URL="mssql+pyodbc://?odbc_connect=DRIVER={ODBC Driver 18 for SQL Server};SERVER=host;DATABASE=dbname;UID=user;PWD=password;ENCRYPT=yes;TrustServerCertificate=yes;"`
If Mage fails to start because an orchestration database migration failed, use `mage db diagnose` first. See [Troubleshoot database migration failures](/development/databases/default#troubleshoot-database-migration-failures) for migration history, table schema inspection, migration, rollback, and Alembic version stamping commands.
### ulimit
ulimit is a built-in Linux command that allows viewing or limiting system resource used in processes.
If you get `OSError: [Errno 24] Too many open files` error when running in Mage in production,
you can set the environment variable `ULIMIT_NO_FILE` to a larger number (e.g. 8192 or 16384) and restart
Mage server to increase the maximum number of open files allowed in Mage processes.
If you get `OSError: [Errno 24] Too many open files` error when running Mage locally on your laptop,
you can increase the limit with the command `ulimit -n 8192` (or a larger number). Add this command to
`~/.bash_profile` or `~/.zshrc` file so that this config can be applied automatically in the terminal.
# Architecture
Source: https://docs.mage.ai/production/deploying-to-cloud/architecture
## Server
Mage server runs in an instance or container. Here are the responsibilities of the Mage server.
* Handle all the API requests.
* Interact with the storage to read and write data.
* Run the websocket server to handle websocket requests.
* Execute the pipeline code in the kernel.
* Handle user authentication.
Mage server creates the scheduler process once the server starts. The scheduler process is responsible for
scheduling pipeline runs for active pipeline triggers.
## Storage
In the storage layer, Mage stores the following data:
* **Code files**: Mage stores code files on the local disk. You can use [git](/production/data-sync/git-integration) to sync the code between your local disk and remote the git repository. If you deploy Mage in containers, you can mount an Network File System share in your container to persist the code and data on your disk.
* **Data and log files**: By default, Mage stores data and log files on the local disk. You can also configure it to store [logs](/development/observability/logging#logging-to-external-destination) in an external storage.
* **Pipeline Trigger, Pipeline Run, User data, etc**.: Orchestration related objects and authentication related objects are stored in a database. By default, Mage uses a local SQLite database. In a production environment, You can configure Mage to use a [Postgres database](/production/configuring-production-settings/overview#databases).
* **Secrets**: Mage supports storing [secrets](/development/variables/secrets) in the database. It's also recommended to use an external [Secret Manager](/production/deploying-to-cloud/secrets/AWS) to store secrets.
## Executors
Mage supports different types of executors to execute jobs:
* Local process executor
* ECS executor (AWS)
* GCP Cloud Run executor
* Kubernetes executor
Check out this [doc](/production/configuring-production-settings/compute-resource#2-customize-the-compute-resource-of-the-mage-executor) to learn about how to configure these executors.
## Scale web server and schedulers
By default, Mage runs the web server and scheduler in the same container. Mage supports horizontally scaling by separating the web server and scheduler,
and run multiple replicas of web servers and schedulers.
To separate the web server and scheduler, you can override the container command with the commands below:
* Run scheduler only (The scheduler doesn't need dedicated IP and port)
```bash theme={"system"}
/app/run_app.sh mage start project_name --instance-type scheduler
```
* Run web server only
```bash theme={"system"}
/app/run_app.sh mage start project_name --instance-type web_server
```
To run multiple schedulers simultaneously, you will need to specify the `REDIS_URL` environment variable in your container. The Redis storage is used to coordinate the job scheduling between multiple containers to resolve race conditions. Here is an example Redis URL format: `redis://redis:6379/0`
## Autoscaling
Do you want to run your pipelines at any scale without having to deal with infrastructure setup,
configurations, deployment, and maintenance?
1. Autoscaling orchestration scheduler for maximum pipeline trigger frequency
2. Vertical and horizontal autoscaling data pipelines and block run executions
# Mage and Native Cloud Environments
Source: https://docs.mage.ai/production/deploying-to-cloud/aws-without-terraform
Just like a traditional notebook, Mage supports execution in native cloud environments. Below are guides on how to integrate with native cloud resources.
* [Amazon EC2](#amazon-ec2)
* [Amazon ECS](#amazon-ecs). For Amazon ECS, we recommend using Terraform. Here
is the [guide](/production/deploying-to-cloud/using-terraform).
***
# Amazon EC2
Mage can be run in an Amazon EC2 instance, either by
* using SSH to port forward localhost requests to your EC2 instance
* opening port 6789 on your EC2 instance for access
***
## Prerequisites
Your EC2 instance must have `python3` installed. All Python versions between
3.7.0 (inclusive) and 3.10.0 (exclusive) are supported.
When launching an EC2 instance, we suggest using the following AMI:
| AMI ID | AMI name |
| ----------------------- | ---------------------------------------------------------------- |
| `ami-0d70546e43a941d70` | `ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20220609` |
***
## Quick Connection via SSH
### Install Python
Once the EC2 instance is in a running state, SSH into the instance. You can SSH
using the UI in AWS or you can SSH from your terminal using a command like the
following:
```bash theme={"system"}
ssh -i [path_to_key_pair] [ec2_username]@[ec2_public_dns_name]
```
Note: the `ec2_username` is typically `ubuntu` if you used the AMI we
recommended above.
An example SSH command could look like this:
```bash theme={"system"}
ssh -i ~/.ssh/aws-ec2.pem ubuntu@ec2-55-186-46-136.us-west-2.compute.amazonaws.com
```
Once you SSH into the instance, run the following command to install Python and
pip:
```bash theme={"system"}
sudo apt update && \
sudo apt install -y software-properties-common && \
sudo add-apt-repository -y ppa:deadsnakes/ppa && \
sudo apt install -y python3.7 \
sudo apt install -y python3-pip
```
### Run script to start Mage
We provide the `scripts/run_ec2.sh` script to launch Mage in an EC2 instance.
Currently only connections via SSH are supported. To access [this script](https://github.com/mage-ai/mage-ai/blob/master/scripts/run_ec2.sh)
clone this [repository](https://github.com/mage-ai/mage-ai).
To run Mage in an EC2 instance, you need to provide
* Path to the key pair used with the EC2 instance
* Your EC2 instance username
* Your EC2 public DNS name
```bash theme={"system"}
./scripts/run_ec2.sh [path_to_key_pair] [ec2_user_name] [ec2_public_dns_name]
```
Note: the `ec2_username` is typically `ubuntu` if you used the AMI we
recommended above.
You can optionally add `--name custom_repo_name` to the end of the command above
to name your project something different than the default of `default_repo`.
An example command could look like this:
```bash theme={"system"}
./scripts/run_ec2.sh ~/.ssh/aws-ec2.pem ubuntu@ec2-55-186-46-136.us-west-2.compute.amazonaws.com --name demo_project
```
This script will
1. Connect to your EC2 instance
2. Install Mage if not already installed on instance
3. Create a default repository (default repository name is 'default\_repo', use
`--name` to specify your own custom repository name)
4. Launch the Mage tool pointing at this repository
To access the Mage tool, open [localhost:6789](http://localhost:6789). All
actions made here will be forwarded to your EC2 instance for execution.
To quit out of Mage, stop execution in your current terminal window (Ctrl+C).
This will shutdown the Mage tool alongside closing the connection to your EC2
instance.
***
## Manual connection via Open Port
You can also connect to the Mage app running on your EC2 instance by editing
your security group to expose the port that Mage runs on.
1. When creating your EC2 instance, edit your security group rules to allow a
Custom TCP connection to port 6789 (the port that Mage runs on).
2. Connect to your EC2 instance via SSH and install Mage if you haven't already:
```bash theme={"system"}
ssh -i path/to/key/pair.pem ec2_username@ec2_public_dns_name
pip install mage-ai
```
3. Start Mage on your EC2 instance:
```bash theme={"system"}
mage init
mage start
```
4. Then from your browser, access `:6789` to access the
Mage app, where `ec2-instance-public-ip` is the Public IPv4 address of your
EC2 instance.
***
## Manual connection via SSH
You can also manually start a connection to the Mage tool running on an EC2
instance. This process involves opening two separate SSH connections:
1. Connect to the EC2 instance to run the Mage Tool. First, open an SSH
connection with your EC2 Key Pair and your EC2 login information to connect
the EC2 instance.
```bash theme={"system"}
ssh -i path/to/key/pair.pem ec2_username@ec2_public_dns_name
```
If the Mage tool is not installed, install Mage using `pip`. It is
recommended to use a virtual environment to store the Python packages to
isolate your Python dependencies per project.
```bash theme={"system"}
python3 -m venv env
source ./env/bin/activate
pip3 install mage-ai
```
Next, initialize your Mage repository using `mage init`. You can change your
repository name as per your preference.
```bash theme={"system"}
mage init default_repo
```
Then start the Mage app. This will start the notebook at
[localhost:6789](http://localhost:6789), but you can't directly access it yet
as this notebook is started and is executed on the EC2 instance
```bash theme={"system"}
mage start default_repo
```
2. Forward all requests sent to [localhost:6789](http://localhost:6789) on
*your* computer back to the EC2 instance. This means that as you interact the
user interface on your computer, the data will be forwarded to the EC2
instance for execution. Open another terminal window and run
```bash theme={"system"}
ssh -i path/to/key/pair.pem -N -L 6789:localhost:6789 ec2_username@ec2_public_dns_name
```
which opens up a SSH connection to your EC2 instance which forwards requests
to your localhost.
* The `-N` option tells your SSH client to not send user commands to the EC2
server through this connection since this connection is only for port
forwarding
Now you can access the Mage tool at [localhost:6789](http://localhost:6789) on
your computer, and all executions will be processed on your EC2 instance!
***
# Amazon ECS
You can launch Mage in an Amazon Elastic Container Service (ECS) cluster,
allowing you to perform tasks like:
* [Using the Mage code editor in the ECS cluster](#running-the-mage-app-in-ecs)
* [Running a Mage ETL pipeline in production](#running-mage-pipeline-in-ecs)
## Running the Mage App in ECS
Follow the steps below to launch the Mage app on your ECS cluster:
1. Create a new Docker image for launching the Mage Appโ you can start with [our template](https://github.com/mage-ai/docker/blob/master/Dockerfile). Upload this Docker image to a repository like Amazon ECR so ECS tasks can pull and create Docker containers using this image.
2. Create a new ECS cluster. Both Fargate and EC2 based clusters work for this
task.
3. Create a new task definition based off the template below. This task, when
run:
1. Creates a new Mage repository named "default\_repo".
2. Starts the Mage app, serving requests sent to "localhost:6789"
```json theme={"system"}
{
"containerDefinitions": [
{
"name": "mage-data-prep-start",
"image": "[aws-account-id].dkr.ecr.[region].amazonaws.com/[your-image-repo-name]:[tag]",
"portMappings": [
{
"containerPort": 6789,
"hostPort": 6789,
"protocol": "tcp"
}
],
"essential": true,
"entryPoint": ["sh", "-c"],
"command": ["mage init default_repo && mage start default_repo"],
"interactive": true,
"pseudoTerminal": true
}
],
"family": "mage-data-prep",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE", "EC2"],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::[aws-account-id]:role/[ecs-task-execution-role-name]"
}
```
You must specify:
* `"image"` - Docker image URI. If using ECR, you can use the template above
and fill in the following information:
| Parameter | Description |
| ---------------------- | ---------------------------------------- |
| `aws-account-id` | AWS Account ID |
| `region` | Region in which ECR is being used |
| `your-image-repo-name` | Name of the repository holding your repo |
| `tag` | tag for the version of the image to use |
* `"executionRoleArn"` - Name of the
[ECS Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
Assigning this role enables this task to make AWS API calls. This is needed
to pull the Docker image from an ECR repository. Fill in the following
information:
| Parameter | Description |
| ------------------------------ | ------------------------------- |
| `aws-account-id` | AWS Account ID |
| `ecs-task-execution-role-name` | Name of task execution IAM role |
This parameter can be ignored if you are not using Amazon ECR
In addition, you may want to edit the allocated CPU and Memory resources
based on the type of data you plan to handle and intensity of the
computations performed.
4. Launch the task in your new cluster. Make sure the following conditions are
met:
* Task is launched in a VPC with a public subnet and has a Public IP
assigned. Allows you to connect to the ECS task and use the code editor.
* Task security group allows an inbound TCP connection to port 6789 from your
IP (or the IP with which the app will be accessed). Enables you to connect
to port 6789 where the Mage app is running.
* The task has network routes to any service that you plan to connect to,
such as data warehouses or Amazon ECR. Simplest way to add these network
routes are to add outbound connection rules to the security group.
5. Find the public IP of your task. Then to access the Mage app, go to
`your-public-ip:6789`. If you followed the previous steps correctly, you
should be able to access the Mage app running in your cluster.
## Running Mage Pipeline in ECS
If you already have a Mage pipeline developed, you can deploy the pipeline as an
ECS task that periodically executes.
Follow the steps below to setup running a Mage pipeline in ECS.
**Prerequisite:** You must have already created a Mage repository containing the
pipeline you wish to deploy to ECS. Make sure this repository includes any data
files and configuration setting files needed for the pipeline to run.
1. Create a new Docker image for running your pipelineโ you can start with [our template](https://github.com/mage-ai/docker/blob/master/Dockerfile). Make sure to add any other environment variables defined locally to the image to be able to access those secrets in the cluster. Upload this Docker image to a repository like Amazon ECR so ECS tasks can pull and deploy containers using the image.
2. If not already created, create a new ECS cluster. This task can be run on both Fargate and EC2 based instances
3. Create a new task definition based off the template below. This task calls `mage run` on your pipeline when started.
```json theme={"system"}
{
"containerDefinitions": [
{
"name": "mage-data-prep-deploy",
"image": "[aws-account-id].dkr.ecr.[region].amazonaws.com/[your-image-repo-name]:[tag]",
"essential": true,
"entryPoint": ["sh", "-c"],
"command": ["mage run default_repo [your-pipeline-name]"],
"interactive": true,
"pseudoTerminal": true
}
],
"family": "mage-data-prep",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "1024",
"memory": "2048",
"executionRoleArn": "arn:aws:iam::[aws-account-id]:role/[ecs-task-execution-role-name]"
}
```
You must specify:
* `"image"` - Docker image URI. If using ECR, you can use the template above
and fill in the following information:
| Parameter | Description |
| ---------------------- | ---------------------------------------- |
| `aws-account-id` | AWS Account ID |
| `region` | Region in which ECR is being used |
| `your-image-repo-name` | Name of the repository holding your repo |
| `tag` | tag for the version of the image to use |
* `"executionRoleArn"` - Name of the
[ECS Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html).
Assigning this role enables this task to make AWS API calls. This is needed
to pull the Docker image from an ECR repository. Fill in the following
information:
| Parameter | Description |
| ------------------------------ | ------------------------------- |
| `aws-account-id` | AWS Account ID |
| `ecs-task-execution-role-name` | Name of task execution IAM role |
This parameter can be ignored if you are not using Amazon ECR
* `your-pipeline-name` - name of the pipeline to run. This pipeline must be
stored in the repository you added to your Docker image.
In addition, you may want to edit the allocated CPU and Memory resources
based on the type of data you plan to handle and intensity of the
computations performed.
4. Launch the task in your new cluster. Make sure your task has network routes
to any service that you plan to connect to, such as data warehouses or Amazon
ECR. Simplest way to add these network routes are to add outbound connection
rules to the security group.
5. Your pipeline is now running in the ECS task
# AWS ECS deployment architecture
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/aws-ecs-architecture
## Overview
The most common AWS deployment of Mage is with AWS ECS. Our Terraform scripts manage the process of creating an ECS cluster, a load balancer, and a database. [Here's a link](/production/deploying-to-cloud/aws/resources) to the default output of `terraform plan` on AWS ECS.
We use the following AWS services in our deployment:
## Elastic Container Service (ECS)
Amazon ECS is a service provided by AWS for managing and deploying containerized applications. The ECS service in Terraform is used to define and configure the ECS clusters and services.
ECS allows you to run containers at scale, and the Terraform configuration for ECS service includes specifying the task definition, desired task count, launch type, and other settings related to container orchestration.
The Mage application runs as multiple containers in an Amazon ECS service.
## Application Load Balancer (ALB)
An Application Load Balancer is a service provided by AWS that distributes incoming application traffic across multiple targets, such as EC2 instances, ECS containers, or Lambda functions.
In Terraform, the ALB resource is used to define and configure the load balancer, including listeners, target groups, and rules for routing incoming requests to the appropriate backend targets.
Our ALB is configured to route traffic to the ECS cluster and provides a public URL for communication with clients. Read more [here](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-ip-address-type.html).
## Elastic File System (EFS)
Amazon EFS provides a fully managed, scalable, and shared file storage service that can be used with AWS services and on-premises resources.
We use EFS to store a persistent volume where the `mage_data` for your project is stored. You can learn more about what is stored in `mage_data` [here](/design/abstractions/project-structure#project-data). Without EFS, your project data would be lost when your ECS service is terminated. EFS allows us to persist data between containers within ECS.
## Relational Database Service (RDS)
The RDS Instance resource is used to define and configure a specific database instance within the Amazon RDS service.
We use a Postgres RDS to hold the the backend database for your Mage instance. This database contains the metadata that powers Mage. Read more [here](/design/abstractions/project-structure#project-data).
## Virtual Private Cloud (VPC)
Amazon VPC allows you to create a virtual network within AWS, enabling you to isolate and control resources in a private, logically isolated section of the AWS cloud.
The VPC resource in Terraform is used to define and configure the virtual network, including its subnets, route tables, security groups, and other networking-related settings.
The Mage application, ALB, ECS, EFS, and RDS are all deployed within a VPC in our Terraform scripts.
## Read More
You can refer to the instructions found [here](/production/deploying-to-cloud/aws/setup) to learn more about deploying to ECS.
# Deploy to AWS ECS with AWS CodePipeline
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/code-pipeline
## Pre-requisites
### Terraform and AWS ECS
This document assumes you're using Terraform to modify and deploy your AWS instructure. If you're
not using Terraform yet, you can set it up by reading [this document](production/deploying-to-cloud/using-terraform).
This document also assumes that you have already set up your ECS cluster, services, and tasks.
* If you are not using ECS to run Mage, then this document will not apply for you.
You can submit a feature request in our [Slack channel](https://www.mage.ai/chat) if you want Mage to
support deploying to your infrastructure through AWS CodePipeline.
* If you have not set up your ECS yet, you can check out [this document](/production/deploying-to-cloud/aws/setup#deploy-to-ecs-cluster)
to set it up with Terraform.
### AWS CodeCommit
The pipeline that is created through the Terraform scripts has a AWS CodeCommit repo as a source. You
will need to have a CodeCommit repo already set up that has your Mage repository code.
Your CodeCommit repository should have a Dockerfile in the top level of your CodeCommit.
[Here is a reference Dockerfile](https://github.com/mage-ai/docker/blob/master/Dockerfile)
that you can use for your own Dockerfile.
You will also need to add a `buildspec.yml` file to your repository.
The `buildspec.yml` file can follow the following format:
```
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker image...
- docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
```
### AWS ECR
One pipeline created by the script will push a Docker build to AWS ECR, and the other pipeline will read from the same
AWS ECR repo. Make sure you have already created your AWS ECR repository.
## Setting up AWS CodePipeline pipeline with Terraform
Use Terraform scripts in [aws-code-pipeline](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/aws-code-pipeline)
folder to provision resources for AWS CodePipeline.
***
### Terraform plan
You can run the following command to see all the resources that will be created
by Terraform:
```bash theme={"system"}
cd aws
terraform plan
```
***
### 1. Environment variables
If you donโt have the AWS CLI installed, youโll need to create this file:
`~/.aws/credentials`.
In that file, add the following values:
```
[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX
```
Alternatively, you can install the AWS CLI by following
[Amazonโs instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
***
### 2. Customize Terraform variables
In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-code-pipeline/variables.tf),
you will need to update the default values for the following variables:
**AWS settings**
```
variable "AWS_ACCESS_KEY_ID" {
type = string
default = "AWS_ACCESS_KEY_ID"
}
variable "AWS_SECRET_ACCESS_KEY" {
type = string
default = "AWS_SECRET_ACCESS_KEY"
}
variable "aws_region" {
type = string
description = "AWS Region"
default = "us-west-2"
}
variable "aws_account_id" {
type = string
description = "AWS Account ID"
default = "12345678910"
}
```
**S3 bucket name**
```
variable "s3_bucket_name" {
type = string
description = "S3 bucket to store pipeline artifacts"
default = "s3-bucket-name"
}
```
**Code pipeline name (optional)**
```
variable "code_pipeline_name" {
type = string
description = "Name for AWS CodePipeline"
default = "mage-codepipeline"
}
```
**ECR repo values**
```
variable "ecr_repo_name" {
type = string
description = "ECR repo name"
default = "ecr-repo-name"
}
variable "ecr_image_tag" {
type = string
description = "ECR docker image tag"
default = "latest"
}
```
**ECS values**
```
variable "ecs_cluster_name" {
type = string
description = "ECS cluster name"
default = "ecs-cluster-name"
}
variable "ecs_service_name" {
type = string
description = "ECS service name"
default = "ecs-service-name"
}
variable "ecs_container_name" {
type = string
description = "ECS container name"
default = "ecs-container-name"
}
variable "ecs_task_role_name" {
type = string
description = "ECS execution task role name"
default = "ecs-task-role-name"
}
```
**AWS CodeCommit values**
```
variable "codecommit_repo_name" {
type = string
description = "Codecommit repo name"
default = "codecommit_repo_name"
}
variable "codecommit_branch" {
type = string
description = "Codecommit repo branch"
default = "main"
}
```
### 3. IAM Permissions
The IAM permissions are defined in the `aws-code-pipeline/iam.tf` file. Make adjustments to the
permissions as needed.
### 4. Apply Terraform
**Using CLI**
1. Change directory into scripts folder:
```bash theme={"system"}
cd aws-code-pipeline
```
2. Initialize Terraform:
```bash theme={"system"}
terraform init
```
* If you run into errors like the following:
```
โ Error: Failed to install provider
โ
โ Error while installing hashicorp/template v2.2.0: the local package for registry.terraform.io/hashicorp/template 2.2.0 doesn't match any of the checksums previously recorded in the dependency lock file (this might be because the available checksums are for packages targeting different
โ platforms); for more information: https://www.terraform.io/language/provider-checksum-verification
```
then run the following commands to resolve this:
```
brew install kreuzwerker/taps/m1-terraform-provider-helper
m1-terraform-provider-helper activate
m1-terraform-provider-helper install hashicorp/template -v v2.2.0
rm .terraform.lock.hcl
terraform init --upgrade
```
3. Deploy:
```bash theme={"system"}
terraform apply
```
### Results
After running the Terraform script, you should see two AWS CodePipeline pipelines created, `mage-codepipeline`
and `mage-codepipeline-ecr` (unless you changed the `code_pipeline_name` variable). The `mage-codepipeline` pipeline
will be triggered every time a change is detected in the AWS CodeCommit repo. The `mage-codepipeline-ecr` pipeline
will be triggered every time a new image is pushed with into the AWS ECR repo.
# EMR policies
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/emr-policy
```json theme={"system"}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"elasticmapreduce:AddJobFlowSteps",
"elasticmapreduce:CancelSteps",
"elasticmapreduce:DescribeCluster",
"elasticmapreduce:DescribeStep",
"elasticmapreduce:ListClusters",
"elasticmapreduce:ListSteps",
"elasticmapreduce:ModifyCluster",
"elasticmapreduce:ModifyInstanceFleet",
"elasticmapreduce:ModifyInstanceGroups",
"elasticmapreduce:PutAutoScalingPolicy",
"elasticmapreduce:PutAutoTerminationPolicy",
"elasticmapreduce:PutManagedScalingPolicy",
"elasticmapreduce:RemoveAutoScalingPolicy",
"elasticmapreduce:RemoveAutoTerminationPolicy",
"elasticmapreduce:RemoveManagedScalingPolicy",
"elasticmapreduce:RunJobFlow",
"elasticmapreduce:SetTerminationProtection",
"elasticmapreduce:TerminateJobFlows"
],
"Resource": "*"
}
]
}
```
# Terraform AWS resources
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/resources
Here is the default output of `terraform plan`:
```bash theme={"system"}
data.http.myip: Reading...
data.http.myip: Read complete after 0s [id=http://ipv4.icanhazip.com]
data.aws_iam_policy_document.assume_role_policy: Reading...
data.aws_iam_policy_document.assume_role_policy: Read complete after 0s [id=320642683]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
<= read (data resources)
Terraform will perform the following actions:
# data.aws_ecs_task_definition.main will be read during apply
# (depends on a resource or a module with changes pending)
<= data "aws_ecs_task_definition" "main" {
+ arn = (known after apply)
+ arn_without_revision = (known after apply)
+ execution_role_arn = (known after apply)
+ family = (known after apply)
+ id = (known after apply)
+ network_mode = (known after apply)
+ revision = (known after apply)
+ status = (known after apply)
+ task_definition = "mage-data-prep-task"
+ task_role_arn = (known after apply)
}
# data.template_file.env_vars will be read during apply
# (config refers to values not yet known)
<= data "template_file" "env_vars" {
+ id = (known after apply)
+ rendered = (known after apply)
+ template = jsonencode(
[
+ {
+ name = "ENV"
+ value = "production"
},
+ {
+ name = "AWS_ACCESS_KEY_ID"
+ value = "${aws_access_key_id}"
},
+ {
+ name = "AWS_SECRET_ACCESS_KEY"
+ value = "${aws_secret_access_key}"
},
+ {
+ name = "AWS_REGION_NAME"
+ value = "${aws_region_name}"
},
+ {
+ name = "MAGE_DATABASE_CONNECTION_URL"
+ value = "${database_connection_url}"
},
+ {
+ name = "MAGE_EC2_SUBNET_ID"
+ value = "${ec2_subnet_id}"
},
]
)
+ vars = {
+ "aws_access_key_id" = "AWS_ACCESS_KEY_ID"
+ "aws_region_name" = "us-west-2"
+ "aws_secret_access_key" = "AWS_SECRET_ACCESS_KEY"
+ "database_connection_url" = (sensitive value)
+ "ec2_subnet_id" = (known after apply)
}
}
# aws_alb.application_load_balancer will be created
+ resource "aws_alb" "application_load_balancer" {
+ arn = (known after apply)
+ arn_suffix = (known after apply)
+ desync_mitigation_mode = "defensive"
+ dns_name = (known after apply)
+ drop_invalid_header_fields = false
+ enable_deletion_protection = false
+ enable_http2 = true
+ enable_tls_version_and_cipher_suite_headers = false
+ enable_waf_fail_open = false
+ enable_xff_client_port = false
+ id = (known after apply)
+ idle_timeout = 60
+ internal = false
+ ip_address_type = (known after apply)
+ load_balancer_type = "application"
+ name = "mage-data-prep-production-alb"
+ preserve_host_header = false
+ security_groups = (known after apply)
+ subnets = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-alb"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-alb"
}
+ vpc_id = (known after apply)
+ xff_header_processing_mode = "append"
+ zone_id = (known after apply)
}
# aws_cloudwatch_log_group.log-group will be created
+ resource "aws_cloudwatch_log_group" "log-group" {
+ arn = (known after apply)
+ id = (known after apply)
+ name = "mage-data-prep-production-logs"
+ name_prefix = (known after apply)
+ retention_in_days = 0
+ skip_destroy = false
+ tags = {
+ "Application" = "mage-data-prep"
+ "Environment" = "production"
}
+ tags_all = {
+ "Application" = "mage-data-prep"
+ "Environment" = "production"
}
}
# aws_db_instance.rds will be created
+ resource "aws_db_instance" "rds" {
+ address = (known after apply)
+ allocated_storage = 20
+ apply_immediately = false
+ arn = (known after apply)
+ auto_minor_version_upgrade = true
+ availability_zone = (known after apply)
+ backup_retention_period = (known after apply)
+ backup_window = (known after apply)
+ ca_cert_identifier = (known after apply)
+ character_set_name = (known after apply)
+ copy_tags_to_snapshot = false
+ db_name = "mage"
+ db_subnet_group_name = (known after apply)
+ delete_automated_backups = true
+ endpoint = (known after apply)
+ engine = "postgres"
+ engine_version = "13.7"
+ engine_version_actual = (known after apply)
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ identifier = "mage-data-prep-production-db"
+ identifier_prefix = (known after apply)
+ instance_class = "db.t3.micro"
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ latest_restorable_time = (known after apply)
+ license_model = (known after apply)
+ listener_endpoint = (known after apply)
+ maintenance_window = (known after apply)
+ master_user_secret = (known after apply)
+ master_user_secret_kms_key_id = (known after apply)
+ monitoring_interval = 0
+ monitoring_role_arn = (known after apply)
+ multi_az = false
+ name = (known after apply)
+ nchar_character_set_name = (known after apply)
+ network_type = (known after apply)
+ option_group_name = (known after apply)
+ parameter_group_name = (known after apply)
+ password = (sensitive value)
+ performance_insights_enabled = false
+ performance_insights_kms_key_id = (known after apply)
+ performance_insights_retention_period = (known after apply)
+ port = (known after apply)
+ publicly_accessible = true
+ replica_mode = (known after apply)
+ replicas = (known after apply)
+ resource_id = (known after apply)
+ skip_final_snapshot = true
+ snapshot_identifier = (known after apply)
+ status = (known after apply)
+ storage_throughput = (known after apply)
+ storage_type = (known after apply)
+ tags = {
+ "Environment" = "production"
}
+ tags_all = {
+ "Environment" = "production"
}
+ timezone = (known after apply)
+ username = "mageuser"
+ vpc_security_group_ids = (known after apply)
}
# aws_db_subnet_group.rds_subnet_group will be created
+ resource "aws_db_subnet_group" "rds_subnet_group" {
+ arn = (known after apply)
+ description = "mage-data-prep RDS subnet group"
+ id = (known after apply)
+ name = "mage-data-prep-production-rds-subnet-group"
+ name_prefix = (known after apply)
+ subnet_ids = (known after apply)
+ supported_network_types = (known after apply)
+ tags = {
+ "Environment" = "production"
}
+ tags_all = {
+ "Environment" = "production"
}
+ vpc_id = (known after apply)
}
# aws_ecs_cluster.aws-ecs-cluster will be created
+ resource "aws_ecs_cluster" "aws-ecs-cluster" {
+ arn = (known after apply)
+ capacity_providers = (known after apply)
+ id = (known after apply)
+ name = "mage-data-prep-production-cluster"
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-ecs"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-ecs"
}
+ setting {
+ name = "containerInsights"
+ value = "enabled"
}
}
# aws_ecs_service.aws-ecs-service will be created
+ resource "aws_ecs_service" "aws-ecs-service" {
+ cluster = (known after apply)
+ deployment_maximum_percent = 200
+ deployment_minimum_healthy_percent = 100
+ desired_count = 1
+ enable_ecs_managed_tags = false
+ enable_execute_command = false
+ force_new_deployment = true
+ iam_role = (known after apply)
+ id = (known after apply)
+ launch_type = "FARGATE"
+ name = "mage-data-prep-production-ecs-service"
+ platform_version = (known after apply)
+ scheduling_strategy = "REPLICA"
+ tags_all = (known after apply)
+ task_definition = (known after apply)
+ triggers = (known after apply)
+ wait_for_steady_state = false
+ load_balancer {
+ container_name = "mage-data-prep-production-container"
+ container_port = 6789
+ target_group_arn = (known after apply)
}
+ network_configuration {
+ assign_public_ip = true
+ security_groups = (known after apply)
+ subnets = (known after apply)
}
}
# aws_ecs_task_definition.aws-ecs-task will be created
+ resource "aws_ecs_task_definition" "aws-ecs-task" {
+ arn = (known after apply)
+ arn_without_revision = (known after apply)
+ container_definitions = (known after apply)
+ cpu = "512"
+ execution_role_arn = (known after apply)
+ family = "mage-data-prep-task"
+ id = (known after apply)
+ memory = "1024"
+ network_mode = "awsvpc"
+ requires_compatibilities = [
+ "FARGATE",
]
+ revision = (known after apply)
+ skip_destroy = false
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-ecs-td"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-ecs-td"
}
+ task_role_arn = (known after apply)
+ volume {
+ name = "mage-data-prep-fs"
+ efs_volume_configuration {
+ file_system_id = (known after apply)
+ root_directory = "/"
+ transit_encryption = "ENABLED"
}
}
}
# aws_efs_file_system.file_system will be created
+ resource "aws_efs_file_system" "file_system" {
+ arn = (known after apply)
+ availability_zone_id = (known after apply)
+ availability_zone_name = (known after apply)
+ creation_token = (known after apply)
+ dns_name = (known after apply)
+ encrypted = true
+ id = (known after apply)
+ kms_key_id = (known after apply)
+ number_of_mount_targets = (known after apply)
+ owner_id = (known after apply)
+ performance_mode = "generalPurpose"
+ size_in_bytes = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-efs"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-efs"
}
+ throughput_mode = "elastic"
}
# aws_efs_mount_target.mount_target[0] will be created
+ resource "aws_efs_mount_target" "mount_target" {
+ availability_zone_id = (known after apply)
+ availability_zone_name = (known after apply)
+ dns_name = (known after apply)
+ file_system_arn = (known after apply)
+ file_system_id = (known after apply)
+ id = (known after apply)
+ ip_address = (known after apply)
+ mount_target_dns_name = (known after apply)
+ network_interface_id = (known after apply)
+ owner_id = (known after apply)
+ security_groups = (known after apply)
+ subnet_id = (known after apply)
}
# aws_efs_mount_target.mount_target[1] will be created
+ resource "aws_efs_mount_target" "mount_target" {
+ availability_zone_id = (known after apply)
+ availability_zone_name = (known after apply)
+ dns_name = (known after apply)
+ file_system_arn = (known after apply)
+ file_system_id = (known after apply)
+ id = (known after apply)
+ ip_address = (known after apply)
+ mount_target_dns_name = (known after apply)
+ network_interface_id = (known after apply)
+ owner_id = (known after apply)
+ security_groups = (known after apply)
+ subnet_id = (known after apply)
}
# aws_iam_role.ecsTaskExecutionRole will be created
+ resource "aws_iam_role" "ecsTaskExecutionRole" {
+ arn = (known after apply)
+ assume_role_policy = jsonencode(
{
+ Statement = [
+ {
+ Action = "sts:AssumeRole"
+ Effect = "Allow"
+ Principal = {
+ Service = "ecs-tasks.amazonaws.com"
}
+ Sid = ""
},
]
+ Version = "2012-10-17"
}
)
+ create_date = (known after apply)
+ force_detach_policies = false
+ id = (known after apply)
+ managed_policy_arns = (known after apply)
+ max_session_duration = 3600
+ name = "mage-data-prep-execution-task-role"
+ name_prefix = (known after apply)
+ path = "/"
+ role_last_used = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-iam-role"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-iam-role"
}
+ unique_id = (known after apply)
}
# aws_iam_role_policy_attachment.ecsTaskExecutionRole_policy will be created
+ resource "aws_iam_role_policy_attachment" "ecsTaskExecutionRole_policy" {
+ id = (known after apply)
+ policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
+ role = "mage-data-prep-execution-task-role"
}
# aws_internet_gateway.aws-igw will be created
+ resource "aws_internet_gateway" "aws-igw" {
+ arn = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-igw"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-igw"
}
+ vpc_id = (known after apply)
}
# aws_lb_listener.listener will be created
+ resource "aws_lb_listener" "listener" {
+ arn = (known after apply)
+ id = (known after apply)
+ load_balancer_arn = (known after apply)
+ port = 80
+ protocol = "HTTP"
+ ssl_policy = (known after apply)
+ tags_all = (known after apply)
+ default_action {
+ order = (known after apply)
+ target_group_arn = (known after apply)
+ type = "forward"
}
}
# aws_lb_target_group.target_group will be created
+ resource "aws_lb_target_group" "target_group" {
+ arn = (known after apply)
+ arn_suffix = (known after apply)
+ connection_termination = false
+ deregistration_delay = "300"
+ id = (known after apply)
+ ip_address_type = (known after apply)
+ lambda_multi_value_headers_enabled = false
+ load_balancing_algorithm_type = (known after apply)
+ load_balancing_cross_zone_enabled = (known after apply)
+ name = "mage-data-prep-production-tg"
+ port = 6789
+ preserve_client_ip = (known after apply)
+ protocol = "HTTP"
+ protocol_version = (known after apply)
+ proxy_protocol_v2 = false
+ slow_start = 0
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-lb-tg"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-lb-tg"
}
+ target_type = "ip"
+ vpc_id = (known after apply)
+ health_check {
+ enabled = true
+ healthy_threshold = 3
+ interval = 30
+ matcher = "200"
+ path = "/api/kernels"
+ port = "traffic-port"
+ protocol = "HTTP"
+ timeout = 5
+ unhealthy_threshold = 2
}
}
# aws_route.public will be created
+ resource "aws_route" "public" {
+ destination_cidr_block = "0.0.0.0/0"
+ gateway_id = (known after apply)
+ id = (known after apply)
+ instance_id = (known after apply)
+ instance_owner_id = (known after apply)
+ network_interface_id = (known after apply)
+ origin = (known after apply)
+ route_table_id = (known after apply)
+ state = (known after apply)
}
# aws_route_table.public will be created
+ resource "aws_route_table" "public" {
+ arn = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ propagating_vgws = (known after apply)
+ route = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-routing-table-public"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-routing-table-public"
}
+ vpc_id = (known after apply)
}
# aws_route_table_association.public[0] will be created
+ resource "aws_route_table_association" "public" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_route_table_association.public[1] will be created
+ resource "aws_route_table_association" "public" {
+ id = (known after apply)
+ route_table_id = (known after apply)
+ subnet_id = (known after apply)
}
# aws_security_group.load_balancer_security_group will be created
+ resource "aws_security_group" "load_balancer_security_group" {
+ arn = (known after apply)
+ description = "Managed by Terraform"
+ egress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = [
+ "::/0",
]
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = false
+ to_port = 0
},
]
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = [
+ "11.111.11.111/32",
]
+ description = ""
+ from_port = 443
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 443
},
+ {
+ cidr_blocks = [
+ "11.111.11.111/32",
]
+ description = ""
+ from_port = 80
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 80
},
]
+ name = (known after apply)
+ name_prefix = (known after apply)
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-sg"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-sg"
}
+ vpc_id = (known after apply)
}
# aws_security_group.mount_target_security_group will be created
+ resource "aws_security_group" "mount_target_security_group" {
+ arn = (known after apply)
+ description = "Managed by Terraform"
+ egress = (known after apply)
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = []
+ description = ""
+ from_port = 2049
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = (known after apply)
+ self = false
+ to_port = 2049
},
]
+ name = (known after apply)
+ name_prefix = (known after apply)
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-efs-sg"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-efs-sg"
}
+ vpc_id = (known after apply)
}
# aws_security_group.rds_sg will be created
+ resource "aws_security_group" "rds_sg" {
+ arn = (known after apply)
+ description = "mage-data-prep RDS Security Group"
+ egress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = false
+ to_port = 0
},
]
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = []
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = true
+ to_port = 0
},
+ {
+ cidr_blocks = []
+ description = ""
+ from_port = 5432
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = (known after apply)
+ self = false
+ to_port = 5432
},
]
+ name = "mage-data-prep-production-rds-sg"
+ name_prefix = (known after apply)
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-production-rds-sg"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-production-rds-sg"
}
+ vpc_id = (known after apply)
}
# aws_security_group.service_security_group will be created
+ resource "aws_security_group" "service_security_group" {
+ arn = (known after apply)
+ description = "Managed by Terraform"
+ egress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = [
+ "::/0",
]
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = false
+ to_port = 0
},
]
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = [
+ "11.111.11.111/32",
]
+ description = ""
+ from_port = 6789
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = (known after apply)
+ self = false
+ to_port = 6789
},
]
+ name = (known after apply)
+ name_prefix = (known after apply)
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-service-sg"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-service-sg"
}
+ vpc_id = (known after apply)
}
# aws_subnet.private[0] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "us-west-2a"
+ availability_zone_id = (known after apply)
+ cidr_block = "10.32.0.0/24"
+ enable_dns64 = false
+ enable_resource_name_dns_a_record_on_launch = false
+ enable_resource_name_dns_aaaa_record_on_launch = false
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ ipv6_native = false
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ private_dns_hostname_type_on_launch = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-private-subnet-1"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-private-subnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.private[1] will be created
+ resource "aws_subnet" "private" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "us-west-2b"
+ availability_zone_id = (known after apply)
+ cidr_block = "10.32.1.0/24"
+ enable_dns64 = false
+ enable_resource_name_dns_a_record_on_launch = false
+ enable_resource_name_dns_aaaa_record_on_launch = false
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ ipv6_native = false
+ map_public_ip_on_launch = false
+ owner_id = (known after apply)
+ private_dns_hostname_type_on_launch = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-private-subnet-2"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-private-subnet-2"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[0] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "us-west-2a"
+ availability_zone_id = (known after apply)
+ cidr_block = "10.32.100.0/24"
+ enable_dns64 = false
+ enable_resource_name_dns_a_record_on_launch = false
+ enable_resource_name_dns_aaaa_record_on_launch = false
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ ipv6_native = false
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ private_dns_hostname_type_on_launch = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-public-subnet-1"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-public-subnet-1"
}
+ vpc_id = (known after apply)
}
# aws_subnet.public[1] will be created
+ resource "aws_subnet" "public" {
+ arn = (known after apply)
+ assign_ipv6_address_on_creation = false
+ availability_zone = "us-west-2b"
+ availability_zone_id = (known after apply)
+ cidr_block = "10.32.101.0/24"
+ enable_dns64 = false
+ enable_resource_name_dns_a_record_on_launch = false
+ enable_resource_name_dns_aaaa_record_on_launch = false
+ id = (known after apply)
+ ipv6_cidr_block_association_id = (known after apply)
+ ipv6_native = false
+ map_public_ip_on_launch = true
+ owner_id = (known after apply)
+ private_dns_hostname_type_on_launch = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-public-subnet-2"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-public-subnet-2"
}
+ vpc_id = (known after apply)
}
# aws_vpc.aws-vpc will be created
+ resource "aws_vpc" "aws-vpc" {
+ arn = (known after apply)
+ cidr_block = "10.32.0.0/16"
+ default_network_acl_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_group_id = (known after apply)
+ dhcp_options_id = (known after apply)
+ enable_classiclink = (known after apply)
+ enable_classiclink_dns_support = (known after apply)
+ enable_dns_hostnames = true
+ enable_dns_support = true
+ enable_network_address_usage_metrics = (known after apply)
+ id = (known after apply)
+ instance_tenancy = "default"
+ ipv6_association_id = (known after apply)
+ ipv6_cidr_block = (known after apply)
+ ipv6_cidr_block_network_border_group = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-vpc"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "mage-data-prep-vpc"
}
}
Plan: 28 to add, 0 to change, 0 to destroy.
```
# Deploy to AWS with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/setup
## Pre-requisites
### Setup
[Download Mage maintained Terraform scripts](/production/deploying-to-cloud/using-terraform).
### Permissions
In order to create all the required resources in AWS, your IAM user account must
have the following
[permissions](/production/deploying-to-cloud/aws/terraform-apply-policy).
In order to terminate and remove all the resources created from Terraform, you
need the following
[permissions](/production/deploying-to-cloud/aws/terraform-destroy-policy).
***
## Deploy to ECS cluster
Use Terraform scripts in [aws](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/aws)
folder to provision resources for AWS ECS cluster.
***
### Terraform plan
You can run the following command to see all the resources that will be created
by Terraform:
```bash theme={"system"}
cd aws
terraform plan
```
By default, here are the **[resources](/production/deploying-to-cloud/aws/resources)** that will be
created.
***
### 1. Environment variables
If you donโt have the AWS CLI installed, youโll need to create this file:
`~/.aws/credentials`.
In that file, add the following values:
```
[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX
```
Alternatively, you can install the AWS CLI by following
[Amazonโs instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
***
### 2. Customize Terraform settings
**Application name (optional)**
In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you can change the default application name that will appear in AWS ECS:
```
variable "app_name" {
type = string
description = "Application Name"
default = "mage-data-prep"
}
```
**Docker image (optional)**
In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you can change the default Docker image:
```
variable "docker_image" {
description = "Docker image url used in ECS task."
default = "mageai/mageai:latest"
}
```
**Custom Docker images**
If you previously tagged a Docker image you built when following this
[CI/CD guide](/production/ci-cd), you must push that locally tagged Docker
image to Amazon Elastic Container Registry (ECR) before deploying using
Terraform. Read the
[AWS documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
to learn more.
1. `docker build --platform linux/amd64 -tag [image_name]:latest .`
2. `docker tag [image_name]:latest [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest`
3. `docker push [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest`
**Region (optional)**
In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you can change the region:
```
variable "aws_region" {
type = string
description = "AWS Region"
default = "us-west-2"
}
```
**Availability zones (optional)**
In the file
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf),
you must change the availability zones to match your region from above:
```
variable "availability_zones" {
description = "List of availability zones"
default = ["us-west-2a", "us-west-2b"]
}
```
**More**
Other variables defined in
[./aws/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/variables.tf)
can also be customized to your needs.
***
### 3. Configurable environment variables
In the
[`./aws/env_vars.json`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/env_vars.json)
file, you can edit the following variables, which are used by the tool while
running in the cloud:
Change the value of the variables with the following names to match the actual
values you want the tool to use while running in the cloud:
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
These variable values are used by the tool to retrieve AWS resources like
CloudWatch events, etc.
**Other environment variables**
You can add any environment variable you want in this file. These will be set on
the running container.
> Values
>
> The values in your `env_vars.json` must be strings. Wrap each value in double
> quotes.
***
### 4. Deploy
**Using CLI**
1. Change directory into scripts folder:
```bash theme={"system"}
cd aws
```
2. Initialize Terraform:
```bash theme={"system"}
terraform init
```
* If you run into errors like the following:
```
โ Error: Failed to install provider
โ
โ Error while installing hashicorp/template v2.2.0: the local package for registry.terraform.io/hashicorp/template 2.2.0 doesn't match any of the checksums previously recorded in the dependency lock file (this might be because the available checksums are for packages targeting different
โ platforms); for more information: https://www.terraform.io/language/provider-checksum-verification
```
then run the following commands to resolve this:
```
brew install kreuzwerker/taps/m1-terraform-provider-helper
m1-terraform-provider-helper activate
m1-terraform-provider-helper install hashicorp/template -v v2.2.0
rm .terraform.lock.hcl
terraform init --upgrade
```
3. Deploy:
```bash theme={"system"}
terraform apply
```
Once itโs finished deploying, you can access Mage in your browser.
1. Open your EC2 dashboard.
2. View all load balancers.
3. Click on the load balancer with the name `mage-data-prep` in it (if you
changed the app name, then find the load balancer with that app name).
4. Find the public DNS name, copy that, and paste it in your browser.
***
**Using Docker**
From the root directory of Mage, run the following commands:
1. Initialize Terraform:
```bash theme={"system"}
docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
hashicorp/terraform:latest init
```
2. Deploy:
```bash theme={"system"}
docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
--env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
--env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
hashicorp/terraform:latest apply
```
***
### Errors
**Page isnโt loading**
If you run into connection issues, check to see if your IP is whitelisted in the
appropriate [security group](#security).
**503 Forbidden**
Check to see if the service task in your ECS cluster is running or if it
stopped.
503 typically means that the service task isnโt running and that can be caused
by a variety of things.
Open the service task that stopped running and click on the "Logs" tab to see
what issues occurred.
**`exec /bin/sh: exec format error`**
If your cluster is running and the service launches a task that eventually
stops, check the logs for the stopped task. If the logs contain the error above,
try re-building your image with the platform flag `--platform linux/amd64`.
***
### Updating Mage versions
After Mage is running in your cloud, you can update the Mage Docker image
version by running the following command in your CLI tool:
```bash theme={"system"}
aws ecs update-service --cluster [cluster_name] --service [service_name] --force-new-deployment
```
> `cluster_name`
>
> This is the name of your AWS ECS cluster. Go to the AWS ECS dashboard to view
> the clusters and find the cluster with the name that contains the `app_name`
> from `variables.tf`.
> `service_name`
>
> This is the name of your AWS ECS service running in your AWS ECS cluster with
> the name identified from above.
***
### Misc
#### Security
To enable other IP addresses access to Mage, open the security group named
`mage-data-prep-sg` to whitelist IPs.
Add a new inbound rule for HTTP port 80 and use your IP address.
#### HTTPS enabling
Terraform creates a load balancer with a HTTP listener by default.
To enable HTTPS for your cloud app, you need to have SSL certificates. You can
create a SSL certificate in
[AWS Certificate Manager](https://console.aws.amazon.com/acm/home?#/certificates/list).
After you get a certificate, add a HTTPS listener in your load balancer group.
1. Open your EC2 dashboard.
2. View all load balancers.
3. Click on the load balancer with the name `mage-data-prep` in it (if you
changed the app name, then find the load balancer with that app name).
4. Click "Listeners and rules" tab.
5. Click "Add listener" button
6. For "Protocol" and "Port", choose HTTPS and keep the default port or enter a
different port.
7. In "Default actions", select "Forward to target groups" and forward to the target group that
contains your app name (`mage-data-prep-production-tg` by default)
8. In "Security Policy", keep the default security policy.
9. For "Default SSL certificate", do one of the following:
1. If you created or imported a certificate using AWS Certificate Manager,
choose From ACM and choose the certificate.
2. If you uploaded a certificate using IAM, choose From IAM and choose the
certificate.
10. Click "Add" to create the HTTPS listener.
11. Edit your HTTP listener, change the default action to redirect traffic to
your HTTPS listener.
The detailed instructions can be found in:
[https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html)
#### Terminate all resources
If you want to delete all resources created from deploying, run the following
command:
```bash theme={"system"}
terraform destroy
```
A sample output could look like this:
```
Destroy complete! Resources: 10 destroyed.
```
### Authentication with ECS task execution role
Mage creates an ECS task execution role with the format `{app_name}-execution-task-role`.
You can grant the ECS task permissions to access other AWS services by attaching IAM policies to this ECS task execution role.
For example, if you want to list the AWS events that can be used for event triggers, you can attach IAM Policy `CloudWatchEventsReadOnlyAccess`
to your ECS task execution role.
***
## Deploy to EKS cluster
Use Terraform scripts in [aws-eks](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/aws-eks)
folder to provision resources for AWS EKS cluster.
You can run the following command to see all the resources that will be created
by Terraform:
```bash theme={"system"}
cd aws-eks
terraform plan
```
Similar with deploying to ECS cluster, you need to set up the [environment variables](/production/deploying-to-cloud/aws/setup#1-environment-variables) first, and you can customize the Terraform settings in [./aws-eks/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-eks/variables.tf) file.
Then, follow the steps below to set up EKS cluster and deploy Mage.
Run `terraform apply` to provision the EKS cluster
Configure `kubectl` with command `aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)`
If you are planning to use workspaces in EFS, you will need to add additional configurations to the StorageClass.
```yaml theme={"system"}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
# needed for EFS CSI driver dynamic provisioning
parameters:
provisioningMode: efs-ap
fileSystemId:
directoryPerms: "700"
```
Update the `EFS_VOLUME_ID` in [kube/efs-pvc.yaml](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-eks/kube/efs-pvc.yaml) to the efs id in terraform output
For workspaces support, you may need to install the EFS CSI driver add-on directly from the AWS console. Make sure
to use a driver version that supports dynamic provisioning.
Run command `kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"` to install efs csi driver
Run `kubectl apply -f kube/` to deploy app or use Helm to deploy Mage.
Get the node IP via command `kubectl get nodes -o wide` and access Mage with `http://[ip]:{node_port}`
# AWS policies for Terraform apply
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/terraform-apply-policy
```json theme={"system"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"application-autoscaling:DescribeScalingPolicies",
"cloudwatch:GetMetricData",
"ec2:AssociateRouteTable",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateInternetGateway",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeNetworkAcls",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcClassicLink",
"ec2:DescribeVpcClassicLinkDnsSupport",
"ec2:DescribeVpcs",
"ec2:ModifySubnetAttribute",
"ec2:ModifyVpcAttribute",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress",
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:CreateRepository",
"ecr:DescribeImages",
"ecr:DescribeRepositories",
"ecr:GetAuthorizationToken",
"ecr:GetRegistryScanningConfiguration",
"ecr:InitiateLayerUpload",
"ecr:ListTagsForResource",
"ecr:PutImage",
"ecr:TagResource",
"ecr:UploadLayerPart",
"ecs:CreateCluster",
"ecs:CreateService",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:ListAccountSettings",
"ecs:ListClusters",
"ecs:ListServices",
"ecs:RegisterTaskDefinition",
"ecs:TagResource",
"ecs:UpdateService",
"elasticfilesystem:CreateFileSystem",
"elasticfilesystem:CreateMountTarget",
"elasticfilesystem:CreateMountTarget",
"elasticfilesystem:DescribeFileSystems",
"elasticfilesystem:DescribeLifecycleConfiguration",
"elasticfilesystem:DescribeMountTargetSecurityGroups",
"elasticfilesystem:DescribeMountTargets",
"elasticfilesystem:TagResource",
"elasticfilesystem:UpdateFileSystem",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:SetSecurityGroups",
"iam:AttachRolePolicy",
"iam:CreatePolicy",
"iam:CreateRole",
"iam:CreateServiceLinkedRole",
"iam:GetPolicy",
"iam:GetPolicyVersion",
"iam:GetRole",
"iam:ListAttachedRolePolicies",
"iam:ListRolePolicies",
"iam:PassRole",
"iam:TagRole",
"lambda:CreateFunction",
"lambda:GetFunction",
"lambda:GetFunctionCodeSigningConfig",
"lambda:ListVersionsByFunction",
"logs:CreateLogGroup",
"logs:DescribeLogGroups",
"logs:FilterLogEvents",
"logs:ListTagsLogGroup",
"logs:TagResource",
"rds:AddTagsToResource",
"rds:CreateDBInstance",
"rds:CreateDBSubnetGroup",
"rds:DescribeDBInstances",
"rds:DescribeDBSubnetGroups",
"rds:ListTagsForResource",
"rds:ModifyDBInstance"
],
"Resource": "*"
}
]
}
```
# AWS policies for Terraform destroy
Source: https://docs.mage.ai/production/deploying-to-cloud/aws/terraform-destroy-policy
```json theme={"system"}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DeleteInternetGateway",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteVpc",
"ec2:DescribeAccountAttributes",
"ec2:DescribeInternetGateways",
"ec2:DescribeNetworkAcls",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcClassicLink",
"ec2:DescribeVpcClassicLinkDnsSupport",
"ec2:DescribeVpcs",
"ec2:DetachInternetGateway",
"ec2:DetachNetworkInterface",
"ec2:DisassociateRouteTable",
"ecs:DeleteCluster",
"ecs:DeleteService",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeClusters",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:UpdateService",
"elasticfilesystem:DeleteFileSystem",
"elasticfilesystem:DeleteMountTarget",
"elasticfilesystem:DescribeFileSystems",
"elasticfilesystem:DescribeLifecycleConfiguration",
"elasticfilesystem:DescribeMountTargetSecurityGroups",
"elasticfilesystem:DescribeMountTargets",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetGroups",
"iam:DeletePolicy",
"iam:DeleteRole",
"iam:DetachRolePolicy",
"iam:GetPolicy",
"iam:GetPolicyVersion",
"iam:GetRole",
"iam:ListAttachedRolePolicies",
"iam:ListInstanceProfilesForRole",
"iam:ListPolicyVersions",
"iam:ListRolePolicies",
"lambda:DeleteFunction",
"lambda:GetFunction",
"lambda:GetFunctionCodeSigningConfig",
"lambda:ListVersionsByFunction",
"logs:DeleteLogGroup",
"logs:DescribeLogGroups",
"logs:ListTagsLogGroup",
"rds:DeleteDBInstance",
"rds:DeleteDBSubnetGroup",
"rds:DescribeDBInstances",
"rds:DescribeDBSubnetGroups",
"rds:ListTagsForResource"
],
"Resource": "*"
}
]
}
```
# Terraform Azure resources
Source: https://docs.mage.ai/production/deploying-to-cloud/azure/resources
Here is the default output of `terraform plan`:
```bash theme={"system"}
data.http.myip: Reading...
data.http.myip: Read complete after 0s [id=http://ipv4.icanhazip.com]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# azurerm_application_gateway.network will be created
+ resource "azurerm_application_gateway" "network" {
+ id = (known after apply)
+ location = "westus2"
+ name = "mage-data-prep-production-app-gateway"
+ private_endpoint_connection = (known after apply)
+ resource_group_name = "mage-data-prep-production"
+ backend_address_pool {
+ fqdns = []
+ id = (known after apply)
+ ip_addresses = (known after apply)
+ name = "mage-data-prep-production-backend-pool"
}
+ backend_http_settings {
+ cookie_based_affinity = "Disabled"
+ id = (known after apply)
+ name = "mage-data-prep-production-http-setting"
+ pick_host_name_from_backend_address = false
+ port = 6789
+ probe_id = (known after apply)
+ protocol = "Http"
+ request_timeout = 60
+ trusted_root_certificate_names = []
}
+ frontend_ip_configuration {
+ id = (known after apply)
+ name = "mage-data-prep-production-frontend-ip-config"
+ private_ip_address = (known after apply)
+ private_ip_address_allocation = "Dynamic"
+ private_link_configuration_id = (known after apply)
+ public_ip_address_id = (known after apply)
}
+ frontend_port {
+ id = (known after apply)
+ name = "mage-data-prep-production-frontend-port"
+ port = 80
}
+ gateway_ip_configuration {
+ id = (known after apply)
+ name = "mage-data-prep-production-ip-configuration"
+ subnet_id = (known after apply)
}
+ http_listener {
+ frontend_ip_configuration_id = (known after apply)
+ frontend_ip_configuration_name = "mage-data-prep-production-frontend-ip-config"
+ frontend_port_id = (known after apply)
+ frontend_port_name = "mage-data-prep-production-frontend-port"
+ host_names = []
+ id = (known after apply)
+ name = "mage-data-prep-production-listener"
+ protocol = "Http"
+ ssl_certificate_id = (known after apply)
+ ssl_profile_id = (known after apply)
}
+ request_routing_rule {
+ backend_address_pool_id = (known after apply)
+ backend_address_pool_name = "mage-data-prep-production-backend-pool"
+ backend_http_settings_id = (known after apply)
+ backend_http_settings_name = "mage-data-prep-production-http-setting"
+ http_listener_id = (known after apply)
+ http_listener_name = "mage-data-prep-production-listener"
+ id = (known after apply)
+ name = "mage-data-prep-production-routing-rule"
+ priority = 10
+ redirect_configuration_id = (known after apply)
+ rewrite_rule_set_id = (known after apply)
+ rule_type = "Basic"
+ url_path_map_id = (known after apply)
}
+ sku {
+ capacity = 2
+ name = "Standard_v2"
+ tier = "Standard_v2"
}
+ ssl_policy {
+ cipher_suites = (known after apply)
+ disabled_protocols = (known after apply)
+ min_protocol_version = (known after apply)
+ policy_name = (known after apply)
+ policy_type = (known after apply)
}
}
# azurerm_container_group.container_group will be created
+ resource "azurerm_container_group" "container_group" {
+ exposed_port = (known after apply)
+ fqdn = (known after apply)
+ id = (known after apply)
+ ip_address = (known after apply)
+ ip_address_type = "Private"
+ location = "westus2"
+ name = "mage-data-prep-production"
+ network_profile_id = (known after apply)
+ os_type = "Linux"
+ resource_group_name = "mage-data-prep-production"
+ restart_policy = "Always"
+ tags = {
+ "Environment" = "production"
}
+ container {
+ commands = (known after apply)
+ cpu = 1
+ image = "mageai/mageai:latest"
+ memory = 1.5
+ name = "mage-data-prep-production-container"
+ ports {
+ port = 6789
+ protocol = "TCP"
}
+ volume {
+ empty_dir = false
+ mount_path = "/home/src"
+ name = "mage-data-prep-fs"
+ read_only = false
+ share_name = "mage-data-prep-production-data"
+ storage_account_key = (sensitive value)
+ storage_account_name = "magedataprepstorage"
}
}
}
# azurerm_network_profile.containergroup_profile will be created
+ resource "azurerm_network_profile" "containergroup_profile" {
+ container_network_interface_ids = (known after apply)
+ id = (known after apply)
+ location = "westus2"
+ name = "mage-data-prep-production-profile"
+ resource_group_name = "mage-data-prep-production"
+ container_network_interface {
+ name = "mage-data-prep-production-nic"
+ ip_configuration {
+ name = "aciipconfig"
+ subnet_id = (known after apply)
}
}
}
# azurerm_network_security_group.nsg-aci will be created
+ resource "azurerm_network_security_group" "nsg-aci" {
+ id = (known after apply)
+ location = "westus2"
+ name = "mage-data-prep-production-nsg-aci"
+ resource_group_name = "mage-data-prep-production"
+ security_rule = [
+ {
+ access = "Allow"
+ description = ""
+ destination_address_prefix = ""
+ destination_address_prefixes = [
+ "10.0.10.0/24",
]
+ destination_application_security_group_ids = []
+ destination_port_range = ""
+ destination_port_ranges = [
+ "22",
+ "443",
+ "445",
+ "6789",
+ "8000",
]
+ direction = "Inbound"
+ name = "from-gateway-subnet"
+ priority = 100
+ protocol = "Tcp"
+ source_address_prefix = ""
+ source_address_prefixes = [
+ "10.0.20.0/24",
]
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
+ {
+ access = "Allow"
+ description = ""
+ destination_address_prefix = "*"
+ destination_address_prefixes = []
+ destination_application_security_group_ids = []
+ destination_port_range = ""
+ destination_port_ranges = [
+ "443",
+ "445",
+ "80",
]
+ direction = "Outbound"
+ name = "to-internet"
+ priority = 100
+ protocol = "Tcp"
+ source_address_prefix = "*"
+ source_address_prefixes = []
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
+ {
+ access = "Deny"
+ description = ""
+ destination_address_prefix = "*"
+ destination_address_prefixes = []
+ destination_application_security_group_ids = []
+ destination_port_range = "*"
+ destination_port_ranges = []
+ direction = "Inbound"
+ name = "DenyAllInBound-Override"
+ priority = 900
+ protocol = "*"
+ source_address_prefix = "*"
+ source_address_prefixes = []
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
+ {
+ access = "Deny"
+ description = ""
+ destination_address_prefix = "*"
+ destination_address_prefixes = []
+ destination_application_security_group_ids = []
+ destination_port_range = "*"
+ destination_port_ranges = []
+ direction = "Outbound"
+ name = "DenyAllOutBound-Override"
+ priority = 900
+ protocol = "*"
+ source_address_prefix = "*"
+ source_address_prefixes = []
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
]
}
# azurerm_network_security_group.nsg-public will be created
+ resource "azurerm_network_security_group" "nsg-public" {
+ id = (known after apply)
+ location = "westus2"
+ name = "mage-data-prep-production-nsg-public"
+ resource_group_name = "mage-data-prep-production"
+ security_rule = [
+ {
+ access = "Allow"
+ description = ""
+ destination_address_prefix = "*"
+ destination_address_prefixes = []
+ destination_application_security_group_ids = []
+ destination_port_range = ""
+ destination_port_ranges = [
+ "80",
]
+ direction = "Inbound"
+ name = "whitelist-inbound-ip"
+ priority = 100
+ protocol = "Tcp"
+ source_address_prefix = ""
+ source_address_prefixes = [
+ "11.111.11.111/32",
]
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
+ {
+ access = "Allow"
+ description = ""
+ destination_address_prefix = "*"
+ destination_address_prefixes = []
+ destination_application_security_group_ids = []
+ destination_port_range = "65200-65535"
+ destination_port_ranges = []
+ direction = "Inbound"
+ name = "AllowInfraComms"
+ priority = 200
+ protocol = "Tcp"
+ source_address_prefix = "*"
+ source_address_prefixes = []
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
+ {
+ access = "Deny"
+ description = ""
+ destination_address_prefix = "*"
+ destination_address_prefixes = []
+ destination_application_security_group_ids = []
+ destination_port_range = "*"
+ destination_port_ranges = []
+ direction = "Inbound"
+ name = "DenyAllInBound-Override"
+ priority = 900
+ protocol = "*"
+ source_address_prefix = "*"
+ source_address_prefixes = []
+ source_application_security_group_ids = []
+ source_port_range = "*"
+ source_port_ranges = []
},
]
}
# azurerm_public_ip.public_ip will be created
+ resource "azurerm_public_ip" "public_ip" {
+ allocation_method = "Static"
+ fqdn = (known after apply)
+ id = (known after apply)
+ idle_timeout_in_minutes = 4
+ ip_address = (known after apply)
+ ip_version = "IPv4"
+ location = "westus2"
+ name = "mage-data-prep-production-public-ip"
+ resource_group_name = "mage-data-prep-production"
+ sku = "Standard"
+ sku_tier = "Regional"
}
# azurerm_resource_group.resource_group will be created
+ resource "azurerm_resource_group" "resource_group" {
+ id = (known after apply)
+ location = "westus2"
+ name = "mage-data-prep-production"
}
# azurerm_storage_account.aci_storage will be created
+ resource "azurerm_storage_account" "aci_storage" {
+ access_tier = (known after apply)
+ account_kind = "StorageV2"
+ account_replication_type = "LRS"
+ account_tier = "Standard"
+ allow_nested_items_to_be_public = true
+ cross_tenant_replication_enabled = true
+ default_to_oauth_authentication = false
+ enable_https_traffic_only = true
+ id = (known after apply)
+ infrastructure_encryption_enabled = false
+ is_hns_enabled = false
+ large_file_share_enabled = (known after apply)
+ location = "westus2"
+ min_tls_version = "TLS1_2"
+ name = "magedataprepstorage"
+ nfsv3_enabled = false
+ primary_access_key = (sensitive value)
+ primary_blob_connection_string = (sensitive value)
+ primary_blob_endpoint = (known after apply)
+ primary_blob_host = (known after apply)
+ primary_connection_string = (sensitive value)
+ primary_dfs_endpoint = (known after apply)
+ primary_dfs_host = (known after apply)
+ primary_file_endpoint = (known after apply)
+ primary_file_host = (known after apply)
+ primary_location = (known after apply)
+ primary_queue_endpoint = (known after apply)
+ primary_queue_host = (known after apply)
+ primary_table_endpoint = (known after apply)
+ primary_table_host = (known after apply)
+ primary_web_endpoint = (known after apply)
+ primary_web_host = (known after apply)
+ public_network_access_enabled = true
+ queue_encryption_key_type = "Service"
+ resource_group_name = "mage-data-prep-production"
+ secondary_access_key = (sensitive value)
+ secondary_blob_connection_string = (sensitive value)
+ secondary_blob_endpoint = (known after apply)
+ secondary_blob_host = (known after apply)
+ secondary_connection_string = (sensitive value)
+ secondary_dfs_endpoint = (known after apply)
+ secondary_dfs_host = (known after apply)
+ secondary_file_endpoint = (known after apply)
+ secondary_file_host = (known after apply)
+ secondary_location = (known after apply)
+ secondary_queue_endpoint = (known after apply)
+ secondary_queue_host = (known after apply)
+ secondary_table_endpoint = (known after apply)
+ secondary_table_host = (known after apply)
+ secondary_web_endpoint = (known after apply)
+ secondary_web_host = (known after apply)
+ shared_access_key_enabled = true
+ table_encryption_key_type = "Service"
+ blob_properties {
+ change_feed_enabled = (known after apply)
+ change_feed_retention_in_days = (known after apply)
+ default_service_version = (known after apply)
+ last_access_time_enabled = (known after apply)
+ versioning_enabled = (known after apply)
+ container_delete_retention_policy {
+ days = (known after apply)
}
+ cors_rule {
+ allowed_headers = (known after apply)
+ allowed_methods = (known after apply)
+ allowed_origins = (known after apply)
+ exposed_headers = (known after apply)
+ max_age_in_seconds = (known after apply)
}
+ delete_retention_policy {
+ days = (known after apply)
}
}
+ network_rules {
+ bypass = (known after apply)
+ default_action = (known after apply)
+ ip_rules = (known after apply)
+ virtual_network_subnet_ids = (known after apply)
+ private_link_access {
+ endpoint_resource_id = (known after apply)
+ endpoint_tenant_id = (known after apply)
}
}
+ queue_properties {
+ cors_rule {
+ allowed_headers = (known after apply)
+ allowed_methods = (known after apply)
+ allowed_origins = (known after apply)
+ exposed_headers = (known after apply)
+ max_age_in_seconds = (known after apply)
}
+ hour_metrics {
+ enabled = (known after apply)
+ include_apis = (known after apply)
+ retention_policy_days = (known after apply)
+ version = (known after apply)
}
+ logging {
+ delete = (known after apply)
+ read = (known after apply)
+ retention_policy_days = (known after apply)
+ version = (known after apply)
+ write = (known after apply)
}
+ minute_metrics {
+ enabled = (known after apply)
+ include_apis = (known after apply)
+ retention_policy_days = (known after apply)
+ version = (known after apply)
}
}
+ routing {
+ choice = (known after apply)
+ publish_internet_endpoints = (known after apply)
+ publish_microsoft_endpoints = (known after apply)
}
+ share_properties {
+ cors_rule {
+ allowed_headers = (known after apply)
+ allowed_methods = (known after apply)
+ allowed_origins = (known after apply)
+ exposed_headers = (known after apply)
+ max_age_in_seconds = (known after apply)
}
+ retention_policy {
+ days = (known after apply)
}
+ smb {
+ authentication_types = (known after apply)
+ channel_encryption_type = (known after apply)
+ kerberos_ticket_encryption_type = (known after apply)
+ versions = (known after apply)
}
}
}
# azurerm_storage_share.container_share will be created
+ resource "azurerm_storage_share" "container_share" {
+ access_tier = (known after apply)
+ enabled_protocol = "SMB"
+ id = (known after apply)
+ metadata = (known after apply)
+ name = "mage-data-prep-production-data"
+ quota = 100
+ resource_manager_id = (known after apply)
+ storage_account_name = "magedataprepstorage"
+ url = (known after apply)
}
# azurerm_subnet.sn-aci will be created
+ resource "azurerm_subnet" "sn-aci" {
+ address_prefixes = [
+ "10.0.10.0/24",
]
+ enforce_private_link_endpoint_network_policies = (known after apply)
+ enforce_private_link_service_network_policies = (known after apply)
+ id = (known after apply)
+ name = "aci"
+ private_endpoint_network_policies_enabled = (known after apply)
+ private_link_service_network_policies_enabled = (known after apply)
+ resource_group_name = "mage-data-prep-production"
+ service_endpoints = [
+ "Microsoft.Storage",
]
+ virtual_network_name = "mage-data-prep-production"
+ delegation {
+ name = "acidelegationservice"
+ service_delegation {
+ actions = [
+ "Microsoft.Network/virtualNetworks/subnets/join/action",
+ "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
]
+ name = "Microsoft.ContainerInstance/containerGroups"
}
}
}
# azurerm_subnet.sn-public will be created
+ resource "azurerm_subnet" "sn-public" {
+ address_prefixes = [
+ "10.0.20.0/24",
]
+ enforce_private_link_endpoint_network_policies = (known after apply)
+ enforce_private_link_service_network_policies = (known after apply)
+ id = (known after apply)
+ name = "public"
+ private_endpoint_network_policies_enabled = (known after apply)
+ private_link_service_network_policies_enabled = (known after apply)
+ resource_group_name = "mage-data-prep-production"
+ virtual_network_name = "mage-data-prep-production"
}
# azurerm_subnet_network_security_group_association.sn-nsg-aci will be created
+ resource "azurerm_subnet_network_security_group_association" "sn-nsg-aci" {
+ id = (known after apply)
+ network_security_group_id = (known after apply)
+ subnet_id = (known after apply)
}
# azurerm_subnet_network_security_group_association.sn-nsg-public will be created
+ resource "azurerm_subnet_network_security_group_association" "sn-nsg-public" {
+ id = (known after apply)
+ network_security_group_id = (known after apply)
+ subnet_id = (known after apply)
}
# azurerm_virtual_network.virtual_network will be created
+ resource "azurerm_virtual_network" "virtual_network" {
+ address_space = [
+ "10.0.0.0/16",
]
+ dns_servers = [
+ "10.0.0.4",
+ "10.0.0.5",
]
+ guid = (known after apply)
+ id = (known after apply)
+ location = "westus2"
+ name = "mage-data-prep-production"
+ resource_group_name = "mage-data-prep-production"
+ subnet = (known after apply)
+ tags = {
+ "Environment" = "production"
}
}
Plan: 14 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ id = (known after apply)
+ ip = (known after apply)
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
```
# Deploy to Azure with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/azure/setup
## Overview
Here is a high level diagram of how Mage interacts with various Azure resources deployed using
this Terraform template.
## Pre-requisites
* Download Mage maintained [Terraform templates](/production/deploying-to-cloud/using-terraform).
***
## 1. Install Azure CLI
Follow these
[instructions](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) to
install the Azure CLI on your workstation.
***
## 2. Log into Azure from CLI
If you donโt already have an account in Azure, create one
[here](https://portal.azure.com/).
Once you created an account, from your terminal run the following command to log
in:
```bash theme={"system"}
az login
```
Your browser will open and have you sign in with an existing Azure account. Once
completed, your terminal will output something like this:
```json theme={"system"}
[
{
"cloudName": "AzureCloud",
"homeTenantId": "7712a6c4-1684-402f-8d1a-f1bb656e5d40",
"id": "d17fcd32-b98c-4f81-a881-42b8e811a0e8",
"isDefault": true,
"managedByTenants": [],
"name": "Azure subscription 1",
"state": "Enabled",
"tenantId": "7712a6c4-1684-402f-8d1a-f1bb656e5d40",
"user": {
"name": "eng@mage.com",
"type": "user"
}
}
]
```
***
## 3. Customize Terraform settings
**Storage account name (REQUIRED)**
The `storage_account_name` must be unique globally. Before running any Terraform
commands, please change the `default` value of variable named
`storage_account_name` in the
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf)
file.
```js theme={"system"}
variable "storage_account_name" {
description = "Storage account name. It must be globally unique across Azure."
default = "something_very_unique"
}
```
**Key vault name (REQUIRED)**
The `key_vault_name` must be unique globally. Before running any Terraform
commands, please change the `default` value of variable named `key_vault_name`
in the
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf)
file.
```js theme={"system"}
variable "key_vault_name" {
description = "Key vault name. It must be globally unique across Azure."
default = "something_very_unique"
}
```
**Virtual network name**
In the file
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf),
you can change the `default` value under `app_name`:
```js theme={"system"}
variable "app_name" {
type = string
description = "Application Name"
default = "mage-data-prep"
}
```
**Docker image**
In the file
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf),
you can change the `default` value under `docker_image`:
```js theme={"system"}
variable "docker_image" {
description = "Docker image url."
default = "mageai/mageai:latest"
}
```
**Custom Docker images**
If you previously tagged a Docker image you built when following this
[CI/CD guide](/production/ci-cd), you must push that locally tagged Docker image
to Azure Container Registry before deploying using Terraform.
**Region**
In the file
[./azure/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/main.tf),
you can change the `location` value under `resource_group`:
```js theme={"system"}
resource "azurerm_resource_group" "resource_group" {
name = "${var.app_name}-${var.app_environment}"
location = "West US 2"
}
```
### More
Other variables defined in
[./azure/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/variables.tf)
can also be customized to your needs.
### Environment variables for application
Set your environment variables in your running cloud environment by adding the
following under the resource named `azurerm_container_group` in the file
[./azure/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/azure/main.tf):
```js theme={"system"}
resource "azurerm_container_group" "container_group" {
...
container {
...
environment_variables = {
...
"ENVIRONMENT_VARIABLE_NAME" = "ENVIRONMENT_VARIABLE_VALUE"
}
}
}
```
***
## 4. Deploy
1. Change directory into scripts folder:
```bash theme={"system"}
cd azure
```
2. Initialize Terraform
```bash theme={"system"}
terraform init
```
A sample output could look like this:
```
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/azurerm...
- Finding latest version of hashicorp/http...
- Installing hashicorp/azurerm v3.24.0...
- Installed hashicorp/azurerm v3.24.0 (signed by HashiCorp)
- Installing hashicorp/http v3.1.0...
- Installed hashicorp/http v3.1.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```
### Terraform plan
You can run the following command to see all the resources that will be created
by Terraform: `terraform plan`
By default, here are the **[resources](/production/deploying-to-cloud/azure/resources)** that will be
created.
3. Deploy
```bash theme={"system"}
terraform apply
```
A sample output could look like this:
```
Apply complete! Resources: 4 added, 2 changed, 0 destroyed.
Outputs:
id = "/subscriptions/d17fcd12-b89c-4f81-a221-40b8e768a0e8/resourceGroups/mage-data-prep2-production/providers/Microsoft.ContainerInstance/containerGroups/mage-data-prep-production"
ip = "20.3.85.62"
```
4. In your browser, go to [`http://[IP_address]/`](http://IP_address/).
Change the `IP_address` to the IP address that was output in your terminal
after successfully running `terraform apply`.
***
## Updating Mage versions
After Mage is running in your cloud, you can update the Mage Docker image
version by running the following command in your CLI tool:
```bash theme={"system"}
az container restart --name mage-data-prep-production --resource-group mage-data-prep-production
```
***
## Misc
### Security
By default, `terraform apply` will attempt to add your current workstationโs IP
address to the above mentioned security group.
In order to access Mage on Azure from another local workstation, you must add
that IP address to the security group named
`mage-data-prep-production-nsg-public`.
> Security group name
>
> Your security group might be named differently because you may have changed
> your virtual network name (e.g. `app_name`). The security group you must add
> the IP to will end with `-nsg-public`.
### HTTPS enabling
TBD
### Terminate all resources
If you want to delete all resources created from deploying, run the following
command:
```bash theme={"system"}
terraform destroy
```
A sample output could look like this:
```
Destroy complete! Resources: 10 destroyed.
```
***
# Deploy to DigitalOcean with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/digitalocean/setup
## Pre-requisites
### Setup
[Download Mage maintained Terraform scripts](/production/deploying-to-cloud/using-terraform).
## Terraform plan
You can run the following command to see all the resources that will be created
by Terraform:
```bash theme={"system"}
cd
terraform plan
```
***
## 1. Configure access token
Generate DigitalOcean access token by following the guide: [https://developer.hashicorp.com/terraform/tutorials/applications/digitalocean-provider#generate-digitalocean-access-token](https://developer.hashicorp.com/terraform/tutorials/applications/digitalocean-provider#generate-digitalocean-access-token)
Set the token in `DIGITALOCEAN_ACCESS_TOKEN` environment variable
***
## 2. Create ssh key
Create ssh key with command: `ssh-keygen`. In the file
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf), update the value of `ssh_pub_key` to your path of public key.
***
## 3. Customize Terraform settings
**Application name (optional)**
In the file
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf),
you can change the default application name that will appear in DigitalOcean:
```
variable "app_name" {
type = string
description = "Application Name"
default = "mage-data-prep"
}
```
**Region (optional)**
In the file
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf),
you can change the region:
```
variable "region" {
type = string
description = "Region of the datacenters"
default = "sfo3"
}
```
**More**
Other variables defined in
[./digitalocean/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/digitalocean/variables.tf)
can also be customized to your needs.
***
## 4. Provision resources
**Using CLI**
1. Change directory into scripts folder:
```bash theme={"system"}
cd digitalocean
```
2. Initialize Terraform:
```bash theme={"system"}
terraform init
```
3. Deploy:
```bash theme={"system"}
terraform apply
```
Once itโs finished deploying, you can get the IP from the output.
## 5. Run app in the provisioned Droplet
SSH into the instance: `ssh -i [path_to_private_key] root@[ip]`
Run command to start Mage app in :
```bash theme={"system"}
docker run -d -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start default_repo
```
To update Mage to latest version, you can pull latest docker image with command:
```bash theme={"system"}
docker pull mageai/mageai:latest
```
Then, just restart the Mage app with the same `docker run` command above.
## 6. Open browser with Mage
Once youโve launched the Droplet and started Mage, you can access Mage from your web browser by entering the following:
`http://[ip]:[port]`
Change the IP to match the IP of your Droplet and the port that youโre using for Mage.
***
## Misc
### Terminate all resources
If you want to delete all resources created from deploying, run the following
command:
```bash theme={"system"}
terraform destroy
```
A sample output could look like this:
```
Destroy complete! Resources: 10 destroyed.
```
# Push Docker Image to GCP Artifact Registry
Source: https://docs.mage.ai/production/deploying-to-cloud/gcp/gcp-artifact-registry
While you may pull an image directly from Docker Hub, here are our legacy instructions for pushing a Docker image to GCP Artifact Registry:
1. Create a repository on
[GCP Artifact Registry](https://console.cloud.google.com/artifacts) by
clicking the **`+ CREATE REPOSITORY`** button.
* Fill in the **Name**
* Choose Docker as the **Format**
* Choose any **Location type**
* Choose any **Encryption**
* Click **`CREATE`**
2. Click on the newly created repository (e.g. `mage-docker`).
3. Near the top of the page, click the link **`SETUP INSTRUCTIONS`** or read these
[instructions](https://cloud.google.com/artifact-registry/docs/docker/authentication)
to set up authentication for Docker.
* Run the following command in your terminal:
```bash theme={"system"}
gcloud auth configure-docker [region]-docker.pkg.dev
```
* An example command could look like this:
```bash theme={"system"}
gcloud auth configure-docker us-west2-docker.pkg.dev
```
4. Pull the Mage Docker image:
```bash theme={"system"}
docker pull mageai/mageai:latest
```
* If your local workstation is using macOS and a silicon chip (e.g. M1, M2,
etc), then run this command instead:
```bash theme={"system"}
docker pull --platform linux/amd64 mageai/mageai:latest
```
5. Tag the pulled Mage docker image or use a previously tagged Docker image you
built when following this [CI/CD guide](/production/ci-cd):
* Sample commands if using vanilla Mage Docker image:
```bash theme={"system"}
docker tag mageai/mageai:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest
```
* An example command could look like this:
```bash theme={"system"}
docker tag mageai/mageai:latest us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest
```
* Sample commands if using previously tagged a custom Docker image using the tag `mageprod:latest`:
```bash theme={"system"}
docker tag mageprod:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest
```
* An example command could look like this if you previously tagged your custom Docker image with the tag `mageprod:latest`:
```bash theme={"system"}
docker tag mageprod:latest us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest
```
6. Push the local Docker image to GCP Artifact Registry:
```bash theme={"system"}
docker push [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest
```
* An example command could look like this:
```bash theme={"system"}
docker push us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest
```
# Terraform GCP resources
Source: https://docs.mage.ai/production/deploying-to-cloud/gcp/resources
Here is the default output of `terraform plan`:
```bash theme={"system"}
data.http.myip: Reading...
data.http.myip: Read complete after 0s [id=http://ipv4.icanhazip.com]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# google_cloud_run_service.run_service will be created
+ resource "google_cloud_run_service" "run_service" {
+ autogenerate_revision_name = true
+ id = (known after apply)
+ location = "us-west2"
+ name = "mage-data-prep"
+ project = (known after apply)
+ status = (known after apply)
+ metadata {
+ annotations = {
+ "run.googleapis.com/ingress" = "internal-and-cloud-load-balancing"
+ "run.googleapis.com/launch-stage" = "BETA"
}
+ generation = (known after apply)
+ labels = (known after apply)
+ namespace = (known after apply)
+ resource_version = (known after apply)
+ self_link = (known after apply)
+ uid = (known after apply)
}
+ template {
+ metadata {
+ annotations = (known after apply)
+ generation = (known after apply)
+ name = (known after apply)
+ namespace = (known after apply)
+ resource_version = (known after apply)
+ self_link = (known after apply)
+ uid = (known after apply)
}
+ spec {
+ container_concurrency = (known after apply)
+ service_account_name = (known after apply)
+ serving_state = (known after apply)
+ timeout_seconds = (known after apply)
+ containers {
+ image = "region-docker.pkg.dev/project_id/repository_name/mageai"
+ env {
+ name = "FILESTORE_IP_ADDRESS"
+ value = (known after apply)
}
+ env {
+ name = "FILE_SHARE_NAME"
+ value = "share1"
}
+ ports {
+ container_port = 6789
+ name = (known after apply)
}
+ resources {
+ limits = {
+ "cpu" = "2000m"
+ "memory" = "1G"
}
}
}
}
}
+ traffic {
+ latest_revision = true
+ percent = 100
+ url = (known after apply)
}
}
# google_cloud_run_service_iam_member.run_all_users will be created
+ resource "google_cloud_run_service_iam_member" "run_all_users" {
+ etag = (known after apply)
+ id = (known after apply)
+ location = "us-west2"
+ member = "allUsers"
+ project = (known after apply)
+ role = "roles/run.invoker"
+ service = "mage-data-prep"
}
# google_compute_backend_service.backend will be created
+ resource "google_compute_backend_service" "backend" {
+ connection_draining_timeout_sec = 300
+ creation_timestamp = (known after apply)
+ fingerprint = (known after apply)
+ id = (known after apply)
+ load_balancing_scheme = "EXTERNAL"
+ name = "mage-data-prep-backend"
+ port_name = "http"
+ project = (known after apply)
+ protocol = "HTTP"
+ security_policy = "mage-data-prep-security-policy"
+ self_link = (known after apply)
+ session_affinity = (known after apply)
+ timeout_sec = 30
+ backend {
+ balancing_mode = "UTILIZATION"
+ capacity_scaler = 1
+ group = (known after apply)
+ max_connections = (known after apply)
+ max_connections_per_endpoint = (known after apply)
+ max_connections_per_instance = (known after apply)
+ max_rate = (known after apply)
+ max_rate_per_endpoint = (known after apply)
+ max_rate_per_instance = (known after apply)
+ max_utilization = (known after apply)
}
+ cdn_policy {
+ cache_mode = (known after apply)
+ client_ttl = (known after apply)
+ default_ttl = (known after apply)
+ max_ttl = (known after apply)
+ negative_caching = (known after apply)
+ serve_while_stale = (known after apply)
+ signed_url_cache_max_age_sec = (known after apply)
+ cache_key_policy {
+ include_host = (known after apply)
+ include_named_cookies = (known after apply)
+ include_protocol = (known after apply)
+ include_query_string = (known after apply)
+ query_string_blacklist = (known after apply)
+ query_string_whitelist = (known after apply)
}
+ negative_caching_policy {
+ code = (known after apply)
+ ttl = (known after apply)
}
}
+ log_config {
+ enable = (known after apply)
+ sample_rate = (known after apply)
}
}
# google_compute_global_address.ip will be created
+ resource "google_compute_global_address" "ip" {
+ address = (known after apply)
+ creation_timestamp = (known after apply)
+ id = (known after apply)
+ name = "mage-data-prep-service-ip"
+ project = (known after apply)
+ self_link = (known after apply)
}
# google_compute_global_forwarding_rule.frontend will be created
+ resource "google_compute_global_forwarding_rule" "frontend" {
+ id = (known after apply)
+ ip_address = (known after apply)
+ ip_protocol = (known after apply)
+ label_fingerprint = (known after apply)
+ load_balancing_scheme = "EXTERNAL"
+ name = "mage-data-prep-frontend"
+ network = (known after apply)
+ port_range = "80"
+ project = (known after apply)
+ psc_connection_id = (known after apply)
+ psc_connection_status = (known after apply)
+ self_link = (known after apply)
+ target = (known after apply)
}
# google_compute_region_network_endpoint_group.cloudrun_neg will be created
+ resource "google_compute_region_network_endpoint_group" "cloudrun_neg" {
+ id = (known after apply)
+ name = "mage-data-prep-neg"
+ network_endpoint_type = "SERVERLESS"
+ project = (known after apply)
+ region = "us-west2"
+ self_link = (known after apply)
+ cloud_run {
+ service = "mage-data-prep"
}
}
# google_compute_security_policy.policy will be created
+ resource "google_compute_security_policy" "policy" {
+ fingerprint = (known after apply)
+ id = (known after apply)
+ name = "mage-data-prep-security-policy"
+ project = (known after apply)
+ self_link = (known after apply)
+ type = (known after apply)
+ advanced_options_config {
+ json_parsing = (known after apply)
+ log_level = (known after apply)
+ json_custom_config {
+ content_types = (known after apply)
}
}
+ rule {
+ action = "allow"
+ description = "Whitelist IP"
+ preview = (known after apply)
+ priority = 100
+ match {
+ versioned_expr = "SRC_IPS_V1"
+ config {
+ src_ip_ranges = [
+ "11.111.11.111/32",
]
}
}
}
+ rule {
+ action = "deny(403)"
+ description = "default rule"
+ preview = (known after apply)
+ priority = 2147483647
+ match {
+ versioned_expr = "SRC_IPS_V1"
+ config {
+ src_ip_ranges = [
+ "*",
]
}
}
}
}
# google_compute_target_http_proxy.http_proxy will be created
+ resource "google_compute_target_http_proxy" "http_proxy" {
+ creation_timestamp = (known after apply)
+ id = (known after apply)
+ name = "mage-data-prep-http-proxy"
+ project = (known after apply)
+ proxy_bind = (known after apply)
+ proxy_id = (known after apply)
+ self_link = (known after apply)
+ url_map = (known after apply)
}
# google_compute_url_map.url_map will be created
+ resource "google_compute_url_map" "url_map" {
+ creation_timestamp = (known after apply)
+ default_service = (known after apply)
+ fingerprint = (known after apply)
+ id = (known after apply)
+ map_id = (known after apply)
+ name = "mage-data-prep-urlmap"
+ project = (known after apply)
+ self_link = (known after apply)
}
# google_filestore_instance.instance will be created
+ resource "google_filestore_instance" "instance" {
+ create_time = (known after apply)
+ etag = (known after apply)
+ id = (known after apply)
+ location = "us-west2-a"
+ name = "mage-data-prep"
+ project = (known after apply)
+ tier = "BASIC_HDD"
+ zone = (known after apply)
+ file_shares {
+ capacity_gb = 1024
+ name = "share1"
}
+ networks {
+ connect_mode = "DIRECT_PEERING"
+ ip_addresses = (known after apply)
+ modes = [
+ "MODE_IPV4",
]
+ network = "default"
+ reserved_ip_range = (known after apply)
}
}
# google_project_service.artifactregistry will be created
+ resource "google_project_service" "artifactregistry" {
+ disable_on_destroy = false
+ id = (known after apply)
+ project = (known after apply)
+ service = "artifactregistry.googleapis.com"
}
# google_project_service.cloudrun will be created
+ resource "google_project_service" "cloudrun" {
+ disable_on_destroy = false
+ id = (known after apply)
+ project = (known after apply)
+ service = "run.googleapis.com"
}
# google_project_service.iam will be created
+ resource "google_project_service" "iam" {
+ disable_on_destroy = false
+ id = (known after apply)
+ project = (known after apply)
+ service = "iam.googleapis.com"
}
# google_project_service.resourcemanager will be created
+ resource "google_project_service" "resourcemanager" {
+ disable_on_destroy = false
+ id = (known after apply)
+ project = (known after apply)
+ service = "cloudresourcemanager.googleapis.com"
}
# google_project_service.vpcaccess will be created
+ resource "google_project_service" "vpcaccess" {
+ disable_on_destroy = false
+ id = (known after apply)
+ project = (known after apply)
+ service = "vpcaccess.googleapis.com"
}
# google_vpc_access_connector.connector will be created
+ resource "google_vpc_access_connector" "connector" {
+ id = (known after apply)
+ ip_cidr_range = "10.8.0.0/28"
+ max_throughput = 300
+ min_throughput = 200
+ name = "mage-data-prep-connector"
+ network = "default"
+ project = (known after apply)
+ region = "us-west2"
+ self_link = (known after apply)
+ state = (known after apply)
}
Plan: 16 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ service_ip = (known after apply)
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
```
# Deploy to GCP with Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/gcp/setup
## Pre-requisites
### Setup
[Download Mage maintained Terraform scripts](/production/deploying-to-cloud/using-terraform).
### Permissions
Go to the IAM management dashboard, find the service account associated to the
account you just logged into, and then add these roles to that service account
(e.g. choose your account as the principal when adding new roles):
1. Artifact Registry Read
2. Artifact Registry Writer
3. Cloud Run Developer
4. Cloud SQL
5. Service Account Token Creator
***
## Terraform plan
You can run the following command to see all the resources that will be created
by Terraform:
```bash theme={"system"}
cd gcp
```
```bash theme={"system"}
terraform plan
```
By default, here are the **[resources](/production/deploying-to-cloud/gcp/resources)** that will be
created.
***
## 1. Install `gcloud` CLI
Follow these [instructions](https://cloud.google.com/sdk/docs/install) to
install the CLI on your workstation.
For macOS and Homebrew, you can do the following:
```bash theme={"system"}
brew install --cask google-cloud-sdk
```
***
## 2. Log into GCP from CLI
If you donโt already have an account in GCP, create one
[here](https://cloud.google.com/).
Once you created an account, from your terminal run the following command to log
in:
```bash theme={"system"}
gcloud init
```
```bash theme={"system"}
gcloud auth application-default login
```
## 3. Customize Terraform settings
**Project ID (REQUIRED)**
Before running any Terraform commands, please change the `default` value of the
variable named `project_id` in the
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
file.
```js theme={"system"}
variable "project_id" {
type = string
description = "The name of the project"
default = "unique-gcp-project-id"
}
```
**Docker Image (REQUIRED)**
Google Cloud Run supports pulling Docker images directly from Docker Hub. To use the latest Mage image (default):
```js theme={"system"}
variable "docker_image" {
type = string
description = "The docker image to deploy to Cloud Run."
default = "mageai/mageai:latest"
}
```
Simply change the tag to adjust the version of Mage you want to deploy. If you need to build your own image, you can push it to Docker Hub or follow our guide to use [GCP Artifact Registry](/production/deploying-to-cloud/gcp/gcp-artifact-registry).
**Application Name**
In the file
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf),
you can change the `default` value under `app_name`:
```js theme={"system"}
variable "app_name" {
type = string
description = "Application Name"
default = "mage-data-prep"
}
```
**Region**
In the file
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf),
you can change the `default` value under `region`:
```js theme={"system"}
variable "region" {
type = string
description = "The default compute region"
default = "us-west2"
}
```
### Environment variables for application
Set your environment variables in your running cloud environment by adding the
following under the resource named `google_cloud_run_service` in the file
[./gcp/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf):
```js theme={"system"}
resource "google_cloud_run_service" "run_service" {
...
template {
spec {
containers {
...
env {
name = "NAME_OF_ENVIRONMENT_VARIABLE"
value = "value of environment variable"
}
}
}
}
}
```
### Secrets
See our [docs](/production/deploying-to-cloud/secrets/GCP) for configuring secrets in Google Secret Manager and using secrets locally while developing.
#### Method A: Terraform configurations
1. Once you save a secret in Google Secret Manager, click on the
**PERMISSIONS** tab.
2. Click the button **`+ GRANT ACCESS`**.
3. Under the field labeled **New principles**, add the service account that is
associated to your Google Cloud Run. If you canโt find this, try deploying
GCP with Terraform and youโll receive an error that looks like this:
```
โท
โ Error: resource is in failed state "Ready:False", message: Revision 'mage-2992l' is not ready and cannot serve traffic. spec.template.spec.volumes[1].secret.items[0].key: Permission denied on secret: projects/1054912425479/secrets/bigquery_credentials/versions/latest for Revision service account 1054912425479-compute@developer.gserviceaccount.com. The service account used must be granted the 'Secret Manager Secret Accessor' role (roles/secretmanager.secretAccessor) at the secret, project or higher level.
โ
โ with google_cloud_run_service.run_service,
โ on main.tf line 88, in resource "google_cloud_run_service" "run_service":
โ 88: resource "google_cloud_run_service" "run_service" {
โ
โต
```
If you still get this error when re-running `terraform apply` even after granting access,
try **Method B** down below.
By default, the service account used by Cloud Run that needs access to the secret is the `Compute Engine` default service account.
If you want to grant access to a different service account,
configure the `service_account_name` argument in the `spec` block in
[`main.tf`](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf#L99):
```
# Create the Cloud Run service
resource "google_cloud_run_service" "run_service" {
name = var.app_name
location = var.region
template {
spec {
service_account_name = "[your service account email here]"
}
}
}
```
4. Under the field labeled **Select a role**, enter the value
`Secret Manager Secret Accessor`.
5. Click the button **SAVE**.
6. Mount secrets to Google Cloud Run via Terraform in the file
[./gcp/main.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/main.tf):
```
resource "google_cloud_run_service" "run_service" {
...
template {
spec {
containers {
...
env {
name = "path_to_keyfile"
value = "/secrets/bigquery/bigquery_credentials"
}
volume_mounts {
name = "secrets-bigquery_credentials"
mount_path = "/secrets/bigquery"
}
}
volumes {
name = "secrets-bigquery_credentials"
secret {
secret_name = "bigquery_credentials"
items {
key = "latest"
path = "bigquery_credentials"
}
}
}
}
}
}
```
7. In your Python code or in any field that support variable interpolation,
enter the following to get the value from the secret: 1
```python Python theme={"system"}
import os
with open(os.getenv('path_to_keyfile'), 'r') as f:
print(f.read())
```
```yaml YAML theme={"system"}
demo:
target: prod
outputs:
prod_bigquery:
dataset: mage_demo
keyfile: "{{ env_var('path_to_keyfile') }}"
method: service-account
project: my_cool_gcp_project
threads: 1
type: bigquery
```
#### Method B: through GCP UI (aka console but not to be confused with CLI)
1. Go to your Google Cloud Run dashboard, find the service running Mage, and
click on it to view its details.
2. Follow these
[Google provided instructions](https://cloud.google.com/run/docs/configuring/secrets#mounting-secrets-service)
to manually mount the secret to the running Mage Cloud Run service and grant
the right permissions for that service to access those secrets.
After you mount the secrets, you can create environment variables in your Cloud Run that reference
your secrets. Follow the below example to setup the mounted secret and environment variable for that
secret:
| Secrets | |
| ---------------- | ---------------------- |
| Secret name | `bigquery_credentials` |
| Reference method | `Mounted as volume` |
| Mount path | `/secrets/bigquery` |
| Path 1 | `bigquery_credentials` |
| Version 1 | `latest` |
| Environment variables | |
| ---------------------------- | ---------------------------------------- |
| Name | `path_to_keyfile` |
| Value | `/secrets/bigquery/bigquery_credentials` |
### More
Other variables defined in
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
can also be customized to your needs.
## 4. Deploy
1. Change directory into scripts folder:
```bash theme={"system"}
cd gcp
```
2. Initialize Terraform:
```bash theme={"system"}
terraform init
```
A sample output could look like this:
```
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/google versions matching ">= 3.3.0"...
- Finding latest version of hashicorp/http...
- Installing hashicorp/google v4.38.0...
- Installed hashicorp/google v4.38.0 (signed by HashiCorp)
- Installing hashicorp/http v3.1.0...
- Installed hashicorp/http v3.1.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```
3. Deploy:
```bash theme={"system"}
terraform apply
```
A sample output could look like this:
```
Apply complete! Resources: 7 added, 1 changed, 0 destroyed.
Outputs:
service_ip = "34.107.187.208"
```
Itโll take a few minutes for GCP Cloud Run to start up and be ready to
receive requests.
After a few minutes, open a browser and go to `http://[IP_address]`
Change the `IP_address` to the IP address that was output in your terminal
after successfully running `terraform apply`.
### Errors
#### 403 Forbidden
If you get this error when trying to open the above IP address in your browser,
open the security group named
[`[application_name]-security-policy`](https://console.cloud.google.com/net-security/securitypolicies/list).
Click on that security group and verify your IP address is whitelisted.
If it isnโt, add a new rule with the following values:
| Mode | Match | Action | Priority |
| ---------- | ------------------------------------------------------- | ------ | -------- |
| Basic mode | [Enter your IP address](https://whatismyipaddress.com/) | Allow | 100 |
If you still get 403, check to see if youโre using `http://` and NOT `https://`.
#### Container canโt start
If Cloud Run is deployed but the service failed because the container couldnโt start,
it is usually caused by the Docker image not being built using the correct platform tag.
Pull the Mage Docker image using this command:
```
docker pull --platform linux/amd64 mageai/mageai:latest
```
Once pulled, tag your image and push to GCP Artifact Registry again.
***
## Updating Mage versions
After Mage is running in your cloud, you can update the Mage Docker image
version by running the following command in your CLI tool:
```bash theme={"system"}
gcloud run deploy [app_name] --image [docker_image]
```
> `app_name`
>
> This is the value you changed when editing the
> [./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
> file.
***
## Misc
### Security
To enable other IP addresses access to Mage, open the security group named
[`[application_name]-security-policy`](https://console.cloud.google.com/net-security/securitypolicies/list).
Click on that security group and add a new rule with the following values:
| Mode | Match | Action | Priority |
| ---------- | ------------------------------------------------------- | ------ | -------- |
| Basic mode | [Enter your IP address](https://whatismyipaddress.com/) | Allow | 100 |
To enable IP addresses access to specific endpoints, you can follow
[this example](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/load_balancer.tf#L23-L33)
to add rules to security policy. Another way is to add a new rule through UI
with the following values:
| Mode | Match | Action | Priority |
| ------------- | ----------------------------------------------------------------------------------------------------------------- | ------ | -------- |
| Advanced mode | `request.path.startsWith('/api/pipeline_schedules/123/pipeline_runs') && inIpRange(origin.ip, '[IP address]/32')` | Allow | 200 |
### HTTPS enabling
To enable HTTPS for Mage app deployed on GCP, you need to firstly make sure you
have a domain. Then follow the steps below to set up HTTPS:
1. In
[./gcp/variables.tf](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/gcp/variables.tf)
file, update the `default` value of `ssl` to be `true`, and set the `default`
value of `domain` variable to be the domain url you want to use.
2. Run `terraform apply` to create the HTTPS load balancer.
3. Get the `service_ip` from the output of terraform command.
4. Update the DNS A and AAAA records to point the domain url to the load
balancer's IP address:
[https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs#update-dns](https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs#update-dns)
5. Visit the domain url to access your Mage app.
### Terminate all resources
If you want to delete all resources created from deploying, run the following
command:
```bash theme={"system"}
terraform destroy
```
A sample output could look like this:
```
Destroy complete! Resources: 10 destroyed.
```
***
## dbt
For information about how to serve dbt docs in production,
[see here](/dbt/docs).
# AWS Secrets Manager
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/AWS
Attention Magers: This classified document contains vital intel on securing sensitive data within the AWS Secrets Manager vault. Study the protocols carefully to create and operationalize confidential secrets, granting you clearance to integrate them into your Mage data pipeline projects.
.gif)
### **Creating a New Secret in AWS Secrets Manager**
Before you can use AWS Secrets Manager with Mage, you need to create a new secret in the AWS Secrets Manager console. Follow these steps:
1. Open the AWS Secrets Manager [console](https://console.aws.amazon.com/secretsmanager/)
2. Click "Store a new secret."
3. On the "Choose Secret Type" page, select the appropriate secret type for your use case, and then click "Next."
4. On the "Configure Secret" page, provide the necessary details for your secret, such as the key-value pairs or the plaintext value, and then click "Next."
5. (Optional) On the "Configure Rotation" page, you can enable automatic rotation for your secret. Automatic rotation periodically updates the secret value, ensuring that your sensitive data remains secure. Configure the rotation settings according to your requirements, and then click "Next."
6. Review the secret details on the "Review" page, and if everything looks correct, click "Store."
Your new secret is now stored in AWS Secrets Manager. Make a note of the secret's ARN (Amazon Resource Name), as you'll need it to access the secret from Mage.
For more detailed information on managing and storing secrets in AWS Secrets Manager, refer to the official [AWS Documentation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/managing-secrets.html)
### Working with Secrets in Mage
Mage works with AWS secrets in two ways: directly accessing them through Python code, and referencing them in YAML configuration files using placeholders that Mage resolves at runtime.
**Python:**
To use secrets in Python code, read theseย [**instructions**](https://docs.mage.ai/design/data-loading#aws-secret-loader).
**YAML:**
Mage allows you to add your secrets directly through the UI. Hereโs how:
1. You can manage secrets by clicking on the **Secrets** icon in the side navigation, or from the pipeline edit page by selecting the **Secrets** tab in the sidekick.
2. Enter each secret by clicking the **New** or **New secret** button:
* **AWS\_ACCESS\_KEY\_ID:** A unique identifier that forms part of the credentials required to authenticate with AWS services. (not needed if using IAM role to authenticate)
* **AWS\_SECRET\_ACCESS\_KEY:** The secret key that, combined with the access key ID, is used to cryptographically sign requests to AWS services. (not needed if using IAM role to authenticate)
* **AWS\_DEFAULT\_REGION:** The default AWS region to use for service API calls when not explicitly specified.(orย **AWS\_REGION\_NAME**)
3. Make the changes as pictured below to represent your secrets in the io\_config.yml file
4. Each secret is now stored and referenced for configuration to AWS
By following these steps, you can securely store and reference your AWS credentials and configuration through the Mage UI, allowing you to seamlessly integrate with AWS services like Secrets Manager throughout your data pipelines.
Add Secrets directly into your environment variables through the io\_config.yml file or your docker compose file. You can also add secrets in a .env file using a similar strategy.
To use secrets in YAML files (e.g. in data integration pipelines):
1. Add the following keys and values to your environment variables
1. **AWS\_ACCESS\_KEY\_ID:** A unique identifier that forms part of the credentials required to authenticate with AWS services. (not needed if using IAM role to authenticate)
2. **AWS\_SECRET\_ACCESS\_KEY:** The secret key that, combined with the access key ID, is used to cryptographically sign requests to AWS services. (not needed if using IAM role to authenticate)
3. **AWS\_DEFAULT\_REGION:** The default AWS region to use for service API calls when not explicitly specified.(orย **AWS\_REGION\_NAME**)
2. Store a secret in AWS Secrets Manager.
3. Use the following syntax in your YAML file to interpolate secret values from AWS Secrets Manager:
```yaml theme={"system"}
"{{ aws_secret_var('some_name_for_secret') }}"
```
For Example
```yaml theme={"system"}
api_key: "{{ aws_secret_var('API_KEY') }}"
access_token: "{{ aws_secret_var('ACCESS_TOKEN') }}"
```
4. Make the changes as pictured below to represent your secrets in the io\_config.yml file
5. Each secret is now stored and referenced for configuration to AWS
By following these steps, you can securely store and reference your AWS credentials and configuration through the Mage UI, allowing you to seamlessly integrate with AWS services like Secrets Manager throughout your data pipelines.
### **Working with AWS Secrets Manager**
.gif)
If you need a more secure environment for handling sensitive data, you can directly integrate your YAML files with AWS Secrets Manager. For example, to reference a secret stored in AWS Secrets Manager with the ARN, you would use the following syntax in your YAML file:
```yaml theme={"system"}
AWS_SECRET_ACCESS_KEY = arn:aws:secretsmanager:::secret:
```
By integrating with AWS Secrets Manager, you can ensure that your sensitive data is securely stored and accessed throughout your Mage data pipelines.
1. **AWS\_SECRET\_ACCESS\_KEY:** The secret value you want get from AWS Secrets Manager
2. **arn:** This is a prefix that identifies the resource as an ARN.
3. **aws:** This is the partition that the resource is in. In this case, it's the AWS partition.
4. **secretsmanager:** This is the service namespace, which identifies the AWS service. In this case, it's the AWS Secrets Manager service.
5. **AWS Region:** This is the AWS region where the secret is stored (i.e. us-west-2).
6. **AWS Account ID:** This is the 12-digit AWS account ID that the secret belongs to.
7. **secret** This is the resource type, which indicates that the resource is a secret.
8. **AWS\_SECRET\_ACCESS\_KEY Name:** This is the friendly name or ID of the specific secret you want to reference. It can include alphanumeric characters, hyphens, and underscores.
In the context of the YAML file, this ARN is being used with the syntax to reference the secret value stored in AWS Secrets Manager. The YAML file itself does not contain the actual secret value; instead, it references the secret by its ARN.
# Azure Key Vault
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/Azure
### YAML
To use secrets in YAML files (e.g. in data integration pipelines):
1. Add the following keys and values to your environment variables so that
Mage can access the Azure Key Vault:
1. `AZURE_KEY_VAULT_URL`: The Azure Key Vault URI.
2. `AZURE_CLIENT_ID`: Azure Active Directory [client ID](https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-client-application-configuration#client-id).
3. `AZURE_CLIENT_SECRET`: Azure Active Directory [client secret](https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret).
4. `AZURE_TENANT_ID`: Azure Active Directory [tenant ID](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id#find-your-azure-ad-tenant).
2. Store a secret in Azure Key Vault.
3. Use the following syntax in your YAML file to interpolate secret values from Azure Key Vault:
```yaml theme={"system"}
"{{ azure_secret_var('secret_name') }}"
```
For example:
```yaml theme={"system"}
api_key: "{{ azure_secret_var('API_KEY') }}"
access_token: "{{ azure_secret_var('ACCESS_TOKEN') }}"
```
### Python
Here is the example python code to get secrets from Azure Key Vault:
```python theme={"system"}
from mage_ai.services.azure.key_vault.key_vault import get_secret
get_secret('secret_name')
```
# GCP Secret Management
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/GCP
## Creating secrets
1. Go to
[Google Secret Manager UI](https://console.cloud.google.com/security/secret-manager?referrer=search\&authuser=1).
2. Click the button at the top labeled **`+ CREATE SECRET`**.
3. Fill in the name of your secret; e.g. `bigquery_credentials`.
4. Under **Secret value**, upload your service account credentials JSON file or
paste the JSON into the text area labeled **Secret value**.
5. Scroll all the way down and click the button **`CREATE SECRET`**.
You can mount secrets from Google Secret Manager through Terraform configurations or through the Google Console UI.
## Using secrets locally
### Download credentials from GCP UI
1. Download the credentials JSON file from GCP.
2. Run Mage and mount the secrets as a volume in Docker.
Follow these [instructions](/getting-started/setup#using-secret-credential-files-in-docker) to learn how to do this.
3. Here are example code snippets to read from that credentials JSON file:
```python Python theme={"system"}
with open('/home/secrets/gcp_credentials.json', 'r') as f:
print(f.read())
```
```yaml YAML theme={"system"}
demo:
target: prod
outputs:
prod_bigquery:
dataset: mage_demo
keyfile: /home/secrets/gcp_credentials.json
method: service-account
project: my_cool_gcp_project
threads: 1
type: bigquery
```
> Note
>
> This code example assumes your credentials JSON file downloaded from GCP is named
> `gcp_credentials.json` and that the mount path (e.g. `-v`) you used when running Docker
> is `/home/secrets`.
***
### Download credentials using `gcloud` CLI
1. Authenticate locally by running this command in your local terminal:
```
gcloud auth application-default login
```
2. Create a new `.env` file in your Mage project folder with the following values:
```
GOOGLE_APPLICATION_CREDENTIALS="[PATH TO YOUR USER CREDENTIALS, MOST LIKELY: ~/.config/gcloud/application_default_credentials.json]"
GCS_BUCKET=[YOUR DEV BUCKET]
GCLOUD_PROJECT=[YOUR PROJECT]
```
3. Run Mage using Docker and set the environment variable `GOOGLE_APPLICATION_CREDENTIALS`.
Follow these [instructions](/getting-started/setup#environment-variables) to learn how to do this.
For example, set the environment variable to:
```
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/FILE_NAME.json
```
4. Run Mage and mount the secrets as a volume in Docker.
Follow these [instructions](/getting-started/setup#using-secret-credential-files-in-docker) to learn how to do this.
For example:
```
-v ~/.config/gcloud/application_default_credentials.json:/tmp/keys/FILE_NAME.json
```
5. Here is an example code snippet:
```python theme={"system"}
from google.cloud import storage
from pandas import DataFrame
from datetime import datetime
import pytz
import os
โ
โ
@data_exporter
def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
โ
bucket_name = os.getenv('GCS_BUCKET')
โ
now = datetime.utcnow()
pt = pytz.timezone("America/Los_Angeles")
now_pst = pytz.utc.localize(now).astimezone(pt)
โ
object_key = f'test_file_{now_pst.strftime("%Y-%m-%d")}.csv'
โ
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(object_key)
โ
blob.upload_from_string(df.to_csv())
โ
print(f'df uploaded to {bucket}/{object_key}.')
```
## GCP Secret Manager Integration in Mage Pro
Securely manage and inject sensitive credentials into your Mage Pro pipelines using **Google Cloud Secret Manager**. This Pro-only feature helps enterprises meet cloud security and compliance standards by keeping secrets like API keys, database passwords, and tokens outside of source code.
### Required Environment Variables
Before accessing secrets, configure your Mage Pro cluster with the following environment variables:
* `GOOGLE_APPLICATION_CREDENTIALS`:
Full path to your **GCP service account credentials JSON file**.
*You can upload the credentials file directly to the Mage Pro cluster using the file browser interface.*
* `GCP_PROJECT_ID`:
The **Google Cloud Project ID** associated with your Secret Manager secrets.
These credentials must have `Secret Manager Secret Accessor` permissions for the secrets you intend to retrieve.
### How to Use the GCP Secret Variable
You can reference GCP secrets programmatically in Python blocks or declaratively in YAML configs.
#### Python code
Use this approach inside a block in your Mage pipeline:
```python theme={"system"}
from mage_ai.services.gcp.secret_manager.secret_manager import get_secret
# Retrieve a secret by ID from GCP Secret Manager
secret_value = get_secret('secret_id')
```
#### YAML config
To inject a secret dynamically into a YAML config (e.g., for a data source, destination, or authentication setting):
```yaml theme={"system"}
{{gcp_secret_var('secret_id')}}
```
Mage will automatically resolve and substitute the secret value at runtime.
# HashiCorp Vault
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/Vault
Securely manage and inject sensitive credentials into your Mage pipelines using HashiCorp Vault. This feature helps enterprises meet security and compliance standards by keeping secrets like API keys, database passwords, and tokens outside of source code.
## Overview
HashiCorp Vault is a secrets management tool that provides secure storage and access to sensitive data. Mage integrates with Vault to retrieve secrets at runtime, ensuring that sensitive credentials are never stored in your codebase or configuration files.
## Prerequisites
Before using Vault with Mage, ensure you have:
1. A running HashiCorp Vault instance
2. Appropriate permissions to read secrets from Vault
3. A valid Vault token or authentication method
## Configuration
### Required Environment Variables
Configure your Mage environment with the following environment variables:
* `VAULT_URL`: The URL of your Vault server (default: `http://127.0.0.1:8200`)
* `VAULT_TOKEN`: Your Vault authentication token
### Optional Environment Variables
* `VAULT_MOUNT_POINT`: The KV v2 mount point for secrets (default: `secret`)
### Example Environment Setup
```bash theme={"system"}
export VAULT_URL="https://vault.company.com"
export VAULT_TOKEN="hvs.xxxxxxxxxxxxxxxxxxxx"
export VAULT_MOUNT_POINT="secret"
```
## Using Secrets in Mage
### Python Code
Use the Vault integration directly in your Python blocks. The `get_secret()` function accepts two parameters:
* `path`: The secret path in Vault (e.g., `'mage/database'`)
* `key`: Optional specific key within the secret (e.g., `'password'`)
**Syntax:**
```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
# Get a specific key from a secret
value = get_secret('secret/path', 'key_name')
# Get all key-value pairs from a secret
all_data = get_secret('secret/path')
```
**Example:**
```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
# Get a specific key from a secret
db_password = get_secret('mage/database', 'password')
# Get all key-value pairs from a secret
db_config = get_secret('mage/database')
print(f"Database host: {db_config['host']}")
print(f"Database username: {db_config['username']}")
```
### YAML Configuration
Reference Vault secrets in your YAML configuration files using the `vault_secret_var()` function. This function accepts two parameters:
* `path`: The secret path in Vault (e.g., `'mage/database'`)
* `key`: The specific key within the secret (e.g., `'password'`)
**Syntax:**
Use the `"{{ vault_secret_var('secret/path', 'key_name') }}"` syntax in any YAML field that supports variable interpolation:
```yaml theme={"system"}
version: 0.1.1
default:
VARIABLE_NAME: "{{ vault_secret_var('secret/path', 'key_name') }}"
```
**Example 1 - IO Config:**
```yaml theme={"system"}
version: 0.1.1
default:
# PostgreSQL configuration using Vault secrets
POSTGRES_HOST: "{{ vault_secret_var('mage/database', 'host') }}"
POSTGRES_PORT: 5432
POSTGRES_DBNAME: "my_database"
POSTGRES_USER: "{{ vault_secret_var('mage/database', 'username') }}"
POSTGRES_PASSWORD: "{{ vault_secret_var('mage/database', 'password') }}"
```
**Example 2 - Data Integration Pipeline:**
```yaml theme={"system"}
# API source configuration
url: "https://api.example.com/data"
method: GET
headers:
Authorization: "Bearer {{ vault_secret_var('mage/api', 'token') }}"
X-API-Key: "{{ vault_secret_var('mage/api', 'key') }}"
response_parser: data.items
```
## Creating Secrets in Vault
### Using Vault CLI
1. **Authenticate with Vault:**
```bash theme={"system"}
vault auth -method=userpass username=your-username
```
2. **Create a secret:**
```bash theme={"system"}
vault kv put secret/mage/database \
username=dbuser \
password=dbpassword \
host=db.example.com
```
3. **Verify the secret:**
```bash theme={"system"}
vault kv get secret/mage/database
```
### Using Vault UI
1. Navigate to your Vault UI:
* **Local development**: `http://127.0.0.1:8200/ui`
* **Production**: `https://vault.company.com/ui`
2. Go to **Secrets** โ **KV v2**
3. Click **Create secret**
4. Enter the secret path (e.g., `mage/database`)
5. Add key-value pairs for your credentials
6. Click **Save**
## Error Handling
The Vault integration provides specific error handling for common scenarios:
```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
try:
secret_value = get_secret('mage/database', 'password')
except KeyError as e:
print(f"Secret or key not found: {e}")
except PermissionError as e:
print(f"Access denied: {e}")
except Exception as e:
print(f"Vault error: {e}")
```
## Security Best Practices
### 1. Token Management
* Use short-lived tokens when possible
* Implement token rotation
* Store tokens securely (environment variables, not in code)
### 2. Secret Organization
* Use hierarchical paths (e.g., `mage/prod/database`, `mage/dev/database`)
* Follow consistent naming conventions
* Separate secrets by environment
### 3. Access Control
* Implement least-privilege access
* Use Vault policies to restrict access
* Regularly audit secret access
### 4. Monitoring
* Enable Vault audit logs
* Monitor secret access patterns
* Set up alerts for unusual activity
## Troubleshooting
### Common Issues
**1. Authentication Failed**
```
Exception: Unable to authenticate to Vault at https://vault.company.com
```
* Verify your `VAULT_TOKEN` is valid
* Check if the token has expired
* Ensure the Vault URL is correct
**2. Secret Not Found**
```
KeyError: Secret path 'mage/database' not found in Vault
```
* Verify the secret path exists
* Check the mount point configuration
* Ensure you have read permissions
**3. Access Denied**
```
PermissionError: Access denied to secret path 'mage/database'
```
* Check your Vault token permissions
* Verify the secret path is accessible
* Contact your Vault administrator
## Advanced Usage
### Custom Mount Points
If your Vault instance uses a different mount point:
```bash theme={"system"}
export VAULT_MOUNT_POINT="custom-secrets"
```
### Multiple Vault Instances
For different environments, you can create multiple Vault clients:
```python theme={"system"}
import os
from mage_ai.services.hashicorp.vault.vault import Vault
# Override environment variables for specific operations
os.environ['VAULT_URL'] = 'https://vault-prod.company.com'
os.environ['VAULT_TOKEN'] = 'prod-token'
prod_vault = Vault()
```
## Integration Examples
### Database Connection
```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
import psycopg2
# Get database credentials from Vault
db_config = get_secret('mage/database')
# Create database connection
conn = psycopg2.connect(
host=db_config['host'],
port=db_config['port'],
database=db_config['database'],
user=db_config['username'],
password=db_config['password']
)
```
### API Authentication
```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
import requests
# Get API credentials from Vault
api_token = get_secret('mage/api', 'token')
# Make authenticated API request
headers = {'Authorization': f'Bearer {api_token}'}
response = requests.get('https://api.example.com/data', headers=headers)
```
### Cloud Provider Credentials
```python theme={"system"}
from mage_ai.services.hashicorp.vault.vault import get_secret
import boto3
# Get AWS credentials from Vault
aws_creds = get_secret('mage/aws')
# Create AWS client
session = boto3.Session(
aws_access_key_id=aws_creds['access_key_id'],
aws_secret_access_key=aws_creds['secret_access_key'],
region_name=aws_creds['region']
)
s3_client = session.client('s3')
```
## Migration from Other Secret Managers
If you're migrating from other secret management solutions:
### From Mage Built-in Secrets
```python theme={"system"}
# Old way
from mage_ai.data_preparation.shared.secrets import get_secret_value
password = get_secret_value('database_password')
# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')
```
### From Environment Variables
```python theme={"system"}
# Old way
import os
password = os.getenv('DATABASE_PASSWORD')
# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')
```
## Support
For additional help with Vault integration:
* [HashiCorp Vault Documentation](https://www.vaultproject.io/docs)
* [Vault Python Client (hvac) Documentation](https://hvac.readthedocs.io/)
* [Mage Community Forum](https://www.mage.ai/chat)
# Mageโs built-in secret manager
Source: https://docs.mage.ai/production/deploying-to-cloud/secrets/mage
Storing and using secrets through Mage.
In various surfaces in Mage, you may be asked to input config for certain integrations
such as cloud databases or services. In these cases, you may need to input a password
or an api key, but you don't want it to be shown in plain text. To get around this issue,
we created a way to store your secrets in the Mage database.
Your secrets are encrypted before being stored into the Mage database. The encryption key
will be stored in the Mage data folder which by default will be created at `~/.mage_data`. If
you need more secure encryption, we recommend using a secrets manager.
## Secret management
### Creating secrets
To manage your secrets, you can go to the **Secrets** page by clicking on the **Secrets**
icon in the side navigation. This will open the global secrets management page where
you can see all your secrets, create new ones, and delete existing ones.
You can also manage secrets directly from the pipeline edit page. In the sidekick,
you should see a `Secrets` tab. To create a secret, press the `New` button on this tab.
Input a name for the secret and the value of the secret, and press Enter or Return to save.
The secrets can be shared across the project that they are created in.
Pipeline level secrets coming soon...
### Secrets in different environments
Secrets are stored to the Mage database. If you are using the same database across multiple
Mage environments, i.e. development and production, the secrets will not be shared across the environments
unless you shared the same Mage data directory for every environment.
Each secret will be saved to the database with a name and a uuid. The name will be used to identify
the secret, and the uuid is used to identify the environment.
Example:
```
Development environment
project
โโโโmage_data
โ โโโโsecrets
โ โ โ key: abc123
โ โ โ uuid: dev-uuid
Production environment
project
โโโโmage_data
โ โโโโsecrets
โ โ โ key: def456
โ โ โ uuid: prod-uuid
```
Secrets table in database
| Name | Uuid | Value |
| --------- | ----------- | --------------------------- |
| `secret1` | `dev-uuid` | sooper dooper secret string |
| `secret2` | `dev-uuid` | xXxSecreTxXx |
| `secret3` | `dev-uuid` | dev only secret |
| `secret1` | `prod-uuid` | real production secret |
| `secret2` | `prod-uuid` | m4g3r0x! |
| `secret1` | `random` | random secret |
| Command | Development Value | Production Value |
| ----------------------------- | --------------------------- | ---------------------- |
| `get_secret_value('secret1')` | sooper dooper secret string | real production secret |
| `get_secret_value('secret2')` | xXxSecreTxXx | m4g3r0x! |
| `get_secret_value('secret3')` | dev only secret | `None` |
Coming soon:
* secrets scoped to a pipeline
## Using Secrets
You can use the following syntax to have Mage interpolate the secret when reading from the config:
```yaml theme={"system"}
field_name: "{{ mage_secret_var('your_secret_name') }}"
```
You can also fetch the secret value in a Mage code block by importing a helper method:
```python theme={"system"}
from mage_ai.data_preparation.shared.secrets import get_secret_value
get_secret_value('your_secret_name')
```
# Helm
Source: https://docs.mage.ai/production/deploying-to-cloud/using-helm
Deploy Mage to a Kubernetes cluster with Helm. Use the chart's `values.yaml` to customize deployment topology, dependencies, storage, and Mage Pro features.
***
# Prerequisites
## Install Helm
```bash From Homebrew (macOS) theme={"system"}
brew install helm
```
```bash From Apt (Debian/Ubuntu) theme={"system"}
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
```
Full installation guide: [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/)
## Setup a Kubernetes Cluster
### Local
[Enable Kubernetes](https://docs.docker.com/desktop/kubernetes/#enable-kubernetes) in Docker Desktop to start a Kubernetes cluster locally.
Other options for starting a Kubernetes cluster locally:
* [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
* [Minikube](https://minikube.sigs.k8s.io/docs/start/)
### AWS EKS
Follow the [guide](/production/deploying-to-cloud/aws/setup#deploy-to-eks-cluster) to set up the EKS cluster.
### Google Kubernetes Engine (GKE)
Follow the [guide](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#create_cluster) to create a GKE cluster.
***
# Deploy Mage using Helm
Mage Helm repo: [https://mage-ai.github.io/helm-charts/](https://mage-ai.github.io/helm-charts/)
### Add Helm repository
```bash theme={"system"}
helm repo add mageai https://mage-ai.github.io/helm-charts
```
### Install with default config
```bash theme={"system"}
helm install my-mageai mageai/mageai
```
### Get the default values
To customize the deployment, save the chart's default values and edit them:
```bash theme={"system"}
helm show values mageai/mageai > values.yaml
```
Then install or upgrade using your file:
```bash theme={"system"}
helm install --values values.yaml my-mageai mageai/mageai
# or
helm upgrade --install my-mageai mageai/mageai --values values.yaml
```
***
# Customize deployment
The chart supports many options. Key areas are summarized below; for the full reference, see the [Mage Helm Chart โ Values Reference](https://github.com/mage-ai/helm-charts/blob/master/charts/mageai/values.yaml) in the chart repository.
## Deployment topology
* **`replicaCount`** โ Number of Mage replicas (default: `1`). Increase for high availability or more concurrent requests and pipeline runs. When using multiple replicas, you must enable Redis (see [Dependencies](#dependencies)).
* **`standaloneScheduler`** โ When `true`, splits Mage into separate **web server** and **scheduler** deployments so you can scale and schedule them independently. Use `scheduler.replicaCount` and `webServer.replicaCount` instead of `replicaCount` when enabled.
## Dependencies
* **PostgreSQL** (`postgresql.enabled`) โ Required for production: pipelines, triggers, and metadata. Enable and set `auth.username`, `auth.password`, and `auth.database` (or use an external database and configure Mage accordingly).
* **Redis** (`redis.enabled`) โ Required when `replicaCount` > 1, for distributed locking and caching. You can use an external Redis by setting `customRedisURL` and leaving `redis.enabled: false`.
## Container image
* **`image.repository`**, **`image.tag`**, **`image.pullPolicy`** โ Control which Mage image is used (default repository: `mageai/mageai`).
* **`imagePullSecrets`** โ Use for private registries or Mage Pro registry; the secret must exist in the target namespace.
## Storage and volumes
Project code must be available at `/home/src` inside the container. You can provide it in two ways:
1. **`extraVolumes` / `extraVolumeMounts`** (default) โ The chart typically mounts a volume named `mage-fs` at `/home/src`. Point it to your project:
* **Local / hostPath:** set `extraVolumes` to a `hostPath` with your Mage project path.
* **Cloud (EKS, GKE, AKS):** provision a **ReadWriteMany** PersistentVolume (e.g. EFS, Filestore, Azure Files) and a PersistentVolumeClaim, then use that claim in `extraVolumes` / `extraVolumeMounts`.
2. **`persistence.enabled: true`** โ Use the chart's built-in PersistentVolume. Set `persistence.storageClassName` and `persistence.size` (e.g. `5Gi`). For cloud filesystems (e.g. EFS), you can use `persistence.csi` if supported by the chart.
### Example: PVC for project code (EKS / GKE / Azure)
After creating a ReadWriteMany PVC (e.g. [EKS EFS PVC](https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws-eks/kube/efs-pvc.yaml), [GKE Filestore](https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#dynamic_provisioning), or [AKS storage](https://learn.microsoft.com/en-us/azure/aks/concepts-storage#persistent-volume-claims)):
```yaml theme={"system"}
extraVolumeMounts:
- name: mage-fs
mountPath: /home/src
extraVolumes:
- name: mage-fs
persistentVolumeClaim:
claimName: your-pvc-name
```
## Service and ingress
* **`service.type`** โ Default is `LoadBalancer`; set to `ClusterIP` or `NodePort` if needed. **`service.port`** defaults to `6789`.
* **`service.annotations`** โ Use for cloud load balancers (e.g. AWS NLB, GCP).
* **`ingress`** โ Set `ingress.enabled: true` and configure `ingress.className`, `ingress.hosts`, and `ingress.tls` for external access. For production, consider the [Mage-Ingress chart](https://github.com/mage-ai/helm-charts/tree/master/charts/ingress).
## Environment and configuration
* **`config`** โ Key-value environment variables for the Mage container (e.g. `USER_CODE_PATH: /home/src/default_repo`).
* **`secrets`** โ Chart-created secrets injected as env vars. Prefer a secret manager (e.g. Vault, AWS Secrets Manager) in production.
* **`existingSecret`** โ Use an existing Secret for env vars; takes precedence over `secrets` if both are set.
* **`extraEnvs`** โ Additional env vars, including `valueFrom` (e.g. downward API for `metadata.namespace`).
## Resources and scheduling
* **`resources`** โ Set CPU/memory `limits` and `requests` for the Mage container.
* **`nodeSelector`**, **`tolerations`**, **`affinity`** โ Control pod placement.
* **`livenessProbe`** / **`readinessProbe`** โ Default to `/api/status` on the http port; customize or replace with `customLivenessProbe` / `customReadinessProbe`.
* **`hpa`** โ Horizontal Pod Autoscaler (commented out by default).
## Quick reference: common overrides
| Use case | Key values |
| ------------------------ | ------------------------------------------------------------------------------- |
| Production with database | `postgresql.enabled: true`, `redis.enabled: true` |
| Multiple replicas | `replicaCount: 2`, `redis.enabled: true` |
| Private registry | `image.repository`, `imagePullSecrets` |
| Custom persistent volume | `extraVolumes` / `extraVolumeMounts` with your path or PVC |
| Persistent storage | `persistence.enabled: true`, `persistence.storageClassName`, `persistence.size` |
| Mage Pro workspaces | `workspaceConfig.enabled: true` |
| Mage Pro K8s executor | `k8sExecutorConfig.enabled: true` |
***
# Default workspace and K8s executor config
When using **Mage Pro**, you can configure default workspace and Kubernetes executor settings via the chart so the control plane (and, for the K8s executor, workspace pods) get the config file mounted and the corresponding environment variables set automatically.
* **`workspaceConfig`** โ Default workspace configuration (e.g. `container_config`, `service_account_name`, `lifecycle_config`). The chart creates a ConfigMap, mounts it on the control plane at `/etc/mage/workspace-config.yaml`, and sets `WORKSPACE_DEFAULT_CONFIG_PATH`. See [Workspaces โ Default config file](/guides/developer-ux/workspaces#default-workspace-config-file).
* **`k8sExecutorConfig`** โ Default K8s executor configuration (e.g. `resource_limits`, `resource_requests`, `pod`). The chart creates a ConfigMap, mounts it at `/etc/mage/k8s-executor-config.yaml`, and sets `K8S_DEFAULT_CONFIG_PATH`. See [K8s executor โ Default config file](/production/executors/k8#default-config-file).
Enable and set `enforce` as needed in `values.yaml`; do not set `WORKSPACE_DEFAULT_CONFIG_PATH` or `K8S_DEFAULT_CONFIG_PATH` manually when using these chart options.
***
# Further reading
* [Mage Helm Chart source and values](https://github.com/mage-ai/helm-charts) โ full values reference in the chart repo
* [Helm chart releases](https://github.com/mage-ai/helm-charts/releases)
* [Mage Pro Workspaces](/guides/developer-ux/workspaces)
* [Kubernetes Executor](/production/executors/k8)
# Terraform
Source: https://docs.mage.ai/production/deploying-to-cloud/using-terraform
Deploy Mage to your cloud provider using Terraform.
Cloud providers currently supported with Mage Terraform scripts:
* AWS
* Azure
* GCP
* DigitalOcean
***
## Pre-install
Clone Mageโs
[Terraform template repository](https://github.com/mage-ai/mage-ai-terraform-templates):
```bash theme={"system"}
git clone https://github.com/mage-ai/mage-ai-terraform-templates.git
```
***
## Install Terraform CLI
```bash Using Homebrew theme={"system"}
brew tap hashicorp/tap && brew install hashicorp/tap/terraform
```
```bash Using Docker theme={"system"}
docker pull hashicorp/terraform:latest
```
### Windows
Follow [Microsoftโs instructions](https://learn.microsoft.com/en-us/azure/developer/terraform/get-started-windows-bash?tabs=bash#4-install-terraform-for-windows) to install Terraform on Windows.
### Errors
If youโre installing Terraform using Homebrew on an Apple M1 laptop, you may get this error:
```
Error: Incompatible provider version
Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms.
Other versions of this provider may have different platforms supported.
```
Follow this [guide](https://medium.com/@sssanjaya/terraform-error-incompatible-provider-version-m1-or-darwin-arm64-fae0159f913f) to fix the issue.
## Cloud specific Terraform instructions
Read the following documentation for the cloud provider of your choice:
* [Amazon Web Services (AWS)](/production/deploying-to-cloud/aws)
* [Microsoft Azure](/production/deploying-to-cloud/azure)
* [Google Cloud Platform (GCP)](/production/deploying-to-cloud/gcp)
* [DigitalOcean](/production/deploying-to-cloud/digitalocean)
***
# AWS ECS executor
Source: https://docs.mage.ai/production/executors/aws
Execute block runs in separate tasks.
You can choose to launch separate AWS ECS tasks to executor blocks by specifying
block executor\_type to be `ecs` in pipeline's metadata.yaml file.
There're 2 ways to customize the ECS executor config,
1. Specify the `ecs_config` in project's metadata.yaml file. Example config:
```yaml theme={"system"}
ecs_config:
cpu: 1024
memory: 2048
```
2. Add the `executor_config` at block level in pipeline's metadata.yaml file. Example config:
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
executor_type: ecs
executor_config:
cpu: 1024
memory: 2048
```
To run the whole pipeline in one ECS executor, you can set the `executor_type` at pipeline level and set `run_pipeline_in_one_process` to true.
`executor_config` can also be set at pipeline level. Here is the example pipeline metadata.yaml:
```yaml theme={"system"}
blocks:
- ...
- ...
executor_type: ecs
run_pipeline_in_one_process: true
name: example_pipeline
...
```
## Configurations
| Field name | Description | Example values |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| assign\_public\_ip | Whether to assign public IP to the ECS task. | true/false (default: true) |
| cpu | The CPU allocated to the ECS task. | 1024 |
| enable\_execute\_command | Whether to [enable execute command for debugging](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) | true/false (default: false) |
| launch\_type | The launch type of the ECS task. | FARGATE |
| memory | The memory allocated to the ECS task. | 2048 |
| tags | The tags of the ECS task. | \['tag1', 'tag2'] |
| wait\_timeout | The maximum wait time for the ECS task (in seconds). The default wait timeout for the ECS task is 10 minutes. Setting to -1 will disable waiting. | 1200 (default: 600) |
### Example
```yaml theme={"system"}
ecs_config:
cpu: 1024
memory: 2048
assign_public_ip: false
enable_execute_command: true
wait_timeout: 1200
```
## IAM permissions
Required IAM permissions for using ECS executor:
```json theme={"system"}
[
"ec2:DescribeNetworkInterfaces",
"ecs:DescribeTasks",
"ecs:ListTasks",
"ecs:RunTask",
"iam:PassRole"
]
```
***
## Resource management
A fully managed service, where we maintain the infrastructure, guarantee uptime,
automatically scale your workloads to handle any volume of pipeline runs,
automatically upgrade new versions of Mage Pro only features,
monitor your production pipelines, and provide enterprise level support.
# Azure Container Instance executor
Source: https://docs.mage.ai/production/executors/azure
Execute block runs in separate container instances.
If your Mage app is deployed on Microsoft Azure with Mageโs [Terraform](https://github.com/mage-ai/mage-ai-terraform-templates/tree/master/azure)
scripts, you can choose to launch separate Azure container instances to execute blocks.
How to configure pipeline to use Azure Container Instance executor:
1. Update Project's metadata.yaml
```yaml theme={"system"}
azure_container_instance_config:
cpu: 1
memory: 2
```
2. Update the `executor_type` of the block to `azure_container_instance` in pipeline's metadata.yaml and specify `executor_config` optionally.
The block level executor\_config will override the global executor\_config.
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
executor_type: azure_container_instance
executor_config:
cpu: 1
memory: 2
...
```
***
## Resource management
A fully managed service, where we maintain the infrastructure, guarantee uptime,
automatically scale your workloads to handle any volume of pipeline runs,
automatically upgrade new versions of Mage Pro only features,
monitor your production pipelines, and provide enterprise level support.
# GCP Cloud Run executor
Source: https://docs.mage.ai/production/executors/gcp
Execute block runs in separate jobs.
If your Mage app is deployed on GCP Cloud Run, you can choose to launch separate GCP Cloud Run jobs to execute blocks.
How to configure pipeline to use GCP cloud run executor:
1. Update Project's metadata.yaml
```yaml theme={"system"}
gcp_cloud_run_config:
path_to_credentials_json_file: "/path/to/credentials_json_file"
project_id: project_id
timeout_seconds: 600
```
2. Update the `executor_type` of block to `gcp_cloud_run` in pipeline's metadata.yaml:
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
executor_type: gcp_cloud_run
...
```
Customizing compute resource for GCP Cloud Run executor is coming soon.
***
## Resource management
A fully managed service, where we maintain the infrastructure, guarantee uptime,
automatically scale your workloads to handle any volume of pipeline runs,
automatically upgrade new versions of Mage Pro only features,
monitor your production pipelines, and provide enterprise level support.
# Kubernetes executor
Source: https://docs.mage.ai/production/executors/k8
Run Mage pipeline blocks in separate Kubernetes pods for scalability, isolation, and resource control.
If your Mage app is deployed in a **Kubernetes cluster**, you can configure the **Kubernetes Executor** (`executor_type: k8s`) to run each block in its own Kubernetes pod.
**Defaults in Mage Pro:**
* Namespace: `default`
* **Job name format:** `mage-data-prep-block-{block_run_id}`
You can override these defaults at the **project level**, **pipeline level**, **block level**, or with a **full Kubernetes job template**.
## Basic Setup
To configure a pipeline block to use Kubernetes executor, you simply just need to update the `executor_type` of the block to `k8s` in pipeline's metadata.yaml:
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
executor_type: k8s
...
```
By default, Mage uses `default` as the Kubernetes namespace. You can customize the namespace by setting the `KUBE_NAMESPACE` environment variable.
```bash theme={"system"}
export KUBE_NAMESPACE=my-namespace
```
## Configuration Methods
You can configure Kubernetes Executor in several places. Mage builds the
effective config by deep-merging the less specific config first, then applying
more specific config on top:
1. Environment variable defaults and hardcoded defaults
2. Project-level `k8s_executor_config`
3. Pipeline-level `executor_config`
4. Block-level `executor_config`
For a normal block-level Kubernetes executor run, block settings win over
pipeline settings, pipeline settings win over project settings, and project
settings win over environment defaults. For `run_pipeline_in_one_process: true`,
Mage uses the pipeline-level Kubernetes executor and merges only project-level
and pipeline-level settings because the whole pipeline runs in one pod.
Most nested dictionaries are merged, so you can set broad defaults at the
project level and override only the keys that need to change at the pipeline or
block level. Resource requests and limits are applied to the job container after
the pod template is built, so `executor_config.resource_requests` and
`executor_config.resource_limits` take precedence over `container.resources`.
### 1. Environment Variable Defaults
Use environment variables to set default resource requests, resource limits, and
service account for all Kubernetes executor jobs. These values are only applied
when a more specific project, pipeline, or block config does not set the same
resource group.
```bash theme={"system"}
export K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_CPU=250m
export K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_MEMORY=512Mi
export K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_CPU=500m
export K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_MEMORY=1Gi
export K8S_EXECUTOR_DEFAULT_SERVICE_ACCOUNT_NAME=default
```
โ **Use when:**
You want a cluster-wide fallback for jobs that do not define their own
Kubernetes resources.
### 2. Project-Level Configuration
Set `k8s_executor_config` in the project's `metadata.yaml` to apply defaults to
all blocks and pipeline runs that use the Kubernetes Executor in that project:
```yaml theme={"system"}
k8s_executor_config:
job_name_prefix: data-prep
namespace: default
resource_limits:
cpu: 1000m
memory: 2048Mi
resource_requests:
cpu: 500m
memory: 1024Mi
service_account_name: default
# Node scheduling configurations
pod:
node_selector:
node-type: gpu
instance-type: c5.large
scheduler_name: custom-scheduler
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-type"
operator: "In"
values: ["gpu", "compute"]
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: mage-pipeline
topologyKey: "kubernetes.io/hostname"
volumes:
- name: data-volume
persistent_volume_claim:
claim_name: mage-data-pvc
image_pull_secrets: my-registry-secret
```
โ **Use when:**
You want consistent Kubernetes settings across a project, such as default
CPU/RAM, namespace, service account, or node scheduling.
### 3. Pipeline-Level Configuration
Add `executor_type: k8s` and `executor_config` to the pipeline's
`metadata.yaml`. Pipeline-level config applies to the pipeline executor when
`run_pipeline_in_one_process: true`; it also acts as a shared default for block
runs in that pipeline before block-level overrides are applied.
```yaml theme={"system"}
executor_type: k8s
run_pipeline_in_one_process: true
executor_config:
namespace: default
resource_limits:
cpu: 2000m
memory: 4096Mi
resource_requests:
cpu: 1000m
memory: 2048Mi
```
โ **Use when:**
You want every block run in one pipeline to inherit the same Kubernetes config,
or you run the entire pipeline in one Kubernetes pod with
`run_pipeline_in_one_process: true`.
### 4. Block-Level Configuration
Add `executor_type: k8s` and `executor_config` to a block in the pipeline's
`metadata.yaml`:
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
executor_type: k8s
executor_config:
namespace: default
resource_limits:
cpu: 1000m
memory: 2048Mi
resource_requests:
cpu: 500m
memory: 1024Mi
```
โ **Use when:**
You only want to:
* Run certain blocks in the Kubernetes executor
* Override the Kubernetes executor config for specific blocks
* The Kubernetes job name is in this format: `mage-{job_name_prefix}-block-{block_run_id}`.
The default `job_name_prefix` is `data-prep`.
You can customize it in the `k8s_executor_config`.
You can interpolate the trigger name in the `job_name_prefix` field with the format `{trigger_name}`.
- **GPU Support:**
```yaml theme={"system"}
k8s_executor_config:
resource_limits:
nvidia.com/gpu: 1 # request 1 GPU
pod:
node_selector:
accelerator: nvidia-tesla-v100
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
```
Make sure [GPU device plugins](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#using-device-plugins) are installed.
- **Custom container & job spec:**
```yaml theme={"system"}
k8s_executor_config:
container_config:
image: mageai/mageai:0.9.7
env:
- name: USER_CODE_PATH
value: /home/src/k8s_project
job:
active_deadline_seconds: 120
backoff_limit: 3
ttl_seconds_after_finished: 86400
```
โ **Use when:**
You need specialized resources or scheduling for one expensive block without
changing the rest of the pipeline.
### 5. Full Kubernetes Job Template (maximum control)
Set the `K8S_CONFIG_FILE` environment variable to the path of a YAML
configuration file. This is the raw job template for Kubernetes executor jobs.
Use `K8S_DEFAULT_CONFIG_PATH` instead when you want a reusable policy that
merges with user config as described in [Default config file](#default-config-file).
**Example template:**
```yaml theme={"system"}
metadata:
annotations:
application: "mage"
component: "executor"
labels:
application: "mage"
type: "spark"
namespace: "default"
pod:
service_account_name: "mage-service-account"
image_pull_secrets: "my-registry-secret"
node_selector:
node-type: "compute"
instance-type: "c5.xlarge"
zone: "us-west-1a"
scheduler_name: "custom-scheduler"
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
- key: "spot-instance"
operator: "Equal"
value: "true"
effect: "NoSchedule"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-type"
operator: "In"
values: ["compute", "gpu"]
- key: "instance-type"
operator: "In"
values: ["c5.xlarge", "c5.2xlarge"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: "zone"
operator: "In"
values: ["us-west-1a", "us-west-1b"]
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: "mage-pipeline"
topologyKey: "kubernetes.io/hostname"
volumes:
- name: data-pvc
persistent_volume_claim:
claim_name: mage-data-pvc
- name: config-volume
config_map:
name: mage-config
- name: secret-volume
secret:
secret_name: mage-secrets
- name: empty-dir-volume
empty_dir: {}
container:
name: "mage-data"
image: "mageai/mageai:latest"
image_pull_policy: "IfNotPresent"
env:
- name: "KUBE_NAMESPACE"
value: "default"
- name: "secret_key"
value: "somesecret"
resources:
limits:
cpu: "1"
memory: "1Gi"
nvidia.com/gpu: 1
requests:
cpu: "0.1"
memory: "0.5Gi"
nvidia.com/gpu: 1
volume_mounts:
- mount_path: "/tmp/data"
name: "data-pvc"
- mount_path: "/etc/config"
name: "config-volume"
- mount_path: "/etc/secrets"
name: "secret-volume"
read_only: true
```
## Node Scheduling Configuration
### Node Selector
Use `node_selector` to schedule pods on specific nodes based on labels:
```yaml theme={"system"}
k8s_executor_config:
pod:
node_selector:
node-type: "gpu" # Schedule on GPU nodes
instance-type: "c5.large" # Schedule on specific instance types
zone: "us-west-1a" # Schedule in specific availability zones
```
### Tolerations
Use `tolerations` to allow pods to be scheduled on tainted nodes:
```yaml theme={"system"}
k8s_executor_config:
pod:
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
- key: "spot-instance"
operator: "Equal"
value: "true"
effect: "NoSchedule"
```
### Affinity Rules
Use `affinity` for advanced scheduling rules:
```yaml theme={"system"}
k8s_executor_config:
pod:
affinity:
# Node affinity - prefer specific node types
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-type"
operator: "In"
values: ["compute", "gpu"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: "zone"
operator: "In"
values: ["us-west-1a", "us-west-1b"]
# Pod anti-affinity - avoid co-locating similar pods
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: "mage-pipeline"
topologyKey: "kubernetes.io/hostname"
```
### Custom Scheduler
Use a custom scheduler for advanced scheduling logic:
```yaml theme={"system"}
k8s_executor_config:
pod:
scheduler_name: "custom-scheduler"
```
โ **Use when:**
You need full control over pod specs, annotations, volume mounts, and other Kubernetes settings.
## Multi-Container Pods
If your Mage deployment runs in a **multi-container pod**, set the `MAGE_CONTAINER_NAME` environment variable to specify which container runs Mage:
```yaml theme={"system"}
env:
- name: MAGE_CONTAINER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
```
If not set, Mage defaults to the **first container** in the pod.
## Default config file
You can provide a **default K8s executor config file** that Mage reads from the path in `K8S_DEFAULT_CONFIG_PATH` environment variable. The file is loaded as a policy and merged with block-level and project-level config. Use this to enforce cluster-wide defaults (e.g. resource limits, service account) without editing every project.
**Helm chart:** When deploying **Mage Pro** with the [Mage Helm chart](https://github.com/mage-ai/helm-charts), configure the default via **`k8sExecutorConfig`** in `values.yaml`. The chart creates a ConfigMap from your config, mounts it into the control-plane (and workspace pods), and sets `K8S_DEFAULT_CONFIG_PATH` automatically. Do not set the env var manually when using the chart. See [Using Helm](/production/deploying-to-cloud/using-helm#default-workspace-and-k8s-executor-config) for details. For non-Helm deployments, set the env var and ensure the file exists at that path (e.g. via your own ConfigMap or volume mount).
**File format:**
```yaml theme={"system"}
enabled: true # Set to false to disable this policy
enforce: true # If true, policy overrides user config. If false, user config overrides.
resource_limits:
cpu: "500m"
memory: "1Gi"
resource_requests:
cpu: "250m"
memory: "512Mi"
service_account_name: default
pod:
node_selector:
node-type: compute
tolerations: []
volumes: []
container:
env: []
volume_mounts: []
```
**Policy fields:**
| Field | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | If `false`, the policy is not applied. |
| `enforce` | Controls how the default file is merged with **user config** (block + pipeline + project). See [Priority when merging](#priority-when-merging-default-file-with-user-config) below. |
| *(other keys)* | Same structure as [project-level k8s\_executor\_config](/production/executors/k8#2-project-level-configuration) (e.g. `resource_limits`, `resource_requests`, `pod`, `container`, `metadata`, `job`). |
The file must exist at the given path on the Mage control-plane. Use a ConfigMap or volume mount to provide it. When **workspaces** are created, only the K8s default config file is injected (volume + volume mount + env var) so workspace pods can read it; see [Workspaces](/guides/developer-ux/workspaces#default-workspace-config-file).
**Priority when merging default file with user config**
The default file is **merged** with the combined user config (block + pipeline + project). The **effective priority order** (who wins for a given key) depends on `enforce`:
| `enforce` | Effective priority (highest โ lowest) | Result |
| ----------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`true`** | **1. Default config file** โ 2. Block โ 3. Pipeline โ 4. Project โ 5. Env vars โ 6. Hardcoded | **Policy is highest priority.** Values from the default file override block, pipeline, and project for the same keys. Use this to enforce cluster-wide defaults (e.g. resource limits, service account) that users cannot override. Conflicting values can cause an error on update when strict conflict checking is used. |
| **`false`** | **1. Block** โ 2. Pipeline โ 3. Project โ 4. Default config file (fills gaps only) โ 5. Env vars โ 6. Hardcoded | **User wins.** Block, pipeline, and project override the policy for the same keys. The policy only fills in keys that user config did not set. Use this to provide optional defaults while allowing per-block, per-pipeline, or per-project overrides. |
So when **`enforce: true`**, the policy has the **highest** priority; when **`enforce: false`**, block, pipeline, and project config are higher and the policy only supplies missing values.
## Environment Variables Reference
### General Configuration
| Variable | Purpose | Default |
| --------------------- | --------------------------------------------- | ------- |
| `KUBE_NAMESPACE` | Namespace for job execution | default |
| `K8S_CONFIG_FILE` | Path to full Kubernetes job config file | โ |
| `MAGE_CONTAINER_NAME` | Container to run Mage in multi-container pods | โ |
### Resource Usage Metrics
In Mage Pro, completed Kubernetes executor runs can store CPU and memory usage
summaries for the run detail **Resources** tab and the pipeline monitoring
dashboard. Mage uses Prometheus first when
`K8S_RESOURCE_USAGE_PROMETHEUS_URL` is configured; otherwise it uses the
Kubernetes Metrics API fallback when available.
Pipeline-level recommendations are available in Mage Pro after completed K8s
executor runs have stored CPU or memory samples. Mage calculates the suggested
`resource_requests` and `resource_limits` from observed usage, then shows the
values in the pipeline monitoring page so you can copy them into
`executor_config` or `k8s_executor_config`.
**How recommendations are calculated:**
| Resource | Basis | Request | Limit |
| -------- | ---------------------------------------------------------- | ------------------------ | ------------------------------- |
| CPU | `p95_cores`, then `max_cores`, then latest `current_cores` | `max(basis * 0.75, 50m)` | `max(basis * 1.5, request * 2)` |
| Memory | `p95_bytes`, then `max_bytes`, then latest `current_bytes` | `basis * 1.1` | `basis * 1.3` |
Mage marks the recommendation as `low` confidence when the run has fewer than
three samples or an OOM warning, `medium` when enough samples exist from the
Kubernetes Metrics API fallback, and `high` when Prometheus provides at least
six samples. Prometheus samples are queried with
`K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` and Mage can wait once after a run
completes, controlled by
`K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS`, before retrying sparse
completed runs. When unset, the completion delay defaults to
`K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS`.
| Variable | Purpose | Default |
| -------------------------------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------- |
| `K8S_RESOURCE_USAGE_PROMETHEUS_URL` | Prometheus base URL for K8s resource usage summaries | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_BEARER_TOKEN` | Bearer token for Prometheus requests | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_USERNAME` | Username for Prometheus basic authentication | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_PASSWORD` | Password for Prometheus basic authentication | โ |
| `K8S_RESOURCE_USAGE_PROMETHEUS_TIMEOUT_SECONDS` | Timeout for each Prometheus query | `5` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` | Query range step for CPU and memory samples | `15` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_COMPLETION_DELAY_SECONDS` | Delay before retrying Prometheus for completed runs with too few samples | `K8S_RESOURCE_USAGE_PROMETHEUS_STEP_SECONDS` |
| `K8S_RESOURCE_USAGE_PROMETHEUS_VERIFY_TLS` | Set to `0`, `false`, `no`, or `off` to skip TLS certificate verification | `true` |
See [Monitoring with Prometheus](/observability/external/prometheus#kubernetes-resource-usage-in-mage-pro) for Prometheus query behavior and authentication details.
### Default Resource Configuration
These environment variables set default resource limits and requests for all K8s executor jobs.
They are applied when a project-level, pipeline-level, or block-level configuration does not
specify resource requests or limits.
| Variable | Purpose | Example Value |
| ----------------------------------------------- | ----------------------------------------- | ------------- |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_CPU` | Default CPU request for executor jobs | `250m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_REQUESTS_MEMORY` | Default memory request for executor jobs | `512Mi` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_CPU` | Default CPU limit for executor jobs | `500m` |
| `K8S_EXECUTOR_DEFAULT_RESOURCE_LIMITS_MEMORY` | Default memory limit for executor jobs | `1Gi` |
| `K8S_EXECUTOR_DEFAULT_SERVICE_ACCOUNT_NAME` | Default service account for executor jobs | `default` |
**Priority order without a default config file** (highest to lowest):
1. Block-level `executor_config` in the pipeline's `metadata.yaml`
2. Pipeline-level `executor_config` in the pipeline's `metadata.yaml`
3. Project-level `k8s_executor_config` in the project's `metadata.yaml`
4. Environment variable defaults (listed above)
5. Hardcoded defaults
When Mage Pro resource usage recommendations suggest new Kubernetes
`resource_requests` or `resource_limits`, apply the values at the narrowest
level you want to tune: block-level for one block, pipeline-level for all runs
of one pipeline, or project-level for all Kubernetes executor jobs in the
project.
# Local python executor
Source: https://docs.mage.ai/production/executors/python
Local python executors are ran within the same container as the scheduler service.
You can customize the compute resource with the same way mentioned in this
[document](/production/configuring-production-settings/compute-resource).
section.
### Force local Python executor
If you want to use `local_python` executor when `DEFAULT_EXECUTOR_TYPE` is set to another executor type,
you can set the `executor_type` to `local_python_force`.
# PySpark executor
Source: https://docs.mage.ai/production/executors/spark
If the pipeline type is `pyspark`, we use PySpark executors for pipeline and
block executions.
You can customize the compute resource of PySpark executor by
updating the instance types of `emr_config` in project's metadata.yaml file.
## Example:
```yaml theme={"system"}
emr_config:
ec2_key_name: "xxxxx"
master_instance_type: "r5.2xlarge"
slave_instance_type: "r5.2xlarge"
```
## Spark compute resource manager
Manage your Spark compute resources and track Spark pipeline execution metrics.
[Learn more](/integrations/compute/management)
# Mage Pro
Source: https://docs.mage.ai/production/mage/pro
A powerful, cloud-native solution to orchestrate and scale your data pipelines โ available as fully managed, private, or hybrid cloud deployments.
## What is Mage Pro?
**Mage Pro** is a production-grade platform for building, running, and scaling data pipelines โ with zero infrastructure hassle. You can deploy it as:
* A **fully managed cloud service** (Mage-hosted)
* A **private cloud** installation (your cloud, your control)
* A **hybrid cloud** setup (Mage manages control plane; you host workloads)
Mage Pro supports all major cloud providers including **AWS**, **GCP**, and **Azure**, and is designed for teams that need high scalability, security, and reliability.
***
## Deployment Options
### โ๏ธ Fully Managed Cloud (Mage-hosted)
Let Mage handle everything: we manage infrastructure, upgrades, availability, and monitoring.
### ๐ Private Cloud (Your Infrastructure)
Run Mage Pro securely in your own cloud environment. Ideal for teams with data residency or compliance requirements.
### ๐ Hybrid Cloud (Split Control)
Keep data and compute in your environment while Mage manages the control plane. The best of both worlds โ reduced overhead, full data privacy.
***
## Migrating to Mage Pro
Visit [Mage Pro](https://cloud.mage.ai) and sign in. Spin up dedicated environments/clusters for dev, staging, and prod โ with isolated workspaces, users, and resources.
Supports **multi-tenant workspaces**, **RBAC**, and custom compute/resource settings per team or project.
Get real-time insights into pipeline performance with built-in **CPU utilization charts** and system metrics.
Monitor and alert on memory usage across pipelines to avoid resource exhaustion and bottlenecks.
Migrate your existing projects into Mage Pro using our **Git-based import tool**.
You can connect to a remote Git repository to keep your pipeline code in sync across environments.
๐ Follow the [Version Control & Sync Guide](/guides/data-sync/version-control-guide) to learn how to integrate Mage with your Git workflows.
***
## Why Choose Mage Pro?
Mage Pro is more than just a hosted version of Mage โ itโs a powerful upgrade with exclusive features, deeper integrations, and a refined developer experience for teams running data pipelines at scale.
### ๐ Mage ProโOnly Features
#### ๐ง AI & Developer Experience
* **AI Sidekick**: Get context-aware code suggestions, explanations, and debugging for every block.
* **Enhanced Code Editor**: Smarter autocomplete, live error checking, and documentation hints.
* **Personalized UI Themes & Light Mode**: Match your workflow preferences.
* **VSCode & Cursor Integration**: Edit and sync directly with your local IDE.
#### ๐ ๏ธ Advanced Orchestration & Infrastructure
* **Workspaces**: Create isolated environments for dev, staging, and prod.
* **Pipeline-Level Resource Monitoring**: Track memory and CPU usage per pipeline.
* **Stateful Streaming Pipelines**: Run real-time pipelines with stateful checkpointing.
* **Deployment App & Version Control Terminal**: Seamlessly deploy and manage pipeline code using built-in Git tools.
* **More Variable Interpolation Syntax**: Dynamically pass values with support for env vars, secrets, and config vars.
#### ๐ Pipeline Engine & Execution Enhancements
* **Stateful Streaming Pipelines**: Run real-time pipelines with built-in state management and checkpointing.
* **Unified IO Config Framework**: Use a single config structure across batch pipelines, streaming pipelines, and data integration pipelines.
* **New Data Sources & Destinations**: Access exclusive connectors not available in Mage OSS.
* **Enhanced Existing Integrations**: Benefit from optimized, production-ready versions of popular sources and destinations.
* **Simplified Spark & Databricks Integration**: Run distributed compute jobs with less setup and config.
* **Native Iceberg Support**: Work with open table formats natively for modern data lakes.
### ๐ข Enterprise-Grade Platform
* โ๏ธ **Flexible Deployments**: Fully managed, private cloud (your VPC), or hybrid cloud.
* ๐ **Security Built-In**: RBAC, workspace isolation, secrets vault, audit logs.
* ๐ **Full Observability**: Built-in dashboards for logs, CPU/memory, errors, and alerts.
* ๐ **Autoscaling & High Availability**: Handle bursty, concurrent workloads across projects.
* ๐ค **Dedicated Support & SLAs**: White-glove onboarding, expert guidance, and 24/7 help if needed.
* ๐ **Instant Migration**: Import OSS pipelines via Git and sync across environments.
***
## ๐ Get Started with Mage Pro
A fully managed service, where we maintain the infrastructure, guarantee uptime,
automatically scale your workloads to handle any volume of pipeline runs,
automatically upgrade new versions of Mage Pro only features,
monitor your production pipelines, and provide enterprise level support.
```
With the `