#topbar-cta-button {}

#navbar {}

#sidebar {}

#content-area {}

#table-of-contents {}

footer {}

a.block {
  display: block;
}

p.normal,
span.normal {
  font-weight: normal !important;
}


Mage project setup

Buildkite setup for deployment to AWS

Buildkite

Buildkite deployments

Mage AI

A powerful, cloud-native solution to orchestrate and scale your data pipelines — available as fully managed, private, or hybrid cloud deployments.

Mage Pro

Overview

Deployment architecture overview

SaaS

Managed cloud

Deploy the data plane in your cloud environment

Hybrid Cloud

Hybrid cloud deployment

Deploy securely in your private cloud environment

Private Cloud

Private cloud deployment

Deploy securely from your own on-prem servers

On-prem

On-prem deployment

Cluster Management

Mage Pro API

Install dependencies using requirements.txt

Install dependencies

Set up Mage Pro workspaces for organizing your own personal environment.

Workspaces

Mage Pro workspaces

Build a pipeline

Mage Pro data pipeline

Trigger your first Mage Pro data pipeline

Trigger a pipeline

Mage Pro pipeline trigger

Mage Pro CI/CD overview

Connect your magical data pipelines from GitHub to your Mage Pro workspace using either the sacred OAuth tokens or the mystical SSH keys. Below lies the ancient knowledge for both rituals. Choose your path wisely!

Version control guide

Mage Pro Version Control

New collaborative terminal with Git authentication and functionality built-in for easy and fast version control.

Git terminal

Git terminal with built-in authentication and shortcuts

Local file edit tracking and version history for restoring changes made in the past.

File versioning

File versioning and history

Global Hooks are a powerful feature in Mage that allows you to execute custom functions before or after an API request. These hooks can be used to perform additional operations, validate data, or integrate with external systems.

Global Hooks

Follow this step by step guide on creating a global hook in Mage.

Setup

Global Hooks Guide

Mage Pro API overview

Create Deployment

It’s magic.

Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. 🦸‍♀️

Quickstart

Every user experience and technical design decision adheres to these principles.

Design

Core Design Principles

Pipelines

Pipeline Management

How to set up a folder containing your Mage project code and other files.

Project setup

There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.

Version control

Get started with Git

Database

The `io_config` YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Credentials

IO Config

Learn how to set up and configure the io_config file for your Mage project

IO Config Setup

We support multiple kernels in the code editor.

Code execution

Python kernels

Display timestamps throughout the app in your local timezone.

Timezones

Local timezone

Settings

Configuring Mage settings backend

How to get the latest and greatest from the Mage team.

Version upgrades

Updating Mage

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Contributing

Want to get started contributing code? You're in the right place! Read on to learn how to set up your development environment.

Dev environment

Configure your development environment

We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.

Code of conduct

Code of Conduct

Please contribute usage statistics to help improve the developer experience for you and everyone in the community 🤝.

Usage statistics

Help improve the tool

Here are some frequently asked questions about Mage and our best answers.

Frequently Asked Questions

Changelog

Releases

Mage guides for getting started, building pipelines, and more.

Guides

Mage Guides

Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.

Tips & tricks

Mage tips & tricks

Code editor

Writing data pipeline code

Money does grow on trees... if you plant them with magic beans.

Dependency tree

Block dependency tree

Add, edit, move, and delete files or folders in your project.

File browser

Browse all files

Have you ever wished you had a sidekick like Batman and Robin? Now you have one.

Sidekick

Pipeline sidekick

This page will walk you through the process of authenticating with SSH or HTTPS and configuring Git for use with Mage Git Sync.

Configurations

Configure Git

Authenticate with a Git provider then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.

Git application

Version control application with Git

Enabling one-way Git sync in Mage ensures data will be synced in one direction with a specified git repository.

Git sync

One-way code sync using Git

Mage's Git integration provides a way for you to push changes to a remote repository— for example collaborating on a Mage project with your teammates.

Git integration

Git integration for code version control

Reorder blocks in the notebook when editing a pipeline.

Reorder

Reordering blocks

Detach other pipeline associations from a block shared by multiple pipelines.

Detach

Detaching blocks

Dynamic

Dynamic blocks

Serialize and deserialize data using Polars DataFrames.

Polars

Native support for Polars

Reuse the same block multiple times within a single pipeline.

Replicate

Replicating blocks

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

Transformers

Transformer blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

Custom

Custom blocks

A markdown block is a standalone text block that supports markdown and can be used to make notes or organize your blocks.

Markdown

Markdown blocks

Execute SQL commands directly in your database, data warehouse, etc.

SQL blocks

You can write R language to transform data in blocks.

R blocks

Build a data pipeline that loads restaurant data, transforms it, then exports it to a DuckDB database. 🦆

Batch

ETL pipeline tutorial

Here are the high level steps to build a data integration pipeline:

Data integration

How to build a data integration pipeline

Import a pipeline zip file into your project.

Importing

Importing pipelines

A collection of Mage projects and examples created by our community.

Community Examples

Learn how Mage integrates with dbt to orchestrate, schedule, and monitor dbt models as part of your data pipelines.

Use dbt with Mage: Simplify Your Data Transformation Pipelines

Learn how to develop your favorite dbt models in Mage.

Getting started

Develop dbt in Mage

⏰ Run dbt models in Mage in under a minute.

New project

Make dbt magical

There are 4 development and deployment workflows.

Setup repository

Repository setup

Development (local) and production (cloud) using GitHub Actions.

GitHub

GitHub Actions

Development (local) and production (cloud) using GitLab CI/CD.

GitLab

GitLab deployments

Manage, deploy, and rollback new code changes, from a remote repository to any environment, all from within the Mage platform.

Continuous deployment with Mage Pro

Production settings

Production Configuration

Architecture

Compute resources

Local python executors are ran within the same container as the scheduler service.

Python

Local python executor

Kubernetes

Kubernetes executor

Spark

PySpark executor

AWS ECS executor

Execute block runs in separate container instances.

Azure

Azure Container Instance executor

GCP Cloud Run executor

Built into Mage

Mage’s built-in secret manager

Attention Magers: This classified document contains vital intel on securing sensitive data within the AWS Secrets Manager vault. Study the protocols carefully to create and operationalize confidential secrets, granting you clearance to integrate them into your Mage data pipeline projects.

Deployment

Cloud infrastructure

Compute resources

Secret manager

Authentication

Buildkite deployments

Mage project setup

Buildkite setup for deployment to AWS

Deployment

Cloud infrastructure

Compute resources

Secret manager

Authentication

​Mage project setup

​Buildkite setup for deployment to AWS

Mage project setup

Buildkite setup for deployment to AWS