#topbar-cta-button {}

#navbar {}

#sidebar {}

#content-area {}

#table-of-contents {}

footer {}

a.block {
  display: block;
}

p.normal,
span.normal {
  font-weight: normal !important;
}


Configuration

Amazon S3

Mage AI

Documentation

Development

Production

Observability

Integrations

Extensibility

Get a demo

GitHub

Community

Mage Pro

Overview

It’s magic.

Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. 🦸‍♀️

Quickstart

Every user experience and technical design decision adheres to these principles.

Design

Core Design Principles

These are the fundamental concepts that Mage uses to operate.

Core abstractions

Abstractions

Details about how the Mage file directory works and how typical projects are structured.

Project Structure

Pipelines

Pipeline Management

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

Use the output of a block to dynamically create more blocks.

Dynamic blocks

Data integration is the process of synchronizing data between two systems.

There are many ways to configure your data integration pipeline.

Configure data integration pipelines

Load data from data integration sources and export to data integration destinations as blocks inside batch pipelines.

Batch pipelines

Using data integrations in batch pipelines (in Beta)

Configure triggers in triggers.yaml under pipeline folder.

Configure triggers in code

You can trigger another pipeline from a block within a different pipeline.

Trigger from block

Trigger pipeline from a block

You can trigger a pipeline by making an API request.

Webhooks

Triggering pipeline via API request

Schedules

Database query

Query trigger metadata from database

Retrying block runs

Retrying block runs from a pipeline run

Saving block output as CSV

Saving block output as CSV file

Query pipeline run metadata from database

Backfilling pipelines

Guide

Backfill guide

How to set up a folder containing your Mage project code and other files.

Setup

Project setup

There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.

Version control

Get started with Git

Here's how you can connect a local database to Mage in Docker for development.

Connect local database

Connect a local db in Docker

Run Mage docker image as non-root user to restrict access to the system

Run as non-root user

Run docker image as non-root user

There are a few ways to set up variables in Mage, here's how you can use them to make the best pipelines the world has ever seen. 🌎

Mage variables

You've got all of these great variables and secrets, now let's put them to use. Learn how to access them in your code. 🦸🏻‍♂️

Accessing variables

Accessing variables in Mage

Mage allows you to easily integrate third-party Python packages into your project and pipelines. This guide will walk you through the process of adding and managing external dependencies.

Python packages

Install Python packages using `requirements.txt`

You can import any Python code from any file that is in your Mage project directory.

Custom modules

Import your own Python code

Database

The `io_config` YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Credentials

IO Config

Learn how to set up and configure the io_config file for your Mage project

IO Config Setup

We support multiple kernels in the code editor.

Code execution

Python kernels

Display timestamps throughout the app in your local timezone.

Timezones

Local timezone

Settings

Configuring Mage settings backend

How to get the latest and greatest from the Mage team.

Version upgrades

Updating Mage

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Contributing

Want to get started contributing code? You're in the right place! Read on to learn how to set up your development environment.

Dev environment

Configure your development environment

Mage backend code is written in Python 🐍 and our server uses the Tornado 🌪️ framework. Here are some guides on adding features to the Mage backend.

Contributing to the backend server

Guides on adding features to the front-end client.

Contributing to the front-end client

Charts

How to add a chart

Making Mage invincible so that nothing can break it, not even itself.

Testing

Frontend testing

Like every other part of Mage, our documentation is open-source. Read on to learn how to make edits or even write entirely new docs for Mage!

We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.

Code of conduct

Code of Conduct

Please contribute usage statistics to help improve the developer experience for you and everyone in the community 🤝.

Usage statistics

Help improve the tool

Here are some frequently asked questions about Mage and our best answers.

Frequently Asked Questions

Changelog

Releases

Mage guides for getting started, building pipelines, and more.

Guides

Mage Guides

Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.

Tips & tricks

Mage tips & tricks

Code editor

Writing data pipeline code

Money does grow on trees... if you plant them with magic beans.

Dependency tree

Block dependency tree

Manage cross-pipeline dependencies and execution flow across every pipeline within a project.

Pipeline dependencies

Cross-pipeline dependencies

Build, run, and manage complex pipelines with the new canvas mode.

Pipeline canvas

Pipeline editor canvas

Add, edit, move, and delete files or folders in your project.

File browser

Browse all files

Code browser and editor with syntax highlighting, code formatting, and version history.

Code browser

Code editor with enhanced productivity

Have you ever wished you had a sidekick like Batman and Robin? Now you have one.

Sidekick

Pipeline sidekick

Customize workspace themes, personalize accounts, and re-configure UI component layouts.

Personalize

Personalize your workspace

Set up Mage Pro workspaces for organizing your own personal environment.

Workspaces

Mage Pro workspaces

Integrate your Mage Pro environment with Visual Studio Code.

Visual Studio Code

Mage Pro Visual Studio Code integration

Connect your magical data pipelines from GitHub to your Mage Pro workspace using either the sacred OAuth tokens or the mystical SSH keys. Below lies the ancient knowledge for both rituals. Choose your path wisely!

Version control guide

Mage Pro Version Control

This page will walk you through the process of authenticating with SSH or HTTPS and configuring Git for use with Mage Git Sync.

Configurations

Configure Git

Authenticate with a Git provider then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.

Git application

Version control application with Git

Enabling one-way Git sync in Mage ensures data will be synced in one direction with a specified git repository.

Git sync

One-way code sync using Git

New collaborative terminal with Git authentication and functionality built-in for easy and fast version control.

Git terminal

Git terminal with built-in authentication and shortcuts

Mage's Git integration provides a way for you to push changes to a remote repository— for example collaborating on a Mage project with your teammates.

Git integration

Git integration for code version control

Local file edit tracking and version history for restoring changes made in the past.

File versioning

File versioning and history

Reorder blocks in the notebook when editing a pipeline.

Reorder

Reordering blocks

Detach other pipeline associations from a block shared by multiple pipelines.

Detach

Detaching blocks

Dynamic

Execute 100,000+ dynamically created block runs concurrently with upgraded memory management and concurrency.

Dynamic blocks 2.0

Level 99 dynamic abilities

Granular block settings for controlling read/write data partitions using output size, number of chunks, and item count.

Batch read/write

Serialize and deserialize data using Polars DataFrames.

Polars

Native support for Polars

Reuse the same block multiple times within a single pipeline.

Replicate

Replicating blocks

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

Transformers

Transformer blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

Key	Description	Sample Value	Required
`aws_access_key_id`	AWS access key ID.	`abc123`	✅
`aws_secret_access_key`	AWS secret access key.	`xyz456`	✅
`aws_region`	Region where the S3 bucket is located.	`us-west-2`
`role_arn`	(Optional) ARN of the IAM role to assume for accessing the S3 bucket.	`arn:aws:iam::111111:role/example-role`
`bucket`	Name of the AWS S3 bucket to read data from.	`user_generated_content`	✅
`file_type`	Type of files in the bucket. Supported values: `parquet`, `csv`, `infer`. If `infer`, the type is detected based on file extension.	`infer`
`prefix`	Path within the bucket where files are located. Do not include `s3://`, the bucket name, or the table name in this path.	`users/ds/20221225`
`search_pattern`	Regular expression to match files within the prefix.	`test_table\\/.*\\.csv`
`single_stream_in_prefix`	If `true`, all files in the prefix are treated as a single stream. Defaults to `false`.	`false`
`table_configs`	A list of table configurations to configure multiple streams. Each table config must include `prefix`, `search_pattern`, and `table_name`.	`[{"prefix": "users/", "search_pattern": "test_table\\/.*\\.csv", "table_name": "test_table"}]`
`aws_endpoint`	(Optional) Custom S3-compatible endpoint (e.g., for MinIO).	`https://play.min.io`

Adventure starts here

Concepts

Project

Contributing

About

Amazon S3

Configuration

Adventure starts here

Concepts

Project

Contributing

About

​Configuration

Configuration