Singer spec

Folder structure

Contribute

Mage AI

Documentation

Development

Production

Observability

Integrations

Extensibility

Demo playground

GitHub

Community

Mage Pro

A source is a system that you want to load data from and synchronize it into another system. A source can be a 3rd party API, SaaS, database, data warehouse, or a data lake.

Overview

Sources technical documentation

It’s magic.

Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. 🦸‍♀️

Quickstart

Every user experience and technical design decision adheres to these principles.

Design

Core Design Principles

These are the fundamental concepts that Mage uses to operate.

Core abstractions

Abstractions

Details about how the Mage file directory works and how typical projects are structured.

Project Structure

Pipelines

Pipeline Management

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

Use the output of a block to dynamically create more blocks.

Dynamic blocks

Data integration is the process of synchronizing data between two systems.

There are many ways to configure your data integration pipeline.

Configuration

Configure data integration pipelines

Load data from data integration sources and export to data integration destinations as blocks inside batch pipelines.

Batch pipelines

Using data integrations in batch pipelines (in Beta)

Configure triggers in triggers.yaml under pipeline folder.

Configure triggers in code

You can trigger another pipeline from a block within a different pipeline.

Trigger from block

Trigger pipeline from a block

You can trigger a pipeline by making an API request.

Webhooks

Triggering pipeline via API request

Schedules

Database query

Query trigger metadata from database

Retrying block runs

Retrying block runs from a pipeline run

Saving block output as CSV

Saving block output as CSV file

Query pipeline run metadata from database

Backfilling pipelines

Guide

Backfill guide

How to set up a folder containing your Mage project code and other files.

Setup

Project setup

There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.

Version control

Get started with Git

Here's how you can connect a local database to Mage in Docker for development.

Connect local database

Connect a local db in Docker

Run Mage docker image as non-root user to restrict access to the system

Run as non-root user

Run docker image as non-root user

There are a few ways to set up variables in Mage, here's how you can use them to make the best pipelines the world has ever seen. 🌎

Mage variables

You've got all of these great variables and secrets, now let's put them to use. Learn how to access them in your code. 🦸🏻‍♂️

Accessing variables

Accessing variables in Mage

Mage allows you to easily integrate third-party Python packages into your project and pipelines. This guide will walk you through the process of adding and managing external dependencies.

Python packages

Install Python packages using `requirements.txt`

You can import any Python code from any file that is in your Mage project directory.

Custom modules

Import your own Python code

Database

The `io_config` YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Credentials

IO Config

We support multiple kernels in the code editor.

Code execution

Python kernels

Display timestamps throughout the app in your local timezone.

Timezones

Local timezone

Settings

Configuring Mage settings backend

How to get the latest and greatest from the Mage team.

Version upgrades

Updating Mage

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Contributing

Want to get started contributing code? You're in the right place! Read on to learn how to set up your development environment.

Environment setup

Configure your development environment

Mage backend code is written in Python 🐍 and our server uses the Tornado 🌪️ framework. Here are some guides on adding features to the Mage backend.

Contributing to the backend server

Guides on adding features to the front-end client.

Contributing to the front-end client

Charts

How to add a chart

Making Mage invincible so that nothing can break it, not even itself.

Testing

Frontend testing

Like every other part of Mage, our documentation is open-source. Read on to learn how to make edits or even write entirely new docs for Mage!

We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.

Code of conduct

Code of Conduct

Please contribute usage statistics to help improve the developer experience for you and everyone in the community 🤝.

Usage statistics

Help improve the tool

Here are some frequently asked questions about Mage and our best answers.

Frequently Asked Questions

Changelog

Releases

Mage guides for getting started, building pipelines, and more.

Guides

Mage Guides

Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.

Tips & tricks

Mage tips & tricks

Code editor

Writing data pipeline code

Money does grow on trees... if you plant them with magic beans.

Dependency tree

Block dependency tree

Manage cross-pipeline dependencies and execution flow across every pipeline within a project.

Pipeline dependencies

Cross-pipeline dependencies

Build, run, and manage complex pipelines with the new canvas mode.

Pipeline canvas

Pipeline editor canvas

Add, edit, move, and delete files or folders in your project.

File browser

Browse all files

Code browser and editor with syntax highlighting, code formatting, and version history.

Code browser

Code editor with enhanced productivity

Have you ever wished you had a sidekick like Batman and Robin? Now you have one.

Sidekick

Pipeline sidekick

Customize workspace themes, personalize accounts, and re-configure UI component layouts.

Personalize

Personalize your workspace

This page will walk you through the process of authenticating with SSH or HTTPS and configuring Git for use with Mage Git Sync.

Configurations

Configure Git

Authenticate with a Git provider then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.

Git application

Version control application with Git

Enabling one-way Git sync in Mage ensures data will be synced in one direction with a specified git repository.

Git sync

One-way code sync using Git

New collaborative terminal with Git authentication and functionality built-in for easy and fast version control.

Git terminal

Git terminal with built-in authentication and shortcuts

Mage's Git integration provides a way for you to push changes to a remote repository— for example collaborating on a Mage project with your teammates.

Git integration

Git integration for code version control

Local file edit tracking and version history for restoring changes made in the past.

File versioning

File versioning and history

Reorder blocks in the notebook when editing a pipeline.

Reorder

Reordering blocks

Detach other pipeline associations from a block shared by multiple pipelines.

Detach

Detaching blocks

Dynamic

Execute 100,000+ dynamically created block runs concurrently with upgraded memory management and concurrency.

Dynamic blocks 2.0

Level 99 dynamic abilities

Granular block settings for controlling read/write data partitions using output size, number of chunks, and item count.

Batch read/write

Serialize and deserialize data using Polars DataFrames.

Polars

Native support for Polars

Reuse the same block multiple times within a single pipeline.

Replicate

Replicating blocks

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

Transformers

Transformer blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

Custom

Custom blocks

A markdown block is a standalone text block that supports markdown and can be used to make notes or organize your blocks.

Markdown

Markdown blocks

Execute SQL commands directly in your database, data warehouse, etc.

SQL blocks

You can write R language to transform data in blocks.

R blocks

Build a data pipeline that loads restaurant data, transforms it, then exports it to a DuckDB database. 🦆

Adventure starts here

Concepts

Project

Contributing

About

Sources technical documentation

Singer spec

Folder structure

Contribute

Adventure starts here

Concepts

Project

Contributing

About

​Singer spec

​Folder structure

​Contribute

Singer spec

Folder structure

Contribute