Mage project setup

Buildkite setup for deployment to AWS

Releases

Roadmap

Troubleshooting

Mage

Docs

Guides

Dev UX

Deploy

Platform

Contribute

Slack

Documentation

Buildkite

Mage is an open-source, hybrid framework for transforming and integrating data. ✨

Overview

🧙‍♀️ Welcome to Mage

Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. 🦸‍♀️

Quickstart

Every user experience and technical design decision adheres to these principles.

Design

Core Design Principles

These are the fundamental concepts that Mage uses to operate.

Core abstractions

Abstractions

Details about how the Mage file directory works and how typical projects are structured.

Project Structure

Backfilling pipelines

Guide

Backfill guide

Global Hooks are a powerful feature in Mage that allows you to execute custom functions before or after an API request. These hooks can be used to perform additional operations, validate data, or integrate with external systems.

Global Hooks

Follow this step by step guide on creating a global hook in Mage.

Global Hooks Guide

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

Use the output of a block to dynamically create more blocks.

Dynamic blocks

Data integration is the process of synchronizing data between two systems.

There are many ways to configure your data integration pipeline.

Configuration

Configure data integration pipelines

Load data from data integration sources and export to data integration destinations as blocks inside batch pipelines.

Batch pipelines

Using data integrations in batch pipelines (in Beta)

dbt Integration

Upgraded dbt developer experience in Mage.

dbt DX 2.0

dbt developer experience level-up

Generate and orchestrate the final output of a pipeline (aka data product). Data products can be globally referenced from any pipeline and its data output can be used in any block.

Global data products

Pipelines

Pipeline Management

Retrying block runs

Retrying block runs from a pipeline run

Saving block output as CSV

Saving block output as CSV file

Database query

Query pipeline run metadata from database

Configure triggers in triggers.yaml under pipeline folder.

Configure triggers in code

You can trigger another pipeline from a block within a different pipeline.

Trigger from block

Trigger pipeline from a block

You can trigger a pipeline by making an API request.

Webhooks

Triggering pipeline via API request

Schedules

Query trigger metadata from database

Create custom block and pipeline templates for your project.

Custom templates

The `io_config` YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

IO Config

We support multiple kernels in the code editor.

Kernels

Enable features in your Mage project settings.

Project features

Display timestamps throughout the app in your local timezone.

Local timezone

How to set up a folder containing your Mage project code and other files.

Project setup

There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.

Get started with Git

Git in Mage

This page will walk you through the process of authenticating with SSH or HTTPS and configuring Git for use with Mage Git Sync.

Configure Git

Mage can automatically keep track of previous versions of a file.

File versions

File versioning in Mage

We love our UI, but here's how to develop in your favorite IDE, too.

Use an IDE

You can monitor many metrics for each of your pipelines and blocks.

Monitoring

Logging

Every data loader and transformer block has data validation capabilities built-in.

Data validation

Test the data produced by pipeline blocks using Great Expectations.

Great Expectations

Unit tests

Mage uses AI to help you build data pipelines faster— with our OpenAI integration, you can automate the tedious parts of pipeline development and focus on the fun stuff.

AI generated code

There are 2 dashboards: a dashboard for all your pipelines and a dashboard for each pipeline. You can add charts of various types with different sources of data. Use these dashboards for observability or for analytics.

Dashboards

Customizable dashboards

How to get the latest and greatest from the Mage team.

Updating Mage

We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.

Code of conduct

Code of Conduct

Please contribute usage statistics to help improve the developer experience for you and everyone in the community 🤝.

Usage statistics

Help improve the tool

Here are some frequently asked questions about Mage and our best answers.

Frequently Asked Questions

Mage guides for getting started, building pipelines, and more.

Mage Guides

Build a data pipeline that loads restaurant data, transforms it, then exports it to a DuckDB database. 🦆

ETL pipeline

ETL pipeline tutorial

Here are the high level steps to build a data integration pipeline:

Data integration

How to build a data integration pipeline

⏰ Run dbt models in Mage in under a minute.

Run dbt

Make dbt magical

In this tutorial, we’ll create a DAG in Airflow for scheduling and running a data pipeline; all from the Mage UI.

Trigger Mage from Airflow

Integrate Mage into an existing Airflow project

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

Transformer blocks

Execute SQL commands directly in your database, data warehouse, etc.

SQL blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

Custom blocks

A markdown block is a standalone text block that supports markdown and can be used to make notes or organize your blocks.

Markdown blocks

R blocks

Reuse the same block multiple times within a single pipeline.

Replicating blocks

Reorder blocks in the notebook when editing a pipeline.

Reordering blocks

Detach other pipeline associations from a block shared by multiple pipelines.

Detaching blocks

Build pipelines that ingest data from event streaming sources like Kafka.

Kafka streaming pipeline

This project uses sample data from the Google NYC Taxi Pubsub to create a streaming pipeline in Mage that reads and transforms a sample stream, ultimately writing output to a SCD Type-2 table in Postgres.

Streaming + dbt transform

🎏 Local stream to dbt transform

Build pipelines that ingest data from event streaming sources like RabbitMQ.

RabbitMQ streaming pipeline

Build a machine learning pipeline to train a model on the Titanic dataset.

ML pipeline

Machine learning pipeline tutorial

Import a pipeline zip file into your project.

Importing pipelines

Learn how to develop your favorite dbt models in Mage.

Develop dbt

Develop dbt in Mage

Use AI to perform various actions in Mage.

Using AI

Using AI (artificial intelligence)

AI Clients

Here's how you can connect a local database to Mage in Docker for development.

Connect a local db in Docker

A collection of Mage projects and examples created by our community.

Community Examples

Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.

Tips & tricks

Mage tips & tricks

Our API powers everything in the app— our UI makes calls to the backend, which operates the app. You can learn more about how these work and make your own calls in this section.

API overview

Resources

API resources

Policies for resources determine what actions can be taken on the resource, what attributes can be read, and which attributes can be written.

Policies

API policies

This will determine what attributes of the resource are returned to the client in the response. You can have different sets of attributes be included in the response based on the action (e.g. create, detail, delete, list, update) or a custom format.

Presenters

API presenters

Programmatically create multiple pipeline runs to backfill a pipeline.

Read backfills

Create backfill

Update backfill

Delete backfill

The Blocks API is used to create, update, and delete blocks objects from a pipeline.

Read block

Create block

Update block

Delete block

Fetch log entries for your Mage instance.

Read logs

Read OAuth tokens

Status and metrics for an individual run of a pipeline.

Read pipeline runs

Trigger pipeline

Read pipeline schedules

Create pipeline schedule

Data pipeline containing execution settings, resources, block information, and block dependencies.

Get Started

Concepts

Buildkite

Mage project setup

Buildkite setup for deployment to AWS

Get Started

Concepts

​Mage project setup

​Buildkite setup for deployment to AWS

Mage project setup

Buildkite setup for deployment to AWS