A source is a system that you want to load data from and synchronize it into another system. A source can be a 3rd party API, SaaS, database, data warehouse, or a data lake.

Overview

Sources technical documentation

mage

Releases

Roadmap

Troubleshooting

Mage

Docs

Guides

Deploy

Contribute

Slack

Documentation

Mage is an open-source, hybrid framework for transforming and integrating data. ✨

🧙‍♀️ Welcome to Mage

Go from zero to Mage hero in under a minute. We'll walk you through installing Mage and running your first pipeline. 🦸‍♀️

Quickstart

Every user experience and technical design decision adheres to these principles.

Design

Core Design Principles

These are the fundamental concepts that Mage uses to operate.

Core abstractions

Abstractions

Details about how the Mage file directory works and how typical projects are structured.

Project Structure

Backfilling pipelines

Guide

Backfill guide

Each block in a pipeline maps to an individual file within a project. Blocks can be reused and shared across multiple pipelines within the same project.

Use the output of a block to dynamically create more blocks.

Dynamic blocks

Data integration is the process of synchronizing data between two systems.

There are many ways to configure your data integration pipeline.

Configuration

Configure data integration pipelines

dbt Integration

Generate and orchestrate the final output of a pipeline (aka data product). Data products can be globally referenced from any pipeline and its data output can be used in any block.

Global data products

Pipelines

Pipeline Management

Retrying block runs

Retrying block runs from a pipeline run

Saving block output as CSV

Saving block output as CSV file

Configure triggers in triggers.yaml under pipeline folder.

Configure triggers in code

You can trigger another pipeline from a block within a different pipeline.

Trigger from block

Trigger pipeline from a block

You can trigger a pipeline by making an API request.

Webhooks

Triggering pipeline via API request

Schedules

Create custom block and pipeline templates for your project.

Custom templates

We support multiple kernels in the code editor.

Kernels

How to set up a folder containing your Mage project code and other files.

Project setup

There are a number of ways to enable version control, collaboration, and easy deployment using Git in Mage. We'll cover them all on this page.

Using Git in Mage

Git with SSH

Configure Git with SSH or HTTPS

Mage can automatically keep track of previous versions of a file.

File versions

File versioning in Mage

We love our UI, but here's how to develop in your favorite IDE, too.

Use an IDE

You can monitor many metrics for each of your pipelines and blocks.

Monitoring

Logging

Every data loader and transformer block has data validation capabilities built-in.

Data validation

Test the data produced by pipeline blocks using Great Expectations.

Great Expectations

Unit tests

There are 2 dashboards: a dashboard for all your pipelines and a dashboard for each pipeline. You can add charts of various types with different sources of data. Use these dashboards for observability or for analytics.

Dashboards

Customizable dashboards

Mage uses AI to help you build data pipelines faster— with our OpenAI integration, you can automate the tedious parts of pipeline development and focus on the fun stuff.

AI generated code

How to get the latest and greatest from the Mage team.

Updating Mage

We've adopted the [Contributor Covenant](https://www.contributor-covenant.org/) Code of Conduct for the Mage community. Please review the following to understand our standards and expectations for participation.

Code of conduct

Code of Conduct

Please contribute usage statistics to help improve the developer experience for you and everyone in the community 🤝.

Usage statistics

Help improve the tool

Here are some frequently asked questions about Mage and our best answers.

Frequently Asked Questions

Mage guides for getting started, building pipelines, and more.

Mage Guides

Build a data pipeline that loads restaurant data, visualizes it, transforms it, then exports it to a PostgreSQL database.

ETL pipeline

ETL pipeline tutorial

Here are the high level steps to build a data integration pipeline:

Data integration

How to build a data integration pipeline

dbt quickstart

Set up dbt models & orchestrate dbt runs

In this tutorial, we’ll create a DAG in Airflow for scheduling and running a data pipeline; all from the Mage UI.

Trigger Mage from Airflow

Integrate Mage into an existing Airflow project

Transformer Actions are a library of modular, reusable data transformations, reducing the boilerplate to perform common transformations on your data.

Transformer blocks

Execute SQL commands directly in your database, data warehouse, etc.

SQL blocks

Custom blocks are generic blocks for organizing your pipeline that may not fit into any of the other category of blocks. Unlike Scratchpad blocks, Custom blocks are executed as part of the pipeline.

Custom blocks

A markdown block is a standalone text block that supports markdown and can be used to make notes or organize your blocks.

Markdown blocks

R blocks

Reuse the same block multiple times within a single pipeline.

Replicating blocks

Build pipelines that ingest data from event streaming sources like Kafka.

Kafka streaming pipeline

This project uses sample data from the Google NYC Taxi Pubsub to create a streaming pipeline in Mage that reads and transforms a sample stream, ultimately writing output to a SCD Type-2 table in Postgres.

Streaming + dbt transform

🎏 Local stream to dbt transform

Build pipelines that ingest data from event streaming sources like RabbitMQ.

RabbitMQ streaming pipeline

Build a machine learning pipeline to train a model on the Titanic dataset.

ML pipeline

Machine learning pipeline tutorial

Use AI to perform various actions in Mage.

Using AI

Using AI (artificial intelligence)

A collection of Mage projects and examples created by our community.

Community Examples

Here are all the great ways you can make the most of Mage.

Tips & tricks

Mage tips & tricks

The front-end client (e.g. browser) creates, reads, updates, and deletes (CRUD) resources by using the backend server’s API.

API overview

Resources

API resources

Policies for resources determine what actions can be taken on the resource, what attributes can be read, and which attributes can be written.

Policies

API policies

This will determine what attributes of the resource are returned to the client in the response. You can have different sets of attributes be included in the response based on the action (e.g. create, detail, delete, list, update) or a custom format.

Presenters

API presenters

Programmatically create multiple pipeline runs to backfill a pipeline.

Read backfills

Create backfill

Update backfill

Delete backfill

The Blocks API is used to create, update, and delete blocks objects from a pipeline.

Read block

Create block

Update block

Delete block

Fetch log entries for your Mage instance.

Read logs

Read OAuth tokens

Status and metrics for an individual run of a pipeline.

Read pipeline runs

Trigger pipeline

Read pipeline schedules

Data pipeline containing execution settings, resources, block information, and block dependencies.

Read pipelines

Create pipeline

Update pipeline

Delete pipeline

Create session

Architecture

Terraform

Helm

Create users, manage users, and require sign in to authenticate and use Mage.

User authentication

Enable signing in with Microsoft account in Mage.

Microsoft

Sign in with Microsoft

There are 4 development and deployment workflows.

Repository setup

Development (local) and production (cloud) using GitHub Actions.

GitHub Actions

Development (local) and production (cloud) using GitLab CI/CD.

GitLab CI/CD

Buildkite

Production Configuration

Compute resources

Databases

AWS Secrets Manager

Azure

Azure Key Vault

GCP Secrets Management

Mage is designed and built to be cloud native.

Multi-development environment in the cloud

Mage integrates with Git— you can version control and deploy Mage from a repo.

Git integration

Authenticate with GitHub then pull from a remote repository, push local changes to a remote repository, and create pull requests for a remote repository.

GitHub integration

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Get Started

Concepts

Reference

About

Sources technical documentation

Singer spec

Folder structure