Setup and install

Introduction

Setup and install

Documentation
Watch 2 min demo
Play with live tool
Get instant help

Introduction

Mage

Setup

Contributing

Tutorials

ETL pipeline

Data integration pipeline

DBT orchestration

Machine learning pipeline

Design

Core design principles

Core abstractions

Orchestration

Blocks

Data integrations

Overview

Sources

Destinations

DBT

Development

Project setup

Pipelines

Blocks

Dependencies

Streaming pipelines

Runtime variables

Testing

Secrets

Versioning

Environment variables

Team management

Developing w/o UI

Troubleshooting

Orchestration

Schedules and triggers

Backfills

Pipeline runs

Production

Authentication

CI/CD

Deployment

Configuration

Version control

Observability

Logging

Monitoring

Alerting

Integrations

Spark

Databricks

Data

Airflow

Prefect

Guides

Visualizations

Misc

Community

Contributing

Backend

Frontend

Usage statistics

About

Features

Releases

Roadmap

Using Docker

Create new project and launch tool

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage start [project_name]

Windows

If you are running Mage in Docker on Windows OS, $(pwd) won’t work. Instead, use the following command:

docker run -it -p 6789:6789 -v %cd%:/home/src mageai/mageai /app/run_app.sh mage start [project_name]

Open http://localhost:6789 in your browser and build a pipeline.

Run pipeline after building it in the tool

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage run [project_name] [pipeline]

Initialize new project

If you want to create a different project with a different name, run the following:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  mage init [project_name]

Using PostgreSQL as database

Mage uses SQLite as the default database engine. To use PostgreSQL as the database engine, you’ll need to run your Docker container slightly differently:

Create Docker network
```
docker network create mage-app
```

Start PostgreSQL Docker container in a separate window

docker run --network mage-app --network-alias postgres_db \
   -it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
   -e POSTGRES_USER=<username> -e POSTGRES_PASSWORD=<password> \
   -e POSTGRES_DB=<database> -e PG_DATA=/var/lib/postgresql/data/pgdata \
   postgres:13-alpine3.17 postgres

Launch Mage with MAGE_DATABASE_CONNECTION_URL environment variable

docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
   -e MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://<username>:<password>@postgres_db:5432/<database> \
   mageai/mageai \
   /app/run_app.sh mage start another_repo

Using `pip` or `conda`

Install Mage
```
pip install mage-ai
```
_{If you run into errors, see the errors section below.}
_{If you’re installing on Linux and need help, check out this community provided video walkthrough.}
Create new project and launch tool
```
mage start [project_name]
```
Open http://localhost:6789 in your browser and build a pipeline.
Run pipeline after building it in the tool
```
mage run [project_name] [pipeline]
```

Initialize new project

If you want to create a different project with a different name, run the following:

mage init [project_name]

Using Kubernetes

Start a Kubernetes cluster locally

Enable Kubernetes in Docker Desktop to start a kubernetes cluster locally.

Other options for starting a Kubernetes cluster locally:
- Kind
- Minikube
Download and update Kubernetes config file
- Download Mage Kubernetes config file. https://github.com/mage-ai/mage-ai/blob/master/kube/app.yaml
- Replace the /path/to/mage_project in the config yaml with the path that you want to use to store your Mage projects.
Run Mage app in Kubernetes cluster
- Install command line tool kubectl if you haven’t.
- Run command kubectl create -f kube/app.yaml to run Mage in a Kubernetes pod.
- You can check the pod status with command kubectl get pods -o wide.
- Set up port forwarding with command kubectl port-forward mage-server 6789:6789.
Open http://localhost:6789 in your browser and build a pipeline.
(Optional) Enable blocks to be executed in separate Kubernetes pods.

If you want the pipeline blocks to be executed in separate containers, you can follow the guide to set executor_type of a block to k8s in pipeline’s metadata.yaml.

Download new version of Mage

Using Docker

docker pull mageai/mageai:latest

Using `pip`

pip install -U mage-ai

In cloud provider

Using the cloud provider’s UI

Use your cloud provider’s UI to deploy a new container (e.g. revision, task definition, etc).

Using Mage terminal

In Mage, go to /terminal.
Type the following command in your terminal: ps aux|grep mage
In the first row, under the 2nd column, note the process ID (e.g. 3).
Kill the Mage process by running this command in your terminal: kill -9 [process_id].

An example command killing process ID 3 could look like this: kill -9 3.

Environment variables

If you’re running Mage using pip or conda, your local machine’s environment variables are accessible within the running Mage app.

If you’re running Mage using Docker, you must add the following command line flags:

-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2

The command to run Mage using Docker could look like this:

docker run -it -p 6789:6789 -v $(pwd):/home/src \
    -e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2 \
    mageai/mageai /app/run_app.sh mage start demo_project

See here for a list of environment variables you can configure.

Using secret credential files in Docker

If you have files containing secret credentials (e.g. GCP credentials JSON files), you’ll need to mount them as a volume so that those files appear in your running Docker container.

macOS

If you have a file at this location /Users/dragon/secrets/credentials.json on your local machine, then run the following Docker command to start Mage and add files in the secrets/ folder into your Docker container:

docker run -it -p 6789:6789
    -v /Users/dragon/secrets:/home/secrets \
    -v $(pwd):/home/src mageai/mageai \
    /app/run_app.sh mage start demo_project

This flag in the command -v /Users/dragon/secrets:/home/secrets takes all the files from /Users/dragon/secrets and puts them in the Docker container under the folder /home/secrets.

Now inside Mage, you can load or refer to your credentials.json file using the absolute path: /home/secrets/credentials.json.

Windows

Alter the Docker command slightly to work for Windows file path syntax:

docker run -it -p 6789:6789 ^
    -v "C:\Some Path\dragon\secrets\credentials.json":/home/secrets ^
    -v "C:\Some Path\To Your Current Directory":/home/src mageai/mageai ^
    /app/run_app.sh mage start [project_name]

Local Docker resources

Memory

When running Mage locally using Docker, you may need to increase your Docker container’s memory limit. Add the following flag and value to your Docker run command: -m 2048m (this sets the memory limit to 2,048 megabytes).

Your Docker run command could look like this:

docker run -it -p 6789:6789 \
    -v $(pwd):/home/src mageai/mageai \
    -m 2048m \
    /app/run_app.sh mage start demo_project

Install

Installing extra packages

Mage also has the following extras:

azure: to install Azure related packages
bigquery: to connect to BigQuery for data import or export
clickhouse: to connect to Clickhouse for data import or export
dbt: to install DBT packages
google-cloud-storage: to connect to Google Cloud Storage for data import or export
hdf5: to process HDF5 file format
mysql: to connect to MySQL for data import or export
postgres: to connect to PostgreSQL for data import or export
redshift: to connect to Redshift for data import or export
s3: to connect to S3 for data import or export
snowflake: to connect to Snowflake for data import or export
spark: to use Spark (EMR) in your Mage pipeline
streaming: to use Streaming pipelines
all: to install all of the above to use all functionalities

Example:

pip install "mage-ai[spark]"

Errors

You may need to install development libraries for MIT Kerberos to use some Mage features.

On Ubuntu, this can be installed as:

apt install libkrb5-dev

Source code development

To setup a development environment for editing source code, please check out this document.

Building your own Docker image

docker build -t mage/dangerous .

Testing from branch

Install mage-ai library from your branch:

$ pip3 install git+https://github.com/mage-ai/mage-ai.git@branch_name#egg=mage-ai

Build the front-end code:

$ cd mage_ai/frontend
$ yarn install
$ yarn export_prod

Testing

Backend

$ ./scripts/server/test.sh

Front-end

To check proper types in TypeScript, run:

$ ./scripts/test.sh

Introduction

Tutorials

Design

Data integrations

DBT

Development

Orchestration

Production

Observability

Integrations

Guides

Community

Contributing

About

​Using Docker

​Initialize new project

​Using PostgreSQL as database

​Using pip or conda

​Initialize new project

​Using Kubernetes

​Download new version of Mage

​Using Docker

​Using pip

​In cloud provider

​Using the cloud provider’s UI

​Using Mage terminal

​Environment variables

​Using secret credential files in Docker

​macOS

​Windows

​Local Docker resources

​Memory

​Install

​Installing extra packages

​Errors

​Source code development

​Building your own Docker image

​Testing from branch

​Testing

​Backend

​Front-end

Using Docker

Initialize new project

Using PostgreSQL as database

Using `pip` or `conda`

Initialize new project

Using Kubernetes

Download new version of Mage

Using Docker

Using `pip`

In cloud provider

Using the cloud provider’s UI

Using Mage terminal

Environment variables

Using secret credential files in Docker

macOS

Windows

Local Docker resources

Memory

Install

Installing extra packages

Errors

Source code development

Building your own Docker image

Testing from branch

Testing

Backend

Front-end