Introduction
Setup and install
mage start

Using Docker

  1. Create new project and launch tool

    docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
      /app/run_app.sh mage start [project_name]
    

    Windows

    If you are running Mage in Docker on Windows OS, $(pwd) won’t work. Instead, use the following command:

    docker run -it -p 6789:6789 -v %cd%:/home/src mageai/mageai /app/run_app.sh mage start [project_name]
    
  2. Open http://localhost:6789 in your browser and build a pipeline.

  3. Run pipeline after building it in the tool

    docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
      /app/run_app.sh mage run [project_name] [pipeline]
    

Initialize new project

If you want to create a different project with a different name, run the following:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  mage init [project_name]

Using PostgreSQL as database

Mage uses SQLite as the default database engine. To use PostgreSQL as the database engine, you’ll need to run your Docker container slightly differently:

  1. Create Docker network

    docker network create mage-app
    
  2. Start PostgreSQL Docker container in a separate window

    docker run --network mage-app --network-alias postgres_db \
       -it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
       -e POSTGRES_USER=<username> -e POSTGRES_PASSWORD=<password> \
       -e POSTGRES_DB=<database> -e PG_DATA=/var/lib/postgresql/data/pgdata \
       postgres:13-alpine3.17 postgres
    
  3. Launch Mage with MAGE_DATABASE_CONNECTION_URL environment variable

    docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
       -e MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://<username>:<password>@postgres_db:5432/<database> \
       mageai/mageai \
       /app/run_app.sh mage start another_repo
    

Using pip or conda

  1. Install Mage

    pip install mage-ai
    
    If you run into errors, see the errors section below.
    If you’re installing on Linux and need help, check out this community provided video walkthrough.
  2. Create new project and launch tool

    mage start [project_name]
    
  3. Open http://localhost:6789 in your browser and build a pipeline.

  4. Run pipeline after building it in the tool

    mage run [project_name] [pipeline]
    

Initialize new project

If you want to create a different project with a different name, run the following:

mage init [project_name]

Using Kubernetes

  1. Start a Kubernetes cluster locally

    Enable Kubernetes in Docker Desktop to start a kubernetes cluster locally.

    Other options for starting a Kubernetes cluster locally:

  2. Download and update Kubernetes config file

  3. Run Mage app in Kubernetes cluster

    • Install command line tool kubectl if you haven’t.
    • Run command kubectl create -f kube/app.yaml to run Mage in a Kubernetes pod.
    • You can check the pod status with command kubectl get pods -o wide.
    • Set up port forwarding with command kubectl port-forward mage-server 6789:6789.
  4. Open http://localhost:6789 in your browser and build a pipeline.

  5. (Optional) Enable blocks to be executed in separate Kubernetes pods.

    If you want the pipeline blocks to be executed in separate containers, you can follow the guide to set executor_type of a block to k8s in pipeline’s metadata.yaml.


Download new version of Mage

Using Docker

docker pull mageai/mageai:latest

Using pip

pip install -U mage-ai

In cloud provider

Using the cloud provider’s UI

Use your cloud provider’s UI to deploy a new container (e.g. revision, task definition, etc).

Using Mage terminal

  1. In Mage, go to /terminal.

  2. Type the following command in your terminal: ps aux|grep mage

  3. In the first row, under the 2nd column, note the process ID (e.g. 3).

  4. Kill the Mage process by running this command in your terminal: kill -9 [process_id].

    An example command killing process ID 3 could look like this: kill -9 3.


Environment variables

If you’re running Mage using pip or conda, your local machine’s environment variables are accessible within the running Mage app.

If you’re running Mage using Docker, you must add the following command line flags:

-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2

The command to run Mage using Docker could look like this:

docker run -it -p 6789:6789 -v $(pwd):/home/src \
    -e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2 \
    mageai/mageai /app/run_app.sh mage start demo_project

See here for a list of environment variables you can configure.


Using secret credential files in Docker

If you have files containing secret credentials (e.g. GCP credentials JSON files), you’ll need to mount them as a volume so that those files appear in your running Docker container.

macOS

If you have a file at this location /Users/dragon/secrets/credentials.json on your local machine, then run the following Docker command to start Mage and add files in the secrets/ folder into your Docker container:

docker run -it -p 6789:6789
    -v /Users/dragon/secrets:/home/secrets \
    -v $(pwd):/home/src mageai/mageai \
    /app/run_app.sh mage start demo_project

This flag in the command -v /Users/dragon/secrets:/home/secrets takes all the files from /Users/dragon/secrets and puts them in the Docker container under the folder /home/secrets.

Now inside Mage, you can load or refer to your credentials.json file using the absolute path: /home/secrets/credentials.json.

Windows

Alter the Docker command slightly to work for Windows file path syntax:

docker run -it -p 6789:6789 ^
    -v "C:\Some Path\dragon\secrets\credentials.json":/home/secrets ^
    -v "C:\Some Path\To Your Current Directory":/home/src mageai/mageai ^
    /app/run_app.sh mage start [project_name]

Local Docker resources

Memory

When running Mage locally using Docker, you may need to increase your Docker container’s memory limit. Add the following flag and value to your Docker run command: -m 2048m (this sets the memory limit to 2,048 megabytes).

Your Docker run command could look like this:

docker run -it -p 6789:6789 \
    -v $(pwd):/home/src mageai/mageai \
    -m 2048m \
    /app/run_app.sh mage start demo_project

Install

Installing extra packages

Mage also has the following extras:

  • azure: to install Azure related packages
  • bigquery: to connect to BigQuery for data import or export
  • clickhouse: to connect to Clickhouse for data import or export
  • dbt: to install DBT packages
  • google-cloud-storage: to connect to Google Cloud Storage for data import or export
  • hdf5: to process HDF5 file format
  • mysql: to connect to MySQL for data import or export
  • postgres: to connect to PostgreSQL for data import or export
  • redshift: to connect to Redshift for data import or export
  • s3: to connect to S3 for data import or export
  • snowflake: to connect to Snowflake for data import or export
  • spark: to use Spark (EMR) in your Mage pipeline
  • streaming: to use Streaming pipelines
  • all: to install all of the above to use all functionalities

Example:

pip install "mage-ai[spark]"

Errors

You may need to install development libraries for MIT Kerberos to use some Mage features.

On Ubuntu, this can be installed as:

apt install libkrb5-dev

Source code development

To setup a development environment for editing source code, please check out this document.


Building your own Docker image

docker build -t mage/dangerous .

Testing from branch

Install mage-ai library from your branch:

$ pip3 install git+https://github.com/mage-ai/mage-ai.git@branch_name#egg=mage-ai

Build the front-end code:

$ cd mage_ai/frontend
$ yarn install
$ yarn export_prod

Testing

Backend

$ ./scripts/server/test.sh

Front-end

To check proper types in TypeScript, run:

$ ./scripts/test.sh