Setup and install

Introduction
Setup and install

Using Docker

Create new project and launch tool

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage start [project_name]

Windows

If you are running Mage in Docker on Windows OS, $(pwd) won’t work. Instead, use the following command:

docker run -it -p 6789:6789 -v %cd%:/home/src mageai/mageai /app/run_app.sh mage start [project_name]

Open http://localhost:6789 in your browser and build a pipeline.

Run pipeline after building it in the tool

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage run [project_name] [pipeline]

Initialize new project

If you want to create a different project with a different name, run the following:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  mage init [project_name]

Using PostgreSQL as database

Mage uses SQLite as the default database engine. To use PostgreSQL as the database engine, you’ll need to run your Docker container slightly differently:

Create Docker network
```
docker network create mage-app
```

Start PostgreSQL Docker container in a separate window

docker run --network mage-app --network-alias postgres_db \
   -it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
   -e POSTGRES_USER=<username> -e POSTGRES_PASSWORD=<password> \
   -e POSTGRES_DB=<database> -e PG_DATA=/var/lib/postgresql/data/pgdata \
   postgres:13-alpine3.17 postgres

Launch Mage with MAGE_DATABASE_CONNECTION_URL environment variable

docker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \
   -e MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://<username>:<password>@postgres_db:5432/<database> \
   mageai/mageai \
   /app/run_app.sh mage start another_repo

Using `pip` or `conda`

Install Mage
```
pip install mage-ai
```
_{If you run into errors, see the errors section below.}
Create new project and launch tool
```
mage start [project_name]
```
Open http://localhost:6789 in your browser and build a pipeline.
Run pipeline after building it in the tool
```
mage run [project_name] [pipeline]
```

Initialize new project

If you want to create a different project with a different name, run the following:

mage init [project_name]

Using Kubernetes

Start a Kubernetes cluster locally

Enable Kubernetes in Docker Desktop to start a kubernetes cluster locally.

Other options for starting a Kubernetes cluster locally:
- Kind
- Minikube
Download and update Kubernetes config file
- Download Mage Kubernetes config file. https://github.com/mage-ai/mage-ai/blob/master/kube/app.yaml
- Replace the /path/to/mage_project in the config yaml with the path that you want to use to store your Mage projects.
Run Mage app in Kubernetes cluster
- Install command line tool kubectl if you haven’t.
- Run command kubectl create -f kube/app.yaml to run Mage in a Kubernetes pod.
- You can check the pod status with command kubectl get pods -o wide.
- Set up port forarding with command kubectl port-forward mage-server 6789:6789.
Open http://localhost:6789 in your browser and build a pipeline.
(Optional) Enable blocks to be executed in separate Kubernetes pods.

If you want the pipeline blocks to be executed in separate containers, you can follow the guide to set executor_type of a block to k8s in pipeline’s metadata.yaml.

Download new version of Mage

Using Docker

docker pull mageai/mageai:latest

Using `pip`

pip install -U mage-ai

Environment variables

If you’re running Mage using pip or conda, your local machine’s environment variables are accessible within the running Mage app.

If you’re running Mage using Docker, you must add the following command line flags:

-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2

The command to run Mage using Docker could look like this:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
    -e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2 \
    /app/run_app.sh mage start demo_project

Using secret credential files in Docker

If you have files containing secret credentials (e.g. GCP credentials JSON files), you’ll need to mount them as a volume so that those files appear in your running Docker container.

macOS

If you have a file at this location /Users/dragon/secrets/credentials.json on your local machine, then run the following Docker command to start Mage and add files in the secrets/ folder into your Docker container:

docker run -it -p 6789:6789
    -v /Users/dragon/secrets:/home/secrets \
    -v $(pwd):/home/src mageai/mageai mageai/mageai \
    /app/run_app.sh mage start demo_project

This flag in the command -v /Users/dragon/secrets:/home/secrets takes all the files from /Users/dragon/secrets and puts them in the Docker container under the folder /home/secrets.

Now inside Mage, you can load or refer to your credentials.json file using the absolute path: /home/secrets/credentials.json.

Windows

Alter the Docker command slightly to work for Windows file path syntax:

docker run -it -p 6789:6789
    -v "C:\Some Path\dragon\secrets\credentials.json":/home/secrets
    -v "C:\Some Path\To Your Current Directory":/home/src mageai/mageai \
    /app/run_app.sh mage start [project_name]

Local Docker resources

Memory

When running Mage locally using Docker, you may need to increase your Docker container’s memory limit. Add the following flag and value to your Docker run command: -m 2048m (this sets the memory limit to 2,048 megabytes).

Your Docker run command could look like this:

docker run -it -p 6789:6789 \
    -v $(pwd):/home/src mageai/mageai \
    -m 2048m \
    /app/run_app.sh mage start demo_project

Install

Installing extra packages

Mage also has the following extras:

spark: to use Spark in your Mage pipeline
bigquery: to connect to BigQuery for data import or export
hdf5: to process HDF5 file format
postgres: to connect to PostgreSQL for data import or export
redshift: to connect to Redshift for data import or export
s3: to connect to S3 for data import or export
snowflake: to connect to Snowflake for data import or export
all: to install all of the above to use all functionalities

Example:

pip install "mage-ai[spark]"

Errors

You may need to install development libraries for MIT Kerberos to use some Mage features.

On Ubuntu, this can be installed as:

apt install libkrb5-dev

Source code development

To setup a development environment for editing source code, please check out this document.

Building your own Docker image

docker build -t mage/dangerous .

Testing from branch

Install mage-ai library from your branch:

$ pip3 install git+https://github.com/mage-ai/mage-ai.git@branch_name#egg=mage-ai

Build the front-end code:

$ cd mage_ai/frontend
$ yarn install
$ yarn export_prod

Testing

Backend

$ ./scripts/server/test.sh

Front-end

To check proper types in TypeScript, run:

$ ./scripts/test.sh

​Using Docker

​Initialize new project

​Using PostgreSQL as database

​Using pip or conda

​Initialize new project

​Using Kubernetes

​Download new version of Mage

​Using Docker

​Using pip

​Environment variables

​Using secret credential files in Docker

​macOS

​Windows

​Local Docker resources

​Memory

​Install

​Installing extra packages

​Errors

​Source code development

​Building your own Docker image

​Testing from branch

​Testing

​Backend

​Front-end

Using Docker

Initialize new project

Using PostgreSQL as database

Using `pip` or `conda`

Initialize new project

Using Kubernetes

Download new version of Mage

Using Docker

Using `pip`

Environment variables

Using secret credential files in Docker

macOS

Windows

Local Docker resources

Memory

Install

Installing extra packages

Errors

Source code development

Building your own Docker image

Testing from branch

Testing

Backend

Front-end