Setup and install

Using Docker
-
Create new project and launch tool
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \ /app/run_app.sh mage start [project_name]
Windows
If you are running Mage in Docker on Windows OS,
$(pwd)
won’t work. Instead, use the following command:docker run -it -p 6789:6789 -v %cd%:/home/src mageai/mageai /app/run_app.sh mage start [project_name]
-
Open http://localhost:6789 in your browser and build a pipeline.
-
Run pipeline after building it in the tool
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \ /app/run_app.sh mage run [project_name] [pipeline]
Initialize new project
If you want to create a different project with a different name, run the following:
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
mage init [project_name]
Using PostgreSQL as database
Mage uses SQLite as the default database engine. To use PostgreSQL as the database engine, you’ll need to run your Docker container slightly differently:
-
Create Docker network
docker network create mage-app
-
Start PostgreSQL Docker container in a separate window
docker run --network mage-app --network-alias postgres_db \ -it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \ -e POSTGRES_USER=<username> -e POSTGRES_PASSWORD=<password> \ -e POSTGRES_DB=<database> -e PG_DATA=/var/lib/postgresql/data/pgdata \ postgres:13-alpine3.17 postgres
-
Launch Mage with
MAGE_DATABASE_CONNECTION_URL
environment variabledocker run --network mage-app -it -p 6789:6789 -v $(pwd):/home/src \ -e MAGE_DATABASE_CONNECTION_URL=postgresql+psycopg2://<username>:<password>@postgres_db:5432/<database> \ mageai/mageai \ /app/run_app.sh mage start another_repo
Using pip
or conda
-
Install Mage
If you run into errors, see the errors section below.pip install mage-ai
If you’re installing on Linux and need help, check out this community provided video walkthrough. -
Create new project and launch tool
mage start [project_name]
-
Open http://localhost:6789 in your browser and build a pipeline.
-
Run pipeline after building it in the tool
mage run [project_name] [pipeline]
Initialize new project
If you want to create a different project with a different name, run the following:
mage init [project_name]
Using Kubernetes
-
Start a Kubernetes cluster locally
Enable Kubernetes in Docker Desktop to start a kubernetes cluster locally.
Other options for starting a Kubernetes cluster locally:
-
Download and update Kubernetes config file
- Download Mage Kubernetes config file. https://github.com/mage-ai/mage-ai/blob/master/kube/app.yaml
- Replace the
/path/to/mage_project
in the config yaml with the path that you want to use to store your Mage projects.
-
Run Mage app in Kubernetes cluster
- Install command line tool kubectl if you haven’t.
- Run command
kubectl create -f kube/app.yaml
to run Mage in a Kubernetes pod. - You can check the pod status with command
kubectl get pods -o wide
. - Set up port forwarding with command
kubectl port-forward mage-server 6789:6789
.
-
Open http://localhost:6789 in your browser and build a pipeline.
-
(Optional) Enable blocks to be executed in separate Kubernetes pods.
If you want the pipeline blocks to be executed in separate containers, you can follow the guide to set
executor_type
of a block tok8s
in pipeline’smetadata.yaml
.
Download new version of Mage
Using Docker
docker pull mageai/mageai:latest
Using pip
pip install -U mage-ai
In cloud provider
Using the cloud provider’s UI
Use your cloud provider’s UI to deploy a new container (e.g. revision, task definition, etc).
Using Mage terminal
-
In Mage, go to
/terminal
. -
Type the following command in your terminal:
ps aux|grep mage
-
In the first row, under the 2nd column, note the process ID (e.g.
3
). -
Kill the Mage process by running this command in your terminal:
kill -9 [process_id]
.An example command killing process ID
3
could look like this:kill -9 3
.
Environment variables
If you’re running Mage using pip
or conda
, your local machine’s environment variables
are accessible within the running Mage app.
If you’re running Mage using Docker, you must add the following command line flags:
-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2
The command to run Mage using Docker could look like this:
docker run -it -p 6789:6789 -v $(pwd):/home/src \
-e SOME_VARIABLE_NAME_1=secret_value_1 -e SOME_VARIABLE_NAME_2=secret_value_2 \
mageai/mageai /app/run_app.sh mage start demo_project
See here for a list of environment variables you can configure.
Using secret credential files in Docker
If you have files containing secret credentials (e.g. GCP credentials JSON files), you’ll need to mount them as a volume so that those files appear in your running Docker container.
macOS
If you have a file at this location /Users/dragon/secrets/credentials.json
on your local machine,
then run the following Docker command to start Mage and add files in the secrets/
folder into your Docker container:
docker run -it -p 6789:6789
-v /Users/dragon/secrets:/home/secrets \
-v $(pwd):/home/src mageai/mageai \
/app/run_app.sh mage start demo_project
This flag in the command -v /Users/dragon/secrets:/home/secrets
takes all the files from /Users/dragon/secrets
and puts them in the Docker container under the folder /home/secrets
.
Now inside Mage, you can load or refer to your credentials.json
file using the absolute path:
/home/secrets/credentials.json
.
Windows
Alter the Docker command slightly to work for Windows file path syntax:
docker run -it -p 6789:6789 ^
-v "C:\Some Path\dragon\secrets\credentials.json":/home/secrets ^
-v "C:\Some Path\To Your Current Directory":/home/src mageai/mageai ^
/app/run_app.sh mage start [project_name]
Local Docker resources
Memory
When running Mage locally using Docker, you may need to increase your Docker container’s
memory limit. Add the following flag and value to your Docker run command:
-m 2048m
(this sets the memory limit to 2,048 megabytes).
Your Docker run command could look like this:
docker run -it -p 6789:6789 \
-v $(pwd):/home/src mageai/mageai \
-m 2048m \
/app/run_app.sh mage start demo_project
Install
Installing extra packages
Mage also has the following extras:
- azure: to install Azure related packages
- bigquery: to connect to BigQuery for data import or export
- clickhouse: to connect to Clickhouse for data import or export
- dbt: to install DBT packages
- google-cloud-storage: to connect to Google Cloud Storage for data import or export
- hdf5: to process HDF5 file format
- mysql: to connect to MySQL for data import or export
- postgres: to connect to PostgreSQL for data import or export
- redshift: to connect to Redshift for data import or export
- s3: to connect to S3 for data import or export
- snowflake: to connect to Snowflake for data import or export
- spark: to use Spark (EMR) in your Mage pipeline
- streaming: to use Streaming pipelines
- all: to install all of the above to use all functionalities
Example:
pip install "mage-ai[spark]"
Errors
You may need to install development libraries for MIT Kerberos to use some Mage features.
On Ubuntu, this can be installed as:
apt install libkrb5-dev
Source code development
To setup a development environment for editing source code, please check out this document.
Building your own Docker image
docker build -t mage/dangerous .
Testing from branch
Install mage-ai
library from your branch:
$ pip3 install git+https://github.com/mage-ai/mage-ai.git@branch_name#egg=mage-ai
Build the front-end code:
$ cd mage_ai/frontend
$ yarn install
$ yarn export_prod
Testing
Backend
$ ./scripts/server/test.sh
Front-end
To check proper types in TypeScript, run:
$ ./scripts/test.sh