Go from zero to Mage hero in under a minute. Weβll walk you through installing Mage and running your first pipeline. π¦ΈββοΈ
pip
or conda
.
If youβre familiar with Docker Compose or plan on adding or extending images (e.g. Postgres) in your project, we recommend starting from the Docker compose template. Otherwise, we recommend Docker run.
Docker
http://localhost:6789
in your browser and explore! If using any cloud provider use external IP address with the port eg 123.456.78:6789. We recommend one of our guides for inspiration.docker run
, which runs a Docker container from the Mage image. additionally, weβre mounting a volume (the -v
command) to persist our project files on our container. This mounts the current folder to /home/src
in the mage container. Weβre executing our docker image mageai/mageai
and running a script to start up the container.Docker Compose
http://localhost:6789
in your browser and explore! We recommend one of our guides to for inspiration.dev.env
to .env
, which is ignored by git in our configuration (to avoid exposing secrets), then running docker compose up
to start a Mage server.pip/conda
Install Mage
Create new project and launch tool
Open Mage
Kubernetes
Start a Kubernetes cluster locally
Download and update Kubernetes config file
/path/to/mage_project
in the config yaml with the path that you want to use to store your Mage projects.Run Mage app in Kubernetes cluster
kubectl create -f kube/app.yaml
to run Mage in a Kubernetes pod. You can check the pod status with command kubectl get pods -o wide
.
Set up port forwarding with command kubectl port-forward mage-server 6789:6789
.Open Mage
http://localhost:6789
. From the pipelines page, select example_pipeline
and open the notebook view by selecting Edit pipeline
from the left side nav.
π€ What's a block?
custom
, loader
, transformer
, etc) will be executed. The first argument to the function must be a dataframe or JSON serializable object. The function can return a dataframe or JSON serializable object, or it can return nothing.You can think of this function like a python main()
function, when we run the block, this function will be executed. For example, the following block will return the first 10 rows of the dataframe passed in:Package | Install | Description |
---|---|---|
all | mage-ai[all] | install all add-ons |
azure | mage-ai[azure] | install Azure related packages |
clickhouse | mage-ai[clickhouse] | use Clickhouse for data import or export |
dbt | mage-ai[dbt] | install dbt packages |
google-cloud-storage | mage-ai[google-cloud-storage] | use Google Cloud Storage for data import or export |
hdf5 | mage-ai[hdf5] | process data in HDF5 file format |
mysql | mage-ai[mysql] | use MySQL for data import or export |
postgres | mage-ai[postgres] | use PostgreSQL for data import or export |
redshift | mage-ai[redshift] | use Redshift for data import or export |
s3 | mage-ai[s3] | use S3 for data import or export |
snowflake | mage-ai[snowflake] | use Snowflake for data import or export |
spark | mage-ai[spark] | use Spark (EMR) in your Mage pipeline |
streaming | mage-ai[streaming] | use Streaming pipelines |
requirements.txt
file:
requirements.txt
here.