We support running the pipeline in Airflow DAGs.
- You need to firstly install
mage_ai
library by addingmage_ai
to your requirements.txt file. - Then you need to download the mage pipeline code into your Airflow directory. You can achieve it by using a git submodule in your Airflow directory.
- In your Mage project’s metadata.yaml file, please specify a variables_dir that you want to store the output of each block execution (You need to have write permission to the variables_dir). Example:
variables_dir: /tmp
.
- Create DAGs for all the pipelines in Mage project
- Run pipeline in a BashOperator
- Run pipeline in a PythonOperator
- Run pipeline as an Airflow DAG