Install Python packages using `requirements.txt`
Mage allows you to easily integrate third-party Python packages into your project and pipelines. This guide will walk you through the process of adding and managing external dependencies.
Add Python packages to requirements.txt
Location
At the root of your Mage project directory, you’ll find a file named requirements.txt.
This file is used to specify all the Python packages that your project depends on.
Path
At the root of your project directory, there is a file named requirements.txt. Open this file
and add the Python package you want installed and available throughout your entire project.
For example, if your project is named demo_project then the file will be located at
demo_project/requirements.txt.
Adding packages
To add a package, simply open the requirements.txt file and list each package on a new line. You can also specify version requirements if needed.
Example requirements.txt:
In this example the requirements.txt file will install:
- matplotlib version 3.5.2
- tqdm
Automatic installation
When installing Mage using Docker, Mage provides two main methods for automatically installing third-party packages at runtime.
These methods utilize the USER_CODE_PATH environment variable, which defines your project path (e.g. /home/src/project_name),
to locate your requirements.txt file and install the packages listed there.
Using Command Line Flags
Using Command Line Flags
Using command line flags
If you installed Mage using Docker, the USER_CODE_PATH can be set using command line flags. You will need to add these command line flags at every run when running a new Docker container or installing a new version of Mage.
Setting USER_CODE_PATH in Docker
The USER_CODE_PATH environment variable tells Mage where your project code is located within the Docker container. This is essential for features like automatic package installation from requirements.txt.
Docker run command
To set the USER_CODE_PATH when running a Mage Docker container, use the following command structure:
Let’s break down this command:
it: Runs the container interactively with a pseudo-TTY.p 6789:6789: Maps port 6789 from the container to your host machine.v $(pwd):/home/src: Mounts your current directory to/home/srcin the container.e USER_CODE_PATH=/home/src/project_name: Sets theUSER_CODE_PATHenvironment variable.mageai/mageai: Specifies the Mage Docker image to use./app/run_app.sh mage start [project_name]: The command to run inside the container.
Important Notes
- Persistence: The
USER_CODE_PATHsetting is not persistent across container restarts or when installing new versions of Mage. You must include these flags each time you run a new Docker container. - Project Name: Replace
[project_name]with your actual Mage project name in both theUSER_CODE_PATHand the start command. - Path Consistency: Ensure that the path in
USER_CODE_PATHmatches the mount point specified in thevflag. In this example, we mount to/home/src, so theUSER_CODE_PATHstarts with/home/src. - Working Directory: The
$(pwd)in thevflag assumes you’re running the Docker command from your project’s parent directory. Adjust this path if you’re running from a different location.
For more detailed information on environment variables and their impact on Mage’s functionality, please refer to our comprehensive environment variable documentation.
Using Dockerfile
Using Dockerfile
Using Dockerfiles
One important aspect of configuring Mage is setting the USER_CODE_PATH, which tells Mage where your project code is located.
While this can be done using command-line flags when running Docker containers, it’s also possible to set the USER_CODE_PATH
directly in your Dockerfile. This approach can be more convenient for consistent deployments and when building custom images.
Breakdown of the Dockerfile:
FROM mageai/mageai:latest- Specifies the base image as the latest version of the official Mage AI image.
- Project-specific arguments:
PROJECT_NAME: Should be replaced with your actual project name.MAGE_CODE_PATH: Sets the base path for Mage code.USER_CODE_PATH: Constructs the full path to your project, effectively setting theUSER_CODE_PATHfor Mage.
WORKDIR ${MAGE_CODE_PATH}- Sets the working directory inside the container to the path specified by
MAGE_CODE_PATH.
- Sets the working directory inside the container to the path specified by
COPY ${PROJECT_NAME} ${PROJECT_NAME}- Copies your project files from the host machine into the container.
CMD ["/bin/sh", "-c", "/app/run_app.sh"]- Specifies the command to run when the container starts, which launches the Mage application.
To use this Dockerfile:
- Replace
[project_name]with your actual project name. - Ensure your project files are in a directory named after your project.
- Build the Docker image using this Dockerfile.
- Run the container to start your Mage environment.
This Dockerfile demonstrates how to set the USER_CODE_PATH using build arguments. By defining it this way, you ensure that Mage will
consistently know where to find your project code within the container, without needing to specify it each time you run the container.
Follow the Repository setup documentation to deploy this Dockerfile to fully automate the installation of third-party Python pacakges.
If you’re running Mage using pip or conda, your local machine’s environment variables are accessible within the running Mage app.
Prerequisites
-
Ensure that the environment variable
USER_CODE_PATHis set to your project path, using the command line flags or the Dockerfile methods.Example:
/home/src/project_name -
Mage uses a script named
run_app.shto manage the application startup process.
Installation process
When you start your Mage server:
- The
run_app.shscript checks theUSER_CODE_PATHenvironment variable. - It locates the
requirements.txtfile within the specified project directory. - All packages listed in
requirements.txtare automatically installed before the server starts.
This ensures that all required packages are available for your project and pipelines.