Details about how the Mage file directory works and how typical projects are structured.
mage_data
) and individual projects are housed. You can think of a project as an environment: it contains configurations, pipelines, and other project-specific data.
pipelines
folder under the Mage project directory: [project_dir]/pipelines/[pipeline_name]/metadata.yaml
utils
folder is meant to hold custom utilities for your project. For example, Python scripts.extensions
folder is used for Mage extensions that integrate other data tools, like Great Expectations.io_config.yaml
file.metadata.yaml
file contains project-level metadata. There’s a metadata file in each pipeline as well.data_loaders
folder with subdirectories for api
, database
, and file
data loaders.
You can do the same when creating blocks via the UI— simply name the block subfolder_name/block_name
and the subfolder will be automatically created.
This might look like:
mage_data
directory— located at the same level as your project folder.
mage_data
holds project-level cache data and stores the result of Block runs, which are then returned to the user via the UI. Here are the locations for various components of mage_data
:
mage_data/[project_folder]/mage-ai.db
tables
schema:jdbc:sqlite:PATH
, where PATH is /home/src/mage_data/[project_folder]/mage-ai.db
for Docker installs or ~/.mage_data/[project_folder]/mage-ai.db
for pip installs.mage_data/[project_folder]/pipelines/[pipeline_uuid]/.variables/
mage_data/[project_folder]/pipelines/[pipeline_uuid]/.logs/
mage_data/[project_folder]/.cache
USER_CODE_PATH
to the absolute path where your project lives.
For example, if your original project name was demo_magic
and the directory is located at /home/src/demo_magic
,
then the value of USER_CODE_PATH
should be: /home/src/demo_magic
.
mage start [project_name]
on the command line to start running Mage,
the 1st command line argument value is used for your project’s name.