We love our UI, but here’s how to develop in your favorite IDE, too.
pipelines/
under the Mage
project directory.
For example, if your project is named demo_project
and your pipeline is named etl_demo
then
you’ll have a folder structure that looks like this:
demo_project/pipelines/
directory. Name this new folder after the name
of your pipeline.
Add 2 files in this new folder:
__init__.py
metadata.yaml
metadata.yaml
file, add the following content:
etl_demo
to whatever name you’re using for your new pipeline.metadata.yaml
will produce the following block dependencies:
metadata.yaml
sectionsdatabricks
integration
pyspark
python
(most common)streaming
ecs
gcp_cloud_run
azure_container_instance
k8s
local_python
(most common)pyspark
block_run_limit
: Maximum number of blocks that can run in parallel.pipeline_run_limit
pipeline_run_limit_all_triggers
on_pipeline_run_limit_reached
ecs
gcp_cloud_run
azure_container_instance
k8s
local_python
(most common)pyspark
python
(most common)r
sql
yaml
chart
custom
(most common)data_exporter
data_loader
dbt
scratchpad
sensor
transformer
type
is data_loader
, then the file must be in the [project_name]/data_loaders/
folder. It can be
nested in any number of subfolders.load_data
and the language
is python
, then the file name
will be load_data.py
.