dbt sources and upstream dependencies
dbt model blocks can depend on other dbt models and other blocks: sensors, Python, SQL, R.
dbt models depending on other dbt models
When you add a dbt model block to your pipeline, if it contains any ref
s,
those dbt models will be added to the current pipeline as upstream dependencies
to the dbt model you initially added.
dbt models depending on other blocks
When a dbt model depends on a block that isn’t another dbt model, a YAML file is
created named mage_sources.yml
. This file is automatically generated and
updated whenever a new non-dbt model block is an upstream dependency of a dbt
model block.
If you have a Mage project named demo_project
with a dbt project named demo
,
here is an example of where the mage_sources.yml
file could be located:
demo_project/
| dbt/
| | demo/
| | | models/
| | | | mage_sources.yml
| | | | example/
| | | | | my_first_dbt_model.sql
| | | | | my_second_dbt_model.sql
Sources
The contents of the mage_sources.yml
could look like this:
sources:
- loader: mage
name: mage_demo
schema: public
tables:
- identifier: mage_dbt_demo_pipeline_load_data
meta:
block_uuid: load_data
pipeline_uuid: dbt_demo_pipeline
name: dbt_demo_pipeline_load_data
version: 2
The source name follows this convention: mage_[DBT_project_name]
.
The table names under the source follow this convention:
[pipeline_uuid]_[block_uuid]
.
To reference the preceding block in a dbt block use this convention:
{{ source('[source_name]', '[pipeline_uuid]_[block_uuid]') }
e.g.
{{ source('mage_demo', 'dbt_demo_pipeline_load_data') }
For more information on sources, read dbt’s documentation.
Was this page helpful?