dbt sources and upstream dependencies

dbt model blocks can depend on other dbt models and other blocks: sensors, Python, SQL, R.

​

dbt models depending on other dbt models

When you add a dbt model block to your pipeline, if it contains any refs, those dbt models will be added to the current pipeline as upstream dependencies to the dbt model you initially added.

​

dbt models depending on other blocks

When a dbt model depends on a block that isn’t another dbt model, a YAML file is created named mage_sources.yml. This file is automatically generated and updated whenever a new non-dbt model block is an upstream dependency of a dbt model block.

If you have a Mage project named demo_project with a dbt project named demo, here is an example of where the mage_sources.yml file could be located:

demo_project/
|   dbt/
|   |   demo/
|   |   |   models/
|   |   |   |   mage_sources.yml
|   |   |   |   example/
|   |   |   |   |   my_first_dbt_model.sql
|   |   |   |   |   my_second_dbt_model.sql

​

Sources

The contents of the mage_sources.yml could look like this:

sources:
- loader: mage
  name: mage_demo
  schema: public
  tables:
  - identifier: mage_dbt_demo_pipeline_load_data
    meta:
      block_uuid: load_data
      pipeline_uuid: dbt_demo_pipeline
    name: dbt_demo_pipeline_load_data
version: 2

The source name follows this convention: mage_[DBT_project_name].

The table names under the source follow this convention: [pipeline_uuid]_[block_uuid].

To reference the preceding block in a dbt block use this convention: {{ source('[source_name]', '[pipeline_uuid]_[block_uuid]') } e.g. {{ source('mage_demo', 'dbt_demo_pipeline_load_data') }

For more information on sources, read dbt’s documentation.