Configure data integration pipelines

Interpolation

You can interpolate pipeline variables, environment variables, secrets, and more into the source config file, schema settings, and destination config file.

Pipeline variables

{{ variables('variable_name') }}

Variable names

You can add your own custom variables to the pipeline. For more information, read the documentation. Here are the default variables available to interpolate:

Name	Description	Sample value
`pipeline.name`	Name of the current pipeline.	`ETL demo`
`pipeline.uuid`	UUID of the current pipeline.	`etl_demo`
`ds`	Execution date string.	`2023-12-25`
`hr`	Execution hour string from `00` to `23`.	`12`
`env`	Environment Mage is running in.	`dev`, `prod`
`execution_partition`	The current partition of the execution: datetime in this format `%Y%m%dT%H%M%S`.	`20231225T122520`

Common prefixes

Mage supports table prefixes for integration sources. To add a prefix, add the following to the source configuration for a data integration pipeline:

_patterns:
  destination_table: MY_PREFIX_{{ variables('stream') }}

For example, configuring a PostgreSQL source:

Environment variables

{{ env_var('environment_variable_name') }}

Secrets

Use the following syntax to interpolate secrets:

{{ mage_secret_var('your_secret_name') }}

For more information on secrets, read the documentation.

Adventure starts here

Concepts

Project

Contributing

About

Configure data integration pipelines

Interpolation

Pipeline variables

Variable names

Common prefixes

Environment variables

Secrets

​Interpolation

​Pipeline variables

​Variable names

​Common prefixes

​Environment variables

​Secrets

Interpolation

Pipeline variables

Variable names

Common prefixes

Environment variables

Secrets