io_config
file. You can learn more about variable interpolation and referencing secrets via our variables overview here.
Unified config
Only in Mage Pro.Try our fully managed solution to access this advanced feature.
io_config.yaml
across different types of pipelines:
- Batch pipeline (Python block, SQL block)
- Data integration pipeline
- Streaming pipeline
Batch pipeline
Example: Python block (Postgres)
Example: SQL block
Just select the desired IO Config profile in the UI (under the “Profile” dropdown). The profile must exist in your io_config.yaml and contain valid credentials for the source or destination.Data integration pipeline
Data integration pipelines use source and destination blocks. These blocks can reference connection credentials fromio_config.yaml
via the io_config_profile
key.
The following sources and destinations systems currently support this configuration method:
- BigQuery
- Microsolft SQL Server
- MySQL
- Postgres
- Redshift
- Snowflake
Example source config (Postgres)
Example destination config (Postgres)
Streaming pipeline
Streaming pipelines also supportio_config.yaml
for setting up destinations.
Example: Destination block config (BigQuery)
profile
field points to the profile in your io_config.yaml
, and additional block-specific settings can still be defined under config.