io_config
file. You can learn more about variable interpolation and referencing secrets via our variables overview here.
ℹ️ To learn how to set up yourio_config.yaml
file, check the setup guide. Theio_config.yaml
file should be placed under your project folder.
Unified config
Only in Mage Pro.Try our fully managed solution to access this advanced feature.
io_config.yaml
across different types of pipelines:
- Batch pipeline (Python block, SQL block)
- Data integration pipeline
- Streaming pipeline
Batch pipeline
Example: Python block (Postgres)
Example: SQL block
Just select the desired IO Config profile in the UI (under the “Profile” dropdown). The profile must exist in your io_config.yaml and contain valid credentials for the source or destination.Data integration pipeline
Data integration pipelines use source and destination blocks. These blocks can reference connection credentials fromio_config.yaml
via the io_config_profile
key.
The following sources and destinations systems currently support this configuration method:
- BigQuery
- Microsolft SQL Server
- MySQL
- Postgres
- Redshift
- Snowflake
Example source config (Postgres)
Example destination config (Postgres)
Streaming pipeline
Streaming pipelines also supportio_config.yaml
for setting up destinations.
Example: Destination block config (BigQuery)
profile
field points to the profile in your io_config.yaml
, and additional block-specific settings can still be defined under config.