IO Config
The io_config
YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.
Here’s a sample io_config
file. You can learn more about variable interpolation and referencing secrets via our variables overview here.
Unified config
Try our fully managed solution to access this advanced feature.
Mage Pro provides a unified framework for configuring connections using io_config.yaml
across different types of pipelines:
- Batch pipeline (Python block, SQL block)
- Data integration pipeline
- Streaming pipeline
Each pipeline reads from the same io_config.yaml file, which allows you to centralize connection information and switch between pipeline types with minimal configuration changes.
Batch pipeline
Example: Python block (Postgres)
Example: SQL block
Just select the desired IO Config profile in the UI (under the “Profile” dropdown). The profile must exist in your io_config.yaml and contain valid credentials for the source or destination.
Data integration pipeline
Data integration pipelines use source and destination blocks.
These blocks can reference connection credentials from io_config.yaml
via the io_config_profile
key.
Example source config (Postgres)
Example destination config (Postgres)
With this setup, you don’t need to hard-code connection details in your block configs—just use the profile name.
Streaming pipeline
Streaming pipelines also support io_config.yaml
for setting up destinations.
Example: Destination block config (BigQuery)
The profile
field points to the profile in your io_config.yaml
, and additional block-specific settings can still be defined under config.
Was this page helpful?