Here’s a sample io_config
file. You can learn more about variable interpolation and referencing secrets via our variables overview here.
version: 0.1.1
default:
# Default profile created for data IO access.
# Add your credentials for the source you use, and delete the rest.
# AWS
AWS_ACCESS_KEY_ID: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
AWS_SECRET_ACCESS_KEY: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
AWS_SESSION_TOKEN: session_token (Used to generate Redshift credentials)
AWS_REGION: region
# Azure
AZURE_CLIENT_ID: "{{ env_var('AZURE_CLIENT_ID') }}"
AZURE_CLIENT_SECRET: "{{ env_var('AZURE_CLIENT_SECRET') }}"
AZURE_STORAGE_ACCOUNT_NAME: "{{ env_var('AZURE_STORAGE_ACCOUNT_NAME') }}"
AZURE_TENANT_ID: "{{ env_var('AZURE_TENANT_ID') }}"
# ClickHouse
CLICKHOUSE_DATABASE: default
CLICKHOUSE_HOST: host.docker.internal
CLICKHOUSE_INTERFACE: http
CLICKHOUSE_PASSWORD: null
CLICKHOUSE_PORT: 8123
CLICKHOUSE_USERNAME: null
# Druid
DRUID_HOST: hostname
DRUID_PASSWORD: password
DRUID_PATH: /druid/v2/sql/
DRUID_PORT: 8082
DRUID_SCHEME: http
DRUID_USER: user
# Google
GOOGLE_SERVICE_ACC_KEY:
type: service_account
project_id: project-id
private_key_id: key-id
private_key: "-----BEGIN PRIVATE KEY-----\nyour_private_key\n-----END_PRIVATE_KEY"
client_email: your_service_account_email
auth_uri: "https://accounts.google.com/o/oauth2/auth"
token_uri: "https://accounts.google.com/o/oauth2/token"
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/your_service_account_email"
GOOGLE_SERVICE_ACC_KEY_FILEPATH: "/path/to/your/service/account/key.json"
GOOGLE_LOCATION: US # Optional
# MongoDB
MONGODB_DATABASE: database
MONGODB_HOST: host
MONGODB_PASSWORD: password
MONGODB_PORT: 27017
MONGODB_COLLECTION: collection
MONGODB_USER: user
# MSSQL
MSSQL_DATABASE: database
MSSQL_SCHEMA: schema
MSSQL_DRIVER: "ODBC Driver 18 for SQL Server"
MSSQL_HOST: host
MSSQL_PASSWORD: password
MSSQL_PORT: 1433
MSSQL_USER: SA
# MySQL
MYSQL_DATABASE: database
MYSQL_HOST: host
MYSQL_PASSWORD: password
MYSQL_PORT: 3306
MYSQL_USER: root
# Pinot
PINOT_HOST: hostname
PINOT_PASSWORD: password
PINOT_PATH: /query/sql
PINOT_PORT: 8000
PINOT_SCHEME: http
PINOT_USER: user
# PostgreSQL
POSTGRES_CONNECT_TIMEOUT: 10
POSTGRES_DBNAME: postgres
POSTGRES_SCHEMA: public # Optional
POSTGRES_USER: username
POSTGRES_PASSWORD: password
POSTGRES_HOST: hostname
POSTGRES_PORT: 5432
# Redshift
REDSHIFT_SCHEMA: public # Optional
REDSHIFT_DBNAME: redshift_db_name
REDSHIFT_HOST: redshift_cluster_id.identifier.region.redshift.amazonaws.com
REDSHIFT_PORT: 5439
REDSHIFT_TEMP_CRED_USER: temp_username
REDSHIFT_TEMP_CRED_PASSWORD: temp_password
REDSHIFT_DBUSER: redshift_db_user
REDSHIFT_CLUSTER_ID: redshift_cluster_id
REDSHIFT_IAM_PROFILE: default
# Snowflake
SNOWFLAKE_USER: username
SNOWFLAKE_PASSWORD: password
SNOWFLAKE_ACCOUNT: account_id.region
SNOWFLAKE_DEFAULT_WH: null # Optional default warehouse
SNOWFLAKE_DEFAULT_DB: null # Optional default database
SNOWFLAKE_DEFAULT_SCHEMA: null # Optional default schema
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null # Optional private key passphrase
SNOWFLAKE_PRIVATE_KEY_PATH: null # Optional private key path
SNOWFLAKE_ROLE: null # Optional role name
SNOWFLAKE_TIMEOUT: null # Optional timeout in seconds
# Trino
trino:
catalog: postgresql # Change this to the catalog of your choice
host: 127.0.0.1
http_headers:
X-Something: 'mage=power'
http_scheme: http
password: mage1337 # Optional
port: 8080
schema: core_data
session_properties: # Optional
acc01.optimize_locality_enabled: false
optimize_hash_generation: true
source: trino-cli # Optional
user: admin
verify: /path/to/your/ca.crt # Optional
data_type_properties:
timestamp_precision: 3 # Optional. Set TIMESTAMP(n) precision
Unified config
Only in Mage Pro.
Try our fully managed solution to access this advanced feature.
Mage Pro provides a unified framework for configuring connections using io_config.yaml
across different types of pipelines:
- Batch pipeline (Python block, SQL block)
- Data integration pipeline
- Streaming pipeline
Each pipeline reads from the same io_config.yaml file, which allows you to centralize connection information and switch between pipeline types with minimal configuration changes.
Batch pipeline
Example: Python block (Postgres)
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path
@data_loader
def load_data_from_postgres(**kwargs):
query = 'SELECT 1'
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
return loader.load(query)
Example: SQL block
Just select the desired IO Config profile in the UI (under the “Profile” dropdown).
The profile must exist in your io_config.yaml and contain valid credentials for the source or destination.
Data integration pipeline
Data integration pipelines use source and destination blocks.
These blocks can reference connection credentials from io_config.yaml
via the io_config_profile
key.
The following sources and destinations systems currently support this configuration method:
- BigQuery
- Microsolft SQL Server
- MySQL
- Postgres
- Redshift
- Snowflake
Example source config (Postgres)
io_config_profile: default # profile in the io_config.yaml
Example destination config (Postgres)
io_config_profile: default # profile in the io_config.yaml
schema: test # Override the schema
With this setup, you don’t need to hard-code connection details in your block configs—just use the profile name.
Streaming pipeline
Streaming pipelines also support io_config.yaml
for setting up destinations.
Example: Destination block config (BigQuery)
connector_type: bigquery
profile: default # profile in the io_config.yaml
config:
table_id: table_id
if_exists: "append"
The profile
field points to the profile in your io_config.yaml
, and additional block-specific settings can still be defined under config.
Responses are generated using AI and may contain mistakes.