After completing data transformations, utilize the data exporter blocks to either load the processed data or store a machine learning model in an external data storage system.
io_config.yml
file to ensure seamless operation.
The io_config.yml
file typically includes connection details such as host, port, database name, username, and password.
Meanwhile, the exporter block needs to be configured with the appropriate export parameters, such as target table names,
schema details, and conflict resolution strategies.
config_profile
parameter set to 'default'
by default. This parameter can be customized
to use different configuration profiles if you have multiple profiles or have renamed them.io_config.yml
:
host
: The server address of the storage system.port
: The port number for the connection.database
: The name of the target database or data storage container.username
and password
: Authentication credentials.Data Warehouse
io_config.yml
file to connect your Mage pipeline to a snowflake data warehouse. While optional,
depending on how your Snowflake DW is configured you may need to enter all information into the .yml file.
It’s recommended to store sensitive information as Secrets. See the general Secrets documentation
for more information.Data Lake
io_config.yml
file to connect your Mage pipeline to a Azure Blob Storage. Configure some Secrets and enter them into io_config.yml
file.
If you need more information on entering secrets see this documentation.io_config.yml
file.Example Code:Database
io_config.yml
file to connect your Mage pipeline to a PostgreSQL database. Configure some Secrets and enter them into io_config.yml
file.
If you need more information on entering secrets see this documentation.POSTGRES_HOST:
io_config.yml
file.Example Code:Delta Lake
io_config.yml
file.
They contain the necessary configurations within the exporter block itself. Lets break that down.Storage Options'AWS_ACCESS_KEY_ID'
: Your AWS access key ID.'AWS_SECRET_ACCESS_KEY'
: Your AWS secret access key.'AWS_REGION'
: The AWS region where your S3 bucket is located.'AWS_S3_ALLOW_UNSAFE_RENAME'
: This option allows unsafe rename operations on S3, which might be necessary for some workflows.uri
: The S3 URI where the Delta Table is stored.