Data exporter
After completing data transformations, utilize the data exporter blocks to either load the processed data or store a machine learning model in an external data storage system.
Mage natively supports integration with a variety of data storage systems. However, these integrations often
require specific configurations in both the exporter block and the io_config.yml
file to ensure seamless operation.
The io_config.yml
file typically includes connection details such as host, port, database name, username, and password.
Meanwhile, the exporter block needs to be configured with the appropriate export parameters, such as target table names,
schema details, and conflict resolution strategies.
Technical Details
- Data Exporter Blocks:
- These blocks are designed to facilitate the movement of transformed data or trained models to external systems.
- Configuration parameters might include destination paths, file formats, table names, schemas, and update strategies.
- Most data exporters include a
config_profile
parameter set to'default'
by default. This parameter can be customized to use different configuration profiles if you have multiple profiles or have renamed them.
- Supported Data Storage Systems:
- Mage supports a wide range of storage systems including PostgreSQL, MySQL, AWS S3, Google Cloud Storage, Azure Blob Storage, and many more.
- Each system may have unique requirements and configurations to ensure compatibility and optimal performance.
- Configuration in
io_config.yml
:- This file serves as the central configuration hub for defining connection parameters.
- Typical parameters include:
host
: The server address of the storage system.port
: The port number for the connection.database
: The name of the target database or data storage container.username
andpassword
: Authentication credentials.- Additional parameters as required by specific storage systems (e.g., SSL settings, API tokens).
Examples
By correctly configuring these components, you can effectively streamline the data loading process into your chosen storage system, whether it be a relational database, a data lake, or a machine learning model repository.
Was this page helpful?