• AWS S3
• Azure Blob Storage
• Azure Data Lake Storage (Pro only)
• Filesystem
• Google Cloud Storage

• AWS Redshift
• ClickHouse
• Databricks SQL (Pro only)
• DuckDB
• MySQL
• MSSQL
• MongoDB
• OracleDB
• PostgreSQL
• SQLite
• Snowflake
• Spark
• Trino

• Google BigQuery
• Microsoft Fabric Warehouse (Pro only)

• Google Sheets

• File-Loading Clients - import/export data between files and your pipeline. Includes both local filesystem storage and external filesystem storage (like AWS S3)
• Database Clients - imports data from an external database into your pipeline and exports data frames back into that database. These database clients include the execute method which execute your queries in the connected database. The Google BigQuery client follows this structure.
  • A subcategory of database clients are connection-based clients, which wrap a connection to the database. This connection is used to execute transactions (sets of queries) on the database, which are either committed (saved to database) or rolled-back (deleted). Clients for PostgreSQL, Redshift, and Snowflake follow this structure.

​

Example: Loading data from a file

loader = FileIO(verbose=True)
df = loader.load(
    'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
)

loader.export(df, './titanic_survival.json', orient='records')

FileIO initialized
├─ Loading data frame from 'https://raw.githubusercontent.com/datasciencedojodatasets/master/titanic.csv'...DONE
└─ Exporting data frame to './titanic_survival.json'...DONE

​

Example: Loading data from Snowflake warehouse

• Account Username
• Account Password
• Account ID (including your region) (Ex: example.us-west-2)

default:
    SNOWFLAKE_USER: my_username
    SNOWFLAKE_PASSWORD: my_password
    SNOWFLAKE_ACCOUNT: example.us-west-2

config = ConfigFileLoader('io_config.yaml', 'default')
loader = Snowflake.with_config(config, verbose=True)

• execute() - executes an arbitrary query on your data warehouse. In this case, the warehouse, database, and schema to use are selected.
• load() - loads the results of a SELECT query into a Pandas DataFrame.
• export() - stores data frame as a table in your data warehouse. If the table exists, then the data is appended by default (and can be configured with other behavior, see Snowflake API). If the table doesn’t exist, then the table is created with the given schema name and table name (the table data types are inferred from the Python data type).

with loader:
    loader.execute('USE WAREHOUSE my_warehouse;')
    loader.execute('USE DATABASE my_database;')
    loader.execute('USE SCHEMA my_schema;')
    df = loader.load('SELECT * FROM test_table;')
    loader.export(df, 'my_schema', 'test_table')

​

Client APIs

​

FileIO

• CSV
• JSON
• Parquet
• HDF5
• XML
• Excel

​

Constructor

• Args
  • verbose (bool): Enables verbose output printing. Defaults to False.

​

Methods

• Args:
  • df (DataFrame): Data frame to export.
  • filepath (os.PathLike): Filepath to export data frame to.
  • format (FileFormat | str, Optional): Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Writer
    CSV	DataFrame.to_csv
    JSON	DataFrame.to_json
    Parquet	DataFrame.to_parquet
    HDF5	DataFrame.to_hdf
    XML	DataFrame.to_xml
    Excel	DataFrame.to_excel

• Args:
  • filepath (os.PathLike): Filepath to load data frame from.
  • format (FileFormat | str, Optional): Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  • limit (int, optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Reader
    CSV	read_csv
    JSON	read_json
    Parquet	read_parquet
    HDF5	read_hdf
    XML	read_xml
    Excel	read_excel

• Returns: (DataFrame) Data frame object loaded from data in the specified file.

​

S3

• CSV
• JSON
• Parquet
• HDF5
• XML
• Excel

AWS_ACCESS_KEY_ID: AWS Access Key ID credential
AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
AWS_REGION: AWS Region

​

Constructor

• Args
  • verbose (bool): Enables verbose output printing. Defaults to False.
  • If IAM profile is not set up using aws configure and Mage’s configuration loader is not used, then specify your AWS credentials through the following keyword arguments:
    • aws_access_key_id (str): AWS Access Key ID credential
    • aws_secret_access_key (str): AWS Secret Access Key credential
    • aws_region (str): Region associated with AWS IAM profile

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (S3) The constructed dataloader using this method

​

Methods

• Args:
  • data (Union[DataFrame, str]): Data frame or file path to export.
  • bucket_name (str): Name of the bucket to export data frame to.
  • object_key (str): Object key in S3 bucket to export data frame to.
  • format (FileFormat | str, Optional): Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Writer
    CSV	DataFrame.to_csv
    JSON	DataFrame.to_json
    Parquet	DataFrame.to_parquet
    HDF5	DataFrame.to_hdf
    XML	DataFrame.to_xml
    Excel	DataFrame.to_excel

• Args:
  • bucket_name (str): Name of the bucket to load data from.
  • object_key (str): Key of the object in S3 bucket to load data from.
  • format (FileFormat | str, Optional): Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  • limit (int, optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Reader
    CSV	read_csv
    JSON	read_json
    Parquet	read_parquet
    HDF5	read_hdf
    XML	read_xml
    Excel	read_excel

• Returns: (DataFrame) Data frame object loaded from data in the specified file.

​

GoogleCloudStorage

• CSV
• JSON
• Parquet
• HDF5
• XML
• Excel

• Define the GOOGLE_APPLICATION_CREDENTIALS environment variable holding the filepath to your service account key
• Define the GOOGLE_SERVICE_ACC_KEY_FILEPATH key with your configuration loader or the path_to_credentials keyword argument with the client constructor holding the filepath to your service account key
• Define the GOOGLE_SERVICE_ACC_KEY key with your configuration loader or the credentials_mapping keyword argument with the client constructor holding a mapping sharing the same contents as your service key
  • if using a configuration file, be careful to wrap your service key values in quotes so the YAML parser reads the settings correctly
• Manually pass the google.oauth2.service_account.Credentials object with the keyword argument credentials

​

Constructor

• Args
  • verbose (bool): Enables verbose output printing. Defaults to False.
  • credentials_mapping (Mapping[str, str]) - Mapping object corresponding to your service account key. See instructions above on when to use this keyword argument
  • path_to_credentials (str) - Filepath to service account key. See instructions above on when to use this keyword argument.

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (GoogleCloudStorage) The constructed dataloader using this method

​

Methods

• Args:
  • data (Union[DataFrame, str]): Data frame or file path to export.
  • bucket_name (str): Name of the bucket to export data frame to.
  • object_key (str): Object key in GoogleCloudStorage bucket to export data frame to.
  • format (FileFormat | str, Optional): Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Writer
    CSV	DataFrame.to_csv
    JSON	DataFrame.to_json
    Parquet	DataFrame.to_parquet
    HDF5	DataFrame.to_hdf
    XML	DataFrame.to_xml
    Excel	DataFrame.to_excel

• Args:
  • bucket_name (str): Name of the bucket to load data from.
  • object_key (str): Key of the object in GoogleCloudStorage bucket to load data from.
  • format (FileFormat | str, Optional): Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  • limit (int, optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Reader
    CSV	read_csv
    JSON	read_json
    Parquet	read_parquet
    HDF5	read_hdf
    XML	read_xml
    Excel	read_excel

• Returns: (DataFrame) Data frame object loaded from data in the specified file.

​

AzureBlobStorage

• CSV
• JSON
• Parquet
• HDF5
• XML
• Excel

• AZURE_CLIENT_ID
• AZURE_CLIENT_SECRET
• AZURE_TENANT_ID
• AZURE_STORAGE_ACCOUNT_NAME

​

Constructor

• Args
  • verbose (bool): Enables verbose output printing. Defaults to False.

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (AzureBlobStorage) The constructed dataloader using this method

​

Methods

• Args:
  • df (DataFrame): Data frame to export.
  • container_name (str): Name of the Azure container to export data to.
  • blob_path (str): The desired output path of the data in your Azure Blob
  • format (FileFormat | str, Optional): Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats at:

    Format	Pandas Writer
    CSV	DataFrame.to_csv
    JSON	DataFrame.to_json
    Parquet	DataFrame.to_parquet
    HDF5	DataFrame.to_hdf
    XML	DataFrame.to_xml
    Excel	DataFrame.to_excel

​

GoogleSheets

​

BigQuery

• Define the GOOGLE_APPLICATION_CREDENTIALS environment variable holding the filepath to your service account key
• Define the GOOGLE_SERVICE_ACC_KEY_FILEPATH key with your configuration loader or the path_to_credentials keyword argument with the client constructor holding the filepath to your service account key
• Define the GOOGLE_SERVICE_ACC_KEY key with your configuration loader or the credentials_mapping keyword argument with the client constructor holding a mapping sharing the same contents as your service key
  • if using a configuration file, be careful to wrap your service key values in quotes so the YAML parser reads the settings correctly
• Manually pass the google.oauth2.service_account.Credentials object with the keyword argument credentials

​

Constructor

• Args
  • verbose (bool): Enables verbose output printing. Defaults to False.
  • credentials_mapping (Mapping[str, str]) - Mapping object corresponding to your service account key. See instructions above on when to use this keyword argument
  • path_to_credentials (str) - Filepath to service account key. See instructions above on when to use this keyword argument.
  All other keyword arguments can be found in the Google BigQuery Python Client docs

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (BigQuery) BigQuery data loading client constructed using this method

• Args:
  • path_to_credentials (str): Path to the credentials file.
  • **kwargs: Additional parameters to pass to BigQuery client constructor.
• Returns: (BigQuery) BigQuery data loading client constructed using this method

• Args:
  • credentials (Mapping[str, str]): Mapping containing same key-value pairs as a service account key.
  • **kwargs: Additional parameters to pass to BigQuery client constructor.
• Returns: (BigQuery) BigQuery data loading client constructed using this method

​

Methods

• Args:
  • query_string (str): Query to execute on the BigQuery warehouse.
  • **kwargs: Additional arguments to pass to query, such as query configurations. See Client.query() docs for additional arguments.

• Args:
  • df (DataFrame): Data frame to export
  • table_id (str): ID of the table to export the data frame to. If of the format "your-project.your_dataset.your_table_name". If this table exists, the table schema must match the data frame schema. If this table doesn’t exist, the table schema is automatically inferred.
  • if_exists (str): Specifies export policy if table exists. Either - 'fail': throw an error. - 'replace': drops existing table and creates new table of same name. - 'append': appends data frame to existing table. In this case the schema must match the original table. Defaults to 'replace'. If write_disposition is specified as a keyword argument, this parameter is ignored (as both define the same functionality).
  • **configuration_params: Configuration parameters for export job. See valid configuration parameters at LoadJobConfig docs.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional parameters to pass to the query. See Google BigQuery Python client docs for additional arguments.

• Args:
  • schema (str): The schema to select the table from.
  • size (int): The number of rows to sample. Defaults to 10,000,000.
  • table (str): The table to sample from in the connected database.
• Returns: (DataFrame) Sampled data from the table.

​

PostgreSQL

POSTGRES_DBNAME: PostgreSQL database name
POSTGRES_USER: PostgreSQL database login username
POSTGRES_PASSWORD: PostgreSQL database login password
POSTGRES_HOST: PostgreSQL database hostname
POSTGRES_PORT: PostgreSQL database port

​

Constructor

• Args:
  • dbname (str): The name of the database to connect to.
  • user (str): The user with which to connect to the database with.
  • password (str): The login password for the user.
  • host (str): Host address for database.
  • port (str): Port on which the database is running.
  • verbose (bool): Enables verbose output printing. Defaults to False.
  • **kwargs: Additional settings for creating psycopg2 connection

​

Attributes

• conn (psycopg2.connection.Connection) - underlying psycopg2 Connection object

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (Postgres) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the PostgreSQL database.
  • **kwargs: Additional parameters to pass to the query. See psycopg2 docs for configuring query parameters.

• Args:
  • df (DataFrame): Data frame to export to the PostgreSQL database.
  • table_name (str): Name of the table to export data to (excluding database and schema).
  • database (str): Name of the database in which the table is located.
  • schema (str): Name of the schema in which the table is located.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table.
    Defaults to 'replace'.
  • index (bool): If True, the data frame index is also exported alongside the table. Defaults to False.
  • **kwargs: Additional arguments to pass to writer. See Advanced Export Parameters for details on UPSERT (unique_constraints, unique_conflict_method) and other advanced options like overwrite_types, query_string, auto_clean_name, etc.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional parameters to pass to the query. See psycopg2 docs for configuring query parameters.
• Returns: (DataFrame) Data frame containing the queried data.

• Args:
  • schema (str): The schema to select the table from.
  • size (int): The number of rows to sample. Defaults to 10,000,000.
  • table (str): The table to sample from in the connected database.
• Returns: (DataFrame) Sampled data from the table.

​

Advanced Export Parameters for SQL Databases

Note: UPSERT (UPDATE or INSERT) functionality is only supported on a subset of SQL databases (e.g., PostgreSQL, MySQL, MSSQL, Snowflake, BigQuery, SQLite). It is not supported on Redshift, ClickHouse, Trino or Databricks SQL. See below for details.

​

UPSERT (UPDATE or INSERT)

• unique_constraints (List[str]): A list of column names that form the unique constraint. These columns are used to identify existing rows for updates.
• unique_conflict_method (str): Specifies how to handle conflicts when unique constraints are violated. Can be either:
  • 'UPDATE' (or use constant UNIQUE_CONFLICT_METHOD_UPDATE): Updates existing rows that match the unique constraints with new values from the data frame, and inserts new rows that don’t match.
  • 'IGNORE' (or use constant UNIQUE_CONFLICT_METHOD_IGNORE): Ignores rows that would violate unique constraints (does not update or insert).

from mage_ai.io.postgres import Postgres
from mage_ai.io.constants import UNIQUE_CONFLICT_METHOD_UPDATE

loader = Postgres(...)
with loader:
    loader.export(
        df,
        table_name='users',
        schema='public',
        if_exists='append',
        unique_constraints=['user_id', 'email'],  # Columns that form unique constraint
        unique_conflict_method=UNIQUE_CONFLICT_METHOD_UPDATE,  # Update existing, insert new
    )
    loader.commit()

• If a row with the same user_id and email already exists, it will be updated with the new values from the data frame.
• If no matching row exists, a new row will be inserted.

from mage_ai.io.postgres import Postgres
from mage_ai.io.constants import UNIQUE_CONFLICT_METHOD_IGNORE

loader = Postgres(...)
with loader:
    loader.export(
        df,
        table_name='users',
        schema='public',
        if_exists='append',
        unique_constraints=['user_id'],
        unique_conflict_method=UNIQUE_CONFLICT_METHOD_IGNORE,  # Skip rows that already exist
    )
    loader.commit()

​

Additional Export Parameters

• allow_reserved_words (bool): If True, allows column names that are SQL reserved words. Defaults to False.
• auto_clean_name (bool): If True, automatically cleans column and table names (removes special characters, handles reserved words). Defaults to True.
• case_sensitive (bool): If True, preserves case sensitivity for identifiers. Defaults to False.
• cascade_on_drop (bool): If True, adds CASCADE when dropping tables. Defaults to False.
• drop_table_on_replace (bool): If True, drops the table when using if_exists='replace' instead of deleting all rows. Defaults to False.
• overwrite_types (Dict[str, str]): A dictionary mapping column names to SQL data types to override automatic type inference. Example: {'price': 'DECIMAL(10,2)', 'description': 'TEXT'}.
• query_string (str): A custom SQL query string to use for creating the table. If provided, the table is created using CREATE TABLE AS SELECT ... syntax.
• skip_check_table_exists (bool): If True, skips checking if the table exists before export. Defaults to False.
• skip_create_schema (bool): If True, skips creating the schema if it doesn’t exist. Defaults to False.
• skip_semicolon_at_end (bool): If True, omits the semicolon at the end of SQL statements. Defaults to False.

loader.export(
    df,
    table_name='products',
    schema='public',
    overwrite_types={
        'price': 'DECIMAL(10,2)',
        'description': 'TEXT',
        'created_at': 'TIMESTAMP',
    }
)

loader.export(
    df,
    table_name='filtered_data',
    schema='public',
    query_string='SELECT * FROM source_table WHERE status = \'active\'',
)

​

OracleDB

ORACLEDB_USER: OracleDB username
ORACLEDB_PASSWORD: OracleDB password
ORACLEDB_HOST: OracleDB host
ORACLEDB_PORT: OracleDB port
ORACLEDB_SERVICE: OracleDB service
ORACLEDB_MODE: OracleDB mode

​

Constructor

• Args:
  • user (str): The user for connecting to the database with.
  • password (str): The login password for the user.
  • host (str): Host address for database.
  • port (int): Port on which the database is running Defaults to 3306.
  • service (str): OracleDB service
  • mode (str): switch between OracleDB mode of thin or thick
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating psycopg2 connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (OracleDB) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the OracleDB database.
  • **kwargs: Additional parameters to pass to the query.

• Args:
  • df (DataFrame): Data frame to export to the OracleDB database.
  • schema_name (str): Not required for OracleDB loaders. Set to None.
  • table_name (str): Name of the table to export data to (excluding database).
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table.
    Defaults to 'replace'.
  • index (bool): If True, the data frame index is also exported alongside the table. Defaults to False.
  • **kwargs: Additional arguments to pass to writer.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • verbose (bool): Enables verbose output printing. Defaults to True.
• Returns: (DataFrame) Data frame containing the queried data.

​

MySQL

MYSQL_DATABASE: MySQL database name
MYSQL_HOST: MySQL database hostname
MYSQL_PASSWORD: MySQL database login password
MYSQL_PORT: MySQL database port
MYSQL_USER: MySQL database login username
MYSQL_ALLOW_LOCAL_INFILE: Pass boolean value to `allow_local_infile` param during connection to enable the capability to load local data file.

​

Constructor

• Args:
  • database (str): The name of the database to connect to.
  • user (str): The user for connecting to the database with.
  • password (str): The login password for the user.
  • host (str): Host address for database.
  • port (int): Port on which the database is running Defaults to 3306.
  • allow_local_infile (bool): Enables verbose capability to load local file. Defaults to False.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating psycopg2 connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (MySQL) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the MySQL database.
  • **kwargs: Additional parameters to pass to the query.

• Args:
  • df (DataFrame): Data frame to export to the MySQL database.
  • schema_name (str): Not required for MySQL loaders. Set to None.
  • table_name (str): Name of the table to export data to (excluding database).
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table.
    Defaults to 'replace'.
  • index (bool): If True, the data frame index is also exported alongside the table. Defaults to False.
  • **kwargs: Additional arguments to pass to writer. See Advanced Export Parameters for details on UPSERT (unique_constraints, unique_conflict_method) and other advanced options.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
• Returns: (DataFrame) Data frame containing the queried data.

• Args:
  • database (str): The database to select the table from.
  • table (str): The table to sample from in the connected database.
  • size (int): The number of rows to sample. Defaults to 10,000,000.
• Returns: (DataFrame) Sampled data from the table.

​

DuckDB

DUCKDB_DATABASE: database # Database name
DUCKDB_SCHEMA: main # Schema name. Default is main.

​

Constructor

__init__(self,
         database: str
         schema: str = None,
         verbose: bool = True,
         **kwargs
        )

• Args:
  • database (str): The name of the database to connect to.
  • schema: Schema to use.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating psycopg2 connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (DuckDB) The constructed dataloader using this method

​

Methods

• Args:
  • schema_name: Name of the schema to use.
  • table_name: Name of the table.
• Returns: true or false.

• Args:
  • df: Data in dataframe type.

• Args:
  • column: Metadata associated with the column.
  • dtype: Input data type.
• Returns: DuckDB data type in string format.

​

Snowflake

SNOWFLAKE_USER: username
SNOWFLAKE_PASSWORD: password
SNOWFLAKE_ACCOUNT: account_id.region
SNOWFLAKE_DEFAULT_WH: null                  # Optional default warehouse
SNOWFLAKE_DEFAULT_DB: null                  # Optional default database
SNOWFLAKE_DEFAULT_SCHEMA: null              # Optional default schema
SNOWFLAKE_PRIVATE_KEY_PASSPHRASE: null      # Optional private key passphrase
SNOWFLAKE_PRIVATE_KEY_PATH: null            # Optional private key path
SNOWFLAKE_ROLE: null                        # Optional role name
SNOWFLAKE_TIMEOUT: null                     # Optional timeout in seconds

​

Constructor

• user (str): Username for the Snowflake user.
• password (str): Login Password for the user.
• account (str): Snowflake account identifier (including region, excluding snowflake-computing.com suffix).

• verbose (bool): Specify whether to print verbose output.
• database (str): Name of the default database to use. If unspecified no database is selected on login.
• schema (str): Name of the default schema to use. If unspecified no schema is selected on login.
• warehouse (str): Name of the default warehouse to use. If unspecified no warehouse is selected on login.

​

Attributes

• conn (snowflake.connector.Connection) - underlying Snowflake Connection object

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • verbose (bool): Enables verbose output printing. Defaults to False.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (Snowflake) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the Snowflake warehouse.
  • **kwargs: Additional parameters to pass to the query. See Snowflake Connector Docs for additional parameters.

• Args:
  • df (DataFrame): Data frame to export to a Snowflake warehouse.
  • table_name (str): Name of the table to export data to (excluding database and schema).
  • database (str): Name of the database in which the table is located.
  • schema (str): Name of the schema in which the table is located.
  • if_exists (str, optional): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table.
    Defaults to 'append'.
  • **kwargs: Additional arguments to pass to writer. See Advanced Export Parameters for details on UPSERT (unique_constraints, unique_conflict_method) and other advanced options.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • *args, **kwargs: Additional parameters to pass to the query. See Snowflake Connector Docs for additional parameters.
• Returns: (DataFrame) Data frame containing the queried data

• Args:
  • schema (str): The schema to select the table from.
  • size (int): The number of rows to sample. Defaults to 10,000,000.
  • table (str): The table to sample from in the connected database.
• Returns: (DataFrame) Sampled data from the data frame.

​

Redshift

1. Pre-generate temporary credentials and specify them in the configuration settings. Add the following keys to the configuration settings for Redshift to use the temporary credentials:
   Copy

   REDSHIFT_DBNAME: Name of Redshift database to connect to
   REDSHIFT_HOST: Redshift Cluster hostname
   REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
   REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
   REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
   
2. Provide an IAM Profile to automatically generate temporary credentials for connection. The IAM profile is read from ~/.aws/ and is used with the GetClusterCredentials endpoint to generate temporary credentials. Add the following keys to the configuration settings for Redshift to generate temporary credentials
   Copy

   REDSHIFT_DBNAME: Name of Redshift database to connect to
   REDSHIFT_DBUSER: Redshift database user to generate credentials for
   REDSHIFT_CLUSTER_ID: Redshift cluster ID
   REDSHIFT_IAM_PROFILE: Name of the IAM profile to generate temp credentials with
   
   If an IAM profile is not setup using aws configure, manually specify the AWS credentials in the configuration settings as well.
   Copy

   AWS_ACCESS_KEY_ID: AWS Access Key ID credential
   AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
   AWS_SESSION_TOKEN: AWS Session Token (used to generate temp DB credentials)
   AWS_REGION: AWS Region
   

​

Constructor

• Args
  • verbose: Enables verbose output. Defaults to False.
  See other keyword arguments in Redshift’s Python connector docs.

​

Attributes

• conn (redshift_connector.Connection) - the underlying Redshift Connection object.

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor.
• Returns: (Redshift) Redshift data loading client constructed using this method

• Args:
  • database (str): Name of the database to connect to.
  • host (str): The hostname of the Redshift cluster which the database belongs to.
  • user (str): Temporary credentials username for use in authentication.
  • password (str): Temporary credentials password for use in authentication.
  • port (int, optional): Port number of the Redshift cluster. Defaults to 5439.
  • **kwargs: Additional parameters passed to the loader constructor.
• Returns: (Redshift) Redshift data loading client constructed using this method

• Args:
  • cluster_identifier (str): Identifier of the cluster to connect to.
  • database (str): The database to connect to within the specified cluster.
  • db_user (str): Database username
  • profile (str, optional): The profile to use from stored credentials file. Defaults to ‘default’.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (Redshift) Redshift data loading client constructed using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the Redshift cluster.
  • **kwargs: Additional parameters to pass to the query. See redshift-connector docs for configuring query parameters.

• Args:
  • df (DataFrame): Data frame to export to database in Redshift cluster.
  • table_name (str): Name of the table to export the data to. Table must already exist.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • *args, **kwargs: Additional parameters to send to query, including parameters for use with format strings. See redshift-connector docs for configuring query parameters.
• Returns: (DataFrame) Data frame containing the queried data.

• Args:
  • schema (str): The schema to select the table from.
  • size (int): The number of rows to sample. Defaults to 10,000,000.
  • table (str): The table to sample from in the connected database.
• Returns: (DataFrame) Sampled data from the table.

​

MSSQL

MSSQL_DATABASE: MSSQL database name
MSSQL_USER: MSSQL database login username
MSSQL_PASSWORD: MSSQL database login password
MSSQL_HOST: MSSQL database hostname
MSSQL_PORT: MSSQL database port (defaults to 1433)
MSSQL_SCHEMA: MSSQL database schema (defaults to 'dbo')
MSSQL_AUTHENTICATION: Authentication method (optional)
MSSQL_CONNECTION_METHOD: Connection method ('direct' or 'ssh_tunnel')
MSSQL_SSH_HOST: SSH tunnel host (if using SSH tunnel)
MSSQL_SSH_PORT: SSH tunnel port
MSSQL_SSH_USERNAME: SSH tunnel username
MSSQL_SSH_PASSWORD: SSH tunnel password
MSSQL_SSH_PKEY: SSH tunnel private key path

​

Constructor

• Args:
  • database (str): The name of the database to connect to.
  • user (str): The user for connecting to the database with.
  • password (str): The login password for the user.
  • host (str): Host address for database.
  • port (int): Port on which the database is running. Defaults to 1433.
  • schema (str): Schema name. Defaults to ‘dbo’.
  • authentication (str, optional): Authentication method.
  • connection_method (str): Connection method (‘direct’ or ‘ssh_tunnel’). Defaults to ‘direct’.
  • ssh_host (str, optional): SSH tunnel host.
  • ssh_port (int, optional): SSH tunnel port.
  • ssh_username (str, optional): SSH tunnel username.
  • ssh_password (str, optional): SSH tunnel password.
  • ssh_pkey (str, optional): SSH tunnel private key path.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (MSSQL) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the MSSQL database.
  • **kwargs: Additional parameters to pass to the query.

• Args:
  • df (DataFrame): Data frame to export to the MSSQL database.
  • schema_name (str): Name of the schema in which the table is located. Defaults to ‘dbo’.
  • table_name (str): Name of the table to export data to.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table. Defaults to 'replace'.
  • index (bool): If True, the data frame index is also exported alongside the table. Defaults to False.
  • **kwargs: Additional arguments to pass to the writer. See the Advanced Export Parameters section for details on using options such as unique_constraints, unique_conflict_method, and other advanced export options.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional parameters to pass to the query.
• Returns: (DataFrame) Data frame containing the queried data.

​

ClickHouse

CLICKHOUSE_HOST: ClickHouse hostname
CLICKHOUSE_PORT: ClickHouse port
CLICKHOUSE_USERNAME: ClickHouse username
CLICKHOUSE_PASSWORD: ClickHouse password
CLICKHOUSE_DATABASE: ClickHouse database name
CLICKHOUSE_INTERFACE: ClickHouse interface (optional)
CLICKHOUSE_SSL_CA_CERT: Path to CA certificate for SSL (optional)

​

Constructor

• Args:
  • host (str): ClickHouse hostname.
  • port (int): ClickHouse port.
  • username (str): ClickHouse username.
  • password (str): ClickHouse password.
  • database (str): Database name. Defaults to ‘default’.
  • interface (str, optional): ClickHouse interface.
  • secure (bool, optional): Enable SSL.
  • ca_cert (str, optional): Path to CA certificate.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings passed to ClickHouse client

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (ClickHouse) The constructed dataloader using this method

​

Methods

• Args:
  • command_string (str): Command to execute on the ClickHouse warehouse.
  • **kwargs: Additional arguments to pass to command.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded dataframe to. Defaults to 10,000,000.
  • **kwargs: Additional arguments to pass to query.
• Returns: (DataFrame) Data frame associated with the given query.

• Args:
  • df (DataFrame): Data frame to export to ClickHouse warehouse.
  • table_name (str): Name of the table to export data to.
  • database (str): Name of the database in which the table is located.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table. Defaults to 'replace'.
  • **kwargs: Advanced export parameters. See Advanced Export Parameters. (UPSERT not supported.)

​

SQLite

​

Constructor

• Args:
  • database (str): Path to the SQLite database file.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (SQLite) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the SQLite database.
  • **kwargs: Additional parameters to pass to the query.

• Args:
  • df (DataFrame): Data frame to export to the SQLite database.
  • table_name (str): Name of the table to export data to.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table. Defaults to 'replace'.
  • index (bool): If True, the data frame index is also exported alongside the table. Defaults to False.
  • **kwargs: Advanced export options such as unique_constraints, unique_conflict_method, overwrite_types, auto_clean_name, etc. See the Advanced Export Parameters section for details.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional parameters to pass to the query.
• Returns: (DataFrame) Data frame containing the queried data.

​

Trino

TRINO_CATALOG: Trino catalog name
TRINO_HOST: Trino hostname
TRINO_PORT: Trino port (defaults to 8080)
TRINO_USER: Trino username
TRINO_PASSWORD: Trino password
TRINO_SCHEMA: Trino schema name

​

Constructor

• Args:
  • catalog (str): Trino catalog name.
  • host (str): Trino hostname.
  • user (str): Trino username.
  • password (str, optional): Trino password.
  • port (int): Trino port. Defaults to 8080.
  • schema (str, optional): Trino schema name.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (Trino) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the Trino warehouse.
  • **kwargs: Additional parameters to pass to the query.

• Args:
  • df (DataFrame): Data frame to export to Trino warehouse.
  • table_name (str): Name of the table to export data to.
  • schema (str): Name of the schema in which the table is located.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table. Defaults to 'replace'.
  • **kwargs: Advanced export parameters. See Advanced Export Parameters. (UPSERT not supported.)

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional parameters to pass to the query.
• Returns: (DataFrame) Data frame containing the queried data.

​

MongoDB

MONGODB_CONNECTION_STRING: MongoDB connection string (alternative to individual settings)
MONGODB_HOST: MongoDB hostname
MONGODB_PORT: MongoDB port (defaults to 27017)
MONGODB_USER: MongoDB username
MONGODB_PASSWORD: MongoDB password
MONGODB_DATABASE: MongoDB database name
MONGODB_COLLECTION: MongoDB collection name

​

Constructor

• Args:
  • connection_string (str, optional): MongoDB connection string. If provided, other connection parameters are ignored.
  • host (str, optional): MongoDB hostname.
  • port (int, optional): MongoDB port. Defaults to 27017.
  • user (str, optional): MongoDB username.
  • password (str, optional): MongoDB password.
  • database (str, optional): MongoDB database name.
  • collection (str, optional): MongoDB collection name.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (MongoDB) The constructed dataloader using this method

​

Methods

• Args:
  • collection (str, optional): MongoDB collection name. Defaults to collection specified in constructor.
  • query (Dict, optional): Filter the result by using a query object. Examples: { "address": "Park Lane 38" }, { "address": { "$gt": "S" } }
• Returns: (DataFrame) Data frame object loaded from the MongoDB collection.

• Args:
  • data (Union[DataFrame, List[Dict]]): Data frame or List of Dictionary to export.
  • collection (str, optional): MongoDB collection name. Defaults to collection specified in constructor.

​

Databricks SQL

DATABRICKS_ACCESS_TOKEN: Databricks access token
DATABRICKS_HOST: Databricks workspace hostname
DATABRICKS_HTTP_PATH: Databricks SQL warehouse HTTP path
DATABRICKS_DATABASE: Databricks database/catalog name
DATABRICKS_SCHEMA: Databricks schema name

​

Constructor

• Args:
  • access_token (str): Databricks access token (required).
  • host (str): Databricks workspace hostname (required).
  • http_path (str): Databricks SQL warehouse HTTP path (required).
  • database (str, optional): Database/catalog name.
  • schema (str, optional): Schema name.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating connection

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (DatabricksSQL) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): The query to execute on the Databricks SQL warehouse.
  • **kwargs: Additional parameters to pass to the query.

• Args:
  • df (DataFrame): Data frame to export to Databricks SQL warehouse.
  • table_name (str): Name of the table to export data to.
  • database (str): Name of the database/catalog in which the table is located.
  • schema (str): Name of the schema in which the table is located.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table. Defaults to 'replace'.
  • **kwargs: See Advanced Export Parameters for available options.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional parameters to pass to the query.
• Returns: (DataFrame) Data frame containing the queried data.

​

Spark

SPARK_HOST: Spark host/master URL
SPARK_METHOD: Spark connection method
SPARK_SCHEMA: Spark database/schema name

​

Constructor

• Args:
  • host (str): Spark master URL (e.g., ‘local’, ‘spark://host:port’).
  • method (str, optional): Spark connection method.
  • database (str, optional): Database/schema name.
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings for creating Spark session

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (Spark) The constructed dataloader using this method

​

Methods

• Args:
  • query_string (str): Query to execute on the Spark session.
  • **kwargs: Additional arguments to pass to query.

• Args:
  • query_string (str): Query to fetch a table or subset of a table.
  • limit (int, Optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs: Additional arguments to pass to query.
• Returns: (DataFrame) Data frame containing the queried data.

• Args:
  • df (DataFrame): Data frame to export to Spark session.
  • table_name (str): Name of the table to export data to.
  • database (str): Name of the database in which the table is located.
  • if_exists (ExportWritePolicy): Specifies export policy if table exists. Either
    • 'fail': throw an error.
    • 'replace': drops existing table and creates new table of same name.
    • 'append': appends data frame to existing table. Defaults to 'replace'.
  • **kwargs: Additional arguments to pass to writer.

​

Azure Data Lake Storage

• CSV
• JSON
• Parquet

• AZURE_CLIENT_ID
• AZURE_CLIENT_SECRET
• AZURE_TENANT_ID
• AZURE_STORAGE_ACCOUNT_NAME

​

Constructor

• Args:
  • storage_account_name (str): Azure Storage account name (required).
  • verbose (bool): Enables verbose output printing. Defaults to True.
  • **kwargs: Additional settings. If AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_TENANT_ID are provided, they will be used for authentication. Otherwise, DefaultAzureCredential will be used.

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (AzureDataLakeStorage) The constructed dataloader using this method

​

Methods

• Args:
  • df (DataFrame): Data frame to export.
  • container_name (str): Name of the Azure container to export data to.
  • file_path (str): The desired output path of the data in your Azure Data Lake Storage.
  • format (FileFormat | str, Optional): Format of the file to export data frame to. Defaults to None, in which case the format is inferred.
  • **kwargs - Additional keyword arguments to pass to the file writer. See possible arguments by file formats in FileIO section.

• Args:
  • container_name (str): Name of the container to load data from.
  • file_path (str): Path of the file in Azure Data Lake Storage to load data from.
  • format (FileFormat | str, Optional): Format of the file to load data frame from. Defaults to None, in which case the format is inferred.
  • limit (int, optional): The number of rows to limit the loaded data frame to. Defaults to 10,000,000.
  • **kwargs - Additional keyword arguments to pass to the file reader. See possible arguments by file formats in FileIO section.
• Returns: (DataFrame) Data frame object loaded from data in the specified file.

​

Microsoft Fabric Warehouse

​

Constructor

​

Factory Methods

• Args:
  • config (BaseConfigLoader): Configuration loader object.
  • **kwargs: Additional parameters passed to the loader constructor
• Returns: (MicrosoftFabricWarehouse) The constructed dataloader using this method

​

Configuration Settings

• Configuration File - ConfigFileLoader
• Environment Variables - EnvironmentVariableLoader
• AWS Secrets Manager - AWSSecretLoader

from mage_ai.io.config import AWSSecretLoader
from mage_ai.io.redshift import Redshift

config = AWSSecretLoader()
loader = Redshift.from_config(config)

​

Configuration Loader APIs

• contains - checks if the configuration source contains the requested key. Commonly, the in operation is used to check for setting existence (but is not always identical as contains can accept multiple parameters while the in keyword only accepts the key).
  Copy

  if config.contains(ConfigKey.POSTGRES_PORT):
      ...
  # alternatively
  if ConfigKey.POSTGRES_PORT in config:
      ...
  
• get - gets the configuration setting associated with the given key. If the key doesn’t exist, returns None. Commonly, the data model overload __getitem__ is used to fetch a configuration setting (but is not always identical as get can accept multiple parameters while __getitem__ does not).
  Copy

  user = config.get(ConfigKey.REDSHIFT_DBUSER)
  # alternatively
  user = config[ConfigKey.REDSHIFT_DBUSER]
  

​

Configuration File

from mage_ai.io.config import ConfigKey, ConfigFileLoader

config = ConfigFileLoader('path/to/my/io_config.yaml', 'my_profile')
postgres_db = config[ConfigKey.POSTGRES_DBNAME]

• Standard: contains a subset of the configuration keys specified in ConfigKey. This is the default and recommended format. Below is an example configuration file using this format.
  Copy

  version: 0.1.0
  default:
      AWS_ACCESS_KEY_ID: AWS Access Key ID credential
      AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
      AWS_REGION: AWS Region
      REDSHIFT_DBNAME: Name of Redshift database to connect to
      REDSHIFT_HOST: Redshift Cluster hostname
      REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
      REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
      REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
  
  The above configuration file has a single profile named 'default'. Each profile organizes a set of keys to use (for example, distinguishing production keys versus development keys). A configuration file can have multiple profiles.
• Verbose: Instead of configuration keys, each profile stores an object of settings associated with each data migration client. This format was used in previous versions of this tool, and exists for backwards compatibility. Below is an example configuration file using this format.
  Copy

  version: 0.0.0
  default:
      AWS:
          Redshift:
              database: Name of Redshift database to connect to
              host: Redshift Cluster hostname
              port: Redshift Cluster port. Optional, defaults to 5439
              user: Redshift temp credentials username
              password: Redshift temp credentials password
          access_key_id: AWS Access Key ID credential
          secret_access_key: AWS Secret Access Key credential
          region: AWS Region
  

version: 0.1.0
default:
    GOOGLE_SERVICE_ACC_KEY_FILEPATH: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS') }}"

• filepath (os.PathLike, optional): Path to IO configuration file. Defaults to '[repo_path]/io_config.yaml'
• profile (str, optional): Profile to load configuration settings from. Defaults to 'default'.

• Args:
  • key (str): Name of the configuration setting to check.
• Returns (bool) Returns True if configuration setting exists, otherwise returns False.

• Args:
  • key (str): Key name of the configuration setting to load
• Returns: (Any) Configuration setting corresponding to the given key

​

Environment Variables

from mage_ai.io.config import ConfigKey, EnvironmentVariableLoader

config = EnvironmentVariableLoader()
postgres_db = config[ConfigKey.POSTGRES_DBNAME]

• Args:
  • key (ConfigKey | str): Name of the configuration setting to check existence of.
• Returns (bool) Returns True if configuration setting exists, otherwise returns False.

• Args:
  • env_var (str): Name of the environment variable to load configuration setting from
• Returns: (Any) The configuration setting stored under env_var

​

AWS Secret Loader

• Configure your AWS profile using the AWS CLI
  Copy

  aws configure
  
• Manually specify your AWS Credentials when constructing the configuration loader python config = AWSSecretLoader( aws_access_key_id = 'your access key id', aws_secret_access_key = 'your secret key', region_name = 'your region' )

from mage_ai.io.config import ConfigKey, AWSSecretLoader

config = AWSSecretLoader()
postgres_db = config[ConfigKey.POSTGRES_DBNAME]
# with finer control on version
postgres_db = config.get(ConfigKey.POSTGRES_DBNAME, version_id='my_version_id')

• Keyword Arguments:
  • aws_access_key_id (str, Optional): AWS access key ID credential.
  • aws_secret_access_key (str, Optional): AWS secret access key credential.
  • region_name (str, Optional): AWS region which Secrets Manager is created in.

• both version_id and version_stage_label are specified, both must agree on the secret version.
• neither of version_id or version_stage_label are specified, any version is checked.
• one of version_id and version_stage_label are specified, the associated version is checked.

• Args:
  • secret_id (str): ID of the secret to load
  • version_id (str, Optional): ID of the version of the secret to load. Defaults to None.
  • version_stage_label (str, Optional): Staging label of the version of the secret to load. Defaults to None.
• Returns: (bool) Returns true if secret exists, otherwise returns false.

• both version_id and version_stage_label are specified, both must agree on the secret version.
• neither of version_id or version_stage_label are specified, the current version is loaded.
• one of version_id and version_stage_label are specified, the associated version is loaded.

• Args:
  • secret_id (str): ID of the secret to load
  • version_id (str, Optional): ID of the version of the secret to load. Defaults to None.
  • version_stage_label (str, Optional): Staging label of the version of the secret to load. Defaults to None.
• Returns: (bytes | str) The secret stored under secret_id in AWS secret manager. If secret is a binary value, returns a bytes object; else returns a string object