Add credentials
- Create a new pipeline or open an existing pipeline.
- Expand the left side of your screen to view the file browser.
- Scroll down and click on a file named
io_config.yaml
.
- Enter the following keys and values under the key named
default
(you can
have multiple profiles, add it under whichever is relevant to you)
version: 0.1.1
default:
AWS_ACCESS_KEY_ID: ...
AWS_SECRET_ACCESS_KEY: ...
Using Python block
-
Create a new pipeline or open an existing pipeline.
-
Add a data loader or transformer block (the code snippet below is for a data
loader).
-
Select Generic (no template)
.
-
Enter this code snippet (note: change the config_profile
from default
if
you have a different profile):
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.s3 import S3
from os import path
from pandas import DataFrame
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
@data_loader
def load_from_s3_bucket(**kwargs) -> DataFrame:
config_path = path.join(get_repo_path(), 'io_config.yaml')
config_profile = 'default'
bucket_name = '...' # Change to your bucket name
object_key = '...' # Change to your object key
return S3.with_config(ConfigFileLoader(config_path, config_profile)).load(
bucket_name,
object_key,
)
-
Run the block.
MinIO Support
To execute requests on MinIO databases, simply add:
AWS_ENDPOINT: your_endpoint:port #example: "https://play.min.io:9000"
to the io_config.yaml
file
Errors
S3 connection endpoint URL error
Open the io_config.yaml
file at the root of your project (e.g.
default_repo/io_config.yaml
).
Change the value for the AWS_REGION
key to the region of your S3 bucket (e.g.
us-west-2
).
Permissions
Ensure your AWS IAM user or role has the following permissions:
s3:GetObject
s3:PutObject
s3:ListBucket
These permissions are required to:
- Read data from S3 (e.g.
.csv
, .parquet
, .json
)
- Write query results or transformed data to S3
- List contents of a bucket when needed
Responses are generated using AI and may contain mistakes.