
Add credentials
- Create a new pipeline or open an existing pipeline.
- Expand the left side of your screen to view the file browser.
- Scroll down and click on a file named
io_config.yaml
. - Enter the following keys and values under the key named
default
(you can have multiple profiles, add it under whichever is relevant to you)
Using Python block
- Create a new pipeline or open an existing pipeline.
- Add a data loader or transformer block (the code snippet below is for a data loader).
-
Select
Generic (no template)
. -
Enter this code snippet (note: change the
config_profile
fromdefault
if you have a different profile): - Run the block.
MinIO Support
To execute requests on MinIO databases, simply add:io_config.yaml
file
Errors
S3 connection endpoint URL error Open theio_config.yaml
file at the root of your project (e.g.
default_repo/io_config.yaml
).
Change the value for the AWS_REGION
key to the region of your S3 bucket (e.g.
us-west-2
).
Permissions
Ensure your AWS IAM user or role has the following permissions:s3:GetObject
s3:PutObject
s3:ListBucket
- Read data from S3 (e.g.
.csv
,.parquet
,.json
) - Write query results or transformed data to S3
- List contents of a bucket when needed