bucket
: The S3 bucket you want to use to store the data.prefix
: The S3 path prefix. The whole S3 path will be s3://{bucket}/{prefix}
.file_type
: parquet
or csv
buffer_size_mb
and buffer_timeout_seconds
: Mage puts messages in a buffer before uploading to S3.
You can configure the size and timeout of the buffer to control the file size and the delay.region
parameter in the default config template, it can be ignored. If you specify any valid AWS region names or null, you can still access to S3 buckets in other regions. However, if you do include the parameter value for region
, it must be a valid region name for AWS or null so it won’t error out.
date_partition_format
to the config. Example values: %Y%m%d
, %Y%m%dT%H
.
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION