Configuration

To set up the Amazon S3 source, provide the following configuration parameters:

KeyDescriptionSample ValueRequired
aws_access_key_idAWS access key ID.abc123
aws_secret_access_keyAWS secret access key.xyz456
aws_regionRegion where the S3 bucket is located.us-west-2
role_arn(Optional) ARN of the IAM role to assume for accessing the S3 bucket.arn:aws:iam::111111:role/example-role
bucketName of the AWS S3 bucket to read data from.user_generated_content
file_typeType of files in the bucket. Supported values: parquet, csv, infer. If infer, the type is detected based on file extension.infer
prefixPath within the bucket where files are located. Do not include s3://, the bucket name, or the table name in this path.users/ds/20221225
search_patternRegular expression to match files within the prefix.test_table\\/.*\\.csv
single_stream_in_prefixIf true, all files in the prefix are treated as a single stream. Defaults to false.false
table_configsA list of table configurations to configure multiple streams. Each table config must include prefix, search_pattern, and table_name.[{"prefix": "users/", "search_pattern": "test_table\\/.*\\.csv", "table_name": "test_table"}]
aws_endpoint(Optional) Custom S3-compatible endpoint (e.g., for MinIO).https://play.min.io