Basic config
bucket
: The Google Cloud Storage bucket you want to use to store the data.prefix
: The Google Cloud Storage path prefix. The whole S3 path will bes3://{bucket}/{prefix}
.file_type
:parquet
orcsv
buffer_size_mb
andbuffer_timeout_seconds
: Mage puts messages in a buffer before uploading to Google Cloud Storage. You can configure the size and timeout of the buffer to control the file size and the delay.path_to_credentials_json_file
: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate.
Configure time-based partition
date_partition_format
to the config. Example values: %Y%m%d
, %Y%m%dT%H
.