​
Basic config

connector_type: google_cloud_storage
bucket: bucket
prefix: prefix
file_type: parquet
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
  • bucket: The Google Cloud Storage bucket you want to use to store the data.
  • prefix: The Google Cloud Storage path prefix. The whole S3 path will be s3://{bucket}/{prefix}.
  • file_type: parquet or csv
  • buffer_size_mb and buffer_timeout_seconds: Mage puts messages in a buffer before uploading to Google Cloud Storage. You can configure the size and timeout of the buffer to control the file size and the delay.
  • path_to_credentials_json_file: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate.

​
Configure time-based partition

date_partition_format: "%Y%m%dT%H"

If you want to store the Google Cloud Storage data in time-based paratition folders. You can add date_partition_format to the config. Example values: %Y%m%d, %Y%m%dT%H.