Skip to main content

​
Basic config

connector_type: google_cloud_storage
bucket: bucket
prefix: prefix
file_type: parquet
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
  • bucket: The Google Cloud Storage bucket you want to use to store the data.
  • prefix: The Google Cloud Storage path prefix. The whole GCS path will be gs://{bucket}/{prefix}.
  • file_type: parquet, csv, or delta (for Delta Lake tables)
  • buffer_size_mb and buffer_timeout_seconds: Mage puts messages in a buffer before uploading to Google Cloud Storage. You can configure the size and timeout of the buffer to control the file size and the delay.
  • path_to_credentials_json_file: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate. See Referencing Variables for configuring credentials via environment variables or secret variables.
  • prefix: The Google Cloud Storage path prefix. The whole S3 path will be s3://{bucket}/{prefix}.
  • file_type: parquet or csv
  • buffer_size_mb and buffer_timeout_seconds: Mage puts messages in a buffer before uploading to Google Cloud Storage. You can configure the size and timeout of the buffer to control the file size and the delay.
  • path_to_credentials_json_file: Google service account credential json file. If Mage is running on GCP, you can leave this value null and then Mage will use the instance service account to authenticate.

​
Configure time-based partition

date_partition_format: "%Y%m%dT%H"
If you want to store the Google Cloud Storage data in time-based partition folders. You can add date_partition_format to the config. Example values: %Y%m%d, %Y%m%dT%H.

​
Delta Lake configuration

Note: Delta Lake format (file_type: delta) is a Mage Pro only feature. When using file_type: delta, you can write data to Delta Lake tables in Google Cloud Storage. 
connector_type: google_cloud_storage
bucket: my-bucket
prefix: my-prefix
file_type: delta
table_uri: gs://my-bucket/path/to/delta-table  # Optional: if not provided, constructed from bucket/prefix
mode: append  # 'append' or 'overwrite'
buffer_size_mb: 5
buffer_timeout_seconds: 300
path_to_credentials_json_file: /path/to/your/service/account/key.json
  • table_uri: (Optional) Full URI for the Delta Lake table (e.g., gs://bucket/path/to/table). If not provided, the URI will be automatically constructed from bucket and prefix.
  • mode: Write mode for Delta Lake tables. Valid values:
    • append: Adds new rows to the existing table (default)
    • overwrite: Replaces all existing rows in the table
Note: When using Delta Lake format, time-based partitioning (date_partition_format) is not applied as Delta Lake manages its own internal file structure. Security Note: Storing credentials in configuration files can be insecure. Prefer using environment variables (e.g., GOOGLE_APPLICATION_CREDENTIALS) or secret variables when possible, especially for production deployments. See Referencing Variables for configuring credentials via environment variables or secret variables. If you want to store the Google Cloud Storage data in time-based paratition folders. You can add date_partition_format to the config. Example values: %Y%m%d, %Y%m%dT%H.