​
Overview

Use Google Cloud Storage (GCS) as a destination in Mage to write data from your pipelines directly into a GCS bucket. This is ideal for exporting pipeline outputs in formats like Parquet or CSV for downstream analytics, storage, or ML workflows. Mage supports native integration with GCS using either a service account key file or automatic authentication via GCP instance metadata.

​
Configuration Parameters

You must provide the following parameters when configuring GCS as a destination:
KeyDescriptionExample ValueRequired
bucketName of the target GCS bucket where files will be stored.user_generated_content✅
google_application_credentialsPath to the GCP service account JSON key file. Leave as null if Mage is running on GCP and should use the default instance service account./path/to/key.json✅
file_typeFormat of the files to be written. Supported values: parquet, csv.parquet or csv✅
object_key_pathRelative path inside the bucket where files will be stored. Do not include gs://, bucket name, or table name.users/ds/20221225✅
date_partition_format(Optional) Datetime format to use for partitioned folders. If null, partitioning will be skipped. Examples: %Y%m%d, %Y%m%dT%H.null or %Y%m%d❌

​
Notes

  • The object_key_path defines where in the GCS bucket the data will be written. Mage will automatically append file extensions based on the selected file_type.
  • If you’re running Mage inside a GCP service (like Cloud Run, GKE, or Compute Engine), and you leave google_application_credentials as null, Mage will use the instance’s IAM identity for authentication.
  • Use date_partition_format to enable directory-based partitioning by time, which is useful for managing time-series datasets or incremental loads.

​
Frequently Asked Questions

​
Can Mage write to a GCS bucket if it’s running inside GCP?

Yes. If Mage is running inside a GCP environment (like GKE, Cloud Run, or Compute Engine), you can leave google_application_credentials as null. Mage will use the default service account attached to the instance for authentication.

​
What file formats are supported when exporting data to GCS?

Mage currently supports parquet and csv as the file_type options when writing data to Google Cloud Storage.

​
How does object_key_path work?

object_key_path specifies the path inside the bucket where your files will be stored. It should exclude the bucket name and should not include gs://. For example, users/2025_07_08.

​
What does date_partition_format do?

When provided, this setting tells Mage to create date-based subfolders (e.g., 20250708, 20250708T13) based on the run time. If set to null, partitioning is disabled and all files will go into the specified object_key_path.

​
Do I need to manually set file extensions?

No. Mage automatically adds the appropriate extension (.parquet or .csv) to each file based on the file_type you select.

​
Can I write to a multi-region or dual-region GCS bucket?

Yes. Mage can write to any standard, multi-region, or dual-region GCS bucket as long as the configured credentials have write access.