How to configure Google Cloud Storage (GCS) as a destination in Mage to write data files in Parquet or CSV format with optional date partitioning.
Key | Description | Example Value | Required |
---|---|---|---|
bucket | Name of the target GCS bucket where files will be stored. | user_generated_content | ✅ |
google_application_credentials | Path to the GCP service account JSON key file. Leave as null if Mage is running on GCP and should use the default instance service account. | /path/to/key.json | ✅ |
file_type | Format of the files to be written. Supported values: parquet , csv . | parquet or csv | ✅ |
object_key_path | Relative path inside the bucket where files will be stored. Do not include gs:// , bucket name, or table name. | users/ds/20221225 | ✅ |
date_partition_format | (Optional) Datetime format to use for partitioned folders. If null , partitioning will be skipped. Examples: %Y%m%d , %Y%m%dT%H . | null or %Y%m%d | ❌ |
object_key_path
defines where in the GCS bucket the data will be written. Mage will automatically append file extensions based on the selected file_type
.google_application_credentials
as null
, Mage will use the instance’s IAM identity for authentication.date_partition_format
to enable directory-based partitioning by time, which is useful for managing time-series datasets or incremental loads.google_application_credentials
as null
. Mage will use the default service account attached to the instance for authentication.
parquet
and csv
as the file_type
options when writing data to Google Cloud Storage.
object_key_path
work?object_key_path
specifies the path inside the bucket where your files will be stored. It should exclude the bucket name and should not include gs://
. For example, users/2025_07_08
.
date_partition_format
do?20250708
, 20250708T13
) based on the run time. If set to null
, partitioning is disabled and all files will go into the specified object_key_path
.
.parquet
or .csv
) to each file based on the file_type
you select.