GCP Secret Management

​

Creating secrets

1. Go to Google Secret Manager UI.
2. Click the button at the top labeled + CREATE SECRET.
3. Fill in the name of your secret; e.g. bigquery_credentials.
4. Under Secret value, upload your service account credentials JSON file or paste the JSON into the text area labeled Secret value.
5. Scroll all the way down and click the button CREATE SECRET.

You can mount secrets from Google Secret Manager through Terraform configurations or through the Google Console UI.

​

Using secrets locally

​

Download credentials from GCP UI

1. Download the credentials JSON file from GCP.

2. Run Mage and mount the secrets as a volume in Docker. Follow these instructions to learn how to do this.

3. Here are example code snippets to read from that credentials JSON file:

   Python

   YAML

   Copy

   with open('/home/secrets/gcp_credentials.json', 'r') as f:
       print(f.read())
   

   Note

   This code example assumes your credentials JSON file downloaded from GCP is named gcp_credentials.json and that the mount path (e.g. -v) you used when running Docker is /home/secrets.

​

Download credentials using gcloud CLI

1. Authenticate locally by running this command in your local terminal:

   Copy

   gcloud auth application-default login
   

2. Create a new .env file in your Mage project folder with the following values:

   Copy

   GOOGLE_APPLICATION_CREDENTIALS="[PATH TO YOUR USER CREDENTIALS, MOST LIKELY: ~/.config/gcloud/application_default_credentials.json]"
   GCS_BUCKET=[YOUR DEV BUCKET]
   GCLOUD_PROJECT=[YOUR PROJECT]
   

3. Run Mage using Docker and set the environment variable GOOGLE_APPLICATION_CREDENTIALS. Follow these instructions to learn how to do this. For example, set the environment variable to:

   Copy

   -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/FILE_NAME.json
   

4. Run Mage and mount the secrets as a volume in Docker. Follow these instructions to learn how to do this. For example:

   Copy

   -v ~/.config/gcloud/application_default_credentials.json:/tmp/keys/FILE_NAME.json
   

5. Here is an example code snippet:

   Copy

   from google.cloud import storage
   from pandas import DataFrame
   from datetime import datetime
   import pytz
   import os
   ​
   ​
   @data_exporter
   def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
   ​
       bucket_name = os.getenv('GCS_BUCKET')
   ​
       now = datetime.utcnow()
       pt = pytz.timezone("America/Los_Angeles")
       now_pst = pytz.utc.localize(now).astimezone(pt)
   ​
       object_key = f'test_file_{now_pst.strftime("%Y-%m-%d")}.csv'
   ​
       storage_client = storage.Client()
       bucket = storage_client.bucket(bucket_name)
       blob = bucket.blob(object_key)
   ​
       blob.upload_from_string(df.to_csv())
   ​
       print(f'df uploaded to {bucket}/{object_key}.')
   

​

GCP Secret Manager Integration in Mage Pro

Securely manage and inject sensitive credentials into your Mage Pro pipelines using Google Cloud Secret Manager. This Pro-only feature helps enterprises meet cloud security and compliance standards by keeping secrets like API keys, database passwords, and tokens outside of source code.

​

Required Environment Variables

Before accessing secrets, configure your Mage Pro cluster with the following environment variables:

• GOOGLE_APPLICATION_CREDENTIALS: Full path to your GCP service account credentials JSON file. You can upload the credentials file directly to the Mage Pro cluster using the file browser interface.

• GCP_PROJECT_ID: The Google Cloud Project ID associated with your Secret Manager secrets.

These credentials must have Secret Manager Secret Accessor permissions for the secrets you intend to retrieve.

​

How to Use the GCP Secret Variable

You can reference GCP secrets programmatically in Python blocks or declaratively in YAML configs.

​

Python code

Use this approach inside a block in your Mage pipeline:

from mage_ai.services.gcp.secret_manager.secret_manager import get_secret

# Retrieve a secret by ID from GCP Secret Manager
secret_value = get_secret('secret_id')

​

YAML config

To inject a secret dynamically into a YAML config (e.g., for a data source, destination, or authentication setting):

{{gcp_secret_var('secret_id')}}

Mage will automatically resolve and substitute the secret value at runtime.