GCP Secret Management

​

Creating secrets

1. Go to Google Secret Manager UI.
2. Click the button at the top labeled + CREATE SECRET.
3. Fill in the name of your secret; e.g. bigquery_credentials.
4. Under Secret value, upload your service account credentials JSON file or paste the JSON into the text area labeled Secret value.
5. Scroll all the way down and click the button CREATE SECRET.

​

Using secrets locally

​

Download credentials from GCP UI

1. Download the credentials JSON file from GCP.
2. Run Mage and mount the secrets as a volume in Docker. Follow these instructions to learn how to do this.
3. Here are example code snippets to read from that credentials JSON file:
   Python

   YAML

   Copy

   with open('/home/secrets/gcp_credentials.json', 'r') as f:
       print(f.read())
   

   Note This code example assumes your credentials JSON file downloaded from GCP is named gcp_credentials.json and that the mount path (e.g. -v) you used when running Docker is /home/secrets.

​

Download credentials using gcloud CLI

1. Authenticate locally by running this command in your local terminal:
   Copy

   gcloud auth application-default login
   

2. Create a new .env file in your Mage project folder with the following values:
   Copy

   GOOGLE_APPLICATION_CREDENTIALS="[PATH TO YOUR USER CREDENTIALS, MOST LIKELY: ~/.config/gcloud/application_default_credentials.json]"
   GCS_BUCKET=[YOUR DEV BUCKET]
   GCLOUD_PROJECT=[YOUR PROJECT]
   

3. Run Mage using Docker and set the environment variable GOOGLE_APPLICATION_CREDENTIALS. Follow these instructions to learn how to do this. For example, set the environment variable to:
   Copy

   -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/FILE_NAME.json
   

4. Run Mage and mount the secrets as a volume in Docker. Follow these instructions to learn how to do this. For example:
   Copy

   -v ~/.config/gcloud/application_default_credentials.json:/tmp/keys/FILE_NAME.json
   

5. Here is an example code snippet:
   Copy

   from google.cloud import storage
   from pandas import DataFrame
   from datetime import datetime
   import pytz
   import os
   ​
   ​
   @data_exporter
   def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
   ​
       bucket_name = os.getenv('GCS_BUCKET')
   ​
       now = datetime.utcnow()
       pt = pytz.timezone("America/Los_Angeles")
       now_pst = pytz.utc.localize(now).astimezone(pt)
   ​
       object_key = f'test_file_{now_pst.strftime("%Y-%m-%d")}.csv'
   ​
       storage_client = storage.Client()
       bucket = storage_client.bucket(bucket_name)
       blob = bucket.blob(object_key)
   ​
       blob.upload_from_string(df.to_csv())
   ​
       print(f'df uploaded to {bucket}/{object_key}.')
   

​

GCP Secret Manager Integration in Mage Pro

​

Required Environment Variables

• GOOGLE_APPLICATION_CREDENTIALS: Full path to your GCP service account credentials JSON file. You can upload the credentials file directly to the Mage Pro cluster using the file browser interface.
• GCP_PROJECT_ID: The Google Cloud Project ID associated with your Secret Manager secrets.

​

How to Use the GCP Secret Variable

​

Python code

from mage_ai.services.gcp.secret_manager.secret_manager import get_secret

# Retrieve a secret by ID from GCP Secret Manager
secret_value = get_secret('secret_id')

​

YAML config

{{gcp_secret_var('secret_id')}}