GCP

​

Using Docker

​

Download credentials from GCP UI

1. Download the credentials JSON file from GCP.

2. Run Mage and mount the secrets as a volume in Docker. Follow these instructions to learn how to do this.

3. Here are example code snippets to read from that credentials JSON file:

   Python

   YAML

   with open('/home/secrets/gcp_credentials.json', 'r') as f:
       print(f.read())
   

   Note

   This code example assumes your credentials JSON file downloaded from GCP is named gcp_credentials.json and that the mount path (e.g. -v) you used when running Docker is /home/secrets.

---

​

Download credentials using gcloud CLI

1. Authenticate locally by running this command in your local terminal:

   gcloud auth application-default login
   

2. Create a new .env file in your Mage project folder with the following values:

   GOOGLE_APPLICATION_CREDENTIALS="[PATH TO YOUR USER CREDENTIALS, MOST LIKELY: ~/.config/gcloud/application_default_credentials.json]"
   GCS_BUCKET=[YOUR DEV BUCKET]
   GCLOUD_PROJECT=[YOUR PROJECT]
   

3. Run Mage using Docker and set the environment variable GOOGLE_APPLICATION_CREDENTIALS. Follow these instructions to learn how to do this. For example, set the environment variable to:

   -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/FILE_NAME.json
   

4. Run Mage and mount the secrets as a volume in Docker. Follow these instructions to learn how to do this. For example:

   -v ~/.config/gcloud/application_default_credentials.json:/tmp/keys/FILE_NAME.json
   

5. Here is an example code snippet:

   from google.cloud import storage
   from pandas import DataFrame
   from datetime import datetime
   import pytz
   import os
   ​
   ​
   @data_exporter
   def export_data_to_google_cloud_storage(df: DataFrame, **kwargs) -> None:
   ​
       bucket_name = os.getenv('GCS_BUCKET')
   ​
       now = datetime.utcnow()
       pt = pytz.timezone("America/Los_Angeles")
       now_pst = pytz.utc.localize(now).astimezone(pt)
   ​
       object_key = f'test_file_{now_pst.strftime("%Y-%m-%d")}.csv'
   ​
       storage_client = storage.Client()
       bucket = storage_client.bucket(bucket_name)
       blob = bucket.blob(object_key)
   ​
       blob.upload_from_string(df.to_csv())
   ​
       print(f'df uploaded to {bucket}/{object_key}.')