Mage home pagelight logo
  1. GCP
  2. Deploy to GCP with Terraform

​
Pre-requisites

​
Setup

Download Mage maintained Terraform scripts.

​
Permissions

Go to the IAM management dashboard, find the service account associated to the account you just logged into, and then add these roles to that service account (e.g. choose your account as the principal when adding new roles):

  1. Artifact Registry Read
  2. Artifact Registry Writer
  3. Cloud Run Developer
  4. Cloud SQL
  5. Service Account Token Creator

​
Terraform plan

You can run the following command to see all the resources that will be created by Terraform:

cd gcp

terraform plan

By default, here are the resources that will be created.

​
1. Install gcloud CLI

Follow these instructions to install the CLI on your workstation.

For macOS and Homebrew, you can do the following:

brew install --cask google-cloud-sdk

​
2. Log into GCP from CLI

If you don’t already have an account in GCP, create one here.

Once you created an account, from your terminal run the following command to log in:

gcloud init

gcloud auth application-default login

​
3. Push Docker image to GCP Artifact Registry

GCP doesn’t support using Docker images from Docker Hub. We’ll need to pull the Mage Docker image, tag it, and push it to GCP Artifact Registry.

Here are the steps we’ll take:

  1. Create a repository on GCP Artifact Registry by clicking the + CREATE REPOSITORY button.

    • Fill in the Name
    • Choose Docker as the Format
    • Choose any Location type
    • Choose any Encryption
    • Click CREATE

  2. Click on the newly created repository (e.g. mage-docker).

  3. Near the top of the page, click the link SETUP INSTRUCTIONS or read these instructions to set up authentication for Docker.

    • Run the following command in your terminal:

      gcloud auth configure-docker [region]-docker.pkg.dev

    • An example command could look like this:

      gcloud auth configure-docker us-west2-docker.pkg.dev

  4. Pull the Mage Docker image:

    docker pull mageai/mageai:latest

    • If your local workstation is using macOS and a silicon chip (e.g. M1, M2, etc), then run this command instead:

      docker pull --platform linux/amd64 mageai/mageai:latest

  5. Tag the pulled Mage docker image or use a previously tagged Docker image you built when following this CI/CD guide:

    • Sample commands if using vanilla Mage Docker image:

      docker tag mageai/mageai:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest

      • An example command could look like this:

        docker tag mageai/mageai:latest us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest

    • Sample commands if using previously tagged a custom Docker image using the tag mageprod:latest:

      docker tag mageprod:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest

      • An example command could look like this if you previously tagged your custom Docker image with the tag mageprod:latest:

        docker tag mageprod:latest us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest

  6. Push the local Docker image to GCP Artifact Registry:

    docker push [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest

    • An example command could look like this:

      docker push us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest

​
4. Customize Terraform settings

Project ID (REQUIRED)

Before running any Terraform commands, please change the default value of the variable named project_id in the ./gcp/variables.tf file.

variable "project_id" {
  type        = string
  description = "The name of the project"
  default     = "unique-gcp-project-id"
}

Docker image (REQUIRED)

In the repository you created in GCP Artifact Repository, you’ll see a list of Docker images. Click on an image name, then copy the full path to the image (e.g. us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai).

In the file ./gcp/variables.tf, you can change the default value under docker_image:

variable "docker_image" {
  type        = string
  description = "The Docker image url in the Artifact Registry repository to be deployed to Cloud Run"
  default     = "us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai"
}

Application Name

In the file ./gcp/variables.tf, you can change the default value under app_name:

variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}

Region

In the file ./gcp/main.tf, you can change the default value under region:

variable "region" {
  type        = string
  description = "The default compute region"
  default     = "us-west2"
}

​
Environment variables for application

Set your environment variables in your running cloud environment by adding the following under the resource named google_cloud_run_service in the file ./gcp/main.tf:

resource "google_cloud_run_service" "run_service" {
  ...

  template {
    spec {
      containers {
        ...
        env {
          name  = "NAME_OF_ENVIRONMENT_VARIABLE"
          value = "value of environment variable"
        }
      }
    }
  }
}

​
Secrets

  1. Go to Google Secret Manager UI.
  2. Click the button at the top labeled + CREATE SECRET.
  3. Fill in the name of your secret; e.g. bigquery_credentials.
  4. Under Secret value, upload your service account credentials JSON file or paste the JSON into the textarea labeled Secret value.
  5. Scroll all the way down and click the button CREATE SECRET.

You can mount secrets from Google Secret Manager through Terraform configurations or through the Google Console UI.

​
Method A: Terraform configurations

  1. Once you save a secret in Google Secret Manager, click on the PERMISSIONS tab.

  2. Click the button + GRANT ACCESS.

  3. Under the field labeled New principles, add the service account that is associated to your Google Cloud Run. If you can’t find this, try deploying GCP with Terraform and you’ll receive an error that looks like this:

    ╷
│ Error: resource is in failed state "Ready:False", message: Revision 'mage-2992l' is not ready and cannot serve traffic. spec.template.spec.volumes[1].secret.items[0].key: Permission denied on secret: projects/1054912425479/secrets/bigquery_credentials/versions/latest for Revision service account 1054912425479-compute@developer.gserviceaccount.com. The service account used must be granted the 'Secret Manager Secret Accessor' role (roles/secretmanager.secretAccessor) at the secret, project or higher level.
│
│   with google_cloud_run_service.run_service,
│   on main.tf line 88, in resource "google_cloud_run_service" "run_service":
│   88: resource "google_cloud_run_service" "run_service" {
│
╵

    If you still get this error when re-running terraform apply even after granting access, try Method B down below.

    By default, the service account used by Cloud Run that needs access to the secret is the Compute Engine default service account. If you want to grant access to a different service account, configure the service_account_name argument in the spec block in main.tf:

    # Create the Cloud Run service
resource "google_cloud_run_service" "run_service" {
  name = var.app_name
  location = var.region

  template {
    spec {
      service_account_name = "[your service account email here]"
    }
  }
}

  4. Under the field labeled Select a role, enter the value Secret Manager Secret Accessor.

  5. Click the button SAVE.

  6. Mount secrets to Google Cloud Run via Terraform in the file ./gcp/main.tf:

    resource "google_cloud_run_service" "run_service" {
  ...

  template {
    spec {
      containers {
        ...
        env {
          name = "path_to_keyfile"
          value = "/secrets/bigquery/bigquery_credentials"
        }
        volume_mounts {
          name       = "secrets-bigquery_credentials"
          mount_path = "/secrets/bigquery"
        }
      }
      volumes {
        name = "secrets-bigquery_credentials"
        secret {
          secret_name  = "bigquery_credentials"
          items {
            key  = "latest"
            path = "bigquery_credentials"
          }
        }
      }
    }
  }
}

  7. In your Python code or in any field that support variable interpolation, enter the following to get the value from the secret: 1

    import os


with open(os.getenv('path_to_keyfile'), 'r') as f:
    print(f.read())

​
Method B: through GCP UI (aka console but not to be confused with CLI)

  1. Go to your Google Cloud Run dashboard, find the service running Mage, and click on it to view its details.

  2. Follow these Google provided instructions to manually mount the secret to the running Mage Cloud Run service and grant the right permissions for that service to access those secrets. After you mount the secrets, you can create environment variables in your Cloud Run that reference your secrets. Follow the below example to setup the mounted secret and environment variable for that secret:

    Secrets
    Secret namebigquery_credentials
    Reference methodMounted as volume
    Mount path/secrets/bigquery
    Path 1bigquery_credentials
    Version 1latest
    Environment variables
    Namepath_to_keyfile
    Value/secrets/bigquery/bigquery_credentials

​
More

Other variables defined in ./gcp/variables.tf can also be customized to your needs.

​
5. Deploy

  1. Change directory into scripts folder:

    cd gcp

  2. Initialize Terraform:

    terraform init

    A sample output could look like this:

    Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/google versions matching ">= 3.3.0"...
- Finding latest version of hashicorp/http...
- Installing hashicorp/google v4.38.0...
- Installed hashicorp/google v4.38.0 (signed by HashiCorp)
- Installing hashicorp/http v3.1.0...
- Installed hashicorp/http v3.1.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

  3. Deploy:

    terraform apply

    A sample output could look like this:

    Apply complete! Resources: 7 added, 1 changed, 0 destroyed.

Outputs:

service_ip = "34.107.187.208"

    It’ll take a few minutes for GCP Cloud Run to start up and be ready to receive requests.

    After a few minutes, open a browser and go to http://[IP_address

    Change the IP_address to the IP address that was output in your terminal after successfully running terraform apply.

​
Errors

403 Forbidden

If you get this error when trying to open the above IP address in your browser, open the security group named [application_name]-security-policy. Click on that security group and verify your IP address is whitelisted.

If it isn’t, add a new rule with the following values:

ModeMatchActionPriority
Basic modeEnter your IP addressAllow100

If you still get 403, check to see if you’re using http:// and NOT https://.

​
Updating Mage versions

After Mage is running in your cloud, you can update the Mage Docker image version by running the following command in your CLI tool:

gcloud run deploy [app_name] --image [docker_image]

app_name

This is the value you changed when editing the ./gcp/variables.tf file.

​
Misc

​
Security

To enable other IP addresses access to Mage, open the security group named [application_name]-security-policy. Click on that security group and add a new rule with the following values:

ModeMatchActionPriority
Basic modeEnter your IP addressAllow100

To enable IP addresses access to specific endpoints, you can follow this example to add rules to security policy. Another way is to add a new rule through UI with the following values:

ModeMatchActionPriority
Advanced moderequest.path.startsWith('/api/pipeline_schedules/123/pipeline_runs') && inIpRange(origin.ip, '[IP address]/32')Allow200

​
HTTPS enabling

To enable HTTPS for Mage app deployed on GCP, you need to firstly make sure you have a domain. Then follow the steps below to set up HTTPS:

  1. In ./gcp/variables.tf file, update the default value of ssl to be true, and set the default value of domain variable to be the domain url you want to use.
  2. Run terraform apply to create the HTTPS load balancer.
  3. Get the service_ip from the output of terraform command.
  4. Update the DNS A and AAAA records to point the domain url to the load balancer’s IP address: https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs#update-dns
  5. Visit the domain url to access your Mage app.

​
Terminate all resources

If you want to delete all resources created from deploying, run the following command:

terraform destroy

A sample output could look like this:

Destroy complete! Resources: 10 destroyed.

​
DBT

For information about how to serve DBT docs in production, see here.