- Local to cloud
- GitHub Actions
Local to cloud
GitHub Actions
Development (local) and production (cloud) using GitHub Actions.
Mage project setup
Follow the Mage project setup instructions.
GitHub Actions setup
- Create a new repository on GitHub.
- Open your repository on GitHub, then click the tab labeled Settings.
- Click the section labeled Security on the left hand side to expand it.
- Click the link labeled Actions.
- Click the button labeled New repository secret in the top right corner.
- Follow the instructions below for your specific cloud provider:
AWS
- If you haven’t already, create a new AWS ECR repository.
- You’ll need AWS credentials with the following policy permissions:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:BatchCheckLayerAvailability", "ecr:CompleteLayerUpload", "ecr:GetAuthorizationToken", "ecr:InitiateLayerUpload", "ecr:PutImage", "ecr:UploadLayerPart", "ecs:DeregisterTaskDefinition", "ecs:DescribeClusters", "ecs:DescribeServices", "ecs:DescribeTaskDefinition", "ecs:RegisterTaskDefinition", "ecs:UpdateService", "iam:PassRole" ], "Resource": "*" } ] }
- In the field labeled Name, enter the value
AWS_ACCESS_KEY_ID
. - In the field labeled Secret, enter your AWS Access Key ID.
- Click the button labeled Add secret to save.
- Add a 2nd secret by clicking the button labeled New repository secret in the top right corner.
- In the field labeled Name, enter the value
AWS_SECRET_ACCESS_KEY
. - In the field labeled Secret, enter your AWS Secret Access Key.
- Click the button labeled Add secret to save.
- Click on the tab labeled Actions.
- On the left side, click the button labeled New workflow.
- Find the link labeled
set up a workflow yourself
and click it. - Copy the contents from the GitHub Action YAML file for AWS at templates/github_actions/build_and_deploy_to_aws_ecs.yml, and paste it into the textarea.
- Change the following values under the key labeled
env
:
env:
AWS_REGION: ...
CONTAINER_NAME: ...
ECR_REPOSITORY: ...
ECS_CLUSTER: ...
ECS_SERVICE: ...
ECS_TASK_DEFINITION: ...
Key | Description | Sample value |
---|---|---|
AWS_REGION | Region of your AWS ECS cluster. | us-west-2 |
CONTAINER_NAME | Set this to the name of the container in the containerDefinitions section of your task definition. | mage-data-production-container |
ECR_REPOSITORY | The name of the AWS ECR repository you created to store your Docker images. | mage-data |
ECS_CLUSTER | The name of your AWS ECS cluster. | mage-production-cluster |
ECS_SERVICE | The name of your AWS ECS service. | mage-production-ecs-service |
ECS_TASK_DEFINITION | Go to your AWS ECS task definition for the above service. Click on the JSON tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | some_path/ecs-task-definition.json |
- Click the button labeled Start commit in the top right corner.
- Click the button labeled Commit new file.
- Every time you merge a pull request into the master branch, this GitHub Action will run, building a Docker image using your GitHub code, then updating AWS ECS to use the new image with the updated code.
GCP
-
You’ll need to add these roles to the service account that you’ll be using to deploy Mage from GitHub:
- Artifact Registry Read
- Artifact Registry Writer
- Cloud Run Admin
- Service Account Token Creator
- Service Account User
-
In the field labeled Name, enter the value
GCP_CREDENTIALS
. -
In the field labeled Secret, enter the JSON string containing your GCP service account credentials. It should look something like this:
json { "type": "service_account", "project_id": "mage-123456", "private_key_id": "...", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", "client_email": "mage@mage-123456.iam.gserviceaccount.com", "client_id": "...", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mage%40mage-123456.iam.gserviceaccount.com" }
-
Click the button labeled Add secret to save.
-
Click on the tab labeled Actions.
-
On the left side, click the button labeled New workflow.
-
Find the link labeled
set up a workflow yourself
and click it. -
Copy the contents from the GitHub Action YAML file for GCP at templates/github_actions/build_and_deploy_to_gcp_cloud_run.yml, and paste it into the textarea.
-
Change the following values under the key labeled
env
:env: GAR_LOCATION: ... GOOGLE_CLOUD_RUN_SERVICE_NAME: ... IMAGE: ... PROJECT_ID: ... REPOSITORY: ...
Key Description Sample value GAR_LOCATION
Region that Mage is already deployed in. us-east1
GOOGLE_CLOUD_RUN_SERVICE_NAME
The name of your Google Cloud Run service. Go to the Google Cloud Run dashboard to find it. mage-data
IMAGE
The name of the Docker image you pushed to the above GCP Artifact Registry. mageai
PROJECT_ID
Project ID of where you launched Mage using Terraform. mage-123456
REPOSITORY
The name of your GCP Artifact Registry that is storing your Docker image. mageprod
-
Click the button labeled Start commit in the top right corner.
-
Click the button labeled Commit new file.
-
Everytime you merge a pull request into the master branch, this GitHub Action will run, building a Docker image using your GitHub code, then updating Google Cloud Run to use the new image with the updated code.
Azure
Coming soon…