Deploy to AWS with Terraform
Pre-requisites
Setup
Download Mage maintained Terraform scripts.
Permissions
In order to create all the required resources in AWS, your IAM user account must have the following permissions.
In order to terminate and remove all the resources created from Terraform, you need the following permissions.
Deploy to ECS cluster
Use Terraform scripts in aws folder to provision resources for AWS ECS cluster.
Terraform plan
You can run the following command to see all the resources that will be created by Terraform:
cd aws
terraform plan
By default, here are the resources that will be created.
1. Environment variables
If you don’t have the AWS CLI installed, you’ll need to create this file:
~/.aws/credentials
.
In that file, add the following values:
[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX
Alternatively, you can install the AWS CLI by following Amazon’s instructions.
2. Customize Terraform settings
Application name (optional)
In the file ./aws/variables.tf, you can change the default application name that will appear in AWS ECS:
variable "app_name" {
type = string
description = "Application Name"
default = "mage-data-prep"
}
Docker image (optional)
In the file ./aws/variables.tf, you can change the default Docker image:
variable "docker_image" {
description = "Docker image url used in ECS task."
default = "mageai/mageai:latest"
}
Custom Docker images
If you previously tagged a Docker image you built when following this CI/CD guide, you must push that locally tagged Docker image to Amazon Elastic Container Registry (ECR) before deploying using Terraform. Read the AWS documentation to learn more.
docker build --platform linux/amd64 -tag [image_name]:latest .
docker tag [image_name]:latest [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest
docker push [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest
Region (optional)
In the file ./aws/variables.tf, you can change the region:
variable "aws_region" {
type = string
description = "AWS Region"
default = "us-west-2"
}
Availability zones (optional)
In the file ./aws/variables.tf, you must change the availability zones to match your region from above:
variable "availability_zones" {
description = "List of availability zones"
default = ["us-west-2a", "us-west-2b"]
}
More
Other variables defined in ./aws/variables.tf can also be customized to your needs.
3. Configurable environment variables
In the
./aws/env_vars.json
file, you can edit the following variables, which are used by the tool while
running in the cloud:
Change the value of the variables with the following names to match the actual values you want the tool to use while running in the cloud:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
These variable values are used by the tool to retrieve AWS resources like CloudWatch events, etc.
Other environment variables
You can add any environment variable you want in this file. These will be set on the running container.
Values
The values in your
env_vars.json
must be strings. Wrap each value in double quotes.
4. Deploy
Using CLI
- Change directory into scripts folder:
cd aws
- Initialize Terraform:
terraform init
- If you run into errors like the following:
then run the following commands to resolve this:
│ Error: Failed to install provider │ │ Error while installing hashicorp/template v2.2.0: the local package for registry.terraform.io/hashicorp/template 2.2.0 doesn't match any of the checksums previously recorded in the dependency lock file (this might be because the available checksums are for packages targeting different │ platforms); for more information: https://www.terraform.io/language/provider-checksum-verification
brew install kreuzwerker/taps/m1-terraform-provider-helper m1-terraform-provider-helper activate m1-terraform-provider-helper install hashicorp/template -v v2.2.0 rm .terraform.lock.hcl terraform init --upgrade
- If you run into errors like the following:
- Deploy:
terraform apply
Once it’s finished deploying, you can access Mage in your browser.
- Open your EC2 dashboard.
- View all load balancers.
- Click on the load balancer with the name
mage-data-prep
in it (if you changed the app name, then find the load balancer with that app name). - Find the public DNS name, copy that, and paste it in your browser.
Using Docker
From the root directory of Mage, run the following commands:
- Initialize Terraform:
docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
hashicorp/terraform:latest init
- Deploy:
docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
--env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
--env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
hashicorp/terraform:latest apply
Errors
Page isn’t loading
If you run into connection issues, check to see if your IP is whitelisted in the appropriate security group.
503 Forbidden
Check to see if the service task in your ECS cluster is running or if it stopped.
503 typically means that the service task isn’t running and that can be caused by a variety of things.
Open the service task that stopped running and click on the “Logs” tab to see what issues occurred.
exec /bin/sh: exec format error
If your cluster is running and the service launches a task that eventually
stops, check the logs for the stopped task. If the logs contain the error above,
try re-building your image with the platform flag --platform linux/amd64
.
Updating Mage versions
After Mage is running in your cloud, you can update the Mage Docker image version by running the following command in your CLI tool:
aws ecs update-service --cluster [cluster_name] --service [service_name] --force-new-deployment
cluster_name
This is the name of your AWS ECS cluster. Go to the AWS ECS dashboard to view the clusters and find the cluster with the name that contains the
app_name
fromvariables.tf
.
service_name
This is the name of your AWS ECS service running in your AWS ECS cluster with the name identified from above.
Misc
Security
To enable other IP addresses access to Mage, open the security group named
mage-data-prep-sg
to whitelist IPs.
Add a new inbound rule for HTTP port 80 and use your IP address.
HTTPS enabling
Terraform creates a load balancer with a HTTP listener by default.
To enable HTTPS for your cloud app, you need to have SSL certificates. You can create a SSL certificate in AWS Certificate Manager.
After you get a certificate, add a HTTPS listener in your load balancer group.
- Open your EC2 dashboard.
- View all load balancers.
- Click on the load balancer with the name
mage-data-prep
in it (if you changed the app name, then find the load balancer with that app name). - Click “Listners” tab.
- Click “Add listener” button
- For “Protocol” and “Port”, choose HTTPS and keep the default port or enter a different port.
- In “Default actions”, select “Forward” and forward to the target group that
contains your app name (
mage-data-prep-production-tg
by default) - In “Security Policy”, keep the default security policy.
- For “Default SSL certificate”, do one of the following:
- If you created or imported a certificate using AWS Certificate Manager, choose From ACM and choose the certificate.
- If you uploaded a certificate using IAM, choose From IAM and choose the certificate.
- Click “Add” to create the HTTPS listener.
- Edit your HTTP listener, change the default action to redirect traffic to your HTTPS listener.
The detailed instructions can be found in: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html
Terminate all resources
If you want to delete all resources created from deploying, run the following command:
terraform destroy
A sample output could look like this:
Destroy complete! Resources: 10 destroyed.
Authentication with ECS task execution role
Mage creates an ECS task execution role with the format {app_name}-execution-task-role
.
You can grant the ECS task permissions to access other AWS services by attaching IAM policies to this ECS task execution role.
For example, if you want to list the AWS events that can be used for event triggers, you can attach IAM Policy CloudWatchEventsReadOnlyAccess
to your ECS task execution role.
Deploy to EKS cluster
Use Terraform scripts in aws-eks folder to provision resources for AWS EKS cluster.
You can run the following command to see all the resources that will be created by Terraform:
cd aws-eks
terraform plan
Similar with deploying to ECS cluster, you need to set up the environment variables first, and you can customize the Terraform settings in ./aws-eks/variables.tf file.
Then, follow the steps below to set up EKS cluster and deploy Mage.
Apply terraform
Run terraform apply
to provision the EKS cluster
Configure kubectl
Configure kubectl
with command aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)
Configure PVC
If you are planning to use workspaces in EFS, you will need to add additional configurations to the StorageClass.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
# needed for EFS CSI driver dynamic provisioning
parameters:
provisioningMode: efs-ap
fileSystemId: <EFS_VOLUME_ID>
directoryPerms: "700"
Update the EFS_VOLUME_ID
in kube/efs-pvc.yaml to the efs id in terraform output
Install EFS CSI Driver
For workspaces support, you may need to install the EFS CSI driver add-on directly from the AWS console. Make sure to use a driver version that supports dynamic provisioning.
Run command kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"
to install efs csi driver
Deploy app to EKS
Run kubectl apply -f kube/
to deploy app or use Helm to deploy Mage.
Whitelist IP in EKS node's security group
Access Mage on EKS
Get the node IP via command kubectl get nodes -o wide
and access Mage with http://[ip]:{node_port}
Was this page helpful?