Deploy to AWS with Terraform

​

Pre-requisites

​

Setup

Download Mage maintained Terraform scripts.

​

Permissions

In order to create all the required resources in AWS, your IAM user account must have the following permissions.

In order to terminate and remove all the resources created from Terraform, you need the following permissions.

​

Deploy to ECS cluster

Use Terraform scripts in aws folder to provision resources for AWS ECS cluster.

​

Terraform plan

You can run the following command to see all the resources that will be created by Terraform:

cd aws
terraform plan

By default, here are the resources that will be created.

​

1. Environment variables

If you don’t have the AWS CLI installed, you’ll need to create this file: ~/.aws/credentials.

In that file, add the following values:

[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX

Alternatively, you can install the AWS CLI by following Amazon’s instructions.

​

2. Customize Terraform settings

Application name (optional)

In the file ./aws/variables.tf, you can change the default application name that will appear in AWS ECS:

variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}

Docker image (optional)

In the file ./aws/variables.tf, you can change the default Docker image:

variable "docker_image" {
  description = "Docker image url used in ECS task."
  default     = "mageai/mageai:latest"
}

1. docker build --platform linux/amd64 -tag [image_name]:latest .
2. docker tag [image_name]:latest [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest
3. docker push [registry_uuid].dkr.ecr.[region].amazonaws.com/[image_name]:latest

Region (optional)

In the file ./aws/variables.tf, you can change the region:

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "us-west-2"
}

Availability zones (optional)

In the file ./aws/variables.tf, you must change the availability zones to match your region from above:

variable "availability_zones" {
  description = "List of availability zones"
  default     = ["us-west-2a", "us-west-2b"]
}

More

Other variables defined in ./aws/variables.tf can also be customized to your needs.

​

3. Configurable environment variables

In the ./aws/env_vars.json file, you can edit the following variables, which are used by the tool while running in the cloud:

Change the value of the variables with the following names to match the actual values you want the tool to use while running in the cloud:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

These variable values are used by the tool to retrieve AWS resources like CloudWatch events, etc.

Other environment variables

You can add any environment variable you want in this file. These will be set on the running container.

Values

The values in your env_vars.json must be strings. Wrap each value in double quotes.

​

4. Deploy

Using CLI

1. Change directory into scripts folder:

   cd aws
   

2. Initialize Terraform:

   terraform init
   

   • If you run into errors like the following:

     │ Error: Failed to install provider
     │
     │ Error while installing hashicorp/template v2.2.0: the local package for registry.terraform.io/hashicorp/template 2.2.0 doesn't match any of the checksums previously recorded in the dependency lock file (this might be because the available checksums are for packages targeting different
     │ platforms); for more information: https://www.terraform.io/language/provider-checksum-verification
     

     then run the following commands to resolve this:

     brew install kreuzwerker/taps/m1-terraform-provider-helper
     m1-terraform-provider-helper activate
     m1-terraform-provider-helper install hashicorp/template -v v2.2.0
     rm .terraform.lock.hcl
     terraform init --upgrade
     

3. Deploy:

   terraform apply
   

Once it’s finished deploying, you can access Mage in your browser.

1. Open your EC2 dashboard.
2. View all load balancers.
3. Click on the load balancer with the name mage-data-prep in it (if you changed the app name, then find the load balancer with that app name).
4. Find the public DNS name, copy that, and paste it in your browser.

Using Docker

From the root directory of Mage, run the following commands:

1. Initialize Terraform:

docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
  hashicorp/terraform:latest init

2. Deploy:

docker run -i -t -v $(pwd):/mage --workdir="/absolute_path_to/mage-ai-terraform-templates/aws" \
  --env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  --env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  hashicorp/terraform:latest apply

​

Errors

Page isn’t loading

If you run into connection issues, check to see if your IP is whitelisted in the appropriate security group.

503 Forbidden

Check to see if the service task in your ECS cluster is running or if it stopped.

503 typically means that the service task isn’t running and that can be caused by a variety of things.

Open the service task that stopped running and click on the “Logs” tab to see what issues occurred.

exec /bin/sh: exec format error

If your cluster is running and the service launches a task that eventually stops, check the logs for the stopped task. If the logs contain the error above, try re-building your image with the platform flag --platform linux/amd64.

​

Updating Mage versions

After Mage is running in your cloud, you can update the Mage Docker image version by running the following command in your CLI tool:

aws ecs update-service --cluster [cluster_name] --service [service_name] --force-new-deployment

cluster_name

This is the name of your AWS ECS cluster. Go to the AWS ECS dashboard to view the clusters and find the cluster with the name that contains the app_name from variables.tf.

service_name

This is the name of your AWS ECS service running in your AWS ECS cluster with the name identified from above.

​

Misc

​

Security

To enable other IP addresses access to Mage, open the security group named mage-data-prep-sg to whitelist IPs.

Add a new inbound rule for HTTP port 80 and use your IP address.

​

HTTPS enabling

Terraform creates a load balancer with a HTTP listener by default.

To enable HTTPS for your cloud app, you need to have SSL certificates. You can create a SSL certificate in AWS Certificate Manager.

After you get a certificate, add a HTTPS listener in your load balancer group.

1. Open your EC2 dashboard.
2. View all load balancers.
3. Click on the load balancer with the name mage-data-prep in it (if you changed the app name, then find the load balancer with that app name).
4. Click “Listeners and rules” tab.
5. Click “Add listener” button
6. For “Protocol” and “Port”, choose HTTPS and keep the default port or enter a different port.
7. In “Default actions”, select “Forward to target groups” and forward to the target group that contains your app name (mage-data-prep-production-tg by default)
8. In “Security Policy”, keep the default security policy.
9. For “Default SSL certificate”, do one of the following:
   1. If you created or imported a certificate using AWS Certificate Manager, choose From ACM and choose the certificate.
   2. If you uploaded a certificate using IAM, choose From IAM and choose the certificate.
10. Click “Add” to create the HTTPS listener.
11. Edit your HTTP listener, change the default action to redirect traffic to your HTTPS listener.

The detailed instructions can be found in: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html

​

Terminate all resources

If you want to delete all resources created from deploying, run the following command:

terraform destroy

A sample output could look like this:

Destroy complete! Resources: 10 destroyed.

​

Authentication with ECS task execution role

Mage creates an ECS task execution role with the format {app_name}-execution-task-role. You can grant the ECS task permissions to access other AWS services by attaching IAM policies to this ECS task execution role.

For example, if you want to list the AWS events that can be used for event triggers, you can attach IAM Policy CloudWatchEventsReadOnlyAccess to your ECS task execution role.

​

Deploy to EKS cluster

Use Terraform scripts in aws-eks folder to provision resources for AWS EKS cluster.

You can run the following command to see all the resources that will be created by Terraform:

cd aws-eks
terraform plan

Similar with deploying to ECS cluster, you need to set up the environment variables first, and you can customize the Terraform settings in ./aws-eks/variables.tf file.

Then, follow the steps below to set up EKS cluster and deploy Mage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
# needed for EFS CSI driver dynamic provisioning
parameters:
  provisioningMode: efs-ap
  fileSystemId: <EFS_VOLUME_ID>
  directoryPerms: "700"