Trigger from AWS events

​

TLDR

1. Create a CloudWatch event rule.
2. Add AWS credentials to environment variables for Mage to use.
3. Create an event trigger for a pipeline.

​

Create a CloudWatch event rule

The following steps will walk you through how to create an event rule for SQS.

1. Go to the AWS CloudWatch dashboard; e.g. https://us-west-2.console.aws.amazon.com/cloudwatch/home
2. Under the Events section, click the Rules link.
3. Click the button Create rule; it should take you to a page like this https://us-west-2.console.aws.amazon.com/cloudwatch/home?#rules:action=create
4. Under the Event Source section, select the option for Event Pattern.
5. In the dropdown menu next to Service Name, select any option; e.g. Simple Queue Service.
6. In the dropdown menu next to Event Type, select an option; e.g. All Events.
7. Under the Targets section, click the + Add target button.
8. In the 1st dropdown menu, select a service; e.g. SQS queue.
9. In the dropdown menu next to Queue, select an option.
10. In the Configure input radio selection, choose Matched event.
11. In the bottom right corner, click the button labeled Configure details.
12. Enter a name for this event rule; e.g. events_from_my_sqs_queue.
13. Check the status box labeled Enabled.
14. In the bottom right corner, click the button labeled Create rule.

​

Add AWS credentials to environment variables

If you’re using Mage in Docker, add the following environment variables when you run Mage:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

Permissions

The AWS credentials need to have the following permissions: events:ListRules.

Follow these instructions to learn more how to add environment variables when running Docker. For example:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  -e AWS_ACCESS_KEY_ID=... \
  -e AWS_SECRET_ACCESS_KEY=... \
  /app/run_app.sh mage start [project_name]

If you’re not using Docker to run Mage, run the following commands:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

​

Set up Lambda Function

For event trigger, Mage uses a Lambda Function to relay the event from CloudWatch to Mage.

If you deploy Mage to AWS with terraform scripts, you can uncomment the code in the following places and run terraform apply to create the Lambda Function:

• https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/lambda.tf#L1-L23
• https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/iam.tf#L29-L73
• https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/main.tf#L46-L47
• Add the LAMBDA_FUNCTION_ARN and LAMBDA_FUNCTION_NAME environment variables to https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/env_vars.json

  {
      "name": "LAMBDA_FUNCTION_ARN",
      "value": "${lambda_func_arn}"
  },
  {
      "name": "LAMBDA_FUNCTION_NAME",
      "value": "${lambda_func_name}"
  },
  

If you don’t use the terraform script to deploy Mage, you can set up the Lambda Function with the following steps:

1. Create a Lambda Function.
   1. Select Python 3.9 as Runtime.
   2. Use the same VPC and subnets of your Mage service.
   3. Create a security group. Edit your security group of the Mage instance to allow traffic from the Lambda Function’s security group.
2. Set the environment variable MAGE_API_HOST in the Lambda Function.
3. Copy the code from the event_handler.py to your Lamdba Function.
4. Test the connection by the executing Lambda Function with a test event json.
5. Add environment variables LAMBDA_FUNCTION_ARN and LAMBDA_FUNCTION_NAME to your Mage service.

​

Create an event trigger

1. Follow these instructions to create a trigger.
2. When you choose the trigger type, select Event.
3. Under the Events section, in the Provider dropdown menu, select the option AWS.
4. Under the Events section, in the Event dropdown menu, select the name of the event you created in the previous steps; e.g. events_from_my_sqs_queue.
5. At the top left corner, click the button Save changes.

​

Test pipeline trigger

1. Go to the AWS SQS dashboard; e.g. https://us-west-2.console.aws.amazon.com/sqs/v2/home
2. Click on a row for the queue that is referenced from the event you created in a previous step.
3. In the top right corner, click the button Send and receive messages.
4. In the input field labeled Message body, type test.
5. In the top right corner, click the button Send message.