Trigger from AWS events
Run a pipeline whenever an AWS event occurs.
TLDR
- Create a CloudWatch event rule.
- Add AWS credentials to environment variables for Mage to use.
- Create an event trigger for a pipeline.
Create a CloudWatch event rule
The following steps will walk you through how to create an event rule for SQS.
- Go to the AWS CloudWatch dashboard; e.g. https://us-west-2.console.aws.amazon.com/cloudwatch/home
- Under the Events section, click the Rules link.
- Click the button Create rule; it should take you to a page like this https://us-west-2.console.aws.amazon.com/cloudwatch/home?#rules:action=create
- Under the Event Source section, select the option for Event Pattern.
- In the dropdown menu next to Service Name, select any option; e.g.
Simple Queue Service
. - In the dropdown menu next to Event Type, select an option; e.g.
All Events
. - Under the Targets section, click the + Add target button.
- In the 1st dropdown menu, select a service; e.g.
SQS queue
. - In the dropdown menu next to Queue, select an option.
- In the Configure input radio selection, choose Matched event.
- In the bottom right corner, click the button labeled Configure details.
- Enter a name for this event rule; e.g.
events_from_my_sqs_queue
. - Check the status box labeled
Enabled
. - In the bottom right corner, click the button labeled Create rule.
Add AWS credentials to environment variables
If you’re using Mage in Docker, add the following environment variables when you run Mage:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
Permissions
The AWS credentials need to have the following permissions:
events:ListRules
.
Follow these instructions to learn more how to add environment variables when running Docker. For example:
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
-e AWS_ACCESS_KEY_ID=... \
-e AWS_SECRET_ACCESS_KEY=... \
/app/run_app.sh mage start [project_name]
If you’re not using Docker to run Mage, run the following commands:
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
Set up Lambda Function
For event trigger, Mage uses a Lambda Function to relay the event from CloudWatch to Mage.
If you deploy Mage to AWS with terraform scripts,
you can uncomment the code in the following places and run terraform apply
to create the Lambda Function:
- https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/lambda.tf#L1-L23
- https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/iam.tf#L29-L73
- https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/main.tf#L46-L47
- Add the
LAMBDA_FUNCTION_ARN
andLAMBDA_FUNCTION_NAME
environment variables to https://github.com/mage-ai/mage-ai-terraform-templates/blob/master/aws/env_vars.json{ "name": "LAMBDA_FUNCTION_ARN", "value": "${lambda_func_arn}" }, { "name": "LAMBDA_FUNCTION_NAME", "value": "${lambda_func_name}" },
If you don’t use the terraform script to deploy Mage, you can set up the Lambda Function with the following steps:
- Create a Lambda Function.
- Select
Python 3.9
as Runtime. - Use the same VPC and subnets of your Mage service.
- Create a security group. Edit your security group of the Mage instance to allow traffic from the Lambda Function’s security group.
- Select
- Set the environment variable
MAGE_API_HOST
in the Lambda Function. - Copy the code from the event_handler.py to your Lamdba Function.
- Test the connection by the executing Lambda Function with a test event json.
- Add environment variables
LAMBDA_FUNCTION_ARN
andLAMBDA_FUNCTION_NAME
to your Mage service.
Create an event trigger
- Follow these instructions to create a trigger.
- When you choose the trigger type, select
Event
. - Under the Events section, in the Provider dropdown menu, select the option
AWS
. - Under the Events section, in the Event dropdown menu, select the name of the event
you created in the previous steps; e.g.
events_from_my_sqs_queue
. - At the top left corner, click the button Save changes.
Test pipeline trigger
- Go to the AWS SQS dashboard; e.g. https://us-west-2.console.aws.amazon.com/sqs/v2/home
- Click on a row for the queue that is referenced from the event you created in a previous step.
- In the top right corner, click the button Send and receive messages.
- In the input field labeled Message body, type
test
. - In the top right corner, click the button Send message.
Was this page helpful?