Trigger from AWS events

Note

Event triggered pipelines only work when Mage is running in a cloud provider like AWS, GCP, or Azure.

​

TLDR

1. Create a CloudWatch event rule.
2. Add AWS credentials to environment variables for Mage to use.
3. Create an event trigger for a pipeline.

​

Create a CloudWatch event rule

The following steps will walk you through how to create an event rule for SQS.

1. Go to the AWS CloudWatch dashboard; e.g. https://us-west-2.console.aws.amazon.com/cloudwatch/home
2. Under the Events section, click the Rules link.
3. Click the button Create rule; it should take you to a page like this https://us-west-2.console.aws.amazon.com/cloudwatch/home?#rules:action=create
4. Under the Event Source section, select the option for Event Pattern.
5. In the dropdown menu next to Service Name, select any option; e.g. Simple Queue Service.
6. In the dropdown menu next to Event Type, select an option; e.g. All Events.
7. Under the Targets section, click the + Add target button.
8. In the 1st dropdown menu, select a service; e.g. SQS queue.
9. In the dropdown menu next to Queue, select an option.
10. In the Configure input radio selection, choose Matched event.
11. In the bottom right corner, click the button labeled Configure details.
12. Enter a name for this event rule; e.g. events_from_my_sqs_queue.
13. Check the status box labeled Enabled.
14. In the bottom right corner, click the button labeled Create rule.

​

Add AWS credentials to environment variables

If you’re using Mage in Docker, add the following environment variables when you run Mage:

• AIRBNB_AWS_ACCESS_KEY_ID
• AIRBNB_AWS_SECRET_ACCESS_KEY

Permissions

The AWS credentials need to have the following permissions: events:ListRules.

Follow these instructions to learn more how to add environment variables when running Docker. For example:

docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  -e AWS_ACCESS_KEY_ID=... \
  -e AWS_SECRET_ACCESS_KEY=... \
  /app/run_app.sh mage start [project_name]

If you’re not using Docker to run Mage, run the following commands:

export AIRBNB_AWS_ACCESS_KEY_ID=...
export AIRBNB_AWS_SECRET_ACCESS_KEY=...

​

Create an event trigger

1. Follow these instructions to create a trigger.
2. When you choose the trigger type, select Event.
3. Under the Events section, in the Provider dropdown menu, select the option AWS.
4. Under the Events section, in the Event dropdown menu, select the name of the event you created in the previous steps; e.g. events_from_my_sqs_queue.
5. At the top left corner, click the button Save changes.

​

Test pipeline trigger

1. Go to the AWS SQS dashboard; e.g. https://us-west-2.console.aws.amazon.com/sqs/v2/home
2. Click on a row for the queue that is referenced from the event you created in a previous step.
3. In the top right corner, click the button Send and receive messages.
4. In the input field labeled Message body, type test.
5. In the top right corner, click the button Send message.