AWS ECS executor

You can choose to launch separate AWS ECS tasks to executor blocks by specifying block executor_type to be ecs in pipeline’s metadata.yaml file.

There’re 2 ways to customize the ECS executor config,

1. Specify the ecs_config in project’s metadata.yaml file. Example config:

   ecs_config:
     cpu: 1024
     memory: 2048
   

   2. Add the executor_config at block level in pipeline’s metadata.yaml file. Example config:

   blocks:
   - uuid: example_data_loader
     type: data_loader
     upstream_blocks: []
     downstream_blocks: []
     executor_type: ecs
     executor_config:
       cpu: 1024
       memory: 2048
   

To run the whole pipeline in one ECS executor, you can set the executor_type at pipeline level and set run_pipeline_in_one_process to true. executor_config can also be set at pipeline level. Here is the example pipeline metadata.yaml:

blocks:
- ...
- ...
executor_type: ecs
run_pipeline_in_one_process: true
name: example_pipeline
...

​

Configurations

​

Example

ecs_config:
  cpu: 1024
  memory: 2048
  assign_public_ip: false
  enable_execute_command: true
  wait_timeout: 1200

​

IAM permissions

Required IAM permissions for using ECS executor:

[
  "ec2:DescribeNetworkInterfaces",
  "ecs:DescribeTasks",
  "ecs:ListTasks",
  "ecs:RunTask"
]

​

Resource management

Get started for free

A fully managed service, where we maintain the infrastructure, guarantee uptime, automatically scale your workloads to handle any volume of pipeline runs, automatically upgrade new versions of Mage Pro only features, monitor your production pipelines, and provide enterprise level support.