You can choose to launch separate AWS ECS tasks to executor blocks by specifying block executor_type to be ecs in pipeline’s metadata.yaml file.

There’re 2 ways to customize the ECS executor config,

  1. Specify the ecs_config in project’s metadata.yaml file. Example config:

    ecs_config:
  cpu: 1024
  memory: 2048
    1. Add the executor_config at block level in pipeline’s metadata.yaml file. Example config:
    blocks:
- uuid: example_data_loader
  type: data_loader
  upstream_blocks: []
  downstream_blocks: []
  executor_type: ecs
  executor_config:
    cpu: 1024
    memory: 2048

To run the whole pipeline in one ECS executor, you can set the executor_type at pipeline level and set run_pipeline_in_one_process to true. executor_config can also be set at pipeline level. Here is the example pipeline metadata.yaml:

blocks:
- ...
- ...
executor_type: ecs
run_pipeline_in_one_process: true
name: example_pipeline
...

​
Configurations

Field nameDescriptionExample values
assign_public_ipWhether to assign public IP to the ECS task.true/false (default: true)
cpuThe CPU allocated to the ECS task.1024
enable_execute_commandWhether to enable execute command for debuggingtrue/false (default: false)
launch_typeThe launch type of the ECS task.FARGATE
memoryThe memory allocated to the ECS task.2048
tagsThe tags of the ECS task.[‘tag1’, ‘tag2’]
wait_timeoutThe maximum wait time for the ECS task (in seconds). The default wait timeout for the ECS task is 10 minutes. Setting to -1 will disable waiting.1200 (default: 600)

​
Example

ecs_config:
  cpu: 1024
  memory: 2048
  assign_public_ip: false
  enable_execute_command: true
  wait_timeout: 1200

​
IAM permissions

Required IAM permissions for using ECS executor:

[
  "ec2:DescribeNetworkInterfaces",
  "ecs:DescribeTasks",
  "ecs:ListTasks",
  "ecs:RunTask",
  "iam:PassRole"
]

​
Resource management

Get started for free


A fully managed service, where we maintain the infrastructure, guarantee uptime, automatically scale your workloads to handle any volume of pipeline runs, automatically upgrade new versions of Mage Pro only features, monitor your production pipelines, and provide enterprise level support.