Use Mage Pro if you need to run your blocks in separate pods or process while developing in the code editor.
Running a block from the code editor, while developing a pipeline, won’t use this executor. Instead, the block will be executed using the same process that the code editor is running on.
Running a block from the code editor, while developing a pipeline, won’t use this executor. Instead, the block will be executed using the same process that the code editor is running on.
executor_type: k8s
) to run each block in its own Kubernetes pod.
Defaults in Mage Pro:
- Namespace:
default
- Job name format:
mage-data-prep-block-{block_run_id}
Basic Setup
To configure a pipeline block to use Kubernetes executor, you simply just need to update theexecutor_type
of the block to k8s
in pipeline’s metadata.yaml:
default
as the Kubernetes namespace. You can customize the namespace by setting the KUBE_NAMESPACE
environment variable.
Configuration Methods
You can configure Kubernetes Executor in three ways:1. Block-Level Configuration
Addexecutor_config
to a block in metadata.yaml
:
- Run certain blocks in the Kubernetes executor
- Override the k8s executor config for specific blocks
2. Project-Level Configuration (applies to all k8s executor blocks)
Setk8s_executor_config
in the project’s metadata.yaml:
- The Kubernetes job name is in this format:
mage-{job_name_prefix}-block-{block_run_id}
. The defaultjob_name_prefix
isdata-prep
. You can customize it in thek8s_executor_config
. You can interpolate the trigger name in thejob_name_prefix
field with the format{trigger_name}
.
-
GPU Support:
Make sure GPU device plugins are installed.
-
Custom container & job spec:
3. Full Kubernetes Job Template (maximum control)
Set theK8S_CONFIG_FILE
environment variable to the path of a YAML configuration file.
Example template:
Node Scheduling Configuration
Node Selector
Usenode_selector
to schedule pods on specific nodes based on labels:
Tolerations
Usetolerations
to allow pods to be scheduled on tainted nodes:
Affinity Rules
Useaffinity
for advanced scheduling rules:
Custom Scheduler
Use a custom scheduler for advanced scheduling logic:Multi-Container Pods
If your Mage deployment runs in a multi-container pod, set theMAGE_CONTAINER_NAME
environment variable to specify which container runs Mage:
Environment Variables Reference
Variable | Purpose | Default |
---|---|---|
KUBE_NAMESPACE | Namespace for job execution | default |
K8S_CONFIG_FILE | Path to full Kubernetes job config file | — |
MAGE_CONTAINER_NAME | Container to run Mage in multi-container pods | — |