Kubernetes executor

If your Mage app is running in a Kubernetes cluster, you can execute the blocks in separate Kubernetes pods with Kubernetes executor.

To configure a pipeline block to use Kubernetes executor, you simply just need to update the executor_type of the block to k8s in pipeline’s metadata.yaml:

blocks:
- uuid: example_data_loader
  type: data_loader
  upstream_blocks: []
  downstream_blocks: []
  executor_type: k8s
  ...

By default, Mage uses default as the Kubernetes namespace. You can customize the namespace by setting the KUBE_NAMESPACE environment variable.

There’re three ways to customize the Kubernetes executor config:

1. Add the executor_config at block level in pipeline’s metadata.yaml file. Example config:

   blocks:
   - uuid: example_data_loader
     type: data_loader
     upstream_blocks: []
     downstream_blocks: []
     executor_type: k8s
     executor_config:
       namespace: default
       resource_limits:
         cpu: 1000m
         memory: 2048Mi
       resource_requests:
         cpu: 500m
         memory: 1024Mi
   

2. Add the k8s_executor_config to project’s metadata.yaml. This k8s_executor_config will apply to all the blocks that use k8s executor in this project. Example config:

   k8s_executor_config:
     job_name_prefix: data-prep
     namespace: default
     resource_limits:
       cpu: 1000m
       memory: 2048Mi
     resource_requests:
       cpu: 500m
       memory: 1024Mi
     service_account_name: default
   

• The kubernetes job name is in this format: mage-{job_name_prefix}-block-{block_run_id}. The default job_name_prefix is data-prep. You can customize it in the k8s executor config. You can interpolate the trigger name in the job_name_prefix field with the format {trigger_name}
• If you want to use GPU resource in your k8s executor, you can configure the GPU resource in the k8s_executor_config like

  k8s_executor_config:
    resource_limits:
      gpu-vendor.example/example-gpu: 1 # requesting 1 GPU
  

  Please make sure the GPU driver is installed and run on your nodes to use the GPUs.
• To further customize the container config of the kubernetes executor, you can specify the container_config or job in the k8s executor config. Here is the example:

  k8s_executor_config:
    container_config:
      image: mageai/mageai:0.9.7
      env:
      - name: USER_CODE_PATH
        value: /home/src/k8s_project
    job:
      active_deadline_seconds: 120
      backoff_limit: 3
      ttl_seconds_after_finished: 86400
  

3. You can configure the job template by using the K8S_CONFIG_FILE environment variable, which should point to the configuration file. Here is the format for the Kubernetes configuration template:

# Kubernetes Configuration Template
metadata:
  annotations:
    application: "mage"
    composant: "executor"
  labels:
    application: "mage"
    type: "spark"
  namespace: "default"
pod:
  service_account_name: ""
  image_pull_secrets: "secret"
  volumes:
  - name: data-pvc
    persistent_volume_claim:
      claim_name: pvc-name
container:
  name: "mage-data"
  env:
    - name: "KUBE_NAMESPACE"
      value: "default"
    - name: "secret_key"
      value: "somesecret"
  image: "mageai/mageai:latest"
  image_pull_policy: "IfNotPresent"
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
    requests:
      cpu: "0.1"
      memory: "0.5Gi"
  volume_mounts:
    - mount_path: "/tmp/data"
      name: "data-pvc"

NB: When deploying Mage in a multi-container pod, you need to specify the environment variable MAGE_CONTAINER_NAME. If this variable is not set, Mage will default to using the first container in the pod. To specify the Mage container, you can use:

    env:
      - name: MAGE_CONTAINER_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name