> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mage.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Management

export const ProOnly = ({button = 'Get started for free', description = 'Try our fully managed solution to access this advanced feature.', source = 'documentation', title = 'Only in Mage Pro.'}) => <a href={`https://cloud.mage.ai/sign-up?source=${source}`} className="block my-4 px-5 py-4 overflow-hidden rounded-xl flex gap-3 border border-emerald-500/20 bg-emerald-50/50 dark:border-emerald-500/30 dark:bg-emerald-500/10" target="_blank">
    <div style={{
  display: 'flex',
  alignItems: 'center',
  width: '100%'
}}>
      <div className="text-sm prose min-w-0 text-emerald-900 dark:text-emerald-200" style={{
  flex: 1
}}>
        {title}
        <p className="normal">{description}</p>
      </div>

      <div> </div>

      <div>
        <ProButton label={button} href={`https://cloud.mage.ai/sign-up?source=${source}`} />
      </div>
    </div>
  </a>;

export const ProButton = ({href, label = 'Get started with Mage Pro for free', source = 'documentation'}) => <div style={{
  height: 32,
  position: 'relative'
}}>
    <a target="_blank" className="group px-4 py-1.5 relative inline-flex items-center text-sm font-medium rounded-full" href={href ?? `https://cloud.mage.ai/sign-up?source=${source}`}>
      <span className="absolute inset-0 bg-primary-dark dark:bg-primary-light/10 border-primary-light/30 rounded-full dark:border group-hover:opacity-[0.9] dark:group-hover:border-primary-light/60">
      </span>

      <div className="mr-0.5 space-x-2.5 flex items-center">
        <span class="z-10 text-white dark:text-primary-light">
          {label}
        </span>

        <svg width="3" height="24" viewBox="0 -9 3 24" class="h-5 rotate-0 overflow-visible text-white/90 dark:text-primary-light">
          <path d="M0 0L3 3L0 6" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"></path>
        </svg>
      </div>
    </a>
  </div>;

<Frame>
  <img alt="Mage orchestration" src="https://mage-ai.github.io/assets/orchestration-overview.gif" />
</Frame>

## Setup

If you haven’t setup a project before, check out the
[setup guide](/getting-started/setup) before starting.

***

## Pipelines

<sub>`http://localhost:3000/pipelines`</sub>

This page will show all the pipelines in your project.

> Core abstraction: [Pipeline](/design/core-abstractions#pipeline)
>
> A pipeline contains references to all the blocks of code you want to run,
> charts for visualizing data, and organizes the dependency between each block
> of code.

![Pipelines](https://mage-ai.github.io/assets/pipelines/pipelines-index.jpg)

Learn more about
[**projects and pipelines here**](/design/core-abstractions#project).

From this page, you can also create a new pipeline by clicking the

**`[+ New pipeline]`** button.

#### Creating new pipeline

Creating a new pipeline will take you to the **Pipeline edit page**; a
notebook-like experience for adding blocks, creating dependencies between
blocks, testing code, and visualizing data with charts.

<sub>
  Learn more about the [Notebook for building data
  pipelines](/about/features#notebook-for-building-data-pipelines)
</sub>

***

## Pipeline runs

<sub>
  [`http://localhost:3000/pipeline-runs`](http://localhost:3000/pipeline-runs)
</sub>

View all the runs for every pipeline in your current project.

> Core abstraction: [Run](/design/core-abstractions#run)
>
> A run record stores information about when it was started, its status, when it
> was completed, any runtime variables used in the execution of the pipeline or
> block, etc.

![Pipeline runs](https://mage-ai.github.io/assets/pipelines/pipeline-runs.jpg)

***

## Pipeline detail

<sub>`http://localhost:3000/pipelines/[uuid]`</sub>

This page contains all the information and history for a single pipeline:

1. [Triggers](/design/core-abstractions#trigger)
2. [Runs](/design/core-abstractions#run)
3. [Logs](/design/core-abstractions#log)

***

## Triggers

<sub>
  [`http://localhost:3000/pipelines/example_pipeline/triggers`](http://localhost:3000/pipelines/example_pipeline/triggers)
</sub>

This page shows all the active and inactive triggers for a single pipeline.

> Core abstraction: [Trigger](/design/core-abstractions#trigger)
>
> A trigger is a set of instructions that determine when or how a pipeline
> should run.

![Pipeline detail](https://mage-ai.github.io/assets/pipelines/pipeline-detail.jpg)

***

#### Create trigger

<sub>`http://localhost:3000/pipelines/[uuid]/triggers/[id]/edit`</sub>

Create a new trigger for this pipeline by clicking the **`[+ Create]`** button
near the top of the page.

You can configure the trigger to run the pipeline on a schedule, when an event
occurs, or when an API is called.

> Core abstraction: [Schedule](/design/core-abstractions#schedule)
>
> A schedule type trigger will instruct the pipeline to run after a start date
> and on a set interval.

***

> Core abstraction: [Event](/design/core-abstractions#event)
>
> An event type trigger will instruct the pipeline to run whenever a specific
> event occurs.

***

> Core abstraction: [API](/design/core-abstractions#api)
>
> An API trigger will instruct the pipeline to run whenever a specific API
> is called.

![Trigger create](https://mage-ai.github.io/assets/pipelines/trigger-create.jpg)

<sub>
  <i>Example page for creating a schedule type trigger.</i>
</sub>

***

#### Trigger detail

On this page, you can start or pause the trigger. Starting the trigger will make
it active. Pausing the trigger will prevent it from running the pipeline.

<Note>
  If you have other triggers for this pipeline, pausing 1 trigger may not stop
  the pipeline from running since other triggers can also run the pipeline.
</Note>

***

![Trigger detail](https://mage-ai.github.io/assets/pipelines/trigger-detail.jpg)

You can also edit the trigger after creating it by clicking the

**`[Edit trigger]`** button.

***

## Runs

<sub>
  [`http://localhost:3000/pipelines/example_pipeline/runs`](http://localhost:3000/pipelines/example_pipeline/runs)
</sub>

View the pipeline runs and block runs for a pipeline.

> Core abstraction: [Run](/design/core-abstractions#run)
>
> A run record stores information about when it was started, its status, when it
> was completed, any runtime variables used in the execution of the pipeline or
> block, etc.

![Pipeline detail runs](https://mage-ai.github.io/assets/pipelines/pipeline-detail-runs.jpg)

#### Retry run

*WIP*

***

## Logs

<sub>
  [`http://localhost:3000/pipelines/example_pipeline/logs`](http://localhost:3000/pipelines/example_pipeline/logs)
</sub>

Browse all logs for a pipeline. You can search and filter logs by log level,
block type, block UUID, and more.

> Core abstraction: [**Log**](/design/core-abstractions#log)
>
> A log is a file that contains system output information.

![Pipeline detail logs](https://mage-ai.github.io/assets/pipelines/pipeline-detail-logs.jpg)

***

## Backfill

*WIP*

***

## Monitor

You can monitor many metrics for each of your pipelines and blocks. Soon, you’ll
be able to monitor aggregate metrics across all pipelines and blocks.

<img alt="Monitoring" src="https://mage-ai.github.io/assets/monitoring.jpg" />

Read more [here](/development/observability/monitoring).

## Concurrency

You can limit the concurrency of the block execution to reduce resource consumption.

### Global concurrency

You can configure the maximum number of concurrent block runs in project’s `metadata.yaml` via `queue_config`.

```yaml theme={"system"}
queue_config:
  concurrency: 100
```

The default value of concurrency is 20.

### Global concurrency by block uuid

<ProOnly source="concurrency" />

You can configure the global block run concurrency by block uuid in project's `metadata.yaml` via `concurrency_config`.

**Example**

```yaml theme={"system"}
concurrency_config:
  block_run_limit_per_block_uuid:
    block_uuid_1: 1
    block_uuid_2: 5
```

In the example above:

* `block_uuid_1` can run at most 1 concurrent execution.
* `block_uuid_2` can run up to 5 concurrent executions.

**Limitations**

This feature is not supported in the following scenarios:

* Replica blocks: Each replica is assigned a unique UUID for block run, so concurrency limits are not enforced across replicas.
* Dynamic blocks: Each dynamic child block has a unique UUID for block run, preventing concurrency limits from applying across them.
* Single-process mode: When `run_pipeline_in_one_process` is set to `true` in the pipeline’s `metadata.yaml`, global concurrency is not enforced.

### Pipeline level concurrency

You can edit the `concurrency_config` in each pipeline's metadata.yaml file to enforce pipeline level concurrency.
Here is the example:

```yaml theme={"system"}
concurrency_config:
  block_run_limit: 5
  pipeline_run_limit: 3
```

The default value of `block_run_limit` and `pipeline_run_limit` can be set via environment variables: `CONCURRENCY_CONFIG_BLOCK_RUN_LIMIT` and `CONCURRENCY_CONFIG_PIPELINE_RUN_LIMIT`

* `block_run_limit`: limit the concurrent blocks runs in one pipeline run.
* `pipeline_run_limit`: limit the concurrent pipeline runs in one pipeline trigger.
* `pipeline_run_limit_all_triggers`: limit the concurrent pipeline runs across all triggers in a pipeline.
* `on_pipeline_run_limit_reached`: choose whether to `wait` or `skip` when the pipeline run limit is reached.

## Variable storage

Mage automatically persists the output of block runs on disk. You can specify the path or the storage for block output variables
in the following ways.

* Specify the data directory path via `MAGE_DATA_DIR` environment variable. If you use Mage docker image, this environment
  variable is set to `/home/src/mage_data` by default.
* If the `MAGE_DATA_DIR` environment variable is not set, you can set the `variables_dir` path in project's metadata.yaml. Here is
  an example:
  ```yaml theme={"system"}
  variables_dir: /home/src/mage_data
  ```
* You can also use an external storage to store the block output variables by specifying the `remote_variables_dir` path in project's
  metadata.yaml.
  * AWS S3 storage:
    ```yaml theme={"system"}
    variables_dir: /home/src/mage_data
    remote_variables_dir: s3://bucket/path_prefix
    ```
  * Google Cloud Storage:
    ```yaml theme={"system"}
    variables_dir: /home/src/mage_data
    remote_variables_dir: gs://bucket/path_prefix
    ```
    When using GCS for your remote variables directory, if you run into a "Your default credentials were not found"
    error, you may need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of your
    `application_default_credentials.json` file with your Google Cloud credentials and the `GOOGLE_CLOUD_PROJECT`
    environment variable to your Google Cloud project ID.

## Variable retention

If you want to clean up the old variables in your variable storage, you can set the `variables_retention_period`
config in project's metadata.yaml. The valid period should end with "d", "h", or "w".

Example config:

```yaml theme={"system"}
variables_retention_period: 30d
```

After configuring `variables_retention_period` in project's metadata.yaml, you can run the following command
to clean up old variables:

```bash theme={"system"}
mage clean-cached-variables [project_path]
```

## Cache block output in memory

By default, Mage persists block output on disk. In pipeline's metadata.yaml, you have the option to configure the pipeline to cache
the block output in memory instead of persisting the block output on disk. The feature is only supported in standard batch
pipeline (without dynamic blocks) for now.

Example config:

```yaml theme={"system"}
cache_block_output_in_memory: true
run_pipeline_in_one_process: true
```

***

## Operation history

Keeps track of recently viewed pipelines so you can easily navigate back to them
in the Pipelines Dashboard (`/pipelines`). Adds a `RECENTLY VIEWED` tab to the Pipelines
Dashboard that lists these pipelines.