> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mage.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Retrying block runs from a pipeline run
export const ProOnly = ({button = 'Get started for free', description = 'Try our fully managed solution to access this advanced feature.', source = 'documentation', title = 'Only in Mage Pro.'}) => ;
export const ProButton = ({href, label = 'Get started with Mage Pro for free', source = 'documentation'}) => ;
Requires version `0.8.76` or greater.
## Automatic retry
Mage supports configuring automatic retry for block runs in the following ways.
Automatic retry applies to standard block runs and data integration pipeline stream block runs.
When multiple retry configs are present, Mage merges them in this order:
project `metadata.yaml` `retry_config`, pipeline `metadata.yaml` `retry_config`, then block `retry_config`.
Later configs override earlier values for matching keys only when the later value is not blank or `null`.
A resolved retry config retries only when `retries` is greater than `0`. If `retries` is blank,
`null`, or not set at the pipeline or block level, Mage inherits the lower-precedence value. To turn
off inherited retries, set `retries: 0` on the pipeline or block. If `delay` or `max_delay` is blank
or `null`, Mage also inherits the lower-precedence value; if the resolved config still does not set
them, Mage uses `5` and `60` seconds.
### 1. Global retry config
The global `retry_config` is saved in the project's `metadata.yaml` and applies
to all block runs by default. Pipeline-level and block-level retry settings can
override the global values.
When retry settings are configured at multiple levels, Mage merges them from
least specific to most specific: global project defaults, then pipeline-level
settings, then block-level settings. For the same retry field, the most specific
configured value wins. Blank or `null` retry fields are ignored during this
merge, so a blank pipeline or block field inherits the next available value from
the less-specific config instead of clearing it.
In Mage Pro, go to **Settings > Workspace > Automatic retry** or open
`/settings/workspace/automatic-retry` to configure the global retry settings for
the project. Users with viewer access can open the page, but the retry controls
are read-only. Set the number of retries, the initial delay in seconds, the
maximum accumulated retry delay in seconds before scheduling the next attempt,
and whether to use exponential backoff. The maximum delay cap applies whether or
not exponential backoff is enabled. Leave a field blank to omit that value from
the saved project metadata, or set retries to `0` to explicitly disable global
automatic retries. Retry count, delay, and max delay must be non-negative
integers; retries can be up to `100`, and delay/max delay can be up to `86400`
seconds. `0` is accepted for numeric retry fields. If `exponential_backoff` is
omitted, Mage uses the default enabled backoff behavior.
You can also edit the project's `metadata.yaml` directly:
```yaml theme={"system"}
retry_config:
# Number of retry times
retries: 3
# Initial delay (in seconds) before retry. If exponential_backoff is true,
# the delay time is multiplied by 2 for the next retry
delay: 5
# Maximum accumulated retry delay before scheduling the next attempt
max_delay: 60
# Whether to use exponential backoff retry
exponential_backoff: true
```
### 2. Block level retry config
Add `retry_config` to the block config in pipeline's `metadata.yaml`. The block level `retry_config` will
override the global and pipeline-level `retry_config` for fields that have
configured values. Leave a block retry field blank or set it to `null` to inherit
that field from the pipeline or global config. Here is an example
```yaml theme={"system"}
blocks:
- uuid: example_data_loader
type: data_loader
upstream_blocks: []
downstream_blocks: []
retry_config:
retries: 2
delay: 5
max_delay: 10
exponential_backoff: false
...
```
### 3. Pipeline level `retry_config`
Add `retry_config` to the pipeline config in pipeline's `metadata.yaml`. The pipeline level `retry_config` will
override the global `retry_config` for fields that have configured values unless
a block sets its own retry value. Leave a pipeline retry field blank or set it to
`null` to inherit that field from the global config. Here is an example
```yaml theme={"system"}
blocks:
- ...
- ...
- ...
retry_config:
retries: 2
delay: 5
max_delay: 10
exponential_backoff: false
...
```
The retry logs show in the same block run log page.
## Retry incomplete blocks
* Supported pipeline types: Standard, Streaming, and Integration pipelines
* If a pipeline run fails, you can retry from the failed block
run instead of restarting the entire pipeline run. In Mage Pro, open the failed
run's actions menu from the pipeline's **Runs** table and select **Retry incomplete
blocks**. The separate **Retry pipeline run** action restarts the entire pipeline run.
For Streaming pipelines, **Retry incomplete blocks** reuses the existing pipeline
run, resets its incomplete block runs, and resumes it instead of creating a new run.
* You can also retry incomplete blocks from the individual pipeline run page
(`/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`).
* At the top of the individual pipeline runs page, there should be a red
`Retry incomplete blocks` button. Click on it, and the block runs should
retry from the failed block.
* For Integration pipelines, this broad retry keeps the existing behavior and
retries every incomplete stream in the run.
## Retry a specific integration stream
* Supported pipeline types: Integration pipelines
* If a data integration pipeline run has a failed stream, go to the individual
pipeline run page (`/pipelines/[pipeline_uuid]/runs/[pipeline_run_id]`).
* Select any block run row for the stream you want to retry. The selected row
can be completed if another block run for that stream failed.
* At the top of the page, click the `Retry` button with the stream name shown
next to it. Mage retries only the stream shown on the button and leaves other
streams unchanged.
## Retry from selected block
* Supported pipeline types: Standard and Integration pipelines
* If you want to retry block runs for a pipeline run starting from a specific
block (regardless of the block run's status), go to the same page for the individual
pipeline run.
* Select a block run row, and a blue `Retry from selected block` button should appear.
* Note that the `Retry from selected block` button will only appear if the pipeline
run is not currently running.
* For Integration pipelines, selecting a block run in a completed pipeline run keeps
the existing selected-block retry behavior. Selecting a stream in an incomplete
run uses the specific stream retry described above.
* If there is a failed block run and you try to retry from a selected block that is
downstream from that failed block, you may get `upstream_failed` block run errors.
### Accessing the individual pipeline runs page
1. Click on a pipeline from the main Pipelines Dashboard.
2. Click on the `Runs` section or on an individual trigger from the `Triggers` section.
3. Underneath the `Block runs` column of the pipeline runs table, there is a link that
says something like `See block runs (x)`. Click on that link to redirect to the
individual pipeline runs page, which lists the blocks runs specific to a pipeline run.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.