Backfilling pipelines

A backfill creates 1 or more pipeline runs for a pipeline. There are 2 types of backfills:

1. Date and time window
2. Custom code

​

Date and time window

Create 1 or more pipeline runs between 2 datetime values.

The datetime of each instance is used as the execution_date for the pipeline run.

For example, if the backfill has the following attributes:

Then the following pipeline runs will be created:

​

Custom code

The output of a backfill code will be used to generate the pipeline runs.

For example, if the backfill code has the following content:

backfill_data = []

for index, _ in range(3):
    backfill_data.append(dict(
        partition=index,
        power=5,
    ))

backfill_data

Then the following pipeline runs will be created: