Backfilling pipelines

A backfill creates 1 or more pipeline runs for a pipeline. There are 2 types of backfills:

Date and time window
Custom code

Date and time window

Create 1 or more pipeline runs between 2 datetime values. The datetime of each instance is used as the execution_date for the pipeline run. For example, if the backfill has the following attributes:

Attribute	Value
Start datetime	`2023-01-01T03:00:00`
End datetime	`2023-01-05T03:00:00`
Interval type	`day`
Interval units	`2`

Then the following pipeline runs will be created:

`id`	`execution_date`	`ds`	`hr`
1	`2023-01-01T03:00:00`	`2023-01-01`	`03`
2	`2023-01-03T03:00:00`	`2023-01-03`	`03`

Custom code

The output of a backfill code will be used to generate the pipeline runs. For example, if the backfill code has the following content:

backfill_data = []

for index, _ in range(3):
    backfill_data.append(dict(
        partition=index,
        power=5,
    ))

backfill_data

Then the following pipeline runs will be created:

`id`	`execution_date`	`ds`	`hr`	`partition`	`power`
1	`2023-01-01T00:00:00`	`2023-01-01`	`00`	`0`	`5`
2	`2023-01-01T00:00:00`	`2023-01-01`	`00`	`1`	`5`
3	`2023-01-01T00:00:00`	`2023-01-01`	`00`	`2`	`5`

Extra runtime variables from pipeline run

If your pipeline run belongs to a trigger that is scheduled, then the following extra variables are available in your Python block’s keyword arguments (e.g. kwargs).

Key	Description	Example
`interval_start_datetime`	The `datetime` when the pipeline run is scheduled for.	`datetime.datetime(2023, 7, 23, 7, 0, 0, 0)`
`interval_end_datetime`	The `datetime` when the next pipeline run is scheduled for.	`datetime.datetime(2023, 7, 24, 7, 0, 0, 0)`
`interval_seconds`	The number of seconds between the current pipeline run and the next pipeline run.	`86400`
`interval_start_datetime_previous`	The `datetime` when the previous pipeline run was scheduled for.	`datetime.datetime(2023, 7, 22, 7, 0, 0, 0)`

SQL block

If you’re using a SQL block, here is an example of how you can access these variables:

SELECT
    '{{ interval_start_datetime }}' AS interval_start_datetime
    , '{{ interval_end_datetime }}' AS interval_end_datetime
    , '{{ interval_seconds }}' AS interval_seconds
    , '{{ interval_start_datetime_previous }}' AS interval_start_datetime_previous

Adventure starts here

Concepts

Project

Contributing

About

Backfilling pipelines

Date and time window

Custom code

Extra runtime variables from pipeline run

SQL block

​Date and time window

​Custom code

​Extra runtime variables from pipeline run

​SQL block

Date and time window

Custom code

Extra runtime variables from pipeline run

SQL block