Backfills
Backfill object
{
"block_uuid": null,
"completed_at": null,
"created_at": "2023-03-23 18:27:58.848959+00:00",
"end_datetime": "2023-03-05 00:00:00+00:00",
"failed_at": null,
"id": 9,
"interval_type": "hour",
"interval_units": 4,
"metrics": null,
"name": "sparkling glitter",
"pipeline_schedule_id": null,
"pipeline_uuid": "example_pipeline",
"start_datetime": "2023-03-01 00:00:00+00:00",
"started_at": null,
"status": null,
"total_run_count": 25,
"updated_at": "2023-03-23 18:28:22.379725+00:00",
"variables": null
}
If the backfill uses custom code to generate the pipeline runs, this block UUID contains the code that will be used.
Date and time the backfill finished running successfully.
Date and time the record was created.
The date and time the backfill should stop on.
Date and time the backfill failed.
Unique ID for the record.
second
, minute
, hour
, day
, week
, month
, year
, custom
The number of interval_type
to use in between each pipeline run that the backfill creates.
For example, if interval_type
is day
and interval_units
is 3, then the backfill will create
pipeline runs spanning 3 day increments between the start_datetime
and end_datetime
.
Various metrics related to the pipeline run; e.g. number of rows processed in a data integration pipeline sync.
Name of the record.
Pipeline schedule that this record belongs to.
Pipeline that this record belongs to.
The date and time the backfill should start on.
Date and time the backfill started running.
initial
, running
, completed
, failed
, cancelled
Number of pipeline runs that will be created for this backfill.
Date and time the record was last updated.
Object containing variables for the pipeline that is being overwritten by this backfill.
The property name is user defined.
Read all backfills
GET /api/backfills
Maximum number of logs to be returned in the API response.
Example: 20
Read logs after N number of logs, where N equals _offset
.
Example: 10
curl --request GET \
--url 'http://localhost:6789/api/backfills?_limit=20&_offset=0&include_run_count=true&pipeline_uuid=example_pipeline&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'Authorization: Bearer ZKXScwLKLZUujbw7vwMeWnhtTTSLqifw8_RnBHYipww' \
{
"backfills": [
...
],
"metadata": {
"count": 3,
"next": false
}
}
Create backfill
POST /api/pipelines/:pipeline_uuid/backfills
Pipeline UUID to create the backfill for.
curl --request PUT \
--url 'http://localhost:6789/api/pipelines/example_pipeline/backfills?api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
--header 'Authorization: Bearer ZKXScwLKLZUujbw7vwMeWnhtTTSLqifw8_RnBHYipww' \
--data '{
"backfill": {
"name": "fire wind"
}
}'
{
"backfill": {
"id": 10,
"block_uuid": null,
"completed_at": null,
"created_at": "2023-04-01 01:02:18.411997+00:00",
"end_datetime": null,
"failed_at": null,
"interval_type": null,
"interval_units": null,
"metrics": null,
"name": "fire wind",
"pipeline_schedule_id": null,
"pipeline_uuid": "example_pipeline",
"start_datetime": null,
"started_at": null,
"status": null,
"updated_at": "2023-04-01 01:02:18.411997+00:00",
"variables": null
}
}
Read backfill
WIP
Update backfill
WIP
Delete backfill
WIP