Schedules
You can schedule your pipeline to run on these intervals:
- Once
- Hourly
- Weekly
- Monthly
- Custom schedule using CRON Expression
- Every minute using CRON expression:
* * * * * - Streaming pipeline: coming soon Learn more about how to schedule pipelines.
Landing times
If the trigger type is a schedule then instead of choosing when the pipeline will run, you can choose the pipeline’s landing time.
This is useful if you want your data pipeline to finish at a specific time instead of start at a specific time.
For example, a pipeline is scheduled to land daily at 17:00 UTC. If this pipeline takes 10 hours on average to complete, then the pipeline will be triggered and start running at 05:00 UTC each day.
Enable landing time
When editing a trigger with schedule type, there is a toggle called Enable landing time.
Turn that on to enable landing time for the trigger.
Configure landing time
Depending on the frequency, you’ll be able to configure the following times the pipeline should complete:
- Day of the month
- Day of the week
- Hour of the day
- Minute of the hour
- Second of the minute
Extra runtime variables from pipeline run
If your pipeline run belongs to a trigger that is scheduled, then the following extra
variables are available in your Python block’s keyword arguments (e.g. kwargs).
| Key | Description | Example |
|---|---|---|
interval_start_datetime | The datetime when the pipeline run is scheduled for. | datetime.datetime(2023, 7, 23, 7, 0, 0, 0) |
interval_end_datetime | The datetime when the next pipeline run is scheduled for. | datetime.datetime(2023, 7, 24, 7, 0, 0, 0) |
interval_seconds | The number of seconds between the current pipeline run and the next pipeline run. | 86400 |
interval_start_datetime_previous | The datetime when the previous pipeline run was scheduled for. | datetime.datetime(2023, 7, 22, 7, 0, 0, 0) |
SQL block
If you’re using a SQL block, here is an example of how you can access these variables:
SELECT
'{{ interval_start_datetime }}' AS interval_start_datetime
, '{{ interval_end_datetime }}' AS interval_end_datetime
, '{{ interval_seconds }}' AS interval_seconds
, '{{ interval_start_datetime_previous }}' AS interval_start_datetime_previous