Data pipeline management
- 8.Backfill (WIP)
- 9.Monitor (WIP)
This page will show all the pipelines in your project.
From this page, you can also create a new pipeline by clicking the
[+ New pipeline]button.
Creating a new pipeline will take you to the Pipeline edit page; a notebook-like experience for adding blocks, creating dependencies between blocks, testing code, and visualizing data with charts.
View all the runs for every pipeline in your current project.
This page contains all the information and history for a single pipeline:
This page shows all the active and inactive triggers for a single pipeline.
Create a new trigger for this pipeline by clicking the
[+ Create]button near the top of the page.
You can configure the trigger to run the pipeline on a schedule or when an event occurs.
Example page for creating a schedule type trigger.
On this page, you can start or pause the trigger. Starting the trigger will make it active. Pausing the trigger will prevent it from running the pipeline.
Note: if you have other triggers for this pipeline, pausing 1 trigger may not stop the pipeline from running since other triggers can also run the pipeline.
You can also edit the trigger after creating it by clicking the
View the pipeline runs and block runs for a pipeline.
Pipeline detail runs
Browse all logs for a pipeline. You can search and filter logs by log level, block type, block UUID, and more.
Pipeline detail logs
You can monitor many metrics for each of your pipelines and blocks. Soon, you’ll be able to monitor aggregate metrics across all pipelines and blocks.