Pipelines
Overview
Data pipeline containing execution settings, resources, block information, and block dependencies.
Pipeline object
blocks
array of objects
Array of block objects. See the blocks section for more details.
description
string
Description for the pipeline.
extensions
array of objects
Array of extension block objects. Same shape as blocks
.
name
string
requiredHuman friendly name of the pipeline.
schedules
array of objects
requiredArray of trigger objects.
type
string
requiredThe type of the pipeline:
integration
, pyspark
, python
, streaming
Note that python
is a standard (batch) pipeline with a python backend, while pyspark
is a batch pipeline with a spark backend.
uuid
string
requiredUnique identifier for the pipeline.
variables
object
Object containing variables for the pipeline.
[key]
string
The property name is user defined.
widgets
array of objects
Array of widget block objects. Same shape as blocks
.
Was this page helpful?