Create pipeline

  curl --request POST 
  -H 'Content-Type: application/json' 
  -H 'Cookie: oauth_token=some_really_long_string'
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
  -d '{"pipeline": {"name": "arwen-starlight", "type": "python"}}' 
  --url http://localhost:6789/api/pipelines

{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "python",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "conditionals": [],
    "widgets": []
  }
}

POST /api/pipelines

string

required

Human readable name of the pipeline.

string

required

The type of the pipeline: integration, pyspark, python, streaming Note that python is a standard (batch) pipeline with a python backend, while pyspark is a batch pipeline with a spark backend.

string

If supplied, the uuid of a pipeline to clone.

array of objects

Array of extension block objects. Same shape as blocks.

array of objects

Array of callback block objects. Same shape as blocks.

array of objects

Array of conditional block objects. Same shape as blocks.

  curl --request POST 
  -H 'Content-Type: application/json' 
  -H 'Cookie: oauth_token=some_really_long_string'
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
  -d '{"pipeline": {"name": "arwen-starlight", "type": "python"}}' 
  --url http://localhost:6789/api/pipelines

{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "python",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "conditionals": [],
    "widgets": []
  }
}

Update pipeline

  curl --request POST 
  -H 'Content-Type: application/json' 
  -H 'Cookie: oauth_token=some_really_long_string'
  -H 'X-API-KEY: zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2'
  -d '{"pipeline": {"name": "arwen-starlight", "type": "python"}}' 
  --url http://localhost:6789/api/pipelines

{
  "pipeline": {
    "data_integration": null,
    "description": null,
    "executor_config": {},
    "executor_count": 1,
    "executor_type": null,
    "name": "arwen-starlight",
    "notification_config": {},
    "type": "python",
    "updated_at": null,
    "uuid": "arwen_starlight",
    "spark_config": {},
    "blocks": [],
    "callbacks": [],
    "conditionals": [],
    "widgets": []
  }
}