Callback blocks

Callback blocks are part of your pipeline but don’t run as individual steps in your pipeline like data loader blocks, transformer blocks, etc. However, callback blocks are associated to other blocks within the pipeline (e.g. data loaders, transformers, data exporters, etc).

After those other blocks are executed, any associated callback blocks will also be executed depending on the status of the parent block.

​

Example

1. Your pipeline has the following blocks:

   1. load_data_from_api (data loader)
   2. clean_column_names (transformer)
   3. save_data (data exporter)

2. Then, you add the following callback block to your pipeline:

   Copy

   @callback('success')
   def hello_world(parent_block_data, **kwargs):
       # some code...
   
   
   @callback('failure')
   def alert_me(parent_block_data, **kwargs):
       # some code...
   

3. You associate the above callback block to the following 2 blocks:

   • load_data_from_api (data loader)
   • clean_column_names (transformer)

4. When you run the pipeline, it’ll execute the load_data_from_api (data loader) block first. It successfully completes. Then, the callback block will run and execute the function hello_world.

5. Next, the pipeline will execute the clean_column_names (transformer) block. It fails due to some error. The callback block will run and execute the function alert_me.

​

How to add callbacks to your pipeline

1. Create a new pipeline or open an existing pipeline.
2. Edit the pipeline.
3. On the right side of the page, click the Add-ons icon in the navigation. If you don’t see it, try expanding the right area of the page.
4. Click the button Callbacks.
5. Click the button + Callback block.
6. In the callback block’s code, add a function and decorate it with @callback. You can have as many functions in a callback block as you want.
7. Use @callback or @callback('success') to decorate a function that should only run when the parent block successfully completes.
8. Use @callback('failure') to decorate a function that should only run when the parent block fails.

​

Supported callbacks

​

success

Execute callback function when block runs successfully.

@callback('success')
def only_run_this_function_on_success(parent_block_data, **kwargs):
    pass

​

failure

Execute callback function when block fails.

@callback('failure')
def only_run_this_function_on_failure(parent_block_data, **kwargs):
    pass

​

Positional arguments

The 1st positional argument is the data output of the callback block’s parent block.

​

Keyword arguments

Here are the keyword arguments that are available in each callback function:

​

__input keyword argument

If your callback function is defined to only accept keyword arguments and no positional arguments, you can access the callback block’s parent block’s data output by accessing the key labeled __input in the callback function’s keyword arguments.

The value of the __input key is a dictionary. The keys in that dictionary are named after the callback block’s parent block’s UUID.

For example, if the callback block’s parent block’s UUID is load_api_data, then this is how you would access load_api_data’s data output:

@callback('success')
def callback_for_load_api_data(**kwargs):
    data_from_parent_block = kwargs['__input']['load_api_data']

​

Data integration pipelines only

The following keyword arguments are only available in callback functions with parent blocks in a data integration pipelines:

​

Legacy

​

Adding a callback to your block

1. Create a block as normal.
2. In the top right of the block, click the triple dot icon which should open up a menu with more options. Click “Add callback”.
3. You should now see another code editor under the main editor for your block.
4. You can add an on_success callback and/or a on_failure callback. These callbacks will run after your block run completes or fails. When you run your block in the pipeline edit page, it will also run the callback depending on the status of the block.
5. The callback function will be passed your pipeline’s runtime variables as keyword arguments, so you can use the same kwargs.get('<variable>') syntax in your callback.