A callback block is associated to another block. When the parent block succeeds or fails, the callback block functions are executed.
0.8.61
or greater.Callback blocks are part of your pipeline but don’t run as individual steps in your pipeline like data loader blocks, transformer blocks, etc. However, callback blocks are associated to other blocks within the pipeline (e.g. data loaders, transformers, data exporters, etc).
After those other blocks are executed, any associated callback blocks will also be executed depending on the status of the parent block.
Your pipeline has the following blocks:
load_data_from_api
(data loader)clean_column_names
(transformer)save_data
(data exporter)Then, you add the following callback block to your pipeline:
You associate the above callback block to the following 2 blocks:
load_data_from_api
(data loader)clean_column_names
(transformer)When you run the pipeline, it’ll execute the load_data_from_api
(data loader) block first.
It successfully completes. Then, the callback block will run and execute the function hello_world
.
Next, the pipeline will execute the clean_column_names
(transformer) block.
It fails due to some error. The callback block will run and execute the function alert_me
.
@callback
.
You can have as many functions in a callback block as you want.@callback
or @callback('success')
to decorate a function that should only run when the
parent block successfully completes.@callback('failure')
to decorate a function that should only run when the
parent block fails.success
Execute callback function when block runs successfully.
failure
Execute callback function when block fails.
The 1st positional argument is the data output of the callback block’s parent block.
Positional arguments are only supported in Mage versions >= 0.8.81
.
Here are the keyword arguments that are available in each callback function:
Name | Description | Sample value |
---|---|---|
block_uuid | Callback block UUID. | 'fireball_callback' |
parent_block_uuid | The block UUID of the parent block. | 'parent_block' |
ds | Date string when the parent block started executing. | '2023-12-25' |
event | A dictionary containing metadata from an event triggered pipeline. | {} |
execution_date | Python datetime object for when the parent block started executing. | datetime.datetime(2023, 4, 26, 20, 28, 17, 335254, tzinfo=datetime.timezone.utc) |
execution_partition | Partition used for the parent block when it was executed. | '207/20230426T202817' |
hr | Hour string when the parent block started executing. | '20' |
pipeline_run | Python pipeline run object associated to the current run. | PipelineRun(id=2357, pipeline_uuid=fire_etl, execution_date=2023-04-26 20:28:17.335254+00:00) |
pipeline_uuid | UUID of the current pipeline. | 'fire_etl' |
__input | For version >= 0.8.81 . See below for more details. | { 'some_block_uuid': {} } |
__input
keyword argument>= 0.8.81
.If your callback function is defined to only accept keyword arguments and no positional arguments,
you can access the callback block’s parent block’s data output by accessing the key
labeled __input
in the callback function’s keyword arguments.
The value of the __input
key is a dictionary. The keys in that dictionary are named after the
callback block’s parent block’s UUID.
For example, if the callback block’s parent block’s UUID is load_api_data
, then this is how
you would access load_api_data
’s data output:
The following keyword arguments are only available in callback functions with parent blocks in a data integration pipelines:
Name | Description | Sample value |
---|---|---|
destination_table | The table name that the source data is being exported to. | 'dim_users_v1' |
index | The order in which the parent block executed compared to other blocks of the same type. | 0 |
is_last_block_run | Boolean value for whether or not the parent block was the last block ran for the block type in the pipeline. | True or False |
stream | The name of the source stream (or table) that the parent block is syncing for. | 'stripe_transactions' |
< 0.8.61
.on_success
callback and/or a on_failure
callback. These callbacks
will run after your block run completes or fails. When you run your block in the
pipeline edit page, it will also run the callback depending on the status of the block.kwargs.get('<variable>')
syntax in your callback.