Writing data pipeline code
Build the future you want to see.
Adding blocks made easy
You can quickly add blocks of code to your pipeline with these methods:
Drag blocks from the file browser and drop it on the pipeline
Drag blocks from the file browser and drop it on the pipeline
Supported block type directories: callbacks, conditionals, custom, data_exporters, data_loaders, dbt,
dbts, markdowns, scratchpads, sensors, and transformers
Version 0.9.64 or greater is required to add callback and conditional
blocks from the file browser.
- Open the file browser on the Pipeline Editor page.
- Click a block file and keep holding down.
- Drag the block file onto the center of the page (the Notebook area).
- Release the button and it’ll be added to the pipeline.
Search blocks across entire projects
Search blocks across entire projects
Find existing code and add it to the current pipeline.
You must first enable the feature named Add new block v2 in your project settings. This feature is available in standard (batch) pipelines.
If you just recently enabled the feature Add new block v2, you must restart Mage so that your blocks can be indexed.
Version 0.9.64 or greater is required to add callback and conditional
blocks from the block search.
Instead of clicking a button to add a new block, type in the search bar to find an existing block across all your projects.
⌘ + / (Command + Forward Slash) keyboard shortcut will instantly focus your typing on the search bar from anywhere on the page (as long as you aren’t already typing in a block of code).
Use the arrow up and arrow down key to browse through the results.
Custom code templates for reducing boilerplate work
Custom code templates for reducing boilerplate work
Create a template for a block or a pipeline that can be reused throughout your project.
Re-arrange and hide or show blocks
Re-arrange the order of blocks
Re-arrange the order of blocks
Drag a block and move it up or down by dropping it on another block.
Show blocks in current pipeline
Show blocks in current pipeline
When editing a pipeline, you can switch the file browser view to show only the blocks in the current pipeline.
After toggling this view, you can select which blocks to collapse and which to expand. This can help optimize vertical screen space while writing code.
Pre-built data integrations for 3rd party sources
Sync data from 3rd party sources without writing boilerplate code by using pre-built data integrations (Singer Taps and Targets) provided by the data engineering community.
Data integrations to sync your data to 3rd party destinations are also available in the same feature set.
These sources and destinations can be combined and chained together with your existing blocks.
Use AI to write code for you
A line of code you don’t write is a line of code you never have to debug. Use AI to generate the code for you.
Document and comment code (AI)
“Work smart, not hard.”
Let the robots document your code so you can focus on the fun and creative side of coding.
Document the code from a block
Document the code from a block
Write comments inline with the code
Write comments inline with the code
Document the entire pipeline and each block in it
Document the entire pipeline and each block in it
Block upgrades
Enhance your code with these quick power-ups:
Recursively dynamic code made simple
Recursively dynamic code made simple
Replicate the code of an existing block
Replicate the code of an existing block
Conditionally execute code
Conditionally execute code
Callbacks
Callbacks
A callback block is associated to another block. When the parent block succeeds or fails, the callback block functions are executed.
Add new block v2
An improved experience for adding new blocks to your pipeline in the
Pipeline Editor (/pipelines/[pipeline_uuid]/edit), which allows you to
search for existing block files and use data integration blocks inside of
a standard (batch) pipeline.
Code and output side-by-side view
Run code and see its output right next to where you’re writing.
This makes it so that you don’t have to keep scrolling down every time you make a code change and want to see its results.
To enable this feature:
- Go to your project settings.
- Enable the feature named Notebook block output split view.
- Open the edit view of a pipeline.
- Under the header menu named View, click the option Show output next to code.
Check out this demo for a quick overview
Check out this demo for a quick overview