Writing data pipeline code
Build the future you want to see.
Code and output side-by-side view
Run code and see its output right next to where you’re writing.
This makes it so that you don’t have to keep scrolling down every time you make a code change and want to see its results.
To enable this feature:
- Go to your project settings.
- Enable the feature named Notebook block output split view.
- Open the edit view of a pipeline.
- Under the header menu named View, click the option Show output next to code.
Adding blocks made easy
You can quickly add blocks of code to your pipeline with these methods:
Supported block type directories: callbacks, conditionals, custom, data_exporters, data_loaders, dbt,
dbts, markdowns, scratchpads, sensors, and transformers
Version 0.9.64 or greater is required to add callback and conditional
blocks from the file browser.
- Open the file browser on the Pipeline Editor page.
- Click a block file and keep holding down.
- Drag the block file onto the center of the page (the Notebook area).
- Release the button and it’ll be added to the pipeline.
Re-arrange and hide or show blocks
Drag a block and move it up or down by dropping it on another block.
Pre-built data integrations for 3rd party sources
Sync data from 3rd party sources without writing boilerplate code by using pre-built data integrations (Singer Taps and Targets) provided by the data engineering community.
Data integrations to sync your data to 3rd party destinations are also available in the same feature set.
These sources and destinations can be combined and chained together with your existing blocks.
Use AI to write code for you
A line of code you don’t write is a line of code you never have to debug. Use AI to generate the code for you.
Document and comment code (AI)
“Work smart, not hard.”
Let the robots document your code so you can focus on the fun and creative side of coding.
Customize layout of buttons
The buttons that are used to add new blocks (alongside the search bar) can be personalized.
The buttons in the dropdown menu and the buttons that are visible can be configured using custom design layouts.
The default layout is:
The personalized layout shows Transformer and Global data product as opposed to
Custom and Markdown; and the dropdown menu is in a different order.
A personalized layout can look like this:
Browse dbt models across multiple projects
Dedicated file browser to discover existing models across multiple projects.
Block upgrades
Enhance your code with these quick power-ups:
Was this page helpful?