Quickstart
Developer experience
Version control
Blocks
Pipelines
dbt
- Overview
- Getting started
- New project
- Configuration
- Models
- Commands
Mage tips & tricks
Here are some neat ways you can make the most of Mage! Tips and tricks to help you make the most magical pipelines possible.
Mage has native version control for your files, so you’ll never lose work again! Check it out in our docs here or see the video below for a tutorial.
Mage integrates with DuckDB! Watch the video below to learn more or read on here.
Mage has an integration with the OpenAI API. Watch the video to learn about how you can generate pipelines using the power of AI.
See our full docs here for more info.
Mage supports streaming data to any of our destinations. Watch the video below to learn about configuring a streaming pipeline in under a minute!
See our full docs here for more info.
Watch this video to learn how you can use Conditional Blocks to add conditional execution and branching logic to your next data pipeline!
See our full docs here for more info.
Watch this video to learn how you can use Replica Blocks to add conditional execution and branching logic to your next data pipeline!
See our full docs here for more info.
The Mage pipeline editor can be used in a single, notebook-style interface or in a split-pane view, with block output displayed on the right and blocks themselves on the left.
Split pane view can be enabled by selecting the “split” icon in the pipeline editor, as shown below:
Can’t wait to try it out? Check out the video below for a demo!
This week’s Mage tip is all about organization!
Did you know you can tag, group, and filter your pipelines from the overview page? Watch the video below to learn more 😀
Read more in our docs here!
We’ve received a ton of feedback from our users, and we’ve been hard at work making updates to the tree view based on your suggestions. So, what’s new? Let me give you the lowdown. 👇
🔹 Enhanced Functionality: The tree view now comes with a whole right-click menu that allows you to easily run, rename, insert, or delete blocks. You have more control than ever before!
🔹 Seamless Connections: You can now remove and add connections with a simple click. Plus, you can insert new blocks between existing ones effortlessly.
🔹 Enhanced Visualizations: With the new tree view, you get grouping and better lineage for certain block types 🔄
Not only does the new tree view offer improved functionality, but it also makes your pipelines look absolutely stunning! After all, our goal is to make data beautiful. 💫
What are dynamic blocks? Think of them as the magical way to create tasks dynamically in our tool. If you’re familiar with Airflow and the concept of Dynamic Task Mapping, this is pretty similar.
If you have a bunch of small tasks that you want to execute in parallel, Dynamic Blocks are here to save the day!
Dynamic Blocks in Mage give you the power to parallelize your workflow and reduce output effortlessly. It’s a pattern that has been around in data engineering, but we’ve made it incredibly easy and efficient.
Learn more here.
Did you know you can trigger pipelines directly with API calls in Mage? 🧙♂️💻
With API Triggers, you can kick-off your ETL runs in just about any way possible— a Python request, a cURL command, or a well-timed carrier pigeon (hopefully not). 🐦⬛
API triggers carry the added benefit of supporting runtime parameters, meaning you can trigger your pipelines and supply external variables to customize functionality/output.
You can read more in our docs here & get started today.
Managing pipelines can be tricky, especially when it comes to backfilling missing data or re-running pipelines. But with Mage, backfills are a breeze! No need for custom scripts or complex setups. 🙌
You can easily define a backfill for a specific date range. Whether you’re missing data or want to add new columns, Mage has got you covered. Simply specify the start and end dates, and let Mage do the rest.
Backfills simulate multiple runs with different execution dates, fetching the data you need and transforming it as required. 📅 🔃
To learn more about backfills using Mage, check out our documentation here.
Was this page helpful?