Build a data pipeline that loads restaurant data, transforms it, then exports it to a DuckDB database. 🦆
Load data from an online endpoint
Clean column names and add a new column
number of meals
, that counts the number of meals for each user.Write the transformed data to a local DuckDB database
https://localhost:6789
in your browser to see the pipeline in action!
1️⃣ Create a pipeline
/pipelines
). This is the default page when navigating to Mage in your web browser. If you have Mage running, just click here.In the top left corner, select the button labeled + New, then select the option labeled Standard (batch) to create a new pipeline.In the left vertical navigation, click the last link labeled Pipeline settings. Change the pipeline’s name to ETL demo
. Click the button labeled Save pipeline settings.2️⃣ Load data from an API
load data
. Click the button labeled Save and add block. Paste the following code in the data loader block:⌘ + Enter
Control + Enter
Shift + Enter
3️⃣ Transform data
⌘ + Enter
Control + Enter
Shift + Enter
4️⃣ Export data to DuckDB
Data Exporter
SQL block and change the connection type to be DuckDB
.Next, change the Table name
to be magic_duck.restaurant_user_transactions
and make sure the content of the block is:magic_duck.restaurant_user_transactions
! 🎉After you run the block (⌘ + Enter), you’ll see a sample of the data that was exported.