dbt-spark
with a pySpark session.
dbt_spark
, and add a Scratchpad
to
test out the connection with PySpark, with the following code:Terminal
icon on the right side of the Mage UI, and create a dbt
project spark_demo
, with the following commands:demo_project/dbt/spark_demo/
.
Click the file named profiles.yml
, and add the following settings to this file:profiles.yml
file by pressing Command (⌘) + S
, then close the file
by pressing the X button on the right side of the file name dbt/spark_demo/profiles.yml
.
dbt model
, and choose the option New model
. Enter model_1
as the Model name
, and spark_demo/models/example
as the folder location.
model_1
, next to the label Target
at the top,
choose dev
in the dropdown list. You can also check Manually enter target
,
and enter dev
in the input field.
model_1
:
Compile & preview
button to execute this new model, which would
generate the results similar to the following: