This guide explains how to use PySpark in Mage Pro, including how to run PySpark blocks in batch pipelines, manage dependencies like Apache Iceberg, and access cloud storage securely.
Data Loader
, Transformer
, Data Exporter
, or Custom
.SparkSession
.metadata.yaml
file to ensure PySpark works properly:
SparkSession
in any block using .builder.config(...)
to tune performance or integrate external tools.