Spark

Add credentials

Create a new pipeline or open an existing pipeline.
Expand the left side of your screen to view the file browser.
Scroll down and click on a file named io_config.yaml.
Enter the following keys and values under the key named default (you can have multiple profiles, add it under whichever is relevant to you)

For a local Spark session:

version: 0.1.1
default:
  SPARK_HOST: local
  SPARK_METHOD: session
  SPARK_SCHEMA: default

For a remote Spark cluster, set SPARK_HOST to the Spark master URL (for example, spark://host:7077, local[*], or another valid SparkSession.builder.master(...) value). You can also set SPARK_METHOD (e.g. session) and SPARK_SCHEMA to control how the session is created and which default database/schema is used.

Using Python block

Create a new pipeline or open an existing pipeline.
Add a data loader, transformer, or data exporter block (the code snippet below is for a data loader).
Select Generic (no template).
Enter this code snippet (note: change the config_profile from default if you have a different profile):

from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.spark import Spark
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_data_from_spark(**kwargs) -> DataFrame:
    query = 'SELECT 1'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    loader = Spark.with_config(ConfigFileLoader(config_path, config_profile))
    return loader.load(query)

Run the block.

Data Preview for Spark paths

In Mage Pro, Data Preview can read file-backed Spark datasets without requiring a persistent Hive metastore table. Use a configured Spark profile and select a project-relative path with one of the supported Spark file formats: parquet, csv, json, or orc. For example, a Spark Data Preview source can target a Parquet dataset stored under the Mage project:

provider: spark
profile: default
format: parquet
path: datasets/events

The path must resolve inside the Mage project directory. If you need catalog table discovery across separate Spark sessions or worker processes, configure a durable Spark catalog or metastore and use the table name instead.

Export a dataframe to Spark

from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.spark import Spark
from os import path
from pandas import DataFrame

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter


@data_exporter
def export_data_to_spark(df: DataFrame, **kwargs) -> None:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'
    table_name = 'your_table_name'
    database = 'default'

    loader = Spark.with_config(ConfigFileLoader(config_path, config_profile))
    loader.export(df, table_name=table_name, database=database, if_exists='replace')

Notes

Spark runs in-process; ensure PySpark and any required cluster dependencies are installed in your Mage environment.
For local development, SPARK_HOST: local typically creates a session with SparkSession.builder.master('local').getOrCreate().
Use SPARK_SCHEMA to set the default database/schema for queries and exports.

Infrastructure

3rd party

Add credentials

Using Python block

Data Preview for Spark paths

Export a dataframe to Spark

Notes

​Add credentials

​Using Python block

​Data Preview for Spark paths

​Export a dataframe to Spark

​Notes

Add credentials

Using Python block

Data Preview for Spark paths

Export a dataframe to Spark

Notes