Migrate from Dagster to Mage Pro

Documentation Index

Fetch the complete documentation index at: https://docs.mage.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Why migrate from Dagster to Mage Pro?

• Visual pipeline development and debugging
• Native data integration (ETL/ELT workflows)
• Mixing SQL and Python naturally in the same pipeline
• Streaming or event-driven data processing
• Built-in data previews and interactive development
• AI-powered pipeline development

​

Mage Pro vs Dagster: Benefits Overview

​

Step-by-Step Migration Instructions

​

Mapping Dagster Concepts to Mage Pro

​

Assets & Jobs → Pipelines

from dagster import asset, job, op

@asset
def raw_data():
    return pd.read_csv("data.csv")

@asset
def transformed_data(raw_data):
    return raw_data.dropna()

@job
def my_etl_job():
    transformed_data(raw_data())

• Create a new pipeline in the UI
• Each asset/op becomes a block (data loader, transformer, or data exporter)
• Dependencies are defined by connecting blocks in the UI
• No need for explicit job definitions—pipeline structure defines execution

​

Assets & Ops → Blocks

@asset
def extract_data():
    return pd.read_csv("data.csv")

@op
def transform_data(context, df):
    return df.dropna()

@asset
def load_data(transformed_data):
    transformed_data.to_sql("table", engine)

• extract_data → Data loader block (Python or connector)
• transform_data → Transformer block (Python, SQL, or dbt block)
• load_data → Data exporter block (Python or connector)

​

Dagster Resources → Mage Blocks

​

Partitions & Materializations

from dagster import DailyPartitionsDefinition, asset

daily_partitions = DailyPartitionsDefinition(start_date="2024-01-01")

@asset(partitions_def=daily_partitions)
def partitioned_asset(context):
    partition_date = context.partition_key
    # Process data for specific partition
    ...

• Use pipeline variables or dynamic blocks for partition logic
• Configure incremental processing in SQL blocks or Python blocks
• Use backfill triggers for historical data processing
• Schedule pipelines with cron expressions for partition-based runs

​

Scheduling & Sensors

from dagster import schedule, sensor, RunRequest

@schedule(cron_schedule="0 0 * * *", job=my_etl_job)
def daily_schedule(context):
    return RunRequest()

@sensor(job=my_etl_job)
def file_sensor(context):
    if file_exists("new_data.csv"):
        return RunRequest()

• Create a schedule trigger in the pipeline UI
• Configure cron expression, interval, or event-based triggers
• Use file triggers for file-based sensors
• Use event triggers for webhook-based sensors
• Use the Deployment App to manage deployments across environments (dev, staging, prod)

​

Secrets & Configuration

from dagster import ConfigurableResource

class MyResource(ConfigurableResource):
    api_key: str

@asset
def use_resource(my_resource: MyResource):
    key = my_resource.api_key

• Use workspace variables or pipeline variables
• Access via variables dictionary in blocks
• Built-in secret manager with encryption, or integrate with external secret managers (AWS Secrets Manager, HashiCorp Vault, etc.)
• Secrets are isolated per workspace and pipeline

​

Convert Dagster Code with AI Sidekick

​

How to Use It

1. Click the “Ask AI” button in the top-right corner of the Mage Pro UI.
2. Paste your Dagster code — including @asset, @op, @job, or asset-based workflows.
3. Ask: “Convert this Dagster job to a Mage pipeline.”
4. Sidekick will:
   • Parse the asset/op structure and dependencies
   • Generate the corresponding Mage blocks (data loader, transformer, data exporter)
   • Define block dependencies and scheduling logic
   • Convert Dagster resources to Mage connectors where applicable
   • Optimize the generated code for performance and best practices
5. Review and insert the generated pipeline directly into your project.

​

Why Use Sidekick?

• Faster migration: turn entire jobs into Mage pipelines in seconds
• Less error-prone: Sidekick understands asset dependencies, partitions, and op types
• Code optimization: automatically optimizes generated code for performance
• Context-aware: uses your project setup and prior block structure
• Fully editable: review, tweak, and insert blocks before saving

​

Tips for Migrating Complex Jobs

​

Handling Dagster Asset Dependencies

@asset
def parent_asset():
    ...

@asset
def child_asset(parent_asset):
    ...

• Dependencies are automatically inferred from block connections in the UI
• Use upstream blocks to define data flow
• Blocks with no dependencies run in parallel automatically

​

Dagster Partitions

from dagster import DailyPartitionsDefinition

daily_partitions = DailyPartitionsDefinition(start_date="2024-01-01")

@asset(partitions_def=daily_partitions)
def partitioned_asset(context):
    partition_date = context.partition_key
    ...

• Use pipeline variables to pass partition information
• Configure incremental processing in SQL blocks
• Use backfill triggers to process historical partitions
• Schedule pipelines with appropriate cron expressions

​

Dagster Resources & Configuration

from dagster import ConfigurableResource, asset

class DatabaseResource(ConfigurableResource):
    connection_string: str

@asset
def use_database(database: DatabaseResource):
    conn = database.connection_string
    ...

• Use workspace variables or pipeline variables for configuration
• Access via variables dictionary in blocks
• Use data loader/exporter blocks with native connectors for databases
• Secrets are managed through built-in or external secret managers

​

Dagster Retries & Error Handling

@asset(retry_policy=RetryPolicy(max_retries=3, delay=60))
def risky_asset():
    ...

• Configure retry settings in block or pipeline configuration
• Set retry count, delay, and backoff strategy
• Built-in error handling and alerting

​

Dagster Caching & Materializations

@asset(
    compute_kind="pandas",
    metadata={"cache": True}
)
def cached_asset():
    ...

• Use block output caching in pipeline settings
• Configure cache expiration and invalidation rules
• Preview cached outputs in the UI

​

After Migration: What You Get with Mage Pro

• Unified data platform: orchestration + data integration + transformation + streaming in one tool
• Visual pipeline builder with real-time data previews
• AI-powered block generation, optimization, and debugging (via AI Sidekick)
• Git-backed version control and CI/CD with UI-based Deployment App
• Built-in access controls, audit logs, and workspace isolation
• Support for SQL, Python, R, streaming, dbt blocks, Delta Lake, Iceberg, and more
• 200+ native connectors for databases, APIs, and cloud storage
• Streaming pipelines for real-time data processing
• Flexible secrets management: built-in secret manager or integrate with external systems (AWS Secrets Manager, Vault, etc.)
• Auto-scaled executors on Kubernetes or Docker
• Transparent pricing: usage-based pricing (SaaS) or cluster/workspace license-based pricing (self-hosted)

Dagster excels at asset-based orchestration. But when you need data engineering, visual development, AI assistance, and native data integration — Mage Pro is built for modern data teams.