Your AI data engineer

Mage Pro AI Sidekick is your intelligent assistant for data pipelines, analytics, and ML workflows. It helps you design, debug, and optimize your pipelines — faster and smarter — using natural language. Whether you’re building ETL processes, exploring data, or resolving code issues, Mage Pro AI understands your intent and delivers code, context, and insights on the spot.

Use it to:

• Generate production-grade pipelines with a simple prompt
• Fix broken blocks or refactor complex logic
• Analyze pipeline behavior and explain errors
• Visualize outputs and transform them into shareable, interactive data products

It’s like having a senior data engineer available 24/7 — inside your repo.

​

AI-Powered Answers for Data Engineering, Analytics, and ML

Mage Pro AI is your always-on data expert. Whether you’re building data pipelines, designing warehouse schemas, troubleshooting integrations, or making sense of your analytics stack, Mage Pro AI has the answer. It’s trained to understand the language of modern data teams — from ETL to orchestration to compliance — so you can move faster, with confidence.

​

How it works

Just type your question — natural language is all it takes. Mage Pro AI instantly parses your query and provides accurate, actionable responses across a wide range of data topics: infrastructure, modeling, governance, ML, and more. It’s built into the Mage Pro platform and fine-tuned specifically for data engineers, analysts, and architects. No docs to dig through. No syntax to memorize. Just ask.

​

Why it matters

Data teams lose countless hours searching for answers across scattered documentation, Slack threads, and tribal knowledge. Mage Pro AI changes that. With expert-level responses at your fingertips, your team can solve problems faster, onboard more easily, and stay focused on high-impact work. It’s like having a senior engineer on call — 24/7.

​

Design Complex Data Pipelines in Seconds

Stop wasting hours piecing together pipelines by hand. Mage Pro AI can design and build complex data workflows in seconds. Just describe your goal in plain English, and Mage Pro AI auto-generates the full pipeline—every block, every dependency, every line of code.

​

How it works

Tell Mage Pro AI what you want to build—ingest from S3, transform with Polars, join with a Delta Lake table, run daily, alert on failure. Mage instantly maps out the entire pipeline and generates working code for each step, fully integrated with your existing tools and infrastructure.

​

Why it matters

Manually building pipelines is slow, error-prone, and often blocks teams from experimenting or scaling quickly. Mage Pro AI turns hours of setup into seconds of generation—so engineers can focus on what matters: logic, not boilerplate.

​

AI-Generated Code Blocks for Data Pipelines

Design your pipeline one step at a time—without writing a single line of code. Mage Pro AI can generate complete, production-ready code blocks for every stage of your data workflow.

​

How it works

Tell Mage what you need:

“Load this CSV from S3,”
“Join with this Delta Lake table,”
“Clean nulls in this column,”
“Calculate weekly aggregates.”

​

Why it matters

Writing every block by hand is slow and repetitive. Mage Pro AI speeds up development by giving you smart, reusable code blocks that integrate cleanly with upstream and downstream steps.

​

Smarter, Faster Code for Every Pipeline Block

Mage Pro AI helps you evolve your workflows by intelligently updating, refactoring, and improving code blocks in place.

​

How it works

Highlight a block. Ask Mage Pro AI to optimize it, update logic, or align with new requirements. The AI understands your pipeline context and rewrites the code accordingly.

​

Why it matters

Mage Pro AI handles updates safely, intelligently, and instantly—saving hours of debugging while ensuring consistency and performance across your pipeline.

​

Fix Pipeline Code Errors with AI-Powered Debugging

Mage Pro AI helps you resolve issues in your pipeline code with intelligent, context-aware fixes.

​

How it works

Highlight the problematic block or describe the issue. Mage Pro AI analyzes the code and suggests a corrected version that works within the full pipeline context.

​

Why it matters

Debugging pipeline code wastes valuable time. Mage Pro AI reduces that friction by helping your team quickly find and fix issues.

​

Self-Healing Data Pipelines with AI

Mage Pro AI makes it possible to build pipelines that detect failures, diagnose errors, and fix themselves.

​

How it works

When a pipeline breaks, Mage Pro AI iterates through debugging cycles, rewriting blocks and rerunning them until validation checks pass.

​

Why it matters

Mage Pro AI automatically fixes pipelines, ensuring that data quality checks are met without wasting hours chasing down errors.

​

Understand Your Pipeline Data with AI-Powered Q&A and Analysis

With Mage Pro AI, you can ask questions about your pipeline data and get clear answers.

​

How it works

Ask Mage Pro AI questions like “Why did this block fail?” or “Are there outliers in this column?” It responds with insights, context, and suggested actions.

​

Why it matters

Mage Pro AI eliminates the need to query logs or write SQL—helping you understand your data instantly and take informed action.

​

AI-Powered Data Visualizations for Every Step in Your Pipeline

Mage Pro AI transforms pipeline outputs into insightful, ready-to-use visualizations.

​

How it works

After a block runs, Mage Pro AI analyzes its output and auto-generates the most appropriate visualization—bar charts, line graphs, histograms, heatmaps, and more.

​

Why it matters

Visual feedback helps you debug faster and understand your data more deeply at every step—without needing to configure or write plotting code.

​

Build Interactive Data Products Instantly with AI-Generated Tables

Mage Pro AI lets you turn raw pipeline outputs into smart, explorable tables—instantly.

​

How it works

Mage Pro AI creates interactive tables with sorting, filtering, and searching automatically after your pipeline runs—no front-end or dashboard setup needed.

​

Why it matters

Don’t stop at raw output. Instantly turn your data into interactive, insightful products for analysts, stakeholders, or pipelines.

​

Enable RAG

Empower Mage Pro AI to give smarter, context-aware answers by enabling RAG (Retrieval-Augmented Generation).

When RAG is enabled, Mage Pro AI can reference your actual project files — including pipeline code, configurations, and documentation — to provide deeper, more accurate answers. This dramatically improves the AI’s ability to help with debugging, code generation, and pipeline understanding.

​

Why Enable RAG

Without RAG, Mage Pro AI answers based solely on general knowledge and standard behaviors. With RAG enabled:

• ✅ The AI can reference your exact project structure and code
• ✅ Responses are context-specific and align with your real pipelines
• ✅ You get accurate fixes, relevant suggestions, and smarter debugging
• ✅ Answers reflect your own naming conventions, tools, and logic

If RAG is not enabled, the AI will ask for permission before reading any of your files each time you make a request.

​

How to Enable RAG

Create a file at the root of your project named ai_config.yaml. This file defines which folders the AI is allowed to read and which files to exclude.

​

Option 1: Enable RAG for the entire project

Create a file at the root of your project named ai_config.yaml

rag:
  folders:
    include:
      - /home/src/default_repo
  files:
    exclude:
      # gitignore patterns
      - secrets.json
# Add this if you permit the AI to read your data when asking it questions.
# If this is not present, then the AI will ask for your permission each time.
data:
  all: true

​

Option 2: Enable RAG for selected folders only

rag:
  folders:
    include:
      - /home/projects/mage-pro-tutorials/callbacks
      - /home/projects/mage-pro-tutorials/conditionals
      - /home/projects/mage-pro-tutorials/custom
      - /home/projects/mage-pro-tutorials/data_exporters
      - /home/projects/mage-pro-tutorials/data_loaders
      - /home/projects/mage-pro-tutorials/dbt
      - /home/projects/mage-pro-tutorials/extensions
      - /home/projects/mage-pro-tutorials/sensors
      - /home/projects/mage-pro-tutorials/transformers
      - /home/projects/mage-pro-tutorials/services
      - /home/projects/mage-pro-tutorials/utils
  files:
    exclude:
      # gitignore patterns
      - .env

​

How to Trigger Reindexing

Once you’ve updated your ai_config.yaml, call the following API to reindex your project files into the vector store:

import requests

url = "https://cluster.mage.ai/[cluster UUID]/api/vector_store"

headers = {
    "Authorization": "Bearer IwMQiEhGerIOt2DxJNiCwOlSiRigpda2FM4mTfjphq8",
    "Content-Type": "application/json",
}

payload = {
    "vector_store": {
        "path": "/home/src/default_repo",
    },
    "api_key": "[API key]",
}

response = requests.post(url, headers=headers, json=payload)

After reindexing, Mage Pro AI will be able to answer questions using your real project data—enabling more personalized, precise assistance.

​

Real-World Use Cases for Mage Pro AI

​

Build AI-Generated Data Pipelines

Create complete data pipelines from a single prompt. Describe your data goals in plain English and Mage Pro AI generates end-to-end ETL workflows with correct dependencies and block-level logic.

​

Automate Machine Learning Pipelines

Use Mage Pro AI to generate data preparation, model training, and evaluation pipelines in seconds. Easily experiment and scale machine learning workflows without writing boilerplate code.

​

Generate Code Blocks On Demand

Need to load a CSV, clean nulls, or join with a Delta Lake table? Mage Pro AI generates clean, reusable code blocks tailored to your infrastructure, tools, and pipeline logic.

​

Debug and Fix Pipelines with AI

Highlight a broken step or describe the issue—Mage Pro AI suggests fixes, explains errors, and updates the code in context. Eliminate hours of debugging time.

​

Build Self-Healing Pipelines

Mage Pro AI can detect failed blocks, rerun them, and rewrite logic to meet validation rules—creating fault-tolerant pipelines that repair themselves automatically.

​

Ask Questions About Your Data

Want to understand why a block failed or check for anomalies? Mage Pro AI provides contextual answers and analysis—no SQL or log digging required.

​

Auto-Generate Visualizations and Tables

Mage Pro AI analyzes pipeline outputs and automatically creates helpful visualizations and interactive tables—ready for review or sharing with stakeholders.

​

Rapidly Prototype and Iterate

Test new data workflows, transform logic, or refactor pipelines in seconds using natural language. Mage Pro AI makes it easy to try new ideas without slowing down.