Core design principles

πŸ’» Easy developer experience

Open-source engine that comes with a custom notebook UI for building data pipelines.

  • Mage comes with a specialized notebook UI for building data pipelines.

  • Use Python and SQL (more languages coming soon) together in the same pipeline for ultimate flexibility.

  • Set up locally and get started developing with a single command.

  • Deploying to production is fast using native integrations with major cloud providers.

🚒 Engineering best practices built-in

Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts.

  • Writing reusable code is easy because every block in your data pipeline is a standalone file.

  • Data validation is written into each block and tested every time a block is run.

  • Operationalizing your data pipelines is easy with built-in observability, data quality monitoring, and lineage.

  • Each block of code has a single responsibility: load data from a source, transform data, or export data anywhere.

πŸ’³ Data is a first class citizen

Designed from the ground up specifically for running data-intensive workflows.

  • Every block run produces a data product (e.g. dataset, unstructured data, etc.)

  • Every data product can be automatically partitioned.

  • Each pipeline and data product can be versioned.

  • Backfilling data products is a core function and operation.

πŸͺ Scaling is made simple

Analyze and process large data quickly for rapid iteration.

  • Transform very large datasets through a native integration with Spark.

  • Handle data intensive transformations with built-in distributed computing (e.g. Dask, Ray) [coming soon].

  • Run thousands of pipelines simultaneously and manage transparently through a collaborative UI.

  • Execute SQL queries in your data warehouse to process heavy workloads.