Core Design Principles

​

💻 Easy developer experience

Open-source engine that comes with a custom notebook UI for building data pipelines.

• Mage comes with a specialized notebook UI for building data pipelines.

• Use Python and SQL (more languages coming soon) together in the same pipeline for ultimate flexibility.

• Set up locally and get started developing with a single command.

• Deploying to production is fast using native integrations with major cloud providers.

​

🚢 Engineering best practices built-in

Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts.

• Writing reusable code is easy because every block in your data pipeline is a standalone file.

• Data validation is written into each block and tested every time a block is run.

• Operationalizing your data pipelines is easy with built-in observability, data quality monitoring, and lineage.

• Each block of code has a single responsibility: load data from a source, transform data, or export data anywhere.

​

💳 Data is a first class citizen

Designed from the ground up specifically for running data-intensive workflows.

• Every block run produces a data product (e.g. dataset, unstructured data, etc.)

• Every data product can be automatically partitioned.

• Each pipeline and data product can be versioned.

• Backfilling data products is a core function and operation.

​

🪐 Scaling is made simple

Analyze and process large data quickly for rapid iteration.

• Transform very large datasets through a native integration with Spark.

• Handle data intensive transformations with built-in distributed computing (e.g. Dask, Ray) [coming soon].

• Run thousands of pipelines simultaneously and manage transparently through a collaborative UI.

• Execute SQL queries in your data warehouse to process heavy workloads.