Core Design Principles

​

💻 Easy developer experience

• Mage comes with a specialized notebook UI for building data pipelines.
• Use Python and SQL (more languages coming soon) together in the same pipeline for ultimate flexibility.
• Set up locally and get started developing with a single command.
• Deploying to production is fast using native integrations with major cloud providers.

​

🚢 Engineering best practices built-in

• Writing reusable code is easy because every block in your data pipeline is a standalone file.
• Data validation is written into each block and tested every time a block is run.
• Operationalizing your data pipelines is easy with built-in observability, data quality monitoring, and lineage.
• Each block of code has a single responsibility: load data from a source, transform data, or export data anywhere.

​

💳 Data is a first class citizen

• Every block run produces a data product (e.g. dataset, unstructured data, etc.)
• Every data product can be automatically partitioned.
• Each pipeline and data product can be versioned.
• Backfilling data products is a core function and operation.

​

🪐 Scaling is made simple

• Transform very large datasets through a native integration with Spark.
• Handle data intensive transformations with built-in distributed computing (e.g. Dask, Ray) [coming soon].
• Run thousands of pipelines simultaneously and manage transparently through a collaborative UI.
• Execute SQL queries in your data warehouse to process heavy workloads.