Data lineage

Overview

Mage Pro can notify Autopilot when project code and pipeline configuration change. Autopilot owns graph indexing, graph storage, and lineage display data. Mage Pro’s responsibility is limited to:

project opt-in through metadata.yaml;
metadata-only change events after successful project mutations;
OpenLineage-compatible event envelopes for Autopilot ingestion; and
short-lived source snapshots when Autopilot needs source context.

Mage Pro does not send block source, file contents, environment variables, credentials, run variables, logs, block outputs, or row-level data in lineage events. Source is available only through the snapshot API after Autopilot presents a short-lived grant scoped to the customer cluster, workspace, project, and snapshot.

Enable lineage

Lineage requires both cluster-level service configuration and project-level metadata opt-in. Set these values in the Mage Pro cluster environment. Keep credentials in the cluster secret manager instead of project files.

Variable	Required	Purpose
`AUTOPILOT_LINEAGE_ENABLED`	Yes	Enables the Mage Pro lineage integration for the cluster. Set to `1`.
`MAGE_PRO_API_ENDPOINT`	Yes, unless `AUTOPILOT_LINEAGE_API_URL` is set	Mage Pro API gateway host used for lineage calls. In development this normally points at staging.
`MAGE_PRO_API_API_KEY`	Yes, unless `MAGE_AUTOPILOT_GATEWAY_ACCESS_TOKEN` is set	Mage Pro API key used to mint the Autopilot gateway token.
`MAGE_AUTOPILOT_GATEWAY_ACCESS_TOKEN`	No	Optional pre-minted gateway token. When unset, Mage Pro mints a short-lived token from `MAGE_PRO_API_ENDPOINT` with `MAGE_PRO_API_API_KEY`.
`AUTOPILOT_LINEAGE_API_URL`	No	Optional direct Autopilot lineage API base URL. When unset, lineage routes through Mage Pro API `/api/v1/autopilot/lineage/...`.
`AUTOPILOT_LINEAGE_API_TOKEN`	Yes only with `AUTOPILOT_LINEAGE_API_URL`	Service token used for direct Mage Pro to Autopilot calls.
`AUTOPILOT_LINEAGE_TENANT_ID`	Yes, unless `MAGE_AUTOPILOT_TENANT_ID` is set	Autopilot tenant identity sent with lineage event requests.
`AUTOPILOT_LINEAGE_WORKSPACE_UUID`	Yes, unless `MAGE_AUTOPILOT_WORKSPACE_ID` or `MAGE_WORKSPACE_UUID` is set	Durable workspace identity used with the customer cluster and project UUID.
`AUTOPILOT_LINEAGE_API_KEY`	No	Optional additional service credential expected by Autopilot.
`AUTOPILOT_LINEAGE_ROUTINE_ID`	No	Logical Mage Pro lineage caller sent to Autopilot. Defaults to `mage-pro-lineage`.
`AUTOPILOT_LINEAGE_FEATURE_FLAG`	No	Optional project `features` key or global boolean gate applied after `lineage.enabled`.
`AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_SECRET`	Yes for source snapshots	Shared verification secret for short-lived Autopilot snapshot grants.
`AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_ISSUER`	No	Expected grant issuer. Defaults to `autopilot-lineage`.
`AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_AUDIENCE`	No	Expected grant audience. Defaults to `mage-pro-lineage-snapshot`.

Then enable lineage for a project:

lineage:
  enabled: true

The Data lineage app at /apps/lineage/dashboard provides the same opt-in by writing lineage.enabled: true to the active project’s metadata.yaml. Lineage events are sent only when:

AUTOPILOT_LINEAGE_ENABLED=1;
the active project has lineage.enabled: true;
either direct Autopilot delivery or Mage Pro API gateway delivery is configured;
tenant ID, customer cluster UUID, workspace UUID, and project UUID are all available; and
the optional AUTOPILOT_LINEAGE_FEATURE_FLAG gate passes, when configured.

For gateway delivery, Mage Pro prefers the explicitly configured MAGE_PRO_API_ENDPOINT over any agent-development service link so lineage traffic continues through the intended Mage Pro API environment. For direct delivery, set both AUTOPILOT_LINEAGE_API_URL and AUTOPILOT_LINEAGE_API_TOKEN. MAGE_CLUSTER_UUID must be explicitly configured, nonblank, durable, and unique for the customer cluster. Mage Pro does not use Mage’s implicit mage fallback for lineage scope.

Optional feature gate

lineage.enabled is the project opt-in. AUTOPILOT_LINEAGE_FEATURE_FLAG is an additional rollout gate. If AUTOPILOT_LINEAGE_FEATURE_FLAG is unset, any project with lineage.enabled: true is eligible. Its values also support cluster-wide or feature-flag behavior:

1, true, yes, or on allows all opted-in projects.
0, false, no, or off disables lineage for every project.
Any other value is treated as a features key in project metadata.yaml.

For example:

lineage:
  enabled: true
features:
  data_lineage: true

With AUTOPILOT_LINEAGE_FEATURE_FLAG=data_lineage, both keys must be true.

What sends events

Mage Pro logs and schedules a non-blocking lineage event delivery attempt after these successful changes:

project files and block contents;
pipelines, blocks, and execution-framework resources;
triggers and schedules;
successful deployment application.

AI Sidekick block edits use the same persistence path as project mutations: they update the block content through the block model, create file versions, refresh pipeline and block caches, and publish the lineage hook after the cache refresh succeeds. Each event has an immutable ID and the durable scope {customer_cluster_uuid, workspace_uuid, project_uuid}. Mage Pro does not store lineage events locally. Autopilot owns durable ingestion, queueing, graph processing, and graph storage. A failed lineage delivery does not change the result of the original Mage Pro mutation.

Event payloads

Mage Pro sends its internal metadata-only event plus an openlineage envelope for Autopilot. The OpenLineage envelope uses the same durable namespace for the project scope and represents pipeline or block mutations as job metadata updates. File mutations are represented as dataset metadata updates. A Mage-specific facet carries the event ID, mutation type, object identity, safe payload hashes, and scope. The OpenLineage payload intentionally excludes source code, credentials, config secrets, and data values. Autopilot can request a source snapshot separately when it needs code context for graph rebuilds.

Source snapshots

Source snapshots are a service-to-service API for Autopilot. Browser sessions cannot call these endpoints. Every request requires an HS256 Bearer grant signed with AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_SECRET.

Endpoint	Required grant action	Behavior
`POST /api/v1/lineage/source-snapshots`	`lineage:source_snapshot:create`	Creates a manifest with scope, generation, expiry, project metadata, pipeline/block topology, integration and trigger references, owner descriptors, byte counts, and a manifest hash.
`GET /api/v1/lineage/source-snapshots/:snapshot_id/owners`	`lineage:source_snapshot:read`	Returns paginated owner descriptors. `cursor` is a non-negative integer; `limit` defaults to 100 and is capped at 500.
`POST /api/v1/lineage/source-snapshots/:snapshot_id/contents`	`lineage:source_snapshot:read`	Accepts bounded owner descriptors and returns redacted content only when the source still matches the manifest expectations.

Grants must contain iss, aud, sub, iat, and exp claims, an actions or permissions claim, and a lineage_scope equal to the target customer cluster, workspace, and project. A read grant must also contain the exact snapshot_id in the request path. Mage Pro rejects a token whose lifetime is longer than twice the configured snapshot TTL. Mage Pro applies these source controls:

file and total-source byte limits;
a cap on filesystem entries inspected while discovering project source files;
only files under the project root;
an explicit project-file allowlist;
redaction of common secrets, authorization values, and bearer tokens;
stable owner keys based on project UUID and object identity; and
source_changed or content_changed markers instead of stale content when a file changes after the manifest was created.

Snapshot data is kept in process memory and becomes inaccessible after AUTOPILOT_LINEAGE_SNAPSHOT_TTL_SECONDS (15 minutes by default). If Mage Pro restarts, Autopilot must request a new snapshot.

Capacity settings

Use these controls when a project is unusually large:

Variable	Default	Purpose
`AUTOPILOT_LINEAGE_EVENT_TIMEOUT_SECONDS`	`5`	Outbound Autopilot request timeout.
`AUTOPILOT_LINEAGE_MAX_FILE_BYTES`	`524288`	Largest source file included in a snapshot.
`AUTOPILOT_LINEAGE_MAX_SOURCE_SCAN_ENTRIES`	`50000`	Maximum filesystem entries inspected while finding allowlisted project-source files.
`AUTOPILOT_LINEAGE_MAX_SOURCE_BYTES`	`5242880`	Total redacted source bytes included in one snapshot.
`AUTOPILOT_LINEAGE_MAX_SNAPSHOT_OWNERS`	`10000`	Maximum owners included in one snapshot manifest.
`AUTOPILOT_LINEAGE_SNAPSHOT_TTL_SECONDS`	`900`	Lifetime, in seconds, of a snapshot manifest.
`AUTOPILOT_LINEAGE_SNAPSHOT_EXPIRED_RETENTION_SECONDS`	`86400`	How long expired snapshots remain only to return a stable `410` before cleanup.

Monitoring

Alerting

Logging

Testing

Overview

Enable lineage

Optional feature gate

What sends events

Event payloads

Source snapshots

Capacity settings

​Overview

​Enable lineage

​Optional feature gate

​What sends events

​Event payloads

​Source snapshots

​Capacity settings

Overview

Enable lineage

Optional feature gate

What sends events

Event payloads

Source snapshots

Capacity settings