> ## Documentation Index > Fetch the complete documentation index at: https://docs.mage.ai/llms.txt > Use this file to discover all available pages before exploring further. # Data lineage > Configure Autopilot-managed lineage hooks for Mage Pro projects. export const ProOnly = ({button = 'Get started for free', description = 'Try our fully managed solution to access this advanced feature.', source = 'documentation', title = 'Only in Mage Pro.'}) =>
{title}

{description}

 
{button}
; ## Overview Mage Pro can notify Autopilot when project code and pipeline configuration change. Autopilot owns graph indexing, graph storage, and lineage display data. Mage Pro's responsibility is limited to: * project opt-in through `metadata.yaml`; * metadata-only change events after successful project mutations; * OpenLineage-compatible event envelopes for Autopilot ingestion; and * short-lived source snapshots when Autopilot needs source context. Mage Pro does not send block source, file contents, environment variables, credentials, run variables, logs, block outputs, or row-level data in lineage events. Source is available only through the snapshot API after Autopilot presents a short-lived grant scoped to the customer cluster, workspace, project, and snapshot. ## Enable lineage Lineage requires both cluster-level service configuration and project-level metadata opt-in. Set these values in the Mage Pro cluster environment. Keep credentials in the cluster secret manager instead of project files. | Variable | Required | Purpose | | ------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `AUTOPILOT_LINEAGE_ENABLED` | Yes | Enables the Mage Pro lineage integration for the cluster. Set to `1`. | | `MAGE_PRO_API_ENDPOINT` | Yes, unless `AUTOPILOT_LINEAGE_API_URL` is set | Mage Pro API gateway host used for lineage calls. In development this normally points at staging. | | `MAGE_PRO_API_API_KEY` | Yes, unless `MAGE_AUTOPILOT_GATEWAY_ACCESS_TOKEN` is set | Mage Pro API key used to mint the Autopilot gateway token. | | `MAGE_AUTOPILOT_GATEWAY_ACCESS_TOKEN` | No | Optional pre-minted gateway token. When unset, Mage Pro mints a short-lived token from `MAGE_PRO_API_ENDPOINT` with `MAGE_PRO_API_API_KEY`. | | `AUTOPILOT_LINEAGE_API_URL` | No | Optional direct Autopilot lineage API base URL. When unset, lineage routes through Mage Pro API `/api/v1/autopilot/lineage/...`. | | `AUTOPILOT_LINEAGE_API_TOKEN` | Yes only with `AUTOPILOT_LINEAGE_API_URL` | Service token used for direct Mage Pro to Autopilot calls. | | `AUTOPILOT_LINEAGE_TENANT_ID` | Yes, unless `MAGE_AUTOPILOT_TENANT_ID` is set | Autopilot tenant identity sent with lineage event requests. | | `AUTOPILOT_LINEAGE_WORKSPACE_UUID` | Yes, unless `MAGE_AUTOPILOT_WORKSPACE_ID` or `MAGE_WORKSPACE_UUID` is set | Durable workspace identity used with the customer cluster and project UUID. | | `AUTOPILOT_LINEAGE_API_KEY` | No | Optional additional service credential expected by Autopilot. | | `AUTOPILOT_LINEAGE_ROUTINE_ID` | No | Logical Mage Pro lineage caller sent to Autopilot. Defaults to `mage-pro-lineage`. | | `AUTOPILOT_LINEAGE_FEATURE_FLAG` | No | Optional project `features` key or global boolean gate applied after `lineage.enabled`. | | `AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_SECRET` | Yes for source snapshots | Shared verification secret for short-lived Autopilot snapshot grants. | | `AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_ISSUER` | No | Expected grant issuer. Defaults to `autopilot-lineage`. | | `AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_AUDIENCE` | No | Expected grant audience. Defaults to `mage-pro-lineage-snapshot`. | Then enable lineage for a project: ```yaml theme={"system"} lineage: enabled: true ``` The **Data lineage** app at `/apps/lineage/dashboard` provides the same opt-in by writing `lineage.enabled: true` to the active project's `metadata.yaml`. Lineage events are sent only when: * `AUTOPILOT_LINEAGE_ENABLED=1`; * the active project has `lineage.enabled: true`; * either direct Autopilot delivery or Mage Pro API gateway delivery is configured; * tenant ID, customer cluster UUID, workspace UUID, and project UUID are all available; and * the optional `AUTOPILOT_LINEAGE_FEATURE_FLAG` gate passes, when configured. For gateway delivery, Mage Pro prefers the explicitly configured `MAGE_PRO_API_ENDPOINT` over any agent-development service link so lineage traffic continues through the intended Mage Pro API environment. For direct delivery, set both `AUTOPILOT_LINEAGE_API_URL` and `AUTOPILOT_LINEAGE_API_TOKEN`. `MAGE_CLUSTER_UUID` must be explicitly configured, nonblank, durable, and unique for the customer cluster. Mage Pro does not use Mage's implicit `mage` fallback for lineage scope. ## Optional feature gate `lineage.enabled` is the project opt-in. `AUTOPILOT_LINEAGE_FEATURE_FLAG` is an additional rollout gate. If `AUTOPILOT_LINEAGE_FEATURE_FLAG` is unset, any project with `lineage.enabled: true` is eligible. Its values also support cluster-wide or feature-flag behavior: * `1`, `true`, `yes`, or `on` allows all opted-in projects. * `0`, `false`, `no`, or `off` disables lineage for every project. * Any other value is treated as a `features` key in project `metadata.yaml`. For example: ```yaml theme={"system"} lineage: enabled: true features: data_lineage: true ``` With `AUTOPILOT_LINEAGE_FEATURE_FLAG=data_lineage`, both keys must be true. ## What sends events Mage Pro logs and schedules a non-blocking lineage event delivery attempt after these successful changes: * project files and block contents; * pipelines, blocks, and execution-framework resources; * triggers and schedules; * successful deployment application. AI Sidekick block edits use the same persistence path as project mutations: they update the block content through the block model, create file versions, refresh pipeline and block caches, and publish the lineage hook after the cache refresh succeeds. Each event has an immutable ID and the durable scope `{customer_cluster_uuid, workspace_uuid, project_uuid}`. Mage Pro does not store lineage events locally. Autopilot owns durable ingestion, queueing, graph processing, and graph storage. A failed lineage delivery does not change the result of the original Mage Pro mutation. ## Event payloads Mage Pro sends its internal metadata-only event plus an `openlineage` envelope for Autopilot. The OpenLineage envelope uses the same durable namespace for the project scope and represents pipeline or block mutations as job metadata updates. File mutations are represented as dataset metadata updates. A Mage-specific facet carries the event ID, mutation type, object identity, safe payload hashes, and scope. The OpenLineage payload intentionally excludes source code, credentials, config secrets, and data values. Autopilot can request a source snapshot separately when it needs code context for graph rebuilds. ## Source snapshots Source snapshots are a service-to-service API for Autopilot. Browser sessions cannot call these endpoints. Every request requires an HS256 Bearer grant signed with `AUTOPILOT_LINEAGE_SNAPSHOT_GRANT_SECRET`. | Endpoint | Required grant action | Behavior | | ------------------------------------------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /api/v1/lineage/source-snapshots` | `lineage:source_snapshot:create` | Creates a manifest with scope, generation, expiry, project metadata, pipeline/block topology, integration and trigger references, owner descriptors, byte counts, and a manifest hash. | | `GET /api/v1/lineage/source-snapshots/:snapshot_id/owners` | `lineage:source_snapshot:read` | Returns paginated owner descriptors. `cursor` is a non-negative integer; `limit` defaults to 100 and is capped at 500. | | `POST /api/v1/lineage/source-snapshots/:snapshot_id/contents` | `lineage:source_snapshot:read` | Accepts bounded owner descriptors and returns redacted content only when the source still matches the manifest expectations. | Grants must contain `iss`, `aud`, `sub`, `iat`, and `exp` claims, an `actions` or `permissions` claim, and a `lineage_scope` equal to the target customer cluster, workspace, and project. A read grant must also contain the exact `snapshot_id` in the request path. Mage Pro rejects a token whose lifetime is longer than twice the configured snapshot TTL. Mage Pro applies these source controls: * file and total-source byte limits; * a cap on filesystem entries inspected while discovering project source files; * only files under the project root; * an explicit project-file allowlist; * redaction of common secrets, authorization values, and bearer tokens; * stable owner keys based on project UUID and object identity; and * `source_changed` or `content_changed` markers instead of stale content when a file changes after the manifest was created. Snapshot data is kept in process memory and becomes inaccessible after `AUTOPILOT_LINEAGE_SNAPSHOT_TTL_SECONDS` (15 minutes by default). If Mage Pro restarts, Autopilot must request a new snapshot. ## Capacity settings Use these controls when a project is unusually large: | Variable | Default | Purpose | | ------------------------------------------------------ | --------- | ------------------------------------------------------------------------------------ | | `AUTOPILOT_LINEAGE_EVENT_TIMEOUT_SECONDS` | `5` | Outbound Autopilot request timeout. | | `AUTOPILOT_LINEAGE_MAX_FILE_BYTES` | `524288` | Largest source file included in a snapshot. | | `AUTOPILOT_LINEAGE_MAX_SOURCE_SCAN_ENTRIES` | `50000` | Maximum filesystem entries inspected while finding allowlisted project-source files. | | `AUTOPILOT_LINEAGE_MAX_SOURCE_BYTES` | `5242880` | Total redacted source bytes included in one snapshot. | | `AUTOPILOT_LINEAGE_MAX_SNAPSHOT_OWNERS` | `10000` | Maximum owners included in one snapshot manifest. | | `AUTOPILOT_LINEAGE_SNAPSHOT_TTL_SECONDS` | `900` | Lifetime, in seconds, of a snapshot manifest. | | `AUTOPILOT_LINEAGE_SNAPSHOT_EXPIRED_RETENTION_SECONDS` | `86400` | How long expired snapshots remain only to return a stable `410` before cleanup. |