Deterministic lineage & dependency tracking in the discover inventory (#23)

[matthewmoorcroft]

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

📦 Merge order (3-PR set): This is the base of a 2-PR stack — merge this before the discover-insights PR (agentic insights), which is based on this branch and consumes the lineage it adds. Independent of the Switch-parsing PR (Switch parser fix), which can merge in any order. When this merges, GitHub auto-retargets the discover-insights PR to main.

---

Closes #23.

What

Adds a purely deterministic (no LLM) lineage block to the discover-phase inventory (metadata/inventory.json), assembled from the already-parsed AdfDefinitions:

• Control lineage — ExecutePipeline caller→callee call edges (recursing into ForEach/If containers; unresolved callees from partial exports are recorded, not dropped).
• Data lineage — dataset producer→consumer edges (sink/outputs = producer; source/inputs + Lookup/Delete/GetMetadata = consumer), joined on resolved physical identity (schema.table or storage path), falling back to dataset name. Identity is null when it can't be resolved deterministically — never a guess.

Design: compose, don't duplicate

The physical-identity resolvers already existed in translator/activity_translators/copy.py but ran only at convert time. Rather than duplicate them, they are extracted into a shared parser-layer module parser/dataset_resolvers.py that both convert (copy.py) and discover (parser/lineage.py) import. The relocation is behavior-preserving — copy.py's call sites are unchanged apart from imports, and the existing convert/copy tests are the safety net. A tiny shared read_execute_pipeline_ref helper keeps the parser and the convert-time translator reading the ExecutePipeline fields identically.

Example output

"lineage": {
  "control_edges": [
    {"caller_pipeline": "pipeline_execute_pipeline_nested", "callee_pipeline": "pipeline_copy_sql_to_delta",
     "activity_name": "Run Ingestion Pipeline", "wait_on_completion": true}
  ],
  "data_edges": [
    {"dataset_name": "ds_curated_orders", "identity": "curated.orders",
     "producer_pipeline": "child_a", "producer_activity": "Write", 
     "consumer_pipeline": "parent", "consumer_activity": "Lookup latest"}
  ]
}

The block always emits control_edges/data_edges as lists (empty, never null) for stable golden diffs.

Testing

• New tests/unit/test_lineage.py (control fan-out, nested-in-ForEach recursion, unresolved callee, data-edge direction, identity join across differently-named datasets, identity: null fallback, no self-edge, no duplicate edges) and tests/unit/test_dataset_resolvers.py (identity resolution incl. parameterized→null).
• Serialization + backward-compat tests extended in test_adf_loader.py; reporting/coverage.py reads only pipelines/summary and is unaffected.
• Full unit suite: 738 passed, 1 skipped; make fmt (ruff + mypy) clean.
• Verified end-to-end: a real discover run over the test fixtures emits the lineage block with 6 control edges and 5 data edges.

Scope / non-goals

Deterministic extraction only. Explicitly out of scope, tracked separately:

• Consuming control lineage for cross-pipeline deploy/run ordering → [FEATURE]: Ordered cross-pipeline deploy/run from control lineage (package phase) #24 (open, depends on this).
• Agentic "holistic view" enrichment → [FEATURE]: Agentic insights in inventory.json (discover phase) #25 (open, depends on this).
• Column-level lineage; Switch-case-body traversal (a pre-existing AST limitation shared with the current inventory walker).

This pull request and its description were written by Isaac.

---

Validated against a real factory + a limitation this surfaced

Ran the extractor against a real customer ADF factory (Factory A — 6 pipelines, 163 activities, the one in #23's evidence):

• Control lineage — production-quality. Correctly extracted all 18 ExecutePipeline fan-out calls from the master orchestrator to its 5 child pipelines (nested inside the orchestrator's control flow), 0 unresolved callees. Exactly the documented shape.
• Data lineage — a real limitation confirmed and handled. Every dataset in Factory A is a parameterized file dataset (@dataset() folderPath/fileName, build tokens) with no literal table/path, so all resolve to identity: null. Initial name-fallback join produced 416 data edges, 396 of them spurious: one dummy dataset is reused by ~40 activities that each point it at a different physical file via parameters — so joining on the shared ADF name manufactured hand-offs that don't exist.

Resolution (in this PR): data edges now join only on resolved physical identity (schema.table / concrete storage path), never on dataset name. An edge therefore always means a provable physical hand-off. Factory A data edges: 416 → 0 (all were false); control edges unchanged.

Limitation (by design)

Data lineage forms only where a dataset's physical identity is statically resolvable. Parameterized/expression-driven paths — common in real factories (Factory A is 100% this) — yield identity: null and therefore no data edge; the true coupling (e.g. the per-entity watermark CSV hand-off) is knowable only after runtime substitution, which the "never guess" rule declines to fake. Recovering those runtime-only couplings is exactly the remit of the agentic enrichment follow-up (#25), which can read the @concat(...) expressions and annotate the logical hand-off. Control lineage has no such limitation (it keys off static ExecutePipeline references).

---

Update: data edges now use a two-tier deterministic join (recovers parameterized hand-offs)

Validation on Factory A showed identity-only join missed a real, common pattern: the CDC watermark hand-off (a per-entity version CSV written by one activity and read back by another) uses fully parameterized paths, so identity is null and no edge formed. This is exactly the coupling #23's evidence said "get wrong and the CDC increment breaks silently."

Data edges now join on two deterministic tiers, tagged per edge via match_kind + match_key:

1. identity (high confidence) — both ends resolve to the same physical asset (schema.table / concrete storage path).
2. expression (structural) — when identity is unresolvable, both ends build the same normalized path signature: literal path segments kept, runtime references (pipeline().parameters.X, item().X, variables/dataset/activity refs) collapsed to <P> slots. Anchored on a literal folder segment (a file extension alone is too weak), so it recovers genuine hand-offs without over-matching reused dummy datasets.

Each DataEdge carries match_kind ("identity"/"expression") and match_key (the identity or the signature) so consumers and the #25 enrichment know the confidence tier and what coupled the two activities.

Re-validated on both real factories:

• Factory A: 25 data edges, all expression — 24 on the watermark path (the [FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23 hand-off, recovered), 1 reference-data lookup. No dummy-dataset explosion (the 396 false name-joins stay gone; the dummy logging writes split off by their distinct literal path).
• Factory B (327 pipelines): 0 data edges — no parameterized file hand-offs; the new tier adds zero false positives at scale.

A residual limitation remains: expression edges are a structural match (same literal folder skeleton + slot shape), not a proven physical identity — they can in principle coincide, which is why they're tagged as a distinct, lower-confidence tier for #25 to weigh. Fully resolving the runtime value (e.g. which entity) is still out of scope.