fix(agentic): leak-check cannot see orphaned Aspire process descendants, so it reports clean through a real leak

[rickylabs]

Recorded as C-D86 (plus two addenda) in the 0.0.5 orchestration drift log. Demonstrated twice, not inferred.

Evidence

• Row 70 (scaffold.runtime RED): an aspire-managed nuget search survived with PPID 1 for 1h53m, working-dir inside that run's own workspace plugin-smoke-20260809-204236. That run's receipt had recorded a secondary TypeError: Child process has already terminated during teardown, which was read at the time as cosmetic noise. It was not — the teardown had lost track of a live child.
• Row 74 (scaffold.runtime GREEN): six helpers from that run's own workspace plugin-smoke-20260809-224559 were still running 17 minutes after it finished, while the same receipt recorded cleanup.aspire-stop passing in 1.003s and run-owned survivors: 0.
• Census at that moment across sibling worktrees: 16 in ns005-w2c, 5 in ns005-w3a, 3 in ns005-w3b2, all ~20-21h old. This is systemic, not a one-off.

The defect

agentic:leak-check measures containers and AppHosts. Detached process descendants are outside what it can observe, so "zero run-owned survivors" is true about containers and simultaneously compatible with several run-owned processes still executing. A check that cannot observe the failure it is trusted to rule out is worse than no check, because it is quoted as proof.

Concretely: every scaffold.runtime execution appears to leak these helpers, and no gate reports it.

Important caveat for whoever implements this

Do not reclaim on PPID 1 plus path containment alone. Aspire spawns these helpers detached by design, so PPID 1 is normal operation — during the row-74 run itself, 40 such processes belonged to the live gate. A tool that reclaimed on containment alone would have killed an authorized run mid-flight.

Reclaim only when all three hold:

1. path containment in the slice worktree (proves ownership),
2. the workspace directory is not the active run's (proves the owning run ended),
3. age far exceeds the operation's normal duration (a nuget search is sub-second).

That is the rule used to clean these safely by targeted PID; foreign helpers in sibling worktrees were left untouched.

Wording correction owed to existing evidence

Expensive-gate ledger rows 67, 69, 73 and 75 each state "zero run-owned survivors" on the strength of this blind check. They should be read as zero run-owned containers and AppHosts. No gate verdict changes — the leaked helpers are idle nuget search calls, not resource holders — but the claims are narrower than their wording implies.

Refs #1333.

[rickylabs]

Additional reproduction — 317 orphaned descendants survived --cleanup and starved a canonical gate

Independent reproduction from the #1443/#1445 lane, offered as evidence on this issue rather than a duplicate.

Observed. After ~10 deno task e2e:cli run scaffold.runtime --cleanup invocations in one worktree, 317 orphaned aspire-managed descendants were alive. --cleanup had run to completion each time and docker ps was empty, so container teardown worked; the process descendants did not.

leak-check reported clean throughout — the visibility gap this issue describes. It surfaced only through ps -eo pid,cmd | grep aspire-managed.

Consequence, which is the part worth adding. This is not only untidy: the orphans starved NuGet restore until the canonical gate failed.

> runtime.aspire-restore: Restore Aspire TypeScript SDK
  FAILED 540629ms (attempt durations: 180.3s + 180.1s + 180.3s; failure class: infrastructure)
    ❌ Failed to prepare: A task was canceled.

Two consecutive scaffold.runtime runs aborted this way. Killing the 24 run-owned orphans by path containment cleared it immediately and the next run reached passed=84 failed=0. So the leak is a self-inflicted gate failure, and any agent iterating on scaffold.runtime will hit it.

Machine-level backlog. 90 further aspire-managed processes from other runs (ns005-w2c, ns005-w3a, w6-billing-run-sol-high-pass3, oldest 2026-08-09) were present and left untouched per the foreign-resource rule. They degrade every Aspire-dependent gate on the host, which suggests the accumulation is not confined to one lane.

Suggested scope amendment, if it is not already implied: teardown should be able to reap run-owned process descendants by path containment — the same ownership proof it already uses for containers — since that is what made the manual cleanup safe here. Ownership was provable in every case: docker inspect mount paths and process cmdlines both contained the owning worktree.

Rate: roughly 30 orphans per suite run.

[rickylabs]

Correction and upstream root cause: this is microsoft/aspire#18948, and my earlier causal framing was half wrong

Following up on my 317-process report above. I went looking for the upstream issue rather than leaving the diagnosis at "our leak", and it exists.

Upstream: microsoft/aspire#18948 — "aspire ls leaks orphaned aspire-managed nuget search helpers that accumulate over days and deadlock unrelated dotnet restore via NuGet lock contention." It cites #18779 (same root cause, different command/platform) and #18050 (closed — PR #18566 did not fix this path), and confirms the same behaviour on WSL, which is this host.

The mechanism, from the related #12672: an orphaned aspire-managed nuget search process becomes suspended while holding NuGet's exclusive HTTP-cache lock. Any subsequent dotnet restore then blocks indefinitely on that lock. The upstream reporter's workaround is the one I stumbled into: kill the orphans and the stuck restore completes immediately.

What that changes about my earlier comment

I framed the 317 orphans as our leak starving NuGet. That was half right, and the half that was wrong matters for whoever picks this up:

• The leak is upstream. Aspire spawns these helpers detached by design and does not reap them. Nothing in our teardown created them, and the reason PPID 1 plus path containment is unsafe to reclaim on — already documented in this issue's body — is the same design fact.
• The visibility gap is ours, and it is exactly what this issue's title says. leak-check reporting clean through a live leak is the NetScript defect. The leak itself is not.

So the scope here should stay as filed — visibility and safe reclamation — and should not grow into "stop Aspire leaking", which we cannot fix.

Likely duplicate of a closed issue

#1227 (runtime.aspire-restore intermittently hangs 15min to timeout against NuGet feeds, closed) is very probably this same upstream bug rather than an independent flaky-feed problem. Its recorded signature — restore hanging for the full gate timeout while api.nuget.org and pkgs.dev.azure.com/dnceng probe sub-second minutes later — is the lock-contention symptom, not network flakiness. I hit the identical signature twice during the #1443/#1445 lane (three 180s attempts, exit 6 Failed to prepare AppHost server, both feeds responding in ~0.2s), and clearing this run's own orphans cleared it.

That matters because #1227's accepted mitigation (pre-seeded offline NuGet cache) treats the symptom. Worth a note there so the next person does not re-diagnose it as a feed problem.

Not confirmed

I could not verify a merged upstream fix landing in Aspire 13.5. The 13.4 change log does ship "reliable cleanup of extension Aspire CLI child processes", but that is VS Code-extension scoped, and #18948 was filed against the CLI path after it. If someone has the 13.5 linkage, adding it here would let us date the point at which our reclamation logic can be simplified — but I am not asserting it on the evidence I found.

Suggested disposition

Keep this issue's scope. Add microsoft/aspire#18948 as the upstream reference, treat the leak as an upstream-owned precondition our tooling must survive, and keep the ask narrow: leak-check should see these descendants, and teardown should be able to reclaim run-owned ones by the path-containment proof it already applies to containers — without reclaiming on PPID 1 alone, since upstream's detached-by-design behaviour makes that unsafe.

[rickylabs]

Milestone audit: the latest evidence identifies microsoft/aspire#18948 as the upstream orphaned-helper root cause. Keep this open as an upstream-tracked cleanup/mitigation item in Backlog / Triage; it is not a framework-surface blocker for 0.0.6.