Account for known eager MoE FC2 input and output storage

[bradhilton]

The MoE FC2 memory estimate counts its base and adapter outputs, but the recognized eager path also keeps the input and creates their combined output. With compatible LoRA input metadata, price those known tensors as topk * (input_width + 3 * output_width) * dtype_bytes. Unknown input metadata keeps the existing fallback; execution-type and topology exclusions remain unchanged.

This is a partial correction for #848, which remains open. The runtime change stays within art.trainer_rank; public APIs and art.megatron are unchanged. It also raises some warm admission and retention estimates when the new static term dominates observed profiles. Hold for Brad’s behavioral decision; do not merge automatically. It is not a complete or compiled-model memory bound.

Validation:

• All 35 focused native CPU cases passed in the pinned runtime image, with CUDA uninitialized and exact resource cleanup verified. The implementation and test blobs here are identical to those tested.
• In two fixed-input H200 pressure runs, the earlier version admitted and then hit a forward OOM; the helper-only successor refused before any forward. Inputs, initial checkpoint values, and the 63.75 GiB pressure allocation matched. Available-memory readings differed, so allocator-state equality is not claimed. The successor’s nonzero task exit is preserved; its refusal evidence and cleanup passed independent review.
• Ten additional scalar controls cover warm retention behavior, inherited malformed-metadata errors, unknown-layer fallback, and limits of the high-rank workspace estimate. These do not establish additional GPU coverage.
• Ruff, formatting, and lock checks pass. Local full type checking reports the same 11 diagnostics on this head and an isolated unchanged base. Hosted CI remains pending.

The physical test used a frozen qualification harness with only this helper delta applied; it is not qualification of the whole current main tree. Additional workspace, arbitrary active ranks, compiled liveness, other topologies, and general admission correctness remain unqualified. The exception’s rejected-split estimate is not presented as an observed full-batch plan.

Private durable evidence: /home/brad/.local/share/schulman/art848-fc2-integration-20260912/validation.json; /home/brad/.local/share/schulman/art848-fc2-pressure-launch-root-review-20260912/final-result/REPORT.md; /home/brad/.local/share/schulman/art848-fc2-fallback-audit-20260912-memory/. Original failures and frozen source packages are retained.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. The MoE FC2 memory estimate counted only base and adapter outputs, but the eager path also keeps the input and their sum. With compatible LoRA metadata, art.trainer_rank now prices all four tensors; unknown metadata keeps the prior fallback. Partial correction for #848.

Review. Minsky, McCarthy and Taravangian reviewed merged head 75d984c; all three gave source-correctness CLEAR within the partial eager-FC2 scope, with no public API or art.megatron change, but flagged a significant admission-behavior change (stricter warm admission and retention) for Brad's decision. Minsky ran the MoE memory unit tests (36 passed); McCarthy verified the blobs match the natively tested source; retained evidence showed 35 native CPU passes, ten scalar controls, and an H200 pressure run refusing before forward where the prior version OOMed. Lint, format and lock checks passed; type checking matched base (11 diagnostics); hosted CI was pending.

Deferred / follow-ups.

• TrainerRank MoE memory admission: backward-recompute OOM and profile calibration failures #848 remains open; rank-dependent workspace, compiled storage reuse and other topologies remain unqualified.