Coordinate rank-local planning failures before collective admission

[bradhilton]

A rank-local estimator, materializer, or memory-pricing error could leave another rank waiting in the next collective. Exchange each pure planning phase's success before proceeding, so peers fail promptly and the originating rank keeps its original exception and chain. Legal unavailable estimates still take the existing fallback path; empty and unequal local groups use the same collective sequence.

Fixes #906. Runtime changes stay within art.trainer_rank; public APIs, planner arithmetic, checkpoint ordering, and art.megatron are unchanged. This does not adopt #900. Memory-estimation work remains open in #848 and #870.

Validation: 153 distinct focused repository tests passed, including a two-process CPU/Gloo regression covering six failure/empty/unequal/unavailable cases and eight exception-chain controls. Changed-file Ruff, formatting, type checks, and diff checks pass. The exact patch also composes without conflict on main e9c68ccc, with identical changed-file blobs. Hosted CI is pending.

The simple scheduler path adds four status reductions; repeated width/layout checks can add more. Native NCCL and latency are not qualified. Checkpoint-loading errors, later CUDA-counter failures, and communicator failures remain outside this narrow fix. The original failed reproduction and its evidence are preserved in #906.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. Exchanges each planning phase's success between ranks in art.trainer_rank so a rank-local estimator, materializer, or memory-pricing error fails peers promptly rather than stranding them in the next collective; the originating rank keeps its original exception and chain. Fixes #906.

Review. McCarthy and Minsky reviewed head 5ea4548; both returned CLEAR on source correctness within scope, no changes requested. Minsky ran the new tests (44 passed, including the two-process Gloo case) plus 175 backend-stage tests with and without Megatron, and confirmed the new tests fail on the base runtime; McCarthy hash-verified the retained 153-case evidence and ran 42 control-flow checks. Both reproduced the composition onto main; CI was still running when the reviews posted.

Deferred / follow-ups.

• The four added WORLD MIN status reductions are unqualified under native NCCL; McCarthy left that performance judgment to Brad.
• Memory-estimation work remains open in TrainerRank MoE memory admission: backward-recompute OOM and profile calibration failures #848 and TrainerRank reusable-cache admission can leave insufficient physical CUDA memory for backward library allocations #870; Bound native CUDA cache recovery by measured forward work #900 not adopted.
• Checkpoint-loading, CUDA-counter, and communicator failures remain out of scope.