You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A selective-recompute runtime can pass TrainerRank's activation-memory admission check and then OOM on its first forward: the current static estimate is calibrated for full recompute. Reject recompute_granularity="selective" during TrainerRank construction with TrainerRankRuntimeSupportError and an actionable hint to restore full/uniform/one-layer recompute.
This is the first step for #913. Draft follow-up #915 models retained activations and removes the temporary selective-recompute refusal; it is stacked on this PR and awaits native GPU calibration. The provider and other Megatron entry points keep their existing behavior.
Validation:
All 276 tests across topology, TrainerRank validation, and split admission passed (269 initially; seven multiprocessing cases passed after correcting the stdin-based test launcher).
The selective-recompute rejection tests pass at TP 1/2/4 and all three fail against the original constructor. Full/disabled recompute acceptance is covered as well.
Changed-file Ruff lint, formatting, type checks, and uv lock --check pass.
Repository-wide type checking reports 23 existing diagnostics in untouched files.
Consolidated review record (automated agent review traffic removed 2026-09-17)
Change. Rejects recompute_granularity="selective" at TrainerRank construction with TrainerRankRuntimeSupportError and a hint to restore full/uniform/one-layer recompute: the static activation-memory admission estimate is calibrated for full recompute, so selective runtimes passed admission and OOMed on the first forward (#913). Other Megatron entry points are unchanged.
Review. Minsky and McCarthy posted source-correctness CLEAR at 8aa6d4f; Shannon posted LGTM. Verified: topology tests pass (14/14, new selective-refusal cells at TP 1/2/4), the base fails exactly the three selective cases as a negative control, 72 further constructor controls pass, Ruff/format/ty clean. Hosted quality checks and the 2x H200 job were pending at review time.
Non-blocking hint gaps: EP overlap normalizes full recompute to disabled; none is not a workaround; TrainerRank docstring omits the new refusal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A selective-recompute runtime can pass TrainerRank's activation-memory admission check and then OOM on its first forward: the current static estimate is calibrated for full recompute. Reject
recompute_granularity="selective"during TrainerRank construction withTrainerRankRuntimeSupportErrorand an actionable hint to restore full/uniform/one-layer recompute.This is the first step for #913. Draft follow-up #915 models retained activations and removes the temporary selective-recompute refusal; it is stacked on this PR and awaits native GPU calibration. The provider and other Megatron entry points keep their existing behavior.
Validation:
uv lock --checkpass.