Tune the YOLO baseline's peak LR / warmup before reporting it — the collapse is an untuned default (#51)

[jonfroehlich]

Parent: #51 · Related: #70 (stabilized rerun), #71 (pre-registered protocol)

Why this is a fairness issue, not a performance one

The supervised-YOLO baseline currently runs Ultralytics defaults. Per the provenance
block in scripts/model_comparison/yolo_baseline/README.md, optimizer=auto resolves to
MuSGD(lr=0.01, momentum=0.9) in all six jobs — overriding the passed lr0/momentum —
and the realized schedule peaks at lr/pg0 = 0.029 at the end of warmup_epochs=3.0.

Every config's validation collapse begins at exactly that peak. That is documented with
figures (figures/fig1–fig4) and the record rules out the alternatives: not a crash (no
NaN/Inf, train loss falls straight through), not preemption (requeues don't line up), not
small batch (the y11l_pano / y26_pano batch=4 control), not architecture (both families
collapse and both recover), not a data fault.

The record's own conclusion is that the peak LR is too high for this task — 150k
single-class, small-object images fine-tuned from COCO weights — and it flags that
hypothesis as untested.

That leaves one hole in an otherwise airtight protocol. #71 exists to defend the baseline
against "you sandbagged it," and every selection decision is already made on validation
data under rules fixed in advance. But an untuned default that visibly destabilizes
training is still a sandbagging exposure: a reviewer can say we reported YOLO at a
learning rate its own curves show to be wrong, and they would be right. Until the
hypothesis is tested we cannot distinguish "YOLO is unstable on this task" from "we ran
YOLO at a bad LR."

Scope — deliberately bounded in both directions

Fairness runs both ways. RampNet's published recipe is itself lightly tuned (one epoch,
~9.4k steps, constant LR, no scheduler — see #84), so an exhaustive HPO on YOLO would
overcorrect and produce a differently-unfair comparison. The goal is parity of tuning
effort, not a maximally-tuned YOLO.

So: a small, principled grid on one config, decided here before it runs.

• Config: y11l_pano — it collapsed and recovered, so the effect is visible in both
  directions, and its ~2.6–2.8 h epoch is among the cheapest.
• Grid (6 runs): lr0 ∈ {0.01 (control), 0.005, 0.0025} × warmup_epochs ∈ {3 (control), 5}.
• Length: enough epochs to clear the collapse window and show the trend — ~12–15, not 60.
• Everything else held: batch=4, imgsz=1280, seed=0, nbs=64, same data, same
  optimizer=auto resolution path.

Pre-registered decision rules (fixed before any run)

1. Selection on val only, exactly as Pre-register the YOLO-baseline evaluation & checkpoint-selection protocol #71: internal YOLO val fitness at best.pt.
   No test/benchmark number is looked at during or after this sweep to choose a setting.
2. The sweep chooses a training config, not a reported number. Whatever wins is then run
   under Pre-register the YOLO-baseline evaluation & checkpoint-selection protocol #71 unchanged, and its benchmark row is reported alongside the default-LR rows —
   both stay in the table. The default-LR results are not retracted or replaced.
3. Success criterion, stated in advance: a setting "fixes" the instability iff its val
   curve shows no collapse-to-zero (no epoch at literal 0.000 after ep1) and its val
   fitness at the same epoch count is ≥ the control's. Anything less and we report the
   instability as a genuine property of YOLO on this task — which is a legitimate finding,
   not a failure of the experiment.
4. Negative result is publishable and will be recorded either way, per the repo's
   standing rule.

Companion: seeds

Protocol item 7 lists ≥3 seeds of the headline configs as aspirational, gated on ckpt
capacity — all runs are currently seed=0, n=1. Tillicum removes that gate (see
docs/tillicum.md). Seeds and this sweep share a justification: both convert
single-run instability commentary into a measured quantity. Worth running the seed arm on
whatever LR setting this issue selects, so the two land together.

Feasibility

On Tillicum (H200, non-preemptable, $0.90/GPU-hour): 6 runs × ~15 epochs. At an estimated
~1 h/epoch for a pano config — unmeasured, and the first thing docs/tillicum.md says to
measure — that is roughly 90 GPU-hours, on the order of $80, and it fits inside the
100 free demo hours. Re-cost this from the measured epoch time before committing.

This is not runnable on klone ckpt in its current state: five arms there have produced
zero completed epochs in the last ~14 h under an effective preemption interval of minutes
(see #51 status comments).

Acceptance

• Grid above run to ~12–15 epochs, results.csv + args.yaml committed to the record
• Val curves plotted alongside the control, same figure style as fig1
• Winner (or "no setting qualified") recorded in yolo_baseline/README.md
• If a setting qualifies: the headline configs re-run under it, reported under Pre-register the YOLO-baseline evaluation & checkpoint-selection protocol #71
  unchanged, with the default-LR rows retained
• README.md provenance updated to state which LR the reported numbers used