Skip to content

Stage 8 exit audit: close three criteria gaps, three stale documents - #90

Open
AdamClemens wants to merge 3 commits into
mainfrom
audit/stage-8-exit
Open

Stage 8 exit audit: close three criteria gaps, three stale documents#90
AdamClemens wants to merge 3 commits into
mainfrom
audit/stage-8-exit

Conversation

@AdamClemens

Copy link
Copy Markdown
Owner

Stage 8's end-of-stage exit audit, and the fixes it produced.

make ci was green throughout — 1200 tests, 144 scenarios, 99% coverage — and three of the nine Completion Criteria were marked Met against checks that did not test what the criterion said. None needed judgement to find; each is a sentence the criterion had already written down. This is docs/practices.md's own "An exit audit reads each criterion to its last sentence", recurring on the one stage whose previous audit is why Criteria 6-9 exist at all.

The three criteria gaps

Criterion What it said What checked it
6 "A keyboard seek and a mouse drag each change window.renderer.snapshot()'s content" Neither scrub test calls snapshot(); both assert on PlaybackState.position, and one's docstring says so
8 "Checked directly against a real multi-hundred-frame pyflow record run" The CLI test used --max-frames 20; TASK-049's by-hand check used 25
1 Recording "imports neither rendering nor anything that transitively imports it" The criterion's own text admits this "was checked by hand at implementation time"

All three were true; none were gated. Each fix is mutation-tested.

The part worth reading

The Criterion 6 pair took three attempts, and the second one is the useful finding:

  1. "The pixels changed after seeking" — worthless. The scrub thumb moves with the index, so any seek repaints something whether or not the field was rebuilt.
  2. "A seek-reached frame matches an autoplay-reached frame at the same index" — the obvious repair, and it passed against a deliberately frozen field. Both sides of the comparison came from the same run, so freezing the field froze the reference too.
  3. What works: the reference comes from a separate play() window launched at the target frame (where that frame is index 0, drawn by the initial scene build, never by the seek path), compared over the field region only, cropping away the scrub bar. Both halves then fail correctly — against a frozen field, and against an off-by-one field, checked separately.

Recorded as a standing rule in docs/practices.md: a reference the run under test produced proves consistency, not correctness.

Stale documentation, all in files no Stage 8 task opened

  • docs/architecture/CLAUDE.md said Stage 8 was complete with "all five" criteria met — falsified two days after it was written by the reopening that added four more.
  • docs/repository-manifest.md said TASK-046/047 were "still unbuilt" — stale from the day it was written, since both landed alongside the TASK-045 the same sentence credits.
  • docs/architecture/sequences.md (Checked-by: stage-boundary) described only TASK-045/046/047, and still said playback had "no declared-field/scalar-colormap path yet" two days after TASK-051 built exactly that. All four reopening tasks landed without it being touched. Its Maintenance section now records why a task anchor cannot reach a task the file has never named.

Also

  • pyflow play --help documented none of the controls the subcommand exists for — pause, speed, scrub. README was the only place to find them. Now in the epilog, pinned by a test, including the line saying seeking stops at the loaded window.
  • Linux CI never ran the live-window tests. No DISPLAY, no xvfb, so all 8 display-guarded tests — every check Criterion 6 rests on — skipped there while passing on Windows. A green two-platform matrix was proving live-window behaviour on one platform. Now run under xvfb-run. This is the one change in this PR I could not verify locally; watching the Linux job is the point of the PR.
  • 19 stray checkpoint_*.pt from a 2026-09-07 manual run removed from the repository root; .gitignore widened to match the files, not only the default directory.

Deliberately not done

The Goal's "scrubbed to any point" is still satisfied only within the window one pyflow play invocation loads. The maintainer was asked at the audit and chose to defer. That is now recorded as a standing exclusion under Criterion 6 — where a decision goes — rather than in a task's Design decisions, where it was hiding. Building it needs a new task and a new criterion, not a quiet widening of Criterion 6.

1200 → 1209 tests. make ci green locally.

🤖 Generated with Claude Code

AdamClemens and others added 3 commits September 11, 2026 14:06
make ci was green throughout -- 1200 tests, 144 scenarios, 99% coverage
-- and three of Stage 8's nine Completion Criteria were marked Met
against checks that did not test what the criterion said. None needed
judgement to find; each is a sentence the criterion had already written
down. This is docs/practices.md's own "An exit audit reads each
criterion to its last sentence", recurring on the one stage whose
previous audit is why Criteria 6-9 exist at all.

Criterion 6's rendered-pixel clause ("a keyboard seek and a mouse drag
each change window.renderer.snapshot()'s content") was checked by
nothing: neither scrub test calls snapshot() at all, and the keyboard
one's own docstring says it proves PlaybackState.position. Two new
tests close it.

Criterion 8 named a scale nothing had ever run ("a real
multi-hundred-frame pyflow record run"); the CLI test used
--max-frames 20 and TASK-049's by-hand check used 25. Closed by a real
300-frame CLI run, plus a separate test pinning peak on-disk count
during the run -- the claim _advance_and_checkpoint's docstring makes
("as the recording grows, not only once it finishes") and which the
final file set cannot prove either way.

Criterion 1 rested on a by-hand check and said so in its own text.
Re-verified by hand here, then gated: a fresh-subprocess check over the
transitive closure for all four headless modules.

All three fixes mutation-tested before being trusted green. The
Criterion 6 pair took three attempts, and the second is the useful
part: comparing a seek-reached frame against an autoplay-reached frame
at the same index PASSED against a deliberately frozen field, because
both sides came from the same run and were equally frozen. The working
version takes its reference from a separate play() window launched at
the target frame, and compares the field region only, cropping away the
scrub bar. Recorded as a standing rule in docs/practices.md: a
reference the run under test produced proves consistency, not
correctness.

Documentation the stage invalidated and never updated, all in files no
Stage 8 task opened:

- docs/architecture/CLAUDE.md said "all five" criteria met, falsified
  two days later by the reopening that added four more.
- docs/repository-manifest.md said TASK-046/047 were "still unbuilt" --
  stale from the day it was written, since both landed alongside the
  TASK-045 the same sentence credits.
- docs/architecture/sequences.md (Checked-by: stage-boundary) described
  only TASK-045/046/047, and still said playback had "no
  declared-field/scalar-colormap path yet" two days after TASK-051
  built exactly that. All four reopening tasks landed without it being
  touched. Its Maintenance section now records why a task anchor cannot
  reach a task the file has never named.

Also: pyflow play --help documented none of the controls the subcommand
exists for (pause, speed, scrub) -- README was the only place to find
them, now pinned by a test. Linux CI had no DISPLAY and no xvfb, so all
8 display-guarded tests skipped there while passing on Windows, making
a green two-platform matrix prove live-window behaviour on one
platform; now run under xvfb-run. 19 stray checkpoint_*.pt from a
2026-09-07 manual run removed from the repository root, and .gitignore
widened to match the files rather than only the default directory.

Not done, deliberately, and now recorded where a decision goes rather
than in a task's Design decisions: the Goal's "scrubbed to any point"
is still satisfied only within the window one pyflow play invocation
loads. The maintainer was asked at the audit and chose to defer.
Building it needs a new task and a new criterion, not a quiet widening
of Criterion 6. A user meets the boundary in pyflow play --help.

Verified live, not only unit-tested: pyflow play --help and pyflow
record --help run and checked by eye; a real 300-frame capped record
run leaves exactly {0, 290, 295, 300} on disk from 61 written.

1200 -> 1209 tests. make ci green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
That paragraph claims every edit to it has landed in the same change as
the roadmap event it describes -- so the exit audit is an edit it owes.
It also needed the distinction it was inviting a reader to miss: this
section staying current says nothing about the three criteria and three
documents that did not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Linux CI crashed a worker outright on the first run that had a virtual
display -- no Python traceback, the hard process abort GLFW produces
rather than an exception -- while the run immediately before it passed
the identical test code. Two runs of the same tests with opposite
results is contention, not a test failure: 8 xdist workers were each
creating software-GL contexts against one Xvfb display.

--dist loadgroup plus an xdist_group("display") mark on all 10
window-opening tests keeps them on one worker. Ordinary tests still
distribute exactly as before; the group is 10 of 1209, so nothing else
slows down (make ci: 183s, against 197s and 214s on the two runs before
it).

This is a hypothesis under test, not a confirmed fix -- the crash was
never reproducible locally, so CI on this branch is the only place it
can be judged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant