Skip to content

fix(check): Edit/MultiEdit acquitted every added comment - #57

Merged
ryanleecode merged 5 commits into
masterfrom
fix/edit-fragment-restate-detection
Aug 24, 2026
Merged

fix(check): Edit/MultiEdit acquitted every added comment#57
ryanleecode merged 5 commits into
masterfrom
fix/edit-fragment-restate-detection

Conversation

@systemfsoftware-maker

Copy link
Copy Markdown
Collaborator

The defect

On Edit and MultiEdit, new_comments stamped every newly-added comment's CommentContext as unreliable. classify reads that flag twice — reliable_adjacent withholds the adjacent code from the context-aware detectors, and the terminal fallback then rewrites the resulting evidence-free RestatesCode into Justification::NonObviousIntent.

Net effect: on the tools an agent actually uses to modify an existing file, only the text-only rules could ever fire (AgentMemo, CommentedOutCode, VacuousTodo). RestatesCode and NarratesControlFlow were dead code there.

Identical comment text, two payload shapes, v0.1.5:

payload verdict
Write// set b to 2, // loop over items exit 2, both flagged
Edit — same text, comment mid-fragment with code above and below exit 0
MultiEdit — same exit 0

The third row is the one that rules out "conservative at the edge": the context was demonstrably not truncated, and it was still acquitted.

The fix

The adjudication plan specifies unreliability at the fragment edge (its risk register and DoD both say so); the implementation applied it to the whole fragment. This restores the documented scope:

ctx.unreliable = ctx.adjacent_code.is_none() || ctx.position == PositionRole::Trailing;

A fragment is a contiguous slice of file text, so a comment with a following code sibling inside the slice has that same sibling in the whole file — the boundary cannot have inserted code between them. Only two shapes are genuinely truncated: no adjacent code captured at all, and a comment derive_context could read only as Trailing, which it yields exactly when the next code sibling is absent.

classify.rs is untouched, so the classifier mutation gate is not triggered by this change.

Measured

The 60-case labelled corpus driven through the Edit seam, before vs after:

before after
false positives (justified but blocked) 0 0
false negatives (unnecessary but spared) 17 0
correct 43/60 60/60

The 17 recovered misses were 15 RestatesCode + 2 NarratesControlFlow. No justified case is newly blocked, so there is no false-positive cost against the corpus. The Write path is unchanged on all 60 with zero changed verdicts — the fix is confined to the adapted paths.

Operator mutation of the new predicate is killed: ||&& (3 failing tests), == Trailing!= (2), and each constant (2 and 3).

Why three green gates missed it

Coverage composes over (stage × entry path), not over stages alone:

  • the mutation gate pins the pure classifier, but the defect sat in the adapter that constructs the classifier's inputs — every mutant of the core still dies, because those tests build contexts directly;
  • the labelled corpus only ever synthesises a whole-content parse, so the edit paths had zero corpus coverage;
  • two composition tests had pinned the blanket acquittal as the contract, so the bug was a protected invariant.

tests/edit_path.rs closes the second one: the corpus driven through check as Edit payloads, asserting no justified case blocks and no unnecessary case is spared. Verified red on the pre-fix check.rs (17 spared, named) and green with the fix.

Contents

  • fix(check) — the edge-scoped predicate; two tests that asserted the old contract now assert Block, two new tests pin the genuinely-unreliable shapes
  • test(edit-path) — the corpus gate for the edit seam
  • test(common) — one payload builder; pipeline.rs and edit_path.rs had byte-identical JSON escapers
  • docs(plans) — a plan for a separate, larger defect found while measuring this one (below)
  • docs(solutions) — the gate-coverage learning

Found while measuring, not fixed here

A keyword acquits a proven restatement. Conviction is evidence-backed and default-deny; acquittal is bare substring membership. 13 of 18 paraphrases of // set x to 1 above const x = 1; are spared by adding one marker — because, note:, security, 1-based, ref:, based on. Padding is not the escape (+9 words of unrelated prose still blocks), so the entire surface is is_non_obvious_intent and is_attribution. Since JUSTIFIED is consulted before any evidence is computed, the precedence — not the table contents — is what makes it launderable. The plan in this PR inverts that precedence, narrows attribution to lead-position tags, and re-derives the corpus floors. Not started; eval/corpus.json is untouched by this PR.

Two related claims checked and found sound, so they are not holes: is_bdd compares for exact equality on the stripped text, and DIRECTIVE_PREFIXES carries only tool-specific tokens (// Allows the caller to… blocks). Unsupported extensions still pass unenforced, which is by design.

One unrelated observation: is_docstring_head over-promotes module-scope leading comments to DocstringHead in synthesised fragments. Harmless today because is_public_api_doc additionally filters on doc markup, and independent of this change.

Verification

cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets

Green: 61 unit, 10 classify, 2 exit_codes, 2 edit_path, 6 f1, 15 pipeline. Run in the repo's own nix develop shell.

An Edit/MultiEdit fragment marked every newly-added comment's context
unreliable, so classify's catch-all rewrote the verdict to Justified and
the hook passed. Restatement and flow-narration detection were dead on
the tool agents use to modify existing files: only AgentMemo,
CommentedOutCode and VacuousTodo could ever fire.

A fragment is contiguous file text, so a comment with code after it
inside the fragment annotates that code in the file too. Mark unreliable
only where the boundary could have taken the adjacent code away: no
adjacent code at all, or nothing after the comment. This restores the
edge-scoped rule the U2 plan specifies (plan lines 233, 248).

Gate: fmt, clippy -D warnings, 15 pipeline tests, F1 unchanged.
A keyword acquits a proven restatement: 13 of 18 paraphrases of one
restatement are spared by adding a marker word. Conviction is
evidence-backed and default-deny; acquittal is bare substring
membership. The plan inverts that precedence, narrows attribution to
lead-position tags, re-derives the corpus floors, and stops the terminal
path claiming a restatement it never cited.

Checked and NOT holes: is_bdd (exact equality), DIRECTIVE_PREFIXES
(tool-specific tokens only).
f1.rs drives the Write path only, so the Edit/MultiEdit path carried no
corpus coverage — which is how a blanket fragment acquittal shipped with
every gate green. Two invariants over the 60 labelled cases, driven
through check() as Edit payloads: no justified case blocks, no
unnecessary case is spared.

Verified to fail on the pre-fix check.rs with 17 spared cases (15
RestatesCode, 2 NarratesControlFlow) and pass with the fix.
pipeline.rs inlined a JSON escaper and edit_path.rs added a byte-identical
copy. Both now call common::write_payload / common::edit_payload, so an
escaping bug is fixed once. Applied from the reuse review.
Coverage composes over stage x entry path: the mutation gate pinned the
pure classifier while the defect sat in the adapter upstream of it, the
labelled corpus only ever constructed the whole-content payload, and two
composition tests had pinned the blanket acquittal as the contract.
@ryanleecode
ryanleecode merged commit c8ddf1c into master Aug 24, 2026
11 checks passed
@ryanleecode
ryanleecode deleted the fix/edit-fragment-restate-detection branch August 24, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants