fix(check): Edit/MultiEdit acquitted every added comment - #57
Merged
Conversation
An Edit/MultiEdit fragment marked every newly-added comment's context unreliable, so classify's catch-all rewrote the verdict to Justified and the hook passed. Restatement and flow-narration detection were dead on the tool agents use to modify existing files: only AgentMemo, CommentedOutCode and VacuousTodo could ever fire. A fragment is contiguous file text, so a comment with code after it inside the fragment annotates that code in the file too. Mark unreliable only where the boundary could have taken the adjacent code away: no adjacent code at all, or nothing after the comment. This restores the edge-scoped rule the U2 plan specifies (plan lines 233, 248). Gate: fmt, clippy -D warnings, 15 pipeline tests, F1 unchanged.
A keyword acquits a proven restatement: 13 of 18 paraphrases of one restatement are spared by adding a marker word. Conviction is evidence-backed and default-deny; acquittal is bare substring membership. The plan inverts that precedence, narrows attribution to lead-position tags, re-derives the corpus floors, and stops the terminal path claiming a restatement it never cited. Checked and NOT holes: is_bdd (exact equality), DIRECTIVE_PREFIXES (tool-specific tokens only).
f1.rs drives the Write path only, so the Edit/MultiEdit path carried no corpus coverage — which is how a blanket fragment acquittal shipped with every gate green. Two invariants over the 60 labelled cases, driven through check() as Edit payloads: no justified case blocks, no unnecessary case is spared. Verified to fail on the pre-fix check.rs with 17 spared cases (15 RestatesCode, 2 NarratesControlFlow) and pass with the fix.
pipeline.rs inlined a JSON escaper and edit_path.rs added a byte-identical copy. Both now call common::write_payload / common::edit_payload, so an escaping bug is fixed once. Applied from the reuse review.
Coverage composes over stage x entry path: the mutation gate pinned the pure classifier while the defect sat in the adapter upstream of it, the labelled corpus only ever constructed the whole-content payload, and two composition tests had pinned the blanket acquittal as the contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
On
EditandMultiEdit,new_commentsstamped every newly-added comment'sCommentContextasunreliable.classifyreads that flag twice —reliable_adjacentwithholds the adjacent code from the context-aware detectors, and the terminal fallback then rewrites the resulting evidence-freeRestatesCodeintoJustification::NonObviousIntent.Net effect: on the tools an agent actually uses to modify an existing file, only the text-only rules could ever fire (
AgentMemo,CommentedOutCode,VacuousTodo).RestatesCodeandNarratesControlFlowwere dead code there.Identical comment text, two payload shapes,
v0.1.5:Write—// set b to 2,// loop over itemsEdit— same text, comment mid-fragment with code above and belowMultiEdit— sameThe third row is the one that rules out "conservative at the edge": the context was demonstrably not truncated, and it was still acquitted.
The fix
The adjudication plan specifies unreliability at the fragment edge (its risk register and DoD both say so); the implementation applied it to the whole fragment. This restores the documented scope:
A fragment is a contiguous slice of file text, so a comment with a following code sibling inside the slice has that same sibling in the whole file — the boundary cannot have inserted code between them. Only two shapes are genuinely truncated: no adjacent code captured at all, and a comment
derive_contextcould read only asTrailing, which it yields exactly when the next code sibling is absent.classify.rsis untouched, so the classifier mutation gate is not triggered by this change.Measured
The 60-case labelled corpus driven through the
Editseam, before vs after:The 17 recovered misses were 15
RestatesCode+ 2NarratesControlFlow. No justified case is newly blocked, so there is no false-positive cost against the corpus. TheWritepath is unchanged on all 60 with zero changed verdicts — the fix is confined to the adapted paths.Operator mutation of the new predicate is killed:
||→&&(3 failing tests),== Trailing→!=(2), and each constant (2 and 3).Why three green gates missed it
Coverage composes over (stage × entry path), not over stages alone:
tests/edit_path.rscloses the second one: the corpus driven throughcheckasEditpayloads, asserting no justified case blocks and no unnecessary case is spared. Verified red on the pre-fixcheck.rs(17 spared, named) and green with the fix.Contents
fix(check)— the edge-scoped predicate; two tests that asserted the old contract now assertBlock, two new tests pin the genuinely-unreliable shapestest(edit-path)— the corpus gate for the edit seamtest(common)— one payload builder;pipeline.rsandedit_path.rshad byte-identical JSON escapersdocs(plans)— a plan for a separate, larger defect found while measuring this one (below)docs(solutions)— the gate-coverage learningFound while measuring, not fixed here
A keyword acquits a proven restatement. Conviction is evidence-backed and default-deny; acquittal is bare substring membership. 13 of 18 paraphrases of
// set x to 1aboveconst x = 1;are spared by adding one marker —because,note:,security,1-based,ref:,based on. Padding is not the escape (+9 words of unrelated prose still blocks), so the entire surface isis_non_obvious_intentandis_attribution. SinceJUSTIFIEDis consulted before any evidence is computed, the precedence — not the table contents — is what makes it launderable. The plan in this PR inverts that precedence, narrows attribution to lead-position tags, and re-derives the corpus floors. Not started;eval/corpus.jsonis untouched by this PR.Two related claims checked and found sound, so they are not holes:
is_bddcompares for exact equality on the stripped text, andDIRECTIVE_PREFIXEScarries only tool-specific tokens (// Allows the caller to…blocks). Unsupported extensions still pass unenforced, which is by design.One unrelated observation:
is_docstring_headover-promotes module-scope leading comments toDocstringHeadin synthesised fragments. Harmless today becauseis_public_api_docadditionally filters on doc markup, and independent of this change.Verification
Green: 61 unit, 10 classify, 2 exit_codes, 2 edit_path, 6 f1, 15 pipeline. Run in the repo's own
nix developshell.