fix: three ask/drift bugs matching issues #39, #40, #41 - #46
Merged
Conversation
Confirmed and fixed three real bugs found during a user bug sweep, each already diagnosed with a precise repro in an open GitHub issue but not yet fixed in code. - verify._COLLAPSE (src/disclosed/ask/verify.py) missed "does not report", "did not report", "not published", and "never reported" -- the most natural phrasing for a disclosure project's own absence-collapse defect. A correctly cited claim using it passed the verifier untouched. Closes the gap in the regex and adds four cases to tests/test_narrate_verify.py. (#39) - verify._check_claim's classification-fidelity check was skipped whenever a claim named a state but cited no ClassificationRecord -- e.g. a claim citing only a drift or contradiction id, both of which are citable per Pack.citable_ids() but prove nothing about one institution's one field. Such a claim now withholds outright, with the same exemption a note-only citation has always had. New test in tests/test_narrate_verify.py reproduces the drift-only case from the issue. (#40) - drift.FieldDrift.direction returned "lost" when rate_change is exactly 0.0 (the applicable population and the reporter count both moved, in exact proportion, so the record is not skipped by compare()) -- the same "absence rendered as a value" defect this module's own docstring argues against, on its own direction word. "unchanged" is now its own case. Updated the narrate.py prompt to describe the third direction word, fixed the self-referential assertion in tests/test_evidence.py's test_direction_is_the_projects_not_recomputed (it reasserted the buggy formula rather than the stated intent), and added a dedicated repro in tests/test_grading.py. (#41) make verify green: ruff, ruff format --check, strict mypy, pytest 702 passed (696 + 6 new), 98.43% branch coverage (floor 90%). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAQ3ihpVCk6n3HBswNEcEi
ChelseaKR
force-pushed
the
bugfix/sweep-2026-08-23
branch
from
August 28, 2026 03:45
0e8568f to
d3ebaa1
Compare
This was referenced Sep 2, 2026
Closed
ChelseaKR
added a commit
that referenced
this pull request
Sep 2, 2026
Merging master brought in #46's classification-fidelity block, which withholds a claim that names a state while citing no ClassificationRecord. Combined with this branch's contradiction and note number allowances, `_check_claim` went to a cyclomatic complexity of 11 and `ruff`'s C901 (limit 10) failed the lint gate. Neither branch tripped it alone; the merge did. `_classification_reason` is that block, moved out verbatim in behaviour: the same three outcomes in the same order, the same reason strings, the same note-only exemption. No test changed, which is the point of lifting it rather than raising the limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FyQD4ePgV5H7xeZBreFvYY
ChelseaKR
added a commit
that referenced
this pull request
Sep 2, 2026
…006, part 5) (#38) * feat: grounded narration, the verifier before display, and the service narrate: the model's second and last job, claims that cite record ids and verbatim quotes, in a schema; anything outside it is a malformed, empty narration. verify: the control. Every claim is checked against the pack that was actually sent and withheld, not softened, when it cites nothing or something foreign, names a classification none of its cited records is in, renders an absence as a non-state, carries a number the model was never given, or contains a judgement or recommendation. Quotes verify verbatim against the corpus passage they name. Withheld counts and reasons are shown so silence is never mistaken for completeness. service: limits before the model (per-client hourly window, hard daily cap, 429 with Retry-After and a body that says the page still works), structure -> lookup -> narrate -> verify -> render, every answer labelled AI-generated and unofficial, no request body kept, provenance on every quote. A Lambda Function URL handler and a standard-library dev server share one request handler; CORS is granted to one origin. On the CLI: disclosed ask, and disclosed serve. Run live once through Bedrock (global.anthropic.claude-sonnet-4-6): the 0%-admission question yields three verified claims and a verbatim glossary quote with nothing withheld; the ranking question is refused with one model call; the cached system prompt is read from cache on the second request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: the evaluation harness, five suites, and the oracle and adversary results Five committed suites (167 cases): ranking refusal over every phrasing kind, five-way classification fidelity scored per state with constructed suppressed cases labelled as such, citation grounding, drift direction judged per cited record, and question structuring including refused-to-guess. Three kinds of model can sit behind a run: live, an oracle that narrates the pack faithfully (proves the scorer accepts a correct system: it passes every suite), and an adversary that emits judgements, wrong states, invented numbers and paraphrased quotes (proves the verifier stops them: zero shown). Every result carries provider, model, prompt version, commit and date; a test rejects one that does not, and the committed scripted numbers are re-derived in the gate. The first live run exposed five scorer false positives, all fixed here rather than in the results: a pack-fact claim citing a note is not a rendering of the record's state; 'fell' describing a count beside 'gained' describing the rate is the distinction the README argues, not a contradiction; numbers in a cited note are citable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * evals: the live results on Bedrock Sonnet 4.6, and the numbers in the README Measured 2026-08-22 on global.anthropic.claude-sonnet-4-6 (the code default, claude-sonnet-5, returned 403 on this account and is recorded as unmeasured), prompt version 2026-08-21.1, harness commit 40b5a84: 0 leaked of 59 ranking questions (57 refused as performance, 2 under other codes, 0 served); 0 wrong states shown of 46, all 46 correct, the model's own claims wrong in 1 and withheld; 43 of 53 model claims shown with 11 verbatim quotes verified and 8 withheld; 0 wrong drift directions of 12; 0 guesses on 19 guarded questions and 11/11 on clear ones. The README gains the section that describes the layer and carries the same table; evals/README.md explains the suites, the three kinds of model, and how to refresh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: the classification-fidelity check is its own function Merging master brought in #46's classification-fidelity block, which withholds a claim that names a state while citing no ClassificationRecord. Combined with this branch's contradiction and note number allowances, `_check_claim` went to a cyclomatic complexity of 11 and `ruff`'s C901 (limit 10) failed the lint gate. Neither branch tripped it alone; the merge did. `_classification_reason` is that block, moved out verbatim in behaviour: the same three outcomes in the same order, the same reason strings, the same note-only exemption. No test changed, which is the point of lifting it rather than raising the limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FyQD4ePgV5H7xeZBreFvYY --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Confirmed and fixed three real bugs found during a user bug sweep, each already
diagnosed with a precise repro in an open GitHub issue but not yet fixed in code.
verify._COLLAPSE (src/disclosed/ask/verify.py) missed "does not report", "did not
report", "not published", and "never reported" -- the most natural phrasing for a
disclosure project's own absence-collapse defect. A correctly cited claim using it
passed the verifier untouched. Closes the gap in the regex and adds four cases to
tests/test_narrate_verify.py. (ask.verify: 'does not report'/'did not report' phrasing bypasses the five-state-collapse check #39)
verify._check_claim's classification-fidelity check was skipped whenever a claim
named a state but cited no ClassificationRecord -- e.g. a claim citing only a
drift or contradiction id, both of which are citable per Pack.citable_ids() but
prove nothing about one institution's one field. Such a claim now withholds
outright, with the same exemption a note-only citation has always had. New test
in tests/test_narrate_verify.py reproduces the drift-only case from the issue. (ask.verify: classification-fidelity check silently skipped when a claim cites only a drift or contradiction record #40)
drift.FieldDrift.direction returned "lost" when rate_change is exactly 0.0 (the
applicable population and the reporter count both moved, in exact proportion, so
the record is not skipped by compare()) -- the same "absence rendered as a value"
defect this module's own docstring argues against, on its own direction word.
"unchanged" is now its own case. Updated the narrate.py prompt to describe the
third direction word, fixed the self-referential assertion in
tests/test_evidence.py's test_direction_is_the_projects_not_recomputed (it
reasserted the buggy formula rather than the stated intent), and added a
dedicated repro in tests/test_grading.py. (drift.FieldDrift.direction labels an unchanged reporting rate as 'lost' #41)
make verify green: ruff, ruff format --check, strict mypy, pytest 702 passed
(696 + 6 new), 98.43% branch coverage (floor 90%).
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01QAQ3ihpVCk6n3HBswNEcEi
Recovered work. This commit (
0e8568f) was finished on 2026-08-23 and had been sitting unpushed on a local branch since, existing nowhere but one laptop. It was found during a portfolio-wide sweep that turned up eleven such commits across eleven repositories, all from the same dated branch.It is pushed and opened as-is, unmodified, so the original work is preserved and reviewable. CI on this PR is the first time it has been exercised anywhere but locally; the push itself passed this repository pre-push gate.
Not reviewed or revised by the sweep that recovered it, so read it on its own merits.
Claude Code session: https://claude.ai/code/session_01Eiyip1cjoUeAPGkZAsZsd9