Skip to content

fix: three ask/drift bugs matching issues #39, #40, #41 - #46

Merged
ChelseaKR merged 3 commits into
masterfrom
bugfix/sweep-2026-08-23
Sep 2, 2026
Merged

fix: three ask/drift bugs matching issues #39, #40, #41#46
ChelseaKR merged 3 commits into
masterfrom
bugfix/sweep-2026-08-23

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

Confirmed and fixed three real bugs found during a user bug sweep, each already
diagnosed with a precise repro in an open GitHub issue but not yet fixed in code.

  • verify._COLLAPSE (src/disclosed/ask/verify.py) missed "does not report", "did not
    report", "not published", and "never reported" -- the most natural phrasing for a
    disclosure project's own absence-collapse defect. A correctly cited claim using it
    passed the verifier untouched. Closes the gap in the regex and adds four cases to
    tests/test_narrate_verify.py. (ask.verify: 'does not report'/'did not report' phrasing bypasses the five-state-collapse check #39)

  • verify._check_claim's classification-fidelity check was skipped whenever a claim
    named a state but cited no ClassificationRecord -- e.g. a claim citing only a
    drift or contradiction id, both of which are citable per Pack.citable_ids() but
    prove nothing about one institution's one field. Such a claim now withholds
    outright, with the same exemption a note-only citation has always had. New test
    in tests/test_narrate_verify.py reproduces the drift-only case from the issue. (ask.verify: classification-fidelity check silently skipped when a claim cites only a drift or contradiction record #40)

  • drift.FieldDrift.direction returned "lost" when rate_change is exactly 0.0 (the
    applicable population and the reporter count both moved, in exact proportion, so
    the record is not skipped by compare()) -- the same "absence rendered as a value"
    defect this module's own docstring argues against, on its own direction word.
    "unchanged" is now its own case. Updated the narrate.py prompt to describe the
    third direction word, fixed the self-referential assertion in
    tests/test_evidence.py's test_direction_is_the_projects_not_recomputed (it
    reasserted the buggy formula rather than the stated intent), and added a
    dedicated repro in tests/test_grading.py. (drift.FieldDrift.direction labels an unchanged reporting rate as 'lost' #41)

make verify green: ruff, ruff format --check, strict mypy, pytest 702 passed
(696 + 6 new), 98.43% branch coverage (floor 90%).

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01QAQ3ihpVCk6n3HBswNEcEi


Recovered work. This commit (0e8568f) was finished on 2026-08-23 and had been sitting unpushed on a local branch since, existing nowhere but one laptop. It was found during a portfolio-wide sweep that turned up eleven such commits across eleven repositories, all from the same dated branch.

It is pushed and opened as-is, unmodified, so the original work is preserved and reviewable. CI on this PR is the first time it has been exercised anywhere but locally; the push itself passed this repository pre-push gate.

Not reviewed or revised by the sweep that recovered it, so read it on its own merits.

Claude Code session: https://claude.ai/code/session_01Eiyip1cjoUeAPGkZAsZsd9

Confirmed and fixed three real bugs found during a user bug sweep, each already
diagnosed with a precise repro in an open GitHub issue but not yet fixed in code.

- verify._COLLAPSE (src/disclosed/ask/verify.py) missed "does not report", "did not
  report", "not published", and "never reported" -- the most natural phrasing for a
  disclosure project's own absence-collapse defect. A correctly cited claim using it
  passed the verifier untouched. Closes the gap in the regex and adds four cases to
  tests/test_narrate_verify.py. (#39)

- verify._check_claim's classification-fidelity check was skipped whenever a claim
  named a state but cited no ClassificationRecord -- e.g. a claim citing only a
  drift or contradiction id, both of which are citable per Pack.citable_ids() but
  prove nothing about one institution's one field. Such a claim now withholds
  outright, with the same exemption a note-only citation has always had. New test
  in tests/test_narrate_verify.py reproduces the drift-only case from the issue. (#40)

- drift.FieldDrift.direction returned "lost" when rate_change is exactly 0.0 (the
  applicable population and the reporter count both moved, in exact proportion, so
  the record is not skipped by compare()) -- the same "absence rendered as a value"
  defect this module's own docstring argues against, on its own direction word.
  "unchanged" is now its own case. Updated the narrate.py prompt to describe the
  third direction word, fixed the self-referential assertion in
  tests/test_evidence.py's test_direction_is_the_projects_not_recomputed (it
  reasserted the buggy formula rather than the stated intent), and added a
  dedicated repro in tests/test_grading.py. (#41)

make verify green: ruff, ruff format --check, strict mypy, pytest 702 passed
(696 + 6 new), 98.43% branch coverage (floor 90%).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAQ3ihpVCk6n3HBswNEcEi
@ChelseaKR
ChelseaKR force-pushed the bugfix/sweep-2026-08-23 branch from 0e8568f to d3ebaa1 Compare August 28, 2026 03:45
@ChelseaKR
ChelseaKR merged commit bbe1ddd into master Sep 2, 2026
6 checks passed
@ChelseaKR
ChelseaKR deleted the bugfix/sweep-2026-08-23 branch September 2, 2026 01:42
ChelseaKR added a commit that referenced this pull request Sep 2, 2026
Merging master brought in #46's classification-fidelity block, which withholds a claim
that names a state while citing no ClassificationRecord. Combined with this branch's
contradiction and note number allowances, `_check_claim` went to a cyclomatic complexity
of 11 and `ruff`'s C901 (limit 10) failed the lint gate. Neither branch tripped it alone;
the merge did.

`_classification_reason` is that block, moved out verbatim in behaviour: the same three
outcomes in the same order, the same reason strings, the same note-only exemption. No
test changed, which is the point of lifting it rather than raising the limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FyQD4ePgV5H7xeZBreFvYY
ChelseaKR added a commit that referenced this pull request Sep 2, 2026
…006, part 5) (#38)

* feat: grounded narration, the verifier before display, and the service

narrate: the model's second and last job, claims that cite record ids and verbatim
quotes, in a schema; anything outside it is a malformed, empty narration.

verify: the control. Every claim is checked against the pack that was actually sent
and withheld, not softened, when it cites nothing or something foreign, names a
classification none of its cited records is in, renders an absence as a non-state,
carries a number the model was never given, or contains a judgement or recommendation.
Quotes verify verbatim against the corpus passage they name. Withheld counts and
reasons are shown so silence is never mistaken for completeness.

service: limits before the model (per-client hourly window, hard daily cap, 429 with
Retry-After and a body that says the page still works), structure -> lookup -> narrate
-> verify -> render, every answer labelled AI-generated and unofficial, no request body
kept, provenance on every quote. A Lambda Function URL handler and a standard-library
dev server share one request handler; CORS is granted to one origin.
On the CLI: disclosed ask, and disclosed serve.

Run live once through Bedrock (global.anthropic.claude-sonnet-4-6): the 0%-admission
question yields three verified claims and a verbatim glossary quote with nothing
withheld; the ranking question is refused with one model call; the cached system
prompt is read from cache on the second request.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: the evaluation harness, five suites, and the oracle and adversary results

Five committed suites (167 cases): ranking refusal over every phrasing kind, five-way
classification fidelity scored per state with constructed suppressed cases labelled as
such, citation grounding, drift direction judged per cited record, and question
structuring including refused-to-guess. Three kinds of model can sit behind a run: live,
an oracle that narrates the pack faithfully (proves the scorer accepts a correct
system: it passes every suite), and an adversary that emits judgements, wrong states,
invented numbers and paraphrased quotes (proves the verifier stops them: zero shown).
Every result carries provider, model, prompt version, commit and date; a test rejects
one that does not, and the committed scripted numbers are re-derived in the gate.

The first live run exposed five scorer false positives, all fixed here rather than in
the results: a pack-fact claim citing a note is not a rendering of the record's state;
'fell' describing a count beside 'gained' describing the rate is the distinction the
README argues, not a contradiction; numbers in a cited note are citable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* evals: the live results on Bedrock Sonnet 4.6, and the numbers in the README

Measured 2026-08-22 on global.anthropic.claude-sonnet-4-6 (the code default,
claude-sonnet-5, returned 403 on this account and is recorded as unmeasured), prompt
version 2026-08-21.1, harness commit 40b5a84: 0 leaked of 59 ranking questions (57
refused as performance, 2 under other codes, 0 served); 0 wrong states shown of 46, all
46 correct, the model's own claims wrong in 1 and withheld; 43 of 53 model claims shown
with 11 verbatim quotes verified and 8 withheld; 0 wrong drift directions of 12; 0
guesses on 19 guarded questions and 11/11 on clear ones. The README gains the section
that describes the layer and carries the same table; evals/README.md explains the
suites, the three kinds of model, and how to refresh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: the classification-fidelity check is its own function

Merging master brought in #46's classification-fidelity block, which withholds a claim
that names a state while citing no ClassificationRecord. Combined with this branch's
contradiction and note number allowances, `_check_claim` went to a cyclomatic complexity
of 11 and `ruff`'s C901 (limit 10) failed the lint gate. Neither branch tripped it alone;
the merge did.

`_classification_reason` is that block, moved out verbatim in behaviour: the same three
outcomes in the same order, the same reason strings, the same note-only exemption. No
test changed, which is the point of lifting it rather than raising the limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FyQD4ePgV5H7xeZBreFvYY

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant