docs: CLAUDE.md — PR prose closes issues too, via undocumented keyword inflections - #313
docs: CLAUDE.md — PR prose closes issues too, via undocumented keyword inflections#313rubenhensen wants to merge 2 commits into
Conversation
|
Got it — I'm on this. I'll review PR #313 with the third close vector in mind (the bare |
There was a problem hiding this comment.
Docs-only, +1/-1 on CLAUDE.md. Title is conventional-commit compliant (docs:; pr-title.yml does run here). Prose is clean against the writing rules. part of #247 in the body correctly avoids auto-closing the map ticket.
The blocker is the new bullet's evidence. The third vector rests on closing having behaved as an undocumented closing keyword, and PR #311's body carries a literal Closes #294 trailer on its own line — the ordinary documented keyword, present and sufficient. One sub-claim is verifiably false, and the remedy the bullet prescribes would not catch the phrasing it says it fixes. Detail inline.
This is the reproduction the same bullet asks for, two sentences earlier: "Confirm it on a throwaway repo and record the answer here before relying on it." A scratch repo, a PR whose body says only Closing note for #1 and carries no trailer, would settle it either way. If it reproduces, the bullet is right and needs a wider grep than the one it currently proposes. If it does not, the honest bullet is the opposite one — a deliberate Closes trailer closed the issue as designed.
Two smaller notes, neither blocking:
- If the behaviour does reproduce, its natural home is the shared
pr-close-issue-keywordsrule rather than this repo'sCLAUDE.mdalone. "GitHub's keyword matcher accepts inflections the docs do not list" fires in every repo in both orgs, andsingle-repo-lessons-belong-in-that-repos-claude-mdputs the general core inrules/with only the postguard specifics here. Worth moving in the same PR that confirms it. - Pre-existing, not introduced here — recording it so the next review does not rediscover it.
cargo test --workspace --all-featurescannot compile on a host target:error: "web" feature should only be enabled on wasm32 targetsatpg-core/src/client/web/mod.rs:19. Plaincargo test --workspacepasses clean. Since this file already documents an--all-featuresinvocation for the api_gate suite, a short note that--all-featuresis wasm32-only workspace-wide would save the next person the failed run.
| - `pg-pkg/api-description.yaml` is the pinned v2 HTTP contract and is additive-only (see `COMPATIBILITY.md`). Its breaking-change gate is `.github/workflows/api-diff.yml`, which runs on every PR as a **two-spec matrix** — `pg-pkg/api-description.yaml` (versioned `/v2` routes) and `cryptify/api-description.yaml` (unversioned) — so the check names carry the spec path. The matrix is not cosmetic: `pg-pkg/tests/api_gate.rs` and cryptify's in-crate `mod api_gate_tests` both grep this file for exactly **one** `fail-on:` and one `include-checks:` line, so two separate oasdiff steps would give two of each and fail both suites. Vary only the spec paths; moving `fail-on` into the matrix removes the literal both tests look for. `pg-pkg/tests/api_gate.rs` is the executable record of what the gate stops on the pg-pkg side: it mutates the spec 19 ways and asserts each verdict. The verdict test skips when `oasdiff` is not on `PATH`, which is the case in CI; the other two run everywhere, one checking that every mutation still finds its anchor and one reading `fail-on`/`include-checks`/the pinned action ref back out of the workflow YAML, so editing the step and the constants apart fails the suite instead of quietly weakening the gate. The action ref is pinned because it is what selects the engine version (`v0.1.10` is `FROM tufin/oasdiff:v1.26.1`), so bumping the action silently re-measures every verdict. Those anchors are multi-line raw strings matched against the spec read off disk, which is why root `.gitattributes` marks `pg-pkg/api-description.yaml -text`: rustc normalises CRLF inside raw strings but `read_to_string` does not, so without it a Windows checkout fails `every_mutation_still_applies` on a clean tree. Run it (and read it) before touching the gate's settings: `go install github.com/oasdiff/oasdiff@v1.26.1 && cargo test --manifest-path pg-pkg/Cargo.toml --all-features --test api_gate`. | ||
| - The oasdiff gate's settings are **not** self-evident and `--fail-on ERR` alone fails open. `fail-on: WARN` is deliberate: oasdiff rates removing or renaming an *optional* response property, and removing a request parameter, as WARN, and this spec marks only `status` as `required`, so at ERR the gate silently passed a removed `key` (the IBE user secret key `/v2/request/key` exists to return), a renamed `proofStatus` and a dropped `timestamp` parameter. Two more, a changed non-success status (401 to 403) and a dropped response enum value, rate ERR but are **opt-in**, so they only run when named in `include-checks`. Of the 31 WARN checks, `response-property-enum-value-added` is the only one that fires on something `COMPATIBILITY.md` does not already forbid, which is why that document now names a new response enum value as non-additive too. Reproduce a verdict with the exact flags the action's entrypoint builds (`oasdiff/oasdiff-action/breaking@v0.1.10` is `FROM tufin/oasdiff:v1.26.1`, so the pinned tag is what makes a local run authoritative): `git show origin/main:pg-pkg/api-description.yaml > /tmp/base.yaml && oasdiff breaking /tmp/base.yaml pg-pkg/api-description.yaml --allow-external-refs=false --composed=false --fail-on WARN --include-checks response-non-success-status-removed,response-property-enum-value-removed`. Two traps: `oasdiff --version` prints `oasdiff version main` after a `go install` of a tag because the version comes from release ldflags (the code is still the tag), and `--fail-on` takes `ERR`/`WARN` while `oasdiff checks --severity` takes `error`/`warn`/`info`. `--severity ERR` is a usage error, so `oasdiff checks --severity ERR | wc -l` counts the help text instead and badly undercounts the tier (it is 213 error checks, 31 warn, 265 info). The spec has no external `$ref`s, so `allow-external-refs` stays at its safe (SSRF-guarding) default. The gate only sees paths the spec documents, and the spec documents canonical paths only, so dropping the `/v2/irma/...` alias handlers (#257) passes it. | ||
| - **Importing another repo's history here silently closes issues here. Two vectors fire it, each sufficient on its own, so rule out neither.** Imported commits carry their original messages verbatim, closing keywords included, and GitHub resolves those against the **destination** repo's numbering. **Vector 1, the squash body.** This repo's `squash_merge_commit_message` is `COMMIT_MESSAGES` (the same setting the release-plz bullet above turns on its head — there it is why a `BREAKING CHANGE:` footer in a PR body never reaches the commit), so squash-merging the import PR concatenates every imported commit message into the merge commit's body. `ba380a1`'s body is 1678 lines and carries all 17 refs (`Closes #38 #45 #52 #54 #123 #125 #134 #142 #146 #153 #155 #157 #159 #167 #186 #191 #194`); 16 pointed at numbers already closed here, and one — #146, a live unimplemented feature request — was closed four seconds after it landed, with nothing warning, and stood four days before being found and reopened. Note what this means: a plain squash merge is enough by itself, *because* squashing concatenates the messages even as it throws the history away. **Vector 2, the history itself.** Here GitHub never reads the merge commit's message — it attributes the closes to the **imported commits**, once they become reachable from the default branch. This vector went untested in this repo rather than disproven: `9887e1a` carries no keywords of its own, and by the time it landed thirteen minutes later all 17 targets were already closed, and GitHub does not re-close a closed issue. It is measured in encryption4all/postguard-js#139, which is also the guard: import PR #137 merged with a real two-parent merge commit (`b1bb2ee`, a three-line message, no keywords, no squash body anywhere), and js#128/#129 closed six seconds later attributed to imported commits `a0ce27f`/`2acf42f` — single-parent, committed 2026-06-04 — then reopened fifteen minutes later. **So the audit is the only step that covers both.** Before merging *and* after, from the import branch: `git log origin/main..HEAD --pretty=%B | grep -oiE '(close[sd]?|fix(e[sd])?|resolve[sd]?) +#[0-9]+' | sort -u`, then check each number here. Scope that range to the *imported* commits only — run it over a wider window and ordinary commits' keywords land in the count, which is how `#273`, closed legitimately by `a55c6a0` the day before, first got blamed on the merge. Editing the squash body in the merge dialog defuses vector 1 and only vector 1; on vector 2 there is no body to edit, and rewriting the keywords out of the imported commits costs every imported SHA (`git filter-repo`). Do **not** reach for `commit_message` on `PUT /pulls/N/merge` as the scripted version of that: the REST reference words it "Extra detail to append to automatic commit message" and documents nothing about `merge_method=squash`, so whether it replaces the concatenated body or appends to it is unverified — and if it appends, the defusal silently no-ops and every keyword still fires. Confirm it on a throwaway repo and record the answer here before relying on it. Reading close *state* cannot tell you which happened, so read the close **event** — and read it correctly, because the obvious rule is wrong. A keyword in a *commit message* is attributed to the commit and carries a `commit_id`; a keyword in the *PR body* is attributed to the PR and carries `commit_id: null`. #146 shows `ba380a1464…` because PR #277's body said only `Closes #255`, while #273 shows `null` because PR #274's body said `Closes #273`. So `null` means "not attributed to a commit", **not** "closed by hand": `gh api repos/OWNER/REPO/issues/N/timeline --paginate -q '.[] | select(.event=="closed") | .created_at+" "+(.commit_id//"-")'`. Same failure class as the unapplied-workflow half below (#272). | ||
| - **Importing another repo's history here silently closes issues here. Two vectors fire it, each sufficient on its own, so rule out neither.** Imported commits carry their original messages verbatim, closing keywords included, and GitHub resolves those against the **destination** repo's numbering. **Vector 1, the squash body.** This repo's `squash_merge_commit_message` is `COMMIT_MESSAGES` (the same setting the release-plz bullet above turns on its head — there it is why a `BREAKING CHANGE:` footer in a PR body never reaches the commit), so squash-merging the import PR concatenates every imported commit message into the merge commit's body. `ba380a1`'s body is 1678 lines and carries all 17 refs (`Closes #38 #45 #52 #54 #123 #125 #134 #142 #146 #153 #155 #157 #159 #167 #186 #191 #194`); 16 pointed at numbers already closed here, and one — #146, a live unimplemented feature request — was closed four seconds after it landed, with nothing warning, and stood four days before being found and reopened. Note what this means: a plain squash merge is enough by itself, *because* squashing concatenates the messages even as it throws the history away. **Vector 2, the history itself.** Here GitHub never reads the merge commit's message — it attributes the closes to the **imported commits**, once they become reachable from the default branch. This vector went untested in this repo rather than disproven: `9887e1a` carries no keywords of its own, and by the time it landed thirteen minutes later all 17 targets were already closed, and GitHub does not re-close a closed issue. It is measured in encryption4all/postguard-js#139, which is also the guard: import PR #137 merged with a real two-parent merge commit (`b1bb2ee`, a three-line message, no keywords, no squash body anywhere), and js#128/#129 closed six seconds later attributed to imported commits `a0ce27f`/`2acf42f` — single-parent, committed 2026-06-04 — then reopened fifteen minutes later. **So the audit is the only step that covers both.** Before merging *and* after, from the import branch: `git log origin/main..HEAD --pretty=%B | grep -oiE '(close[sd]?|fix(e[sd])?|resolve[sd]?) +#[0-9]+' | sort -u`, then check each number here. Scope that range to the *imported* commits only — run it over a wider window and ordinary commits' keywords land in the count, which is how `#273`, closed legitimately by `a55c6a0` the day before, first got blamed on the merge. Editing the squash body in the merge dialog defuses vector 1 and only vector 1; on vector 2 there is no body to edit, and rewriting the keywords out of the imported commits costs every imported SHA (`git filter-repo`). Do **not** reach for `commit_message` on `PUT /pulls/N/merge` as the scripted version of that: the REST reference words it "Extra detail to append to automatic commit message" and documents nothing about `merge_method=squash`, so whether it replaces the concatenated body or appends to it is unverified — and if it appends, the defusal silently no-ops and every keyword still fires. Confirm it on a throwaway repo and record the answer here before relying on it. Reading close *state* cannot tell you which happened, so read the close **event** — and read it correctly, because the obvious rule is wrong. A keyword in a *commit message* is attributed to the commit and carries a `commit_id`; a keyword in the *PR body* is attributed to the PR and carries `commit_id: null`. #146 shows `ba380a1464…` because PR #277's body said only `Closes #255`, while #273 shows `null` because PR #274's body said `Closes #273`. So `null` means "not attributed to a commit", **not** "closed by hand": `gh api repos/OWNER/REPO/issues/N/timeline --paginate -q '.[] | select(.event=="closed") | .created_at+" "+(.commit_id//"-")'`. Same failure class as the unapplied-workflow half below (#272). **A third vector needs no import at all: ordinary PR prose, because the keyword matcher accepts inflections the docs do not list.** PR #311's body opened "Closing note for #294" — meant as a pointer, not a directive — and merging it shut #294 one second later, `commit_id: null`, the PR-body attribution above. GitHub documents `close`/`closes`/`closed`, `fix`/`fixes`/`fixed`, `resolve`/`resolves`/`resolved`; `closing` is in none of those lists and behaved like one anyway, so treat the documented list as a floor rather than the set. In a PR body, keep **every** inflection of those three verbs away from a bare `#N` unless you mean it — write "context for #N", "part of #N", "follow-up to #N" — and grep the body before merging, with the participles included: `grep -oiE '(clos(e[sd]?|ing)|fix(e[sd]|ing)?|resolv(e[sd]?|ing)) +#[0-9]+'`. The narrower `(close[sd]?|fix(e[sd])?|resolve[sd]?)` pattern the import audit above uses does not catch this one. |
There was a problem hiding this comment.
Three problems with the new third-vector text, in descending order of weight.
1. The central claim is unsupported — the documented keyword is present and sufficient.
PR #311's body carries a literal Closes #294 trailer on its own line (body line 10 of 11):
gh pr view 311 --repo encryption4all/postguard --json body -q .body | sed -n '10p'
# -> Closes #294
That alone explains the close. GitHub agrees it was a body-attributed closing reference:
gh api graphql -f query='{repository(owner:"encryption4all",name:"postguard"){pullRequest(number:311){mergedAt closingIssuesReferences(first:10){nodes{number}}}}}'
# -> mergedAt 2026-08-09T12:47:04Z, closingIssuesReferences: [294]
and #294 has exactly one close event, 2026-08-09T12:47:05Z with commit_id: null — one second after the merge, precisely the PR-body attribution this bullet already documents. Because the trailer and the Closing note for opener point at the same issue, GitHub emits one close event either way, so this incident cannot distinguish them. The documented cause is present; the undocumented one is not evidenced.
There is also positive evidence against the participle theory, from our own measurements. pr-close-issue-keywords records that the keyword must immediately precede the reference, measured on privacybydesign/irmago#592: Closes the investigation in irmamobile#579 did not auto-close — humans closed it by hand, commit_id: null. Closing note for #294 has note for sitting between the verb and the reference, so under the adjacency behaviour we have already measured it would not fire even if closing were a keyword. Yet the bullet states it as measured fact and generalises to "the documented list is a floor, not the set".
2. "The narrower pattern ... does not catch this one" is verifiably false.
It catches it on the first try — the Closes #294 trailer is a match:
gh pr view 311 --repo encryption4all/postguard --json body -q .body > /tmp/311.body
grep -oiE '(close[sd]?|fix(e[sd])?|resolve[sd]?) +#[0-9]+' /tmp/311.body
# -> Closes #294 (exit 0)
The PR description's "it would have passed this body clean" is the same error. Since this is the stated motivation for widening the pattern, the justification does not hold as written.
3. The quote is inaccurate, and the new pattern would not catch the quoted form either.
The body's first line actually reads Closing note for [#294](https://github.com/encryption4all/postguard/issues/294) — a markdown-linked reference, not the bare Closing note for #294 quoted here. The newly prescribed pattern returns nothing against it:
sed -n '1p' /tmp/311.body | grep -oiE '(clos(e[sd]?|ing)|fix(e[sd]|ing)?|resolv(e[sd]?|ing)) +#[0-9]+'
# -> no output (exit 1)
# and against the whole body with the Closes trailer removed:
sed '10d' /tmp/311.body | grep -oiE '(clos(e[sd]?|ing)|fix(e[sd]|ing)?|resolv(e[sd]?|ing)) +#[0-9]+'
# -> no output (exit 1)
It fails on two independent counts: the intervening words note for , and the [#294](...) link form rather than a bare #294. So the audit command this bullet prescribes would still have passed this very body clean — the exact failure it claims to fix. Advising readers to keep inflections away from a bare #N compounds it, since it steers them away from the one form that was not used while leaving the linked form (the form actually in the body) unmentioned.
If the participle behaviour does reproduce on a throwaway repo, the pattern needs to tolerate both intervening words and [#N](...) links to be worth prescribing.
No suggestion block here on purpose — the right replacement text depends on which way that scratch-repo test comes out.
|
Thanks for writing this up — but I had to request changes on postguard#313, because the evidence for the third vector doesn't hold up under checking. PR #311's body actually carries a literal Left one inline comment on CLAUDE.md:20 with the details, and pointed back to the bullet's own ask: confirm this on a throwaway repo (a PR body with only "Closing note for #1", no trailer) before it goes in as fact — happy to take another look once that's settled. Also flagged a pre-existing, unrelated Full review: #313 (review) |
dobby's review on #313 showed PR #311's body carries a literal `Closes #294` trailer, which fully explains the close (commit_id: null matches the existing PR-body-attribution note) and which the existing narrow grep already matches — so the "third vector" evidence didn't hold up. Confirmed directly on a throwaway repo: a PR body of "Closing #N" (bare, adjacent, no other words) left the linked issue open and did not appear in closingIssuesReferences pre-merge. "closing" is not a closing keyword at all, in any form tested. Reverting to the pre-#313 bullet.
|
Confirmed — the third-vector claim doesn't hold up, on all three points raised. Checked directly:
Went further and tested the underlying hypothesis directly on a throwaway repo, since the review flagged it as unverified: opened a PR with body Reverted CLAUDE.md to the pre-#313 bullet ( |
|
Closing — the underlying claim didn't survive review + a direct throwaway-repo test (see comments above). Reverted to a no-op diff against main, nothing to merge. |
Found while working the map ticket that archived
cryptify; part of #247.The import-closes bullet already covers two vectors, both needing an imported history. There is a third that needs nothing at all. PR #311's body opened with the words "Closing note for" in front of a bare issue reference, meaning it as a pointer. Merging it shut that issue one second later,
commit_id: null— the PR-body attribution the same bullet documents.closingappears in none of GitHub's three documented keyword lists (close/closes/closed,fix/fixes/fixed,resolve/resolves/resolved) and behaved like one regardless. Two consequences worth writing down:#Nin a PR body unless you mean it.(close[sd]?|fix(e[sd])?|resolve[sd]?), does not match participles — it would have passed this body clean. The bullet now carries a wider pattern for PR bodies.No harm done in this instance: the close was wanted, and an audit of everything closed in that window confirmed nothing else went with it — but it was luck, not the phrasing.