fix(ci): estate-rescan — fail fast on 403, publish via branch+PR (token grant still required)#641
Merged
Merged
Conversation
…-store
The push step retried ANY push failure with backoff (2/4/8/16s) on the
assumption of writer contention. But `HYPATIA_DISPATCH_PAT` lacks
Contents:write on hyperpolymath/verisimdb-data, so every attempt returned:
remote: Permission to hyperpolymath/verisimdb-data.git denied to hyperpolymath
fatal: ... The requested URL returned error: 403
A permission error can never clear by retrying. The loop turned a hard,
actionable auth failure into five rounds of flaky-looking noise, and the
estate fact-store silently went stale: verisimdb-data pushed_at was stuck
at 2026-07-06 and was still stale on 2026-07-21 (15 days) while the
neurosymbolic pipeline kept reasoning over the frozen snapshot.
Retry remains correct for genuine contention, so this narrows rather than
removes it: classify stderr, fail fast with an actionable ::error on
auth/permission, keep the backoff for everything else.
NOTE: this makes the failure legible; it does not grant the permission.
The PAT still needs Contents:write on verisimdb-data (read already works —
the clone step succeeds — so the token is valid but under-scoped).
Verified with a stubbed `git` on PATH, all four paths:
403 denied -> 1 attempt, exit 78 (was 5 attempts, exit 128)
contention reject -> 5 attempts, exit 1 (unchanged)
immediate success -> 1 attempt, exit 0 (unchanged)
transient x2 then ok -> 3 attempts, exit 0 (unchanged, still self-heals)
shellcheck clean; workflow YAML re-parsed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hyperpolymath
marked this pull request as ready for review
July 21, 2026 05:38
… via PR
Corrects the diagnosis in the previous commit on this branch. That commit
made the 403 fail fast instead of retrying it five times, which was right,
but it named the wrong cause and told the operator to grant the PAT
Contents:write. That would not have worked.
## Actual cause
`hyperpolymath/verisimdb-data` carries an active `Base` ruleset on
~DEFAULT_BRANCH whose rules include `pull_request` and
`required_signatures`, plus `deletion`, `non_fast_forward` and
`required_linear_history`. Its only bypass actor is RepositoryRole(id=5)
in mode `pull_request`. So `git push origin HEAD:main` is refused for
everyone, admin included, whatever the token can do.
Timeline, which is what identified it:
Base ruleset created ............ 2026-04-12
bot pushed fine (16 commits) .... through 2026-06-25 17:40
Base ruleset UPDATED ............ 2026-06-26 08:57
bot pushes since ................ none
The ruleset was updated ~15h after the last successful bot push.
## Two measurement traps this exposed
- The repo's `pushed_at` read 2026-07-21 and looked healthy, because
humans kept committing. Bot-data freshness must be judged by the last
commit AUTHORED BY THE BOT — which was 2026-06-25, i.e. 26 days stale,
longer than the "frozen 2026-07-06" previously recorded.
- A hard permission wall retried five times with backoff is
indistinguishable from flakiness in the log.
## Fix
Push a feature branch (unrestricted — the ruleset targets only the default
branch), open a PR, and merge it. That satisfies `pull_request`, and
satisfies `required_signatures` because GitHub signs the commits it creates
server-side, which a runner's raw `git push` can never do. No ruleset needs
weakening and no new token is needed.
`--rebase`, not `--squash`: squashing would collapse the scans commit and
the index commit into one, breaking the invariant regen-index.sh relies on
(its `last_updated` comes from the newest commit touching scans/, so the
index must land as a separate commit that does not touch scans/). Rebase
also satisfies `required_linear_history`.
Adds a postcondition asserting the PR actually reached MERGED. A refused
merge that exits 0 is precisely the silent-failure class that let this
store go stale for 26 days.
The retained 403 handler now covers the branch push only, where a denial
genuinely *is* a token problem, and says so.
Verified: YAML parses; zero live `push origin HEAD:main` lines remain;
shellcheck -S warning on the step body returns 0 with no findings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
Correction: I named the wrong cause. It is a ruleset, not the token.The first commit on this branch made the 403 fail fast instead of retrying it five times — What is actually blocking the push
So nobody may push directly to The timeline is what settled it:
Updated roughly fifteen hours after the last successful bot push. Two measurement traps worth recording
The fix in this pushPush a feature branch (unrestricted — the ruleset targets only the default branch), open
Added a postcondition asserting the PR actually reached The retained 403 handler now guards only the branch push, where a denial genuinely Verification
Scope noteThe same |
hyperpolymath
changed the title
Owner
Author
Re-correction: the original diagnosis was right. It IS the token.I overturned this branch's first commit on circumstantial evidence and I was wrong. Deployed What the run showedThe rewrite pushes a feature branch, which no ruleset on this repo covers. It still failed: The differential test that settled itSame operation, two different tokens:
A ruleset does not discriminate by token. What misled meThe Confidence levels, stated explicitly this time
Owner action IS requiredGrant
Keeping the branch+PR code regardless
So this branch stays as-is: the code is necessary but was never sufficient, and the first |
hyperpolymath
changed the title
…s scanning The write to verisimdb-data was the LAST step of this job, after a ~300-repo panic-attack scan. Every time the token was wrong, the run burned the whole scan and then died at the final step. That is a large part of why a hard permission wall went unnoticed for 26 days: reaching the failure was expensive, and it presented as a flaky tail-end push rather than as a categorical "this cannot work". Adds a first step that creates a ref on verisimdb-data and immediately deletes it. No content, no commit, about a second. It is the same differential test that finally identified the fault — the one thing that distinguishes a token problem from a ruleset problem, since a ruleset cannot discriminate by token. Skipped when dry_run is set, since a dry run never publishes. The error message names the specific grants required (Contents: Read and write AND Pull requests: Read and write) and notes that a fine-grained PAT must list the repository explicitly rather than inheriting account scope — which is the trap that made the earlier "the token is valid, the clone worked" reasoning misleading. Verified: YAML parses, preflight is step 0, shellcheck -S warning returns 0 with no findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The branch+PR route reached the merge and was refused. GitHub said why,
verbatim:
GraphQL: Base branch requires signed commits.
Rebase merges cannot be automatically signed by GitHub (mergePullRequest)
So my previous commit's claim — that a rebase merge would satisfy
`required_signatures` — was wrong. Measured, not inferred, this time.
## The merge method is fully determined
verisimdb-data's `Base` ruleset requires `pull_request`,
`required_signatures` and `required_linear_history` on main:
direct push -> refused (pull_request required)
merge commit -> refused (required_linear_history)
rebase merge -> refused (cannot be signed) <- measured above
squash merge -> GitHub authors one new commit server-side and signs it
Squash is the only method that satisfies all three.
## Why that forces TWO PRs
A single squash of "scans + index" breaks the index contract.
scripts/regen-index.sh line 44:
last_updated="$(git log -1 --format=%cI -- "$SCAN_DIR")"
i.e. the commit time of the newest commit touching scans/. Squashing both
changes together produces ONE commit touching scans/ *and* index.json,
while index.json still carries a timestamp computed before that commit
existed — so index-freshness.yml, which reruns the same script at HEAD and
diffs, would fail.
Publishing as two squashes keeps them separate: PR 1 lands exactly one
commit touching scans/; PR 2 lands exactly one commit that does not. The
index is regenerated AGAINST MERGED main (fetch + reset --hard FETCH_HEAD
after PR 1 lands), so the timestamp it records is the real post-merge
scans commit.
Both PRs assert they reached MERGED before the step succeeds.
Verified: YAML parses; shellcheck -S warning rc 0, no findings; and the
live-code ordering (comments excluded) is
publish scans -> reset to merged main -> regen index -> publish index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run 29843933209 reported the publish step as failed. It had not failed:
verisimdb-data#54 merged at 15:37:35Z with a verified signature, and the
363-repo rescan is on main.
`gh pr merge --squash` is evaluated the instant the PR is created, before
the base-branch policy has settled, and returns
not mergeable: the base branch policy prohibits the merge
...add the `--auto` flag
then the merge lands ~30s later regardless. So the step exited 1 on a
success, and — because it exited before opening its second PR — left
index.json stale on main while scans/ had moved. That is the exact
"reports failure, actually succeeded" inverse of the fake-gate problem.
Now: --auto queues the merge, and the step POLLS for state==MERGED rather
than inferring it from the exit code of the command that requested it.
Also proven while fixing the stale index by hand (verisimdb-data#55): an
index-only PR passes index-freshness cleanly under squash, because
regen-index.sh's `git log -1 -- scans/` is unaffected by a squash that
touches no file under scans/. So the two-PR split is sound and the
"squash vs regen-index" contradiction I reported does NOT exist.
Verified: YAML parses; shellcheck rc 0; verisimdb-data main is now
d81aec63 with last_updated=2026-07-21T16:37:34 and 498 scans, and the
index-freshness gate is green on main.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
verisimdb-datais the estate fact-store — 302 repos of panic-attack scan output that thewhole neurosymbolic pipeline (
PatternRegistry → TriangleRouter → FleetDispatcher) reasonsover. It stopped updating on 2026-07-06 and was still stale on 2026-07-21 (15 days).
Nothing alerted, because
estate-rescan.ymlwas busy retrying the failure.Root cause
The push step retried any push failure with backoff (2/4/8/16s), on the reasonable
assumption of writer contention. But the actual failure was:
HYPATIA_DISPATCH_PAThas read onverisimdb-data(the clone step at line 141 succeeds)but not Contents:write. A permission error can never clear by retrying — so the loop
converted a hard, actionable auth failure into five rounds of flaky-looking noise, and the
fact-store froze silently.
This is a monitoring failure as much as a permissions one: the retry hid the signal.
Changes
.github/workflows/estate-rescan.yml— narrow the retry rather than remove it. Contentionretry is still correct and is preserved; auth/permission now fails fast with a
::errornaming the exact missing grant.
Verification
Exercised with a stubbed
gitonPATHcovering all four paths:! [rejected])shellcheckclean; workflow YAML re-parsed (jobs: ['rescan']).Action still required: grant
HYPATIA_DISPATCH_PATContents: writeonhyperpolymath/verisimdb-data. Until that lands, this workflow will now fail fast andclearly on the first attempt instead of silently wasting 30s and going stale.
After the grant, verify recovery with:
Related
Companion to #640 (arangodb health check). Both are Phase 0 foundation repair: the estate
audit cannot be trusted while hypatia's own tests are red and its fact-store is frozen.
🤖 Generated with Claude Code