Log every approval ask and how it settles (CL-5666) - #567
Merged
Conversation
Approval volume was unmeasurable: the only durable record was a lifetime "Allow Always" grant, so every allow-once and every deny — the overwhelming majority of answers — was discarded the moment it was given. approvals.jsonl in the session dir now records each consequential decision the permission gate makes: auto-mode allow/deny, or an interactive prompt's allow-once / allow-with-scope / deny / timeout / abort. Each record carries the classifier/auto-shell rule that fired (reusing the existing rule names, not a new taxonomy), whether it was auto or interactive, a shell chain's segment count, and queued/displayed/settled timestamps. displayedAt is set from gate-wire.ts's overlay host the moment a request actually reaches the operator's screen, distinct from when it was raised — the gap is the CL-5664 signal (a queued gate arming its timeout before the operator could see it). No command text, file content, path, or credential is ever recorded — only tool name, rule, mode, segment count, and timing. Writes are fire-and-forget and swallow their own errors; the log defaults to a no-op so nothing depends on it being wired. scripts/approval-forensics.ts aggregates approvals.jsonl across local sessions the same way intervention-forensics.ts does for stop/nudge events: per-tool counts by outcome and mode, duration/display-delay percentiles, mega-chain counts, and a duplicate-rate proxy. It lstats and skips symlinks so the `latest` session link cannot double-count. On by default, per ticket comment: it's a diagnostic sink only, cannot fail a run, and never logs anything sensitive.
…fix) A sub-agent's task-dispatch description is free text the model composes itself, only ever .trim()ed, never constrained to a closed set. It had been flowing verbatim into approvals.jsonl as agentLabel, which is on-by-default and append-forever -- unlike rule (a closed set) and segments (a count), nothing stops it from carrying a path, a token, or content the model just read. Dropped the field entirely: volume, rule mix, mode split, and timing all still work without it, and no concrete question needed a per-agent breakdown badly enough to justify the risk. Added a hard size cap on the serialized record (512 bytes) as defense in depth -- every real field is a fixed enum, a count, or a timestamp, so a well-formed line should never come close to it; an oversized line is dropped rather than truncated, so no partial secret survives. Added the missing test: a fake secret embedded in a sub-agent's dispatch description, driven through the gate, asserted absent from the serialized record; and a test that a rule field stuffed with 10k chars (simulating a future regression) is dropped by the cap rather than logged. Also corrected the doc comment's claim about rule provenance: alongside the existing auto-shell-policy.ts/classify.ts rule names, this file defines its own small set of fixed literals (auto-allowed-tool, non-interactive, mega-chain) for decisions those modules don't otherwise name -- still a closed set, just not literally reused from elsewhere.
TheGreatAxios
enabled auto-merge
August 23, 2026 20:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is now measurable
Approval volume was completely dark: an exhaustive search of the session corpus found zero ask-event logs, and the only durable store was a lifetime "Allow Always" grant — every allow-once and every deny was discarded the instant it was given.
approvals.jsonl(session dir, alongsiderun.json/sent-messages.ndjson) now records one line per settled approval decision, covering all six questions in the ticket's acceptance criteria: volume per session, breakdown by rule, duplicate rate across agents (via rule+tool per session), mega-chain count (segments >= MEGA_CHAIN_SEGMENT_THRESHOLD), auto vs interactive mode, and (viadisplayDelayMs) the CL-5664 queued-vs-displayed gap.Record shape
displayedAtis set byPermissionRequest.markDisplayed, called fromgate-wire.ts'sopen()at the exact moment a queued request reaches the shell's single overlay host (see CL-5643) — not when it was raised. For the headless exec runner, which never queues, it equalsqueuedAt.No
agentLabel/ subject / path / command field exists in the record. An earlier version of this PR carried the sub-agent'staskdispatch description verbatim asagentLabel— review caught that this is model-authored free text (only.trim()ed intask-tool.ts, never constrained to a closed set), and nothing stops a model from putting a path, a token, or content it just read into its own summary of a sub-task. It's dropped entirely rather than replaced with a closed-set id: volume, rule mix, mode split, and timing all still answer the ticket's six questions without a per-agent breakdown, so there was no concrete question that justified the risk.Redaction decisions
Every remaining field is a fixed enum, a count, or a timestamp — nothing free-text:
tool(run_shell,write_file, …) andsegments(a count) are logged for shell.ruleis a closed set: the existingauto-shell-policy.ts/classify.tsrule names (file-mutation,dependency-install,sensitive-path, …), plus three additional fixed literals this file itself defines for decisions those modules don't otherwise name —auto-allowed-tool,non-interactive,mega-chain. (Correction from the original PR body, which saidruleonly reused the existing taxonomy — these three are new fixed strings, not reused ones. No privacy consequence either way since they're not model- or user-authored, but the original description of their provenance was wrong.)agentLabelwas removed (see above) rather than replaced with a closed-set identifier — no caller of the log needed a per-agent breakdown badly enough to justify sourcing and verifying a safe id for it.agentLabeldid.Tests (
src/permission/approval-log.test.ts) assert this directly: one embeds a fake secret in the shell command and asserts it's absent; a new one embeds a fake secret in a sub-agent's dispatch description, drives an approval through the gate, and asserts it's absent from the serialized record and that noagentLabelkey exists at all; another stuffs a 10k-character string intorule(simulating a future field regression) and asserts the record is dropped by the size cap rather than logged.On by default
Stated in the ticket comments before implementation: on by default, same as the CL-6938 intervention log. It's a diagnostic sink only — fire-and-forget, swallows its own write errors, defaults to a no-op when not wired — and never logs anything sensitive, so there's no cost that justifies opt-in, and opt-in would leave the sessions we most need data from unmeasured.
Reading the data back
bun run scripts/approval-forensics.tswalks~/.corbits/projects/**/approvals.jsonl(lstat-skipping thelatestsymlink so it can't double-count) and prints, per tool: total count, auto/interactive split, duration and display-delay percentiles, mega-chain count, an outcome breakdown, and a duplicate-rate proxy (sessions hitting the same rule more than once). It only ever prints aggregate counts and timings — never a subject or command, consistent with what the log itself holds.Gap confirmed
Re-verified against current
origin/mainbefore building:grep -rl approvals.jsonlandgrep -rl "ask.*settle"acrosssrc/anddocs/turned up nothing,src/permission/gate.ts/store.tsconfirmed only lifetime grants persist, and there is no branch onoriginbesides this one touching the gap. Mirrored the shape of the not-yet-mergedcl-6938intervention log (src/subagent/intervention-log.ts, PR #561) — read viagit show origin/cl-6938-...without importing from or merging that branch.What I left out
agentLabelabove) — dropped rather than fixed with a closed-set id.Noticed but did not touch
src/tui/gate-events.tscomment block documents an auto-deny timeout mechanism with "no caller arms this today" — dead plumbing kept for a future generalized auto-continue mechanism.