Skip to content

Log every approval ask and how it settles (CL-5666) - #567

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-5666-approval-ask-settle-event-log
Aug 23, 2026
Merged

Log every approval ask and how it settles (CL-5666)#567
TheGreatAxios merged 2 commits into
mainfrom
cl-5666-approval-ask-settle-event-log

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

What is now measurable

Approval volume was completely dark: an exhaustive search of the session corpus found zero ask-event logs, and the only durable store was a lifetime "Allow Always" grant — every allow-once and every deny was discarded the instant it was given.

approvals.jsonl (session dir, alongside run.json/sent-messages.ndjson) now records one line per settled approval decision, covering all six questions in the ticket's acceptance criteria: volume per session, breakdown by rule, duplicate rate across agents (via rule+tool per session), mega-chain count (segments >= MEGA_CHAIN_SEGMENT_THRESHOLD), auto vs interactive mode, and (via displayDelayMs) the CL-5664 queued-vs-displayed gap.

Record shape

interface ApprovalRecord {
  id: string;                 // correlates the ask with its settle
  tool: string;
  rule?: string;               // classifier/auto-shell rule name, when one fired
  mode: "auto" | "interactive";
  segments?: number;           // real shell chain segment count
  outcome: "allow-once" | "allow-with-scope" | "deny" | "auto-allow" | "auto-deny" | "timeout" | "abort";
  queuedAt: string;            // request raised
  displayedAt: string;         // request actually reached the operator's screen
  settledAt: string;
  durationMs: number;          // settledAt - queuedAt
  displayDelayMs: number;      // displayedAt - queuedAt (CL-5664 signal)
}

displayedAt is set by PermissionRequest.markDisplayed, called from gate-wire.ts's open() at the exact moment a queued request reaches the shell's single overlay host (see CL-5643) — not when it was raised. For the headless exec runner, which never queues, it equals queuedAt.

No agentLabel / subject / path / command field exists in the record. An earlier version of this PR carried the sub-agent's task dispatch description verbatim as agentLabel — review caught that this is model-authored free text (only .trim()ed in task-tool.ts, never constrained to a closed set), and nothing stops a model from putting a path, a token, or content it just read into its own summary of a sub-task. It's dropped entirely rather than replaced with a closed-set id: volume, rule mix, mode split, and timing all still answer the ticket's six questions without a per-agent breakdown, so there was no concrete question that justified the risk.

Redaction decisions

Every remaining field is a fixed enum, a count, or a timestamp — nothing free-text:

  • No command text, ever. Only tool (run_shell, write_file, …) and segments (a count) are logged for shell.
  • No file paths, subjects, or arguments.
  • rule is a closed set: the existing auto-shell-policy.ts / classify.ts rule names (file-mutation, dependency-install, sensitive-path, …), plus three additional fixed literals this file itself defines for decisions those modules don't otherwise name — auto-allowed-tool, non-interactive, mega-chain. (Correction from the original PR body, which said rule only reused the existing taxonomy — these three are new fixed strings, not reused ones. No privacy consequence either way since they're not model- or user-authored, but the original description of their provenance was wrong.)
  • agentLabel was removed (see above) rather than replaced with a closed-set identifier — no caller of the log needed a per-agent breakdown badly enough to justify sourcing and verifying a safe id for it.
  • Hard size cap: the serialized line is capped at 512 bytes and dropped (not truncated) if it would exceed that — defense in depth so a future field reintroducing free text can't turn this into a leak even if it slips past review the way agentLabel did.

Tests (src/permission/approval-log.test.ts) assert this directly: one embeds a fake secret in the shell command and asserts it's absent; a new one embeds a fake secret in a sub-agent's dispatch description, drives an approval through the gate, and asserts it's absent from the serialized record and that no agentLabel key exists at all; another stuffs a 10k-character string into rule (simulating a future field regression) and asserts the record is dropped by the size cap rather than logged.

On by default

Stated in the ticket comments before implementation: on by default, same as the CL-6938 intervention log. It's a diagnostic sink only — fire-and-forget, swallows its own write errors, defaults to a no-op when not wired — and never logs anything sensitive, so there's no cost that justifies opt-in, and opt-in would leave the sessions we most need data from unmeasured.

Reading the data back

bun run scripts/approval-forensics.ts walks ~/.corbits/projects/**/approvals.jsonl (lstat-skipping the latest symlink so it can't double-count) and prints, per tool: total count, auto/interactive split, duration and display-delay percentiles, mega-chain count, an outcome breakdown, and a duplicate-rate proxy (sessions hitting the same rule more than once). It only ever prints aggregate counts and timings — never a subject or command, consistent with what the log itself holds.

Gap confirmed

Re-verified against current origin/main before building: grep -rl approvals.jsonl and grep -rl "ask.*settle" across src/ and docs/ turned up nothing, src/permission/gate.ts/store.ts confirmed only lifetime grants persist, and there is no branch on origin besides this one touching the gap. Mirrored the shape of the not-yet-merged cl-6938 intervention log (src/subagent/intervention-log.ts, PR #561) — read via git show origin/cl-6938-... without importing from or merging that branch.

What I left out

  • Two-line ask/settle join: chose one record per completed decision (written at settle time) instead of a separate line at ask-time, since every decision path here settles synchronously or near-synchronously; a hard crash mid-ask loses that one record, same tradeoff the intervention log makes.
  • Per-agent breakdown (see agentLabel above) — dropped rather than fixed with a closed-set id.
  • Project-grant near-miss detection (CL-5662) and queue-drain coverage (CL-5663) — the ticket's sequence note says these are confirmed by source reading and don't need volume data; out of scope here.

Noticed but did not touch

  • src/tui/gate-events.ts comment block documents an auto-deny timeout mechanism with "no caller arms this today" — dead plumbing kept for a future generalized auto-continue mechanism.
  • Only 2/38 project-scoped grants and 0/58 global grants are cwd-scoped per the ticket's own numbers — consistent with CL-5662 (worktree cwd mismatch making project scope silently useless), left alone per the sequence note.

Approval volume was unmeasurable: the only durable record was a lifetime
"Allow Always" grant, so every allow-once and every deny — the overwhelming
majority of answers — was discarded the moment it was given.

approvals.jsonl in the session dir now records each consequential decision
the permission gate makes: auto-mode allow/deny, or an interactive prompt's
allow-once / allow-with-scope / deny / timeout / abort. Each record carries
the classifier/auto-shell rule that fired (reusing the existing rule names,
not a new taxonomy), whether it was auto or interactive, a shell chain's
segment count, and queued/displayed/settled timestamps. displayedAt is set
from gate-wire.ts's overlay host the moment a request actually reaches the
operator's screen, distinct from when it was raised — the gap is the CL-5664
signal (a queued gate arming its timeout before the operator could see it).

No command text, file content, path, or credential is ever recorded — only
tool name, rule, mode, segment count, and timing. Writes are fire-and-forget
and swallow their own errors; the log defaults to a no-op so nothing depends
on it being wired.

scripts/approval-forensics.ts aggregates approvals.jsonl across local
sessions the same way intervention-forensics.ts does for stop/nudge events:
per-tool counts by outcome and mode, duration/display-delay percentiles,
mega-chain counts, and a duplicate-rate proxy. It lstats and skips symlinks
so the `latest` session link cannot double-count.

On by default, per ticket comment: it's a diagnostic sink only, cannot fail
a run, and never logs anything sensitive.
@linear-code

linear-code Bot commented Aug 23, 2026

Copy link
Copy Markdown

CL-5666

…fix)

A sub-agent's task-dispatch description is free text the model composes
itself, only ever .trim()ed, never constrained to a closed set. It had been
flowing verbatim into approvals.jsonl as agentLabel, which is on-by-default
and append-forever -- unlike rule (a closed set) and segments (a count),
nothing stops it from carrying a path, a token, or content the model just
read. Dropped the field entirely: volume, rule mix, mode split, and timing
all still work without it, and no concrete question needed a per-agent
breakdown badly enough to justify the risk.

Added a hard size cap on the serialized record (512 bytes) as defense in
depth -- every real field is a fixed enum, a count, or a timestamp, so a
well-formed line should never come close to it; an oversized line is
dropped rather than truncated, so no partial secret survives. Added the
missing test: a fake secret embedded in a sub-agent's dispatch description,
driven through the gate, asserted absent from the serialized record; and a
test that a rule field stuffed with 10k chars (simulating a future
regression) is dropped by the cap rather than logged.

Also corrected the doc comment's claim about rule provenance: alongside the
existing auto-shell-policy.ts/classify.ts rule names, this file defines its
own small set of fixed literals (auto-allowed-tool, non-interactive,
mega-chain) for decisions those modules don't otherwise name -- still a
closed set, just not literally reused from elsewhere.
@TheGreatAxios
TheGreatAxios merged commit e9c085d into main Aug 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant