Skip to content

Count shell reads and edits as evidence (CL-6937) - #559

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real
Aug 23, 2026
Merged

Count shell reads and edits as evidence (CL-6937)#559
TheGreatAxios merged 1 commit into
mainfrom
cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Closes CL-6937. Stacked on #558 (cl-6936-… base) — both touch thrash.ts / stop-policy.ts, so they cannot be disjoint lanes. Merge #558 first; this PR then retargets to main.

evaluateSubAgentStop measured real work by typed tool calls only:

  • requireEditeditedPaths.size === 0never-edited, a HARD_BLOCK_SALVAGES class, so the parent is refused an identical re-dispatch for the rest of the session.
  • requireEvidencereadCounts.size === 0incomplete-report.

So sed -i, a heredoc, or > redirection was work classified as no work. The prompt does prohibit shell file work — but buildGrokLeafAntiThrashNote documents grok reaching for shell first anyway, and grok is the family every sampled workbench session runs. A prompt violation should earn a correction, not a verdict that the work never happened.

What changed

  • classifyShellFileEvidence in src/shell/run-shell-authz.ts, reusing expandShellSubjects — the same subject expansion the auto-shell policy uses — so bash -c, env -S, and xargs payloads are inspected rather than trusted. No new parser.
  • Writes: in-place editors (sed -i, sed -i.bak, perl -pi -e), > / >> redirection including fused (>out.txt) and heredoc forms, tee, cp, mv, install, touch, truncate, patch, ln.
  • Reads: cat, head, tail, grep/rg, sed without an in-place flag, awk, diff, find, jq, and friends. Flag values are skipped, so head -n 40 f reads f, not 5.
  • Asymmetric by design: a missed write is the false salvage this exists to prevent; a missed read costs a worker nothing, since typed tools remain the primary evidence. A command with no determinable operand records shell:<program> — the checks only test non-emptiness.
  • nextThrashState folds that into readCounts / editedPaths.
  • ARCHITECTURE / PRODUCT / CHANGELOG updated in the same diff.

The ticket's second option — demoting missing evidence from a hard block to a nudge — is deliberately not done here; the sticky-hard-block machinery is CL-6939 / CL-6704 / CL-6710.

Verification

bun run check green: 5272 pass / 0 fail. 14 new tests covering the classifier and the two salvage paths (a shell-only run with a full envelope now completes under both requireEdit and requireEvidence).

Review

needs-sawyer-review — Guardrail 4: touches stop/nudge policy. Not for auto-merge, and wants an eval matrix run before merge.


Review fix applied: moved classifyShellFileEvidence (and its program/flag tables and helpers) out of src/shell/run-shell-authz.ts into a new src/subagent/shell-evidence.ts — the authz module is the security boundary and evidence detection is a different concern with a single consumer (src/subagent/thrash.ts). Exported tokenizeSegment, programBasename, HEAD_TAIL_VALUE_FLAGS, and GREP_VALUE_FLAGS from run-shell-authz.ts so the new module keeps reusing the shared parsing primitives instead of reimplementing them. Moved the corresponding tests to src/subagent/shell-evidence.test.ts.

@linear-code

linear-code Bot commented Aug 23, 2026

Copy link
Copy Markdown

CL-6937

@TheGreatAxios
TheGreatAxios force-pushed the cl-6936-re-read-thrash-hard-stop-at-rereadlimit-4-false-positives-on branch from 055690c to 928447d Compare August 23, 2026 19:49
@TheGreatAxios
TheGreatAxios force-pushed the cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real branch from 5c8466a to bdb9ced Compare August 23, 2026 19:49
The stop policy measured whether a worker did work by counting typed tool
calls only. A worker that edited with `sed -i`, a heredoc, or `>`
redirection had editedPaths empty and salvaged as never-edited — a
HARD_BLOCK_SALVAGES class, so the parent was then refused an identical
re-dispatch for the session. One that read with cat/head salvaged as
incomplete-report. Both are real work classified as no work.

The prompt does prohibit shell file work, but buildGrokLeafAntiThrashNote
documents grok reaching for shell first anyway, and that is the family we
run. A prompt violation should produce a correction, not a verdict that
the work never happened.

classifyShellFileEvidence lives in run-shell-authz.ts and reuses
expandShellSubjects, so bash -c / env -S / xargs payloads are inspected
rather than trusted. Writes are recognized from redirection as well as
from the program name, since a missed write is exactly the false salvage
this prevents, while a missed read costs a worker nothing.

Stacked on cl-6936 (same files).
@TheGreatAxios
TheGreatAxios force-pushed the cl-6936-re-read-thrash-hard-stop-at-rereadlimit-4-false-positives-on branch from 928447d to d9efb08 Compare August 23, 2026 19:57
@TheGreatAxios
TheGreatAxios force-pushed the cl-6937-shell-based-reads-and-edits-count-as-zero-evidence-so-real branch from bdb9ced to 4fd8c94 Compare August 23, 2026 19:57
@TheGreatAxios
TheGreatAxios changed the base branch from cl-6936-re-read-thrash-hard-stop-at-rereadlimit-4-false-positives-on to main August 23, 2026 20:00
@TheGreatAxios
TheGreatAxios merged commit 8374c0f into main Aug 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant