Tolerate poisoned session entry payloads and sanitize every Postgres text write

[ReganBell]

Rebased onto main after the runtime consolidation merged; this is the same change, eleven small commits, 39 files.

What broke

A session whose stored history contained one legacy entry with a \u0000 escape failed every turn with:

unsupported Unicode escape sequence

That is Postgres refusing to cast text to jsonb when the text contains the six-character escape \u0000. jsonb cannot store NUL; text (and json) can. The compaction-window change (the getContextWindow read) casts every session_entries.payload of the session to jsonb on every turn, so a single such entry made every turn and every background compaction die before the model was called. The affected rows predate the write-side NUL stripping and were only three across one deployment, but one of them sat in an active session.

The producer

The rows came from extractDirectives in the Slack layer: it masks code spans with a \u0000CODE<n>\u0000 placeholder and handed the masked text to the directive callback, so an agent-to-agent task that quoted inline code was stored with the placeholder. Capture groups are now unmasked before the callback, so the placeholder never leaves the function.

Read side

An every-boot maintenance statement defines safe_jsonb(text) (returns jsonb, or NULL when the text will not parse), mirroring the existing safe_json. It is eagerly validated (so a lone-surrogate row no longer throws out of ->>) and repairs the text inside its exception branch when a clean parse fails, with one tokenizing regexp_replace: escaped backslashes and valid surrogate pairs are consumed as tokens and kept; NUL and lone-surrogate escapes are dropped. No lookbehind, so it is linear (100k escapes in about 0.1 s). Verified on Postgres 16 across eighteen cases including adjacent escapes and escaped-backslash literals next to real escapes.

Every read of a stored payload (taint check, summary lookup, tape coverage, admin previews, search authors) parses with it behind a LIKE prefilter, each row once (OFFSET 0 fences), so one poisoned row cannot abort a query. The summary lookup is a flat backward walk of the primary key under LIMIT 1.

clearSecurityTaint reads the candidate rows, drops the flag in JavaScript and writes them back compact through the same serializer every write uses, so cleared rows keep matching the compact substring probes elsewhere in the store and the Postgres and memory stores share one rule through a single withoutSecurityTaint helper. It also clears the mirrored session_tape.security_tainted column for every tainted tape row of the session, so sessions released before this change catch up; the memory store does the same.

The taint check fails closed: candidate rows are those whose text contains the key and colon "securityTainted": (which also matches jsonb's spaced form, since the space follows the colon), the parsed top-level value decides, and a candidate that Postgres cannot parse even after repair counts as tainted on both the read and the clear path, so the two cannot disagree about a row. A tainted row nothing can parse makes clearSecurityTaint clear nothing and throw a typed TaintUnclearableError. The input-approval path clears taint before consuming the approval; on that error it records the detail (session and rows) to the operator error log and the approval audit, then returns a pending-approval result carrying only a plain reason, which Slack renders as a note above a redrawn card with its buttons and the web shows to the user, so the approver learns an operator must repair the history and the approval stays answerable. A transient database error during the clear takes the same shape with a retry hint instead of burning the run's attempts; both branches write the approval audit row. The base swallowed every failure here; other errors now propagate. The admin release endpoint maps it to 409 with the row list and audits the refusal.

Write side

jsonbSafeStringify applies pgTextSafe (NUL dropped, lone surrogates replaced with U+FFFD via the built-in well-formedness methods) to every string and key and is used by every jsonb writer: session entries, tape, run signals, deliveries, surface cache, channel policy, run activity. durable-map had its own copy; it is folded in. TEXT columns that carry user or model text go through pgTextSafe as well (tape text, delivery text, surface-cache author and text, session and participant titles, pins, search rows, run-signal text, standing orders, ack-emoji picks, ambient judgments, file artifact names and mimetypes, audit detail and resource, scoped event sinks, process commands, environment names, directory display names, session channel names, fork titles, memory revision bodies). Identity keys (ACL refs, dedupe keys) are deliberately left alone. The five request fields that are stored and later compared (text, display text, conversation header, overheard messages, attachments) are normalized once where the request enters app-turn with pgSafeValue, which returns the same object when nothing needs sanitizing, so stored text and request text agree in every downstream compare at no cost on the common path. Identifiers (thread refs, actor ids, request and idempotency keys, ACL refs) are never rewritten. Session channel names are sanitized once at the top of the thread lookup so the heal compare and the insert see the same value; attachment names and artifact paths are sanitized where they are built, and the file-write tool rejects a path carrying a NUL or unpaired surrogate instead of silently renaming the file, so ACL grants, audits and artifact rows share one value. The helper only re-serializes when the plain output carries an escape jsonb rejects.

Tests

test/context-window.test.ts (added to test:pg) seeds legacy NUL-escape and lone-surrogate rows, a scalar-string payload quoting the flag, a row whose text literally contains \u0000, a row with a nested flag inside tool arguments, a mid-emoji slice, a poisoned latest summary, and a flag rewritten in jsonb's spaced form, then asserts the window resolves, taint surfaces and clears on every row, healthy rows round-trip byte-identical and compact, and the slice was stored as U+FFFD. Further tests cover the orchestrator's approval branch when release fails (pending result with reason, refused audit row, operator error record, same approval succeeding after repair), a session whose only tainted row is poisoned, a tainted row nothing can parse (fails closed, throws naming the row, leaves other rows untouched), the 409 route with its audit row, directive capture groups carrying quoted code, and key sanitization. All fail on the base branch.

Follow-ups (pre-existing, not changed here)

• entry_search_text (the generated search column) still strips the NUL escape unconditionally and returns NULL when the result will not parse, so a message whose text literally quotes \\u0000, and legacy lone-surrogate rows, preview but do not search. This predates the change. Redefining the function over safe_jsonb is cheap, but the STORED column only recomputes on write, so existing rows need a backfill; deferred with that.
• safe_json has no callers after this change but stays defined by the frozen first migration; dropping it must wait until no older build runs against the database.
• rowToEntry still parses with a bare JSON.parse, so a row malformed beyond the NUL and surrogate classes (nothing writes those) would still fail reads that reach it; TaintUnclearableError names such rows when they are tainted.
• clearSecurityTaint previously re-serialized rows through jsonb, which produced the spaced form "overheard": true that the compact userTurn probes miss; rows already rewritten that way are unaffected by this change.
• Per-site pgTextSafe wrappers on TEXT params could collapse into one sanitizer at the pool layer.
• rowToEntry still parses with a bare JSON.parse; a row malformed beyond the NUL and surrogate classes (nothing writes those) inside the live window would still fail reads, and there is no admin tool to rewrite or drop a single entry. A security_tainted column on session_entries (the tape already has one) would replace the substring probes entirely; both are bounded follow-ups rather than part of this fix.
• Deployments that once carried the older safe_jsonb body may still hold a dead expression index on it; drop it with DROP INDEX CONCURRENTLY out of band.