feat: count shell fetches as docs activation, read Codex's web_search action

[mattrossman]

Changes how we track docs activation for Codex:

• stdout-facing shell commands like curl and wget -O - are now counted as docs calls (with content)
• The web_search tool's action payload is now used to determine whether the model opened a page, instead of inferring this based on a URL-shaped query
• Docs calls where hasContent is undefined are rendered with a distinct question mark instead of falling back to looking like content-ful calls

Example of the new SHELL docs activation tracking in the "cli 001: bootstrap app" eval run from the preview app:

Another example where the hasContent couldn't be determined:

Motivation

Our skill tells agents to fetch https://supabase.com/changelog.md and in several of our runs, Codex achieves this with curl which we're not currently counting. For comparison, Claude Code uses WebFetch which is counted.

This change aims to more accurately count changelog reads, among other docs pages.

The web_search change is less important, but still a technical accuracy thing for how we count page reads in Codex.

What's counted

This only counts commands that write page content to stdout, since we can generally trust that means the content was added to context. It counts supabase.com domains only, excluding service subdomains. resultChars is measured after whatever the agent piped the response through, so it counts what reached the model rather than the size of the whole page. If the command has ambiguous output, we treat hasContent as undefined.

It doesn't count commands that don't direct to stdout (e.g. downloading a page and then reading it from filesystem). The reasoning is that it's harder to reliably track this, as there's tons of ways that our heuristics might be inaccurate. We also don't see evidence of this happening in practice to justify the effort. We can revisit that sort of tracking once we notice it impacting results.

See tests cases for more context on what's counted vs excluded.

Impact on transcript and UI

This adds a shell_fetch type to DocsCall.source similar to existing MCP seach_docs and harness web_fetch and web_search sources.

In this UI this appears as a SHELL chip tagged on the docs call. I label it as such since it represents multiple possible shell fetch strategies, but if desired we could split into the specific binary like CURL, WGET, etc.

Shell invocations that mix multiple commands w/o piping are counted as docs calls but with hasContent undefined, since we can't be sure what content was printed to stdout as context. Those now render distinctly in the UI from known content-ful or content-less calls.

On web_search.action change

Note that although this PR starts reading Codex's web_search.action payload, there's a bug in Codex preventing this helping to its full potential. Similar to this fixed issue, there's inconsistency in the action casing between validation (camelCase vs. snake_case) that causes the open_page action (or any multi-word action) to fall back to other.

You can see this in practice by running:

codex exec --json --ephemeral --skip-git-repo-check \
  'Use web search to open https://raw.githubusercontent.com/openai/codex/main/CHANGELOG.md directly. Do not use curl or shell commands.' \
  | jq -c 'select(.item.type == "web_search") | {query: .item.query, action: .item.action}'

We can consider opening an issue in the Codex repo, and for now this change is a guard for if/when that validation issue gets fixed.

Nonetheless, the other action falls back to the existing URL-shaped query heuristic, so it's not regressing our results.

I've updated results for the Codex experiments on this PR, I didn't refresh other harnesses since I haven't observed them using shell to fetch web pages thusfar.

Closes AI-958

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Jul 31, 2026 6:18pm

[linear-code]

AI-958 Investigate incomplete Codex web search / docs activation results

Motivation: Changelog content activation is suspiciously low.

Initial research suggests that Codex's web search tool masks which underlying pages it read content from which might be under-representing it in the content-ful docs activation results.

Review in Linear

[Rodriguespn]

LGTM. We discussed offline the details of this so I'm in the loop. Nice catch and thkx for taking care of this

[barryroodt]

For the sake of completeness, I had my agent run a thorough review on this, all feedback below, along with the inline comments, is a result of that run.

Nice direction, though the shell heuristic still miscounts a few common cases, all in the direction that moves the number this PR exists to make accurate.

• echo "run curl https://supabase.com/changelog.md next" records a page read with hasContent: true. The word curl anywhere in the segment is enough.
• curl -fsSL <changelog> 2>/dev/null | head -50 records nothing at all.
• A 404 counts as a read, since curl without -f exits 0 and the error body lands in result.

Left those inline with the details. A regression test for each would be the useful part, more than the specific fix.

Separately on the results: they look re-run rather than re-scored. 71 of 190 rows differ, 53 of those changed a run-level field like attempts or checks, search_docs moved 146 to 142 even though this PR doesn't touch that branch, and 7 rows flipped pass/fail. Nothing wrong with re-running, just worth a line in the description so nobody reads those flips as effects of this change.

One structural thing I'd like your take on, and I think it's fine as a follow-up: the Codex action handling lives in docs-results.ts even though that module is meant to stay agent-agnostic.

[mattrossman]

Separately on the results: they look re-run rather than re-scored

Yep, that's just a byproduct of refreshing results on the PR w/ nondeterminism. All good.

[mattrossman]

the Codex action handling lives in docs-results.ts even though that module is meant to stay agent-agnostic.

Agreed it's not ideal, I'm okay leaving it for now because this doesn't affect the persisted results shape, and can be simplified by using the existing normalized url once the casing bug on Codex's end is resolved.