Skip to content

Fix WebMCP tool references and document MiMo/Luna retest - #8

Merged
jo32 merged 1 commit into
mainfrom
codex/webmcp-tool-references
Sep 23, 2026
Merged

jo32 merged 1 commit into
mainfrom
codex/webmcp-tool-references

Conversation

@jo32

@jo32 jo32 commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Problem and change

Models sometimes copy the catalog digest into a WebMCP tool revision or miscopy frame/document identifiers, causing avoidable rejected calls and recovery loops. Discovery now returns session-local toolRef handles; the browser plugin resolves each handle to the exact discovered identity and schema before dispatch.

References expire on tab/document/frame/version/schema changes. Legacy identity calls remain strictly validated. Errors distinguish not_dispatched from an unknown execution outcome and provide recovery guidance; uncertain actions are never automatically replayed. All runtime changes are in the Cordis browser plugin; upstream Harness and Jev changes are excluded.

Benchmark evidence

Retested six error-heavy WebMCP ON cases with MiMo V2.6 Flash and GPT-5.6 Luna, three attempts per case/model. All 36 formal attempts passed. Old baseline is one attempt per case; these tasks previously passed after tool failures, so this measures efficiency, not an increase in task success rate.

Six-case suite, old → new mean MiMo Luna
WebMCP call error rate 75.0% → 27.5% 38.9% → 21.6%
Agent time 20.04 → 5.75 min 3.07 → 3.02 min
Agent steps 108 → 60.3 46 → 41.7
Estimated USD, actual cache usage $0.05645 → $0.02633 $0.04017 → $0.03624

This is a selected historical comparison, not a full-suite or contemporaneous A/B claim. Auth-boundary exploration regresses in steps/cost; Luna is also slower on auth-boundary and course enrollment. The report preserves these counterexamples and documents nine excluded MiMo configuration-calibration attempts and three Luna scoring-auth failures replaced by a complete three-attempt rerun. All 48 recorded attempts are included in the usage ledger. Luna costs are API-equivalent estimates, not subscription charges.

The committed report and sanitized per-attempt data include usage, rates, exclusions, operation-evidence checks, source hashes and raw-report hashes. Raw transcripts and credentials are not published. Paid benchmarks ran on the recorded working-tree snapshot before this PR was assembled; the seven code/test/doc files were copied unchanged onto current main.

Validation

On this isolated PR branch:

  • Browser plugin type check, build and 162 tests across 17 files passed.
  • Real Electron test passed native tool execution, catalog-digest rejection, stale-reference rejection after navigation and generated-tool version replacement.
  • Exported costs were recomputed from recorded token usage; report links and staged diff checks passed.

Before PR isolation, the source workspace also passed pnpm check, pnpm build, and the full test suite with NODE_OPTIONS=--no-webstorage (needed for Node 25's localStorage behavior). These full-workspace checks were not rerun on the isolated PR branch.

@vercel

vercel Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
deepdeck Ready Ready Preview Sep 23, 2026 9:46am UTC

Request Review

@jo32
jo32 merged commit 9f8e152 into main Sep 23, 2026
4 checks passed

This branch was successfully deployed

1 active deployment
Preview — a607ab5a Deployed Sep 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant