From 6ecaf8f95621b0b81cdfe6c2a05794e88ac911e6 Mon Sep 17 00:00:00 2001 From: agentforce314 <273884145+agentforce314@users.noreply.github.com> Date: Sun, 16 Aug 2026 02:40:35 -0700 Subject: [PATCH] docs(harbor): full pi-extension pairing for the nano arm The pi-matched configuration now names both kwargs (--ak vision=openai:gpt-5.6-luna --ak websearch=1 / NANO_VISION + NANO_WEBSEARCH via the runner) with the measured per-task vision/websearch usage that motivated them, and names stock-vs-stock as the other clean pairing. Co-Authored-By: Claude Fable 5 --- eval/harbor/RUN_NANO_TB21.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/eval/harbor/RUN_NANO_TB21.md b/eval/harbor/RUN_NANO_TB21.md index 9907134b..27f0b5a0 100644 --- a/eval/harbor/RUN_NANO_TB21.md +++ b/eval/harbor/RUN_NANO_TB21.md @@ -77,12 +77,15 @@ what the A/B measures. Don't describe the runs as "same tools". **Matching a pi run that used the TB extension** (vision_analyze + websearch — e.g. `jobs/tb21-pi-flash-max-2`, which ran -`vision_model: gpt-5.6-luna`): add `--ak vision=openai:gpt-5.6-luna` to -the nano arm so nano registers its identically-named `vision_analyze` -tool. The 2026-08 run analysis found 4–5 of pi's exclusive wins used -vision/websearch (chess-best-move, video-processing, path-tracing…); -without the kwarg those tasks measure a capability gap, not the harness. -Nano has no websearch analog wired yet — note it when comparing. +`vision_model: gpt-5.6-luna` with TAVILY websearch): add +`--ak vision=openai:gpt-5.6-luna --ak websearch=1` to the nano arm +(runner: `NANO_VISION=openai:gpt-5.6-luna NANO_WEBSEARCH=1`) so nano +registers its identically-named `vision_analyze` plus WebSearch. The +2026-08 run analysis found 4–5 of pi's exclusive wins used +vision/websearch (chess-best-move ×3 vision, video-processing ×11+3, +mteb-leaderboard ×2 web, path-tracing…); without the kwargs those tasks +measure a capability gap, not the harness. Stock-vs-stock (no kwargs on +nano, `--ak tools=off` on pi) is the other clean pairing. For a clean head-to-head against pi, pair **nano** with **stock pi**