feat: Broad-Side — batch reconnaissance over the OpenRouter Batch API (#103) - #144
Merged
Conversation
…103) New codecarto_broadside MCP tool (submit/collect/status) fires six single-turn analysis lenses — architecture, API surface, security, mechanical defect scan, convention extraction, porting — at any git repository as asynchronous OpenRouter batch jobs. Large modules are sliced by top-level directory and oversized slices split rather than truncate. Results land in .codecarto/broadside/<run>/ as JSON plus rendered markdown, with an optional cross-lens synthesis report. Deliberately not the interactive pipeline: batch is text-in/text-out, so findings are framed as unverified scouting leads for the real analysis to confirm. Works without an initialized workspace; init now tolerates a scout-only .codecarto/, and scaffold refresh never touches broadside state, config, or results.
…row guard (#103) Three failures surfaced by running Broad-Side on its own repo: - The poll loop retried a dead key for the whole budget. fetchBatch now surfaces the HTTP status and the poller returns auth-failed fast on 401/403, which collect treats as terminal. - A lens whose globs match no files submitted an empty batch (rejected by the API) and blocked the synthesis terminal check forever. Empty lens slices are now marked skipped before submission, and skipped/rejected are terminal statuses for collect and synthesis gating. - A network-level throw during submission stranded the batch entry in 'submitting' with no batch id. The submission guard now records the failure as rejected with the error message. Also broadens the non-Go api-lens globs (mcp-server/**, routes/router/ handler/endpoint name patterns) so repos whose server code does not live under server/ or api/ still get scanned.
… limit (#103) Before submitting, Broad-Side now estimates the run cost from the collected file sizes (approx 4 chars/token) against the configured model's real per-token pricing: - Live lookup from OpenRouter's model catalog, cached 24h in broadside/pricing-cache.json. Unknown models that cannot be priced refuse to submit rather than silently estimating at default rates. - config.yaml gains model (now actually wired through submissions), max_cost, and pricing.input_per_m/output_per_m overrides. - The MCP tool gains max_cost and force parameters; a submit whose estimate exceeds the limit refuses with a per-lens breakdown and creates no run entry unless force: true. - The submit response reports the pricing used and its source (built-in/config/live/cache), and run-meta.json records both pricing and the limit for audit. Motivated by the expensive end of the batch catalog (e.g. openai/gpt-5.2-pro:batch at ~$84/M output tokens): an estimate against the wrong model's rates is a wrong guardrail.
…bility pre-flight (#103) Wired OpenRouter's model catalog and benchmarks endpoints into Broad-Side (field shapes per the official OpenRouterTeam/skills references): - New action 'models' lists every :batch variant — pricing per million tokens, context window, completion ceiling, structured-output support — cheapest first with the configured model marked, and optionally annotates each with its Artificial Analysis coding index (GET /api/v1/benchmarks, attribution preserved). - The pricing cache becomes a full model-catalog cache (shared by the models action and submit-time resolution). - Submit pre-flight: lens max_tokens clamps to the provider's completion ceiling (a request above it fails the whole batch), deprecated models are flagged in the submit text, and models that do not advertise structured-output support are refused outright — every lens depends on json_schema response_format. - The default model's capabilities are asserted from its shipped configuration (1M ctx, 64K out, structured outputs) so the built-in path needs no network. 5 new tests (catalog filtering/sort, structured-output refusal, max_tokens clamping, benchmark slug mapping, models table rendering); 373 total, all passing.
#133) Reviewing OpenRouter's standard patterns (OpenRouterTeam/skills create-headless-agent) surfaced three adoptable ideas: - Fence-tolerant output parsing: lens content wrapped in markdown code fences now parses as JSON, mirroring the scaffold's --output-schema tolerance. Saved .json files hold clean parsed output instead of verbatim fenced text. - Truncation is spoken: output that still does not parse after fence stripping (the signature of a max_tokens cutoff) is saved verbatim but flagged truncated. Collect summaries, run-meta.json, and the synthesis prompt all report it, so an unscouted module is never mistaken for a clean one. - Invariants written down: the resubmission-safety rule (batch requests are pure, so retries are always safe — load-bearing if server tools ever arrive) and the distinction between the pre-flight max_cost estimate and OpenRouter's runtime cost accounting, in the module header, SKILL.md, and config.yaml. Catalog field shapes are attributed to the official skills repo. New feat issue #143 records the headless-agent lens-queue question and the hybrid verify-pass idea. 4 new tests; 376 total, all passing.
…ndings (#103, #135) Collect now runs a second cross-lens post-pass alongside synthesis. Triage scores every lead by impact × fix difficulty and emits a work order: P0–P3 priority, effort estimates, per-module grouping, deduplicated leads, and explicit omitted notes for dropped items — saved as triage.json/triage.md, surfaced in the collect summary, and framed in-prompt as a starting point for re-verification, never a commitment. Both post-passes submit as separate batches together and poll independently; the state file tracks each so a resumed collect can finish whichever is still pending (older runs without a triage entry are upgraded in place). Skip with include_triage: false on collect. 2 new tests (triage flow, include_triage opt-out); 35 broadside tests, 377 total, all passing.
TheAmericanMaker
added a commit
that referenced
this pull request
Aug 23, 2026
…145) Five defects found by running Broad-Side on its own repo, each verified by hand before fixing: - #128 writeLibraryConfig derives the parent dir with dirname(); the forward-slash match treated Windows paths as bare filenames and ENOENT'd on write. - #130 isWithinPath respects roots that already end in a separator ("/", "C:\") instead of building a double-separator prefix that rejected every subpath. - #131 acquireLock closes the lock descriptor when writeFile throws instead of leaking it to GC. - #129 the phase runner settles the compaction promise with false when no compaction fired, so continuations stop paying the full 30s settle timeout. - #132 completeValidatedPhase re-validates the output inside the status lock; a stale PASS whose output changed refuses with a clear error and leaves status untouched. Validations with no outputPath keep the legacy synthetic path used by unit tests. Six tests in tests/broadside-scan-fixes.test.mjs. Two are true regression guards, failing against the unfixed code: #130 and the #132 stale-PASS refusal. The other three pin behavior around a fix they cannot provoke from this suite — #128 is a Windows-only path bug POSIX cannot reproduce, #131 covers acquire/release rather than the writeFile-throws path, and #129 covers waitForCompaction rather than runPhase where the fix lives. The file header records which is which so a later reader does not assume those three are protected. Merged main (Broad-Side, #144) into the branch to resolve a CHANGELOG conflict under [Unreleased]; both sections kept, Added before Fixed. No source overlap with #144. 383 tests passing, tsc clean. Closes #128, #129, #130, #131, #132.
TheAmericanMaker
added a commit
that referenced
this pull request
Aug 23, 2026
…127) deriveSlug uses only the trailing path segment of source_repo, so acme/whisper and openai/whisper both produce "whisper" and publishing the second wrote it as v2 of the first. The entry's version history then spanned two unrelated codebases, and because buildIndexEntry reads only the newest version's metadata, index.yaml attributed the whole entry to whichever repo published last. Nothing reported it, at publish time or after. publishEntry now reads the source_repo recorded on the newest version and fails before writing anything when it denotes a different repository. The check sits ahead of the content-hash branch: identical spec bytes take the metadata-only path, which overwrote the other project's source_repo and headline in place, so guarding only the new-version path would have left the quieter half of the bug intact. forceNewVersion does not bypass it — that option means "another version of this entry", not "overwrite a different project". Unreadable or malformed recorded metadata skips the check, since refusing on unknown would turn a corrupt v1 into an entry nobody can publish to. Comparison is normalized so re-publishing one repository spelled another way is unaffected: scheme or none, embedded credentials (git@, user:token@), git@host:path SCP syntax, a default port, a www. prefix, a trailing .git, repeated and trailing slashes, and separators. A non-default port is kept, since two services on one host can differ by port alone. Case folds for hosts, for the repository paths the forges serve over them, and for Windows drive paths — all case-insensitive. It does not fold for absolute POSIX paths: /srv/Repos/tool and /srv/repos/tool are two directories on a case-sensitive filesystem, and folding them let a genuine two-project collision through the guard silently. Pi records the analyzed directory as source_repo, so local paths are the common shape on that surface. Host aliases (ssh.github.com) and provider-specific SSH path layouts stay deliberately distinct; docs/library-format.md says so. The refusal names where each remedy lives rather than telling the caller to pass an option it may not expose — /codecarto-publish takes no arguments and allow_source_repo_change is MCP-only. Closing that gap on Pi is #146. library-format.md previously promised auto-suffixing (-2, -3) for this case. It was never implemented, and silent suffixing is its own surprise for an agent-driven tool, so the doc now describes the refusal and the override. Twelve tests. Three fail with the guard disabled (the two-project collision, the metadata-only path, forceNewVersion) and four more fail against the pre-review normalization (the equivalence set, the case rules both ways, an end-to-end refusal of two local directories differing only in case, and the message contract). Merged main to resolve a CHANGELOG conflict under [Unreleased]; no source overlap with #144 or #145. 395 tests passing, tsc clean. Fixes #123. Follow-ups filed: #146, #147, #148.
TheAmericanMaker
added a commit
that referenced
this pull request
Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Broad-Side
Cheap, unattended, multi-lens reconnaissance for any git repository: fire every analysis lens at once as OpenRouter batch jobs (~50% of sync pricing), collect JSON + markdown findings, and optionally synthesize one cross-lens executive report.
What shipped
codecarto_broadsideMCP tool — actionssubmit,collect,status,models. Works on any git repo; no initialized workspace required.json_schemaoutput): architecture, api, security, defect, conventions, porting. Large modules slice by top-level directory; oversized slices split rather than truncate.max_costlimit enforced pre-submit with per-lens breakdown,forceoverride. Models without structured-output support are refused outright (every lens needsjson_schema); lensmax_tokensclamps to the provider's completion ceiling.modelsaction — lists every:batchvariant with pricing, context, output caps, structured-output support, and optional Artificial Analysis coding indices (attribution preserved).truncatedflags surfaced in collect summaries,run-meta.json, and the synthesis prompt).codecarto_inittolerates a scout-only.codecarto/; scaffold refresh never touches broadside state/config/results; GUIDE.md first-read points sessions atbroadside/synthesis.md.The contract
Broad-Side findings are unverified scouting signals, never evidence — file:line leads for the interactive pipeline (or a human) to confirm. Stated in the tool description, every output file,
SKILL.md, and every synthesis prompt. This is the division of labor: a ~$0.50 unattended sweep that tells the expensive interactive run where to look.Verified live, twice
Real-API runs against the ollama fork (311k lines of Go):
f5a7703)Tests
34 new unit tests (fake-fetcher based — no network in CI) + an opt-in live smoke script that skips without
OPENROUTER_API_KEY. 376/376 passing locally; CI runs Node 22 + 24.Out of scope (see ROADMAP.md)
Pi extension UI (#138), pipeline phase (#139), zero-config executive (#140), per-language prompts (#137), concurrent polling (#136), triage lens (#135), auto-resubmit of truncated slices (#133), incremental re-scouting (#142), headless-agent variant (#143).
Closes #103. Related: #128–#143.