CL-6340: live-smoke harness for the code-review MVP - #73
Draft
TheGreatAxios wants to merge 2 commits into
Draft
Conversation
Wires @corbits/github-tools' real diff-fetch/post-review calls and @corbits/code-review's runPullRequestReview loop to a direct Anthropic Messages API call for runReviewerTurn — the seam the MVP left unbound. Confirmed against a live scratch PR: diff fetch resolves and posting works end to end; the inference pass itself needs ANTHROPIC_API_KEY, which is not configured in this sandbox.
ANTHROPIC_API_KEY is still unavailable in this sandbox. To finish proving the loop mechanics end to end, runReviewerTurn falls back to a local Ollama chat completion (localhost:11434) when no Anthropic key is set — harness-only, not a second production inference binding; the production seam remains the known gap. Ran the full loop for real against corbitsdev/pr-review-spike-test-repo#5 with qwen2.5vl:7b: diff fetch, three reviewer passes, aggregate, and a real posted review with anchored inline comments. Per-leg timings are now recorded (diff-fetch, each pass, aggregate+post, total).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/repro/live-smoke-code-review.ts, a harness that wires the merged@corbits/code-reviewreview loop to a real GitHub PR via@corbits/github-toolsand posts a real review.runReviewerTurnprefers a direct Anthropic Messages API call; whenANTHROPIC_API_KEYis absent (as in this sandbox) it falls back to a local Ollama chat completion. The Ollama path is HARNESS-ONLY — it exists to prove the loop mechanics (diff → three passes → aggregate → post) for real without a paid credential, not as a second production inference binding. The productionrunReviewerTurnbinding remains the known gap the MVP report flagged; this harness does not close it.Verified live end to end
arrayUtils.ts).qwen2.5vl:7bvia local Ollama (the strongest non-embedding completion model available on this machine).Promise.all), aggregate+post 1836, total 95310.Honest quality note
A 3-7B local model is not the production bar. Findings were directionally right (it caught the off-by-one, the mutation-in-sort, and flagged missing empty-array guards) but shallow — no distinct blocking/should-fix/later triage, some near-duplicate findings across reviewers, and its
suggestionblocks contain prose ("Change the loop condition to...") rather than literal replacement code, so a "Commit suggestion" on GitHub would not apply cleanly. This is expected from a 7B model and is exactly what the module comment says not to read as a verdict on the loop — the mechanics are what's being proven here, not finding quality.Test plan
bun install, prettier, eslint pass on the new filegh api .../pulls/5/commentsANTHROPIC_API_KEYonce available to compare finding quality against a production-grade model