test(openapi-mcp): measure Wave-0 P1–P3 proofs

[rickylabs]

Summary

Measures the three RFC #1123 Wave-0 proofs before any OpenAPI→MCP contract freezes: Aspire post-allocation discovery (P1), real-scaffold OpenAPI fidelity/size (P2), and auth-guarded spec behavior plus spec_unavailable wording (P3). This PR changes proof/run artifacts only; it does not productize template or package behavior.

All requested local gates and the separate-session IMPL-EVAL are complete. The PR remains draft for the orchestrator's readiness/merge decision. P2's committed verdict is a legitimate explicit FAIL, but #1128 remains open because its required DB-backed measurement could not be completed.

Scope

• Archetype / area: proof/measurement · tooling · service
• Part of Epic: OpenAPI→MCP service introspection — agent-legible service APIs #1126
• Closes [openapi-mcp S1] P1 proof: post-allocation endpoint-manifest seam (arbitrates F1) #1127
• Part of [openapi-mcp S2] P2 proof: spec fidelity + size dry-run against a real scaffold #1128 — closing keyword withheld because the DB-backed acceptance evidence is incomplete
• Closes [openapi-mcp S3] P3 proof: auth-guarded spec fixture + spec_unavailable wording #1129
• No packages/** or plugins/** public-surface change

Slices

• Harness bootstrap — b0be3673e
• Locked Plan/Design + separate PLAN-EVAL PASS — 1fc6e3935
• P1 explicit FAIL and qualified F1(b) arbitration — a9a28c4d1
• P2 explicit FAIL: no-DB fidelity/size measured; DB live spec unavailable — 5b0ba26b5
• P3 auth-guarded fixture and ratified spec_unavailable wording — 5c041bdfd
• RFC §9 and epic Epic: OpenAPI→MCP service introspection — agent-legible service APIs #1126 synchronized with P1
• Final gates, resource hygiene, and evaluator handoff — 65ea2304a, 24ebdc515

Definition of Done

• Separate-session PLAN-EVAL is PASS before implementation
• proofs/P1-verdict.md contains measured evidence and selects qualified F1(b)
• proofs/P2-verdict.md records complete no-DB operationId/size/error/keyword measurements and the explicit failed DB branch
• proofs/P3-verdict.md contains the observed fixture result and ratified wording
• A skipped/unrun proof cannot be represented as PASS
• RFC RFC: OpenAPI→MCP — making a service's own API legible to the agent building it (#1117) #1123 §9 and epic Epic: OpenAPI→MCP service introspection — agent-legible service APIs #1126 record P1's measured outcome
• Scoped check/lint/fmt gates pass for touched TypeScript; no new lint ignores
• deno.lock is unchanged and deno task e2e:cli was not run
• Leak check reports no run-owned residue; foreign resources remain untouched
• Separate-session IMPL-EVAL is PASS
• Every issue named by a closing keyword has linked acceptance evidence

Validation

• Plan-Gate — PASS: separate Qwen 3.7 Max/high verdict in plan-eval.md
• Touched proof/verdict/run-artifact formatting — PASS; IMPL-EVAL records non-blocking historical Markdown drift in older review artifacts
• P1 — explicit FAIL / qualified F1(b); scoped TypeScript check/lint/fmt PASS; exact teardown/resource hygiene PASS; separate Fable re-review APPROVED
• P2 — explicit FAIL: attributed no-DB spec measured (3657 bytes, 3 dotted operationIds, no non-2xx envelope, complete key audit); DB live measurement unavailable; separate Fable re-review APPROVED
• P3 — PASS: focused auth fixture independently rerun (401/403 exact envelopes, authorized 200); D9 wording byte-identical; separate Fable review APPROVED
• Scope/lock — PASS: no packages/**, plugins/**, or deno.lock diff; seed RFC §9 contains the required additive P1 decision record
• Final hygiene — PASS: zero run-owned survivors; foreign/unproven resources untouched; 0 review threads / 0 unanswered
• IMPL-EVAL — PASS: separate guarded Qwen 3.7 Max/high session; evaluate.md committed at 24ebdc515
• Full CLI E2E — intentionally not run per orchestrator contract

Harness

• Run dir: .llm/runs/test-openapi-mcp-wave0-proofs--wave0/
• Phase: complete; draft PR disposition belongs to the orchestrator
• Commit trail: this PR's commit list plus per-slice comments

Drift / Debt

• Owner-addressed Codex supervisor route recorded; implementation/review/formal evaluation remain separate canonical sessions.
• Evaluator preflight/guard retries are recorded without secrets; the successful verdict stayed on the required open Qwen route.
• P1 exposed a generated SQLite --allow-ffi permission defect; product remediation is explicitly outside this proof PR and recorded as rescope guidance.
• P2 records the DB branch as FAIL and leaves [openapi-mcp S2] P2 proof: spec fidelity + size dry-run against a real scaffold #1128 acceptance/closing keyword open; its no-DB measurement is reviewed partial progress.
• A stale-path P3 launch was interrupted before mutation; only the corrected isolated medium-effort P3 thread produced evidence.
• IMPL-EVAL records two non-blocking low findings—missing ## SKILL chapters in six historical advisory briefs and formatting drift in nine historical brief/review Markdown files—without changing the formal PASS.

[rickylabs]

[PHASE: RESEARCH]

Wave-0 proof run bootstrapped at .llm/runs/test-openapi-mcp-wave0-proofs--wave0/ on baseline fb75cf6fc5ad02130ada0ac42e6f44035ac03a9b.

Findings

1. RFC §4/§9 and [openapi-mcp S1] P1 proof: post-allocation endpoint-manifest seam (arbitrates F1) #1127–[openapi-mcp S3] P3 proof: auth-guarded spec fixture + spec_unavailable wording #1129 were re-read against current main; P1 remains genuinely proof-arbitrated.
2. Aspire 13.4 documents and generates a TypeScript onResourceEndpointsAllocated callback plus EndpointReference.getValueAsync(); this is promising research evidence, not yet a P1 verdict.
3. Existing service tests expose the exact P3 401/403/200 fixture shape, while P2 still requires a live real-scaffold measurement.
4. Two foreign AppHosts and six foreign containers were present before this run; they are recorded as immutable external state.

Next

• Lock the measurement plan and Design checkpoint.
• Run separate-session PLAN-EVAL; no experiment implementation starts before PASS.

[rickylabs]

[PHASE: PLAN]

The Wave-0 proof design is locked at commit 1fc6e3935.

Decisions

• Five ordered slices: S0 plan gate, S1 P1/F1 arbitration, S2 DB + no-DB P2 measurement, S3 P3 auth fixture/wording, S4 final hygiene/evaluation.
• D1–D12 fix the artifact paths, serialized two-scaffold method, P1 PASS/FAIL bar, P2 measurement schema, P3 wording, resource ownership, and no-product-change boundary.
• NOT_RUN, incomplete, missing, or skipped proof evidence maps to FAIL.
• No packages/**, plugins/**, template, lock, or CLI E2E scope.

Gates

• Owned run Markdown format: PASS.
• Experiment/runtime/static gates: correctly NOT_RUN before Plan-Gate approval.
• Shared-host foreign AppHosts/containers are inventoried and immutable.

Next

• Separate OPEN-model Qwen PLAN-EVAL; implementation remains blocked until its committed verdict is PASS.

[rickylabs]

[PHASE: PLAN-EVAL] [VERDICT: APPROVED]

Separate Qwen 3.7 Max/high evaluation passed every Plan-Gate item.

Findings

1. Research current — PASS. The evaluator rechecked MCP truncation constants, the auth fixture, generated AppHost sequencing, and RFC §4/§9 against current source.
2. Decisions and sweep — PASS. D1–D12 leave no rework-causing design decision open; P1/P2/P3 values are empirical outputs with locked methods and bars.
3. Slices and gates — PASS. Five slices are ordered, bounded, file-scoped, and paired with static/runtime/service/resource gates.
4. Scope/debt — PASS. No published surface or product template is owned; any required product change is FAIL_RESCOPE to [openapi-mcp S7] Endpoint-manifest emission from the P1-proven seam #1133/later waves.
5. Skip semantics — PASS. Missing, incomplete, timed-out, or skipped evidence cannot become PASS.

Evidence

• Commit: 1fc6e3935
• Verdict: .llm/runs/test-openapi-mcp-wave0-proofs--wave0/plan-eval.md
• The model guard denied one attempted closed default child before provider dispatch; the successful evaluator performed all checks in the top-level open-model session.

Next

• Supervisor launches the separate Codex implementation lane for serialized S1–S3; each slice stops for opposite-family review and supervisor sign-off.

[rickylabs]

[PHASE: IMPL]

Implementation has started in the separately tracked canonical Codex lane.

Lane

• Thread: 019fc95d-ce57-7893-98b3-36977bec0cf1
• Route: OpenAI · gpt-5.6-sol · medium (requested = observed)
• Worktree: dedicated no-upstream test/openapi-mcp-wave0-proofs-impl at approved commit 641ae63ba
• Authority: draft proof/evidence/verdict artifacts only; no commit, push, GitHub, package, plugin, or template changes

Safety

• The extra worktree preserves one-sender-per-worktree because the PR worktree belongs to this supervisor session.
• S1–S3 are serialized; one AppHost maximum; every slice stops for opposite-family review and supervisor sign-off.
• Commit 7f62ff71d records the credential-free thread identity and orchestration drift.

Next

• S1 P1 experiment and evidence-backed F1(a)/(b) verdict draft.

[rickylabs]

IMPL — S1 P1 lifecycle proof

Commit: a9a28c4d1

• Verdict: FAIL under locked D5/D6; qualified F1(b) selected.
• Positive evidence retained: the post-allocation callback plus getValueAsync() atomically wrote a complete identity-bound manifest with the allocated http://localhost:3001 endpoint.
• Failure bar: the owned generated SQLite service exited because its command omitted --allow-ffi; the later HTTP 200 had no captured listener owner or precise ordering, so it is ambiguous and not pass evidence.
• Decision record: RFC RFC: OpenAPI→MCP — making a service's own API legible to the agent building it (#1117) #1123 §9 and epic Epic: OpenAPI→MCP service introspection — agent-legible service APIs #1126 now record F1(b), the causal qualifier, and legitimate future revisitability.
• Review: first native Fable review requested M1–M3/m1–m3 amendments; separate native Fable re-review APPROVED all dispositions.
• Gates: scoped TypeScript check/lint/fmt PASS; no lint ignores; root deno.lock unchanged; exact owned teardown/resource hygiene PASS; foreign resources untouched.

Only DB-backed P2 is affected by the permission defect. No-DB P2 and P3 remain independently runnable.

[rickylabs]

IMPL — S2 P2 fidelity/size proof

Commit: 5b0ba26b5

• Combined verdict: FAIL. D7 requires DB and no-DB live measurements; the DB service remains unavailable because the generated command omits --allow-ffi. The unattributed P1 HTTP 200 was not reused.
• Valid no-DB measurement: attributed healthy OpenAPI 3.1.1 response, 3,657 compact bytes, three dotted operationIds, discovery rows 73/89/88 bytes, no refs, no declared non-2xx responses/common envelope, max array 5/50, max string 34/2000, and no whole-result byte ceiling.
• Auditability: the exact raw spec is retained; its SHA-256 matches runtime evidence; the keyword/non-allowlisted-key union covers every object key.
• Review: first native Fable review found an omitted summary keyword; offline amendment fixed and audited the inventory; native Fable re-review APPROVED.
• Gates: scoped check/lint/fmt PASS for both proof scripts; no lint ignores; exact owned teardown/resource hygiene PASS; no product/template/lock change.

#1128 remains open and has no closing keyword because the required DB-backed measurement is absent.

[rickylabs]

S3 implementation summary — P3 auth fixture

Disposition: PASS, committed as 5c041bdfd.

Evidence

• Executed the focused existing auth fixture against one live preset-auth service lifecycle.
• Observed and exactly asserted 401 {error: UNAUTHORIZED, message: missing-credential} and 403 {error: FORBIDDEN, message: authz.missing-scope:docs:read}.
• Authorized request reached /api/openapi.json with status 200; no unasserted success-body shape is claimed.
• Ratified spec_unavailable text is byte-identical to locked D9.
• Verdict: .llm/runs/test-openapi-mcp-wave0-proofs--wave0/proofs/P3-verdict.md.
• Normalized evidence: proofs/evidence/P3-auth-fixture.json.

Review and validation

• Separate native Fable review: APPROVED; reviewer independently reran the fixture and verified head/blob/version attribution.
• Supervisor rerun: exit 0, 1 passed, 0 failed, 1 filtered out.
• Nine touched S3/run artifacts pass the scoped formatting wrapper; git diff --check passes.
• No packages/**, plugins/**, seed-RFC, or deno.lock diff; no new lint-ignore directives.
• Full CLI E2E was intentionally not run per the orchestrator contract.

Issue semantics

• [openapi-mcp S3] P3 proof: auth-guarded spec fixture + spec_unavailable wording #1129 acceptance is satisfied and the PR now carries Closes #1129.
• [openapi-mcp S2] P2 proof: spec fidelity + size dry-run against a real scaffold #1128 remains open and has no closing keyword: P2's explicit FAIL is committed, but the required DB-backed measurement remains absent.
• Final leak/review-thread gates and separate Qwen IMPL-EVAL remain pending.

[rickylabs]

IMPL-EVAL summary

Formal verdict: PASS · committed at 24ebdc515.

• Evaluator: separate guarded OpenRouter qwen/qwen3.7-max / high session.
• Artifact: .llm/runs/test-openapi-mcp-wave0-proofs--wave0/evaluate.md.
• Independent checks: scoped proof TypeScript check/lint, P3 focused fixture rerun, raw P2 hash/size, P1/P2/P3 evidence consistency, PR taxonomy/milestone, close-gate semantics, scope/lock, leak, and review-thread evidence.
• Ruling: P1/P2 empirical FAIL verdicts are complete and truthful proof outputs; they do not turn the underlying failed runtime/consumer behavior into PASS. P3 is supported PASS.
• Close-gate: Closes #1127 and Closes #1129 are supported; [openapi-mcp S2] P2 proof: spec fidelity + size dry-run against a real scaffold #1128 correctly remains open without a closing keyword because its DB-backed measurement is absent.
• Final hygiene: zero run-owned survivors; foreign/unproven resources untouched; 0 review threads / 0 unanswered; deno task e2e:cli intentionally not run.

The evaluator recorded two non-blocking low findings: six historical advisory briefs lack ## SKILL, and nine historical brief/review Markdown files have formatting drift. These are documented in evaluate.md; the formal verdict remains PASS.

PR remains draft and moves to status:in-review for the orchestrator's readiness/merge decision.