From 2db7d6a72e82e85cb1780ed997af2e9878d9a58c Mon Sep 17 00:00:00 2001 From: Jean Mertz Date: Fri, 21 Aug 2026 17:05:27 +0200 Subject: [PATCH 1/2] chore(ticket): Track fixture review findings Record the bugs and follow-up work identified while reviewing the existing test fixtures. Signed-off-by: Jean Mertz --- ...-argument-constraints-before-every-disp.md | 23 ++++++++++++++++++ ...ovider-tool-rounds-through-a-production.md | 19 +++++++++++++++ ...e-tests-assert-semantic-outcomes-and-co.md | 19 +++++++++++++++ ...-fixtures-into-tests-and-anonymize-them.md | 15 ++++++++++++ ...gentic-shepherd-issue-rendering-fixture.md | 15 ++++++++++++ ...s-deterministic-and-sanitize-recorded-m.md | 22 +++++++++++++++++ ...del-details-when-request-aliases-differ.md | 18 ++++++++++++++ ...and-schema-invalid-structured-responses.md | 24 +++++++++++++++++++ ...nai-prompt-cache-behavior-automatically.md | 16 +++++++++++++ ...formed-streamed-tool-call-argument-json.md | 16 +++++++++++++ ...-preserve-ollama-provider-tool-call-ids.md | 17 +++++++++++++ ...ll-provider-catalogs-with-focused-mappi.md | 18 ++++++++++++++ 12 files changed, 222 insertions(+) create mode 100644 docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md create mode 100644 docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md create mode 100644 docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md create mode 100644 docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md create mode 100644 docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md create mode 100644 docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md create mode 100644 docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md create mode 100644 docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md create mode 100644 docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md create mode 100644 docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md create mode 100644 docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md create mode 100644 docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md diff --git a/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md b/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md new file mode 100644 index 000000000..fdb4b7b80 --- /dev/null +++ b/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md @@ -0,0 +1,23 @@ +# Enforce resolved tool argument constraints before every dispatch + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +Tool-call validation checks missing and unknown names, but does not enforce parameter types, enums, array item schemas, or nested value constraints. It runs only for local tools; MCP and built-in tools bypass it. + +The recorded provider fixtures already contain invalid calls that pass through the test harness, including Cerebras arrays with numeric items, Google values outside the declared enum, and llama.cpp JSON-looking strings where an array was intended. + +`conversation.tools..parameters` documents enums as allowed-value constraints and supports forcing a value. Those constraints must be enforced by JP rather than treated only as model guidance. + +Open PR #998 validates resolved schema definitions but does not validate argument instances at dispatch time. + +Acceptance criteria: + +- Validate required fields, unknown fields, JSON types, complete-value enums, `items`, and nested `properties` against the resolved tool schema. +- Apply the same validation before local, MCP, and built-in dispatch. +- Normalize strict-provider `null` placeholders back to omission or the configured default before validation. Optional nullable values must not leak into a tool whose source schema does not accept `null`. +- Return the existing invalid-arguments tool response without invoking the target. +- Add tests using invalid calls taken from the Cerebras, Google, and llama.cpp fixtures. +- Add tests proving MCP and built-in parameter overrides are enforced. diff --git a/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md b/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md new file mode 100644 index 000000000..f04ef3c66 --- /dev/null +++ b/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md @@ -0,0 +1,19 @@ +# Exercise recorded provider tool rounds through a production-shaped loop + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`TestRequest::tool_call_response` creates a fresh `ChatQuery` with no tools. Production keeps the full tool list on every streaming cycle and resets a forced choice to `Auto` after execution. + +The mismatch is visible in the fixtures: Ollama says no tool is defined after JP sends a result, and some Google forced-tool follow-ups contain no assistant message. These tests do not cover repeated tool calls or continued tool availability. + +Acceptance criteria: + +- Add a recorded-provider test path that retains tool definitions across the post-result request, matching `run_turn_loop`. +- Use a fake executor so the recorded response is produced by the same request, execute, append-result, request cycle used in production. +- Cover a second tool call after the first result. +- Cover parallel calls where the provider supports them. +- Assert that forced choice becomes `Auto` while tools remain declared. +- Keep lower-level provider serialization tests where useful, but name them as such. diff --git a/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md b/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md new file mode 100644 index 000000000..67794b608 --- /dev/null +++ b/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md @@ -0,0 +1,19 @@ +# Make provider cassette tests assert semantic outcomes and consumed interactions + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +The provider VCR tests snapshot whatever happened but usually have no semantic assertion. `Vcr::cassette` also discards the `MockSet` returned by `playback_async`, so recorded interactions are not checked for exactly one use. Optional tool follow-ups can be silently skipped and leave stale cassette entries without failing. + +Current green examples include empty llama.cpp model details, malformed Ollama structured output, invalid tool arguments, and empty Google post-tool responses. + +Acceptance criteria: + +- Assert every cassette interaction is consumed exactly once during playback. +- Fail on unexpected requests and unused recorded responses. +- Require request-kind invariants: forced tool name, tool argument conformance, structured schema conformance, nonempty or explicitly empty post-tool outcome, requested model ID, and nonempty model lists. +- Remove no-op assertion defaults where the request kind has a meaningful contract. +- Make skipped `ToolCallResponse` requests explicit in the test declaration rather than controlled by a boolean that silently returns `None`. +- Add a harness regression test with an intentionally unused second cassette interaction. diff --git a/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md b/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md new file mode 100644 index 000000000..f99ebfa54 --- /dev/null +++ b/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md @@ -0,0 +1,15 @@ +# Wire xct2cli XML fixtures into tests and anonymize them + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`crates/contrib/xct2cli/tests/fixtures/sample-toc.xml` and `time-sample.xml` have no test consumer; the parser tests use inline XML. The files also retain a device name, device UUID, username-bearing absolute path, and process IDs while the README calls the exported fixtures safe to keep. + +Acceptance criteria: + +- Add tests that parse both checked-in fixture files through the production entry points, or delete the files if the inline cases fully replace them. +- Assert the meaningful TOC and time-sample business outcomes exactly. +- Replace device names, UUIDs, user paths, process IDs, and addresses with stable synthetic values unless a raw value is needed by the parser case. +- Clarify the README: exported XML omits the trace bundle's environment secrets but still requires anonymization before commit. diff --git a/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md b/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md new file mode 100644 index 000000000..e7f01726d --- /dev/null +++ b/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md @@ -0,0 +1,15 @@ +# Snapshot the full Agentic Shepherd issue rendering fixture + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`renders_full_issue_fixture` renders a complete user-visible Markdown document but checks selected substrings. Duplicated sections, broken ordering, leaked content, and formatting changes can pass. + +Acceptance criteria: + +- Replace the substring checks with one exact static output assertion or an accepted snapshot. +- Keep focused unit tests for individual rendering rules. +- Normalize only genuinely unstable fields before comparison. +- Verify the test fails when a section is duplicated, reordered, or rendered with incorrect indentation. diff --git a/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md b/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md new file mode 100644 index 000000000..684b63a03 --- /dev/null +++ b/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md @@ -0,0 +1,22 @@ +# Make provider fixtures deterministic and sanitize recorded machine data + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +Several fixtures contain avoidable nondeterminism or local machine data: + +- The common tool prompt asks the model to provide arbitrary arguments. +- `banana.jpg` depicts an apple and the test uses a substring assertion. +- Every generic OpenAI fixture uses the same conversation timestamp and prompt-cache key. +- llama.cpp cassettes record `/Users/jean/...`. +- Other recorded fixtures contain host names, UUIDs, process IDs, and absolute paths. + +Acceptance criteria: + +- Replace the arbitrary tool prompt with separate deterministic scalar, array, default, and nested-value cases. +- Rename the image fixture to match its content and compare the normalized answer exactly. +- Give each cassette a fixed distinct conversation identity while keeping all turns in one cassette on the same identity. +- Extend fixture post-processing to normalize user paths, host names, device UUIDs, process IDs, and timestamps when those values are not under test. +- Add a fixture lint test that rejects common absolute home-directory patterns and known secret header/query fields. diff --git a/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md b/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md new file mode 100644 index 000000000..563735518 --- /dev/null +++ b/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md @@ -0,0 +1,18 @@ +# Resolve llama.cpp model details when request aliases differ from catalog IDs + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +The llama.cpp fixture requests `llamacpp/qwen3.5:9b`, while `/v1/models` reports `unsloth/Qwen3.5-9B-GGUF`. `map_model` strips the vendor prefix to `Qwen3.5-9B-GGUF`, so `model_details("qwen3.5:9b")` misses the loaded model and returns empty details. The accepted snapshot loses the `/props` context window of 8192. + +The chat cassettes are also inconsistent with model discovery: chat responses identify a 35B model while `/v1/models` identifies a 9B model. + +Acceptance criteria: + +- Define and implement model identity matching for llama.cpp request aliases and loaded catalog IDs. +- Preserve the served context window when the configured request name differs from the server model ID. +- Avoid ambiguous fallback when a server exposes more than one model. +- Re-record a self-consistent llama.cpp fixture corpus using one loaded model. +- Make the model-details fixture assert the expected ID and context window instead of accepting empty details. diff --git a/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md b/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md new file mode 100644 index 000000000..87801d0ae --- /dev/null +++ b/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md @@ -0,0 +1,24 @@ +# Reject malformed and schema-invalid structured responses + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +The Ollama structured-output cassette returns plain text despite receiving the title JSON schema. `EventBuilder` wraps the parse failure in `Value::String` and emits `ChatResponse::Structured`, so the background title task treats it as structured data, extracts no title, and returns success without updating the title. + +Evidence: + +- `crates/jp_llm/tests/fixtures/ollama/test_structured_output.yml` +- `crates/jp_llm/tests/fixtures/ollama/test_structured_output.snap` +- `crates/jp_llm/src/event_builder.rs` +- `crates/jp_task/src/task/title_generator.rs` + +Acceptance criteria: + +- Malformed JSON cannot be represented as a successful `ChatResponse::Structured`. +- Parsed JSON is checked against the schema attached to the request. +- A malformed or nonconforming response produces a typed failure that the retry layer or caller can handle. +- Background title generation reports the failure rather than silently succeeding without a title. +- Provider fixture tests assert schema conformance, including the existing Ollama cassette. +- Cover valid JSON with the wrong shape, wrong field types, and wrong array cardinality. diff --git a/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md b/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md new file mode 100644 index 000000000..18cd81f53 --- /dev/null +++ b/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md @@ -0,0 +1,16 @@ +# Assert OpenAI prompt-cache behavior automatically + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`test_gpt_5_6_prompt_cache_read_after_write` asks the person re-recording fixtures to inspect usage fields manually. The cassette currently contains nonzero cache writes followed by nonzero cached tokens, but the test cannot fail if either disappears because provider usage is discarded. + +Acceptance criteria: + +- Capture the relevant cache usage from recorded Responses API events in the test path. +- Assert nonzero cache writes on the first request and nonzero cached tokens on the second. +- Assert the stable prompt-cache key and expected breakpoints on both requests. +- Keep usage out of the persisted conversation unless product behavior requires it; a test-only observer is sufficient. +- Add a negative harness test proving zero cache activity fails the cache test. diff --git a/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md b/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md new file mode 100644 index 000000000..efc542c03 --- /dev/null +++ b/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md @@ -0,0 +1,16 @@ +# Reject malformed streamed tool-call argument JSON + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +`EventBuilder` converts malformed tool-call argument JSON into an empty map and emits a normal `ToolCallRequest` (`crates/jp_llm/src/event_builder.rs`). A truncated provider stream can therefore execute a no-argument tool or a tool whose defaults fill the missing fields. + +Acceptance criteria: + +- A tool-call buffer containing malformed JSON must not become a valid empty argument map. +- Surface a specific invalid-call response or stream error that can be recorded and returned to the model. +- Prove that the tool executor is not invoked. +- Cover truncated JSON, a non-object top-level value, and valid `{}` separately. +- Add a test that fails against the current fallback behavior. diff --git a/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md b/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md new file mode 100644 index 000000000..b6f7c5ba0 --- /dev/null +++ b/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md @@ -0,0 +1,17 @@ +# Preserve Ollama provider tool-call IDs + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +Recorded Ollama responses contain provider-generated call IDs such as `call_77bh2121`, but JP discards them and persists synthetic IDs such as `run_me_2`. The pinned `ollama-rs` response type exposes only the function payload, so the ID is lost during deserialization. + +Acceptance criteria: + +- Preserve the provider ID when Ollama sends one. +- Use a deterministic synthetic ID only for responses from older servers that omit the field. +- Keep parallel calls and repeated calls across turns distinct. +- Round-trip the same ID through `ToolCallRequest`, `ToolCallResponse`, and the next Ollama request. +- Update the Ollama fixtures to assert the recorded provider IDs. +- If this requires an `ollama-rs` change, pin the fixed revision and add an upstream regression test. diff --git a/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md b/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md new file mode 100644 index 000000000..d5517e094 --- /dev/null +++ b/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md @@ -0,0 +1,18 @@ +# Replace duplicated full provider catalogs with focused mapping fixtures + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +Provider model-list fixtures are too large to review and several model-details cassettes duplicate the full catalog byte for byte. OpenRouter has two identical 29,699-line cassettes plus a 10,111-line model snapshot. + +Google, llama.cpp, and Ollama also duplicate their model-list cassette for `models` and `model_details` tests. + +Acceptance criteria: + +- Keep one recorded catalog smoke cassette per provider where live format coverage is useful. +- Test model mapping, sorting, deduplication, deprecation, capability inference, and lookup with small curated static responses. +- Reuse one cassette response for `models` and `model_details` when both production methods call the same endpoint. +- Replace full-catalog snapshots with exact assertions over selected representative records and list invariants. +- Keep a test proving an unknown requested model returns the intended fallback. From 0392b61d0b920ea8846bbfc712d98acdb166fa2f Mon Sep 17 00:00:00 2001 From: Jean Mertz Date: Fri, 21 Aug 2026 17:22:43 +0200 Subject: [PATCH 2/2] fixup! chore(ticket): Track fixture review findings Signed-off-by: Jean Mertz --- ...-argument-constraints-before-every-disp.md | 31 ++++++++++++++----- ...ovider-tool-rounds-through-a-production.md | 18 ++++++++--- ...e-tests-assert-semantic-outcomes-and-co.md | 23 ++++++++++---- ...-fixtures-into-tests-and-anonymize-them.md | 14 ++++++--- ...gentic-shepherd-issue-rendering-fixture.md | 11 +++++-- ...s-deterministic-and-sanitize-recorded-m.md | 21 ++++++++----- ...del-details-when-request-aliases-differ.md | 18 ++++++++--- ...and-schema-invalid-structured-responses.md | 21 +++++++++---- ...nai-prompt-cache-behavior-automatically.md | 15 ++++++--- ...formed-streamed-tool-call-argument-json.md | 11 +++++-- ...-preserve-ollama-provider-tool-call-ids.md | 15 ++++++--- ...ll-provider-catalogs-with-focused-mappi.md | 20 ++++++++---- 12 files changed, 157 insertions(+), 61 deletions(-) diff --git a/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md b/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md index fdb4b7b80..807cb1478 100644 --- a/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md +++ b/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md @@ -5,19 +5,34 @@ - **Authors**: jp - **Date**: 2026-08-21 -Tool-call validation checks missing and unknown names, but does not enforce parameter types, enums, array item schemas, or nested value constraints. It runs only for local tools; MCP and built-in tools bypass it. +Tool-call validation checks missing and unknown names, but does not enforce +parameter types, enums, array item schemas, or nested value constraints. +It runs only for local tools; MCP and built-in tools bypass it. -The recorded provider fixtures already contain invalid calls that pass through the test harness, including Cerebras arrays with numeric items, Google values outside the declared enum, and llama.cpp JSON-looking strings where an array was intended. +The recorded provider fixtures already contain invalid calls that pass through +the test harness, including Cerebras arrays with numeric items, Google values +outside the declared enum, and llama.cpp JSON-looking strings where an array was +intended. -`conversation.tools..parameters` documents enums as allowed-value constraints and supports forcing a value. Those constraints must be enforced by JP rather than treated only as model guidance. +`conversation.tools..parameters` documents enums as allowed-value +constraints and supports forcing a value. +Those constraints must be enforced by JP rather than treated only as model +guidance. -Open PR #998 validates resolved schema definitions but does not validate argument instances at dispatch time. +Open PR \#998 validates resolved schema definitions but does not validate +argument instances at dispatch time. Acceptance criteria: -- Validate required fields, unknown fields, JSON types, complete-value enums, `items`, and nested `properties` against the resolved tool schema. +- Validate required fields, unknown fields, JSON types, complete-value enums, + `items`, and nested `properties` against the resolved tool schema. - Apply the same validation before local, MCP, and built-in dispatch. -- Normalize strict-provider `null` placeholders back to omission or the configured default before validation. Optional nullable values must not leak into a tool whose source schema does not accept `null`. -- Return the existing invalid-arguments tool response without invoking the target. -- Add tests using invalid calls taken from the Cerebras, Google, and llama.cpp fixtures. +- Normalize strict-provider `null` placeholders back to omission or the + configured default before validation. + Optional nullable values must not leak into a tool whose source schema does + not accept `null`. +- Return the existing invalid-arguments tool response without invoking the + target. +- Add tests using invalid calls taken from the Cerebras, Google, and llama.cpp + fixtures. - Add tests proving MCP and built-in parameter overrides are enforced. diff --git a/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md b/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md index f04ef3c66..35adc6f2e 100644 --- a/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md +++ b/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md @@ -5,15 +5,23 @@ - **Authors**: jp - **Date**: 2026-08-21 -`TestRequest::tool_call_response` creates a fresh `ChatQuery` with no tools. Production keeps the full tool list on every streaming cycle and resets a forced choice to `Auto` after execution. +`TestRequest::tool_call_response` creates a fresh `ChatQuery` with no tools. +Production keeps the full tool list on every streaming cycle and resets a forced +choice to `Auto` after execution. -The mismatch is visible in the fixtures: Ollama says no tool is defined after JP sends a result, and some Google forced-tool follow-ups contain no assistant message. These tests do not cover repeated tool calls or continued tool availability. +The mismatch is visible in the fixtures: Ollama says no tool is defined after JP +sends a result, and some Google forced-tool follow-ups contain no assistant +message. +These tests do not cover repeated tool calls or continued tool availability. Acceptance criteria: -- Add a recorded-provider test path that retains tool definitions across the post-result request, matching `run_turn_loop`. -- Use a fake executor so the recorded response is produced by the same request, execute, append-result, request cycle used in production. +- Add a recorded-provider test path that retains tool definitions across the + post-result request, matching `run_turn_loop`. +- Use a fake executor so the recorded response is produced by the same request, + execute, append-result, request cycle used in production. - Cover a second tool call after the first result. - Cover parallel calls where the provider supports them. - Assert that forced choice becomes `Auto` while tools remain declared. -- Keep lower-level provider serialization tests where useful, but name them as such. +- Keep lower-level provider serialization tests where useful, but name them as + such. diff --git a/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md b/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md index 67794b608..a7c1e7d27 100644 --- a/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md +++ b/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md @@ -5,15 +5,26 @@ - **Authors**: jp - **Date**: 2026-08-21 -The provider VCR tests snapshot whatever happened but usually have no semantic assertion. `Vcr::cassette` also discards the `MockSet` returned by `playback_async`, so recorded interactions are not checked for exactly one use. Optional tool follow-ups can be silently skipped and leave stale cassette entries without failing. +The provider VCR tests snapshot whatever happened but usually have no semantic +assertion. +`Vcr::cassette` also discards the `MockSet` returned by `playback_async`, so +recorded interactions are not checked for exactly one use. +Optional tool follow-ups can be silently skipped and leave stale cassette +entries without failing. -Current green examples include empty llama.cpp model details, malformed Ollama structured output, invalid tool arguments, and empty Google post-tool responses. +Current green examples include empty llama.cpp model details, malformed Ollama +structured output, invalid tool arguments, and empty Google post-tool responses. Acceptance criteria: - Assert every cassette interaction is consumed exactly once during playback. - Fail on unexpected requests and unused recorded responses. -- Require request-kind invariants: forced tool name, tool argument conformance, structured schema conformance, nonempty or explicitly empty post-tool outcome, requested model ID, and nonempty model lists. -- Remove no-op assertion defaults where the request kind has a meaningful contract. -- Make skipped `ToolCallResponse` requests explicit in the test declaration rather than controlled by a boolean that silently returns `None`. -- Add a harness regression test with an intentionally unused second cassette interaction. +- Require request-kind invariants: forced tool name, tool argument conformance, + structured schema conformance, nonempty or explicitly empty post-tool outcome, + requested model ID, and nonempty model lists. +- Remove no-op assertion defaults where the request kind has a meaningful + contract. +- Make skipped `ToolCallResponse` requests explicit in the test declaration + rather than controlled by a boolean that silently returns `None`. +- Add a harness regression test with an intentionally unused second cassette + interaction. diff --git a/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md b/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md index f99ebfa54..4d3d1fcf5 100644 --- a/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md +++ b/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md @@ -5,11 +5,17 @@ - **Authors**: jp - **Date**: 2026-08-21 -`crates/contrib/xct2cli/tests/fixtures/sample-toc.xml` and `time-sample.xml` have no test consumer; the parser tests use inline XML. The files also retain a device name, device UUID, username-bearing absolute path, and process IDs while the README calls the exported fixtures safe to keep. +`crates/contrib/xct2cli/tests/fixtures/sample-toc.xml` and `time-sample.xml` +have no test consumer; the parser tests use inline XML. +The files also retain a device name, device UUID, username-bearing absolute +path, and process IDs while the README calls the exported fixtures safe to keep. Acceptance criteria: -- Add tests that parse both checked-in fixture files through the production entry points, or delete the files if the inline cases fully replace them. +- Add tests that parse both checked-in fixture files through the production + entry points, or delete the files if the inline cases fully replace them. - Assert the meaningful TOC and time-sample business outcomes exactly. -- Replace device names, UUIDs, user paths, process IDs, and addresses with stable synthetic values unless a raw value is needed by the parser case. -- Clarify the README: exported XML omits the trace bundle's environment secrets but still requires anonymization before commit. +- Replace device names, UUIDs, user paths, process IDs, and addresses with + stable synthetic values unless a raw value is needed by the parser case. +- Clarify the README: exported XML omits the trace bundle's environment secrets + but still requires anonymization before commit. diff --git a/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md b/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md index e7f01726d..46b1c4319 100644 --- a/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md +++ b/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md @@ -5,11 +5,16 @@ - **Authors**: jp - **Date**: 2026-08-21 -`renders_full_issue_fixture` renders a complete user-visible Markdown document but checks selected substrings. Duplicated sections, broken ordering, leaked content, and formatting changes can pass. +`renders_full_issue_fixture` renders a complete user-visible Markdown document +but checks selected substrings. +Duplicated sections, broken ordering, leaked content, and formatting changes can +pass. Acceptance criteria: -- Replace the substring checks with one exact static output assertion or an accepted snapshot. +- Replace the substring checks with one exact static output assertion or an + accepted snapshot. - Keep focused unit tests for individual rendering rules. - Normalize only genuinely unstable fields before comparison. -- Verify the test fails when a section is duplicated, reordered, or rendered with incorrect indentation. +- Verify the test fails when a section is duplicated, reordered, or rendered + with incorrect indentation. diff --git a/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md b/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md index 684b63a03..b42bc660c 100644 --- a/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md +++ b/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md @@ -9,14 +9,21 @@ Several fixtures contain avoidable nondeterminism or local machine data: - The common tool prompt asks the model to provide arbitrary arguments. - `banana.jpg` depicts an apple and the test uses a substring assertion. -- Every generic OpenAI fixture uses the same conversation timestamp and prompt-cache key. +- Every generic OpenAI fixture uses the same conversation timestamp and + prompt-cache key. - llama.cpp cassettes record `/Users/jean/...`. -- Other recorded fixtures contain host names, UUIDs, process IDs, and absolute paths. +- Other recorded fixtures contain host names, UUIDs, process IDs, and absolute + paths. Acceptance criteria: -- Replace the arbitrary tool prompt with separate deterministic scalar, array, default, and nested-value cases. -- Rename the image fixture to match its content and compare the normalized answer exactly. -- Give each cassette a fixed distinct conversation identity while keeping all turns in one cassette on the same identity. -- Extend fixture post-processing to normalize user paths, host names, device UUIDs, process IDs, and timestamps when those values are not under test. -- Add a fixture lint test that rejects common absolute home-directory patterns and known secret header/query fields. +- Replace the arbitrary tool prompt with separate deterministic scalar, array, + default, and nested-value cases. +- Rename the image fixture to match its content and compare the normalized + answer exactly. +- Give each cassette a fixed distinct conversation identity while keeping all + turns in one cassette on the same identity. +- Extend fixture post-processing to normalize user paths, host names, device + UUIDs, process IDs, and timestamps when those values are not under test. +- Add a fixture lint test that rejects common absolute home-directory patterns + and known secret header/query fields. diff --git a/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md b/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md index 563735518..21a54e820 100644 --- a/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md +++ b/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md @@ -5,14 +5,22 @@ - **Authors**: jp - **Date**: 2026-08-21 -The llama.cpp fixture requests `llamacpp/qwen3.5:9b`, while `/v1/models` reports `unsloth/Qwen3.5-9B-GGUF`. `map_model` strips the vendor prefix to `Qwen3.5-9B-GGUF`, so `model_details("qwen3.5:9b")` misses the loaded model and returns empty details. The accepted snapshot loses the `/props` context window of 8192. +The llama.cpp fixture requests `llamacpp/qwen3.5:9b`, while `/v1/models` reports +`unsloth/Qwen3.5-9B-GGUF`. +`map_model` strips the vendor prefix to `Qwen3.5-9B-GGUF`, so +`model_details("qwen3.5:9b")` misses the loaded model and returns empty details. +The accepted snapshot loses the `/props` context window of 8192. -The chat cassettes are also inconsistent with model discovery: chat responses identify a 35B model while `/v1/models` identifies a 9B model. +The chat cassettes are also inconsistent with model discovery: chat responses +identify a 35B model while `/v1/models` identifies a 9B model. Acceptance criteria: -- Define and implement model identity matching for llama.cpp request aliases and loaded catalog IDs. -- Preserve the served context window when the configured request name differs from the server model ID. +- Define and implement model identity matching for llama.cpp request aliases and + loaded catalog IDs. +- Preserve the served context window when the configured request name differs + from the server model ID. - Avoid ambiguous fallback when a server exposes more than one model. - Re-record a self-consistent llama.cpp fixture corpus using one loaded model. -- Make the model-details fixture assert the expected ID and context window instead of accepting empty details. +- Make the model-details fixture assert the expected ID and context window + instead of accepting empty details. diff --git a/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md b/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md index 87801d0ae..e965d3c2a 100644 --- a/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md +++ b/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md @@ -5,7 +5,11 @@ - **Authors**: jp - **Date**: 2026-08-21 -The Ollama structured-output cassette returns plain text despite receiving the title JSON schema. `EventBuilder` wraps the parse failure in `Value::String` and emits `ChatResponse::Structured`, so the background title task treats it as structured data, extracts no title, and returns success without updating the title. +The Ollama structured-output cassette returns plain text despite receiving the +title JSON schema. +`EventBuilder` wraps the parse failure in `Value::String` and emits +`ChatResponse::Structured`, so the background title task treats it as structured +data, extracts no title, and returns success without updating the title. Evidence: @@ -16,9 +20,14 @@ Evidence: Acceptance criteria: -- Malformed JSON cannot be represented as a successful `ChatResponse::Structured`. +- Malformed JSON cannot be represented as a successful + `ChatResponse::Structured`. - Parsed JSON is checked against the schema attached to the request. -- A malformed or nonconforming response produces a typed failure that the retry layer or caller can handle. -- Background title generation reports the failure rather than silently succeeding without a title. -- Provider fixture tests assert schema conformance, including the existing Ollama cassette. -- Cover valid JSON with the wrong shape, wrong field types, and wrong array cardinality. +- A malformed or nonconforming response produces a typed failure that the retry + layer or caller can handle. +- Background title generation reports the failure rather than silently + succeeding without a title. +- Provider fixture tests assert schema conformance, including the existing + Ollama cassette. +- Cover valid JSON with the wrong shape, wrong field types, and wrong array + cardinality. diff --git a/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md b/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md index 18cd81f53..79e7e8ec2 100644 --- a/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md +++ b/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md @@ -5,12 +5,19 @@ - **Authors**: jp - **Date**: 2026-08-21 -`test_gpt_5_6_prompt_cache_read_after_write` asks the person re-recording fixtures to inspect usage fields manually. The cassette currently contains nonzero cache writes followed by nonzero cached tokens, but the test cannot fail if either disappears because provider usage is discarded. +`test_gpt_5_6_prompt_cache_read_after_write` asks the person re-recording +fixtures to inspect usage fields manually. +The cassette currently contains nonzero cache writes followed by nonzero cached +tokens, but the test cannot fail if either disappears because provider usage is +discarded. Acceptance criteria: -- Capture the relevant cache usage from recorded Responses API events in the test path. -- Assert nonzero cache writes on the first request and nonzero cached tokens on the second. +- Capture the relevant cache usage from recorded Responses API events in the + test path. +- Assert nonzero cache writes on the first request and nonzero cached tokens on + the second. - Assert the stable prompt-cache key and expected breakpoints on both requests. -- Keep usage out of the persisted conversation unless product behavior requires it; a test-only observer is sufficient. +- Keep usage out of the persisted conversation unless product behavior requires + it; a test-only observer is sufficient. - Add a negative harness test proving zero cache activity fails the cache test. diff --git a/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md b/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md index efc542c03..82b564e18 100644 --- a/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md +++ b/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md @@ -5,12 +5,17 @@ - **Authors**: jp - **Date**: 2026-08-21 -`EventBuilder` converts malformed tool-call argument JSON into an empty map and emits a normal `ToolCallRequest` (`crates/jp_llm/src/event_builder.rs`). A truncated provider stream can therefore execute a no-argument tool or a tool whose defaults fill the missing fields. +`EventBuilder` converts malformed tool-call argument JSON into an empty map and +emits a normal `ToolCallRequest` (`crates/jp_llm/src/event_builder.rs`). +A truncated provider stream can therefore execute a no-argument tool or a tool +whose defaults fill the missing fields. Acceptance criteria: -- A tool-call buffer containing malformed JSON must not become a valid empty argument map. -- Surface a specific invalid-call response or stream error that can be recorded and returned to the model. +- A tool-call buffer containing malformed JSON must not become a valid empty + argument map. +- Surface a specific invalid-call response or stream error that can be recorded + and returned to the model. - Prove that the tool executor is not invoked. - Cover truncated JSON, a non-object top-level value, and valid `{}` separately. - Add a test that fails against the current fallback behavior. diff --git a/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md b/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md index b6f7c5ba0..a122473f2 100644 --- a/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md +++ b/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md @@ -5,13 +5,20 @@ - **Authors**: jp - **Date**: 2026-08-21 -Recorded Ollama responses contain provider-generated call IDs such as `call_77bh2121`, but JP discards them and persists synthetic IDs such as `run_me_2`. The pinned `ollama-rs` response type exposes only the function payload, so the ID is lost during deserialization. +Recorded Ollama responses contain provider-generated call IDs such as +`call_77bh2121`, but JP discards them and persists synthetic IDs such as +`run_me_2`. +The pinned `ollama-rs` response type exposes only the function payload, so the +ID is lost during deserialization. Acceptance criteria: - Preserve the provider ID when Ollama sends one. -- Use a deterministic synthetic ID only for responses from older servers that omit the field. +- Use a deterministic synthetic ID only for responses from older servers that + omit the field. - Keep parallel calls and repeated calls across turns distinct. -- Round-trip the same ID through `ToolCallRequest`, `ToolCallResponse`, and the next Ollama request. +- Round-trip the same ID through `ToolCallRequest`, `ToolCallResponse`, and the + next Ollama request. - Update the Ollama fixtures to assert the recorded provider IDs. -- If this requires an `ollama-rs` change, pin the fixed revision and add an upstream regression test. +- If this requires an `ollama-rs` change, pin the fixed revision and add an + upstream regression test. diff --git a/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md b/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md index d5517e094..56712fc02 100644 --- a/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md +++ b/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md @@ -5,14 +5,22 @@ - **Authors**: jp - **Date**: 2026-08-21 -Provider model-list fixtures are too large to review and several model-details cassettes duplicate the full catalog byte for byte. OpenRouter has two identical 29,699-line cassettes plus a 10,111-line model snapshot. +Provider model-list fixtures are too large to review and several model-details +cassettes duplicate the full catalog byte for byte. +OpenRouter has two identical 29,699-line cassettes plus a 10,111-line model +snapshot. -Google, llama.cpp, and Ollama also duplicate their model-list cassette for `models` and `model_details` tests. +Google, llama.cpp, and Ollama also duplicate their model-list cassette for +`models` and `model_details` tests. Acceptance criteria: -- Keep one recorded catalog smoke cassette per provider where live format coverage is useful. -- Test model mapping, sorting, deduplication, deprecation, capability inference, and lookup with small curated static responses. -- Reuse one cassette response for `models` and `model_details` when both production methods call the same endpoint. -- Replace full-catalog snapshots with exact assertions over selected representative records and list invariants. +- Keep one recorded catalog smoke cassette per provider where live format + coverage is useful. +- Test model mapping, sorting, deduplication, deprecation, capability inference, + and lookup with small curated static responses. +- Reuse one cassette response for `models` and `model_details` when both + production methods call the same endpoint. +- Replace full-catalog snapshots with exact assertions over selected + representative records and list invariants. - Keep a test proving an unknown requested model returns the intended fallback.