diff --git a/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md b/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md new file mode 100644 index 000000000..807cb1478 --- /dev/null +++ b/docs/ticket/063sw1z-enforce-resolved-tool-argument-constraints-before-every-disp.md @@ -0,0 +1,38 @@ +# Enforce resolved tool argument constraints before every dispatch + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +Tool-call validation checks missing and unknown names, but does not enforce +parameter types, enums, array item schemas, or nested value constraints. +It runs only for local tools; MCP and built-in tools bypass it. + +The recorded provider fixtures already contain invalid calls that pass through +the test harness, including Cerebras arrays with numeric items, Google values +outside the declared enum, and llama.cpp JSON-looking strings where an array was +intended. + +`conversation.tools..parameters` documents enums as allowed-value +constraints and supports forcing a value. +Those constraints must be enforced by JP rather than treated only as model +guidance. + +Open PR \#998 validates resolved schema definitions but does not validate +argument instances at dispatch time. + +Acceptance criteria: + +- Validate required fields, unknown fields, JSON types, complete-value enums, + `items`, and nested `properties` against the resolved tool schema. +- Apply the same validation before local, MCP, and built-in dispatch. +- Normalize strict-provider `null` placeholders back to omission or the + configured default before validation. + Optional nullable values must not leak into a tool whose source schema does + not accept `null`. +- Return the existing invalid-arguments tool response without invoking the + target. +- Add tests using invalid calls taken from the Cerebras, Google, and llama.cpp + fixtures. +- Add tests proving MCP and built-in parameter overrides are enforced. diff --git a/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md b/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md new file mode 100644 index 000000000..35adc6f2e --- /dev/null +++ b/docs/ticket/063sw3q-exercise-recorded-provider-tool-rounds-through-a-production.md @@ -0,0 +1,27 @@ +# Exercise recorded provider tool rounds through a production-shaped loop + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`TestRequest::tool_call_response` creates a fresh `ChatQuery` with no tools. +Production keeps the full tool list on every streaming cycle and resets a forced +choice to `Auto` after execution. + +The mismatch is visible in the fixtures: Ollama says no tool is defined after JP +sends a result, and some Google forced-tool follow-ups contain no assistant +message. +These tests do not cover repeated tool calls or continued tool availability. + +Acceptance criteria: + +- Add a recorded-provider test path that retains tool definitions across the + post-result request, matching `run_turn_loop`. +- Use a fake executor so the recorded response is produced by the same request, + execute, append-result, request cycle used in production. +- Cover a second tool call after the first result. +- Cover parallel calls where the provider supports them. +- Assert that forced choice becomes `Auto` while tools remain declared. +- Keep lower-level provider serialization tests where useful, but name them as + such. diff --git a/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md b/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md new file mode 100644 index 000000000..a7c1e7d27 --- /dev/null +++ b/docs/ticket/063swa4-make-provider-cassette-tests-assert-semantic-outcomes-and-co.md @@ -0,0 +1,30 @@ +# Make provider cassette tests assert semantic outcomes and consumed interactions + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +The provider VCR tests snapshot whatever happened but usually have no semantic +assertion. +`Vcr::cassette` also discards the `MockSet` returned by `playback_async`, so +recorded interactions are not checked for exactly one use. +Optional tool follow-ups can be silently skipped and leave stale cassette +entries without failing. + +Current green examples include empty llama.cpp model details, malformed Ollama +structured output, invalid tool arguments, and empty Google post-tool responses. + +Acceptance criteria: + +- Assert every cassette interaction is consumed exactly once during playback. +- Fail on unexpected requests and unused recorded responses. +- Require request-kind invariants: forced tool name, tool argument conformance, + structured schema conformance, nonempty or explicitly empty post-tool outcome, + requested model ID, and nonempty model lists. +- Remove no-op assertion defaults where the request kind has a meaningful + contract. +- Make skipped `ToolCallResponse` requests explicit in the test declaration + rather than controlled by a boolean that silently returns `None`. +- Add a harness regression test with an intentionally unused second cassette + interaction. diff --git a/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md b/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md new file mode 100644 index 000000000..4d3d1fcf5 --- /dev/null +++ b/docs/ticket/063swjq-wire-xct2cli-xml-fixtures-into-tests-and-anonymize-them.md @@ -0,0 +1,21 @@ +# Wire xct2cli XML fixtures into tests and anonymize them + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`crates/contrib/xct2cli/tests/fixtures/sample-toc.xml` and `time-sample.xml` +have no test consumer; the parser tests use inline XML. +The files also retain a device name, device UUID, username-bearing absolute +path, and process IDs while the README calls the exported fixtures safe to keep. + +Acceptance criteria: + +- Add tests that parse both checked-in fixture files through the production + entry points, or delete the files if the inline cases fully replace them. +- Assert the meaningful TOC and time-sample business outcomes exactly. +- Replace device names, UUIDs, user paths, process IDs, and addresses with + stable synthetic values unless a raw value is needed by the parser case. +- Clarify the README: exported XML omits the trace bundle's environment secrets + but still requires anonymization before commit. diff --git a/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md b/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md new file mode 100644 index 000000000..46b1c4319 --- /dev/null +++ b/docs/ticket/063swjr-snapshot-the-full-agentic-shepherd-issue-rendering-fixture.md @@ -0,0 +1,20 @@ +# Snapshot the full Agentic Shepherd issue rendering fixture + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`renders_full_issue_fixture` renders a complete user-visible Markdown document +but checks selected substrings. +Duplicated sections, broken ordering, leaked content, and formatting changes can +pass. + +Acceptance criteria: + +- Replace the substring checks with one exact static output assertion or an + accepted snapshot. +- Keep focused unit tests for individual rendering rules. +- Normalize only genuinely unstable fields before comparison. +- Verify the test fails when a section is duplicated, reordered, or rendered + with incorrect indentation. diff --git a/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md b/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md new file mode 100644 index 000000000..b42bc660c --- /dev/null +++ b/docs/ticket/063swpx-make-provider-fixtures-deterministic-and-sanitize-recorded-m.md @@ -0,0 +1,29 @@ +# Make provider fixtures deterministic and sanitize recorded machine data + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +Several fixtures contain avoidable nondeterminism or local machine data: + +- The common tool prompt asks the model to provide arbitrary arguments. +- `banana.jpg` depicts an apple and the test uses a substring assertion. +- Every generic OpenAI fixture uses the same conversation timestamp and + prompt-cache key. +- llama.cpp cassettes record `/Users/jean/...`. +- Other recorded fixtures contain host names, UUIDs, process IDs, and absolute + paths. + +Acceptance criteria: + +- Replace the arbitrary tool prompt with separate deterministic scalar, array, + default, and nested-value cases. +- Rename the image fixture to match its content and compare the normalized + answer exactly. +- Give each cassette a fixed distinct conversation identity while keeping all + turns in one cassette on the same identity. +- Extend fixture post-processing to normalize user paths, host names, device + UUIDs, process IDs, and timestamps when those values are not under test. +- Add a fixture lint test that rejects common absolute home-directory patterns + and known secret header/query fields. diff --git a/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md b/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md new file mode 100644 index 000000000..21a54e820 --- /dev/null +++ b/docs/ticket/063swpy-resolve-llama-cpp-model-details-when-request-aliases-differ.md @@ -0,0 +1,26 @@ +# Resolve llama.cpp model details when request aliases differ from catalog IDs + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +The llama.cpp fixture requests `llamacpp/qwen3.5:9b`, while `/v1/models` reports +`unsloth/Qwen3.5-9B-GGUF`. +`map_model` strips the vendor prefix to `Qwen3.5-9B-GGUF`, so +`model_details("qwen3.5:9b")` misses the loaded model and returns empty details. +The accepted snapshot loses the `/props` context window of 8192. + +The chat cassettes are also inconsistent with model discovery: chat responses +identify a 35B model while `/v1/models` identifies a 9B model. + +Acceptance criteria: + +- Define and implement model identity matching for llama.cpp request aliases and + loaded catalog IDs. +- Preserve the served context window when the configured request name differs + from the server model ID. +- Avoid ambiguous fallback when a server exposes more than one model. +- Re-record a self-consistent llama.cpp fixture corpus using one loaded model. +- Make the model-details fixture assert the expected ID and context window + instead of accepting empty details. diff --git a/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md b/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md new file mode 100644 index 000000000..e965d3c2a --- /dev/null +++ b/docs/ticket/063swpz-reject-malformed-and-schema-invalid-structured-responses.md @@ -0,0 +1,33 @@ +# Reject malformed and schema-invalid structured responses + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +The Ollama structured-output cassette returns plain text despite receiving the +title JSON schema. +`EventBuilder` wraps the parse failure in `Value::String` and emits +`ChatResponse::Structured`, so the background title task treats it as structured +data, extracts no title, and returns success without updating the title. + +Evidence: + +- `crates/jp_llm/tests/fixtures/ollama/test_structured_output.yml` +- `crates/jp_llm/tests/fixtures/ollama/test_structured_output.snap` +- `crates/jp_llm/src/event_builder.rs` +- `crates/jp_task/src/task/title_generator.rs` + +Acceptance criteria: + +- Malformed JSON cannot be represented as a successful + `ChatResponse::Structured`. +- Parsed JSON is checked against the schema attached to the request. +- A malformed or nonconforming response produces a typed failure that the retry + layer or caller can handle. +- Background title generation reports the failure rather than silently + succeeding without a title. +- Provider fixture tests assert schema conformance, including the existing + Ollama cassette. +- Cover valid JSON with the wrong shape, wrong field types, and wrong array + cardinality. diff --git a/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md b/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md new file mode 100644 index 000000000..79e7e8ec2 --- /dev/null +++ b/docs/ticket/063swq0-assert-openai-prompt-cache-behavior-automatically.md @@ -0,0 +1,23 @@ +# Assert OpenAI prompt-cache behavior automatically + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +`test_gpt_5_6_prompt_cache_read_after_write` asks the person re-recording +fixtures to inspect usage fields manually. +The cassette currently contains nonzero cache writes followed by nonzero cached +tokens, but the test cannot fail if either disappears because provider usage is +discarded. + +Acceptance criteria: + +- Capture the relevant cache usage from recorded Responses API events in the + test path. +- Assert nonzero cache writes on the first request and nonzero cached tokens on + the second. +- Assert the stable prompt-cache key and expected breakpoints on both requests. +- Keep usage out of the persisted conversation unless product behavior requires + it; a test-only observer is sufficient. +- Add a negative harness test proving zero cache activity fails the cache test. diff --git a/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md b/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md new file mode 100644 index 000000000..82b564e18 --- /dev/null +++ b/docs/ticket/063swq1-reject-malformed-streamed-tool-call-argument-json.md @@ -0,0 +1,21 @@ +# Reject malformed streamed tool-call argument JSON + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +`EventBuilder` converts malformed tool-call argument JSON into an empty map and +emits a normal `ToolCallRequest` (`crates/jp_llm/src/event_builder.rs`). +A truncated provider stream can therefore execute a no-argument tool or a tool +whose defaults fill the missing fields. + +Acceptance criteria: + +- A tool-call buffer containing malformed JSON must not become a valid empty + argument map. +- Surface a specific invalid-call response or stream error that can be recorded + and returned to the model. +- Prove that the tool executor is not invoked. +- Cover truncated JSON, a non-object top-level value, and valid `{}` separately. +- Add a test that fails against the current fallback behavior. diff --git a/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md b/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md new file mode 100644 index 000000000..a122473f2 --- /dev/null +++ b/docs/ticket/063swq2-preserve-ollama-provider-tool-call-ids.md @@ -0,0 +1,24 @@ +# Preserve Ollama provider tool-call IDs + +- **Status**: Todo +- **Kind**: Bug +- **Authors**: jp +- **Date**: 2026-08-21 + +Recorded Ollama responses contain provider-generated call IDs such as +`call_77bh2121`, but JP discards them and persists synthetic IDs such as +`run_me_2`. +The pinned `ollama-rs` response type exposes only the function payload, so the +ID is lost during deserialization. + +Acceptance criteria: + +- Preserve the provider ID when Ollama sends one. +- Use a deterministic synthetic ID only for responses from older servers that + omit the field. +- Keep parallel calls and repeated calls across turns distinct. +- Round-trip the same ID through `ToolCallRequest`, `ToolCallResponse`, and the + next Ollama request. +- Update the Ollama fixtures to assert the recorded provider IDs. +- If this requires an `ollama-rs` change, pin the fixed revision and add an + upstream regression test. diff --git a/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md b/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md new file mode 100644 index 000000000..56712fc02 --- /dev/null +++ b/docs/ticket/063swq3-replace-duplicated-full-provider-catalogs-with-focused-mappi.md @@ -0,0 +1,26 @@ +# Replace duplicated full provider catalogs with focused mapping fixtures + +- **Status**: Todo +- **Kind**: Chore +- **Authors**: jp +- **Date**: 2026-08-21 + +Provider model-list fixtures are too large to review and several model-details +cassettes duplicate the full catalog byte for byte. +OpenRouter has two identical 29,699-line cassettes plus a 10,111-line model +snapshot. + +Google, llama.cpp, and Ollama also duplicate their model-list cassette for +`models` and `model_details` tests. + +Acceptance criteria: + +- Keep one recorded catalog smoke cassette per provider where live format + coverage is useful. +- Test model mapping, sorting, deduplication, deprecation, capability inference, + and lookup with small curated static responses. +- Reuse one cassette response for `models` and `model_details` when both + production methods call the same endpoint. +- Replace full-catalog snapshots with exact assertions over selected + representative records and list invariants. +- Keep a test proving an unknown requested model returns the intended fallback.