Skip to content

Add API-level tests for the MCP tool surface - #97

Merged
olavgg merged 1 commit into
mainfrom
test/mcp-tool-surface
Aug 19, 2026
Merged

Add API-level tests for the MCP tool surface#97
olavgg merged 1 commit into
mainfrom
test/mcp-tool-surface

Conversation

@JosteinGj

Copy link
Copy Markdown
Contributor

An LLM getting a tool call wrong is a different problem from the tool being wrong. These drive POST /mcp as a real MCP client would, and read every write back — cross-checking against the REST services on the same ApiService wherever they see the same row, since a tool that reports success while writing nothing is indistinguishable from a working one on the MCP side alone.

Rust rather than python_tests/ because the SDK already wires auth with refresh and TLS against the OS trust store (rustls-tls-native-roots). A Python client needed a hand-rolled token exchange and a CA-bundle probe, since a venv's certifi does not carry the dev IdP's issuer. The McpClient here is a private test helper; no new dependency, no public API, and the crate ships identically with or without the module.

Coverage is measured, not declared: try_call_tool records each (tool, field) pair it sends and the sweep diffs that against the live tools/list schema — 125 fields across 37 tools. A parameter added server-side fails the audit until something drives it. Hence one sequential sweep rather than 74 tests: cargo test parallelises with no ordering hook, so a shared registry could not be read reliably at the end of any individual test.

Also adds three polling range-query tests to the Python datapoint suite. They prove the point exists through a second read path first, so an empty range result cannot be excused as eventual consistency — which is what caught /timeseries/data/list answering 200 with a zero-byte body.

mcp_event_update_by_uuid_reindexes_the_external_id is red on purpose: renaming an event by UUID writes the new externalId but never reindexes it, so the event stays reachable only under the old one.

An LLM getting a tool call wrong is a different problem from the tool
being wrong. These drive POST /mcp as a real MCP client would, and read
every write back — cross-checking against the REST services on the same
ApiService wherever they see the same row, since a tool that reports
success while writing nothing is indistinguishable from a working one on
the MCP side alone.

Rust rather than python_tests/ because the SDK already wires auth with
refresh and TLS against the OS trust store (rustls-tls-native-roots).
A Python client needed a hand-rolled token exchange and a CA-bundle
probe, since a venv's certifi does not carry the dev IdP's issuer. The
McpClient here is a private test helper; no new dependency, no public
API, and the crate ships identically with or without the module.

Coverage is measured, not declared: try_call_tool records each
(tool, field) pair it sends and the sweep diffs that against the live
tools/list schema — 125 fields across 37 tools. A parameter added
server-side fails the audit until something drives it. Hence one
sequential sweep rather than 74 tests: cargo test parallelises with no
ordering hook, so a shared registry could not be read reliably at the
end of any individual test.

Also adds three polling range-query tests to the Python datapoint suite.
They prove the point exists through a second read path first, so an
empty range result cannot be excused as eventual consistency — which is
what caught /timeseries/data/list answering 200 with a zero-byte body.

mcp_event_update_by_uuid_reindexes_the_external_id is red on purpose:
renaming an event by UUID writes the new externalId but never reindexes
it, so the event stays reachable only under the old one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@olavgg
olavgg merged commit 85447ec into main Aug 19, 2026
18 checks passed
@olavgg
olavgg deleted the test/mcp-tool-surface branch August 19, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants