feat(eval): log tool selection — query_logs (hosted) vs get_logs (self-hosted) - #100
Draft
jordienr wants to merge 1 commit into
Draft
feat(eval): log tool selection — query_logs (hosted) vs get_logs (self-hosted)#100jordienr wants to merge 1 commit into
jordienr wants to merge 1 commit into
Conversation
…hosted) Two MCP-mode evals asserting description-driven log tool selection: - hosted project -> query_logs - self-hosted project -> get_logs From supabase/mcp#333 review feedback (Rodriguespn). Blocked on query_logs shipping in a released @supabase/mcp-server-supabase version and bumping MCP_SERVER_VERSION in this repo; not runnable until then.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two MCP-mode evals that check the agent selects the right logs tool based on the tool descriptions:
investigate-logs-002-tool-selection-hosted— hosted project → expectsquery_logs(and notget_logs)investigate-logs-003-tool-selection-self-hosted— self-hosted project → expectsget_logs(and notquery_logs)Both are
interface: mcp,suite: regression, scored deterministically onctx.toolCalls[].endpoint.Why
Review feedback on supabase/mcp#333 (Rodriguespn): "curious if the agent will correctly call
query_logswhen working with the hosted version andget_logswhen working with self-hosted … we should have an eval scenario." That PR added thequery_logstool and description-level steering (hosted-only, withget_logsas the CLI/self-hosted fallback); these evals guard that steering.Draft because the tool under test doesn't exist in a released MCP server version:
query_logscurrently only exists in unmerged feat: add query_logs tool for custom log queries mcp#333. The pinnedMCP_SERVER_VERSIONhere is0.8.1, which has noquery_logs, so the hosted eval can't pass until:@supabase/mcp-server-supabaseversion shipsquery_logs, andMCP_SERVER_VERSIONis bumped to that version inpackages/core/src/index.ts./analytics/endpoints/logs) needed forquery_logsto succeed is already onmain. ✅The self-hosted context is conveyed via the prompt (MCP mode runs against hosted platform-lite), so it tests description-driven selection rather than a real self-hosted runtime — flagging in case reviewers prefer a local-stack variant instead.
Validated with
pnpm eval:dry(both scenarios load, parse, and plan asmode=tools runtime=platform-lite). Not executed — that needs the version bump above.