feat(evaluation): evaluate the integration Skill guidance with skill-up

[AlexStocks]

Feature description

Adopt alibaba/skill-up (examined at v0.12.0, 2026-09-18) to add a versioned, reproducible regression suite for the agent-facing PowerContext integration Skill guidance.

The system under test is the Skill's guidance quality: intent routing (which PowerContext tool should or should not be selected for a request) and authorization boundaries (what the agent must not infer from a Skill or from an empty/failed result). This is the instruction layer that determines whether an agent uses PowerContext correctly at all.

This should complement, not replace, the existing integration-guidance qualification record. The current docs already retain multi-surface observations, raw JSONL evidence, and transcript-bound review. The existing CI also gates adapter request/response mapping for selected hosts. The missing signal is narrower and still important: deterministic, turn-scoped assertions for Skill text -> tool selection, tied to a pinned Skill revision.

Scope limit, stated up front: the first version should target the claude_code + MCP runner that skill-up actually provides. Results must not be presented as coverage of every real host integration, real host permission flow, bounded recall, automatic Capture/Flush, or memory quality.

Problem and proposed solution

Problem

docs/en/development/integration-guidance-evaluation.md (and its docs/zh counterpart) records the existing integration guidance matrix and later qualification batches. That evidence is useful, but it is not a focused regression suite for the Skill text itself:

• it is not a small CI gate that maintainers can run when changing SKILL.md;
• a report is not mechanically tied to the exact Skill revision under test;
• routing/authorization expectations are reviewed from transcripts rather than asserted as deterministic pass/fail checks;
• host-specific Skill variants can drift because the repository contains multiple packaged powercontext-project-context Skills with different tool names and host conventions.

The narrow need here is to catch regressions in the instruction layer: ordinary coding should not call PowerContext, explicit memory saves should call the right memory tool, empty search results should not grant write/inventory authority, and candidate inspection should not imply approval.

Proposed solution

Build a small, self-contained skill-up eval project that treats one packaged integration Skill as the system under test and grades tool-selection accuracy with deterministic rule_based assertions. Avoid an LLM judge for these checks; the expected signal is whether the right tool name appears, or does not appear, in the transcript.

Proposed location: evaluation/skill-up/, linked from evaluation/README.md. Maintainers may prefer another path; the shape matters more than the exact directory name.

evaluation/skill-up/
  README.md          # prerequisites, exact commands, limitations, and report interpretation rules
  evals/
    eval.yaml
    cases/01-*.yaml ...
    fixtures/mcp/
      powercontext.yaml  # auth headers / optional mocked responses live here
  sync-skill.sh      # copies or vendors the pinned Skill revision under test

Minimal eval.yaml shape for the first version:

schema_version: v1alpha1

environment:
  type: none            # only if README documents a pre-started PowerContext Server
  # Alternatively use docker + setup_steps when the suite should own the Server lifecycle.

mcp:
  servers:
    - name: powercontext
      mode: real
      transport: http
      endpoint: http://127.0.0.1:8000/mcp
      config_ref: fixtures/mcp/powercontext.yaml

skills:
  - source: local_path
    path: <pinned claude-code skill copy under test>
    include: [SKILL.md, "references/**"]

engine:
  name: claude_code
  model: {provider: anthropic, name: <model>}

cases:
  files: [evals/cases/*.yaml]
  defaults:
    timeout_seconds: 300
    max_turns: 8
  parallelism: 1

judge:
  type: rule_based

benchmark:
  enabled: true

report:
  formats: [json, junit, html]
  artifacts: [transcript]

The MCP authentication file should use skill-up's supported config_ref shape. Do not put headers directly under mcp.servers[]; skill-up v0.12.0's MCPServer schema does not define that field, and non-strict YAML decoding can silently ignore it.

# evals/fixtures/mcp/powercontext.yaml
headers:
  Authorization: ${POWERCONTEXT_CLAUDE_AUTHORIZATION}

Use the Claude Code convention here: POWERCONTEXT_CLAUDE_AUTHORIZATION is the complete Bearer <token> header value. Do not mix it with the Python SDK's bare-token convention unless the README documents the conversion exactly.

Why skill-up fits this narrow need:

Requirement	skill-up facility
Skill loaded vs not-loaded comparison	benchmark.enabled + run --baseline
Tool selection as the measured quantity	judge.type: rule_based with tool_called_in_turn / tool_not_called_in_turn
Multi-turn checks	case-level turns + conversation mode
Transcript evidence	report/transcript artifacts
Skill revision under test	local-path Skill copy plus a scripted sync/pin step
Machine-readable CI output	report.formats: [json, junit, html]

MCP is a first-class part of this runner: skill-up v0.12.0 builds Claude Code MCP configuration through claude mcp add --scope project --transport http <name> <endpoint> --header <k:v>. PowerContext's local MCP endpoint can fit that shape when headers are supplied through the referenced MCP fixture.

Case set

Cases should be derived from the routing and authorization rules already written in the Skill, not from generic coding tasks:

1. Ordinary coding request -> no PowerContext tool is called. Negative control for the Skill's "no PowerContext call is required" rule.
2. Explicit "save this" request -> the memory write tool is called. Positive control.
   • Cases 1 and 2 must ship together. Case 1 alone can pass if the model does nothing at all; case 2 alone can pass if the model calls tools indiscriminately.
3. A search that returns nothing -> the result must not be treated as authorization to write or inventory. Use a fabricated project/scope fixture or mocked response so the outcome is deterministic.
4. Inspecting candidates -> inspection grants no decision authority; approval/publication tools must not be called.
5. A write that fails -> the final response must not claim the memory was persisted. This can remain rule_based: use forbidden output patterns (output_matches.not / output_contains.not) plus a controlled failure response.

Implementation caveats

These caveats are source-verified against skill-up v0.12.0 and can otherwise produce wrong-but-green results. They belong in the README and generated report, not only in comments.

C1 - MCP headers must come from config_ref.

skill-up v0.12.0's MCPServer config includes name, mode, transport, command, args, endpoint, and config_ref; it does not include headers. Headers are loaded from the file referenced by config_ref. If headers are written inline under mcp.servers[], they can be silently ignored, leaving an authentication-enabled Server unusable while negative controls still pass vacuously.

C2 - Tool-name matching is exact string equality.

internal/judge/rule_based.go compares the recorded tool name to the expected rule name directly, and pkg/transcript/transcript.go stores the name as recorded. Do not write assertions with bare names until a real transcript confirms that those are the recorded names.

For Claude Code MCP, expected names may be namespaced, such as mcp__powercontext__remember_memory, while the Skill prose may use bare names such as remember_memory. A tool_not_called_in_turn assertion using the wrong bare name can pass while the namespaced tool was actually called. Also document the expected namespace spelling; the MCP server key powercontext is the likely source for mcp__powercontext__..., not the FastMCP display name.

First task after the first run:

grep -o '"name":"[^"]*"' result.json | sort -u

Assertions must use the exact recorded names.

C3 - Hooks are disabled, so automatic memory paths are unobservable.

skill-up's Claude Code runner constructs commands with --settings '{"disableAllHooks":true}' and --permission-mode=bypassPermissions. Separately, PowerContext intentionally does not project POST /v1/context/prepare as an MCP tool. Therefore this suite cannot observe bounded recall, automatic Capture, or Flush. This is an execution-model limit, not a configuration gap.

C4 - bypassPermissions diverges from real hosts.

Writes that a real host might gate behind user confirmation can succeed unconditionally in this runner. Reports should state that the suite evaluates instruction/tool-selection behavior under the runner's permission model, not host approval UX.

C5 - Real mode is stateful.

PowerContext persists writes, so real-mode cases can affect later cases and later runs. Keep parallelism: 1 until isolation is designed. Also isolate state across runs: use a dedicated fixture Scope per run/case, or document a cleanup/reset mechanism. If mocked MCP is used for deterministic failure cases, record which behavior is mocked and avoid presenting those cases as end-to-end persistence evidence.

C6 - The Server lifecycle must be owned or documented.

environment.type: none does not start PowerContext. If the suite uses real MCP mode, the README must document an explicit pre-step to start the Server and verify readiness, or the eval should use an environment that starts the Server through setup steps. The acceptance command must not imply that a clean checkout alone is enough when an external Server is required.

C7 - The packaged Skill is not fully self-contained.

references/scope-memory.md invokes python3 "${CLAUDE_PLUGIN_ROOT}/scripts/workspace_scope.py", but scripts/ lives at the plugin root, outside the Skill directory. skill-up installs the Skill directory, and CLAUDE_PLUGIN_ROOT is a plugin-runtime variable that may not exist in this context. This should be documented or handled explicitly; if the eval surfaces it, that is a legitimate packaging/runtime-boundary finding.

C8 - First-version host coverage is intentionally narrow.

The repository has multiple PowerContext Skill variants. The first suite should not claim to qualify Codex, Hermes, WorkBuddy, OpenCode, Pi, OpenClaw, or portable Agent Plugin behavior unless those hosts are actually run or their tool namespace is faithfully represented. Per-host Skill variants can be added later as separate suites or cases once the runner boundary is clear.

Acceptance criteria

• skill-up validate and skill-up run succeed on the eval project using exact commands documented in the README.
• The README documents whether the suite owns the PowerContext Server lifecycle or requires a pre-started Server, including the readiness check (C6).
• The Skill revision under test is pinned by commit SHA or content hash, and the sync/vendor step is scripted.
• MCP authentication is configured through config_ref; the referenced fixture uses POWERCONTEXT_CLAUDE_AUTHORIZATION or an explicitly documented equivalent (C1).
• The first report includes a real transcript-derived inventory of recorded tool names, and assertions use those exact names (C2).
• Positive and negative controls ship together, so "the model did nothing" cannot pass the suite.
• The suite runs with parallelism: 1, and documents the state-isolation mechanism across cases and runs (C5).
• benchmark.enabled is on, and reports show the Skill-loaded vs not-loaded delta rather than absolute numbers alone.
• report.formats includes junit in addition to human-readable artifacts, so CI can consume failures naturally.
• The README and generated report state that hooks are disabled, permissions are bypassed, and the suite does not measure bounded recall, automatic Capture/Flush, or memory quality (C3/C4).
• The README states the first-version host boundary: Claude Code + MCP Skill guidance only, unless additional hosts are actually implemented and verified (C8).
• evaluation/README.md and docs/en/development/integration-guidance-evaluation.md / docs/zh/development/integration-guidance-evaluation.md link to the new project and describe it as complementary evidence.

Scope limit

Memory quality is out of scope here and is measured elsewhere: benchmark/locomo/ covers engine-level conversation-memory behavior, while evaluation/ covers SWE-bench Pro OFF/ON task-outcome evaluation. skill-up results should not be aggregated with those results or used as a substitute for them.

This issue also does not require solving all host-specific integration differences in the first PR. It asks for a focused, deterministic regression suite for the Skill guidance layer, with clear report language about what the suite does and does not prove.

Alternatives considered

• Keep only the current integration-guidance record. It preserves useful evidence, but it is not a small Skill-revision regression gate.
• Use an LLM judge for tool selection. Unnecessary for the first version; rule_based assertions express the intended checks deterministically and without grading variance.
• Use only the existing evaluation/ harness. It is the right tool for task-outcome benchmarking, but it does not directly express "was the correct tool selected in this turn with this name/argument boundary", which is the behavior the Skill guidance determines.

Related

• Tracking Issue: expand benchmark coverage #1263 (tracking: expand benchmark coverage) - this can be a small runnable validation under that broader direction.
• feat(evaluation): benchmark PowerContext across agent harnesses and models #1705 (benchmark across agent harnesses and models) - adjacent but different: feat(evaluation): benchmark PowerContext across agent harnesses and models #1705 measures task outcomes per host/model; this measures Skill guidance quality.
• feat: evaluate Experience and Skill applicability with Jev #1647 (evaluate Experience and Skill applicability with Jev) - different layer: read-time candidate applicability, not instruction/tool-selection quality.
• feat: coding agent access capability alignment #1338 (coding agent access capability alignment) - related to capability boundaries and unavailable-tool behavior.
• Tracking Issue: make PowerContext work continuity reliable, explainable, and cross-host #1352 (work continuity roadmap).

Additional context

• skill-up version examined: v0.12.0 (2026-09-18).
• The important v0.12.0 runner constraints are: exact string matching for tool names, disabled Claude Code hooks, bypassed permissions, config_ref-based MCP headers, and mocked-only case-level MCP override behavior.
• This issue is an evaluation-infrastructure enhancement, not a bug report and not a merge blocker for unrelated work.

Prepared with AI assistance.

[AlexStocks]

Source-level verification against skill-up v0.12.0

I checked this proposal against the repository (master 969d9f2e) and against the pinned upstream revision (skill-up v0.12.0, read file by file through the API). Caveats C1, C2, and C5 all hold, but three parts of the proposed configuration are wrong in ways that fail silently.

Confirmed correct

Claim	Evidence
Tool-name matching is exact string equality (C1)	internal/judge/rule_based.go:593 msg.ToolCall.Name == rule.Name; :318 uses !=. No namespace normalization.
tool_called_in_turn / tool_not_called_in_turn exist and are turn-scoped	rule_based.go, resolved through lookupTurn(rule.Turn, ...)
Hooks disabled and permissions bypassed (C2)	internal/agent/claude_code.go:322 / 331 / 341 — all three command constructions carry --settings '{"disableAllHooks":true}' and -p --permission-mode=bypassPermissions
POST /v1/context/prepare is not projected as an MCP tool (C2)	prepare_context is absent from _MCP_OPERATION_IDS in src/powercontext/server/mcp.py
The packaged Skill is not self-contained (C5)	references/scope-memory.md:8,18 invoke python3 "${CLAUDE_PLUGIN_ROOT}/scripts/workspace_scope.py", and scripts/ sits at the plugin root
--header is emitted for the http transport	internal/agent/mcp.go:80-95

Two supporting notes:

• Case 5 ("the final response must not claim the memory was persisted") is expressible with judge.type: rule_based. output_matches supports forbidden regex patterns, and output_contains supports all/any/not modes. No LLM judge is needed.
• The C5 workaround already has precedent in-tree: integrations/workbuddy/README.md:78 copies workspace_scope.py out of the plugin root, since that host cannot reach it either.

Defect 1 (blocking): inline mcp.servers[].headers is silently dropped

The proposed eval.yaml writes headers: {Authorization: Bearer ${POWERCONTEXT_TOKEN}} inside an mcp.servers[] entry. That key does not exist at that level:

• internal/config/schema.go defines MCPServer as {Name, Mode, Transport, Command, Args, Endpoint, ConfigRef}. There is no Headers field.
• Headers are read only from the file that config_ref points to. internal/mcp/provisioner.go:217 defines mcpServerFile{ Headers map[string]string ... }, and the real-mode path in resolveServer calls resolveAuth(server.Name, fileCfg, lookupEnv) — i.e. exclusively from the referenced file.
• Decoding uses non-strict yaml.Unmarshal (internal/config/loader.go:74,119), so the unknown key is neither rejected nor warned about.

Against an authentication-enabled Server the consequence is that every MCP call fails, and case 1 (negative control: "no PowerContext tool is called") passes vacuously because no tool is usable. That is exactly the wrong-but-green failure mode C1 warns about, and it is not currently in the caveat list.

Correct shape — config_ref inline, headers in the referenced file:

mcp:
  servers:
    - name: powercontext
      mode: real
      transport: http
      endpoint: http://127.0.0.1:8000/mcp
      config_ref: fixtures/mcp/powercontext.yaml

# fixtures/mcp/powercontext.yaml  (resolved relative to SkillDir)
headers:
  Authorization: ${POWERCONTEXT_CLAUDE_AUTHORIZATION}

Header-value expansion semantics (provisioner.go:283-345): a whole-value ${NAME} is looked up and exported into the run environment; an embedded "Bearer ${NAME}" also works through resolveReferencedEnv; a value that starts with $ but is not a complete reference is rejected; an unset referenced variable fails loudly rather than silently. Note that fixtures/mcp/ is therefore a required part of the authentication path, not just a mocking facility — the directory note in the proposal should be reworded.

Defect 2: the credential variable name does not match the host convention

The proposal uses POWERCONTEXT_TOKEN. The Claude Code channel uses POWERCONTEXT_CLAUDE_AUTHORIZATION, and its value is the complete Bearer <token> header:

• integrations/claude-code/plugins/powercontext/.mcp.json → "Authorization": "${POWERCONTEXT_CLAUDE_AUTHORIZATION:-}"
• website/content/docs/en/docs/integrations/claude-code.md documents it as the full Bearer <token> header

POWERCONTEXT_TOKEN is the bare token used by the Python SDK / API quickstart path. Using the bare name inside a Bearer template yields Bearer Bearer ...; using the wrong name leaves it unset. The README should pick one convention and state it explicitly.

Defect 3: environment.type: none with mode: real leaves the Server lifecycle unowned

Environment.Type accepts none, opensandbox, and docker. With type: none, nothing in the suite starts the PowerContext Server, yet the first acceptance criterion requires skill-up validate and skill-up run to succeed on a clean checkout using only the documented commands. Either the README must document a pre-step (start the Server / docker compose up), or the config needs type: docker plus setup_steps.

Related and not covered by C4: parallelism: 1 serializes cases but does not isolate state across runs. The existing harness avoids this with a fixture scope; the proposal does not mention scope isolation at all.

Understated scope: the system under test is not a single artifact

The proposal names four host variants. The repository actually contains nine SKILL.md files with seven distinct contents:

md5	lines	hosts
e983c5c7f3c9d07e33042cc3f5691afc	34	claude-code, workbuddy, agent-plugin
a81e7ba085013961fe1c9b8b47221ca3	34	codex
52f15111e30f95ab33201e869e7bc8f7	34	hermes
0405000870ced0734edbe761ab7e4635	45	minimax
507f00093af3affb5344992eab549b65	23	openclaw
0a3d1cb68d5199cef54fcefcca003067	37	opencode
3dac1f3641429f05e824b2806d534f90	49	pi

Three consequences: (a) the claude_code text under test is byte-identical to the workbuddy and agent-plugin copies, so a claude-code-only suite cannot claim independence from those two hosts; (b) the drift cited as motivation is real and measurable, but six variants remain uncovered; (c) scripts/integration_guidance_skills.py already enumerates all nine hosts and diagnoses broken references, so drift-detection tooling already exists.

The novelty claim is weaker than it needs to be

Several rows of the "Why skill-up fits" table describe capabilities the repository already has:

• Skill loaded vs not-loaded: the existing record already carries a loaded/unloaded matrix (44 observations per host).
• Transcript evidence: scripts/evaluate_integration_guidance.py:384 already sha256-binds the transcript.
• Deterministic CI: master.yml already runs the per-host test_native_adapter_handoff_uses_real_request_mapping_and_response_envelopes[...] regressions.

The genuine new signal is narrower: deterministic, turn-scoped, rule-based assertions on tool selection, bound to a pinned Skill revision. Stated that way it is also a stronger case than the current wording, because the existing CI gates the adapter request/response mapping layer deterministically while model-driven routing is covered only by recorded measurement plus Codex review — and the record itself states that the review was "not a human reviewer or an automatic truthfulness classifier", including a case where review rejected an unfairly strengthened claim. The missing deterministic layer is precisely "Skill text → tool selection".

Two smaller points:

• ReportConfig.Formats accepts json, junit, and html. The proposal takes [json, html], leaving the natural machine-readable CI channel unused.
• The MCP namespace has two plausible spellings. Claude Code derives it from the .mcp.json key (powercontext → mcp__powercontext__remember_memory), while the FastMCP server self-declares MCP_SERVER_NAME = "PowerContext Server". An assertion written as mcp__PowerContext_Server__... would pass tool_not_called_in_turn vacuously. Naming both in the README would prevent that.

Finally, the Skill prose itself is currently consistent: search_memory, list_memory_entries, remember_memory, handoff_current_work, and list_artifact_candidates all match OpenAPI operationId values exactly. There is no stale-name defect today — the C1 hazard is purely the bare-name-versus-recorded-namespace gap.

Suggested edits to this issue

• Rewrite the eval.yaml snippet to use config_ref, and move headers into fixtures/mcp/<server>.yaml.
• Fix the credential variable name so it matches the host convention.
• Decide and document the Server lifecycle, and add the scope-isolation mechanism to the C4 caveat.
• Add Defect 1 to the caveats section; it belongs there more than C3 does, because it produces a green suite rather than a narrow one.
• Consider adding junit to report.formats.
• Restate the novelty claim narrowly, and consider promoting "the existing CI gates the adapter layer, not tool selection" from Related into the Problem section.

Prepared with AI assistance.

[Tech-Circle-48]

please assign to me