Run the analysis-orchestrator evals with skillscope

[danielholanda]

Pull Request Template

Note to AMDers:
This is a public repository. Please do not upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.

Wires TraceLens/Agent/Analysis/skills/analysis-orchestrator/evals/evals.json up to danielholanda/skillscope.

What is here

.github/workflows/skill-evals.yml, three jobs, one per test type:

Job	What it grades	Cost
structural	the skill folder, both dataset files, every internal link the skill's markdown makes	free, no agent
routing	which skill fires for each of the 12 pooled prompts	one short agent run per case
behavior	what the agent did once it fired, for the evaluations that assert something beyond routing	one full agent run per case

Each job is a checkout plus a single danielholanda/skillscope@main step pointed at TraceLens/Agent/Analysis/skills/analysis-orchestrator. routing and behavior both wait on structural, so a malformed dataset fails before anything spends a token.

Routing therefore runs solo: analysis-orchestrator is the only skill in the room, graded against its own two near misses plus the five negatives skillscope ships rather than against a neighbour. It is still named explicitly in --routing-skills, because routing has no default — who a skill competes against is what its score means. The behavior job needs no such flag: it grades every skill it can see, which is this one.

Adding a second skill here means widening the skills: lines and deciding whether the new skill belongs in the routing room. Both are diffs a reviewer can see, which is the point.

The one choice worth reviewing

This calls the action, not skillscope's reusable workflow. danielholanda/skillscope/.github/workflows/skill-evals.yml would be a shorter caller, but its steps reference amd/skillscope@bootstrap internally and that repository does not resolve yet, so a caller would fail at job start. Calling danielholanda/skillscope@main directly avoids that and is also more legible as a reference: the three commands are visible in the file instead of behind twenty with: inputs.

The trade-off to know about: this workflow does not run skillscope select, so it does not read evals/machine.yml. Everything runs on ubuntu-latest, which satisfies the os: [Linux] the skill asks for today. A skill that later asks for hardware labels would need either the reusable workflow or a matrix here.

The one change outside the workflow

SKILL.md declared name: tracelens-analysis-orchestrator inside a folder named analysis-orchestrator. The Agent Skills format requires the two to agree, and skillscope structural fails on it — so nothing else could run until it was fixed. This does not change the published catalog entry: amd/skills federation rewrites the frontmatter name to the as: value from .github/federation.json, which is still tracelens-analysis-orchestrator.

No change was needed to evals.json itself. Its {trace_path} / {platform} / {output_dir} placeholders are already filled by evals/hooks.py in behavior mode, and the file validates against skillscope's schema as written.

Verification

Ran against this branch with the harness installed from the ref the workflow uses:

$ uvx --from git+https://github.com/danielholanda/skillscope@main skillscope structural \
    --skills TraceLens/Agent/Analysis/skills/analysis-orchestrator
[evals] OK: 1 skill folder(s) in shape.
[evals] OK: 12 case(s) across 1 skill(s) plus 5 shared negative(s).
[evals] OK: 56 internal reference(s) across 17 markdown file(s).

Also confirmed the composite action's launcher resolves to the same command, that the routing and behavior invocations resolve against this repo's layout, and that the workflow is clean under actionlint.

The routing and behavior jobs could not be run here: they need an authenticated claude CLI, and ANTHROPIC_API_KEY has to exist as a repository secret before this workflow's first run.