feat(evals): widen run diagnostics and route findings to owning surface

docs/ADDING_EVALS.md

@@ -124,6 +124,8 @@ The optional `expected` argument on `wroteFile` is useful when a file is exclude
 
 **A security judge that reads the command trace must know about the redaction marker.** The harness masks credential values as `[REDACTED SECRET]` before the trace reaches any model, so a judge asked "does an actual secret appear?" would answer no on a run that leaked one. Say in the prompt that the marker means a secret was on that command line, as the B2B org eval does.
 
+**Never add a step to a PROMPT.md to satisfy a grader.** `wroteFile` only sees write-tool calls, so an agent that creates a file with a `>` redirect fails it — and the fix is not a prompt that says "use your file-writing tool, not a shell redirect". Ask for the outcome and assert it route-agnostically: `matches('^// FILE: smoke-b2b-manifest\\.json$', …)` checks that some workspace file contains the manifest's self-naming header, whatever put it there (`matches` searches file contents, not paths, so anchor on a marker the artifact itself carries). Reserve `wroteFile` for cases where the write itself is the thing under test, or where you need its `expected` content check. This matters beyond one grader: every instruction in a PROMPT.md is guidance the agent no longer has to derive from the skill, so hand-holding hides the exact defect the eval exists to surface. State the goal, the exact names and identifiers, and the artifact you want back — nothing about how to get there.
+
 **Grade the effect, not one spelling of the command.** When an action can be done through a dedicated subcommand *or* a raw `auth0 api` call, match the shared endpoint/resource substring in `ranCommand` (e.g. `'invitations'`, `'client-grants'`, `'enabled_connections'`) instead of the full subcommand. A grader keyed to `auth0 orgs invitations create` failed a run that correctly used `auth0 api post "organizations/<id>/invitations"`. Where the two routes share no useful substring (`apis` vs `resource-servers`, `apps` vs `clients`, `orgs` vs `organizations`), list both in `ranCommandOneOf` and pin the resource with `args`, so the grader accepts either route while still insisting the command names the thing the task asked for:
 
 ```ts

docs/ARCHITECTURE.md

@@ -212,10 +212,8 @@ sequenceDiagram
         Grade->>Grade: LLM-judge for judge graders
         Grade-->>Score: GraderResult[]
         Score->>Score: 8 dimensions → overall + grade
-        opt skills or MCP active
-            Score->>Recs: ask judge LLM for fixes
-            Note over Recs: see "Recommendations":<br/>grader / skill / mcp / efficiency
-        end
+        Score->>Recs: ask judge LLM for fixes
+        Note over Recs: runs for every agent job, control run included<br/>see "Recommendations": grader / skill / mcp / efficiency
         Recs-->>CLI: scores-*.json (+ recommendations)
     end
 
@@ -330,16 +328,36 @@ The overall score is a **weighted sum** of 8 dimensions, split evenly between *h
 
 Scores diagnose; **recommendations prescribe** — the "every score must point to a fix" principle, in code.
 
-When a run had **skills or MCP enabled**, `generateRunRecommendations` hands the judge LLM the full run context (task, workspace output, injected skill content, grader results, scoring dimensions, efficiency breakdown) and gets back structured JSON: a `severity`-ranked list of fixes, each targeting one of four things to improve.
+`generateRunRecommendations` runs on **every agent job**, including the one with no tools at all. That run is the control: same task, same graders, same workspace, no skill and no MCP. If correct work fails a check there, the check is the suspect — so skipping the diagnosis on exactly those runs threw away the only evidence that separates a grader defect from a documentation defect. The skill is sent only when the skill was actually in the agent's context, and the prompt says so; handing the analyst documentation the agent never saw is how a control run acquires an invented "the skill should say X" finding. (True `--mode baseline` jobs have no workspace and no run record, so they are not analysed at all.)
+
+It hands the judge LLM the full run context (task, workspace output, the run trace, injected skill content, grader results, scoring dimensions, efficiency breakdown) and gets back structured JSON: a `severity`-ranked list of fixes, each naming the surface that has to change.
 
 | Category | What it flags | Example |
 |---|---|---|
 | `grader` | Missing checks, false pos/neg, over-strict criteria | "L4 grader misses the `audience` config key" |
 | `skill` | Skill doc gaps, confusing or outdated instructions | "SKILL.md omits the `cacheLocation` option" |
+| `eval` | The task itself: an ambiguous `PROMPT.md`, a prompt that contradicts a grader, bad provisioning | "The prompt says `role`, the grader wants a `rol_…` id" |
+| `cli` | The `auth0` CLI: a missing subcommand, a misleading flag, an unhelpful error | "`--send-email false` silently parses as a positional" |
+| `docs` | Auth0's published documentation | "The organizations page never says the setting is tenant-wide" |
 | `mcp` | Missing MCP tools, unhelpful responses, poor tool UX | "Add a `get_quickstart` tool returning the canonical snippet" |
 | `efficiency` | Thrashing that better docs/tools would prevent | "Agent retried the redirect-URI config 3× — document it" |
 
-Recommendations are scoped to **custom** skills/MCP tools (never the agent's built-in tools), then persisted alongside scores and surfaced in the leaderboard. The step is safe by construction: it never throws (returns `undefined` on failure) and strips `.env*` from the prompt.
+The list is deliberately wider than the skill. Offered only `skill`, `grader`, `mcp` and `efficiency`, the analyst files everything as a skill gap, including a CLI with no subcommand for the job and a task prompt two models read two different ways — real defects with different owners, folded into "document it harder" and sent to the wrong place. `cli`, `docs` and `mcp` are offered only when the run actually reached that surface, so the analysis cannot invent a complaint about a binary that never ran.
+
+Each finding also carries a diagnosis: `what_happened`, `what_should_have_happened`, an `evidence` quote, and a `root_cause` of `skill`, `model`, `grader`, `eval`, `cli`, or `environment`. `root_cause` is the field to read first. The skill sits in the agent's context for the whole run, so a failure the skill was in a position to prevent and did not is a defect in the documentation rather than in the model — which is what the analyst is asked to separate from an agent that ignored correct guidance, and from a grader that failed work which was actually right.
+
+Two inputs make that attribution possible, and both are easy to lose:
+
+- **The run trace.** Every shell command, MCP call, and failed tool call, in order, with the error text of anything that failed. Aggregate counts ("errors: 7") cannot identify a wrong command, and for a CLI eval the commands *are* the artifact. When the trace exceeds its budget, failures are kept in preference to successful calls.
+- **The reference pool.** `collectSkillFiles` walks `references/` recursively, because a reference is not always one file — the auth0 skill stores each as a directory (`references/feature-mfa/index.md`). Files the agent opened during the run are sent whole; the rest are listed by path even when their content is cut, so the analyst never reports a documented topic as missing.
+
+Recommendations are scoped to **custom** skills/MCP tools (never the agent's built-in tools), then persisted alongside scores and surfaced in the leaderboard. The step is safe by construction: it never throws, and it strips `.env*` from the prompt.
+
+Three properties of that step are worth stating, because each fixes a way the analysis used to mislead:
+
+- **Secrets are masked before anything leaves the machine.** Withholding `.env` is not enough for a CLI eval, where the credentials sit on the command line and in error bodies. `redactSecrets` (in `evals-core`) replaces credential *values* with `[REDACTED SECRET]` in the run trace, in MCP arguments, and in error text, and the same scrubber runs on the trace appended to an LLM judge. The value is replaced rather than the line dropped so a security grader still sees that a secret occupied that position. A judge prompt that checks for secret exposure must say **where** the marker counts as a violation, not treat every marker as one: the marker on the command that *creates* a resource is the harness masking a flag value on the way in, so a blanket "any marker fails" turns correct work into an automatic failure. Auth0 ids (`client_id`, `org_…`) stay readable, since a diagnosis that cannot name the resource is not a diagnosis.
+- **A failed analysis says so.** On a proxy error or an unparseable response the result comes back with an empty list *and* an `error` string, and the report renders the reason. An empty list with no explanation reads as "this run was clean", which is the opposite of what a 500 means.
+- **Findings stay attached to the run that produced them.** The report renders them inside each run's Recommendations panel, where the trace, graders, and metrics that produced them are one tab away, so a finding is read next to the evidence that produced it rather than as a free-floating claim.
 
 ## Sandbox — running untrusted agent code safely
 

packages/evals-core/src/recommendations/types.ts

@@ -2,10 +2,29 @@
  * Types for the post-scoring recommendations engine.
  */
 
-/** A single actionable recommendation produced by the analysis. */
+/**
+ * A single actionable recommendation produced by the analysis.
+ *
+ * `category` is deliberately wider than the skill. Offered only `skill`, `grader`,
+ * `mcp` and `efficiency`, an analyst files everything as a skill gap — including a
+ * CLI that has no subcommand for the job and a task prompt two models read two
+ * different ways. Those are real defects with different owners, and folding them
+ * into "document it harder" sends the fix to the wrong place.
+ */
 export interface Recommendation {
-  /** Which area this recommendation targets. */
-  category: 'grader' | 'skill' | 'mcp' | 'efficiency';
+  /**
+   * Which surface has to change.
+   *
+   * - `skill` — the Auth0 agent skill's own text.
+   * - `grader` — one check in the eval's `graders.ts`.
+   * - `eval` — the task definition: `PROMPT.md`, its scaffold, or its provisioning.
+   * - `cli` — the `auth0` CLI itself: a missing subcommand, a misleading flag, an
+   *   unhelpful error. Product feedback rather than something this repo can patch.
+   * - `docs` — Auth0's published documentation.
+   * - `mcp` — the Auth0 docs MCP server's tools or their output.
+   * - `efficiency` — turns wasted with no defect behind them.
+   */
+  category: 'grader' | 'skill' | 'eval' | 'cli' | 'docs' | 'mcp' | 'efficiency';
   /** Impact level of the issue. */
   severity: 'high' | 'medium' | 'low';
   /** Description of the problem observed. */
@@ -14,6 +33,24 @@ export interface Recommendation {
   suggestion: string;
   /** Optional context — grader name, skill name, tool name, file path, etc. */
   context?: string;
+  /**
+   * Where the fault lies.
+   *
+   * `skill` is the one worth acting on first: the skill was in the agent's context
+   * the whole run, so a failure it was in a position to prevent is a defect in the
+   * documentation, not in the model. `grader` means the agent was right and the
+   * check is wrong; `eval` means the task itself was ambiguous or contradictory, so
+   * neither the agent nor the skill could have got it right; `cli` means the tool
+   * surface was the obstacle. Optional — older stored results and efficiency notes
+   * omit it.
+   */
+  root_cause?: 'skill' | 'model' | 'grader' | 'eval' | 'cli' | 'environment';
+  /** What the agent actually did, with the command or code that did it. */
+  what_happened?: string;
+  /** The correct behaviour, concretely. */
+  what_should_have_happened?: string;
+  /** Verbatim quote from the run trace, workspace, or skill text backing the finding. */
+  evidence?: string;
 }
 
 /** Full recommendations output attached to an AgentJobResult. */
@@ -28,4 +65,12 @@ export interface Recommendations {
   recommendations: Recommendation[];
   /** 2-3 sentence executive summary of the analysis. */
   summary: string;
+  /**
+   * Why the analysis produced nothing, when it produced nothing.
+   *
+   * Present only on failure (proxy error, truncated or unparseable response). Without
+   * it an empty list reads as "the run was clean" in the report, which is the opposite
+   * of what a 500 means.
+   */
+  error?: string;
 }

packages/evals-reporter/src/templates/report.css

@@ -31,13 +31,16 @@
   --clr-red:    #ef4444;
   --clr-blue:   #60a5fa;
   --clr-orange: #f97316;
+  --clr-violet: #a78bfa;
 
   /* ── tinted backgrounds ── */
   --clr-green-bg:  #22c55e22;
+  --clr-lime-bg:   #84cc1622;
   --clr-amber-bg:  #f59e0b22;
   --clr-red-bg:    #ef444422;
   --clr-blue-bg:   #60a5fa22;
   --clr-orange-bg: #f9731622;
+  --clr-violet-bg: #a78bfa22;
   --clr-dim-bg:    #94a3b822;
 }
 
@@ -484,21 +487,53 @@ a { color: var(--link); }
 .tab-panel-empty { padding: 8px; color: var(--text-quat); font-size: 12px; font-style: italic; }
 
 /* ── Recommendations ────────────────────────────────────────────────────── */
-.rec-summary { padding: 8px 0; color: var(--text-sec); font-size: 13px; line-height: 1.5; margin-bottom: 8px; }
+.rec-tally { display: flex; align-items: center; gap: 6px; padding: 8px 0 0; }
+.rec-tally-total { font-size: 12px; font-weight: 600; color: var(--text-medium); margin-right: 2px; }
+.rec-chip { font-size: 11px; font-weight: 600; padding: 1px 7px; border-radius: 999px; }
+.rec-chip--high { background: var(--clr-red-bg); color: var(--clr-red); }
+.rec-chip--medium { background: var(--clr-amber-bg); color: var(--clr-amber); }
+.rec-chip--low { background: var(--clr-dim-bg); color: var(--text-sec); }
+.rec-summary { padding: 8px 0; color: var(--text-sec); font-size: 13px; line-height: 1.5; margin-bottom: 4px; }
 .rec-list { list-style: none; padding: 0; margin: 0; }
-.rec-item { padding: 10px 12px; border: 1px solid var(--border-2); border-radius: 6px; margin-bottom: 8px; }
-.rec-item-header { display: flex; align-items: center; gap: 8px; margin-bottom: 4px; }
-.rec-badge { font-size: 10px; font-weight: 600; text-transform: uppercase; padding: 2px 6px; border-radius: 3px; }
+.rec-item { padding: 10px 12px; border: 1px solid var(--border-2); border-left-width: 3px;
+            border-radius: 6px; margin-bottom: 8px; background: var(--surface-1); }
+.rec-item-header { display: flex; align-items: center; flex-wrap: wrap; gap: 6px; margin-bottom: 6px; }
+.rec-num { font-size: 11px; font-weight: 700; color: var(--text-quat); font-variant-numeric: tabular-nums; }
+.rec-badge { font-size: 10px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.03em;
+             padding: 2px 6px; border-radius: 3px; }
+.rec-badge--neutral { background: var(--clr-dim-bg); color: var(--text-sec); }
 .rec-badge--grader { background: var(--clr-blue-bg); color: var(--clr-blue); }
 .rec-badge--skill { background: var(--clr-green-bg); color: var(--clr-green); }
+.rec-badge--eval { background: var(--clr-violet-bg); color: var(--clr-violet); }
+.rec-badge--cli { background: var(--clr-lime-bg); color: var(--clr-lime); }
+.rec-badge--docs { background: var(--clr-dim-bg); color: var(--text-medium); }
 .rec-badge--mcp { background: var(--clr-amber-bg); color: var(--clr-amber); }
 .rec-badge--efficiency { background: var(--clr-orange-bg); color: var(--clr-orange); }
-.rec-severity-high { border-left: 3px solid var(--clr-red); }
-.rec-severity-medium { border-left: 3px solid var(--clr-amber); }
-.rec-severity-low { border-left: 3px solid var(--text-quat); }
-.rec-issue { font-size: 13px; color: var(--text-primary); margin-bottom: 4px; }
-.rec-suggestion { font-size: 12px; color: var(--text-sec); }
-.rec-context { font-size: 11px; color: var(--text-ter); margin-top: 4px; font-style: italic; }
+.rec-where { font-family: monospace; font-size: 11px; color: var(--text-ter); overflow-wrap: anywhere; }
+.rec-severity-high { border-left-color: var(--clr-red); }
+.rec-severity-medium { border-left-color: var(--clr-amber); }
+.rec-severity-low { border-left-color: var(--text-quat); }
+.rec-issue { font-size: 13px; line-height: 1.5; color: var(--text-primary); }
+/* `Did` / `Should` read as a pair, so they are laid out as one: a fixed label column
+   lines the two values up and lets the eye compare them without re-reading a prefix. */
+.rec-detail { display: grid; grid-template-columns: 52px 1fr; gap: 2px 10px; margin: 6px 0 0; }
+.rec-detail-key { font-size: 10px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.03em;
+                  color: var(--text-quat); padding-top: 2px; }
+.rec-detail-val { margin: 0; font-size: 12px; line-height: 1.5; color: var(--text-sec); }
+.rec-evidence { margin: 8px 0 0; padding: 6px 8px; background: var(--surface-2); border-radius: 4px;
+                font-size: 11px; line-height: 1.45; color: var(--text-medium);
+                white-space: pre-wrap; overflow-wrap: anywhere; overflow-x: auto; }
+.rec-suggestion { display: flex; gap: 8px; margin-top: 8px; padding-top: 8px;
+                  border-top: 1px solid var(--border); font-size: 12px; line-height: 1.5; color: var(--text-medium); }
+.rec-suggestion-key { flex: none; font-size: 10px; font-weight: 600; text-transform: uppercase;
+                      letter-spacing: 0.03em; color: var(--clr-green); padding-top: 2px; }
+
+/* Failed analysis — deliberately not styled like an empty state, since "nothing
+   came back" and "the run was clean" mean opposite things. */
+.rec-panel--failed { padding: 10px 12px; border: 1px solid var(--clr-amber); border-radius: 6px; background: var(--clr-amber-bg); }
+.rec-failed-title { font-size: 13px; font-weight: 600; color: var(--clr-amber); margin-bottom: 4px; }
+.rec-failed-reason { font-size: 12px; color: var(--text-primary); font-family: monospace; overflow-wrap: anywhere; }
+.rec-failed-note { font-size: 11px; color: var(--text-ter); margin-top: 6px; font-style: italic; }
 
 /* ── Metrics compact table (turn metrics / session trace) ────────────────── */
 .metrics-table       { width: 100%; border-collapse: collapse; font-size: 12px; }