Conversation
The legacy engine shells out to the claude CLI with permissions bypassed on the runner, and hand-rolls workspace staging, LLM judging and routing termination around it. This adds a second engine that hands that work to inspect_ai, selected with `--engine inspect` and off by default until a benchmark shows what it changes. Its agent is harness-independent rather than the claude CLI, so a skill is graded on whether its instructions work rather than on one harness's reading of them. That also removes what blocked Windows: with no CLI in the sandbox there is no agent bridge, and the bridge is Linux-only. Both engines produce the same outcome objects, so `summarize`, `render_markdown` and the report writers are untouched. Three pieces are not thin wrappers. The tools are ours because inspect's assume a POSIX guest, which the Windows legs do not have. The judge sees artifacts rather than the agent's prose, attaches images so "did it draw a cat" is answerable, and reports whether a requirement is satisfied so a "must not" is never scored by negating a verdict. Routing is a single model call with no agent loop: the decision is a tool call, visible in the first turn, so there is no turn to pay for and kill. machine.yml gains a `sandbox` key naming a compose file, so a skill that must reach the network to pull a model can say so.
Neither engine reported its own cost. The legacy engine discards the `total_cost_usd` the CLI returns with every result, and inspect keeps usage in its own log where skillscope's report never looks. That was fine with one engine and nothing to compare it against. Both now record into `usage`, which the report meta carries, and `tools/benchmark_engines.py` runs the same dataset through both and reports per-case agreement plus what each run spent. Agreement is compared case by case rather than in aggregate: two runs can post identical accuracy while individual cases flip in both directions and cancel out. `--noise` runs the legacy engine twice so flips can be read against how much it already disagrees with itself. The spend columns are not equally trustworthy and the report says so rather than tabulating them as though they were. Legacy token counts are a floor -- assistant events omit the system prompt and cached input, and a routing case is killed before its totals arrive -- so cost is its reliable figure. inspect has a cost only when the provider supplies pricing, which a gateway does not, so wall time is often the only column comparable on both sides.
The inspect engine grades a skill with a harness-independent agent, which tests whether the skill's instructions work rather than how one product reads them. That choice raises an obvious question, so `--engine claude-code` answers it: run the real harness in the sandbox and see whether it agrees. It slots into the existing flag rather than adding a command, so the benchmark tool can diff its report against either other engine with no new machinery. Reporting only -- harness runs are nondeterministic and the harness is not what we grade, so a divergence is a question about the skill rather than a build failure. Linux only, because inspect_swe shells `bash -c` to find the CLI and the proxy it installs in the guest is a Linux binary. Also fixes how the inspect engines count model calls. They recorded one per sample while the legacy engine records one per assistant response: identical for routing, where a case is a single turn, but a large undercount for behavioral, where the agent loops. The benchmark puts the two columns side by side and names that one comparable, so it had to mean the same thing on both sides. Counted from assistant messages now.
The two legacy commands read the same `claude` stream but recorded from different events: routing counted assistant replies, behavioral counted only the result event, which is one per case. A behavioral run of three cases reported three model calls while costing $0.72 -- far more than three small calls -- and the benchmark puts that column beside the other engine's and calls it comparable. Both now go through one recorder in `usage`. The same three cases report fourteen calls against the new engine's twelve, which is the like-for- like reading: two agents looping to similar depth.
`--engine` changes what drives the agent while leaving the dataset, the CLI and the reports identical, so it belongs in usage rather than being discoverable only from --help. Says what each engine needs, why the inspect one is cheaper, why the claude-code leg never gates, and where the sandbox does and does not exist.
…config Which provider to use is a property of the runner -- docker, podman on a host that has that instead, local where there is no container -- and whoever runs the job knows it while a skill does not. What the sandbox has to provide is the opposite: the skill declares a compose file when it needs network egress or a device. `SKILLSCOPE_SANDBOX` already selected the provider but returned it bare, discarding any compose file the skill had declared. A skill that needs the network would then run without it and fail for a reason nothing in the report explains. The two are now resolved independently. podman works with no registration: `inspect-podman` publishes an `inspect_ai` entry point and inspect resolves the bare name, so installing the extra is the whole setup.
A behavioral report showed the same numbers whether the agent had been contained or had worked directly in the harness's filesystem with permissions bypassed, which invites the reader to assume the former. Both engines now record what held the run, and the markdown says it outright -- the legacy engine reports `host`, and the Windows legs, which have no sandbox available at all, no longer look like the Linux ones. Routing under the inspect engine reports `none` rather than a provider: it offers the skill tool and never calls it, so nothing is executed and there is nothing to isolate. That is not the same claim as unprotected. Also stops the gateway-versus-federation guard firing on runs that reach no provider. `--model mockllm/model` is the wiring check that costs nothing, and refusing it because the shell holds both Anthropic variables broke it on exactly the machines most likely to have an OAuth token.
A graded run on the inspect engine starts a sandbox and installs a skill before it first reaches a provider, so a bad key surfaced as a task that failed after all that work -- a 401 buried in a sample error, eighteen seconds and a container in. One tiny call up front turns that into a message on the first line. The legacy engine already had this; the inspect one skipped it on the grounds that the CLI-based probe tested the wrong thing, which was true and left nothing in its place. mockllm reaches no provider, so it is skipped rather than charged for a round trip that proves nothing.
`list_paths` returned an empty list when the listing command failed, so a sandbox that could not be listed was indistinguishable from one the agent had left empty. That is not a cosmetic difference: `files_exist` fails, and the judge builds its evidence from the same listing, so every judged expectation fails too with "no files were produced". Nine checks blame the skill for what the harness did. It now raises. The scorer reports that it could not list the sandbox, and the judge declines to rule rather than being handed an empty workspace as fact -- a judge told "no files" will confidently conclude the agent did nothing.
A container sandbox starts at `/`. So `write_file("router.json")` landed
at `/router.json`, `find . -type f` walked the whole image and died on
`/proc`, and the judge -- had the listing survived -- would have read
twenty arbitrary system files as the case's artifacts. An agent left to
guess reasonably tried `/app`, then `~`, and scattered its output.
Everything a case does now happens in `/workspace`: fixtures are seeded
there, the file tools resolve against it, shell commands run in it, and
it is what gets listed. The agent is told so, because where its output
lands should not be something the dataset has to predict.
`local` keeps the harness's own working directory -- it already has a
sensible one, and creating `/workspace` on someone's machine would not be.
inspect_swe solves the same problem the same way; its agent cwd falls
back to the home directory when the sandbox default is `/`.
Two reasons a case could not pass in a container. The judge read artifacts by the path `list_paths` reported, which is relative to the case's working directory, while `read_file` resolves against the sandbox's own -- `/`. Every artifact came back unreadable and the judge concluded, reasonably, that nothing had been produced. It now resolves them the same way the tools do. And it never saw the agent's final message, so an expectation about what the agent *told* the user -- "output the curl commands they need" -- was unanswerable however well the agent had done. The legacy judge was given both halves and this one was not. They stay labelled apart, because the distinction is what stops an agent writing "I won't call the cloud API" from settling an expectation that it avoided doing so: actions are judged from tool calls and artifacts, and only what the agent told the user is judged from its message.
The judge's transcript was cut at the end, so the agent's most recent tool calls were the first to go -- and those are usually the ones an expectation is about. An agent writes a file, validates it, and reports the result; losing the validation makes the run look like it claimed something it never did, which is precisely how the last failing check read: "only the agent's unsupported claim". Tool calls are now kept whole, since each is short and the list of them is the record of what happened. Only results are capped, individually, because they are what grows without bound. If the whole still overflows, the middle goes rather than the end.
This repository is public and the fixture named a real internal CI service account alongside the gateway's header, which together describe how model access is fronted. The test is about parsing `Key: value` lines; the values were never the point.
Every bug in this branch was found by running against a real container, and none of them would have been caught by the unit suite. `mockllm` reaches no provider, so the machinery around the model can be driven for free, with no key, on a pull request from a fork -- on Ubuntu, where Docker is the default sandbox, and on Windows, where there is none and the cross-platform tools are the only thing that works. The assertions are about the harness rather than the skill, because the mock satisfies nothing. It checks that no case errored, that the sandbox that ran is the one the platform should have chosen, and that no check failed because the sandbox could not be listed -- which otherwise reports identically to an agent that did nothing. The load-bearing one is the seeded fixture. The file exists because the case staged it, not because the agent acted, so `files_exist` passing proves the whole path: staged into the working directory, listed there, and matched against what the case asked for. That is exactly the chain that was broken in a container an hour ago.
Two failures on the first run of the new job, both worth having. The sandbox-selection tests asserted `docker` on any host, which is only true on a POSIX one -- on the Windows runners the answer is `local`, because inspect's sandbox layer assumes a POSIX guest. They pin the platform now and assert both answers, which is what they should have said in the first place. The engine job pointed at its fixture with `working-directory`, but a `--skills-dir` glob that was passed is resolved against the repo root, and `find_root` takes that from the nearest `.git`. Running from a subdirectory of a checkout still globbed the checkout, so the fixture was never found. `SKILLSCOPE_REPO` is the documented way to say which repo is under test, and is what the action already uses. The local reproduction missed this because the throwaway fixture was not inside a git repository, so the root happened to be the fixture.
The CI pin coerces any non-opus model to opus so paid runs stay comparable between runs. `mockllm` reaches no provider, grades nothing and costs nothing, so pinning it turned the free wiring check into a run that needed a key -- in the one environment where not needing a key is the entire point. The new engine job failed on exactly that.
`mockllm` never calls the submit tool, so the agent looped to the full 120-message budget and every turn was a real sandbox round trip. The CI grade step took 43 seconds to learn what it knows in three turns, and the same thing made a mock run over a real catalogue take 157 seconds for three cases. A model that reaches no provider now gets six turns. The job's grade step drops to about two seconds and the assertion it rests on -- the seeded fixture being found -- is unchanged. Also caches the pip download. inspect-ai pulls in the order of eighty packages, which was the other twenty seconds.
A trial on a self-hosted runner installed the podman binary, selected `SKILLSCOPE_SANDBOX=podman`, and failed with a ValueError from inspect's registry naming neither the variable that chose the provider nor the package that supplies it. The binary being present proves nothing: inspect resolves a third-party provider through an entry point, so the Python package has to be installed too, and that is the `[podman]` extra rather than `[inspect]`. Checked once before any container starts, with the install line in the message. The resolver is injectable so this is testable without the inspect extra, which the unit suite runs without on purpose.
A `react` agent delivers its answer through the submit tool, which lands in `output.completion`. The judge read the last assistant message instead, and that is often the preamble -- so an expectation about what the agent told the user could be graded against "the commands are below" rather than the commands. On a real skill this showed up as the judge reporting that the final message "only describes having printed registration/test commands but contains no actual curl commands", which is exactly what it was shown. Falls back to the last assistant message for agents with no submit tool.
Three things, each found by the previous one failing on a real runner: the provider package, podman's own compose rather than the shim that delegates to Docker's, and a search registry because podman will not guess one for an unqualified image name. Worth the setup where the runner's user cannot reach the Docker socket, which is the case this came from.
rominf
added a commit
to amd/skills
that referenced
this pull request
Sep 14, 2026
The default sandbox has no network, which I had been treating as a reason those skills cannot be containerised. It is not: it is a default, and `sandbox:` in machine.yml exists so a skill can say otherwise. That key has been unit-tested and never used for real, so this is its first outing. `local-ai-app-integration` installs a runtime and fetches models. Its compose file is inspect's own default minus the line that removes the network, with the image fully qualified because podman will not guess a registry. Note this cannot merge ahead of amd/skillscope#19: releases before it reject `sandbox:` as an unknown key outright.
It resolved against the skill root, which is what gets published -- so eval infrastructure would ship with the skill. A path written in a file is also most usefully relative to that file. Both point at evals/, beside the machine.yml that names it. Found by writing the first real one.
A malformed sandbox declaration raised out of the whole behavioral command, so one skill's broken setup discarded results for skills already graded and paid for. On a real run that threw away three passing cases to report that a fourth skill named a compose file that was not there. It reports failed outcomes for that skill and carries on, which is the same rule the structural gate already follows: a neighbour's mistake says nothing about whether this run can proceed.
In an agent loop the user sees every assistant turn, so an expectation about what the agent told them is satisfied by any of those. Reading only the final turn credited the agent with its closing summary and called the work missing -- on a real skill the judge reported that the agent "only claims curl commands were delivered as text" when it had printed them a turn earlier. The submitted answer still comes last and is marked, because it is the answer where the rest is working. The split the prompt relies on is unchanged: actions are judged from tool calls and artifacts, and only what the agent told the user is judged from what it said.
`inspect_swe` runs Claude Code inside the sandbox and reaches the model through a bridge whose proxy is a Linux binary, so it cannot run on Windows at all. This drives the same CLI on the host, the way the legacy engine does, and maps what it did into inspect's messages -- so the scorers, the artifact-reading judge and the .eval transcript all work unchanged. The point is fidelity. A skill is written for this harness, and grading a different agent measures something customers will not experience. With this the real harness runs on both platforms and only the isolation differs: a container on Linux via --engine claude-code, the host on Windows via --engine claude-cli, with the report saying which. Unsandboxed by construction, which is what the legacy engine already does, so it is not a regression. Refuses a container provider outright rather than running the CLI in one filesystem and scoring another. `build_task` gains a solver seam: what drives the agent swaps while staging, scoring, judging and reporting stay put.
Legacy against inspect asks whether a different agent reaches the same verdicts. Legacy against claude-cli asks something narrower and sharper: both drive the same CLI, so agreement means the framework around the agent is faithful, and a flip is a defect in the crossing rather than a property of a different agent. That is the measurement that decides whether the legacy path is redundant or merely superseded. The engine list now lives in the CLI and is imported, so a new engine is offered in both places at once rather than in whichever was remembered. On the fixture the two agree on every case at 9 model calls and ~27s each, which is what the same binary run twice should look like.
Every files_exist check on the claude-code leg of the first trial run failed with "sandbox holds: nothing" -- nine of nine, across every skill that asserts files. The agent was not doing nothing: a container starts at /, so it worked there while the scorers read /workspace. The other engines say this in a prompt, because their agent takes instructions. This one takes a cwd. The directory is created before the solver runs, since this leg's agent brings its own tools and so never touches ours, which is what would otherwise have made it. Read as a comparison, the old numbers said real Claude Code performs worse than a harness-independent agent on these skills. They said nothing of the kind.
Every engine but `legacy` runs on inspect_ai, so any of them can hit the missing-extra path -- but the hint said `--engine inspect` regardless. A Windows CI job that had passed `--engine claude-cli` was told to go look at a flag it had never used. The remedy is the same extra either way, so this only costs a reader their bearings. That is enough.
The guard around the inspect dispatch was a literal tuple listing only inspect and claude-code. `claude-cli` was added to the chain inside it but never to the tuple, so it fell through to the legacy engine: every run that asked for it silently got something else, while the report recorded `engine: claude-cli` throughout. The preflight, which does key off INSPECT_ENGINES, meanwhile demanded an extra the run never used -- which is how a Windows job failed on a flag it had not passed. The absent .eval transcripts were the symptom that gave it away: legacy does not write them. This invalidates every claude-cli measurement taken so far, including the benchmark that reported legacy and claude-cli agreeing 4/4 with cost within 3%. They agreed because they were the same engine; the figure matched the noise floor exactly, which should have been the tell. Tests now assert which runner a flag reaches, and that the guard and the choices list cannot drift apart again.
It graded skills with a react agent carrying five tools where the real harness has many more, so a skill referring to a tool it did not have failed the case for a reason that belonged to the agent rather than the skill -- and nothing in the report told the two apart. Every remaining engine drives the agent a skill is actually written for; what differs between them is where it runs. Its behavioral numbers never exceeded the sandboxed real-CLI leg on any skill, so nothing it measured is lost. Its routing numbers were a different matter: it offered each skill as a tool definition and took one turn, which is not how the harness finds a skill, so those numbers were never interchangeable with the rest. Routing is now legacy-only and says so, until a leg exists that drives the real CLI. Removing it costs one piece of coverage, so that is replaced rather than dropped. The old CI job proved the sandbox layer -- container starts, fixtures stage, guest lists, expectations match -- by grading a fixture with a mock model, and neither replacement engine can run key-free on a fork's pull request. tools/sandbox_smoketest.py exercises the same layer with a stub solver instead: the case seeds a file and expects that same file, so nothing an agent does can pass it or break it, and the assertions move from YAML into reviewable Python that also runs locally. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The old name said which binary it drove, which stopped distinguishing anything once the engine that drove something else was removed. Every engine drives the claude CLI now; what differs is where it runs. The new name says that, and says it against its sibling: claude-code runs the same agent in a container, this one runs it on the host. A reader choosing between them no longer has to know that "cli" was the one without isolation -- which is the property that actually decides which to use, and the one a reader is most likely to get wrong in the direction that overstates what a run proved. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Git tracks this as a rename, so the diff is the identifiers rather than the file. `--engine claude-code-no-sandbox` now traces to `engine/no_sandbox.py` and the solver it exports, instead of stopping at a module named after the old flag value. The guard's message and the sandbox docstring already said which engine they belonged to; this makes the file agree with them. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Four, found by reviewing the removal rather than by running it. The benchmark's routing mode could no longer complete. Routing has one engine, so every pair is either the same engine twice or a candidate that will be refused -- and the baseline leg runs first, twice under --noise, so the refusal arrived after a full routing run had been paid for, reported as a missing file rather than as the engine that was never going to run. It now refuses up front, before spending anything. The sandbox smoke test asked the harness which provider it had chosen and then checked that answer against itself, which cannot fail. A run that quietly got no container -- a stray SKILLSCOPE_SANDBOX, a change to the default, an inverted platform check -- passed every remaining assertion against the host filesystem and stayed green under the name "Sandbox machinery". The expectation now comes from the caller, per platform, the way the job it replaced did. The refusal's own test asserted a word that appears elsewhere in the message, so deleting the clause it was guarding still passed. It asserts the clause now; confirmed by deleting the clause and watching it fail. The rest is prose that outlived what it described: an install hint defaulting to an engine argparse rejects, a module explaining its reason for existing in terms of the deleted one, a comment pointing at a prompt that no longer exists, and the machine.yml schema text skill authors read. Also ignore inspect's .eval transcripts. One was staged by accident while fixing the above, which is argument enough. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`inspect_swe` prepares the guest for the CLI by writing $HOME/.claude/settings.json outright, discarding whatever was there. In a container that file belongs to nobody and the write is setup working as intended. Under SKILLSCOPE_SANDBOX=local, $HOME is the developer's own, and the same write destroys their real configuration -- permissions, model, gateway environment -- with no backup and no warning. `require()` already refused Windows and a missing extra. It did not refuse this, so the combination was reachable by anyone who had set the variable for the other engine, which is the engine that requires it. Found by doing it. The recovered file was four weeks stale; there was no newer copy to recover. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Routing ran on `legacy` alone, which made the sandbox the one thing it could not have -- and routing is where isolation matters most. A stray user-level skill does not spoil one case's grade; it is offered for every prompt, so it changes every decision at once and the run still reports a clean accuracy. Both new legs reach the same `claude` CLI through inspect_ai: `claude-code` inside a container via inspect_swe, `claude-code-no-sandbox` on the host. They surface the agent's tool calls in different shapes, so the crossing happens in one direction only -- a bridged ToolCall is re-wrapped into the stream-json shape `routing.detect_activation` already reads. Two graders that must agree about what counts as an activation would drift; converting data is cheap. The host leg refuses to start without the credential that lets it redirect CLAUDE_CONFIG_DIR. The legacy engine only warns, because it reads the CLI's init event and can name a skill that gate-crashed the room. Neither leg here gets that event, so the same contamination would be invisible -- and invisible is the part that matters. Observed, not feared: a probe of this leg put roughly forty of the host's user-level skills in the room and none of the three that were staged. Stopping at the decision is deliberately absent, and the module says why: on the host leg the subprocess buffers until exit, so the decision is only visible once it is paid for; on the sandboxed leg an approval policy could do it, but approval runs before the bridge adopts the assistant message, so terminating on the tool call that reveals the decision may discard the observation it fired on. That failure is silent and looks exactly like an agent correctly declining to route. It waits on a container run. What contained a run is now passed to `_finish_routing` by the leg that knows, as required keywords, rather than derived. Deriving it from the engine's name is what reported a container for runs that never started one. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`test_a_container_provider_is_allowed` called `verify.require()` and expected None. It passed locally and failed on every unit job: that function checks the sandbox, then imports inspect_swe, and the unit suite runs without the extras on purpose. The test meant something narrower than it asserted -- that a container is not refused *for being a container* -- so it asserts that instead, and tolerates the run stopping on a missing wheel, which is a different answer to a different question. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
They patched `sandbox.is_windows`, which `verify.require()` never consults: it reads `sys.platform` itself, and reads it before the sandbox check. So on a Linux runner these measured the sandbox guard and on a Windows one they measured the platform guard, under the same names -- passing locally and failing on every Windows job. Both levers are pinned now, so the class measures the same thing wherever it runs. Checked by running the whole suite with every platform-branching module told it is on win32. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The docs, the benchmark's refusal, and its docstring all still described routing as legacy-only. They were written when that was true and were left behind by the legs that made it false. The replacement says more than "all three work". Which leg a routing run uses matters more than it does for behavioral, because a stray user-level skill changes every decision at once rather than one case's grade -- so the docs now state what each leg does about that, and that the two inspect-backed legs cost more per case because neither stops at the moment a skill activates. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
It did not reach them at all. The only bound on a sample was the whole command's remaining budget, so one hung prompt could spend the run -- the exact thing the flag exists to prevent, as `deadline`'s own docstring says. The report meanwhile recorded `case_timeout` as though it had applied, which is the same "a cap that is reported and not enforced" defect this branch has now fixed twice elsewhere. inspect's `Task(time_limit=)` is per sample, which is per case, so the flag maps onto it directly. Clipped to what `--timeout` has left, for the reason the behavioral cap keeps a reserve: the command deadline ends the process outright and takes the report with it, so a per-case cap that outlives it turns a timeout into silence. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Legacy kills a routing case the moment a skill activates, counts tool calls, and passes the CLI's own spend cap. The new legs did none of that: a case ran to a message cap whatever happened, so a decision made on turn one was paid for to turn thirty. The sandboxed leg now stops earlier than legacy manages. An approver sees each tool call the bridged CLI *proposes*, so the case ends before the work happens, rather than after it has run and the process is raced to a kill. The tool-call and inspection budgets ride the same path, reusing legacy's own BOOKKEEPING_TOOLS and inspection test so the counts mean the same thing. The skill's name goes into the approver's reason, and the outcome mapper reads the transcript first and that reason second. Approval runs before the bridge adopts the assistant message, so terminating on the call that revealed the decision can discard it -- and a suppressed activation is indistinguishable from an agent that correctly declined to route. Two sources, one of which cannot go missing, rather than one that probably survives. The host leg gets the CLI's own --max-budget-usd, probed first the way legacy probes it. It still cannot stop at the decision or count calls: its stdout is buffered until exit, so the decision is only visible once it has been paid for. Recovering that means driving the subprocess directly, which is what legacy already does. The decision rule is a pure function so it is tested in CI. The unit suite runs without the inspect extra on purpose, and a rule reachable only through an approver would have had no coverage where it matters. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The docs claimed neither inspect-backed leg stops at the decision. That was true when written and stopped being true an hour later: `claude-code` declines the call that reveals the decision before it runs, which is earlier than `legacy` manages. Replaced with the per-leg difference, since that is what decides what a routing run costs -- and with the note that the report records which caps applied rather than which were passed, because a cap that is reported and not enforced has now been the same defect three times on this branch. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
added a commit
to amd/skills
that referenced
this pull request
Sep 23, 2026
The candidate engine has run against one skill in a container on a laptop, and against fixtures in hosted CI. It has never run here: these runners, this catalogue, this gateway. Routing first, because it starts no sandbox and so cannot be blocked by whatever runtime the runner has -- and because one model call per case makes it the cheap half. Scoped to its own branch and reporting rather than gating. It is not part of this repository's CI and deleting the branch removes it.
rominf
added a commit
to amd/skills
that referenced
this pull request
Sep 23, 2026
The default sandbox has no network, which I had been treating as a reason those skills cannot be containerised. It is not: it is a default, and `sandbox:` in machine.yml exists so a skill can say otherwise. That key has been unit-tested and never used for real, so this is its first outing. `local-ai-app-integration` installs a runtime and fetches models. Its compose file is inspect's own default minus the line that removes the network, with the image fully qualified because podman will not guess a registry. Note this cannot merge ahead of amd/skillscope#19: releases before it reject `sandbox:` as an unknown key outright.
A gateway 504 lands in a report as a lower score with nothing in the verdict saying why, so a reader cannot tell it from the skill failing. On one catalogue these have been observed on roughly a third of runs, which makes an unmarked provider error the most likely reason two runs of the same engine disagree -- and therefore the first thing to rule out before a difference between engines means anything at all. Three parts. The reason is kept: a result event puts it in `result` or in `subtype` depending on how it failed, and collapsing both into "result event reported an error" discarded the one thing that made the failure readable. Observed doing exactly that to a 504. Then it is classified, with deliberately generous matching -- a false positive makes a reader look twice at a run that was fine, a false negative lets an outage score as a routing miss, and those costs are not symmetric. Then it is counted, in the totals of both commands and on both engines, so the two cannot describe an outage differently during one. The report says so above the verdict table rather than below it. A reader who has taken in the score has already formed a view, and a note underneath does not undo it. What is not guessed at: an error with no stated reason is neither marked nor excused, and `error_max_turns` is the agent running out of road rather than the gateway. Marking that degraded would excuse a real failure. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Only the legacy engine runs evals/hooks.py. No inspect-backed engine builds the ctx those hooks take, so on those engines the file is simply not read -- and not read leaves no trace. The case is graded as though its setup happened, and the failure turns up later as a skill that mysteriously does not work on this runner. The catalogue this was written against ships one: serving-llms-on-instinct uses setup to clear stale vLLM containers and teardown to remove them. Skipping that on a shared GPU runner leaks containers holding GPU memory into whatever runs next -- the same contamination the sandboxed engines exist to prevent, arriving through the door left open by removing the engine that closed it. So a behavioral run whose skills ship hooks the chosen engine cannot execute now stops, naming every skill affected, before the model probe -- finding out the credentials are fine is no use when the run could not have honoured the setup anyway. Routing is exempt: it installs the skills, asks which one fires, and executes nothing. This is a gate on retiring the legacy engine, not a workaround. Either the inspect path grows hook support or the remaining hooks go; until one of those happens, the refusal is what keeps the choice visible. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
On the real runner the host leg errored two cases the legacy engine graded `correct_trigger` and `true_negative`, both with "the sample produced no agent messages". The runs had happened. The driver appended a closing message only when the CLI had something to say, so a run that finished without a tool call and without a closing sentence left an empty message list -- and the routing mapper, whose job is to tell "reached for no skill" from "never ran", could not tell them apart from nothing. An agent that declines to route is the case a routing eval most needs to grade, so losing exactly those is the worst shape the bug could take. The distinction still exists, in the right place: a stream carrying no result event at all is a CLI that did not reach the end, and still produces nothing. Only a finished run with an empty answer is now recorded, using the placeholder `state.output` has always used for the same case -- the message list was the half that disagreed with it. The rule is a pure function so it is tested without the inspect extra. The first version of these tests needed it, and would have gone untested in CI, which is exactly where this failure came from. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The first sandboxed routing run on a real container graded 24 of 67 cases as "the agent never ran". They had run: the legacy engine saw those same cases activate a skill. The mapper read `sample.messages` alone, and for a bridged agent that record is not reliable -- inspect adopts one conversation onto the sample following heuristics about which thread is the main one, and a run ending inside a sub-agent can leave it holding the wrong thread or none at all. The transcript is strictly more complete: every bridged generation emits a ModelEvent, sub-agents included. This was known before the leg was written and the leg was written without it, which is the whole of the bug. Both records now, in order and de-duplicated by identity, because the same turn appears in both when both have it and the routing decision is the first skill reached for -- counting one turn twice could move it. The distinction the guard exists for survives: a sample with neither record still reports as never having run, because that is an infrastructure failure and grading it as a miss would invent a routing result. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Three faults in the approver, all found by the first sandboxed run that worked, all pushing the same way: terminate a case before it chose. The tally was captured when the approver was built, and an approver is built once per task -- so a per-case budget was really a per-run one. It crept up until it crossed the budget and then terminated any case that made a counted call before reaching for a skill. Visible from outside as four cases stopped at tallies of 10, 11, 12 and 13: consecutive, across different prompts. It lives in inspect's store now, which is scoped to the sample. `_is_skills_inspection` takes (tool_input, skills) and was being handed (tool name, JSON arguments). It therefore never recognised a survey of the installed skills, which is why every budget message in that run read "0 inspection(s)". And a survey was spending both budgets rather than only the inspection one. Legacy makes the three categories exclusive and says why: ending a run mid-survey scores deliberation as a missed trigger. That is precisely what this did. None of it changes a verdict that was reached honestly -- a case that chose is still graded on what it chose. What it changes is how many cases got to choose at all. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Four cases on the real runner were reported as "the sample produced no agent messages, so the run never made a routing decision". They had hit the message cap. What survives on the sample when that trips can be the prompt and nothing else, so the check for an assistant turn found none and called it an infrastructure failure. It is the opposite of one. An agent that spent its whole budget without reaching for a skill is the clearest missed trigger there is, and the legacy engine grades its own equivalent -- `tool_budget` -- as a verdict rather than an error. Reporting it as "never ran" both loses the finding and blames the machine for the agent's behaviour. A sample carrying a limit therefore counts as having run. A sample with neither a limit nor an assistant turn still does not, because that is the case the guard exists for. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The sandboxed leg reported ten tool calls for cases the approver had terminated at five. Both numbers were right about different things: the approver saw each call once, and the mapper saw each twice. De-duplication keyed on object identity. The same turn arrives as two different objects -- one adopted onto the sample, one carried by the transcript event -- so identity never matched and every call was counted in both. Only the sandboxed leg was affected: the host leg has no bridge, so no transcript events to duplicate, which is why its distribution looked sane while the sandboxed one ran to ten. It keys on message id now. The column was the visible half; the effective budget was the other, because a doubled sequence crosses a per-case cap at half the calls it should. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The previous commit went out with two failing tests: the stand-in ToolCall gained an id in the assertions but not in its constructor. I chained the push onto a grep for failures, which matched, succeeded, and let the push run anyway. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Two runs of one engine disagreed on six cases against a noise floor of one, and the reason was not visible in either report: 26 of 67 cases had stopped within a single call of a budget. Those are coin-flips. One more call either way moves them across the line and takes the verdict with them, and in the report they look exactly like cases decided on their merits. The count now sits in the totals and above the verdict table, because a reader who has taken in the score has already formed a view about what the run measured. Not corrected, reported. The budget is doing its job -- stopping a case that is rummaging rather than choosing is the whole point. What was missing is any way to tell a flip that means something from a threshold artefact, which is the difference between two runs disagreeing and two runs being unreadable. Caps a leg did not set are not thresholds: a leg that cannot enforce a budget reports none, and this invents none for it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The cap asks one question -- is this agent still choosing, or has it started work -- and the answer depends on where it is standing. Measured over one 67-case room: legacy averages 0.58 tool calls before deciding and peaks at 4; the same agent in a container averages 2.46 and peaks at 10, because it is orienting in a filesystem it has never seen. That is not worse routing. Held to a flat 4, it put 24 of 67 sandboxed cases within one call of the threshold. A third of that run measured the budget rather than the agent, and the leg scored lower for it -- 0.851 against 0.91 -- which then read as a difference between engines. Two engines judged by one number while doing different amounts of unavoidable work is unfair by construction, and repeating the comparison only measures the bias more precisely. Scaled rather than replaced, so a caller asking for a tighter budget still gets one everywhere. The report states the cap that was enforced rather than the one requested, because otherwise the near-limit count is measured against a threshold that never applied. Three is calibrated from that measurement and clears the observed peak with room to spare. It is a measurement, not a constant; recalibrate it if the room or the image changes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The factor of three came from a mean of 2.46 tool calls per sandboxed case, which summed two unrelated populations: cases that call the skill tool immediately, and cases that never find a skill and rummage until something stops them. Separated, the first group -- the only one the budget can meaningfully cut short -- decides at a mean of 0.31 calls, median zero, peak four. Thirty-two of thirty-nine call the skill tool first with no preamble at all. The cases sitting near the old threshold are all in the second group, and budget cannot rescue them: they score no_activation at four calls or at forty. So the scaling bought no accuracy and spent real money doing it. Two keeps headroom over the observed decision peak without funding the agents that are lost. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
This leg answered the routing question on its first tool call and then went on to do the entire job: downloading trace files, running the analysis, writing reports, spawning subagents. Measured over one 67-case room, 152 of its 220 tool calls happened after the decision it was being asked for, and nothing read any of them. The cause was structural. It awaited inspect's `subprocess`, which returns only once the process has exited, so nothing it produced could arrive in time to end a run. The sandboxed leg has an approver because its calls cross inspect's bridge; this leg's CLI is a plain subprocess with no such interception. So apply the same rule from the other side, against the `stream-json` the CLI prints as it goes, and kill the process group the moment it fires -- which is what the legacy engine has always done, and this leg is what replaces the legacy engine. Opt-in, because the two callers want opposite things from one CLI: a behavioral case is graded on what the skill produced and has to run to the end. Left unset, the driver takes the old path unchanged. The rule is handed over as a factory rather than a closure: the solver is built once and run per sample, so a shared tally would carry one case's spent budget into the next -- a bug already shipped once in the approver. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A bridged turn reaches the mapper twice -- adopted onto sample.messages and carried by a transcript event -- and the two copies were being counted as two turns. De-duplicating by message id looked like the fix and was not. The bridge builds a fresh message when it adopts the turn rather than moving it, so each copy carries an independently generated id and the keys never matched. The provider's tool call ids do survive the crossing, which makes them the only stable name the turn has. Measured on one 67-case room: 299 tool calls reported where the transcript held 226, and case-by-case as bad as 23 against 12. Verdicts were never affected -- an activation is found by presence, not by count -- but every tool-call column was, and so was the headroom the budget was calibrated against. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Both legs that build the CLI's command line pass --effort into it. The sandboxed legs go through inspect_swe, whose `effort` argument they never filled -- and its default is not "match everyone else", it is documented as leaving the model's own default reasoning effort in place. The behavioral leg was worse: `run` takes an effort, and dropped it on the way to `build_task`, which had nowhere to put it. So the trial compared a leg thinking as hard as it was told against one thinking as hard as it liked, across the whole catalogue, and read the difference as a fact about isolation. Any leg-to-leg gap measured before this needs measuring again. Pinned by tests that read the call sites, because the alternative is a live sandboxed run and a silent revert here would look like a finding about skills rather than about the harness. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The previous commit passes `effort` to claude_code(). 0.2.70 does not take it, and the extra's floor was 0.2.70 -- so a runner that already had it satisfied the requirement, never upgraded, and raised `TypeError: Unexpected keyword argument(s): effort` inside a task inspect had already started. The sandboxed leg produced no report at all while the run reported success. Raise the floor, and check the capability in `require()`, which exists for exactly this: a legible refusal before the run rather than a traceback partway through one. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two eval engines built on
inspect_ai, selectable with--engine, alongside the existing one. Nothing about the default path changes:--engine legacystays the default, the new dependencies are optional extras, and a repo that has not migrated installs nothing new.Three engines are now available. All of them drive the agent a skill is actually written for, so what differs is where it runs — not what it is, and not how cases are written or graded:
--enginelegacyclaudeCLI, driven directlyclaude-codeinspect_sweclaude-code-no-sandboxinspect_aiWhy. The existing engine hand-rolls machinery that a framework already provides: subprocess stream pumping, process-group termination, brace-scanning verdict extraction, and hand-rolled activation detection. Moving to
inspect_aibuys a real sandbox, structured scorers, a judge that can read the artifacts a case produced (including images),.evaltranscripts, retries, and time and cost limits.A note on the isolation claim, which this PR no longer makes. An earlier revision argued that isolation was the substantive gain, on the evidence that one skill's cases scored 0/7 under the host-driven engines and 7/7 under a sandboxed one. That evidence does not support the claim, and I would rather say so here than have a reviewer find it.
Checking the trial runner's logs: its sandbox probe falls through to
localwhen no container runtime answers, and it has been doing so since 2026-09-15. Exactly one Linux run was ever genuinely containerized, and the engine it graded was the harness-independent one this PR removes. Everyclaude-coderesult in the table below was produced on the host filesystem, by a leg that reported itself as sandboxed.So the 0/7 → 7/7 split is real and its cause is not established. It is not the container, because there wasn't one. It is not a clean working directory either — both
claude-codeandclaude-code-no-sandboxgive each sample a fresh temporary directory, and they land on opposite sides of the split. One untested possibility is thatinspect_sweoverwrites$HOME/.claude/settings.jsonto set up the agent, which on a shared runner would also clear any user-level skills; that would make the isolation a side effect of a destructive write rather than a property worth shipping.What this PR claims instead is narrower and checkable: the engines are the same agent in different places, the reports say which, and the guards refuse the configurations that silently lied about it — including the one that produced this table.
--engine claude-codenow refuses a sandbox that shares the host's filesystem, which is what surfaced all of the above.Non-obvious decisions
The dataset contract is unchanged.
evals.jsonand the CLI surface are untouched, so a product repo migrates by changing one flag.machine.ymlgains an optionalsandbox:key naming a compose file beside it. Absent means the default container with no network — right for a skill that only reads and writes files. Name one to opt into egress or to bind a device in.examples/skill-with-a-device/evals/is the worked pair: amachine.ymlasking for GPU runners, and the compose file that binds the devices in and grants egress. The two are not substitutes —labels:decides which machine the job lands on,sandbox:decides whether the container on it can see the hardware that machine has.Routing now runs on all three. It previously had one leg, and the other two were refused rather than allowed to fall through to
legacyand report a container that never started. Routing is where isolation matters most — a stray user-level skill changes every case's decision rather than one case's grade — soclaude-codeis the leg to prefer there: its guest has no~/.claudeto contribute.claude-code-no-sandboxcannot read the CLI's session-init event, so it cannot notice or report that contamination, and therefore refuses to run a routing leg withoutANTHROPIC_API_KEY, which is what lets it redirect the config dir. On stopping at the decision,claude-codeis ahead oflegacy: its calls cross inspect's bridge, so the one revealing the decision is declined before it runs, rather than observed after the fact and then raced to a process kill. The tool-call and inspection budgets ride the same path.claude-code-no-sandboxcan do neither — its CLI's output is buffered until exit — so it runs to a message cap, bounded by--case-timeoutand the CLI's own--max-budget-usd, and costs more per case. Each report records which caps actually applied rather than which were passed.There is no harness-independent engine. An earlier revision of this PR added one — a framework-native agent, no
claudeCLI involved. It is gone. It carried a smaller tool set than the real harness, so a skill referring to a tool it did not have failed for a reason belonging to the agent rather than the skill, and nothing in the report distinguished the two. It never scored above the sandboxed real-CLI leg on any skill, and its routing numbers measured a different activation mechanism than the harness uses, so they were never interchangeable with the rest.Windows runs unsandboxed, by construction.
inspect_sweinstalls a Linux proxy binary in the guest and cannot run on a Windows guest at all, soclaude-code-no-sandboxdrives the CLI from the host instead. That is what the legacy engine already does on Windows, so it is not a regression — but it is not an improvement either, and reports now statesandbox_isolatedrather than leaving a reader to assume.inspect_sweis an optional extra, not load-bearing.claude-code-no-sandboxreaches the same real agent throughinspect_aialone, so only theclaude-codecross-check depends on it.The package gets bigger, not smaller. 7,124 lines against 4,938 on
main. Retiring the legacy path later removes 798 measured lines and plausibly ~1,200 with the imports, constants and tests that exist only for it — so roughly a quarter to a third more code than today, in exchange for the machinery above. This PR removes none of it.Non-goals
Retiring the legacy engine, and flipping the default. Both are follow-ups, deliberately: this PR is net-additive so it can be reverted by not passing a flag.
Test plan
Unit tests run in CI on Linux and Windows, alongside a key-free pass over the sandbox layer itself — a case seeds a file and expects that same file, with a stub solver standing in for the agent, so what passes or fails is whether the container started, the fixture staged, the guest listed and the listing matched. No model is called, so it runs on a pull request from a fork, which is where a cross-platform sandbox bug shows up.
The evidence below comes from running the whole catalogue of a downstream skills repository on self-hosted runners, through an enterprise Anthropic-compatible gateway, at one commit.
Routing, measured per case against a noise floor. Routing now runs on all three engines, so it can be compared rather than asserted. Whole room, one commit, one session, no degraded cases:
legacyvsclaude-code-no-sandboxlegacyvs itself, second passFive of the seven engine flips are not explained by run-to-run variance. Aggregate accuracy, meanwhile, is 0.892 against 0.894 — near-identical, and it conceals every one of those flips. That is the reason this comparison is per case: the headline number agrees while the cases underneath it do not.
Cost on the same room:
legacy$3.79 over 261 model calls,claude-code-no-sandbox$14.62 over 869 — 3.9×, which is the price of not stopping at the decision.claude-codestops earlier thanlegacydoes; the host leg cannot, because its CLI's output is buffered until exit.Both legs reported zero provider-degraded cases, which is what makes the comparison readable at all — a gateway outage during either pass would otherwise have shown up as an engine difference.
Behavioral, checks passed per skill (Linux). Read with the note above: the
claude-codecolumn was produced on the host filesystem, not in a container, so this is not a sandboxed-versus-unsandboxed comparison.legacyclaude-codeclaude-code-no-sandboxclaude-code-no-sandboxmatcheslegacyon every skill. Both drive the same binary, so that agreement is the evidence that wrapping the CLI in the framework changes nothing about what it concludes — and it is the one conclusion in this table that does not depend on the sandbox question.The
tracelensrow is the open one. Three engines ran on the same host and one of them scored 7/7 where the others scored 0/7, so something separated them; the note above sets out why it was not a container and not a working directory. Until that is explained the row is a real observation with no established cause, and it is visible at all only because more than one engine ran — which remains the argument for keeping more than one.Windows. Real Claude Code runs there under
inspect_aiviaclaude-code-no-sandbox:lemonade-router-builder21/21 andlocal-ai-use7/7, matching Linux. A second run of the same engine on the same machine scoredlemonade-router-builder15/21 with no change in between, so Windows run-to-run variance is not characterised — treat individual Windows numbers as provisional.Known gaps
legacy. Their sandbox is verified to expose the devices and resolve DNS, and a case still overran an hour without reporting. Pre-existing, and not a regression from this PR.Risk
Low for existing users — the default engine, the dataset format and the CLI surface are unchanged, and the new dependencies are optional extras. The risk is carried entirely by repos that opt in with
--engine.