feat: add AgentRuntimeConfig with Docker and smolvm agent runtimes - #137
Merged
Conversation
- Add evalution/agent_runtime.py with BaseAgentRuntime, DockerAgentRuntime, SmolVmAgentRuntime, and UnsafeLocalRuntime. - Add AgentRuntimeConfig in evalution/config.py and wire local tool-calling agentic suites to it. - Refuse to evaluate tool-calling suites without a configured runtime; host execution requires explicit UnsafeLocalRuntime which warns on construction. - Replace the Docker sandbox helper with command extraction only; scoring now runs through the configured runtime. - Document agent runtimes and sandboxing usage in README. - Update tests for runtimes, CLI construction, guard behavior, and warnings.
…entrally - Add is_agentic and has_tool_calling ClassVar declarations to BaseTestSuite. - Flag all agentic scaffolds is_agentic=True; local Harbor suites also set has_tool_calling=True. - Move the tool-calling security guard into BaseTestSuite.evaluate so any suite declaring has_tool_calling requires a configured AgentRuntime, regardless of construction path (Python or YAML). - Drop the now-redundant evaluate() override in _LocalAgenticBenchmark. - Add flag coverage and central-enforcement regression tests; document the flags in the README Agent Runtimes section.
…e-resume tool loop - Move path and image onto BaseAgentRuntime; path defaults to "auto" which resolves the runtime binary from the environment PATH. - Replace docker_path/smolvm_path kwargs with the shared path kwarg. - Drop suite-level docker_image/docker_timeout; runtime config owns image and timeout, with per-task task.toml images still overriding per call. - Add an intercept-execute-resume tool loop to local agentic suites: explicit tool calls (fenced bash blocks or <bash> tags) execute on the configured runtime, the observation is appended, and inference resumes until a final answer or max_tool_turns. - Support apply_chat_template=True for message-based multi-turn tool loops. - Add try_extract_command for deterministic loop termination. - Add E2E tests running Llama-3.2-1B-Instruct through a real Terminal-Bench task on Docker (passes) and smolvm (skips without bootable KVM); the task command only prints the expected answer inside an Alpine runtime, proving execution is not local. - Update unit tests for scripted multi-turn sessions, auto path resolution, image defaults, loop interception/resume, and turn-cap termination.
- Prepare the Alpine rootfs by docker-exporting into a world-traversable directory: smolvm's per-VM uid isolation (uid 2000005) cannot traverse 0700 pytest tmp dirs, and re-pulling registry images fails offline. - Preserve executable bits when opening up permissions; blanket chmod broke guest exec. - Replace the skipif heuristic with a real boot probe fixture so the test only skips when a microVM genuinely cannot start. - Verified end-to-end on this host: Llama-3.2-1B-Instruct completes the Terminal-Bench task through both DockerAgentRuntime and SmolVmAgentRuntime; the guest kernel differs from the host, confirming VM execution.
…ened runtime config
Tool calling vs code output:
- Add evalution/benchmarks/tool_calling.py with declared protocols; only the
declared protocol is intercepted, so plain code output (fenced snippets,
prose) is inert model text and never executed.
- Replace the merged extractor with per-protocol parsing: bash_tags captures
all <bash></bash> markers (document order, case-insensitive, empty/unclosed
rejected); fenced_shell only executes shell-language fences with console
prompt stripping; native_json parses <|python_tag|>{...},
<tool_call>{...}</tool_call>, and bare JSON responses.
Native vs prompted models:
- GenerationRequest gains a tools field, threaded into chat-template rendering.
- Suites declare tool_call_mode=auto|native|prompted; auto probes the chat
template for native tool support and uses the model's pre-trained format
explicitly, falling back to the generic prompted <bash></bash> syntax
(injected as a system message) for models without native tools.
- Invalid mode/format combinations fail fast at resolve time.
Config flattening:
- Drop AgentRuntimeConfig; suites take agent_runtime=DockerAgentRuntime()
directly. BaseAgentRuntime carries shared path/image settings.
E2E coverage:
- Native: Llama-3.2-1B-Instruct completes a Terminal-Bench task via its
pre-trained tool template through Docker and smolvm runtimes.
- Prompted: Falcon-H1-3B-Instruct (no native tools) completes it via prompted
<bash></bash> markers.
- Fenced-shell variant kept for explicit protocol opt-in.
Strict security tests:
- Source tripwire forbids subprocess/os.system/os.popen/Popen/exec/eval in
benchmark modules.
- Code output is never executed under the default protocol; 100% of declared
tool calls route to the runtime with task images forwarded.
- Mode resolution matrix incl. forced-native-without-support failing closed.
Also normalize the pypcre dependency spelling (same PyPI distribution).
…ive parsing - Replace the permissive bash-tag protocol with strict <tool_call></tool_call> action markers (tool_call_tags): ordinary <bash>/fenced code output can no longer be mistaken for a tool call. - Capture every marker per generation in document order; a truncated final call (opening marker cut at generation stop) still counts; empty markers are dropped; special-token trailers (<|im_end|> etc.) are stripped from bodies. - Harden the prompted system message (mandatory-marker wording) after live compliance testing with Falcon-H1-3B-Instruct. - Native parser: balanced-brace JSON scan plus lenient escape repair for model-emitted invalid escapes (e.g. backslash-before-dollar); covers python_tag, Hermes-style XML, and bare JSON encodings. - Resume turns withhold the tool schema and request verbatim output so small models conclude instead of issuing further calls. - E2E task switched to a deterministic single-character runtime probe (0 inside Alpine, 1 on host); native/prompted/fenced/smolvm all pass repeatedly with real Llama-3.2-1B-Instruct and Falcon-H1-3B-Instruct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a first-class agent runtime abstraction for agentic benchmarks. Tool-calling suites (
terminal_bench_21,deep_swe,toolathlon_verified) execute model-generated shell commands, so they now require an explicitly configured sandboxed runtime before evaluation starts. No backward-compat shims are kept.What changed
evalution/agent_runtime.py:BaseAgentRuntime: ABC with a singlerun(command, *, image, timeout, env, volumes, workdir) -> AgentRuntimeResultcontract.DockerAgentRuntime: disposabledocker run --rmcontainers; configurabledocker_path, default image, timeout, network mode (defaultnone), and pull policy (defaultnever).SmolVmAgentRuntime: ephemeral smolvm microVMs (smolvm machine run) with hardware isolation, per-VM guest kernel, opt-in networking, and cpus/mem overrides.UnsafeLocalRuntime: explicit host-execution escape hatch that emits aRuntimeWarningon construction.AgentRuntimeConfiginevalution/config.py; local agentic suites take anagent_runtime=kwarg._LocalAgenticBenchmark.evaluate()raisesValueErrorwhen no runtime is configured — model-generated commands never run on the host by accident.evalution/benchmarks/agentic_docker.pyis now command extraction only (extract_command); the oldDockerSandbox/use_dockerpaths were removed.runtime_typeand exit code.evalution.Validation
EVALUTION_SKIP_GIL_CHECK=1 pytest tests/test_agentic.py tests/test_agentic_docker.py tests/test_package.py→ 34 passed.UnsafeLocalRuntimewarning + host invocation shape, live Docker smoke tests.compileallandgit diff --checkpass.