Skip to content

Fix tool-call extraction and populate events on the run() path - #504

Open
ambiorix2099 wants to merge 2 commits into
mainfrom
agentspan-evals
Open

Fix tool-call extraction and populate events on the run() path#504
ambiorix2099 wants to merge 2 commits into
mainfrom
agentspan-evals

Conversation

@ambiorix2099

@ambiorix2099 ambiorix2099 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • After run(), AgentResult.tool_calls dropped or misnamed most tool kinds and AgentResult.events was always empty. 13 of the 17 assertions in conductor.ai.agents.testing read one of the two; three pass on the empty list rather than fail.
  • Root cause: _extract_tool_calls keyed on a call_ reference-name prefix, which is the provider's tool-call id format and not Conductor's; _SYSTEM_TASK_TYPES skipped five task types that ToolCompiler.TYPE_MAP compiles tool kinds to; survivors were named task_type.lower(). run() never passed events=.
  • Fix: identify a tool by its compiled task type, name it from the _agent_tool_name key the dispatch script sets, and derive events from the execution already fetched. Polled task list throughout; the SSE path is untouched.

User impact

One runtime.run() per tree, same agent, a getWeather worker tool and an HTTP tool, Conductor 3.32.0-rc18 on openai/gpt-4o-mini:

result.tool_calls result.events
before [('getweather', {'city': 'Tokyo', '_agent_tool_name': 'getWeather'})] []
after [('getWeather', {'city': 'Tokyo'}), ('serverHealth', …)] thinking, tool_call, tool_result, tool_call, tool_result, thinking, done

Both runs scheduled the same tasks. assert_no_errors runs on every eval case and was green without evidence; expect_handoff_to, the runner's documented example, failed every time.

Changes

  • runtime.py: _is_tool_task / _tool_name / _tool_args replace the prefix check, a SUB_WORKFLOW counting as a tool only with that key, which separates an agent_tool from a handoff; events= at the five AgentResult sites; _task_events shared with both polling streams
  • runtime.py: framework agents derive both fields from their execution too, in _run_framework, _run_framework_with_events and both async branches, since the Claude Agent SDK injects a task per tool call
  • result.py: one internal-argument key set, so AgentEvent.args and tool_calls agree
  • tests/unit/ai/test_tool_extraction.py: new, per tool kind and both provider id formats
  • tests/unit/ai/test_runtime.py: test_non_tool_tasks_ignored used a SIMPLE task the server counts as a tool; now INLINE

Fixes #501

Tool tasks were identified by a `call_` reference-name prefix. That prefix is
the LLM provider's tool-call id format, not something the server adds, so an
Anthropic-backed agent (`toolu_…`) recorded no tool calls at all. Five tool
kinds were also filtered out as system tasks and five more were named from the
task type, and `run()` never passed `events=` at all.

Identify a tool by the task type it compiled to, per the server's
`ToolCompiler.TYPE_MAP`, and resolve its name from `_agent_tool_name` — which
the tool-dispatch script sets on every tool kind — falling back to `method` and
the task definition. Nothing reads the reference name for identity, so nothing
depends on provider-controlled data. Names are no longer case-folded, so
`getWeather` stops arriving as `getweather`.

`agent_tool` and a strategy handoff both compile to SUB_WORKFLOW; only the tool
carries `_agent_tool_name`, which is what tells them apart. The agent's own
statically-compiled workers — guardrails, callbacks, gates, routing — compile to
SIMPLE tasks like a worker tool does, so they are excluded by name; that check
is consulted only after the dispatch key has failed to settle the question.

`run()` and `run_async()` now derive `events` from the execution they already
fetch for `tool_calls`, with no extra call. Eight assertions in
`conductor.ai.agents.testing` read `events`, and four of them check for absence —
including `assert_no_errors`, which the eval runner runs on every case — so they
were passing without evidence. `expect_handoff_to`, used by the eval runner's
own documented example, was failing every time.

The task-to-event mapping was copy-pasted across the sync and async polling
streams; both now share `_task_events`, which is also what `run()` uses, so an
assertion reads the same whichever way the agent was run.

Verified against a live server: before, an Anthropic-backed agent with one tool
returned `tool_calls: []` and `events: []`; after, `[('getWeather', {'city':
'Tokyo'})]` and a full event list. The OpenAI path is unchanged apart from the
name no longer being lowercased.
@ambiorix2099

ambiorix2099 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Framework agents: the Claude Agent SDK injects a task per tool call (POST /agent/{id}/tasks) under the Anthropic tool-use id, so those tool calls are in the polled task list and now come back in tool_calls. _run_framework, _run_framework_with_events and both async branches derive tool_calls and events from the execution, as the native path does.

LangChain and LangGraph publish their steps to the event stream rather than as tasks, so their per-step detail still reaches on_event and stream() consumers only. A failed passthrough task shows up either way; an error one of those frameworks handled internally does not.

…-task detection

The Claude Agent SDK injects one task per tool call into the running
execution (POST /agent/{id}/tasks) using the Anthropic tool-use id as the
reference name, so those tool calls sit in the polled task list. But run()
routes every framework agent to _run_framework, which built its result with
neither tool_calls nor events, so the very case #501 captured its payloads
from still came back empty. All four framework result paths now derive both
fields from the execution, as the native path does. LangChain and LangGraph
publish their steps to the event stream rather than as tasks, so their
per-step detail is still stream-only.

_is_agent_internal_task matched its suffixes against the reference name or
the task definition name, which dropped an injected tool of the user's own
named, say, refresh_gate. It now requires both names to look internal, and
strips Conductor's __N turn counter from the reference first — without which
an internal worker inside the agent loop was read as a tool call, since only
the task definition name saved it.

Drops _transfer_check, a suffix nothing mints. Renames the new test doubles
to FakeTask/FakeWorkflowRun, so Workflow no longer stands for an execution.
Comments on the new code rewritten to the module's own style.
@ambiorix2099
ambiorix2099 marked this pull request as ready for review September 9, 2026 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool-call extraction drops or misnames most tool kinds, and run() returns no events

2 participants