fix(runtime): fail fast when the LLM endpoint can't return structured tool calls (#520)

[devin-ai-integration]

Summary

Fixes #520. Strix is entirely tool-driven — every working turn must be a native tool_calls response. Some self-hosted / OpenAI-compatible endpoints (llama.cpp without --jinja, an Ollama model whose template lacks tool wiring, a misconfigured vLLM tool parser) instead return the tool call as plain assistant text. The Agents SDK correctly treats that as a normal final message, so the interactive loop settles to waiting and the scan silently parks on "Send message to resume" with no hint why.

Root cause is inference-server configuration, not the model or Strix, and there is no safe client-side way to execute arbitrary text-form tool calls. So instead of stalling mid-scan, we now detect the missing capability up front and fail loudly with actionable guidance, plus document the exact server settings.

Preflight probe (strix/core/warmup.py), run once in run_strix_scan before the sandbox spins up (skipped on resume):

# only self-hosted / OpenAI-compatible routes, where #520 happens
def requires_tool_call_probe(model_name, settings) -> bool:
    return model_name.startswith("ollama/") or bool(settings.llm.api_base)

async def probe_tool_calling(model_name, settings, *, request_timeout=None):
    if settings.llm.skip_tool_call_probe or not requires_tool_call_probe(...):
        return
    # one cheap request with a throwaway dummy tool (never executed)
    resp = await model.get_response(tools=[dummy], ...)
    if any(isinstance(item, ResponseFunctionToolCall) for item in resp.output):
        return                       # capable → continue
    raise ToolCallingUnsupportedError(_GUIDANCE)   # text-only/empty → abort

Behavior:

• Structured tool call → pass.
• Text-only / empty (the [BUG] Tool calls returned as plain text instead of being executed when using Ollama backend #520 leak) → abort with guidance (--jinja, tool-templated Ollama model + disable thinking, vLLM --tool-call-parser, ...).
• Provider error mentioning tool config (e.g. tools param requires --jinja flag) → abort immediately with the same guidance.
• Unrelated transient error → retried (2 attempts) before surfacing.

Gating keeps hosted providers (OpenAI/Anthropic/... with no custom api_base) untouched — no extra latency or cost — and STRIX_SKIP_TOOL_CALL_PROBE=1 is an escape hatch.

Verified live against a local llama-server (jinja default → passes), --no-jinja (aborts), and llama-cpp-python with no tool-aware format (the exact #520 text-leak → aborts).

Changes

• strix/core/warmup.py — new preflight probe + ToolCallingUnsupportedError.
• strix/core/runner.py — invoke probe_tool_calling before sandbox bringup (non-resume).
• strix/config/settings.py — add STRIX_SKIP_TOOL_CALL_PROBE.
• docs/llm-providers/local.mdx — "Tool calling must return structured tool_calls" section with a per-server fix matrix (llama.cpp / Ollama / vLLM).
• tests/test_warmup_probe.py — deterministic tests (gating, structured-pass, text-only abort, retry, tool-config-error mapping, hosted skip, setting skip). Runner test stubs updated for the new settings fields.

Link to Devin session: https://app.devin.ai/sessions/f74044feadc04f02b17d2ccfdc4df7d7
Requested by: @0xallam

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

Adds a preflight check that verifies structured LLM tool-call support before fresh scans.

• Introduces the probe, capability-specific error guidance, retry behavior, and self-hosted-route gating.
• Adds an environment setting to bypass the probe and invokes it before sandbox provisioning.
• Documents local inference-server tool-call configuration and adds probe and runner tests.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking diagnostics issue when the probe encounters repeated network, authentication, or provider failures.

The capability probe is correctly scoped and integrated, but its final error branch reports every repeated unrelated request failure as unsupported tool calling and directs users toward irrelevant server configuration changes.

Files Needing Attention: strix/core/warmup.py

Important Files Changed

Filename	Overview
strix/core/warmup.py	Implements the capability probe and retries, but misclassifies repeated unrelated provider errors as unsupported tool calling.
strix/core/runner.py	Runs the probe for fresh scans before sandbox provisioning while preserving resume behavior.
strix/config/settings.py	Adds the aliased STRIX_SKIP_TOOL_CALL_PROBE escape-hatch setting.
tests/test_warmup_probe.py	Covers gating and probe outcomes, including codifying the misleading wrapping of unrelated failures.
docs/llm-providers/local.mdx	Documents structured tool-call requirements and configuration guidance for common local servers.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
strix/core/warmup.py:185-189
**Unrelated failures get capability guidance**

After both probe attempts fail with a network, authentication, or provider error, this branch replaces that failure with `ToolCallingUnsupportedError`. The primary CLI message therefore directs users to change Jinja templates or tool parsers instead of presenting the relevant provider remediation.

_{Reviews (1): Last reviewed commit: "fix(runtime): fail fast when the LLM end..." | Re-trigger Greptile}

[greptile-apps]

Unrelated failures get capability guidance

After both probe attempts fail with a network, authentication, or provider error, this branch replaces that failure with ToolCallingUnsupportedError. The primary CLI message therefore directs users to change Jinja templates or tool parsers instead of presenting the relevant provider remediation.

Prompt To Fix With AI

This is a comment left during a code review.
Path: strix/core/warmup.py
Line: 185-189

Comment:
**Unrelated failures get capability guidance**

After both probe attempts fail with a network, authentication, or provider error, this branch replaces that failure with `ToolCallingUnsupportedError`. The primary CLI message therefore directs users to change Jinja templates or tool parsers instead of presenting the relevant provider remediation.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!