docs: staged strategy for scalable HUF agent runs

[esafwan]

Summary

This PR documents a staged, non-breaking strategy to improve HUF agent run scalability. The central change is moving long agent turns away from the web request path and toward queue-first execution, while keeping smaller deployments simple and leaving room for a detached external runner at higher scale.

Why this matters

HUF is not a short-request app. Agent runs can involve long LLM calls, multiple tool calls, and several audit writes across runs, messages, conversations, and tool-call records. If all of that stays on the web request path, concurrent usage can tie up Frappe workers, increase latency, and make the site fragile under bursty load.

Proposed path

1. Keep the current direct execution path available for small or trusted deployments.
2. Make queued execution the default for agent turns, with an explicit now=true escape hatch for trusted immediate execution.
3. Separate execution from request handling so the HTTP layer only creates or resumes a run and returns quickly.
4. Preserve interactive streaming by moving stream delivery to run events instead of tying it to a long-lived web request.
5. Run queued jobs immediately on available workers so capacity is used as soon as it exists.
6. Keep the architecture open for larger production workloads to move to a detached external runner when queue-based execution is no longer enough.

Scale strategies

This is intentionally not a one-size-fits-all model.

• Standard stack: direct execution remains useful for small teams, local installs, trusted admin workflows, and low-concurrency deployments.
• Queue-first stack: agent turns run through Frappe background workers by default, giving 10-20 concurrent users a safer path without requiring a separate runner service.
• Detached runner stack: high-volume B2C or externally embedded workloads can move execution outside the Frappe web and queue workers, using a runner model such as Temporal for durable orchestration.

Streaming behavior

Streaming should remain available, but the transport changes by execution mode.

• Direct mode can continue using the current request-bound streaming path.
• Queued mode should return a run_id quickly, then stream tokens, tool-call updates, status changes, and final output through HUF-owned run events such as Socket.IO or SSE.
• Detached runner mode should publish workflow progress back into HUF run events, so clients consume the same stream contract whether execution happens inside Frappe workers or an external runner.

This keeps chat responsive without holding a Frappe web worker for the full LLM/tool loop.

External runner role

The detached runner is a later scaling option, not the center of this PR. Temporal is a strong candidate for that layer because it provides durable workflows, retries, cancellation, and long-running orchestration outside the web process. The immediate focus remains queue-first execution inside the standard HUF/Frappe deployment path.

Non-breaking principles

• Do not change the current sync path in a way that surprises existing callers.
• Keep the direct path available as an opt-in escape hatch.
• Introduce new run lifecycle APIs alongside the old ones before deprecating anything.
• Keep streaming as a supported interaction model while changing the backend execution path.
• Make queue failure explicit instead of silently falling back to a different mode.
• Preserve auditability for runs, messages, tool calls, and conversation state.

What this PR is

Documentation only. No runtime code, schema migration, or behavior change is included here.