Skip to content

Gateway approvals are lost on restart for chat channels — chat backends never use the durable ApprovalStore #3183

Description

@MervinPraison

Summary

When an Agent requires human approval for a tool call over a chat channel (Slack, Telegram, Discord, Webhook, HTTP), the pending approval is held only in an in-memory coroutine that polls the platform for a reply. If the gateway/bot process restarts (deploy, crash, scale-to-zero wake) while an approval is outstanding, the pending request is lost: the agent run that was blocked on it is stranded, and the human's later "yes/no" resolves nothing.

Human-in-the-loop approval is a core gateway safety feature, so losing it across a restart directly undermines the "robust, production-ready, safe-by-default" goal for the gateway/bot. A durable, resolution-safe ApprovalStore already exists in-tree and the presentation/secure backend already persists + rehydrates — but the chat backends never use it, and behaviour diverges across backends (persistence, actor authorisation, timeout, and an LLM free-text classifier that can mutate the decision).

Current behaviour

Chat approval backends keep all state in a per-call coroutine + polling loop, with no persistence:

  • src/praisonai-bot/praisonai_bot/bots/_slack_approval.py — polls conversations.replies; no store.
  • src/praisonai-bot/praisonai_bot/bots/_telegram_approval.py — inline keyboard + getUpdates poll; no store.
  • src/praisonai-bot/praisonai_bot/bots/_discord_approval.py, .../_webhook_approval.py, .../_http_approval.py — same in-memory pattern.

Verified: none of _slack_approval.py, _discord_approval.py, _webhook_approval.py, _http_approval.py reference ApprovalStore, persist, rehydrate, or sqlite — the outstanding approval exists only for the lifetime of the process.

The durable machinery exists but is wired only to the presentation/secure + opt-in gateway paths:

  • src/praisonai-bot/praisonai_bot/bots/_approval_store.py — SQLite+WAL pending_approvals, refuses to overwrite a resolved decision, idempotent re-persist.
  • src/praisonai-bot/praisonai_bot/bots/_presentation_approval_backend.py — persists, exposes rehydrate(), uses a typed presentation action, and enforces an actor allowlist.
  • src/praisonai-bot/praisonai_bot/gateway/exec_approval.py — durable "allow-always" grants + rehydrate(), but only when PRAISONAI_GATEWAY_DURABLE_APPROVALS is set.

Divergence across backends compounds the problem:

  • _http_approval.py serves an approve/deny endpoint with no authentication or allowlist — any caller reaching host:port decides, and approver is caller-supplied.
  • Default timeouts are inconsistent: gateway backend 120s, chat backends 300s, presentation handler 60s — no single source of truth for the HITL wait window.
  • The three chat backends run non-keyword replies through an LLM free-text classifier that can flip deny → approve and inject modified_args; the durable presentation path deliberately does not.

Desired behaviour

All approval backends resolve through the durable ApprovalStore by default, so an outstanding approval survives a restart and is rehydrated on startup; a decision that arrives after the restart resolves the original request (respecting the original deadline). Behaviour is consistent across channels: durable persistence, actor authorisation required by default (fail-closed), one default timeout, and reliance on typed approval actions rather than per-backend free-text polling.

Layer placement

  • Primary layer: wrapper (src/praisonai-bot/praisonai_bot/bots/*_approval*.py, the channel approval backends and their construction).
  • Why not core: the approval contract already lives in core (praisonaiagents.approval.protocols.ApprovalProtocol) and needs no change; the missing behaviour is that the channel-transport backends do not use the durable store — that is wrapper/bot-tier wiring.
  • Why not wrapper (CLI/YAML) alone: the durable resolution is a runtime behaviour of the backends themselves, not just a config surface; the CLI/YAML change is only backend selection (secondary).
  • Why not tools: approvals are not agent-callable integrations; they are a lifecycle/transport concern.
  • Why not plugins: this is transport delivery + persistence of an approval prompt, not a cross-cutting lifecycle guardrail policy.
  • Secondary touch (optional): core already provides ApprovalProtocol; expose backend selection consistently on the CLI (--approval <name>) and in gateway YAML.
  • 3-way surface (CLI + YAML + Python): yes — approval backend selection should be equally reachable from Python (Agent(approval=...)), YAML (gateway.yaml), and the bot/gateway CLI.

Proposed approach

  • Extension point: the existing ApprovalStore + ApprovalProtocol; make durability the default construction path for every channel backend.
  • Minimal API sketch:
# A shared base that every channel backend uses, so persistence + rehydrate
# + actor auth are uniform rather than per-backend.
class DurableChannelApproval:
    def __init__(self, *, store: ApprovalStore, allowed_actors: set[str],
                 timeout: float = DEFAULT_APPROVAL_TIMEOUT):
        ...

    async def request_approval(self, request) -> ApprovalDecision:
        approval_id = await self.store.persist(request)      # survives restart
        ...                                                  # deliver typed action

    async def rehydrate(self) -> None:                       # called on startup
        for pending in await self.store.list_pending():
            self._resume_waiting(pending)

Resolution sketch

# Before (today): Slack/Discord/Webhook/HTTP hold the pending approval only in
# a coroutine; a restart mid-approval strands the blocked agent run.
approval = SlackApproval(channel="#approvals")          # no persistence
agent = Agent(name="ops", tools=[deploy], approval=approval)
# process restarts while awaiting -> pending request gone, run never resumes

# After (proposed): every channel backend persists + rehydrates by default.
store = ApprovalStore(path="~/.praisonai/state/approvals.sqlite")
approval = SlackApproval(channel="#approvals", store=store,
                         allowed_actors={"U123"})       # durable + actor auth
agent = Agent(name="ops", tools=[deploy], approval=approval)
# on startup: await approval.rehydrate()  -> outstanding approvals resume

Severity

High — a gateway restart during an outstanding approval strands the blocked agent run, and the no-auth HTTP endpoint plus decision-mutating LLM classifier are safety concerns on a human-in-the-loop path.

Validation

  • Traced each chat backend's polling/in-memory resolution: _slack_approval.py, _telegram_approval.py, _discord_approval.py, _webhook_approval.py, _http_approval.py — confirmed none import/use ApprovalStore or any persist/rehydrate/sqlite path.
  • Confirmed the durable store and a durable backend already exist and rehydrate: bots/_approval_store.py, bots/_presentation_approval_backend.py, gateway/exec_approval.py (durability gated behind PRAISONAI_GATEWAY_DURABLE_APPROVALS).
  • Confirmed the missing-auth endpoint and divergent timeouts/LLM-classifier paths in _http_approval.py and _approval_base.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaudeAuto-trigger Claude analysisenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions