gateway: fix funnel-watchdog to probe real public path (b66, interim)

[baron-3dl]

Root cause (enterpriseaiframework-b66)

Camp portal timeouts + long glm-5.2 stream deaths trace to the gateway VM's CGNAT uplink: a middlebox on its path (the on-prem MiniUPnPd/Debian-wheezy router at 192.168.2.1, and/or the carrier CGNAT) flushes long-lived TCP state on a fixed ~34-min timer, regardless of traffic.

Evidence gathered 2026-08-11:

• Gateway tailscaled control+DERP reset every 33–36 min for 6+ h — a metronome, not random NAT churn.
• Silent blackhole: a 40-min tcpdump on :443 caught zero RSTs; the dying flow just had bytes stuck in Send-Q. NAT-state teardown, not an active reset.
• Flows carry packets every 2–13s yet still die on the tick → keepalives cannot prevent a scheduled flush.
• workshop (same LAN, same CGNAT egress, same router, same UPnP/PMP/PCP, same NIC offloads, same Tailscale version) had 0 such resets in the same window → gateway-node-specific, not fixable by copying host config.

Each flush drops the public Funnel path. tailscaled usually self-heals in 1–2 min but occasionally wedges. The prior watchdog checked tailscale status … offline — which the fast self-heal almost never trips — so it fired 0× in 12 h while the portal was actually down.

This change (interim)

Tracked, corrected watchdog under deploy/gateway/funnel-watchdog/. Probes the real public Funnel path every 30s; bounces tailscale only after ~90s sustained outage (debounced past normal self-heal), 120s cooldown to avoid thrash.

Verified end-to-end: injected tailscale down, watchdog auto-recovered the public portal in 107s. (Old watchdog would not have fired at all.)

Not the durable fix

Does not eliminate the ~34-min blips and cannot save an in-flight stream. It is a safety net for genuine wedges.

Durable fix — operator decision (raised on b66)

The CGNAT uplink cannot hold a connection past ~34 min, so no gateway-local change makes the public path reliable:

1. Fix/replace the on-prem router (MiniUPnPd on Debian wheezy, EOL 2016). Rebooting it is a free first test.
2. Move public ingress off the CGNAT path (Cloudflare Tunnel / public-IP reverse proxy). Most resilient, but changes the load-bearing OIDC hostname gateway.tailcb6ef9.ts.net → coordinated auth reconfig + CF account/domain (spend/scope).

🤖 Generated with Claude Code