gateway: fix funnel-watchdog to probe real public path (b66, interim)

deploy/gateway/funnel-watchdog/README.md

@@ -0,0 +1,67 @@
+# funnel-watchdog (gateway VM)
+
+**Interim mitigation for `enterpriseaiframework-b66`. Not the durable fix.**
+
+## What breaks
+
+Off-tailnet users (the camp) reach the portal only through Tailscale Funnel on the
+gateway VM. The gateway is behind CGNAT, and a middlebox on its uplink path flushes
+long-lived TCP connection state on a **fixed ~34-minute timer, regardless of traffic**.
+
+Evidence gathered on 2026-08-11 (`enterpriseaiframework-b66`):
+
+- Gateway `tailscaled` control + DERP connections reset like clockwork every 33–36 min
+  for 6+ hours straight. A metronome, not random NAT churn.
+- The reset is a **silent blackhole**: a 40-min `tcpdump` on `:443` caught **zero** RST
+  packets; the dying flow just had bytes stuck in `Send-Q`. Classic NAT-state teardown.
+- The flows are **not idle** — packets flow every 2–13s — yet they still die on the
+  tick. So keepalives cannot prevent it; the middlebox flushes on a schedule.
+- **workshop** (same LAN, same CGNAT egress IP, same MiniUPnPd router, same UPnP/PMP/PCP
+  portmapping, same virtio NIC offloads, same Tailscale version) had **0** such resets in
+  the same window. The fault is specific to the gateway node's role/path, not the uplink
+  as a whole and not any host config that can be copied over.
+- The on-prem router at `192.168.2.1` reports `MiniUPnPd/2.1` on **Debian "wheezy"**
+  (EOL 2016) — a prime suspect for an aggressive conntrack flush.
+
+Each flush drops the gateway's Funnel path; `tailscaled` usually self-heals in 1–2 min
+but occasionally wedges and needs a manual `tailscale down; up`. Any long HTTP stream
+(e.g. a 3-min glm-5.2 turn) that is in flight when the tick lands is killed.
+
+## What this watchdog does
+
+Probes the **real public Funnel path** (`https://gateway.tailcb6ef9.ts.net/portal/` via
+the Tailscale ingress IPs) every 30s. After ~90s of sustained public-path outage
+(3 consecutive failures) it runs `tailscale down; up`, then holds a 120s cooldown so it
+does not re-bounce into tailscaled's own recovery.
+
+This replaces the previous watchdog, which checked `tailscale status … offline` — a
+signal the fast self-heal almost never trips, so it fired **0 times in 12 h** while the
+portal was actually down.
+
+### What it does NOT do
+
+- It does not eliminate the ~34-min blips (each flush still drops in-flight connections).
+- It cannot save a long stream that is mid-flight when a flush lands.
+- It is a safety net for genuine wedges, not a cure.
+
+## The durable fix (operator decision)
+
+The gateway's CGNAT uplink cannot hold a connection past ~34 min, so **no gateway-local
+change makes the public path reliable.** Options, cheapest first:
+
+1. **Fix/replace the on-prem router** (`192.168.2.1`, MiniUPnPd on Debian wheezy).
+   Reboot it as a free first test; if the ~34-min flush stops, it was the router. Raising
+   its conntrack timeouts or replacing the box may fully resolve it at zero recurring cost.
+2. **Move public ingress off the CGNAT path** — a Cloudflare Tunnel (or a reverse proxy
+   on a public-IP host). Most resilient, but changes the public hostname, which is the
+   load-bearing OIDC issuer (`gateway.tailcb6ef9.ts.net` is baked into Keycloak,
+   oauth2-proxy, and pod `hostAliases`), so it requires a coordinated auth reconfiguration,
+   plus a Cloudflare account + domain (spend/accounts = operator scope).
+
+## Install / refresh
+
+```
+ssh baron@gateway 'sudo bash -s' < install.sh
+# or, on the gateway, from this directory:
+sudo ./install.sh
+```

deploy/gateway/funnel-watchdog/funnel-watchdog

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# funnel-watchdog — recover the public Tailscale Funnel path when it wedges.
+#
+# INTERIM MITIGATION, NOT THE DURABLE FIX. See enterpriseaiframework-b66.
+#
+# Root cause: the gateway VM sits behind CGNAT, and a middlebox on its uplink path
+# (the on-prem router at 192.168.2.1 runs MiniUPnPd on Debian "wheezy", and/or the
+# carrier CGNAT) flushes long-lived TCP connection state on a fixed ~34-minute timer,
+# REGARDLESS of traffic — measured: the gateway's Tailscale control+DERP flows carry
+# packets every 2-13s yet still die, silently (no RST/FIN, bytes stuck in Send-Q), on
+# the tick. Each flush drops the gateway's Tailscale control+DERP connections, so the
+# public Funnel path (off-tailnet client -> Tailscale ingress -> gateway DERP -> Caddy)
+# goes dead. tailscaled usually self-heals in 1-2 min; it occasionally wedges and needs
+# a `tailscale down; up`. Keepalives cannot beat a scheduled flush — the ONLY durable
+# fix is to move public ingress off this CGNAT path (operator decision: replace the
+# ancient router, or a public-IP/Cloudflare-Tunnel ingress with the auth-hostname
+# reconfiguration that implies). This script only bounds the damage until then.
+#
+# The previous version checked `tailscale status ... offline`, which the fast self-heal
+# almost never trips, so it fired 0x in 12h while the portal was actually down. This
+# version probes the REAL public Funnel path and only bounces after a sustained outage
+# (debounced past normal self-heal), so it recovers genuine wedges without fighting
+# tailscaled's own recovery or thrashing.
+
+set -u
+
+HOST="gateway.tailcb6ef9.ts.net"
+# Public DNS A records for the Tailscale Funnel ingress (gateway.tailcb6ef9.ts.net).
+INGRESS_IPS=(199.38.181.54 209.177.145.137)
+PROBE_PATH="/portal/"
+PROBE_TIMEOUT=8          # seconds per curl
+FAIL_THRESHOLD=3         # consecutive failed runs before bouncing
+COOLDOWN_SECS=120        # after a bounce, wait this long before acting again
+
+# Timer fires every 30s, so FAIL_THRESHOLD=3 => ~90s of sustained public-path outage
+# before a bounce. Normal tailscaled self-heal (1-2 min) usually recovers before that,
+# so this only fires on genuine wedges.
+
+FAILS_FILE=/run/funnel-watchdog.fails
+COOLDOWN_FILE=/run/funnel-watchdog.cooldown
+
+now=$(date +%s)
+
+# Respect cooldown after a recent bounce (tailscale needs ~15-30s to re-establish
+# Funnel; do not re-bounce into that window).
+if [[ -f "$COOLDOWN_FILE" ]]; then
+    until=$(cat "$COOLDOWN_FILE" 2>/dev/null || echo 0)
+    if [[ "$now" -lt "$until" ]]; then
+        exit 0
+    fi
+fi
+
+probe() {
+    # Reachable end-to-end over the PUBLIC path if any ingress IP answers the funnel
+    # host with an HTTP status (any status = the request reached Caddy through the
+    # gateway's DERP connection; connection failure/timeout yields 000).
+    local ip code
+    for ip in "${INGRESS_IPS[@]}"; do
+        code=$(curl -s -o /dev/null -w '%{http_code}' --max-time "$PROBE_TIMEOUT" \
+            --resolve "${HOST}:443:${ip}" "https://${HOST}${PROBE_PATH}" 2>/dev/null)
+        case "$code" in
+            2??|3??|401|403) return 0 ;;
+        esac
+    done
+    return 1
+}
+
+if probe; then
+    echo 0 > "$FAILS_FILE"
+    exit 0
+fi
+
+# Probe failed: increment the consecutive-failure counter.
+fails=$(cat "$FAILS_FILE" 2>/dev/null || echo 0)
+[[ "$fails" =~ ^[0-9]+$ ]] || fails=0
+fails=$((fails + 1))
+echo "$fails" > "$FAILS_FILE"
+
+if [[ "$fails" -ge "$FAIL_THRESHOLD" ]]; then
+    logger -t funnel-watchdog "public funnel unreachable for ~$((FAIL_THRESHOLD * 30))s -> bouncing tailscale (see enterpriseaiframework-b66)"
+    tailscale down || true
+    sleep 3
+    tailscale up || true
+    echo 0 > "$FAILS_FILE"
+    echo $((now + COOLDOWN_SECS)) > "$COOLDOWN_FILE"
+fi

deploy/gateway/funnel-watchdog/funnel-watchdog.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=Recover the public Tailscale Funnel path if it wedges (interim; see enterpriseaiframework-b66)
+After=tailscaled.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/funnel-watchdog

deploy/gateway/funnel-watchdog/funnel-watchdog.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Probe the public Tailscale Funnel path every 30s
+
+[Timer]
+OnBootSec=30
+OnUnitActiveSec=30
+AccuracySec=5s
+
+[Install]
+WantedBy=timers.target

deploy/gateway/funnel-watchdog/install.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Install/refresh the funnel-watchdog on the gateway VM. Idempotent.
+# Run ON the gateway (or via: ssh baron@gateway 'sudo bash -s' < install.sh).
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+install -m 0755 "$HERE/funnel-watchdog"          /usr/local/bin/funnel-watchdog
+install -m 0644 "$HERE/funnel-watchdog.service"  /etc/systemd/system/funnel-watchdog.service
+install -m 0644 "$HERE/funnel-watchdog.timer"    /etc/systemd/system/funnel-watchdog.timer
+
+systemctl daemon-reload
+systemctl enable --now funnel-watchdog.timer
+systemctl restart funnel-watchdog.timer
+
+echo "installed. timer status:"
+systemctl status funnel-watchdog.timer --no-pager | tail -4