diff --git a/deploy/gateway/funnel-watchdog/README.md b/deploy/gateway/funnel-watchdog/README.md new file mode 100644 index 0000000..dd5916a --- /dev/null +++ b/deploy/gateway/funnel-watchdog/README.md @@ -0,0 +1,67 @@ +# funnel-watchdog (gateway VM) + +**Interim mitigation for `enterpriseaiframework-b66`. Not the durable fix.** + +## What breaks + +Off-tailnet users (the camp) reach the portal only through Tailscale Funnel on the +gateway VM. The gateway is behind CGNAT, and a middlebox on its uplink path flushes +long-lived TCP connection state on a **fixed ~34-minute timer, regardless of traffic**. + +Evidence gathered on 2026-08-11 (`enterpriseaiframework-b66`): + +- Gateway `tailscaled` control + DERP connections reset like clockwork every 33–36 min + for 6+ hours straight. A metronome, not random NAT churn. +- The reset is a **silent blackhole**: a 40-min `tcpdump` on `:443` caught **zero** RST + packets; the dying flow just had bytes stuck in `Send-Q`. Classic NAT-state teardown. +- The flows are **not idle** — packets flow every 2–13s — yet they still die on the + tick. So keepalives cannot prevent it; the middlebox flushes on a schedule. +- **workshop** (same LAN, same CGNAT egress IP, same MiniUPnPd router, same UPnP/PMP/PCP + portmapping, same virtio NIC offloads, same Tailscale version) had **0** such resets in + the same window. The fault is specific to the gateway node's role/path, not the uplink + as a whole and not any host config that can be copied over. +- The on-prem router at `192.168.2.1` reports `MiniUPnPd/2.1` on **Debian "wheezy"** + (EOL 2016) — a prime suspect for an aggressive conntrack flush. + +Each flush drops the gateway's Funnel path; `tailscaled` usually self-heals in 1–2 min +but occasionally wedges and needs a manual `tailscale down; up`. Any long HTTP stream +(e.g. a 3-min glm-5.2 turn) that is in flight when the tick lands is killed. + +## What this watchdog does + +Probes the **real public Funnel path** (`https://gateway.tailcb6ef9.ts.net/portal/` via +the Tailscale ingress IPs) every 30s. After ~90s of sustained public-path outage +(3 consecutive failures) it runs `tailscale down; up`, then holds a 120s cooldown so it +does not re-bounce into tailscaled's own recovery. + +This replaces the previous watchdog, which checked `tailscale status … offline` — a +signal the fast self-heal almost never trips, so it fired **0 times in 12 h** while the +portal was actually down. + +### What it does NOT do + +- It does not eliminate the ~34-min blips (each flush still drops in-flight connections). +- It cannot save a long stream that is mid-flight when a flush lands. +- It is a safety net for genuine wedges, not a cure. + +## The durable fix (operator decision) + +The gateway's CGNAT uplink cannot hold a connection past ~34 min, so **no gateway-local +change makes the public path reliable.** Options, cheapest first: + +1. **Fix/replace the on-prem router** (`192.168.2.1`, MiniUPnPd on Debian wheezy). + Reboot it as a free first test; if the ~34-min flush stops, it was the router. Raising + its conntrack timeouts or replacing the box may fully resolve it at zero recurring cost. +2. **Move public ingress off the CGNAT path** — a Cloudflare Tunnel (or a reverse proxy + on a public-IP host). Most resilient, but changes the public hostname, which is the + load-bearing OIDC issuer (`gateway.tailcb6ef9.ts.net` is baked into Keycloak, + oauth2-proxy, and pod `hostAliases`), so it requires a coordinated auth reconfiguration, + plus a Cloudflare account + domain (spend/accounts = operator scope). + +## Install / refresh + +``` +ssh baron@gateway 'sudo bash -s' < install.sh +# or, on the gateway, from this directory: +sudo ./install.sh +``` diff --git a/deploy/gateway/funnel-watchdog/funnel-watchdog b/deploy/gateway/funnel-watchdog/funnel-watchdog new file mode 100644 index 0000000..50f5913 --- /dev/null +++ b/deploy/gateway/funnel-watchdog/funnel-watchdog @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# funnel-watchdog — recover the public Tailscale Funnel path when it wedges. +# +# INTERIM MITIGATION, NOT THE DURABLE FIX. See enterpriseaiframework-b66. +# +# Root cause: the gateway VM sits behind CGNAT, and a middlebox on its uplink path +# (the on-prem router at 192.168.2.1 runs MiniUPnPd on Debian "wheezy", and/or the +# carrier CGNAT) flushes long-lived TCP connection state on a fixed ~34-minute timer, +# REGARDLESS of traffic — measured: the gateway's Tailscale control+DERP flows carry +# packets every 2-13s yet still die, silently (no RST/FIN, bytes stuck in Send-Q), on +# the tick. Each flush drops the gateway's Tailscale control+DERP connections, so the +# public Funnel path (off-tailnet client -> Tailscale ingress -> gateway DERP -> Caddy) +# goes dead. tailscaled usually self-heals in 1-2 min; it occasionally wedges and needs +# a `tailscale down; up`. Keepalives cannot beat a scheduled flush — the ONLY durable +# fix is to move public ingress off this CGNAT path (operator decision: replace the +# ancient router, or a public-IP/Cloudflare-Tunnel ingress with the auth-hostname +# reconfiguration that implies). This script only bounds the damage until then. +# +# The previous version checked `tailscale status ... offline`, which the fast self-heal +# almost never trips, so it fired 0x in 12h while the portal was actually down. This +# version probes the REAL public Funnel path and only bounces after a sustained outage +# (debounced past normal self-heal), so it recovers genuine wedges without fighting +# tailscaled's own recovery or thrashing. + +set -u + +HOST="gateway.tailcb6ef9.ts.net" +# Public DNS A records for the Tailscale Funnel ingress (gateway.tailcb6ef9.ts.net). +INGRESS_IPS=(199.38.181.54 209.177.145.137) +PROBE_PATH="/portal/" +PROBE_TIMEOUT=8 # seconds per curl +FAIL_THRESHOLD=3 # consecutive failed runs before bouncing +COOLDOWN_SECS=120 # after a bounce, wait this long before acting again + +# Timer fires every 30s, so FAIL_THRESHOLD=3 => ~90s of sustained public-path outage +# before a bounce. Normal tailscaled self-heal (1-2 min) usually recovers before that, +# so this only fires on genuine wedges. + +FAILS_FILE=/run/funnel-watchdog.fails +COOLDOWN_FILE=/run/funnel-watchdog.cooldown + +now=$(date +%s) + +# Respect cooldown after a recent bounce (tailscale needs ~15-30s to re-establish +# Funnel; do not re-bounce into that window). +if [[ -f "$COOLDOWN_FILE" ]]; then + until=$(cat "$COOLDOWN_FILE" 2>/dev/null || echo 0) + if [[ "$now" -lt "$until" ]]; then + exit 0 + fi +fi + +probe() { + # Reachable end-to-end over the PUBLIC path if any ingress IP answers the funnel + # host with an HTTP status (any status = the request reached Caddy through the + # gateway's DERP connection; connection failure/timeout yields 000). + local ip code + for ip in "${INGRESS_IPS[@]}"; do + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time "$PROBE_TIMEOUT" \ + --resolve "${HOST}:443:${ip}" "https://${HOST}${PROBE_PATH}" 2>/dev/null) + case "$code" in + 2??|3??|401|403) return 0 ;; + esac + done + return 1 +} + +if probe; then + echo 0 > "$FAILS_FILE" + exit 0 +fi + +# Probe failed: increment the consecutive-failure counter. +fails=$(cat "$FAILS_FILE" 2>/dev/null || echo 0) +[[ "$fails" =~ ^[0-9]+$ ]] || fails=0 +fails=$((fails + 1)) +echo "$fails" > "$FAILS_FILE" + +if [[ "$fails" -ge "$FAIL_THRESHOLD" ]]; then + logger -t funnel-watchdog "public funnel unreachable for ~$((FAIL_THRESHOLD * 30))s -> bouncing tailscale (see enterpriseaiframework-b66)" + tailscale down || true + sleep 3 + tailscale up || true + echo 0 > "$FAILS_FILE" + echo $((now + COOLDOWN_SECS)) > "$COOLDOWN_FILE" +fi diff --git a/deploy/gateway/funnel-watchdog/funnel-watchdog.service b/deploy/gateway/funnel-watchdog/funnel-watchdog.service new file mode 100644 index 0000000..c6994bc --- /dev/null +++ b/deploy/gateway/funnel-watchdog/funnel-watchdog.service @@ -0,0 +1,7 @@ +[Unit] +Description=Recover the public Tailscale Funnel path if it wedges (interim; see enterpriseaiframework-b66) +After=tailscaled.service + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/funnel-watchdog diff --git a/deploy/gateway/funnel-watchdog/funnel-watchdog.timer b/deploy/gateway/funnel-watchdog/funnel-watchdog.timer new file mode 100644 index 0000000..13e7a23 --- /dev/null +++ b/deploy/gateway/funnel-watchdog/funnel-watchdog.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Probe the public Tailscale Funnel path every 30s + +[Timer] +OnBootSec=30 +OnUnitActiveSec=30 +AccuracySec=5s + +[Install] +WantedBy=timers.target diff --git a/deploy/gateway/funnel-watchdog/install.sh b/deploy/gateway/funnel-watchdog/install.sh new file mode 100644 index 0000000..e8a6ef9 --- /dev/null +++ b/deploy/gateway/funnel-watchdog/install.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Install/refresh the funnel-watchdog on the gateway VM. Idempotent. +# Run ON the gateway (or via: ssh baron@gateway 'sudo bash -s' < install.sh). +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +install -m 0755 "$HERE/funnel-watchdog" /usr/local/bin/funnel-watchdog +install -m 0644 "$HERE/funnel-watchdog.service" /etc/systemd/system/funnel-watchdog.service +install -m 0644 "$HERE/funnel-watchdog.timer" /etc/systemd/system/funnel-watchdog.timer + +systemctl daemon-reload +systemctl enable --now funnel-watchdog.timer +systemctl restart funnel-watchdog.timer + +echo "installed. timer status:" +systemctl status funnel-watchdog.timer --no-pager | tail -4