From 5655cc74654bb5b36bf35142eec44b4fc2df34b9 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:06:50 -0500 Subject: [PATCH 01/10] Spec: two-host production, and heavy-model modes on top of it Kevin's chosen standing configuration, from three sessions converging tonight: normal production bots on crow, Qwen3.8-Flash-Next at 1M on raven as normal production, and the option to evict production on both boxes to run either GLM-5.3-Flash or DSv4-Flash two-box. Docs only. No code, no catalog edits. Implementation ownership stays with the crow side for the format, orchestrator routing and catalog; the two harness items are pi-lab's. Why this shape works, with the measured numbers: Flash-Next Q4_K_XL is the only one of the three heavies whose weights leave room for a 1M KV on a single box (about 106 GiB of 124, 17.7 GiB free on raven), and single-box on raven beats the two-box split on both prefill and decode, so the 1M model needs one box and crow stays untouched. GLM IQ4_XS at 146 GiB and DSv4 IQ4_XS at 128 GiB exceed one box outright, so they are structurally two-box and cannot join the standard state; they get their own mode with both hosts evicted. The design also sidesteps the harness's biggest limitation. A persistent raven service needs nothing to start it on demand, and the harness can already consume a remote endpoint over the LAN even though it cannot lifecycle one. Carries pi-lab's five window properties verbatim as requirements, plus its veto that a curated catalog must never offer a known-wedge shape as a one-tap option, and crow-34's sequencing: serving.class first, then gates, then build, then topology. Records three open decisions for Kevin: retiring the crow-dsv4 provider row, the priority of the two harness changes against the catalog work, and 524k as a curated rung. --- ...o-host-production-and-heavy-model-modes.md | 224 ++++++++++++++++++ 1 file changed, 224 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md diff --git a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md new file mode 100644 index 00000000..6457bd7d --- /dev/null +++ b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md @@ -0,0 +1,224 @@ +# Spec: two-host production, and heavy-model modes on top of it + +Status: draft for review. Author: `~/r4-tehcy` session, 2026-09-09 evening, from Kevin's direction across +three sessions (`~/r4-tehcy`, `pi-lab-36`, `crow-34`). +Implementation owner: crow (format, orchestrator routing, catalog). Harness changes: pi-lab. First consumer: R4. + +## 1. The configuration Kevin chose + +Verbatim: run the normal production bots on crow, run Qwen3.8-Flash-Next at 1M on raven as normal production, +with the option to evict production on both boxes to run either GLM or DSv4. That is the standard configuration. + +So there are three states, and only three: + +| state | crow | raven | +|---|---|---| +| **standard** | prod bots (35b, vLLM 4b, embed, gemma) | Flash-Next Q4_K_XL @1M | +| **heavy: GLM** | evicted | evicted, GLM-5.3-Flash two-box master | +| **heavy: DSv4** | evicted, RPC worker | evicted, DSv4-Flash two-box master | + +The standard state is the important one: both a fast runner and a 1M-context model are available at the same +time, because they sit on different boxes. No eviction, no RPC, no shared GPU. + +## 2. Why this works, with the numbers + +Per-box GTT is 124 GiB on both machines. Measured, from +`docs/research/2026-08-30-second-strix-halo-box-plan.md` and the serving doc +`~/pi-lab/docs/research/2026-09-09-heavy-model-serving-configs.md`: + +- **Flash-Next Q4_K_XL fits 1M on one box.** Body 76.8 GiB plus 26.9 GiB lazy PLE, about 106 GiB resident at 1M, + leaving **17.7 GiB free on raven**. Rungs measured on raven: 262k leaves 31.0 GiB free, 524k 23.4, 1M 17.7. +- **Single-box beats two-box for this model.** Raven single-box at 120 W measures 354 / 24.7 short and + 234 / 18.4 at depth, ahead of the two-box split on both prefill and decode (§7.16). Two-box earns its keep only + as headroom for the Q5 quant or to host the MTP head on the master. So the 1M production config needs one box, + which is what leaves crow untouched. +- **Flash-Next single-box has never wedged.** All three GPU wedges were DSv4 two-box at 86 to 91 GiB per box, + confirmed against CROW-SCHEDULE's own incident rows, which name DSv4 and dspark and never Flash-Next or GLM. +- **Crow prod is about 60.5 GiB of GTT today**, of which the vLLM 4b is 15.2 GiB by KFD accounting and the 35b is + roughly 43 GiB by subtraction. The Vulkan containers do not appear in ROCm accounting, so the 35b figure is + bounded rather than isolated. It is good enough for gating and should be measured properly when convenient. +- **GLM and DSv4 cannot join the standard state.** Their weights exceed one box at every usable quant + (GLM IQ4_XS 146 GiB, DSv4 IQ4_XS 128 GiB against 124 GiB MemTotal), so they are structurally two-box, and the + second box is crow. A crow-side worker share of 46 to 56 GiB does not fit beside a 43 GiB 35b plus the rest of + prod, and the splits that would make it fit (about 0.73/0.27 toward raven) are untested and put raven at its + ceiling. Hence: they get their own state, with both boxes evicted. + +**Assets are already in place.** Raven holds `qwen38-flash-next` UD-Q4_K_XL (104 GB) and UD-Q5_K_XL (148 GB), both +MTP heads including `shared-Q8_0`, plus `dsv4-flash-vision-exp`, `dsv4-flash-0731` and `glm53-flash`, with 614 GB +free at 66 percent. Crow has 168 GB free at 91 percent and does **not** hold the Flash-Next Q4/Q5 quants. Nothing +needs downloading for this design. + +## 3. What the standard state requires + +### 3.1 Flash-Next as a production service on raven + +Raven has run no production services until now. This makes it a second production host. + +Launch, from the validated single-box config (tree `~/llama-max-stack`, or b10715 for the MTP head): + +``` +llama-server -m /Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \ + -c 1048576 -b 4096 -ub 2048 -ctk q8_0 -ctv q8_0 \ + --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 262144 \ + --override-kv qwen4exp.attention.indexer.top_k=int:1024 \ + --spec-type draft-mtp -md /mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-draft-n-max 2 \ + -ngl 999 -fa on -np 1 --jinja --host 0.0.0.0 --port +``` + +**Run it natively on raven's host, not in a container.** The bind-mount into the stock Vulkan image works +(pi-lab probed it: `llama-bench --list-devices` enumerates the GPU identically). The image does carry Mesa 25.3.6 +against raven's host 25.2.8, and H.4 measured image choice moving 27b prefill by 36 to 57 percent. Every validated +number for this config was taken on the host. A native systemd unit keeps the measurements valid and removes the +container from the trust chain. The existing benchmark arms already run exactly this way. + +Open items for the implementer: +- **Port.** pi-lab reserved 8030 for FN single-box. Confirm nothing on raven claims it. +- **Provider row** pointing at `http://10.0.0.126:/v1`. Note `providers` is in `SYNCED_TABLES` + (`servers/sharing/instance-sync.js:68`), so the row replicates to every paired instance. That is desirable here + and must be deliberate. +- **Restart policy and a memory watchdog.** 17.7 GiB is the permanent headroom on raven in this state. It is a + shape that has run for hours (R25/R25b) and never wedged, and it is still the tightest standing configuration in + the lab. +- **MTP at 1M carries a known quality caveat**: output is not byte-identical between MTP and non-MTP at 1M + (1752 versus 1777 tokens on prompt A), where it was identical at 262k. It has not shown up as a quality loss: + the 1M arms scored 8/9 twice, the same as 262k, missing the same case. MTP is worth 1.82x on prompt A, so keep it, + but record the caveat rather than discovering it. +- **A 262k fallback is worth defining.** 262k leaves 31.0 GiB free instead of 17.7 and is the more exercised rung. + If 1M proves uncomfortable in standing use, 262k is the same service with one flag changed. + +### 3.2 Group and eviction convention + +`group` and `evicts` in `settings.localModels` are global strings and host-blind. `wouldEvict` compares group +names and nothing else. So a raven entry that declared `evicts: ["standard"]` would stop crow's 35b for no reason. + +Convention for this design: +- raven's Flash-Next entry: its own group, `evicts: []`. It never evicts anything. +- crow's existing `standard` group: unchanged, and must not name raven's group. +- the heavy states: group `heavy`, evicting everything on both hosts. This is the only group that may cross hosts. + +### 3.3 The harness can consume this, and cannot manage it + +`startModelNow` runs `docker compose up -d` with `cwd: entry.composeDir` on whichever box runs the harness. There +is no host field, no ssh, no docker context. **The harness cannot start or stop a raven-hosted model.** It can +consume one, because `isRunning` fetches `/models` over the LAN. + +For the standard state this does not matter: the service is persistent, so nothing needs to start it on demand. +This is the main reason this design is cheaper than the alternatives considered. Do not build a `composeDir` +entry for it that implies otherwise, because the first start would fail. + +For the heavy states it does matter, and the window script already ssh-es to raven, so the lifecycle belongs there +rather than in the harness. + +### 3.4 One harness defect this design will hit + +`setRoleModel`'s conflict test is: + +```js +const conflicted = bindings.filter(([role, m]) => role !== modelRole && m && m !== ref && isLocal(m)); +``` + +It fires whenever two roles hold two different local models. It never calls `wouldEvict`, never reads +`group`/`evicts`, and has no host concept. So binding a raven model to one role and a crow model to another raises +"local models share one server slot" and offers to collapse them onto one model. It is a `confirm()`, so Cancel is +safe and nothing is destroyed, and the harness will argue against this configuration every time a binding changes. + +The fix is to ask whether starting one actually evicts the other. `wouldEvict` is already exported and already the +right predicate, and with the convention in 3.2 it returns false for the raven/crow pair. The work is exposing the +eviction relation to the client, which today receives only a boolean `local` flag from `annotate()`. This is +pi-lab's tree and needs its own change with a test. + +## 4. The heavy states + +Both are window modes, not services. Entry conditions, per pi-lab's classes: + +- **GLM-5.3-Flash IQ4_XS two-box**, pipeline parallelism off, `-b` equal to `-ub` at 2048 (#28360 asserts at 512 + at every rung, and at 1024/2048 with PP on). Least exercised of the three. Its top_k quality gate has not run. +- **DSv4-Flash IQ4_XS two-box**, tree `~/llama-hc-sop`, now equivalent to master plus #28571 since #26578 merged + on 2026-09-07 (`7a333e724`). Highest quality of the three at 9/9, and the exact shape that wedged the box three + times. Operator present, explicit override, never a one-tap option. + +Both now require **evicting production on two hosts** and restoring both. Today `dsv4-window.sh` restores crow +prod only. Teaching it raven prod is a prerequisite for this design, and it is the one new capability the heavy +states need. + +Context caps: on the max-stack lineage, keep two-box context below 128k, because the RPC worker aborts at +`ggml-rpc.cpp:1386` on 128k prompts on both cache and no-cache builds. This is lineage-scoped, not universal: +R24 W1b ran a 120k prompt two-box on `llama-hc-sop`, which carries `supports_op`, successfully. For qwen4exp above +64k on the #28571 lineage it is untested, and Thursday's W0 tests exactly that. + +## 5. The window contract, which is not negotiable + +Whatever wraps a heavy state must preserve these five properties. Each was written after an incident, and the +orchestrator should **call** `~/pi-lab/scripts/dsv4-window.sh` rather than reimplement it. + +1. The deadman is **out of process**, a detached watchdog with its own wall clock, and restores production even if + the orchestrator dies, hangs or is killed. This is a lab-wide rule. +2. The memory gate reads **live** state, MemAvailable plus GTT residue after eviction, never a static + `min_vram_gb` floor. A static floor cannot see the 86 GiB a dead process is still pinning, which is the exact + condition that produced the wedges. +3. Production restore is **verified**, health check plus DB integrity, before a window is called closed. With this + design that now means both hosts. +4. The box reservation file is written, because that is what makes the gateway refuse a competing on-demand start. +5. The two-strike ownership guard stays. + +**Veto, from pi-lab, carried here as a requirement:** a curated catalog must never be able to offer a known-wedge +shape as a one-tap option. + +### 5.1 Open seam: window expiry during a live turn + +Raised by crow-34, and jointly owned. A window expiring restores production out from under a live conversation, +which is correct for the box and bad for the bot. + +Proposed answer, for review: **drain, never extend.** The out-of-process deadman keeps its absolute wall clock, +untouched, because property 1 above is not negotiable. A separate soft deadline before it stops accepting new +turns and lets the in-flight turn finish. If that turn is still running at the hard deadline, the deadman wins and +the turn dies. The operator sees the soft deadline in the session drawer. + +Extending the deadman on activity is explicitly rejected: that is how a cap stops being a cap. + +Note that in the standard state this seam is milder than it looks, because evicting for a heavy state stops +raven's Flash-Next but leaves crow's 35b as a genuine fallback, which is not true of any design where the heavy +model displaces crow prod. + +## 6. Catalog work, in shippable order + +`registry/model-catalog.json` v1 already carries `qwen3.8-flash-next`, `glm-5.3-flash` and `deepseek-v4-flash`. +Their `launch` blocks carry only `ctx`, `ngl`, `flash_attn`, `no_mmap`, `parallel`, `jinja`. None of the validated +tuning is expressible, there are no topology or rung variants, `min_vram_mb` is 0 on every heavy quant, and +`min_runtime_version` says `b10068` while each entry's own notes admit the stock runtime will not load it. + +crow-34's sequencing, adopted: land this in pieces rather than as one block. + +1. **`serving.class`** first. Independently shippable, needs no build, topology or window support, and protects the + box on day one by letting the orchestrator refuse a known-wedge shape without an explicit override. Classes: + `resident` (single box, no RPC, safe behind a cap), `windowed` (operator present, two-box, evicts), and + `wedge-risk` (above roughly 85 GiB per box, explicit override, never one-tap). +2. **`gates`**, live state rather than a static floor: max GTT per host, required MemAvailable. The measured peaks + per config are in the serving doc's memory table and belong here. +3. **`build`**, naming the tree or PR set a config needs, since `min_runtime_version` against a stock release is + known-false for all three models. +4. **`topology`**, host roles and link preconditions. Last, and only needed once a two-box state is curated. + +**Throughput numbers need care.** Any figure the catalog publishes per config must come from a run in the +configuration as deployed. The doc's numbers are host-native. If a config is ever containerized, it needs one +confirmation run first, for the Mesa reason in 3.1. + +## 7. Decisions still open + +- **The `crow-dsv4` provider row.** It points at `http://127.0.0.1:8020/v1`, is enabled, and validates in + `resolveModel` while nothing listens there, so a bot pointed at it fails every turn on connection refused. + Confirmed live on both the crow and R4 instances. Because `providers` syncs, this is a fleet decision and no + session has touched it. Under this spec DSv4 is a window mode with no standing endpoint, so the row should be + retired rather than repointed. Kevin's call. +- **Priority of the two harness changes** (remote lifecycle in 3.3, the swap predicate in 3.4) against the catalog + work. Only the swap predicate affects the standard state. Kevin's call. +- **524k as a curated rung.** Deferred. It has exactly one two-box run, no byte-identity check and no quality arm, + where 262k and 1M each have several runs, a full corpus twice and 8/9 twice. It wants an identity check and one + zoo arm before curation. + +## 8. Scheduling + +Raven stops being single-tenant. It is master for every two-box arm, so a heavy state and any two-box benchmark +work are mutually exclusive in time, and Flash-Next production must be evicted before either. pi-lab has added a +standing note to CROW-SCHEDULE; it needs updating from "raven is single-tenant" to "raven runs Flash-Next +production, which two-box arms must evict and restore." From 2c7c8b11925d4c96672785ef0c3cab6fc6d99243 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:11:17 -0500 Subject: [PATCH 02/10] Spec review round: two ownership decisions, a ceiling rule, and an ordering constraint Folds in review from both other sessions. crow-34, as format/catalog owner, made two calls. serving.class is a catalog schema field on the model entry and must NOT live in settings.localModels: a curated safety property held in per-instance settings is one edit away from a wedge-risk shape being relabelled resident, and a veto needs git review. Instance settings may narrow what a box will run, never widen it. The group convention goes the other way, into settings.localModels, because group/evicts describe what else runs on a box rather than a property of a model. Its refinement, adopted: class is a property of a config, not a model, and the catalog has no variant concept yet. So the entry-level field is now defined as a ceiling rather than a description, the most dangerous supported shape for that model, with class moving onto the variant when topology lands and the entry value kept as an enforced ceiling. The ratchet is one-way by design. The group convention is now labelled a workaround, with a host-aware eviction relation named as the intended end state, so future multi-host entries do not re-learn the trap. pi-lab supplied a hard ordering constraint that was missing: the two-host window must land BEFORE Flash-Next production goes onto raven. dsv4-window.sh evicts crow prod only and its preflight requires raven idle under 2 GiB GTT residue, so once raven carries prod every two-box arm either fails preflight or contends with a live service. New 3.0. Also: crow prod measured at 61 to 62 GiB across two window teardowns, anchoring the 43 GiB bound for the 35b; port 8030 confirmed free on raven, closing an open item; the catalog memory fields described exactly (min_vram_mb absent at entry level, present-but-zero per quant, min_ram_mb populated and useful); and pi-lab's offer of an isolated per-container GTT measurement recorded as needing a deliberate go. --- ...o-host-production-and-heavy-model-modes.md | 89 ++++++++++++++++--- 1 file changed, 75 insertions(+), 14 deletions(-) diff --git a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md index 6457bd7d..8f17473f 100644 --- a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md +++ b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md @@ -34,9 +34,13 @@ Per-box GTT is 124 GiB on both machines. Measured, from which is what leaves crow untouched. - **Flash-Next single-box has never wedged.** All three GPU wedges were DSv4 two-box at 86 to 91 GiB per box, confirmed against CROW-SCHEDULE's own incident rows, which name DSv4 and dspark and never Flash-Next or GLM. -- **Crow prod is about 60.5 GiB of GTT today**, of which the vLLM 4b is 15.2 GiB by KFD accounting and the 35b is - roughly 43 GiB by subtraction. The Vulkan containers do not appear in ROCm accounting, so the 35b figure is - bounded rather than isolated. It is good enough for gating and should be measured properly when convenient. +- **Crow prod is 61 to 62 GiB of GTT**, measured two ways. A live read gives 60.5 GiB; two independent window + teardowns gave MemAvailable swings of 64 and 61 GiB on evicting all four containers, with 17 MiB of GTT residue + left behind. Of that, the vLLM 4b is 15.2 GiB by KFD accounting, which puts **the 35b at roughly 43 GiB** by + subtraction. Cross-checked from the other direction: its weights are 27.16 GB plus a 0.9 GB mmproj, about + 26.1 GiB, and at `-c 262144` another ~17 GiB of KV and compute buffers is plausible. The Vulkan containers do not + appear in ROCm accounting at all, so 43 GiB is a **bound rather than an isolated measurement**, and every gate in + this spec treats it as one. - **GLM and DSv4 cannot join the standard state.** Their weights exceed one box at every usable quant (GLM IQ4_XS 146 GiB, DSv4 IQ4_XS 128 GiB against 124 GiB MemTotal), so they are structurally two-box, and the second box is crow. A crow-side worker share of 46 to 56 GiB does not fit beside a 43 GiB 35b plus the rest of @@ -50,6 +54,19 @@ needs downloading for this design. ## 3. What the standard state requires +### 3.0 Ordering constraint: the two-host window lands FIRST + +**Flash-Next production must not go onto raven until `dsv4-window.sh` can evict and restore raven prod.** This is a +hard ordering constraint rather than a preference. + +`dsv4-window.sh` evicts and restores crow prod only, and its `preflight` requires raven to be idle with under 2 GiB +of GTT residue. The moment raven carries a production service, every two-box arm either fails pre-flight on "raven +busy", which is the good outcome, or contends with a live production service, which is the bad one. Thursday's +chain is safe only because raven is idle today. + +So the deployment order is: teach the window two hosts, then stand up Flash-Next on raven. Doing it the other way +round breaks the two-box benchmark program on its next run. + ### 3.1 Flash-Next as a production service on raven Raven has run no production services until now. This makes it a second production host. @@ -72,7 +89,9 @@ number for this config was taken on the host. A native systemd unit keeps the me container from the trust chain. The existing benchmark arms already run exactly this way. Open items for the implementer: -- **Port.** pi-lab reserved 8030 for FN single-box. Confirm nothing on raven claims it. +- **Port 8030, confirmed free on raven** (2026-09-09: only 22, 53, 631 and two ephemeral ports listen there). + 8031 through 8033 are free too. Closing this before the provider row exists matters, because a provider row + pointing at an occupied port is a subtler form of the `crow-dsv4` bug this spec retires. - **Provider row** pointing at `http://10.0.0.126:/v1`. Note `providers` is in `SYNCED_TABLES` (`servers/sharing/instance-sync.js:68`), so the row replicates to every paired instance. That is desirable here and must be deliberate. @@ -91,11 +110,22 @@ Open items for the implementer: `group` and `evicts` in `settings.localModels` are global strings and host-blind. `wouldEvict` compares group names and nothing else. So a raven entry that declared `evicts: ["standard"]` would stop crow's 35b for no reason. +**Decision (crow-34, owner): the group convention lives in `settings.localModels`, not in the catalog.** +`group`/`evicts` describe what else is running on a given box, which is instance topology rather than a property of +a model. The same Flash-Next entry would need a different group on crow than on raven, so putting it in a curated +file would bake one lab's host layout into content meant to describe models. + Convention for this design: - raven's Flash-Next entry: its own group, `evicts: []`. It never evicts anything. - crow's existing `standard` group: unchanged, and must not name raven's group. - the heavy states: group `heavy`, evicting everything on both hosts. This is the only group that may cross hosts. +**This convention is a workaround and should be recorded as one.** Encoding a *host* distinction inside a global, +host-blind string works only while everyone remembers it, and this section is its own evidence: a raven entry that +declared `evicts: ["standard"]` would silently stop crow's 35b. **The intended end state is a host-aware eviction +relation**, after which the convention becomes unnecessary. Writing that down here is what stops every future +multi-host entry from re-learning the trap the same way. + ### 3.3 The harness can consume this, and cannot manage it `startModelNow` runs `docker compose up -d` with `cwd: entry.composeDir` on whichever box runs the harness. There @@ -183,16 +213,38 @@ model displaces crow prod. ## 6. Catalog work, in shippable order `registry/model-catalog.json` v1 already carries `qwen3.8-flash-next`, `glm-5.3-flash` and `deepseek-v4-flash`. -Their `launch` blocks carry only `ctx`, `ngl`, `flash_attn`, `no_mmap`, `parallel`, `jinja`. None of the validated -tuning is expressible, there are no topology or rung variants, `min_vram_mb` is 0 on every heavy quant, and -`min_runtime_version` says `b10068` while each entry's own notes admit the stock runtime will not load it. +Their `launch` blocks carry only `ctx`, `ngl`, `flash_attn`, `no_mmap`, `jinja` (plus `parallel` on Flash-Next). +None of the validated tuning is expressible and there are no topology or rung variants. On the memory fields, to be +exact: `min_vram_mb` is **absent at the entry level** and **present-but-zero on every quant**, while `min_ram_mb` +*is* populated and meaningful (115,068 MB for Flash-Next UD-Q4_K_XL, 157,911 for GLM UD-IQ4_XS). So the catalog +already carries a RAM figure; what it lacks is measured peak GTT per host per config, which is what a gate needs. +`min_runtime_version` says `b10068` on all three while each entry's own notes admit the stock runtime will not load +them. crow-34's sequencing, adopted: land this in pieces rather than as one block. -1. **`serving.class`** first. Independently shippable, needs no build, topology or window support, and protects the - box on day one by letting the orchestrator refuse a known-wedge shape without an explicit override. Classes: +1. **`serving.class`** first. Independently deployable, needs no build, topology or window support, and protects + the box on day one by letting the orchestrator refuse a known-wedge shape without an explicit override. Classes: `resident` (single box, no RPC, safe behind a cap), `windowed` (operator present, two-box, evicts), and `wedge-risk` (above roughly 85 GiB per box, explicit override, never one-tap). + + **Decision (crow-34, owner): `serving.class` is a catalog schema field on the model entry, and must NOT live in + `settings.localModels`.** The reason is the veto itself. A curated safety property held in per-instance settings + is one settings edit, or one bug in a settings writer, away from a `wedge-risk` shape being relabelled `resident` + locally, after which the orchestrator offers it as one tap. `registry/model-catalog.json` lives in git and gets + reviewed, which is exactly the property a veto needs. **Instance settings may narrow what a box will run, never + widen it.** + + **Class is a property of a config, not of a model**, and the catalog has no variant concept yet. Flash-Next + single-box at 262k is `resident`; DSv4 two-box at 86 to 91 GiB per box is `wedge-risk`; those could be one entry. + So the entry-level field is defined as **a ceiling, not a description: the most dangerous supported shape for + that model.** DSv4 is `wedge-risk` outright and can never be one-tap however it is invoked, Flash-Next is + `resident`, GLM is `windowed`. Over-restrictive in principle, correct in every case that exists today, and it + preserves the veto on day one without waiting for step 4. + + When `topology` lands in step 4, class moves onto the variant, with the entry-level value kept as an **enforced + ceiling: a variant may never declare itself safer than its model's class.** The ratchet is one-way by design + rather than by whoever implements step 4. 2. **`gates`**, live state rather than a static floor: max GTT per host, required MemAvailable. The measured peaks per config are in the serving doc's memory table and belong here. 3. **`build`**, naming the tree or PR set a config needs, since `min_runtime_version` against a stock release is @@ -210,8 +262,15 @@ confirmation run first, for the Mesa reason in 3.1. Confirmed live on both the crow and R4 instances. Because `providers` syncs, this is a fleet decision and no session has touched it. Under this spec DSv4 is a window mode with no standing endpoint, so the row should be retired rather than repointed. Kevin's call. -- **Priority of the two harness changes** (remote lifecycle in 3.3, the swap predicate in 3.4) against the catalog - work. Only the swap predicate affects the standard state. Kevin's call. +- **Priority of the two harness changes** against the catalog work. Note that remote lifecycle (3.3) is *not* + needed for the standard state and should not be built speculatively. That leaves the two-host window (3.0) and the + swap predicate (3.4). pi-lab's read, which this spec endorses: **the two-host window first**, because it is the + hard prerequisite that makes a two-box arm safe once raven carries production, where the swap predicate is a + confirm dialog. Kevin's call. +- **An isolated per-container GTT measurement**, offered by pi-lab and not yet taken. The window restore path + already brings containers back one at a time, so logging GTT between each restore would yield real per-container + figures instead of the 43 GiB bound this spec uses. It is a log-only change in a script that runs unattended with + a chain armed, so it is a five-minute job that wants a deliberate go rather than a quiet edit. - **524k as a curated rung.** Deferred. It has exactly one two-box run, no byte-identity check and no quality arm, where 262k and 1M each have several runs, a full corpus twice and 8/9 twice. It wants an identity check and one zoo arm before curation. @@ -219,6 +278,8 @@ confirmation run first, for the Mesa reason in 3.1. ## 8. Scheduling Raven stops being single-tenant. It is master for every two-box arm, so a heavy state and any two-box benchmark -work are mutually exclusive in time, and Flash-Next production must be evicted before either. pi-lab has added a -standing note to CROW-SCHEDULE; it needs updating from "raven is single-tenant" to "raven runs Flash-Next -production, which two-box arms must evict and restore." +work are mutually exclusive in time, and Flash-Next production must be evicted before either. + +pi-lab has rewritten the CROW-SCHEDULE standing note accordingly: raven is a production host, the service is native +systemd for the Mesa reason in 3.1, and three consequences follow. Raven is no longer free to borrow. The ordering +constraint in 3.0 applies. And property 3 of the window contract, verified restore, now spans two hosts. From bac3d0abef070939840670347b65fbea75b26ab4 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:25:47 -0500 Subject: [PATCH 03/10] Port registry: add raven as a second host, and name two gaps it exposes Kevin: "we might need to add the second box to the port registry if we haven't already." We hadn't. The registry described one machine throughout, with no host dimension anywhere, so under the two-host production design a port number no longer identifies an allocation. Adds a raven section covering 8030 (Flash-Next production), 8031-8033 (two-box masters, reserved) and crow:50052 (the RPC worker on the USB4 link). 8030 through 8033 verified free on raven, which listens only on 22, 53, 631 and two ephemeral ports. The first column is deliberately raven: rather than a bare number, because check-port-allocation.js reads bare numbers in the first cell as crow allocations. Keeping raven rows unparseable to it is correct today and is a stopgap; making the checker host-aware is recorded as follow-up. Checker still passes: 45 ports, all documented, no new collisions. Two gaps found while writing it, recorded rather than silently fixed. Crow's own model ports 8003, 8006 and 8014 are absent from the table (only the copilot's 8010 is listed) because they bind from crow-addons composes, which the conventions already call a separate registry that nothing checks. And 8098 is allocated to searxng on crow's loopback while the two-box zoo arms use 8098 on raven: both correct today because they are different machines, and neither the table nor the checker can express that. --- docs/developers/port-allocation.md | 32 ++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/developers/port-allocation.md b/docs/developers/port-allocation.md index 281095fe..a17ff20a 100644 --- a/docs/developers/port-allocation.md +++ b/docs/developers/port-allocation.md @@ -114,6 +114,38 @@ These bundles use `network_mode: host`. They consume whatever ports their upstre - `tailscale` (MagicDNS, peer connections) - `crowdsec-firewall-bouncer` (deferred to PR 4.5 — upstream does not publish a Docker image; needs a custom Dockerfile and a tested unwind command verified on a throwaway host) — will need host network to manipulate iptables/nftables +## Second host: raven + +Until 2026-09-09 this registry described one machine. The two-host production design +(`docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md`) makes **raven** a production +host, so ports now need a host qualifier to mean anything. + +Raven's ports live in their own namespace and do not collide with crow's. The first column below is deliberately +`raven:` rather than a bare number, because `scripts/check-port-allocation.js` reads bare numbers in the +first cell as **crow** allocations. Keeping raven rows unparseable to it is correct today and is a stopgap: +**making the checker host-aware is follow-up work**, and until it lands, a raven port is only verified by looking +at raven. + +| port | bind | what | status | +|---|---|---|---| +| raven:8030 | 0.0.0.0 | Qwen3.8-Flash-Next @1M, production (native systemd, not a container) | planned | +| raven:8031 | 0.0.0.0 | Flash-Next two-box master (window mode) | reserved | +| raven:8032 | 0.0.0.0 | DeepSeek-V4-Flash two-box master (window mode) | reserved | +| raven:8033 | 0.0.0.0 | GLM-5.3-Flash two-box master (window mode) | reserved | +| crow:50052 | 10.99.0.1 (USB4 link) | `ggml-rpc-server` worker for every two-box arm | existing | + +Verified free on raven 2026-09-09: 8030 through 8033. Raven listens only on 22, 53, 631 and two ephemeral ports. + +### Two gaps this section exposes, both worth closing + +1. **Crow's own model ports are largely unlisted.** 8003 (35b), 8006 (27b solo) and 8014 (27b 512k) are absent + from the allocation table; only 8010 (27b copilot) is recorded. They bind the tailnet IP from `crow-addons/` + composes, which the conventions above already flag as a separate registry, so nothing catches them. +2. **8098 is ambiguous across hosts.** The table allocates it to searxng on crow's loopback, while the two-box + benchmark zoo arms conventionally use 8098 on raven. Both are correct today because they are different + machines, and neither the table nor the checker can say so. This is precisely the class of silent + double-allocation the conventions section warns about, one host further out. + ## Process for amending this file 1. Pick an unallocated port in a sensible range (admin UIs in 3000-3099, backend APIs in 8000-8099, metrics in 19000-19999). From 5b874efd24115cfbccee58d95916d6f47c57b332 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:28:51 -0500 Subject: [PATCH 04/10] Port registry: complete raven's coverage, including ports nothing could catch pi-lab pointed out that raven:8098 has been bound by its zoo arms for days while the new raven section omitted it, which makes the registry look authoritative for raven while missing a port in regular use. That is worse than not covering raven at all, and it was right. Scanning pi-lab/scripts for bound ports rather than adding only the one named turned up three more: raven:8035 (the R24 result-check server on loopback), raven:50053 (a second ggml-rpc-server, for arms running two workers on raven's one GPU), and the fact that 50052 binds on BOTH machines depending on which is the worker, 10.99.0.1 for crow and 10.99.0.2 for raven. That last one is the clearest possible argument for the raven: prefix, since a bare 50052 cannot say which box it means. Section now splits production and reserved from benchmark-transient, the latter being ports bound by scripts rather than compose files, so nothing would ever have caught them. Also records crow:8020 and crow:8021, absent from the main table for the same reason; 8020 matters because the crow-dsv4 provider row points at it. Checker unchanged: 45 ports, all documented, no new collisions. --- docs/developers/port-allocation.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/developers/port-allocation.md b/docs/developers/port-allocation.md index a17ff20a..5496ab57 100644 --- a/docs/developers/port-allocation.md +++ b/docs/developers/port-allocation.md @@ -126,22 +126,45 @@ first cell as **crow** allocations. Keeping raven rows unparseable to it is corr **making the checker host-aware is follow-up work**, and until it lands, a raven port is only verified by looking at raven. +### Production and reserved + | port | bind | what | status | |---|---|---|---| | raven:8030 | 0.0.0.0 | Qwen3.8-Flash-Next @1M, production (native systemd, not a container) | planned | | raven:8031 | 0.0.0.0 | Flash-Next two-box master (window mode) | reserved | | raven:8032 | 0.0.0.0 | DeepSeek-V4-Flash two-box master (window mode) | reserved | | raven:8033 | 0.0.0.0 | GLM-5.3-Flash two-box master (window mode) | reserved | -| crow:50052 | 10.99.0.1 (USB4 link) | `ggml-rpc-server` worker for every two-box arm | existing | -Verified free on raven 2026-09-09: 8030 through 8033. Raven listens only on 22, 53, 631 and two ephemeral ports. +### Benchmark-transient, held only inside a window + +These are bound by `pi-lab/scripts/` and are absent from any compose file, so nothing would ever have caught them. +They are listed because a registry that covers raven while omitting ports in regular use is worse than one that +does not cover raven at all. + +| port | bind | what | +|---|---|---| +| raven:8098 | 0.0.0.0 | zoo arm endpoint, two-box master and single-box arms (`ZOO_PORT`) | +| raven:8035 | 127.0.0.1 | R24 result-check server (`raven-r24-check-results.sh`) | +| raven:50052 | 10.99.0.2 (USB4 link) | `ggml-rpc-server` when **raven** is the worker | +| raven:50053 | 127.0.0.1 | second `ggml-rpc-server`, for arms running two workers on raven's one GPU | +| crow:50052 | 10.99.0.1 (USB4 link) | `ggml-rpc-server` when **crow** is the worker, which is the usual case | +| crow:8020 | 127.0.0.1 | DeepSeek-V4-Flash windowed serve (`dsv4-window.sh`) | +| crow:8021 | 127.0.0.1 | max-stack and indexer phase-0 probes (`PHASE0_PORT`) | + +**50052 binds on both machines** depending on which one is the worker, which is the clearest illustration of why a +bare port number is not an allocation here. `10.99.0.1` is crow and `10.99.0.2` is raven on the USB4 link. + +Verified free on raven 2026-09-09: 8030 through 8033, 8035 and 8098. Raven listens only on 22, 53, 631 and two +ephemeral ports. ### Two gaps this section exposes, both worth closing 1. **Crow's own model ports are largely unlisted.** 8003 (35b), 8006 (27b solo) and 8014 (27b 512k) are absent from the allocation table; only 8010 (27b copilot) is recorded. They bind the tailnet IP from `crow-addons/` composes, which the conventions above already flag as a separate registry, so nothing catches them. -2. **8098 is ambiguous across hosts.** The table allocates it to searxng on crow's loopback, while the two-box +2. **Crow's benchmark ports are unlisted too.** 8020 and 8021 are bound by `pi-lab/scripts/` on crow and appear + nowhere in the table. 8020 matters most, because the `crow-dsv4` provider row points at it. +3. **8098 is ambiguous across hosts.** The table allocates it to searxng on crow's loopback, while the two-box benchmark zoo arms conventionally use 8098 on raven. Both are correct today because they are different machines, and neither the table nor the checker can say so. This is precisely the class of silent double-allocation the conventions section warns about, one host further out. From f5a1d7caf16ade7143b5399455b1dedbc40ed2f7 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:32:23 -0500 Subject: [PATCH 05/10] Port registry: definitive two-host list, and the scanning lesson pi-lab reviewed its own tree against the previous commit and found one wrong attribution, one live collision, and three more ports. Verifying those rather than accepting them found five more that neither of us had, and corrected pi-lab's correction. Fixed: 8021 is bound by 25 raven-* master scripts and is the standard two-box master port from R1 through R26, so it belonged on raven. But it is NOT raven-only as reported: four non-raven scripts default to it and all of them run as duties under dsv4-window.sh, which is the crow window. So 8021 binds on both boxes, a third instance alongside 8099 and 50052. Collision: raven-r23-knob-sanity.sh was already binding 127.0.0.1:8031, inside the range this section reserved for two-box masters. pi-lab moved it to 8037 (KS_PORT) rather than have the reservation move. Found by scanning invocations rather than assignments: crow:8022, 8024, 8025, 8026 and 8027, all DSv4 Vulkan hyper-connection duties. Added from pi-lab's read of its own tree: raven:8036 (the R25/R25b single-box stack, which is the Flash-Next shape chosen for production), crow:8012/8013 (GLM-5.2 windows) and crow:8023. The section now lists twenty-one script-bound ports across both hosts, none of which any compose file declares, and calls out the three that bind on both machines. It also records how the list was built, because three passes each found more and a human reading the tree still found what the patterns missed. Any scan of this kind is a lower bound. Checker unchanged: 45 ports, all documented, no new collisions. --- docs/developers/port-allocation.md | 41 +++++++++++++++++++----------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/docs/developers/port-allocation.md b/docs/developers/port-allocation.md index 5496ab57..594fcab1 100644 --- a/docs/developers/port-allocation.md +++ b/docs/developers/port-allocation.md @@ -126,7 +126,7 @@ first cell as **crow** allocations. Keeping raven rows unparseable to it is corr **making the checker host-aware is follow-up work**, and until it lands, a raven port is only verified by looking at raven. -### Production and reserved +### Production and reserved (raven) | port | bind | what | status | |---|---|---|---| @@ -137,33 +137,44 @@ at raven. ### Benchmark-transient, held only inside a window -These are bound by `pi-lab/scripts/` and are absent from any compose file, so nothing would ever have caught them. -They are listed because a registry that covers raven while omitting ports in regular use is worse than one that -does not cover raven at all. +Bound by `pi-lab/scripts/`, never by a compose file, so `check-port-allocation.js` cannot see any of them. Listed +because a registry that covers a host while omitting ports in regular use is worse than one that omits the host. | port | bind | what | |---|---|---| +| raven:8021 | 127.0.0.1 | **two-box master port**, the standard across R1 to R26 (25 `raven-*` scripts). Thursday's W0 binds it | +| raven:8035 | 127.0.0.1 | R24 result-check server | +| raven:8036 | 127.0.0.1 | R25 / R25b single-box stack runs. This is the Flash-Next config chosen for production, so 8030 and 8036 are the same shape on different ports | +| raven:8037 | 127.0.0.1 | R23 knob-sanity (`KS_PORT`), moved here off 8031 when this section reserved that range | | raven:8098 | 0.0.0.0 | zoo arm endpoint, two-box master and single-box arms (`ZOO_PORT`) | -| raven:8035 | 127.0.0.1 | R24 result-check server (`raven-r24-check-results.sh`) | -| raven:50052 | 10.99.0.2 (USB4 link) | `ggml-rpc-server` when **raven** is the worker | +| raven:8099 | 127.0.0.1 | Q4 MTP smoke | +| raven:50052 | 10.99.0.2 (USB4) | `ggml-rpc-server` when **raven** is the worker | | raven:50053 | 127.0.0.1 | second `ggml-rpc-server`, for arms running two workers on raven's one GPU | -| crow:50052 | 10.99.0.1 (USB4 link) | `ggml-rpc-server` when **crow** is the worker, which is the usual case | -| crow:8020 | 127.0.0.1 | DeepSeek-V4-Flash windowed serve (`dsv4-window.sh`) | -| crow:8021 | 127.0.0.1 | max-stack and indexer phase-0 probes (`PHASE0_PORT`) | +| crow:8012, 8013 | 127.0.0.1 | GLM-5.2 IQ2 and IQ4 windows (`glm52-window.sh` and friends) | +| crow:8020 | 127.0.0.1 | DeepSeek-V4-Flash windowed serve (`dsv4-window.sh`). The `crow-dsv4` provider row points here | +| crow:8021 | 127.0.0.1 | phase-0 Vulkan probes, run as duties under `dsv4-window.sh` (`PHASE0_PORT`) | +| crow:8022, 8024, 8025, 8026, 8027 | 127.0.0.1 | DSv4 Vulkan hyper-connection investigation duties (`dsv4-vk-hc-*`, `dsv4-ubatch-confirm`) | +| crow:8023 | 127.0.0.1 | DSv4 top-k A/B and HC verify (`TOPK_PORT`, `HCV_PORT`) | +| crow:8099 | 127.0.0.1 | GTT ladder duty under `dsv4-window --duty` | -**50052 binds on both machines** depending on which one is the worker, which is the clearest illustration of why a -bare port number is not an allocation here. `10.99.0.1` is crow and `10.99.0.2` is raven on the USB4 link. +### Three ports bind on BOTH machines -Verified free on raven 2026-09-09: 8030 through 8033, 8035 and 8098. Raven listens only on 22, 53, 631 and two -ephemeral ports. +`8021`, `8099` and `50052` each mean two different things depending on the box. 8021 and 8099 are the same number +on the same loopback on two hosts, so nothing about the number distinguishes them; 50052 at least differs by bind +address (`10.99.0.1` is crow, `10.99.0.2` is raven). This is the whole argument for the `raven:`/`crow:` prefix, +and it is why the prefix is worth keeping even after the checker learns host-awareness. A bare number here is not +an allocation, it is an ambiguity. + +Verified free on raven 2026-09-09: 8030 through 8033, 8035, 8036, 8037, 8098 and 8099. Raven listens only on 22, +53, 631 and two ephemeral ports. ### Two gaps this section exposes, both worth closing 1. **Crow's own model ports are largely unlisted.** 8003 (35b), 8006 (27b solo) and 8014 (27b 512k) are absent from the allocation table; only 8010 (27b copilot) is recorded. They bind the tailnet IP from `crow-addons/` composes, which the conventions above already flag as a separate registry, so nothing catches them. -2. **Crow's benchmark ports are unlisted too.** 8020 and 8021 are bound by `pi-lab/scripts/` on crow and appear - nowhere in the table. 8020 matters most, because the `crow-dsv4` provider row points at it. +2. **Crow's own benchmark ports were unlisted too**, and are now in the table above rather than the main one, + because they are script-bound and transient. There are eleven of them. 3. **8098 is ambiguous across hosts.** The table allocates it to searxng on crow's loopback, while the two-box benchmark zoo arms conventionally use 8098 on raven. Both are correct today because they are different machines, and neither the table nor the checker can say so. This is precisely the class of silent From cbade4b65428af56374bbe930d22871cf44a6f03 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:35:06 -0500 Subject: [PATCH 06/10] Port registry: the capacity constraint, the third scan form, and the method note that silently failed to land Three things, one of them a correction to my own last commit. The method note was never in the file. The previous commit's message claimed the section records how the list was built; it does not, because that edit used an unasserted string replace whose pattern did not match, so it no-opped silently while every other edit in the same script succeeded. Exactly the failure mode this section is about. Every edit here asserts its match count first. The note is now actually present. Adds the constraint a port registry structurally cannot express: raven cannot host 8030 and 8036 at the same time however free both ports are, because they are the same Flash-Next single-box config and each wants about 92.6 GiB on a 124 GiB box. Two free ports, one machine's worth of memory. Recorded as the general case, since on a single-tenant box port availability is necessary and never sufficient. Mirrored into the spec's open items, because an R25-lineage arm must now treat the production service as something to evict. Adds the third scan form, which is the easiest to miss and the one that caused the wrong host attribution: PORT="${PHASE0_PORT:-8021}" matches no scan for PORT=8021, port 8021 or :8021. It is multiplying, because parameterising a hardcoded port is the correct fix for the ambiguities here and creates a new hiding place each time. The method note now also names the two failure modes that actually occurred, neither of which was a missing grep: a scan surfaced a colliding port and its author read the hit as confirming their own reservation, and a categorical negative was asserted from a pattern that could not have matched the files in question. Checker unchanged: 45 ports, all documented, no new collisions. --- docs/developers/port-allocation.md | 34 ++++++++++++++++++- ...o-host-production-and-heavy-model-modes.md | 4 +++ 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/docs/developers/port-allocation.md b/docs/developers/port-allocation.md index 594fcab1..465824e2 100644 --- a/docs/developers/port-allocation.md +++ b/docs/developers/port-allocation.md @@ -168,7 +168,19 @@ an allocation, it is an ambiguity. Verified free on raven 2026-09-09: 8030 through 8033, 8035, 8036, 8037, 8098 and 8099. Raven listens only on 22, 53, 631 and two ephemeral ports. -### Two gaps this section exposes, both worth closing +### A constraint this registry cannot express + +**Raven cannot host 8030 and 8036 at the same time, however free both ports are.** They are the same Flash-Next +single-box config, one as the proposed production port and one as the R25/R25b benchmark port, and each wants +about 92.6 GiB on a 124 GiB box. Two free ports, one machine's worth of memory. Allocating a port is not the same +as being able to run the thing, and no port table can say so. + +Read it as the general case rather than one awkward pair: on a single-tenant box, port availability is a necessary +condition and never a sufficient one. Same class as the ordering constraint in +`docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md` §3.0, where the port would have +been free and the arm would still have been unsafe. + +### Three gaps this section exposes, all worth closing 1. **Crow's own model ports are largely unlisted.** 8003 (35b), 8006 (27b solo) and 8014 (27b 512k) are absent from the allocation table; only 8010 (27b copilot) is recorded. They bind the tailnet IP from `crow-addons/` @@ -180,6 +192,26 @@ Verified free on raven 2026-09-09: 8030 through 8033, 8035, 8036, 8037, 8098 and machines, and neither the table nor the checker can say so. This is precisely the class of silent double-allocation the conventions section warns about, one host further out. +### How this list was built, because the method outlasts the list + +Four passes each found more, and the last two were humans reading their own trees: + +1. `PORT=` assignments: four ports. +2. Adding `--port` and `host:port` forms: five more (8022, 8024 through 8027). +3. Adding the shell-default form `${VAR:-NNNN}`: this is the easiest to miss, because a scan for `PORT=8021`, + `port 8021` or `:8021` returns **nothing** on a file that binds 8021 all day via `PORT="${PHASE0_PORT:-8021}"`. + That form is also multiplying, since parameterising a hardcoded port is the right fix for the ambiguities above + and creates a new hiding place every time. +4. Two people reading their own trees: four more ports, one wrong host attribution, and one live collision inside + a range this section had just reserved. + +Two failure modes worth naming, because both happened here and neither is a missing grep. A scan **surfaced** a +colliding port and its author read the hit as confirming their own reservation rather than as a conflict, which +turns evidence into confirmation and is worse than missing it. And a categorical negative, "not one script on that +host uses it", was asserted from a pattern that could not have matched the files in question. **Treat any scan of +this kind as a lower bound.** If the checker grows host-awareness, its companion should read invocations and +shell defaults as well as assignments. + ## Process for amending this file 1. Pick an unallocated port in a sensible range (admin UIs in 3000-3099, backend APIs in 8000-8099, metrics in 19000-19999). diff --git a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md index 8f17473f..04dfe262 100644 --- a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md +++ b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md @@ -102,6 +102,10 @@ Open items for the implementer: (1752 versus 1777 tokens on prompt A), where it was identical at 262k. It has not shown up as a quality loss: the 1M arms scored 8/9 twice, the same as 262k, missing the same case. MTP is worth 1.82x on prompt A, so keep it, but record the caveat rather than discovering it. +- **8030 and 8036 are mutually exclusive on raven.** 8036 is the R25/R25b benchmark port for this same + single-box Flash-Next config. Both ports being free is irrelevant: each instance wants about 92.6 GiB on a + 124 GiB box. Any R25-lineage arm must treat the production service as something to evict, exactly like the + two-box arms in 3.0. - **A 262k fallback is worth defining.** 262k leaves 31.0 GiB free instead of 17.7 and is the more exercised rung. If 1M proves uncomfortable in standing use, 262k is the same service with one flag changed. From 3311bde55abc71efea48449e669e4938f2fa5d27 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:37:03 -0500 Subject: [PATCH 07/10] Port registry: name the failure mode behind all four mistakes pi-lab hit a fourth instance one message after I reported the third, and it completes the pattern well enough to be worth generalising. The four, none of which was a missing grep: a trailing comment on a shell assignment line swallowed the assignments after it, passing bash -n and failing later under set -u; a categorical negative was asserted from a pattern that could not have matched the files; a string replace reported success without applying, because its pattern missed the file's line wrapping; and a grep confirming a correct edit returned zero, because it was single-line and the text wrapped. All four are one failure: the check and the thing checked disagree about shape, and the check returns the reassuring answer. The fourth is the worst, because it makes a correct edit look failed and the natural response is to apply it twice. Records three cheap habits: assert the match count before substituting so a no-op is loud, verify against text normalised the same way the edit was written, and never assert a categorical negative from a pattern scan, since the honest form is "this pattern found none". This commit's own verification was done on whitespace-flattened text for exactly the fourth reason. Checker unchanged: 45 ports, all documented, no new collisions. --- docs/developers/port-allocation.md | 34 ++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/docs/developers/port-allocation.md b/docs/developers/port-allocation.md index 465824e2..962fd3e6 100644 --- a/docs/developers/port-allocation.md +++ b/docs/developers/port-allocation.md @@ -205,12 +205,34 @@ Four passes each found more, and the last two were humans reading their own tree 4. Two people reading their own trees: four more ports, one wrong host attribution, and one live collision inside a range this section had just reserved. -Two failure modes worth naming, because both happened here and neither is a missing grep. A scan **surfaced** a -colliding port and its author read the hit as confirming their own reservation rather than as a conflict, which -turns evidence into confirmation and is worse than missing it. And a categorical negative, "not one script on that -host uses it", was asserted from a pattern that could not have matched the files in question. **Treat any scan of -this kind as a lower bound.** If the checker grows host-awareness, its companion should read invocations and -shell defaults as well as assignments. +**Treat any scan of this kind as a lower bound.** If the checker grows host-awareness, its companion should read +invocations and shell defaults as well as assignments. + +### The failure mode behind every mistake made here + +Building this section produced four errors, and not one was a missing grep: + +1. A trailing comment on a shell assignment line swallowed the assignments after it. `bash -n` passed; it would + have failed at first use under `set -u`. +2. A categorical negative, "not one script on that host uses it", was asserted from a pattern that could not have + matched the files in question. +3. A string replace reported success without applying, because its pattern did not match the file's line + wrapping. Every other edit in the same script worked, so the run looked clean. +4. A grep confirming a correct edit returned zero, because the grep was single-line and the text wrapped across a + newline. Re-probing whitespace-flattened text found it. + +All four are one failure: **the check and the thing checked disagree about shape, and the check returns the +reassuring answer.** The fourth is the most dangerous, because it makes a correct edit look failed, and the +natural response is to apply it again. + +Three habits follow, and they are cheap. Assert the match count before substituting, so a no-op is loud rather +than silent. Verify against text normalised the same way the edit was written, since a single-line probe cannot +see a wrapped phrase. And never assert a categorical negative from a pattern-based scan; the honest form is "this +pattern found none", which is a different claim. + +A related trap, from the same evening: a scan **surfaced** a colliding port and its author read the hit as +confirming their own reservation rather than as a conflict. That turns evidence into confirmation, and it is worse +than missing the port outright. ## Process for amending this file From f541d965c4aa812793a3c76956cdf991d44a5e85 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Wed, 9 Sep 2026 18:38:45 -0500 Subject: [PATCH 08/10] Spec: prefer arithmetic to rules when writing a constraint pi-lab's generalisation of why its version of the raven note beat mine. "Evict raven production before an R25-lineage arm" is a rule someone forgets at 02:00; "92.6 GiB each on a 124 GiB box" is arithmetic anyone can redo and get the same answer. The second form outlives its authors and fails loudly when the hardware changes, where a remembered rule quietly stops being true. New 4.1, so later sections adding constraints have the preference stated rather than inferred. --- ...26-09-09-two-host-production-and-heavy-model-modes.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md index 04dfe262..3e46aeae 100644 --- a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md +++ b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md @@ -180,6 +180,15 @@ Context caps: on the max-stack lineage, keep two-box context below 128k, because R24 W1b ran a 120k prompt two-box on `llama-hc-sop`, which carries `supports_op`, successfully. For qwen4exp above 64k on the #28571 lineage it is untested, and Thursday's W0 tests exactly that. +## 4.1 A note on how the constraints here are written + +Where this spec had a choice, it states arithmetic rather than a rule. "Evict raven production before an +R25-lineage arm" is a rule someone forgets at 02:00. "92.6 GiB each on a 124 GiB box" is arithmetic anyone can +redo from scratch and get the same answer. The second form survives the people who wrote it, and it fails loudly +when the hardware changes underneath it, where a remembered rule just quietly stops being true. + +Prefer it wherever a later section adds a constraint. + ## 5. The window contract, which is not negotiable Whatever wraps a heavy state must preserve these five properties. Each was written after an incident, and the From 16d0ced4b63cfa5612fd65eabb1678b57e6209e5 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Thu, 10 Sep 2026 11:00:19 -0500 Subject: [PATCH 09/10] Spec: loopback rows do not sync, and the raven row is the opposite case shouldSyncRow, not SYNCED_TABLES membership, decides whether a providers row replicates, and it carries a loopback carve-out that gates both emit and apply. crow-dsv4 therefore never synced and had to be retired once per instance; the claim that it was a fleet decision was false and would have sent a reader looking for a sync bug that does not exist. The raven row sits on the other side of that carve-out. Record what follows: host rides the wire and must say raven everywhere, and the row should not be published before the endpoint is reachable from the instances that inherit it. --- ...o-host-production-and-heavy-model-modes.md | 66 ++++++++++++++++--- 1 file changed, 58 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md index 3e46aeae..f87c894d 100644 --- a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md +++ b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md @@ -92,9 +92,18 @@ Open items for the implementer: - **Port 8030, confirmed free on raven** (2026-09-09: only 22, 53, 631 and two ephemeral ports listen there). 8031 through 8033 are free too. Closing this before the provider row exists matters, because a provider row pointing at an occupied port is a subtler form of the `crow-dsv4` bug this spec retires. -- **Provider row** pointing at `http://10.0.0.126:/v1`. Note `providers` is in `SYNCED_TABLES` - (`servers/sharing/instance-sync.js:68`), so the row replicates to every paired instance. That is desirable here - and must be deliberate. +- **Provider row** pointing at `http://10.0.0.126:/v1`. This row **does** replicate to every paired + instance, and that is desirable — but membership in `SYNCED_TABLES` is not what decides it. `shouldSyncRow` + does, and it exempts loopback endpoints; this row syncs because `10.0.0.126` is routable, where the + `crow-dsv4` row §7 retires never synced at all. Two requirements follow, and both were violated on the first + attempt (2026-09-10): + - Set **`host = 'raven'`**, on every instance that carries the row. `host` rides the wire, so whichever copy + wins on `lamport_ts` imposes its value fleet-wide, and `host = 'local'` tells + `resolveWarmableProviderName` the endpoint is startable on the local box when it is not. + - **Publish the row only once the endpoint is reachable from the instances that will inherit it.** raven's + `ufw` rule currently admits crow alone; grackle and black-swan are paired with crow and cannot reach 8030. + Either widen the rule or accept that those instances carry an entry that probes as down. See §7 for the + full argument and the reason `gpu_policy.local_only` is the wrong lever. - **Restart policy and a memory watchdog.** 17.7 GiB is the permanent headroom on raven in this state. It is a shape that has run for hours (R25/R25b) and never wedged, and it is still the tightest standing configuration in the lab. @@ -270,11 +279,52 @@ confirmation run first, for the Mesa reason in 3.1. ## 7. Decisions still open -- **The `crow-dsv4` provider row.** It points at `http://127.0.0.1:8020/v1`, is enabled, and validates in - `resolveModel` while nothing listens there, so a bot pointed at it fails every turn on connection refused. - Confirmed live on both the crow and R4 instances. Because `providers` syncs, this is a fleet decision and no - session has touched it. Under this spec DSv4 is a window mode with no standing endpoint, so the row should be - retired rather than repointed. Kevin's call. +- **The `crow-dsv4` provider row — RESOLVED 2026-09-10.** It pointed at `http://127.0.0.1:8020/v1`, was + enabled, and validated in `resolveModel` while nothing listened there, so a bot pointed at it failed every + turn on connection refused. Kevin approved retirement. Both instances now carry `disabled = 1` rather than a + `DELETE`: `resolveModel`'s predicate is `WHERE id = ? AND disabled = 0`, so the row stops resolving while + staying reversible if DSv4 ever returns as a windowed serve. + + **The retirement had to be performed twice, once per instance, and the reason generalises.** An earlier draft + of this bullet said "because `providers` syncs, this is a fleet decision". That is false for this row. + Membership in `SYNCED_TABLES` is necessary but not sufficient — `shouldSyncRow` decides, and it carries a + loopback carve-out (`servers/sharing/instance-sync.js:328`): + + ```js + if (host === "localhost" || host === "::1" || /^127\./.test(host)) return false; + ``` + + `http://127.0.0.1:8020/v1` matches, and that gate covers **both emit and apply**. A peer dialing 127.0.0.1 + reaches itself and never the origin's service, so loopback provider rows are per-instance by construction. + Retiring one is a per-instance action each session performs on its own box, and waiting to observe it arrive + from a peer yields a false negative rather than a result. + +- **Loopback and routable provider rows sit on opposite sides of that carve-out, and this spec creates one of + each.** They are not one category and their failure modes are opposites, so the distinction is load-bearing + rather than editorial: + + | | `crow-dsv4` | `raven-flash-next` | + |---|---|---| + | base_url | `http://127.0.0.1:8020/v1` | `http://10.0.0.126:8030/v1` | + | syncs? | no — loopback carve-out | **yes** | + | a stale row costs | a local annoyance each box fixes itself | a fleet-wide pointer every paired instance inherits | + + §3.1 already wants the raven row to propagate, and that remains right. What follows from it is that the row + must not be published before the service behind it is real and reachable from the instances that will inherit + it — otherwise the spec fleet-publishes exactly the class of dead pointer the bullet above spent a day + retiring. Two consequences, both verified live on 2026-09-10: + + - **`host` rides the wire** (it is not in `EXCLUDED_COLUMNS.providers`), so whichever copy of a row wins on + `lamport_ts` imposes its `host` value fleet-wide. A row whose service runs on raven must say `host = 'raven'` + on **every** instance. `host = 'local'` is not merely cosmetic: `resolveWarmableProviderName` + (`gpu-orchestrator.js:619`) returns null only for a non-`local` host, so a `local` mislabel falls through to + the sibling scan and is one same-baseUrl bundle away from an instance trying to warm a service on another + machine. + - **Reachability is per-instance and the firewall decides it.** raven's `ufw` rule scopes 8030 to `10.0.0.237` + (crow) alone. The crow instance is paired with grackle and black-swan as well, neither of which can reach + that endpoint — black-swan, off-LAN, never will. Publishing the row to them costs an entry that probes as + down. The honest fix is the firewall rule, not `gpu_policy.local_only`, which would suppress the crow↔R4 + sync this spec actually wants. - **Priority of the two harness changes** against the catalog work. Note that remote lifecycle (3.3) is *not* needed for the standard state and should not be built speculatively. That leaves the two-host window (3.0) and the swap predicate (3.4). pi-lab's read, which this spec endorses: **the two-host window first**, because it is the From 33ef6776f009255ff4bdcb2084c7d36400264b45 Mon Sep 17 00:00:00 2001 From: kh0pper Date: Thu, 10 Sep 2026 11:04:10 -0500 Subject: [PATCH 10/10] Spec: the host field is inferred wrongly, not merely left unset An earlier revision of 3.1 read as though the raven row's host='local' was an omission. It was a derivation: inferHost supplies a host when the caller omits one, and it treats the whole private address space plus Tailscale's CGNAT range as local. Framing it as a forgotten field invites a checklist; framing it as an unsound inference invites the fix. --- ...26-09-09-two-host-production-and-heavy-model-modes.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md index f87c894d..3170adde 100644 --- a/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md +++ b/docs/superpowers/specs/2026-09-09-two-host-production-and-heavy-model-modes.md @@ -100,6 +100,15 @@ Open items for the implementer: - Set **`host = 'raven'`**, on every instance that carries the row. `host` rides the wire, so whichever copy wins on `lamport_ts` imposes its value fleet-wide, and `host = 'local'` tells `resolveWarmableProviderName` the endpoint is startable on the local box when it is not. + + **Setting it explicitly is not a checklist item — it is a workaround for an unsound inference.** When a + caller omits `host`, `inferHost` (`shared/providers-db.js:50`) supplies one, and it classifies every + `10.`, `192.168.` and `100.` address as `local`. That is the whole RFC1918 space plus the entire Tailscale + CGNAT range: by that rule raven, grackle, black-swan, colibri and mockingbird are all "local" to every box + in the lab. The only two answers it can return honestly are loopback and public. Existing rows are shielded + only by `if (existingHost) return existingHost` and by long-lived rows carrying explicit hosts, so the + exposure is the seed path — which means the next new id reproduces it. Until that inference is fixed, treat + an omitted `host` on any non-loopback row as a wrong value rather than a missing one. - **Publish the row only once the endpoint is reachable from the instances that will inherit it.** raven's `ufw` rule currently admits crow alone; grackle and black-swan are paired with crow and cannot reach 8030. Either widen the rule or accept that those instances carry an entry that probes as down. See §7 for the