Stop a busy port from stranding every session a home has - #1175
Merged
Conversation
Route transport is injected into a session's environment once, at spawn, and never revisited. There is no renegotiation and no way to tell a running harness the router moved: the port it was handed is the only one it will ever dial. So when the preferred port was busy, the router bound an ephemeral one, logged a warning about a port, and recorded the fallback as this home's port. Nothing failed at that moment. The failures arrived later, one per session, as a connection error naming a URL — attributable to the network, the provider, or the harness, but not to the daemon that had moved out from under them. Recording the fallback made it permanent: the next start preferred the stand-in, so the home never went back, and every session spawned before the collision was unroutable for the rest of its life. Observed on a live fleet: 67 of 71 running adapters dialing a port nothing had listened on for half an hour. Four changes, all following from the port being part of a home's identity: Wait for it rather than give up on the first refusal — during a restart the likeliest holder is the daemon that just left, and it lets go in moments. Never record a stand-in as the home's port. A home that already has one keeps preferring it; only a home with nothing dialing it yet adopts what it can get, which is how a second home on one machine gets an identity. Report a fallback as the outage it is. The daemon knows at that moment that every session it spawned has lost its route, which is an error about sessions, not a warning about a port. Take the port back when it frees, and serve on it alongside the stand-in so neither generation of sessions is cut off. A pinned port is exempt throughout: it never falls back and is never rewritten.
edwin-zvs
force-pushed
the
router-port-strands-sessions
branch
from
August 2, 2026 19:14
6f93230 to
04e7dfb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
A live fleet, right now:
67 of 71 running adapters are dialing 8917, where nothing has listened since:
Every one of those sessions fails its next model call. What the user sees is the harness's error, three hours and one restart removed from the cause:
Route transport is injected into a session's environment once, at spawn (
PROXY_ENV,router.rs). There is no renegotiation and no way to tell a running harness the router moved — the port it was handed is the only one it will ever try. The module header already says reclaiming the port is mandatory for exactly this reason; the fallback path quietly violated it.And it made it permanent. The fallback was written to
router.port, so the next start preferred 64205 and the home never went back. 8917 has been free for a while now. Nothing will ever reclaim it.What this changes
Wait for the port instead of giving up on first refusal. During a restart the likeliest holder is the daemon that just exec'd away, which lets go in milliseconds. One second of retries at startup, versus every live session losing its route.
Never record a stand-in as the home's port. Binding elsewhere so the daemon still boots is right; recording it is what turned one busy moment into a permanent move. A home that already has a port keeps preferring it. A home with no port yet has nothing dialing it, so it adopts what it gets — that's how a second home on one machine gets a stable identity, and it's preserved.
The rule is extracted as
records_port(pinned, bound, claimed_preferred, established)and tested directly, because the interesting case (a first boot whose default port is taken) can't be staged without binding the machine's real default port.Report a fallback as the outage it is.
tracing::error!, notwarn, naming what was lost — the daemon knows at that instant that every session it spawned is unroutable.Take the port back when it frees, and serve on it alongside the stand-in, so neither the sessions predating the fallback nor those spawned during it are cut off. New sessions get the reclaimed port, so the home converges back to one. Attribution is by proxy credential, not by arrival port, so serving two ports changes nothing about routing.
A pinned
[router] port = Nis exempt throughout — it never falls back and is never rewritten.Behavior change to call out
auto_port_falls_back_and_persists_when_preferred_is_busyasserted the old rule and is replaced bya_stand_in_port_does_not_replace_this_homes_own, which asserts the opposite. That inversion is the fix, not a casualty of it: its own doc comment claimed "reclaim-on-restart is covered", which persisting the fallback is precisely what defeated.Not fixed here
Why 8917 and the webui's 5746 were both busy at 17:32 — almost certainly a second daemon instance. This PR makes that survivable rather than silently destructive.
Spec
0183-a-home-does-not-give-up-its-router-port(new). Module header updated to match.Testing
cargo test --workspacegreen. New: the persist rule across all six cases, a stand-in leaving the recorded port alone, and the restart race — a holder that releases mid-window has its port taken rather than abandoned.This PR touches
crates/daemononly → the relevant binary isconstruct.🤖 Generated with Claude Code