Make node start explain its failures (v0.15.1) - #11
Merged
Conversation
A node that authenticated and was then refused registration retried forever in total silence — the failure that took a live packet trace and hours to trace to a one-line misconfiguration (a grant with role `server`/`gpu` instead of `node`). The cause was never a missing signal: the omnyhub node runtime already logged the Hub's rejection and its reason — it even receives the Hub's `NodeErrorMessage(code: forbidden, message: "principal … may not register node …")` — but its logger defaults to a NoopLogger that OmnyServer never replaced. Every diagnostic went to a silent sink. - Wire a real logger (ConsoleNodeLogger) into the runtime, so what the Hub reports reaches the operator. Problems only by default; the full lifecycle under `node start --verbose`. Repeats within a short window are collapsed, so a node that keeps retrying a rejection it cannot fix says why once, not once per backoff — the chosen behaviour: stay up and recover when the grant is fixed. - `node start` prints `Connecting to <url> …` before it blocks, and a terminal auth failure now exits with `error: <reason>` instead of an unhandled exception and a stack trace. - The Hub logs a refused registration on its own side too, so journalctl shows it, not just the audit trail. - Capability probes get a timeout. `CommandDetector` and the GPU detectors ran `Process.run` with no deadline, and the scanner awaits every probe before a node can register — so a single wedged command (a hung nvidia-smi) would have frozen registration silently. A timed-out probe is killed and treated as "not detected". The headline test reproduces the field failure exactly: a node with a non-`node`-role grant now logs `may not register`, and keeps retrying. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Born from a real debugging session: a node that authenticated and was then refused registration retried forever in total silence. It took a live packet trace to find the cause — a grant with role
server/gpuinstead ofnode.The root cause
The signal was never missing. The omnyhub node runtime already receives the Hub's rejection —
NodeErrorMessage(code: forbidden, message: "principal server-node may not register node sites-menuici")— and logs it. But its logger defaults to aNoopLoggerthat OmnyServer never replaced (node_runtime.dart:287). Every diagnostic — the rejection reason, connection-refused, bad-certificate — went to a silent sink.Changes
ConsoleNodeLogger). Problems-only by default; full lifecycle undernode start --verbose. Repeats within a short window are collapsed, so a node that keeps retrying a rejection it can't fix says why once, not once per backoff. (Chosen behaviour: stay up and recover automatically when the grant is fixed — no restart.)Connecting to <url> …before it blocks, and a terminal auth failure now exits with a cleanerror: <reason>instead ofUnhandled exception:+ stack trace.refused registration: … needs the "node" role), sojournalctlshows it, not just the audit trail.CommandDetector/GPU detectors ranProcess.runwith no deadline, and the scanner awaits every probe before registration — so one wedged command (a hungnvidia-smi) would freeze the node silently. A timed-out probe is killed and treated as "not detected."Before / after
Verification
test/integration/node_registration_logging_test.dartreproduces the field failure exactly — a non-node-role grant logsmay not registerand keeps retrying — plus unit tests for the logger's formatting/dedup and the probe timeout (asleep 30killed at a 300ms deadline).Connecting to…thenerror: Invalid token(no stack trace); an unreachable host →Node connection failed: … Connection refusedinstead of silence.dart analyze --fatal-infos --fatal-warnings,dart format,dependency_validatorclean.Follow-up (separate)
An omnyhub 1.7.0 change — map a terminal-coded
NodeErrorMessageto a typed exception so registration fails fast instead of waiting out a 10s timeout — is planned next. This omnyserver release already delivers the full user-visible fix against omnyhub^1.6.0.🤖 Generated with Claude Code