Skip to content

Make node start explain its failures (v0.15.1) - #11

Merged
gmpassos merged 1 commit into
masterfrom
fix/node-start-observability
Jul 15, 2026
Merged

Make node start explain its failures (v0.15.1)#11
gmpassos merged 1 commit into
masterfrom
fix/node-start-observability

Conversation

@gmpassos

Copy link
Copy Markdown
Contributor

Born from a real debugging session: a node that authenticated and was then refused registration retried forever in total silence. It took a live packet trace to find the cause — a grant with role server/gpu instead of node.

The root cause

The signal was never missing. The omnyhub node runtime already receives the Hub's rejection — NodeErrorMessage(code: forbidden, message: "principal server-node may not register node sites-menuici") — and logs it. But its logger defaults to a NoopLogger that OmnyServer never replaced (node_runtime.dart:287). Every diagnostic — the rejection reason, connection-refused, bad-certificate — went to a silent sink.

Changes

  1. Wire a real logger into the runtime (ConsoleNodeLogger). Problems-only by default; full lifecycle under node start --verbose. Repeats within a short window are collapsed, so a node that keeps retrying a rejection it can't fix says why once, not once per backoff. (Chosen behaviour: stay up and recover automatically when the grant is fixed — no restart.)
  2. Connecting to <url> … before it blocks, and a terminal auth failure now exits with a clean error: <reason> instead of Unhandled exception: + stack trace.
  3. The Hub logs a refused registration on its own side (refused registration: … needs the "node" role), so journalctl shows it, not just the audit trail.
  4. Capability probes get a timeout. CommandDetector/GPU detectors ran Process.run with no deadline, and the scanner awaits every probe before registration — so one wedged command (a hung nvidia-smi) would freeze the node silently. A timed-out probe is killed and treated as "not detected."

Before / after

# before: nothing, until Ctrl-C
$ omnyserver node start --hub wss://hub:8081 --id web-01 --principal p --token t
^C

# after
$ omnyserver node start --hub wss://hub:8081 --id web-01 --principal p --token t
Connecting to wss://hub:8081/node …
hub rejected the connection (forbidden): principal p may not register node web-01
  (the node.register action needs the "node" role)

Verification

  • 285 tests (up from 273). The headline test/integration/node_registration_logging_test.dart reproduces the field failure exactly — a non-node-role grant logs may not register and keeps retrying — plus unit tests for the logger's formatting/dedup and the probe timeout (a sleep 30 killed at a 300ms deadline).
  • Driven against the live Hub: a bad token → Connecting to… then error: Invalid token (no stack trace); an unreachable host → Node connection failed: … Connection refused instead of silence.
  • dart analyze --fatal-infos --fatal-warnings, dart format, dependency_validator clean.

Follow-up (separate)

An omnyhub 1.7.0 change — map a terminal-coded NodeErrorMessage to a typed exception so registration fails fast instead of waiting out a 10s timeout — is planned next. This omnyserver release already delivers the full user-visible fix against omnyhub ^1.6.0.

🤖 Generated with Claude Code

A node that authenticated and was then refused registration retried forever in
total silence — the failure that took a live packet trace and hours to trace to
a one-line misconfiguration (a grant with role `server`/`gpu` instead of `node`).

The cause was never a missing signal: the omnyhub node runtime already logged
the Hub's rejection and its reason — it even receives the Hub's
`NodeErrorMessage(code: forbidden, message: "principal … may not register
node …")` — but its logger defaults to a NoopLogger that OmnyServer never
replaced. Every diagnostic went to a silent sink.

- Wire a real logger (ConsoleNodeLogger) into the runtime, so what the Hub
  reports reaches the operator. Problems only by default; the full lifecycle
  under `node start --verbose`. Repeats within a short window are collapsed, so a
  node that keeps retrying a rejection it cannot fix says why once, not once per
  backoff — the chosen behaviour: stay up and recover when the grant is fixed.
- `node start` prints `Connecting to <url> …` before it blocks, and a terminal
  auth failure now exits with `error: <reason>` instead of an unhandled exception
  and a stack trace.
- The Hub logs a refused registration on its own side too, so journalctl shows
  it, not just the audit trail.
- Capability probes get a timeout. `CommandDetector` and the GPU detectors ran
  `Process.run` with no deadline, and the scanner awaits every probe before a
  node can register — so a single wedged command (a hung nvidia-smi) would have
  frozen registration silently. A timed-out probe is killed and treated as
  "not detected".

The headline test reproduces the field failure exactly: a node with a
non-`node`-role grant now logs `may not register`, and keeps retrying.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gmpassos
gmpassos merged commit 9f8482f into master Jul 15, 2026
5 checks passed
@gmpassos
gmpassos deleted the fix/node-start-observability branch July 15, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant