Broaden hot-reload: unified reload_all + POST /api/config/reload (12/13)

config.example.yaml

@@ -133,7 +133,8 @@ memory:
 # value wins outright, so it disables both the workspace location and the
 # fallback, and `nerve init` will keep regenerating a system.yaml nothing reads.
 #
-# Editing the cron files does not apply itself: POST /api/cron/reload does.
+# Editing the cron files does not apply itself: run `nerve reload`, or let the
+# next workspace sync do it.
 
 # Workflow runs — budget-capped multi-agent jobs (Claude Workflow tool or
 # Codex Ultracode) in dedicated tracked sessions. Nerve meters real dollar

docs/config.md

@@ -22,8 +22,8 @@ inside `settings.yaml` is ignored.
 
 | Layer | Gets |
 |-------|------|
-| `config.yaml` | `workspace`, `deployment`, `provider.aws_profile`, `gateway.ssl.*`, `proxy`, `docker`, `telegram.enabled`, `sync.gmail.accounts`, `external_agents`, `mcp_endpoint` |
-| `settings.yaml` | `timezone`, `gateway.host`/`port`, `provider.type`/`aws_region` (incl. the region-scoped Bedrock model IDs), `agent.*`, `memory.*`, `sessions.*`, `sync.*`, `houseofagents.*`, quiet hours, `telegram.dm_policy`/`stream_mode` |
+| `config.yaml` | `workspace`, `deployment`, `provider.aws_profile`, `gateway.ssl.*`, `proxy`, `docker`, `telegram.enabled`, `sync.gmail.accounts`, `external_agents`, `mcp_endpoint`, `workflows.runs_dir` |
+| `settings.yaml` | `timezone`, `gateway.host`/`port`, `provider.type`/`aws_region` (incl. the region-scoped Bedrock model IDs), `agent.*`, `memory.*`, `sessions.*`, `sync.*`, the rest of `workflows.*` (the budget caps and cadence), `houseofagents.*`, quiet hours, `telegram.dm_policy`/`stream_mode` |
 
 The test is whether the value would be wrong on another machine: filesystem
 paths, credential handles, whose mailboxes this person syncs, which agent
@@ -185,6 +185,118 @@ is too old to read shows up as a startup warning on the instance and in `nerve
 doctor`. If a key is ever retired upstream, `--strict-keys` flags it here first;
 drop the key, or install the ref you deploy (`nerve @ git+...@v1.2.3`) instead.
 
+## Hot-Reload
+
+Many config changes apply without a restart. Not all do, and a config file
+changing on disk is not by itself a reload.
+
+### What triggers a reload
+
+A reload is always explicit. Two things cause one:
+
+- **`nerve reload`**, or `POST /api/config/reload` directly. Re-reads all three
+  layers and reloads every reloadable subsystem at once. This is the one to use
+  after editing a file on the box.
+- **A workspace sync that merged something.** Applies exactly the same set, so a
+  reviewed config PR takes effect on the instance without anyone logging in.
+  `workspace_sync.interval_minutes` (default 1) bounds how long that takes.
+
+### What a reload applies
+
+| Change | Reloaded? |
+|--------|-----------|
+| Cron jobs (`config/cron/*.yaml`) | ✅ |
+| Custom cron gate plugins (new, edited *and* deleted `.py` files) | ✅ the registry is rebuilt from the directory |
+| Cron file locations (`cron.jobs_file`, `system_file`, `gate_plugins_dir`) | ✅ re-read from the new config |
+| Cron sources: `sync.telegram`, `.gmail`, `.github`, `.github_events`, `.github_repos`, `.message_ttl_days`, and each source's `schedule` | ✅ runners are rebuilt and rescheduled. **`sync.codex` is not one of these**; see the restart table |
+| MCP servers (`mcp_servers`) | ✅ new sessions get the new set |
+| Skills (`skills/`) | ✅ re-scanned |
+| `lockdown` | ✅ the write guards and the layer stack both follow |
+| Web gateway auth (`auth.*`) | ✅ read per request |
+| `notifications.*` | ✅ read per notification |
+| `workspace_sync.*` | ✅ from the next sync cycle |
+| `retention.*`, `backup.*`, and the `sessions.*` the background loops read | ✅ from the next cycle of that loop |
+| `external_agents.targets` (including each target's `enabled`), `.sync_interval_minutes`, `.conflict_policy` | ✅ from the next sweep, provided at least one target existed at startup (see the restart table) |
+| `sessions.sticky_period_minutes` | ✅ |
+| `provider.*` and the API keys it selects (`aws_region`, `aws_profile`, `aws_access_key_id`, and the effective Anthropic key) | ✅ for sessions started **after** the reload. Each client's environment is built from the live reference when the session is created, by the same seam as `agent.*` below |
+| **`agent.*` and `codex.*`**: backend choice and models (`agent.backend`, `agent.cron_model`, `agent.model`, `codex.model`, `codex.cron_model`), `max_turns`, `agent.effort`/`cron_effort` and `codex.effort_map`, `agent.thinking`, `agent.context_1m*`, `agent.background_agent_permissions`, idle timeouts, cache TTL, `codex.sandbox`, `.approval_policy`, `.web_search`, `.extra_config`, `.tool_timeout_sec`, `.bin_path`, `.auth`/`.api_key`/`.api_key_env`, `.pricing`, `.min_version`/`.max_version`, `.ultracode.*` | ✅ for sessions and turns **started after** the reload. The engine and both backends resolve these through one live reference, so a key cannot be hot in one and frozen in the other |
+
+All of that is reloaded together, and the response says what happened to each
+piece: `POST /api/config/reload` returns `ok`, a per-subsystem `detail`, and an
+`errors` map. A reload is best-effort by design, because a typo in `settings.yaml`
+must not stop a valid cron edit from being applied, so `ok: false` with `detail`
+showing four subsystems reloaded and one failed is a normal answer. Check `errors`
+rather than reading the 200 as success.
+
+### What still needs a restart
+
+A reload compares the old and new config and reports any of these that changed, as
+`restart_required` on `POST /api/config/reload` and as a log warning. Without that
+report a reload returns success while the process keeps the old value, which
+matters most for `gateway.host`/`port`: they live in `settings.yaml`, so the change
+can arrive by workspace sync rather than by a local edit.
+
+The check covers the unconditional entries below. The conditional ones (turning X
+on, adding the first target, a session already running) depend on runtime state the
+reload cannot inspect, and are documented here only.
+
+| Change | Why |
+|--------|-----|
+| `gateway.host`, `.port`, `.ssl.*` | the socket is already bound |
+| `timezone` | the cron scheduler and every trigger built from it carry the old zone |
+| `agent.max_concurrent` | its semaphore cannot be resized under in-flight turns |
+| `workspace` | the skill manager, the tool context, the memory bridges and each session's working directory all captured it at startup. Following it in one of them and not the others would be worse than not following it at all |
+| `memory.*`, `xmemory.*` | the bridges hold the config they were constructed with |
+| `codex.home_dir` | half-hot, which is why it is here: new sessions are handed the new `CODEX_HOME`, but the directory is only created when the backend is built, so nothing creates the new one. Change it and restart, rather than leaving sessions pointed somewhere that may not exist |
+| `sync.codex.*` (`enabled`, every `origins[*]` field, `store_encrypted_reasoning`, `workspace_filter.*`) | Codex thread sync is a **different service** from the cron sources above, built once at startup with one polling worker per origin. Adding or editing an origin and reloading reports `ok` and ingests nothing |
+| `langfuse.*` | set up before the engine, caching its host, redaction patterns and `LANGFUSE_*` environment exports in process globals |
+| `telegram.*`, including `allowed_users` | the bot is running with the allow-list it was started with. Notification *delivery* does follow a reload, so after changing `allowed_users` the two can disagree until a restart |
+| `mcp_endpoint.*`, including the token check on `/mcp/v1` | fixed when the app was created. This is *not* the same code as the web gateway's auth above, which is hot |
+| `proxy.*` | the proxy process is started at startup, so turning it on, turning it off or moving its port needs one. The backend does read the proxy host and port per session, so those can point somewhere nothing is listening until you restart |
+| Turning `ollama.enabled` **on** while the proxy is not already running | Ollama routes through the proxy as its translation layer, and that process only starts at startup. With the proxy already up, this is read per use and follows a reload |
+| Turning `workspace_sync.enabled` or `retention.enabled` **on** | their loops are only created at startup, so there is nothing running to see the flag change. Turning either **off** is hot |
+| `external_agents.enabled` (**both** directions) and adding the **first** target | the sweeper is only created when the flag is on *and* at least one target is configured; with none it is never created, so a first target added later reloads to `ok` and does nothing (`POST /api/external-agents/sync` answers 503). Once it exists, adding, removing and toggling targets is hot |
+| A session that is **already running** | the agent process was spawned with the options in force at the time; the new ones apply to the next session |
+
+### From the command line
+
+```bash
+nerve reload    # apply config edits to the running daemon
+```
+
+It calls the endpoint above on this box's own gateway and prints the per-subsystem
+result. It exits non-zero on a partial reload, so it can gate a deploy step rather
+than only informing a human. Anything in the restart table that changed is printed
+as a warning: nothing failed, but the new value is not live yet.
+
+With no daemon running there is nothing to reload and the command says so. Config
+is read fresh at startup, so `nerve start` already picks the edit up.
+
+It authenticates the way the gateway asks to be authenticated, which depends on
+`auth.jwt_secret` in the config it just read:
+
+- **Set** → it signs a token with it. If that is not the secret the running daemon
+  started with, the gateway rejects the request and only a restart resolves it.
+- **Empty, unlocked** → the gateway is in dev mode and does not ask for a token
+  (`require_auth` runs open), so the call goes unauthenticated.
+- **Empty, locked** → refused before anything is sent. A locked gateway never takes
+  the open path, so no request from that shell can be authenticated. If the secret
+  comes from `${ENV_VAR}`, export it in that shell too.
+
+`auth.password_hash` is not an alternative here. It gates the browser login, which
+is what mints a token from it; `require_auth` reads `auth.jwt_secret` alone, so a
+password neither makes the endpoint ask for a token nor gives the CLI one to sign.
+
+`POST /api/config/sync` runs the same reload but scores it differently, because it
+answers a different question. Its `ok` is about the *merge*: true once the merged
+config is loaded and in effect, false only when the daemon could not load it, in
+which case the merge applied nothing at all. A subsystem that failed *after* the
+config loaded leaves `ok: true`, because the merged settings really are live, and
+shows up as `applied: false` with the reason in `reload_errors`. So the same skills
+failure gives `ok: false` on `/api/config/reload` and `ok: true, applied: false` on
+`/api/config/sync`. **On the sync endpoint, read `applied`, not `ok`**, unless what
+you want to know is specifically whether the merge took.
+
 ## Git-Backed Workspace Sync
 
 The workspace can be a git repository whose remote is a shared **config repo**.
@@ -213,10 +325,13 @@ Each sync is fetch, then validate, then fast-forward merge. The fetched bundle i
 validated in a throwaway git worktree and the live working tree is fast-forwarded
 only if that passes, so an invalid bundle never lands on disk and there is nothing
 for a later reload or restart to pick up (`POST /api/config/sync` returns 400 and
-leaves the workspace untouched). A pull that changed something reloads cron and MCP
-config, so the merged change takes effect immediately. CI on the PR is still the
-first line of defense. The remote and credentials come from git itself, so
-configure `git remote` and auth in the workspace as usual.
+leaves the workspace untouched). A pull that changed something runs the same reload
+as `nerve reload`, so see [Hot-Reload](#hot-reload) for what that covers; the
+response names which subsystems took the merged change (`applied`, `reload`,
+`reload_errors`). A merge whose config the daemon then refuses to load has applied
+nothing and reports `ok: false`. CI on the PR is still the first line of defense.
+The remote and credentials come from git itself, so configure `git remote` and auth
+in the workspace as usual.
 
 **Keep the config subtree clean.** Sync refuses to merge while
 `<workspace>/config/` has local changes: an edited or deleted tracked file, a
@@ -240,10 +355,13 @@ use `workspace_sync.strict_env: false` rather than letting every sync fail.
 for a one-off. Warnings never block a merge: an unrecognized cron gate type, an
 unknown key, or a skipped validation.
 
-`workspace_sync` changes need a daemon restart. The sync loop reads the current
-config object every cycle, so it adds no staleness of its own, but nothing refreshes
-that object while the process runs. Turning `enabled` on needs a restart in any
-case, because the sync task is only created at startup.
+`workspace_sync` changes apply from the next cycle, once a reload has run. The sync
+loop reads the current config object every cycle rather than a copy taken at
+startup, so an edit to `branch`, `interval_minutes`, `validate` or `strict_env`
+reaches it as soon as something replaces that object: a sync that merged a change,
+or `POST /api/config/reload` after you edited the file yourself. Turning `enabled`
+**on** still needs a restart, because the sync task is only created at startup.
+Turning it off is picked up like any other value.
 
 ## Lockdown (remote-only, read-only)
 
@@ -391,12 +509,12 @@ warns when it sees an env-controlled flag resolve to false, so the gap is at lea
 visible.
 
 **Flipping the flag needs the config reloaded.** A sync does that, and so does
-`POST /api/config/reload`. The write guards, gateway authentication and the sync
-loop pick the change up immediately; cron jobs and gates already built follow on the
-next cron reload, and anything captured at startup needs a restart. A hand edit on
-the box refreshes nothing, and under lockdown it is ignored anyway. If a sync merges
-a config the daemon then cannot load, it reports `ok: false` with `apply_error`
-rather than claiming the change took.
+`POST /api/config/reload` — both run the same reload. The write guards, gateway
+authentication and the sync loop pick the change up immediately; cron jobs and gates
+already built follow on the next cron reload, and anything captured at startup needs
+a restart. A hand edit on the box refreshes nothing, and under lockdown it is
+ignored anyway. If a sync merges a config the daemon then cannot load, it reports
+`ok: false` with `apply_error` rather than claiming the change took.
 
 ### What lockdown does not cover
 

docs/cron.md

@@ -58,7 +58,9 @@ Notes:
   registered gates as they were, so a deleted plugin file only takes its gate
   away once a reload succeeds.
 - A job holding a [reserved id](#job-fields) is **skipped**, at reload and at
-  startup alike. The reload itself still succeeds; only that job is dropped.
+  startup alike. The reload itself still succeeds; only that job is dropped, and
+  its id comes back in the reload's `rejected` list, so a job that never runs is
+  visible from the API and not only in the log.
 - An **invalid schedule** (a crontab whose fields the scheduler rejects, e.g.
   `99 * * * *`) is refused the same way at reload: `400`, nothing applied.
   Startup is the one case that differs — the daemon logs an error naming the job

nerve/agent/backends/__init__.py

@@ -45,9 +45,17 @@ class BackendDeps:
 
     Callables (rather than direct references) where the underlying value
     is hot-reloadable or wired up after engine construction.
+
+    ``config`` is one of the hot ones. A config reload replaces the process
+    config object; a backend that captured the old one would keep sending the
+    options the operator has just changed, while the engine — which *is*
+    re-pointed — reported the change applied and rendered it in the UI. One key
+    live on one side of that seam and frozen on the other is the hardest state
+    to diagnose there is, so backends resolve config per read instead of holding
+    it.
     """
 
-    config: "NerveConfig"
+    config: Callable[[], "NerveConfig"]
     db: "Database"
     registry: Any                                   # ToolRegistry
     tool_ctx_factory: Callable[[str], Any]          # session_id -> ToolContext

nerve/agent/backends/claude.py

@@ -367,7 +367,18 @@ class ClaudeBackend:
 
     def __init__(self, deps: Any):
         self._deps = deps
-        self.config = deps.config
+
+    @property
+    def config(self):
+        """The live config, resolved per read rather than captured.
+
+        Everything below builds SDK options straight off this — the model, the
+        thinking budget, the 1M-context header, the sub-agent permission mode.
+        Holding a reference would freeze all of them at start-up while the
+        engine's own copy moved on with a reload, so the context bar could show
+        a 200k budget for sessions still being opened with the 1M header.
+        """
+        return self._deps.config()
 
     # -- policy -------------------------------------------------------- #
 

nerve/agent/backends/codex/backend.py

@@ -127,14 +127,30 @@ class CodexBackend:
 
     def __init__(self, deps: Any):
         self._deps = deps
-        self.config = deps.config
-        self.codex = deps.config.codex
         Path(self._home_dir()).mkdir(parents=True, exist_ok=True)
         self._preflight_cache: tuple[float, dict[str, Any]] | None = None
         self._preflight_lock = asyncio.Lock()
         self._live_models: set[str] = set()
         self._rate_limits: dict[str, Any] | None = None
 
+    @property
+    def config(self):
+        """The live config, resolved per read rather than captured."""
+        return self._deps.config()
+
+    @property
+    def codex(self):
+        """The live ``codex`` section.
+
+        A property and not an attribute for the same reason as :attr:`config`,
+        and more sharply: a sub-object caches one level deeper, so re-pointing
+        ``config`` alone would leave the sandbox mode, approval policy, effort
+        map and auth choice on their start-up values while everything read off
+        ``config`` had moved. Resolving through the deps callable means there is
+        no second place for the two to disagree.
+        """
+        return self._deps.config().codex
+
     # -- policy ---------------------------------------------------------- #
 
     def default_model(self, source: str) -> str: