Auto-migrate legacy config into workspace/config (scrub + backup) (8/13)

docs/config.md

@@ -18,11 +18,12 @@ inside `settings.yaml` is ignored.
 
 ## Which layer a key belongs in
 
-`nerve init` splits its answers between the first two layers:
+The first two layers divide as below. `nerve init` writes its answers this way,
+and migration splits a legacy `config.yaml` on the same table:
 
 | Layer | Gets |
 |-------|------|
-| `config.yaml` | `workspace`, `deployment`, `provider.aws_profile`, `gateway.ssl.*`, `proxy`, `docker`, `telegram.enabled`, `sync.gmail.accounts`, `external_agents`, `mcp_endpoint` |
+| `config.yaml` | `workspace`, `deployment`, `provider.aws_profile`, `gateway.ssl.*`, `proxy`, `docker`, `telegram.enabled`, `sync.gmail.accounts`, `external_agents`, `mcp_endpoint`, `workflows.runs_dir` |
 | `settings.yaml` | `timezone`, `gateway.host`/`port`, `provider.type`/`aws_region` (incl. the region-scoped Bedrock model IDs), `agent.*`, `memory.*`, `sessions.*`, `sync.*`, `houseofagents.*`, quiet hours, `telegram.dm_policy`/`stream_mode` |
 
 The test is whether the value would be wrong on another machine: filesystem
@@ -188,6 +189,73 @@ is too old to read shows up as a startup warning on the instance and in `nerve
 doctor`. If a key is ever retired upstream, `--strict-keys` flags it here first;
 drop the key, or install the ref you deploy (`nerve @ git+...@v1.2.3`) instead.
 
+## Migrating an Existing Install
+
+Installs from before the workspace-config layout are migrated automatically and
+idempotently on `nerve upgrade` and daemon start. To run it by hand:
+
+```bash
+nerve migrate --dry-run   # show what would change
+nerve migrate             # apply
+```
+
+Nothing is deleted and the effective configuration does not change; values are
+only relocated. Originals are renamed to `*.migrated` breadcrumbs, and an existing
+breadcrumb is never overwritten.
+
+- `config.yaml` is split rather than copied. Shareable keys go to the git-tracked
+  `workspace/config/settings.yaml`; secret values go to machine-local
+  `config.local.yaml`, replaced with `${ENV_VAR}` placeholders; the machine-local
+  keys from the table above are rewritten into `config.yaml`, so a certificate
+  path or an AWS profile handle never travels with the workspace repo. The split
+  follows the table at whatever depth a key is listed, so `gateway.ssl.*` stays
+  local while `gateway.host`/`port` move across. Migration prints which keys it
+  kept. The `workspace` path is the exception that stays in `config.local.yaml`:
+  it is written before anything is consumed, so an interrupted migration cannot
+  leave the instance unable to find its workspace.
+- The legacy `~/.nerve/cron/*` moves to `workspace/config/cron/*`, including any
+  `prompts/` referenced by `prompt_file`.
+- `config.local.yaml`, the rewritten `config.yaml` and the breadcrumb can all hold
+  plaintext credentials, so all three are written `0600`. `settings.yaml` is
+  written normally, so a restrictive `umask` is honored.
+
+A value is treated as secret when any of these hold:
+
+- The key name looks like one: `*api_key*`, `*api_hash*`, `api_id`, `*token*`,
+  `*secret*`, `password*`, `jwt`, `authorization`, `bearer`, `oauth`,
+  `*access_key*`, `*private_key*`, `dsn`, `pw`, `pat`, `session_string`,
+  `webhook_url`. A public identifier is not a secret: `client_id` is left alone,
+  `client_secret` is not.
+- The value's shape is a credential whatever the key is called: `sk-…`, `ghp_…`,
+  `xox…`, `user:password@host`, `?token=…`, `Bearer …`. This fires only when the
+  whole value is the credential, so a category description that mentions
+  `postgres://user:pass@host` stays put.
+- It sits inside an `env` or `headers` block, including inside lists, where MCP
+  `headers` entries live.
+- It is the argument after a flag that names a credential, so both
+  `args: ["--token=abc123"]` and the split `args: ["--token", "abc123"]` are
+  caught. The split form is what `npx` and `uvx` MCP servers use, and the value
+  on its own is unrecognizable.
+
+If one item in a list is a secret, the whole list moves, because a merge replaces
+a list rather than combining it element-wise. Migration reports this, and the copy
+left in `settings.yaml` has no further effect.
+
+Secret detection is best-effort, so **review `workspace/config/settings.yaml`
+before committing it to a shared repo**, then run `nerve config validate` to
+confirm the bundle is well-formed. Migration also lists any value it left in the
+tracked file that still looks like a credential, for you to judge.
+
+Migration only runs on a pre-refactor `config.yaml`, meaning one that holds
+shareable settings rather than only this box's. A `config.yaml` written by
+`nerve init` — or left behind by an earlier migration — holds only machine-local
+keys (workspace, certificate paths, provider handles), so it is left in place even
+when the workspace has no `settings.yaml`. Migration is also a no-op once
+`settings.yaml` carries real keys; the `nerve init` scaffold is all comments and
+counts as empty. A `config.yaml` with shareable keys sitting next to a populated
+`settings.yaml` is reported, because it overrides the tracked file, and you move
+those keys across by hand.
+
 ## Config Directory Resolution
 
 `nerve` commands locate the config directory via a waterfall, so they work

nerve/agent/backends/codex/ultracode.py

@@ -21,6 +21,8 @@
 from pathlib import Path
 from typing import Any
 
+from nerve.utils.fs import atomic_write_text
+
 logger = logging.getLogger(__name__)
 
 _MARKETPLACE = "nerve-ultracode"
@@ -93,15 +95,6 @@ def managed_dir(home: str | Path) -> Path:
     return Path(home).expanduser() / "nerve-managed" / "ultracode"
 
 
-def _atomic_write(path: Path, content: str, mode: int = 0o600) -> None:
-    path.parent.mkdir(parents=True, exist_ok=True)
-    tmp = path.with_name(f".{path.name}.{os.getpid()}.tmp")
-    tmp.write_text(content, encoding="utf-8")
-    os.chmod(tmp, mode)
-    tmp.replace(path)
-    os.chmod(path, mode)
-
-
 def materialize_worker_wrapper(home: str | Path) -> Path:
     """Write an owner-only executable pointing at Nerve's wrapper module."""
     path = managed_dir(home) / "codex-worker"
@@ -110,7 +103,7 @@ def materialize_worker_wrapper(home: str | Path) -> Path:
         "from nerve.agent.backends.codex.worker_wrapper import main\n"
         "raise SystemExit(main())\n"
     )
-    _atomic_write(path, script, mode=0o700)
+    atomic_write_text(path, script, mode=0o700)
     return path
 
 
@@ -123,7 +116,7 @@ def _replace_overlay(path: Path, old: str, new: str) -> None:
             f"Pinned Ultracode source drifted; policy overlay does not match {path}"
         )
     mode = path.stat().st_mode & 0o777
-    _atomic_write(path, content.replace(old, new), mode=mode)
+    atomic_write_text(path, content.replace(old, new), mode=mode)
 
 
 def _sha256(path: Path) -> str:
@@ -156,7 +149,7 @@ def apply_policy_overlay(config: Any, plugin_root: str | Path) -> dict[str, Any]
             "scripts/ultracode-script-runner.js": _sha256(script_runner),
         },
     }
-    _atomic_write(
+    atomic_write_text(
         managed_dir(config.codex.home_dir) / "policy-overlay.json",
         json.dumps(marker, indent=2, sort_keys=True) + "\n",
     )
@@ -318,7 +311,7 @@ async def ensure_installed(config: Any) -> dict[str, Any]:
 
         root = managed_dir(config.codex.home_dir) / "marketplace"
         manifest_path = root / ".agents" / "plugins" / "marketplace.json"
-        _atomic_write(
+        atomic_write_text(
             manifest_path,
             json.dumps(_marketplace_manifest(config), indent=2) + "\n",
         )

nerve/cli.py

@@ -172,7 +172,7 @@ def _docker_compose(
 # Commands that must keep working when the config itself is broken -- they are
 # how an operator diagnoses or repairs it. They receive ctx.obj["config"] = None
 # and ctx.obj["config_error"], and are responsible for reporting.
-_SELF_DIAGNOSING_COMMANDS = frozenset({"config", "doctor", "init"})
+_SELF_DIAGNOSING_COMMANDS = frozenset({"config", "doctor", "init", "migrate"})
 
 
 @click.group()
@@ -276,6 +276,25 @@ def start(ctx: click.Context, foreground: bool) -> None:
     config_dir = Path(ctx.obj["config_dir"])
     config = ctx.obj["config"]
 
+    # Migrate a legacy install to the workspace/config layout if needed
+    # (idempotent, best-effort). Non-destructive — originals kept as *.migrated.
+    if config is not None:
+        from nerve.migrate import maybe_migrate
+        report = maybe_migrate(config_dir, workspace=getattr(config, "workspace", None))
+        if report is not None and report.did_anything:
+            # The config in hand was loaded from the old locations, and migration
+            # has just moved the files out from under it — cron in particular now
+            # lives in the workspace, so the stale object resolves to job files
+            # that no longer exist and a --foreground run would schedule nothing.
+            try:
+                config = load_config(config_dir)
+            except Exception as e:
+                raise click.ClickException(
+                    f"Config could not be reloaded after migration: {e}"
+                ) from e
+            set_config(config)
+            ctx.obj["config"] = config
+
     # Detect fresh install — offer to run setup wizard
     from nerve.bootstrap import is_fresh_install
     if is_fresh_install(config_dir):
@@ -726,6 +745,47 @@ def upgrade(ctx: click.Context, no_frontend: bool, no_deps: bool, no_pull: bool)
         if rc != 0:
             raise click.ClickException("npm run build failed")
 
+    # Migrate a legacy config layout to workspace/config (idempotent).
+    if config is not None:
+        from nerve.migrate import maybe_migrate
+        report = maybe_migrate(
+            Path(ctx.obj["config_dir"]), workspace=getattr(config, "workspace", None)
+        )
+        if report and report.did_anything:
+            click.echo("\nMigrated config to the workspace layout:")
+            for action in report.actions:
+                click.echo(f"  - {action}")
+            if report.error:
+                # The steps above did apply. Saying so beats a bare success
+                # message, because the rest has to be finished by hand or by a
+                # re-run, and the tracked file already exists either way.
+                click.secho(
+                    f"  Migration did not finish: {report.error}\n"
+                    "  What is listed above was applied; re-run `nerve migrate` "
+                    "once the cause is fixed.",
+                    fg="yellow",
+                )
+            if report.secrets_moved:
+                click.echo(
+                    f"  (scrubbed {len(report.secrets_moved)} secret(s) into "
+                    "config.local.yaml; review workspace/config/settings.yaml "
+                    "before committing)"
+                )
+            if report.machine_local_kept:
+                click.echo(
+                    f"  ({len(report.machine_local_kept)} machine-local key(s) kept "
+                    "in config.yaml instead of published: "
+                    f"{', '.join(report.machine_local_kept)})"
+                )
+            if report.suspect_values:
+                click.secho(
+                    f"  ({len(report.suspect_values)} value(s) left in settings.yaml "
+                    f"still look like credentials: {', '.join(report.suspect_values)})",
+                    fg="yellow",
+                )
+            for warning in report.warnings:
+                click.secho(f"  Note: {warning}", fg="yellow")
+
     click.echo("\nUpgrade complete. Restart Nerve for changes to take effect:")
     click.echo("  nerve restart")
 
@@ -1092,6 +1152,56 @@ def doctor(ctx: click.Context) -> None:
         ctx.exit(1)
 
 
+@main.command()
+@click.option("--dry-run", is_flag=True, help="Show what would change without writing.")
+@click.pass_context
+def migrate(ctx: click.Context, dry_run: bool) -> None:
+    """Migrate a legacy config layout into the workspace/config subtree.
+
+    Splits config.yaml: shareable keys → workspace/config/settings.yaml (secrets
+    scrubbed into config.local.yaml as ${ENV_VAR} refs), machine-local keys stay
+    in config.yaml. Also moves ~/.nerve/cron → workspace/config/cron.
+    Non-destructive (originals kept as *.migrated) and idempotent.
+    """
+    from nerve.migrate import migrate as run_migrate
+
+    config = ctx.obj["config"]
+    config_dir = Path(ctx.obj["config_dir"])
+    workspace = getattr(config, "workspace", None) if config is not None else None
+    try:
+        report = run_migrate(config_dir, workspace=workspace, dry_run=dry_run)
+    except Exception as e:  # e.g. a malformed config.local.yaml
+        raise click.ClickException(f"Migration failed: {e}") from e
+
+    if not report.did_anything:
+        click.secho("Nothing to migrate — already on the workspace layout.", fg="green")
+        for warning in report.warnings:
+            click.secho(f"  Note: {warning}", fg="yellow")
+        return
+    prefix = "[dry-run] would " if dry_run else ""
+    for action in report.actions:
+        click.echo(f"  {prefix}{action}")
+    for warning in report.warnings:
+        click.secho(f"\n  Note: {warning}", fg="yellow")
+    if report.secrets_moved:
+        click.echo(f"\n  Secrets scrubbed to config.local.yaml: {', '.join(report.secrets_moved)}")
+    if report.machine_local_kept:
+        click.echo(
+            "\n  Machine-local keys kept in config.yaml: "
+            f"{', '.join(report.machine_local_kept)}"
+        )
+    if report.suspect_values:
+        click.secho(
+            f"\n  {len(report.suspect_values)} value(s) headed for settings.yaml still "
+            f"look like credentials — review: {', '.join(report.suspect_values)}",
+            fg="yellow",
+        )
+    if dry_run:
+        click.secho("\nDry run — no changes written.", fg="yellow")
+    else:
+        click.secho("\nMigration complete. Review workspace/config/settings.yaml before committing.", fg="green")
+
+
 @main.group(name="config")
 def config_group() -> None:
     """Configuration commands."""