Converge the annex plumbing: required=true always, pinned where PATH cannot serve

CLAUDE.md

@@ -327,7 +327,7 @@ user owns:
 | `.python-version` | The exact patch of the interpreter `lc` is running on |
 | `uv.lock`, `.venv` | **Derived** — converged by correctness, not existence: `uv lock --check` / `uv sync --locked --exact --check` decide, then `uv lock` / `uv sync --locked --exact --compile-bytecode` repair |
 | `.gitignore` | One managed block of patterns; convergence ensures each is present |
-| `.git` + the annex | `git init` then `git annex init` — results are versioned in the project's own repository |
+| `.git` + the annex | `git init` then `git annex init` — results are versioned in the project's own repository — then **`filter.annex.required=true`**, the one key git-annex does not set and the difference between a loud refusal and silent corruption (see the storage invariants) |
 | `.gitattributes` | The storage policy: what git-annex holds and what git carries. Line-managed, like `.gitignore` |
 | `.datalad/config` | A `datalad.dataset.id` UUID, generated once. Read back only by `dataset.dataset_id`, through `git config -f`, for the run record's `dsid` |
 | `data/` + `README.md` | Where declared inputs live; annexed, and committed before anything computes on them |
@@ -342,8 +342,9 @@ user owns:
   Universes are discovered by `glob("*.yaml")`, which is empty-not-error on
   a missing directory. `tests/test_project.py::test_a_clone_of_a_converged_project_is_converged`
   pins this: a clone must need nothing but `.venv` and `git annex init`.
-  Those two are the exemptions, and for one reason — they are local state
-  git does not clone.
+  Those (with the annex filter, a `.git/config` entry) are the
+  exemptions, and for one reason — they are local state git does not
+  clone.
 - **Convergence, not scaffolding.** Each item is created if missing,
   offered to a conservative `repair(text) -> str | None` hook otherwise,
   and left alone when the hook returns `None`. `--check` computes the
@@ -545,6 +546,62 @@ for lc's subprocesses exactly as for their own git, and no different
 from which `git` itself runs. Nothing records the annex version anyway
 (see the Recorded decision on the engine's dependency closure).
 
+**"By construction" covers every install channel — except the
+researcher's own shell**, which is the one place `filter=annex` sends
+git looking for git-annex (the filter config `git annex init` writes is
+`PATH`-resolved). Under `uvx` nothing lands on the user's `PATH`, and a
+`git add` whose clean filter cannot start prints an error, **exits 0,
+and stages the raw bytes into git history** (measured, and pinned by
+`test_stock_plumbing_without_required_stages_raw_bytes_silently`). So
+dataset convergence sets one key as an item (`annex-filter`, 2026-08):
+**`filter.annex.required=true`, always** — git's own hard failure
+instead of silent corruption.
+
+**That flag is the whole of it, by decision.** lc writes no absolute
+path into a repository and rewrites no filter driver or hook: how git
+dispatches git-annex stays `PATH` resolution, which is git-annex's own
+design and git-lfs's too (it writes `filter.lfs.required = true` beside
+`PATH`-relative drivers and has never baked a path).
+
+**The hazard is Linux's, the guarantee is ours.** The exits-0-and-stages-raw-bytes
+behaviour is what Linux git does; macOS git aborts the same handshake
+outright (exit 128, `fatal: the remote end hung up unexpectedly`) and
+would have been safe without the flag — found by CI, which is why the
+suite runs on both. So `test_stock_plumbing_without_required_stages_raw_bytes_silently`
+is Linux-only by `skipif`, and the loud-refusal test asserts the facts
+(nonzero exit, nothing staged) rather than either platform's wording.
+Do not read macOS's accident as a reason to scope the flag: an
+implementation detail that can change on the next git release is not a
+guarantee, and Linux is where the venues are.
+
+The rule the neighbours agree on is **route through a filter ⇒ set
+`required`**, and both halves were measured. git-lfs routes and sets
+it. `datalad create` does *neither*: no `required`, and no
+`* filter=annex` in its `.gitattributes` (just `* annex.backend=MD5E`
+and `**/.git* annex.largefiles=nothing`) — so a plain `git add big.bin`
+in a DataLad dataset stages 200 000 raw bytes into git *even with
+git-annex working*, and the safeguard is procedural: use `datalad
+save`. lc made the opposite promise — the researcher types the git they
+already know — so lc owns the failure mode that promise creates. The
+honest cost, also measured: once a project holds committed annexed
+content, the refusal covers `git status`, `git diff` and `git checkout`
+too, not just `git add`. Pinning the filter
+and the four hooks to the engine's bundled executable was implemented
+and reverted before merge: it writes durable state pointing at a
+prunable `uvx` cache, `core.hooksPath` can send the hooks into a
+machine-shared directory that breaks commits in unrelated
+repositories, and uv documents `uvx` environments as disposable. The
+reachability problem is the *install's* to solve, not the
+repository's — `uv tool install lightcone-cli` links the git-annex
+wheel's entry points beside `lc` (verified), so `ambient` git-annex is
+on `PATH` for free and the whole question disappears. `uvx` is fine for
+running lc and cannot support the researcher's bare `git add`; that is
+what the docs and the troubleshooting entry say, and reporting it from
+`lc init`/`lc status` is an open follow-up, not a promise made here.
+`require_git_annex` stays a `PATH` check, deliberately — it gates lc's
+*own* `git annex` subprocesses, which dispatch from lc's environment,
+where every install channel fronts the bundled copy.
+
 **A project can sit inside a larger repository, so `dataset.status` is
 scoped and relativised.** `lc init subdir/` adopts an enclosing work
 tree rather than nesting a new one — that is a supported layout — and

docs/api/dataset.md

@@ -20,6 +20,8 @@ Source: `src/lightcone/engine/dataset.py` (+
 | `last_writer(root, dir)` | Who last touched an output's directory — the foreign-write question. Answers "cannot say" as empty, never an error. |
 | `require_committer(root)` | Refuses a repository with no git identity, before any recipe spends time. Asked as `git var`, the question a commit itself asks. |
 | `dataset_id(root)` | The DataLad dataset UUID, read via `git config -f`. |
+| `set_annex_filter_required(root)` | Set `filter.annex.required=true`, so a `git add` that cannot reach git-annex fails loudly instead of staging raw bytes. |
+| `annex_filter_required(root)` | Whether that flag is already set — `lc init --check`'s question. |
 
 ## What must stay true
 
@@ -49,6 +51,15 @@ Source: `src/lightcone/engine/dataset.py` (+
 - **Committing an archive or dot-named file needs `annex.dotfiles`** —
   git-annex routes dotfiles to git whatever `largefiles` says, and
   without the flag an image archive lands as a git blob, silently.
+- **`filter.annex.required=true` is the storage policy's safety net.**
+  Without it, a `git add` whose shell cannot resolve git-annex prints
+  an error, exits 0, and stages the raw bytes into git history —
+  measured, and pinned by
+  `test_stock_plumbing_without_required_stages_raw_bytes_silently`.
+  It is the *only* thing convergence adds to what `git annex init`
+  wrote: no filter driver is rewritten, and no hook is touched, so how
+  git dispatches git-annex stays git-annex's own business and stays
+  resolved from `PATH`.
 
 ## Tests
 

docs/api/project.md

@@ -38,9 +38,15 @@ Source: `src/lightcone/engine/project.py` (+
   `.gitattributes` are converged entry-wise, order judged against the
   template).
 - **Only what git can carry is converged.** No `src/`, no empty
-  directories — a clone must need nothing but `.venv` and
-  `git annex init`, and
-  `test_a_clone_of_a_converged_project_is_converged` pins it.
+  directories — a clone must need nothing but `.venv`, `git annex
+  init` and the annex filter (all three local state git does not
+  clone), and `test_a_clone_of_a_converged_project_is_converged` pins
+  it.
+- **The annex filter is one config key.** `filter.annex.required=true`,
+  always, so a `git add` that cannot reach git-annex refuses instead of
+  silently staging raw bytes. Nothing else about how git dispatches
+  git-annex is lc's to write: the filter drivers and hooks stay exactly
+  as `git annex init` left them, resolved from `PATH`.
 - **There is no discovery.** The invoked directory is the project or
   it is a clean error; every uv call carries an explicit `--project`.
 - **Templates are files** (`templates/files/*.tmpl`, `string.Template`

docs/architecture.md

@@ -104,6 +104,15 @@ git carries pointers and history, the annex carries bytes, and
 default; `data/` and `results/` opt out). A researcher only ever types
 ordinary `git add` / `git commit`.
 
+That ordinary `git add` dispatches git-annex from the *researcher's*
+`PATH`, and a shell that cannot resolve it stages the raw bytes into
+git history while exiting 0 — so `lc init` sets
+`filter.annex.required=true`, which makes git refuse loudly instead
+(every filtered command, not only `git add`).
+Getting git-annex onto that `PATH` is the install's job, not the
+repository's: `uv tool install lightcone-cli` puts it there alongside
+`lc`.
+
 Each output is committed with a **run record** — a `[DATALAD RUNCMD]`
 commit message whose `cmd` reconstructs the engine
 (`uv run --no-project --with lightcone-cli==<v>`) and re-executes the

docs/cli/init.md

@@ -25,7 +25,9 @@ manages, and never overwrites files you own:
 - **Repaired** — derived artifacts that have drifted: a `uv.lock` that
   no longer matches `pyproject.toml`, a `.venv` that no longer matches
   the lock, a managed `.gitignore` or `.gitattributes` entry that a
-  newer `lc` added. Repairs only ever append or rebuild derived state;
+  newer `lc` added, an annexed repository still missing the
+  `filter.annex.required` flag. Repairs only ever append or rebuild
+  derived state;
   hand-written lines are never reordered or removed.
 - **Blocked** — something convergence can see but must not fix by
   appending: a `.gitignore` rule that would silently swallow
@@ -80,6 +82,27 @@ Two things it deliberately does *not* create: a `src/` directory
 directories), and any dependency in `pyproject.toml` — the lock
 carries only what *your* analysis imports, added with `uv add`.
 
+Inside `.git`, convergence sets one configuration key — reported as the
+`annex-filter` item:
+
+- `filter.annex.required=true`, always. Without it, a `git add` whose
+  shell cannot find git-annex prints an error, **exits 0, and stages
+  the raw bytes into git history** — a 2 GB dataset in git proper, on
+  every clone, forever. With it, the same situation is a hard, loud
+  failure and nothing is staged. Once the project holds committed
+  annexed content that refusal covers every command that must run the
+  filter, `git status` and `git diff` included — a project you cannot
+  use until git-annex is back, rather than one that silently absorbed
+  your data.
+
+That is the only thing `lc init` adds to what `git annex init` wrote.
+How git finds git-annex is still ordinary `PATH` resolution, which is
+why `lc` should be installed with `uv tool install lightcone-cli` — it
+puts `git-annex` on your `PATH` alongside `lc`. If your `git add` ever
+refuses, see
+[`fatal: … clean filter 'annex' failed`](../user/troubleshooting.md#fatal-clean-filter-annex-failed)
+in the troubleshooting guide.
+
 ## Options
 
 | Option | Default | Effect |

docs/user/troubleshooting.md

@@ -114,6 +114,49 @@ read-only verbs (`lc status`, `--check`) never transfer data, so they
 report the fact instead. Fetch by hand only when you want the bytes
 for your own inspection.
 
+## "fatal: … clean filter 'annex' failed"
+
+```
+git-annex filter-process: line 1: git-annex: command not found
+error: could not read greeting from subprocess 'git-annex filter-process'
+error: initialization for subprocess 'git-annex filter-process' failed
+fatal: data/catalog.fits: clean filter 'annex' failed
+```
+
+Your shell's `PATH` has no `git-annex`, so git could not run the filter
+that turns a large file into an annex pointer. **Nothing was staged**,
+which is the point: without `filter.annex.required=true` — which
+`lc init` sets — git would have exited 0 and committed the raw bytes
+into history instead.
+
+Once a project holds committed annexed content, this is not limited to
+`git add`. Any command that has to run the filter over that content
+stops the same way, `git status`, `git diff` and `git checkout`
+included — so the whole project reads as broken until git-annex is back
+on your `PATH`. That is the intended shape of the failure: a repository
+you cannot use is recoverable in one command, and one that quietly
+absorbed a multi-gigabyte file is not.
+
+`git-annex` ships with `lc`, so a tool install puts both on your `PATH`:
+
+```bash
+uv tool install lightcone-cli
+git-annex version
+```
+
+If `lc` runs but `git-annex` does not, uv's tool directory is not on
+your `PATH` — run `uv tool update-shell` and open a new shell. Running
+`lc` through `uvx` puts nothing on your `PATH` at all, so a plain
+`git add` cannot work that way.
+
+This failure is deliberately loud. `lc init` sets
+`filter.annex.required=true` in every project precisely because
+without it git handles the same situation by printing the error,
+**exiting 0, and staging your data's raw bytes into git history** —
+committing a multi-gigabyte dataset into git proper, silently, where
+every clone carries it forever. A refused `git add` costs you one
+`lc init`; the silent version costs you the repository.
+
 ## "… and this is a NERSC login node"
 
 `lc materialize` executes recipes, and on centers `lc` recognizes it

src/lightcone/engine/dataset.py

@@ -102,6 +102,57 @@ def ignore_rule(directory: Path, path: str) -> str | None:
     return proc.stdout.splitlines()[0].split("\t")[0]
 
 
+# =============================================================================
+# The annex filter: a missing git-annex must not be silent
+# =============================================================================
+
+
+def set_annex_filter_required(directory: Path) -> None:
+    """Make git refuse, rather than corrupt, when the annex filter cannot run.
+
+    ``git annex init`` wires ``filter.annex.*`` to a bare ``git-annex``,
+    resolved from the ``PATH`` of whichever git runs — and the
+    researcher's shell, unlike lc's own environment, may hold none.
+    Measured: a ``git add`` whose clean filter cannot start prints an
+    error, **exits 0**, and stages the raw bytes into git history, so a
+    2 GB input lands in git proper on every clone forever. This flag is
+    what turns that into git's own hard failure instead
+    (``fatal: <path>: clean filter 'annex' failed``), which is the whole
+    difference between a loud stop and silent corruption.
+
+    Args:
+        directory: A directory inside the repository.
+    """
+    _git(["config", "filter.annex.required", "true"], cwd=directory)
+
+
+def annex_filter_required(directory: Path) -> bool:
+    """Whether *this repository* already carries that flag.
+
+    ``--local`` because the write is repository-local, and a probe that
+    read the merged config would answer for the *host* instead: a user
+    who once set ``filter.annex.required`` in ``~/.gitconfig`` would have
+    every project report converged while its own ``.git/config`` carried
+    nothing, leaving the protection behind the moment that repository is
+    used under another ``HOME`` — a clone, CI, a container. ``--type=bool``
+    because git's booleans are not one spelling: ``1``, ``yes``, ``on``
+    and a valueless key all mean true to the filter, and reading them as
+    drift would rewrite a repository that was already correct.
+
+    Args:
+        directory: A directory inside the repository.
+
+    Returns:
+        True if ``filter.annex.required`` is set, in this repository, to
+        anything git reads as true.
+    """
+    set_to = _ask(
+        ["config", "--local", "--get", "--type=bool", "filter.annex.required"],
+        cwd=directory,
+    )
+    return (set_to or "").strip() == "true"
+
+
 # =============================================================================
 # What a run does to the repository
 # =============================================================================

src/lightcone/engine/project.py

@@ -407,10 +407,18 @@ def _converge_dataset(c: _Converger, directory: Path) -> None:
     """
     c.item(".git", _in_repository(directory), lambda: dataset.init_git(directory))
     # After the item above, so a fresh project has a repository to annex.
+    # `annexed` is read *before* the item applies: in write mode the init
+    # runs inline, and the filter item below must still know whether this
+    # run created the annex (created) or found one (repaired).
+    annexed = _can_ask_git(directory) and dataset.is_annexed(directory)
+    c.item("git-annex", annexed, lambda: dataset.init_annex(directory))
+    # The one thing `git annex init` does not set, and the reason it
+    # matters is in `dataset.set_annex_filter_required`.
     c.item(
-        "git-annex",
-        _can_ask_git(directory) and dataset.is_annexed(directory),
-        lambda: dataset.init_annex(directory),
+        "annex-filter",
+        annexed,
+        lambda: dataset.set_annex_filter_required(directory),
+        is_current=lambda: dataset.annex_filter_required(directory),
     )
     attributes = directory / ".gitattributes"
     # Read before the repair, not after: the check is on the text a repair

tests/conftest.py

@@ -102,6 +102,10 @@ def tools(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
     # rather than on disk because `.git` is a *file* in a linked
     # worktree, so there is nowhere inside it to leave a marker.
     annexed: set[Path] = set()
+    # What `git config` has been told, per repository — the annex filter
+    # converges through a config write and a config read, and a fake that
+    # forgot the write would report the same item repaired forever.
+    config: dict[tuple[Path, str], str] = {}
 
     def fake_run(argv: list[str], *, cwd: Path) -> MagicMock:
         calls.append(list(argv))
@@ -120,6 +124,15 @@ def fake_run(argv: list[str], *, cwd: Path) -> MagicMock:
             annexed.add(_repo(cwd))
         elif argv[:2] == ["git", "config"] and argv[-1] == "annex.uuid":
             return MagicMock(returncode=0 if _repo(cwd) in annexed else 1)
+        elif argv[:3] == ["git", "config", "--local"] and "--get" in argv:
+            # The key is last; the flags in between are git's own reading
+            # conventions (`--type=bool`), which the store does not model
+            # because the engine only ever writes one value.
+            value = config.get((_repo(cwd), argv[-1]))
+            code = 0 if value is not None else 1
+            return MagicMock(returncode=code, stdout=(value or "") + "\n", stderr="")
+        elif argv[:2] == ["git", "config"] and len(argv) == 4 and not argv[2].startswith("-"):
+            config[(_repo(cwd), argv[2])] = argv[3]
         elif argv[:2] == ["git", "check-ignore"]:
             return _fake_check_ignore(cwd, argv[-1])
         return MagicMock(returncode=0, stdout="", stderr="")