diff --git a/CLAUDE.md b/CLAUDE.md index 7c73800..246ff4b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -327,7 +327,7 @@ user owns: | `.python-version` | The exact patch of the interpreter `lc` is running on | | `uv.lock`, `.venv` | **Derived** — converged by correctness, not existence: `uv lock --check` / `uv sync --locked --exact --check` decide, then `uv lock` / `uv sync --locked --exact --compile-bytecode` repair | | `.gitignore` | One managed block of patterns; convergence ensures each is present | -| `.git` + the annex | `git init` then `git annex init` — results are versioned in the project's own repository | +| `.git` + the annex | `git init` then `git annex init` — results are versioned in the project's own repository — then **`filter.annex.required=true`**, the one key git-annex does not set and the difference between a loud refusal and silent corruption (see the storage invariants) | | `.gitattributes` | The storage policy: what git-annex holds and what git carries. Line-managed, like `.gitignore` | | `.datalad/config` | A `datalad.dataset.id` UUID, generated once. Read back only by `dataset.dataset_id`, through `git config -f`, for the run record's `dsid` | | `data/` + `README.md` | Where declared inputs live; annexed, and committed before anything computes on them | @@ -342,8 +342,9 @@ user owns: Universes are discovered by `glob("*.yaml")`, which is empty-not-error on a missing directory. `tests/test_project.py::test_a_clone_of_a_converged_project_is_converged` pins this: a clone must need nothing but `.venv` and `git annex init`. - Those two are the exemptions, and for one reason — they are local state - git does not clone. + Those (with the annex filter, a `.git/config` entry) are the + exemptions, and for one reason — they are local state git does not + clone. - **Convergence, not scaffolding.** Each item is created if missing, offered to a conservative `repair(text) -> str | None` hook otherwise, and left alone when the hook returns `None`. `--check` computes the @@ -545,6 +546,62 @@ for lc's subprocesses exactly as for their own git, and no different from which `git` itself runs. Nothing records the annex version anyway (see the Recorded decision on the engine's dependency closure). +**"By construction" covers every install channel — except the +researcher's own shell**, which is the one place `filter=annex` sends +git looking for git-annex (the filter config `git annex init` writes is +`PATH`-resolved). Under `uvx` nothing lands on the user's `PATH`, and a +`git add` whose clean filter cannot start prints an error, **exits 0, +and stages the raw bytes into git history** (measured, and pinned by +`test_stock_plumbing_without_required_stages_raw_bytes_silently`). So +dataset convergence sets one key as an item (`annex-filter`, 2026-08): +**`filter.annex.required=true`, always** — git's own hard failure +instead of silent corruption. + +**That flag is the whole of it, by decision.** lc writes no absolute +path into a repository and rewrites no filter driver or hook: how git +dispatches git-annex stays `PATH` resolution, which is git-annex's own +design and git-lfs's too (it writes `filter.lfs.required = true` beside +`PATH`-relative drivers and has never baked a path). + +**The hazard is Linux's, the guarantee is ours.** The exits-0-and-stages-raw-bytes +behaviour is what Linux git does; macOS git aborts the same handshake +outright (exit 128, `fatal: the remote end hung up unexpectedly`) and +would have been safe without the flag — found by CI, which is why the +suite runs on both. So `test_stock_plumbing_without_required_stages_raw_bytes_silently` +is Linux-only by `skipif`, and the loud-refusal test asserts the facts +(nonzero exit, nothing staged) rather than either platform's wording. +Do not read macOS's accident as a reason to scope the flag: an +implementation detail that can change on the next git release is not a +guarantee, and Linux is where the venues are. + +The rule the neighbours agree on is **route through a filter ⇒ set +`required`**, and both halves were measured. git-lfs routes and sets +it. `datalad create` does *neither*: no `required`, and no +`* filter=annex` in its `.gitattributes` (just `* annex.backend=MD5E` +and `**/.git* annex.largefiles=nothing`) — so a plain `git add big.bin` +in a DataLad dataset stages 200 000 raw bytes into git *even with +git-annex working*, and the safeguard is procedural: use `datalad +save`. lc made the opposite promise — the researcher types the git they +already know — so lc owns the failure mode that promise creates. The +honest cost, also measured: once a project holds committed annexed +content, the refusal covers `git status`, `git diff` and `git checkout` +too, not just `git add`. Pinning the filter +and the four hooks to the engine's bundled executable was implemented +and reverted before merge: it writes durable state pointing at a +prunable `uvx` cache, `core.hooksPath` can send the hooks into a +machine-shared directory that breaks commits in unrelated +repositories, and uv documents `uvx` environments as disposable. The +reachability problem is the *install's* to solve, not the +repository's — `uv tool install lightcone-cli` links the git-annex +wheel's entry points beside `lc` (verified), so `ambient` git-annex is +on `PATH` for free and the whole question disappears. `uvx` is fine for +running lc and cannot support the researcher's bare `git add`; that is +what the docs and the troubleshooting entry say, and reporting it from +`lc init`/`lc status` is an open follow-up, not a promise made here. +`require_git_annex` stays a `PATH` check, deliberately — it gates lc's +*own* `git annex` subprocesses, which dispatch from lc's environment, +where every install channel fronts the bundled copy. + **A project can sit inside a larger repository, so `dataset.status` is scoped and relativised.** `lc init subdir/` adopts an enclosing work tree rather than nesting a new one — that is a supported layout — and diff --git a/docs/api/dataset.md b/docs/api/dataset.md index 89c3dd1..53df884 100644 --- a/docs/api/dataset.md +++ b/docs/api/dataset.md @@ -20,6 +20,8 @@ Source: `src/lightcone/engine/dataset.py` (+ | `last_writer(root, dir)` | Who last touched an output's directory — the foreign-write question. Answers "cannot say" as empty, never an error. | | `require_committer(root)` | Refuses a repository with no git identity, before any recipe spends time. Asked as `git var`, the question a commit itself asks. | | `dataset_id(root)` | The DataLad dataset UUID, read via `git config -f`. | +| `set_annex_filter_required(root)` | Set `filter.annex.required=true`, so a `git add` that cannot reach git-annex fails loudly instead of staging raw bytes. | +| `annex_filter_required(root)` | Whether that flag is already set — `lc init --check`'s question. | ## What must stay true @@ -49,6 +51,15 @@ Source: `src/lightcone/engine/dataset.py` (+ - **Committing an archive or dot-named file needs `annex.dotfiles`** — git-annex routes dotfiles to git whatever `largefiles` says, and without the flag an image archive lands as a git blob, silently. +- **`filter.annex.required=true` is the storage policy's safety net.** + Without it, a `git add` whose shell cannot resolve git-annex prints + an error, exits 0, and stages the raw bytes into git history — + measured, and pinned by + `test_stock_plumbing_without_required_stages_raw_bytes_silently`. + It is the *only* thing convergence adds to what `git annex init` + wrote: no filter driver is rewritten, and no hook is touched, so how + git dispatches git-annex stays git-annex's own business and stays + resolved from `PATH`. ## Tests diff --git a/docs/api/project.md b/docs/api/project.md index a8c9f02..d2d0797 100644 --- a/docs/api/project.md +++ b/docs/api/project.md @@ -38,9 +38,15 @@ Source: `src/lightcone/engine/project.py` (+ `.gitattributes` are converged entry-wise, order judged against the template). - **Only what git can carry is converged.** No `src/`, no empty - directories — a clone must need nothing but `.venv` and - `git annex init`, and - `test_a_clone_of_a_converged_project_is_converged` pins it. + directories — a clone must need nothing but `.venv`, `git annex + init` and the annex filter (all three local state git does not + clone), and `test_a_clone_of_a_converged_project_is_converged` pins + it. +- **The annex filter is one config key.** `filter.annex.required=true`, + always, so a `git add` that cannot reach git-annex refuses instead of + silently staging raw bytes. Nothing else about how git dispatches + git-annex is lc's to write: the filter drivers and hooks stay exactly + as `git annex init` left them, resolved from `PATH`. - **There is no discovery.** The invoked directory is the project or it is a clean error; every uv call carries an explicit `--project`. - **Templates are files** (`templates/files/*.tmpl`, `string.Template` diff --git a/docs/architecture.md b/docs/architecture.md index 6d28ba8..af9a3bd 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -104,6 +104,15 @@ git carries pointers and history, the annex carries bytes, and default; `data/` and `results/` opt out). A researcher only ever types ordinary `git add` / `git commit`. +That ordinary `git add` dispatches git-annex from the *researcher's* +`PATH`, and a shell that cannot resolve it stages the raw bytes into +git history while exiting 0 — so `lc init` sets +`filter.annex.required=true`, which makes git refuse loudly instead +(every filtered command, not only `git add`). +Getting git-annex onto that `PATH` is the install's job, not the +repository's: `uv tool install lightcone-cli` puts it there alongside +`lc`. + Each output is committed with a **run record** — a `[DATALAD RUNCMD]` commit message whose `cmd` reconstructs the engine (`uv run --no-project --with lightcone-cli==`) and re-executes the diff --git a/docs/cli/init.md b/docs/cli/init.md index e6429b7..ff36709 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -25,7 +25,9 @@ manages, and never overwrites files you own: - **Repaired** — derived artifacts that have drifted: a `uv.lock` that no longer matches `pyproject.toml`, a `.venv` that no longer matches the lock, a managed `.gitignore` or `.gitattributes` entry that a - newer `lc` added. Repairs only ever append or rebuild derived state; + newer `lc` added, an annexed repository still missing the + `filter.annex.required` flag. Repairs only ever append or rebuild + derived state; hand-written lines are never reordered or removed. - **Blocked** — something convergence can see but must not fix by appending: a `.gitignore` rule that would silently swallow @@ -80,6 +82,27 @@ Two things it deliberately does *not* create: a `src/` directory directories), and any dependency in `pyproject.toml` — the lock carries only what *your* analysis imports, added with `uv add`. +Inside `.git`, convergence sets one configuration key — reported as the +`annex-filter` item: + +- `filter.annex.required=true`, always. Without it, a `git add` whose + shell cannot find git-annex prints an error, **exits 0, and stages + the raw bytes into git history** — a 2 GB dataset in git proper, on + every clone, forever. With it, the same situation is a hard, loud + failure and nothing is staged. Once the project holds committed + annexed content that refusal covers every command that must run the + filter, `git status` and `git diff` included — a project you cannot + use until git-annex is back, rather than one that silently absorbed + your data. + +That is the only thing `lc init` adds to what `git annex init` wrote. +How git finds git-annex is still ordinary `PATH` resolution, which is +why `lc` should be installed with `uv tool install lightcone-cli` — it +puts `git-annex` on your `PATH` alongside `lc`. If your `git add` ever +refuses, see +[`fatal: … clean filter 'annex' failed`](../user/troubleshooting.md#fatal-clean-filter-annex-failed) +in the troubleshooting guide. + ## Options | Option | Default | Effect | diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md index e13edf6..73f0b2a 100644 --- a/docs/user/troubleshooting.md +++ b/docs/user/troubleshooting.md @@ -114,6 +114,49 @@ read-only verbs (`lc status`, `--check`) never transfer data, so they report the fact instead. Fetch by hand only when you want the bytes for your own inspection. +## "fatal: … clean filter 'annex' failed" + +``` +git-annex filter-process: line 1: git-annex: command not found +error: could not read greeting from subprocess 'git-annex filter-process' +error: initialization for subprocess 'git-annex filter-process' failed +fatal: data/catalog.fits: clean filter 'annex' failed +``` + +Your shell's `PATH` has no `git-annex`, so git could not run the filter +that turns a large file into an annex pointer. **Nothing was staged**, +which is the point: without `filter.annex.required=true` — which +`lc init` sets — git would have exited 0 and committed the raw bytes +into history instead. + +Once a project holds committed annexed content, this is not limited to +`git add`. Any command that has to run the filter over that content +stops the same way, `git status`, `git diff` and `git checkout` +included — so the whole project reads as broken until git-annex is back +on your `PATH`. That is the intended shape of the failure: a repository +you cannot use is recoverable in one command, and one that quietly +absorbed a multi-gigabyte file is not. + +`git-annex` ships with `lc`, so a tool install puts both on your `PATH`: + +```bash +uv tool install lightcone-cli +git-annex version +``` + +If `lc` runs but `git-annex` does not, uv's tool directory is not on +your `PATH` — run `uv tool update-shell` and open a new shell. Running +`lc` through `uvx` puts nothing on your `PATH` at all, so a plain +`git add` cannot work that way. + +This failure is deliberately loud. `lc init` sets +`filter.annex.required=true` in every project precisely because +without it git handles the same situation by printing the error, +**exiting 0, and staging your data's raw bytes into git history** — +committing a multi-gigabyte dataset into git proper, silently, where +every clone carries it forever. A refused `git add` costs you one +`lc init`; the silent version costs you the repository. + ## "… and this is a NERSC login node" `lc materialize` executes recipes, and on centers `lc` recognizes it diff --git a/src/lightcone/engine/dataset.py b/src/lightcone/engine/dataset.py index def066f..8b3d91e 100644 --- a/src/lightcone/engine/dataset.py +++ b/src/lightcone/engine/dataset.py @@ -102,6 +102,57 @@ def ignore_rule(directory: Path, path: str) -> str | None: return proc.stdout.splitlines()[0].split("\t")[0] +# ============================================================================= +# The annex filter: a missing git-annex must not be silent +# ============================================================================= + + +def set_annex_filter_required(directory: Path) -> None: + """Make git refuse, rather than corrupt, when the annex filter cannot run. + + ``git annex init`` wires ``filter.annex.*`` to a bare ``git-annex``, + resolved from the ``PATH`` of whichever git runs — and the + researcher's shell, unlike lc's own environment, may hold none. + Measured: a ``git add`` whose clean filter cannot start prints an + error, **exits 0**, and stages the raw bytes into git history, so a + 2 GB input lands in git proper on every clone forever. This flag is + what turns that into git's own hard failure instead + (``fatal: : clean filter 'annex' failed``), which is the whole + difference between a loud stop and silent corruption. + + Args: + directory: A directory inside the repository. + """ + _git(["config", "filter.annex.required", "true"], cwd=directory) + + +def annex_filter_required(directory: Path) -> bool: + """Whether *this repository* already carries that flag. + + ``--local`` because the write is repository-local, and a probe that + read the merged config would answer for the *host* instead: a user + who once set ``filter.annex.required`` in ``~/.gitconfig`` would have + every project report converged while its own ``.git/config`` carried + nothing, leaving the protection behind the moment that repository is + used under another ``HOME`` — a clone, CI, a container. ``--type=bool`` + because git's booleans are not one spelling: ``1``, ``yes``, ``on`` + and a valueless key all mean true to the filter, and reading them as + drift would rewrite a repository that was already correct. + + Args: + directory: A directory inside the repository. + + Returns: + True if ``filter.annex.required`` is set, in this repository, to + anything git reads as true. + """ + set_to = _ask( + ["config", "--local", "--get", "--type=bool", "filter.annex.required"], + cwd=directory, + ) + return (set_to or "").strip() == "true" + + # ============================================================================= # What a run does to the repository # ============================================================================= diff --git a/src/lightcone/engine/project.py b/src/lightcone/engine/project.py index 836ca69..c907216 100644 --- a/src/lightcone/engine/project.py +++ b/src/lightcone/engine/project.py @@ -407,10 +407,18 @@ def _converge_dataset(c: _Converger, directory: Path) -> None: """ c.item(".git", _in_repository(directory), lambda: dataset.init_git(directory)) # After the item above, so a fresh project has a repository to annex. + # `annexed` is read *before* the item applies: in write mode the init + # runs inline, and the filter item below must still know whether this + # run created the annex (created) or found one (repaired). + annexed = _can_ask_git(directory) and dataset.is_annexed(directory) + c.item("git-annex", annexed, lambda: dataset.init_annex(directory)) + # The one thing `git annex init` does not set, and the reason it + # matters is in `dataset.set_annex_filter_required`. c.item( - "git-annex", - _can_ask_git(directory) and dataset.is_annexed(directory), - lambda: dataset.init_annex(directory), + "annex-filter", + annexed, + lambda: dataset.set_annex_filter_required(directory), + is_current=lambda: dataset.annex_filter_required(directory), ) attributes = directory / ".gitattributes" # Read before the repair, not after: the check is on the text a repair diff --git a/tests/conftest.py b/tests/conftest.py index 51536f3..04a0188 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -102,6 +102,10 @@ def tools(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]: # rather than on disk because `.git` is a *file* in a linked # worktree, so there is nowhere inside it to leave a marker. annexed: set[Path] = set() + # What `git config` has been told, per repository — the annex filter + # converges through a config write and a config read, and a fake that + # forgot the write would report the same item repaired forever. + config: dict[tuple[Path, str], str] = {} def fake_run(argv: list[str], *, cwd: Path) -> MagicMock: calls.append(list(argv)) @@ -120,6 +124,15 @@ def fake_run(argv: list[str], *, cwd: Path) -> MagicMock: annexed.add(_repo(cwd)) elif argv[:2] == ["git", "config"] and argv[-1] == "annex.uuid": return MagicMock(returncode=0 if _repo(cwd) in annexed else 1) + elif argv[:3] == ["git", "config", "--local"] and "--get" in argv: + # The key is last; the flags in between are git's own reading + # conventions (`--type=bool`), which the store does not model + # because the engine only ever writes one value. + value = config.get((_repo(cwd), argv[-1])) + code = 0 if value is not None else 1 + return MagicMock(returncode=code, stdout=(value or "") + "\n", stderr="") + elif argv[:2] == ["git", "config"] and len(argv) == 4 and not argv[2].startswith("-"): + config[(_repo(cwd), argv[2])] = argv[3] elif argv[:2] == ["git", "check-ignore"]: return _fake_check_ignore(cwd, argv[-1]) return MagicMock(returncode=0, stdout="", stderr="") diff --git a/tests/test_dataset.py b/tests/test_dataset.py index c21e649..9e4dc1c 100644 --- a/tests/test_dataset.py +++ b/tests/test_dataset.py @@ -10,6 +10,8 @@ from __future__ import annotations import shutil +import subprocess +import sys from pathlib import Path import pytest @@ -499,6 +501,135 @@ def test_the_worker_and_the_shim_are_never_console_scripts() -> None: assert entry.value.startswith("lightcone.cli") or entry.value in theirs, entry +# ---- the annex filter, against a real shell --------------------------------- + + +def _researcher_shell(tmp_path: Path) -> dict[str, str] | None: + """The environment a researcher's own `git add` runs in when nothing + put git-annex on their PATH — the uvx case. The system directories + carry git and the usual utilities; whether they also carry a + git-annex is the host's business, and where they do there is nothing + for these tests to prove. Probed by listing the directories rather + than `shutil.which`, which the autouse `tools` fixture narrows for + exactly the three tools this asks about.""" + dirs = [Path("/usr/bin"), Path("/bin")] + if not any((d / "git").is_file() for d in dirs): + return None + if any((d / "git-annex").exists() for d in dirs): + return None + # `HOME` takes the global config out of play; `GIT_CONFIG_NOSYSTEM` + # does the same for `/etc/gitconfig`, where a site-wide + # `filter.annex.required` would otherwise decide these outcomes. + return { + "PATH": ":".join(str(d) for d in dirs), + "HOME": str(tmp_path), + "GIT_CONFIG_NOSYSTEM": "1", + } + + +def _shell_git(repo: Path, env: dict[str, str], *argv: str) -> subprocess.CompletedProcess[str]: + """Run git exactly as that shell would — not through `project._run`, + whose environment is lc's own and always resolves the bundled annex.""" + git = next(p for d in env["PATH"].split(":") if (p := Path(d) / "git").is_file()) + return subprocess.run( + [str(git), *argv], cwd=repo, env=env, capture_output=True, text=True, check=False + ) + + +@pytest.fixture +def no_annex_shell(tmp_path: Path) -> dict[str, str]: + env = _researcher_shell(tmp_path) + if env is None: + pytest.skip("the system PATH itself resolves git-annex (or lacks git)") + return env + + +def test_required_true_makes_the_missing_filter_loud_not_silent( + repo: Path, no_annex_shell: dict[str, str] +) -> None: + """The whole claim, measured: with the flag set, a shell that cannot + resolve git-annex gets git's own hard refusal — nothing staged, exit + nonzero — instead of raw bytes in git history.""" + assert not dataset.annex_filter_required(repo) + dataset.set_annex_filter_required(repo) + assert dataset.annex_filter_required(repo) + (repo / "data" / "catalog.fits").write_bytes(b"\x00" * 4096) + + added = _shell_git(repo, no_annex_shell, "add", "data/catalog.fits") + + # The facts, not git's wording: Linux git reports the failed handshake + # and then `fatal: : clean filter 'annex' failed`, while macOS + # git dies on the pkt-line read with `fatal: the remote end hung up + # unexpectedly`. Both refuse, which is the whole claim. + assert added.returncode != 0 + assert "annex" in added.stderr + staged = _shell_git(repo, no_annex_shell, "diff", "--cached", "--name-only") + assert staged.stdout.strip() == "" + + +def test_the_flag_is_read_from_the_repository_never_the_users_global_config( + repo: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The write is repository-local, so the read must be too. A user who + once set the flag in `~/.gitconfig` would otherwise have every project + report converged while its own `.git/config` carried nothing — and + since this is local state a clone never receives, the protection would + vanish the moment the repository moved to CI or another account.""" + global_config = tmp_path / "gitconfig" + global_config.write_text('[filter "annex"]\n\trequired = true\n') + monkeypatch.setenv("GIT_CONFIG_GLOBAL", str(global_config)) + + assert not dataset.annex_filter_required(repo) + + dataset.set_annex_filter_required(repo) + assert dataset.annex_filter_required(repo) + assert "required = true" in (repo / ".git" / "config").read_text() + + +@pytest.mark.parametrize("spelling", ["true", "1", "yes", "on"]) +def test_any_spelling_git_reads_as_true_is_not_reported_as_drift( + repo: Path, spelling: str +) -> None: + """git's booleans are not one string. A repository someone configured + by hand as `required = 1` is fully protected — reading it as drift + would rewrite a repository that was already correct, and report every + `lc init --check` on it as unconverged.""" + dataset._git(["config", "filter.annex.required", spelling], cwd=repo) + + assert dataset.annex_filter_required(repo) + + +@pytest.mark.skipif( + sys.platform == "darwin", + reason="macOS git aborts the filter handshake outright (exit 128, " + "'the remote end hung up unexpectedly'), so the silent-passthrough " + "hazard recorded here is the Linux behaviour", +) +def test_stock_plumbing_without_required_stages_raw_bytes_silently( + repo: Path, no_annex_shell: dict[str, str] +) -> None: + """The hazard the flag exists for, recorded as measured: as + `git annex init` leaves a repository, a shell without git-annex + prints an error, *exits 0*, and stages the raw bytes into git + history — a 2 GB dataset in git proper, on every clone, forever. + If this ever starts failing, git changed the behavior and the + `required=true` net is worth re-examining. + + Linux-only, and that asymmetry is the point rather than a gap: macOS + git happens to refuse the same situation on its own, so there the + flag changes nothing — but a platform accident is not a guarantee, + and the flag is what makes the refusal one on every host.""" + (repo / "data" / "catalog.fits").write_bytes(b"\x00" * 4096) + + added = _shell_git(repo, no_annex_shell, "add", "data/catalog.fits") + + assert added.returncode == 0 + # Not the shell's exact wording: bash says "command not found" where + # dash — /bin/sh on Debian and Ubuntu — says only "not found". + assert "not found" in added.stderr + staged = _shell_git(repo, no_annex_shell, "cat-file", "-p", ":data/catalog.fits") + assert not staged.stdout.startswith("/annex/objects/") + # ---- who last wrote a path ------------------------------------------------- diff --git a/tests/test_project.py b/tests/test_project.py index 463af2e..1c6d91f 100644 --- a/tests/test_project.py +++ b/tests/test_project.py @@ -10,7 +10,7 @@ import pytest from conftest import probes, uv_calls -from lightcone.engine import templates +from lightcone.engine import dataset, templates from lightcone.engine.project import ( SPEC_FILENAME, ConvergenceReport, @@ -212,10 +212,11 @@ def test_a_clone_of_a_converged_project_is_converged(tmp_path: Path) -> None: tracks has to be something git can carry, or a fresh clone reports drift forever. - Two items are exempt, and both for the same reason — they are local + Three items are exempt, and all for the same reason — they are local state git does not clone. `.venv` is git-ignored and rebuilt from the - lock, and `git clone` of an annexed repository leaves the annex - uninitialized until someone runs `git annex init`.""" + lock, `git clone` of an annexed repository leaves the annex + uninitialized until someone runs `git annex init`, and the annex + filter is a `.git/config` entry, which a clone starts fresh.""" project = tmp_path / "proj" converge(project) @@ -230,7 +231,7 @@ def test_a_clone_of_a_converged_project_is_converged(tmp_path: Path) -> None: (clone / ".git").mkdir() report = converge(clone, write=False) - assert report.created == ["git-annex", ".venv"], report.created + assert report.created == ["git-annex", "annex-filter", ".venv"], report.created def test_converge_repairs_a_missing_piece(tmp_path: Path) -> None: @@ -505,6 +506,93 @@ def fake_run(argv: list[str], *, cwd: Path) -> MagicMock: converge(tmp_path / "proj") +# ---- the annex filter ------------------------------------------------------ +# +# `git annex init` wires the filter to a bare `git-annex`, resolved from +# the PATH of whichever git runs — and the researcher's shell may hold +# none (the engine invoked through uvx puts nothing on it), where a plain +# `git add` prints an error, exits 0, and stages raw bytes into git +# history. `filter.annex.required=true` makes git refuse instead. What +# that refusal actually looks like is measured in test_dataset.py, +# against a real annex and a real shell. + + +def _config_writes(tools: list[list[str]]) -> dict[str, str]: + """The ``git config `` writes, last value per key.""" + return { + c[2]: c[3] + for c in tools + if c[:2] == ["git", "config"] and len(c) == 4 and not c[2].startswith("-") + } + + +def _annexed_but_unconverged(directory: Path) -> None: + """A repository as `git annex init` alone leaves it: annexed, and + carrying none of the config lc adds on top.""" + directory.mkdir(parents=True, exist_ok=True) + dataset.init_git(directory) + dataset.init_annex(directory) + + +def test_convergence_requires_the_annex_filter(tmp_path: Path, tools: list[list[str]]) -> None: + """The one thing `git annex init` does not set, and the only thing + this convergence writes — no filter driver is rewritten and no hook + is touched, so what git-annex configured stays exactly as it wrote it.""" + project = tmp_path / "proj" + + report = converge(project) + + assert "annex-filter" in report.created + written = _config_writes(tools) + assert written["filter.annex.required"] == "true" + assert [key for key in written if key.startswith("filter.annex.")] == [ + "filter.annex.required" + ] + + +def test_a_converged_project_reconverges_unchanged(tmp_path: Path) -> None: + """The flag is read back, so a second run reports it unchanged rather + than repairing it on every invocation forever.""" + project = tmp_path / "proj" + converge(project) + + report = converge(project) + + assert "annex-filter" in report.unchanged + assert report.converged + + +def test_an_adopted_annex_without_the_flag_is_repaired( + tmp_path: Path, tools: list[list[str]] +) -> None: + """A repository annexed before lc ever saw it: the annex is already + there, so the flag is *repaired* onto it rather than created.""" + project = tmp_path / "proj" + _annexed_but_unconverged(project) + tools.clear() + + report = converge(project) + + assert "annex-filter" in report.repaired + assert _config_writes(tools)["filter.annex.required"] == "true" + + +def test_check_mode_reports_the_missing_flag_and_writes_nothing( + tmp_path: Path, tools: list[list[str]] +) -> None: + """Check mode asks the same question and answers it without writing: + the drift is reported, and no `git config` write is issued.""" + project = tmp_path / "proj" + _annexed_but_unconverged(project) + tools.clear() + + report = converge(project, write=False) + + assert "annex-filter" in report.repaired + assert not report.converged + assert _config_writes(tools) == {} + + # ---- refusals -------------------------------------------------------------