diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..6018ed6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,56 @@ +# dropbox + +A nightly, self-chaining mirror of a Dropbox account into one Proton Drive folder, with +its state in an age-encrypted SQLite database in a bucket. `README.md` says what it +mirrors, how it works step by step, how to fork it and the runbook; +[katoptra/lib](https://github.com/katoptra/lib)'s README is the manual for the toolbox +this mirror includes. This file is what a change must not break. + +Nothing in this repo starts a run: an external scheduler dispatches `sync.yml` nightly. +This mirror includes the toolbox alone: `Taskfile.yml` owns the order, one +`python -m migrator ` per step, and `src/migrator/` owns every decision. The +infrastructure modules there are from donphi/dropbox_proton at `cfd0e57`, MIT; the phases +under `src/migrator/phases/` and the Taskfile are this repository's own. + +## Must knows + +- **Plan by default.** `batches`, `trash`, `reconcile`, `report` and `empty-trash` mutate + only with `--apply`, which `task pipeline` passes and `task plan-pipeline` never does. + No mutation is trusted on its exit status: the state push is trusted once the object + lands, and nothing is recorded as uploaded on a command's exit code alone. +- **A missing state beside history is refused.** A lost state must never look like an + empty mirror. The rollback is `task state-rollback`; never delete the history to make + a run start fresh. +- **A truncated listing can never become a trash list.** `delta` refuses a listing under + `listing_floor_ratio` of the mirrored count, and `trash` runs only when every planned + batch landed. +- **`checkpoint` is the last step of a batch**, a dated copy first and then a server-side + copy to the canonical key, so a killed run repeats at most one batch. A batch that + recorded nothing fails the run rather than chaining, so an identical failure cannot loop. +- **The session is this mirror's own.** Two processes holding it race its rotating + refresh token. Never run `sync` or `empty-trash` from a laptop while an Actions run may + be going. The reconcile walk's workers each run from their own copy of the session, and + the copy a refresh rewrote is adopted. +- **`run_budget_minutes` stays 20 minutes under `sync.yml`'s `timeout-minutes`**, so the + last batch's upload and the report finish. The chain is the workflow's, on `.run/chain`. +- **Files are keyed by lowercased path**, so a case-only rename changes nothing in the + delta; display paths are rebuilt from each ancestor's own name because Dropbox cases + parent segments inconsistently. +- **The logs are public.** The report is built from the state and carries counts only; + errors print as their class unless `MIRROR_VERBOSE=1`; `op run` masks every value. +- **Run flags go after the double dash** (`task sync -- RUN_BUDGET_MIN=30 RECONCILE=true`, + or the workflow's `vars` input); the Taskfile maps them to the environment the migrator + reads. `RECONCILE` takes the literal word `true`. +- **`config/mirror.toml` is strict** and rejects unknown keys; the three account + identifiers come from the environment and override its keys. + +## Verifying a change + +- `task test` runs the pytest suite inside the image, offline; `task lint` is ruff. +- `task check` renders every command of the pipeline inside the image and diffs it + against `render.txt`; `task render-update` accepts a change. The `check` workflow runs + check, test and lint on every pull request, with no secret. +- `task plan` runs the read path against the real account and bucket, changes nothing in + Proton, and prints the report it would make. +- `task status` fetches the state and prints counts; both leave `.run/state.sqlite` for + inspection. diff --git a/README.md b/README.md index c9f0c49..2d8e88e 100644 --- a/README.md +++ b/README.md @@ -1,515 +1,269 @@ -## 🪞 dropbox +# dropbox -A self-chaining GitHub Actions pipeline that mirrors a Dropbox account into Proton Drive. -After each run Proton Drive holds what Dropbox held at listing time, under one folder of -your choosing. Changed files become new Proton revisions, files that left Dropbox move to -Proton's trash, and nothing is recorded as mirrored until Proton's own upload summary has -accounted for it. The only durable state is one SQLite database, age-encrypted in a -Cloudflare R2 bucket; no run is ever told where to start. +[![sync](https://github.com/katoptra/dropbox/actions/workflows/sync.yml/badge.svg)](https://github.com/katoptra/dropbox/actions/workflows/sync.yml) +[![license](https://img.shields.io/github/license/katoptra/dropbox)](LICENSE) -Dropbox stays the primary. Nothing written in Proton Drive flows back. - -``` -Dropbox (primary, read-only credentials) - -> GitHub Actions run (this repo, inside one pinned toolbox image) - inventory -> delta -> plan -> per batch: fetch -> verify -> upload -> confirm -> checkpoint - -> trash -> reconcile (weekly) -> report -> ping -> chain if batches remain - -> Proton Drive (Proton's version history is the cloud-side history) - -> R2 .state/ (encrypted state database, dated history, CLI session) -``` +A nightly, self-chaining mirror of a Dropbox account into one Proton Drive folder. After +each run Proton Drive holds what Dropbox held at listing time. Changed files become new +Proton revisions, files that left Dropbox move to Proton's trash, and nothing is recorded +as mirrored until Proton's own upload summary has accounted for it. The only durable +state is one SQLite database, age-encrypted in a bucket; no run is ever told where to +start. Dropbox stays the primary; nothing written in Proton Drive flows back. The repository is public and holds no account: every credential and every account -identifier (the Dropbox account id, the Proton destination folder and its UID) lives in -one 1Password vault and reaches a run by name. - -### Provenance - -The infrastructure modules under `src/migrator/` (SQLite evidence schema, one-pass hasher, -atomic writes, redacting logger, path guards, and the two providers for the Dropbox API -and the official `proton-drive` CLI) come from +identifier lives in one vault and reaches a run by name. The infrastructure modules under +`src/migrator/` (the SQLite evidence schema, the one-pass hasher, atomic writes, the +redacting logger, path guards, and the two providers for the Dropbox API and the +`proton-drive` CLI) come from [donphi/dropbox_proton](https://github.com/donphi/dropbox_proton) at commit `cfd0e57`, MIT, whose copyright notice is retained in [LICENSE](LICENSE). The mirror phases and the -Taskfile are this repo's own; the toolbox that runs them, its image and the two workflows -this repository calls are [katoptra/lib](https://github.com/katoptra/lib)'s, at `v2`. +Taskfile are this repository's own. + +## How to use + +The files are under the destination folder in Proton Drive, at their Dropbox paths and +with Dropbox's casing. An edit in Dropbox is a new revision in Proton the next morning; +a delete is in Proton's trash; a file too large for the runner to stage is counted as +oversized in the run's report and can be uploaded by hand to its Dropbox path, where the +weekly walk leaves it alone. + +## How it works + +Once a night a GitHub Actions job runs this pipeline inside the toolbox image from +[katoptra/lib](https://github.com/katoptra/lib). The toolbox's own verbs are the solid +boxes; every dashed box is one `python -m migrator `. This mirror includes the +toolbox alone and supplies its own pipeline: the Taskfile owns the order, the Python +owns every decision. + +```mermaid +flowchart LR + clock --> cp["clock"] --> session --> state --> inventory --> delta --> plan --> batches + subgraph b["batches: each until the run's budget"] + direction LR + fetch --> verify --> upload --> confirm --> checkpoint + end + batches --> b --> trash --> reconcile["reconcile
weekly"] --> rp["report"] --> report --> ping + classDef own stroke-dasharray: 5 5 + class cp,session,state,inventory,delta,plan,batches,fetch,verify,upload,confirm,checkpoint,trash,reconcile,rp own +``` -## 🧭 How it works +| Step | Does | +|---|---| +| `clock` | The migrator's own stamp after the toolbox's: the run's start to `.run/clock.json`; clears staging, the report and the chain marker | +| `session` | Fetches the encrypted Proton CLI session from the bucket and unpacks it for every later CLI call | +| `state` | Fetches the state database and starts the run row. A missing state is accepted as an empty mirror only when the history is empty too; a missing state beside history is refused, because a lost state must never look like an empty mirror | +| `inventory` | Walks Dropbox over the API, each page committed with its cursor. Entries with no content hash are recorded as non-downloadable and excluded. Display paths are rebuilt from each ancestor's own name, because Dropbox cases parent segments inconsistently | +| `delta` | Compares the inventory against `mirror_objects` on path, size and content hash. Refuses a listing under half the mirrored file count, so a truncated listing can never become a trash list | +| `plan` | Refuses a tree over `ceiling_gb` or a batch the disk cannot stage. Leaves out files over `max_file_gb` and counts them as oversized. Packs the rest into batches of `batch_gb` and `batch_files` | +| `batches` | Runs each batch through the five steps below, stopping before a batch that would pass the run's budget; stopping with batches left is a success that chains the next run | +| `fetch` | Downloads the batch from Dropbox into staging under its display paths; a path that vanished since listing is counted and skipped | +| `verify` | Recomputes every staged file's Dropbox content hash and records SHA-1 and SHA-256. A mismatch is a file edited since listing: removed and counted, never recorded | +| `upload` | One `proton-drive filesystem upload` of the staging tree; Proton skips files whose content it already holds and makes revisions of changed ones | +| `confirm` | The upload summary must account for every verified file plus every folder, and every failure must name a file in the batch. Those alone are recorded as failed; the rest confirm | +| `checkpoint` | Merges the confirmed rows into `mirror_objects` and pushes the state to the bucket, a dated copy first and then the canonical key. Always the last step of a batch, so a killed run repeats at most one | +| `trash` | Only when every planned batch landed: one listing and one `filesystem trash` per folder of deleted files | +| `reconcile` | On the first run of the configured weekday, or with `RECONCILE=true`: a full Proton walk compared against `mirror_objects`. Rows Proton lacks or mis-sizes are dropped so they re-upload; nodes neither Dropbox nor the state knows are trashed. A walk that does not fit one run resumes on the next, and a partial walk drops and trashes nothing | +| `report` | Builds the step summary from the state alone, finishes the run row, writes the chain marker, and returns the run's status | + +Every step is plan-by-default: `batches`, `trash`, `reconcile`, `report` and +`empty-trash` change anything only with `--apply`, which `task pipeline` passes and +`task plan-pipeline` never does. No mutation is trusted on its exit status. The evidence +has three layers: the upload summary per batch, matched item by item; Proton's own +server-side block hashes, checked at upload and out of this repository's hands; and the +weekly walk, which compares Proton's own listing against the state independently of +anything a batch claimed. + +The run around those steps, from the image and the secrets to the report and the ping, +is the toolbox's and is documented once in +[lib's README](https://github.com/katoptra/lib#the-toolbox). + +## Want your own? + +### 1. Fork it + +Fork [katoptra/dropbox](https://github.com/katoptra/dropbox). Nothing in `Taskfile.yml` +names an account; [`config/mirror.toml`](config/mirror.toml) is the one behaviour input +and its defaults fit a GitHub runner. The account is the environment, all of it `op.env`. + +### 2. Storage + +The bucket holds the state and the session, nothing of the mirrored tree. -One run is `task pipeline`, executed inside the toolbox image. Each step is one -`python -m migrator `; the Taskfile owns sequencing, the Python owns every -decision, and the toolbox from [katoptra/lib](https://github.com/katoptra/lib) owns the -run around them: how it starts, is contained, resolves its secrets, is checked and is -reported. Steps before `inventory` run without a state database and are called -commands; the rest are phases and record their evidence in the state. +``` +.state/state.sqlite.xz.age the state: evidence tables, mirror_objects, runs, batches, deletions +.state/history/-