Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ its own adds to them and says what it adds.
## Checking a change

```sh
task render # dry-run the whole pipeline inside the toolbox, no network
task check # render every command of the pipeline inside the toolbox, no network, and diff it against render.txt
```

The check workflow runs the same render on every pull request and compares it with the
Expand Down
2 changes: 1 addition & 1 deletion PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@

## Test plan

- [ ] `task render` passes and the committed render was updated if the commands changed
- [ ] `task check` passes, and `render.txt` was updated with `task render-update` if the commands changed
- [ ] Verbs that write to a bucket were run on a fork against a scratch bucket, or this change touches none
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This policy applies to every katoptra repository that has no SECURITY file of it
account's token; a pull-request check gets no secret at all.

Everything else on a public mirror is served as upstream serves it. Problems with the
packages themselves belong upstream, to CTAN, TeX Live, CRAN or CPAN. This
packages themselves belong upstream, to CTAN or TeX Live. This
organization copies what they publish.

## Reporting
Expand Down
46 changes: 30 additions & 16 deletions profile/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,41 +2,55 @@

Greek for mirrors. Bytes here are closer than they appear.

Each repository is one mirror: an upstream, a bucket, and a pipeline that syncs the two
on a schedule from GitHub Actions. The mirrors are served from Cloudflare R2 and listed
with live status at [katoptra.org](https://katoptra.org/).
Each repository is one mirror: an upstream, a sink, and a pipeline that syncs the two on
a schedule from GitHub Actions. The public mirrors are served from Cloudflare R2 and
listed with live status at [katoptra.org](https://katoptra.org/).

| Mirror | Upstream | Cadence | Repository |
|---|---|---|---|
| [ctan.katoptra.org](https://ctan.katoptra.org/) | [CTAN](https://ctan.org), all of it | hourly | [ctan](https://github.com/katoptra/ctan) |
| [tlnet.katoptra.org](https://tlnet.katoptra.org/) | CTAN `systems/texlive/tlnet`, what `tlmgr` installs from | daily | [tlnet](https://github.com/katoptra/tlnet) |

Two more mirror into Proton Drive rather than onto the web. What they copy is private;
the pipelines are public and run on the same toolbox, so they are forks like the others.

| Mirror | Upstream | Cadence | Repository |
|---|---|---|---|
| GitHub | every repository under the owners it names, one git bundle each | nightly | [github](https://github.com/katoptra/github) |
| Dropbox | one Dropbox account | nightly | [dropbox](https://github.com/katoptra/dropbox) |

More are on the way.

## Why this exists

- I built these for myself. They live here so the pipelines are namespaced and share one
toolbox.
- Open source in the sense that you can fork and run it. Or just use these published
- Open source in the sense that you can fork and run it. Or just use the published
mirrors like I do.

## How a mirror runs

- `list` the upstream, `diff` it against the `state` the last run left behind, then
`publish` the delta and `checkpoint` the state.
- `verify` data and signatures whenever upstream provides them.
- Works in batches, sized for a GitHub Actions runner.
- Pings a healthcheck after each run if you give it one.
- Keeps its state and checkpoints in R2 or any S3-compatible bucket. Cloudflare is the
default for its versatility and free egress.
- I preferentially use 1Password for secrets, but GitHub secrets work too.
- Everything the mirrors share lives once, in [lib](https://github.com/katoptra/lib):
the toolbox that starts a run, contains it in an image, resolves its secrets, checks
it and reports it, and an engine per transport that moves the bytes. lib's README is
the manual.
- An rsync mirror lists upstream, diffs the listing against the state the last run left
in the bucket, publishes the delta in batches and checkpoints after each. It verifies
data and signatures wherever upstream provides them.
- A Proton mirror stages what its upstream holds and uploads it in one call; unchanged
files are skipped, changed ones become revisions, and Proton's version history is the
history of the mirror.
- Every run happens inside one pinned image, on a laptop and in Actions alike, and pings
a healthcheck when it finishes. Silence is the alert.
- Secrets reach a run by name from a vault. No credential, account identifier, bucket
name or endpoint is in any repository.

## Want your own?

- Fork a mirror.
- Set three vars in its Taskfile: `SOURCE` (the upstream rsync URL), `BUCKET` (your
bucket name) and `HOST` (the domain in front of it).
- Point the workflow at your vault, or your repository secret.
- Fork a mirror and follow its README's "Want your own?": the lines to change, the
bucket, the vault item, its own accounts, the first run.
- Everything the mirrors share, from making a vault and a service account to what the
bucket holds and how the workflows run, is in lib's README, once.
- A full CTAN mirror costs under two dollars a month.

## Contributing
Expand Down