Builds some AURs and my libfprint package for my laptop, pushing the diff to xerootg.github.io
Saves me time building AURs every time someone pushes.
xerootg.github.io lists everything in every repo below — versions, sizes, build dates, when each package was first published, and a rolling changelog. It is generated rather than written; see the package index.
Packages are signed. Trust the key once, then add the repo to your
pacman.conf:
curl -fsSL https://xerootg.github.io/xerootg.asc | sudo pacman-key --add -
sudo pacman-key --lsign-key "$(curl -fsSL https://xerootg.github.io/xerootg.asc \
| gpg --show-keys --with-colons | awk -F: '/^fpr:/{print $10; exit}')"[custom]
Server = https://github.com/xerootg/arch/releases/download/custom-repo
SigLevel = Required DatabaseOptional--lsign-key is the step people miss: --add imports the key, but pacman still
refuses every package until the key is locally signed as trusted. The site
carries the same snippet with the fingerprint filled in.
ghidra-noprompt is ~500 MB, and GitHub refuses any pushed file over 100 MB, so
it cannot live in the Pages repo above. It gets its own pacman repo, served
directly out of a GitHub Release on this repository:
[ghidra]
Server = https://github.com/xerootg/arch/releases/download/pacman-repo
SigLevel = Required DatabaseOptionalThen sudo pacman -Sy ghidra-noprompt.
That is stock Ghidra from
the matching upstream stable release, with one change: opening a program that
has never been analyzed no longer prompts "would you like to analyze it now?".
The prompt moves to a tool option (Edit → Tool Options → Auto Analysis → Ask
To Analyze) that ships disabled, so it can be turned back on without a
rebuild. The patch and the build that produced it live in
xerootg/ghidra. The package
provides/conflicts with ghidra.
.github/workflows/ghidra-noprompt.yml checks that fork daily, repackages any
new release, refreshes the ghidra.db in the fixed pacman-repo release, and
drops the superseded package asset. The release tag never changes, so the
Server URL above is stable.
Its PKGBUILD lives in release-pkgbuilds/, not pkgbuilds/, so the
build-pacman-repo pipeline never sees it — that pipeline would otherwise try
to commit the package to the Pages repo and break the push for every other
package.
The job decides what to build from the release itself, not from the checked-in
pkgver, so it stays correct without needing to commit anything. It does try to
refresh the PKGBUILD afterwards, but the main ruleset requires a pull request
and github-actions[bot] cannot bypass it, so that push is best effort and
non-fatal — expect the checked-in recipe to lag a release behind unless the bot
is granted a bypass.
orca-slicer-git compresses to ~164 MB, so it hits the same 100 MB wall. It gets
its own release-hosted repo:
[orca]
Server = https://github.com/xerootg/arch/releases/download/orca-repo
SigLevel = Required DatabaseOptionalThen sudo pacman -Sy orca-slicer-git.
Ships orca-slicer-git and orca-slicer-git-extras (calibration models, daily
tips, profile validator). The main package provides/conflicts with
orca-slicer.
.github/workflows/orca-slicer.yml computes upstream's pkgver the same way the
PKGBUILD does, skips the run if that revision is already published, and otherwise
does a full from-source build — around 2h45m — before refreshing orca.db in
the fixed orca-repo release and deleting the superseded packages.
Two settings keep that build alive on a 16 GB runner, both in
the PKGBUILD: options=('!lto') at
pkgbase level, and ninja -j2 for the main build. The pkgbase part matters —
an options=() inside a package_*() function only affects packaging, and
makepkg resolves lto before build() runs, so a per-package array cannot
disable it. Without this the job is OOM-killed (exit 137) while compiling
libslic3r's precompiled header, and because that kill takes the whole container
down rather than returning a non-zero status, allow-failure does not contain
it — it stops every other package in the repo from updating too.
xerootg.github.io is regenerated by
.github/workflows/repo-index.yml and never edited by hand.
Its input is each repo's pacman database — the .db tarball repo-add
writes at the end of every build. That file already carries the name, version,
description, URL, licence, compressed and installed size, and %BUILDDATE% of
every package, and it is the same file pacman downloads, so the page cannot
disagree with what a client would actually install. Nothing had to be added to
the builders to produce it.
A .db is a snapshot with no history, though, so "when was this added" cannot
be recovered from it. That is what the CSVs in the Pages repo are for. They
start empty and accumulate:
| file | contents |
|---|---|
data/packages.csv |
one row per (repo, package): current version, sizes, built_utc, first_seen_utc, last_updated_utc |
data/changes.csv |
append-only log of added / updated / removed, capped at 5000 rows |
data/state.json |
last check time, and a digest of the data |
first_seen_utc is written once, the first time a package is observed, and
never rewritten. The first run records its packages as seeded rather than
added, because their real history predates the tool and claiming otherwise
would be a lie the CSV then carries forever. built_utc comes straight from the
database and is accurate from day one.
Rendering reads only the CSVs, never the databases. A repo whose .db could not
be fetched this run keeps its last known contents on the page instead of
silently emptying out, and logs no spurious removals — one failed release
download must not rewrite history.
The job runs after each of the three publishers finishes (workflow_run), plus
once daily. It commits only when the data actually changed, or when the last
commit is over 20 hours old — enough to keep the "last checked" stamp honest
without a commit per trigger, in a repo that has already had to be rewritten
once for size.
It is a separate workflow rather than a step inside each publisher on purpose: those run on different schedules, take between two minutes and three hours, and two of them never check out the Pages repo. One writer avoids the race.
The site carries a .nojekyll, so Pages serves it statically. There is no build
step between publishing a package and it being downloadable, which matters when
the site is mostly a few hundred megabytes of .pkg.tar.zst.
[custom] used to be served from the GitHub Pages repo. It moved to a GitHub
Release, and that removes the constraint the whole layout was built around.
GitHub refuses any pushed file over 100 MB, and the rejection takes the entire
commit with it -- so one oversized package stopped every other package from
updating. That is why ghidra-noprompt and orca-slicer-git had to be split
into repos of their own, and why several packages could not be built here at all.
A release asset may be 2 GB.
It also fixes pruning. Deleting a package from a git repository only adds a
commit; the bytes stay in history forever, which is how the Pages repo reached
8.39 GB and needed rewriting, and why imhex sat there in three versions at
once. Release assets can actually be deleted, so the database is the source of
truth and publish-release-repo.sh removes any asset it no longer names.
pacman needs no special support for this. The release download URL answers 302
to release-assets.githubusercontent.com and libcurl follows it -- exactly what
[ghidra] and [orca] have relied on all along.
A redirect from the Pages URL would have preserved the old Server, but GitHub
Pages cannot issue one: it serves static files, with no server-side redirect
configuration, and jekyll-redirect-from emits an HTML meta-refresh page rather
than an HTTP status -- pacman would save that HTML as the package. The site also
ships .nojekyll. So the Server URL changes, and scripts/enroll.sh rewrites
an existing entry in place.
The Pages repo now holds only the generated index and its CSVs.
yay has to be able to stop being root. It will not run makepkg as root, so
when it is root it looks up $SUDO_USER / $DOAS_USER and drops to that user
— and only if neither resolves to a real user does it fall back to
systemd-run, which cannot work in a container. With neither set, every AUR
dependency install failed with "System has not been booted with systemd as
init system", and because build-pacman-repo drives makepkg itself for
members, this only broke yay's path — so it looked like several unrelated
packages failing for their own reasons. The container creates a builder user
and yay-noninteractive runs yay as it, which is how yay is meant to be used.
That wrapper also has to keep its stdout clean. makepkg parses it to find out
what is missing, and yay's "Avoid running yay as root/sudo." warning went
onto stdout, where makepkg read it as a dependency list and reported
-> Avoid, -> running, -> yay as missing packages.
check() is off. python-beartype builds and then fails its own
test_poetry case, which aborts packaging and takes everything depending on it
down too. An upstream test that does not survive a container is not a reason to
refuse to ship the package, and heavy-build already passed --nocheck.
build-pacman-repo builds every member of build-pacman-repo.yaml in one
container. allow-failure keeps a package that fails from stopping the other
seventy — but it cannot contain the two failure modes that kill the container
itself: the OOM reaper, which returns no status to catch, and a build that
never finishes, which eats the whole job budget. Either one stops every other
package from updating.
.github/workflows/heavy-build.yml gives those packages a job each, two at a
time, so a death is contained to the package that caused it. They publish into
the same custom-repo release, so this is an implementation detail rather than
another repo to add. .github/heavy-packages.yaml is the manifest:
| knob | what it does |
|---|---|
jobs |
cap on make/ninja parallelism — 16 GB and a translation unit needing 4 GB means 2–3, not 6 |
timeout |
minutes for that package alone |
ccache / srccache |
keep a compiler or source cache between runs |
lto |
false to build without link-time optimisation |
preinstall |
packages to install from [custom] before makepkg resolves anything |
Two of those exist because of specific failures. ggml-sycl-f32-git and
llama.cpp-sycl-f32-git link fine until LTO is on, at which point icpx stops
carrying symbols out of a static archive through to the link and both die on
undefined references from ld-temp.o — the same thing that forced
options=('!lto') on orca-slicer. And ilspy-git needs preinstall because
makepkg resolves runtime dependencies before buildtime ones: the runtime pass
pulls stock dotnet-host out of extra, the buildtime pass then wants
dotnet-sdk-preview-bin, whose dotnet-host-preview-bin conflicts with what
is already installed, and pacman abandons the transaction. Whichever lands
first wins, so ours lands first.
build-one.sh also adds [custom] to the container, which is what makes an
incremental build across two pipelines work at all — a dependency built by
either one is installed from the release instead of rebuilt.
One thing a matrix cannot do is order itself. llama.cpp-sycl-f32-git depends
on ggml-sycl-f32-git from the same manifest, and serialising the list to get
that ordering would cost more runner time than the dependency is worth, so it
resolves on the following run.
patches/<name>.sed is applied to that package's PKGBUILD and
patches/<name>.patch with patch -p1, by both pipelines. The sed form suits
a -git package better: its PKGBUILD churns often enough that a context patch
goes stale, while a targeted substitution keeps applying.
A fixup that changes nothing is reported, and prints the recipe lines it was
aimed at. That matters more than it sounds — the llama.cpp rule was matching
$pkgdir against a recipe that writes ${pkgdir}, and without the report a
missed rule costs a two-hour build to learn nothing.
Two pipelines write to one release. Each seeds its copy of the database, builds for up to two hours, and writes it back — so each is always working from a stale copy, and every guard here exists because that produced a real failure.
The prune only deletes packages the database tracks. It removes an asset the database does not name, which is right when there is one writer and wrong with two: heavy-build's packages are unknown to build-repo's database for the whole time its matrix is running. A run deleted all ten of them once. Now an asset whose package name the database has never heard of is someone else's current work, not our superseded copy.
SEED_TIME keeps anything uploaded after the database was read. Same idea,
narrower: a package we do track that another run has just republished.
Re-indexing rebuilds the database from the assets. The release is what exists; the database is only an index of it. Both publishers compare the two before writing and add back anything published that the database does not name. This has to compare against the assets — an earlier version compared against the published database, which fails exactly when it matters, because once one run has written a database without a package, the published copy does not name it either.
Epoch versions are renamed before anything records them. GitHub rewrites
: to . in release asset names, silently. A pacman epoch puts a colon in the
filename, so dotnet-sdk-preview-bin-1:11.0.0… is stored as …-1.11.0.0…
while the database still says otherwise — pacman 404s on every epoch package,
and the prune deletes them for not matching.
.github/scripts/sanitize-epoch-filenames.py applies the substitution up
front. %VERSION% comes from .PKGINFO, not the filename, so the epoch
survives and only the download URL changes. + and ~ are preserved; only
: is affected. The publisher then verifies every %FILENAME% has a matching
asset and fails naming them if not, because this class of bug is otherwise
invisible: a repo that looks published and 404s on download.
Every package and database is signed, so clients can run SigLevel = Required DatabaseOptional — the same setting stock /etc/pacman.conf uses — instead of
Optional TrustAll. The current fingerprint and a ready-to-paste import command
are on xerootg.github.io; the public key is served
at /xerootg.asc.
.github/scripts/sign-pacman-repo.sh does all of it, for all three repos. It
needs only gpg and coreutils, so the same script runs inside the Arch build
containers and directly on the Ubuntu runner for backfills.
Two things drove the design:
build-pacman-repo cannot sign. It shells out to repo-add --quiet --nocolor with no way to pass --sign, and has no GPG support at all. So
signing is a separate pass over the repository directory rather than something
threaded through the builder.
A pass over the directory is also the only correct approach. Once
SigLevel = Required is in play, pacman wants a .sig beside every package,
including ones that were not rebuilt this run. Signing only what a build
produced would leave the untouched packages permanently unverifiable.
The script is idempotent and rotation-aware: an existing signature is kept only if it still verifies against the key currently loaded, so changing the secret and re-running re-signs whatever no longer checks out.
The database is rewritten on every build. If the signing key were ever missing
while signatures were already present, the old .db.sig would no longer match
the new .db, and pacman rejects a repo with a bad database signature far
more harshly than one with none — every client breaks at once. The script
refuses to run in that state rather than publishing it.
.github/workflows/sign-backfill.yml signs what is already published without
rebuilding it. The release-hosted repos only rebuild when upstream moves —
ghidra roughly monthly — so without this, turning signing on would leave those
packages unverifiable for weeks. It downloads the release assets, signs
whatever is missing or stale, uploads the .sig files, and refreshes the
release notes. It is also the tool to run after rotating the key.
Release notes now live in .github/release-notes/. They used to be inline
heredocs that only ran when a release was first created, which meant the
install instructions published on an existing release could never be corrected.
build-repo-index.py reads the issuer fingerprint out of each <repo>.db.sig
with gpg --list-packets. No secret is involved — the issuer is cleartext in
the signature packet — and it is the one source that cannot drift, because it
is whatever key actually signed the database clients download. A repo is
advertised as Required only when its own database signature verifies as
present, so the page can never tell someone to enable a setting that would
then reject the repo.
| secret | contents |
|---|---|
GPG_SIGNING_KEY |
ASCII-armoured signing subkey, secret half |
GPG_PASSPHRASE |
passphrase for it |
Repository-level secrets, so all four workflows can read them. Only the signing subkey is exported — the certifying primary key stays offline, so a compromise of CI costs a subkey that can be revoked without abandoning the identity or asking every user to re-trust a new fingerprint.