diff --git a/rust/api/README.md b/rust/api/README.md index 3ce52c60..6ef95c05 100644 --- a/rust/api/README.md +++ b/rust/api/README.md @@ -2,6 +2,134 @@ WARNING: This file is publicly readable, do NOT put anything secret in here. +## Deploying gkapi + +There is **no CI deployment for this crate**. `deploy.yml` builds the Hugo site and +publishes it to GitHub Pages; it never touches the API. `rust-api-tests.yml` only runs +fmt, build and test. Merging a change to `rust/api` therefore ships nothing: the binary +on vega has to be replaced by hand, and has in the past sat months behind main. + +There is also no Rust toolchain on vega (`~gkapi/.cargo` exists but `bin/` is empty), so +the binary is built elsewhere and copied. vega and nova are both Ubuntu 24.04 on the same +glibc, so a release build from nova runs there as-is. Check with `ldd --version` on both +before relying on that. + +### The step that is easy to miss + +The service runs as the unprivileged `gkapi` user and binds **port 80** for the HTTP-01 +ACME challenge server, which needs `CAP_NET_BIND_SERVICE`. That capability lives on the +binary as a file capability, and **`cp` does not preserve it**. Replace the binary without +re-running `setcap` and the service binds 443, logs `Listening on 0.0.0.0:443`, then panics +at `main.rs` with `PermissionDenied` on the port 80 bind and systemd restart-loops until it +gives up. + +The same trap catches the rollback: copying a backup *back* into place also produces a file +with no capability, so a panicking service stays panicking and it looks like the new build +is at fault. `mv` preserves the capability because it keeps the inode; `cp` does not. + +### Procedure + +```bash +# 1. build on nova, from main, and prove the build contains what you expect +cd ~/code/freenet/web/main/rust && cargo build --release -p ghostkey-api +strings target/release/ghostkey-api | grep -c payment_claim # sanity: expect non-zero + +# 2. upload and re-check after transfer +scp target/release/ghostkey-api vega:/tmp/ghostkey-api.new +ssh vega 'strings /tmp/ghostkey-api.new | grep -q payment_claim && echo ok' + +# 3. stage it at its final location, fully prepared, WITHOUT displacing the live binary +ssh vega ' + sudo cp /tmp/ghostkey-api.new /home/gkapi/bin/ghostkey-api.staged + sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api.staged + sudo chmod 775 /home/gkapi/bin/ghostkey-api.staged + sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api.staged + sudo getcap /home/gkapi/bin/ghostkey-api.staged +' +# Must print: /home/gkapi/bin/ghostkey-api.staged cap_net_bind_service=ep +# If it does not, STOP. Nothing has changed yet and the live binary is untouched. + +# 4. swap with two adjacent renames, then restart +ssh vega ' + set -e + STAMP=$(date +%Y%m%d-%H%M%S) + sudo mv /home/gkapi/bin/ghostkey-api /home/gkapi/bin/ghostkey-api.rollback-$STAMP + sudo mv /home/gkapi/bin/ghostkey-api.staged /home/gkapi/bin/ghostkey-api + echo "rollback binary: /home/gkapi/bin/ghostkey-api.rollback-$STAMP" + sudo systemctl reset-failed gkapi + sudo systemctl restart gkapi +' +``` + +Two things about that shape are deliberate, and both exist to avoid a worse failure than +the one being fixed: + +- **Prepare the capability on the staged file, and verify it, before anything is + displaced.** The `getcap` gate in step 3 is the whole point of splitting the steps: if it + fails you stop with the running service completely untouched. Do not restart to see what + happens. +- **Do not `mv` the old binary away in one command and `cp` the new one in with another.** + Between those two the path does not exist, and anything that restarts the unit in that + window (a crash, an OOM, a reboot, someone running `systemctl restart` out of order) fails + with `status=203/EXEC` and stays down. Step 4 closes that to two adjacent renames. + `mv` is also what carries the capability across, since it keeps the inode. + +`sudo` on `getcap` is not decoration: it lives in `/usr/sbin`, which is not on a normal +user's `PATH`, so a bare `getcap` can fail as "command not found" and skip the check +entirely. + +### Verifying a deploy + +```bash +ssh vega 'systemctl is-active gkapi; strings /home/gkapi/bin/ghostkey-api | grep -c payment_claim' +curl -s https://gkapi.freenet.org/ # {"message":"Hello, world!"} +curl -s -o /dev/null -w '%{http_code}\n' http://gkapi.freenet.org/.well-known/acme-challenge/probe +``` + +That last one matters: a 404 means the port 80 challenge listener is up. Connection refused +means the capability is missing and certificate renewal will fail at the next attempt even +if HTTPS looks healthy. + +Then confirm every donation tier still resolves its notary keypair, which is a separate +failure mode from the binary (see the tier comment in +`hugo-site/themes/freenet/layouts/shortcodes/stripe-donation-form.html`). + +Note that `/create-donation` has no dry-run mode: each call creates a real PaymentIntent in +the live Stripe account. Nothing is charged and no card is attached, so these are harmless +abandoned intents, exactly what a visitor clicking between the amount radios produces. Run +the loop once after a deploy; do not wrap it in a retry-until-success script. + +```bash +for a in 1 5 20 50 100 500 2500 10000; do + curl -s -X POST https://gkapi.freenet.org/create-donation \ + -H 'Content-Type: application/json' -d "{\"amount\":$((a*100)),\"currency\":\"usd\"}" \ + | grep -q notary_certificate_base64 && echo "\$$a ok" || echo "\$$a FAIL" +done +``` + +### Rollback + +Substitute `` with the timestamp step 4 printed. + +```bash +ssh vega ' + set -e + sudo cp /home/gkapi/bin/ghostkey-api.rollback- /home/gkapi/bin/ghostkey-api.staged + sudo chown gkapi:gkapi /home/gkapi/bin/ghostkey-api.staged + sudo chmod 775 /home/gkapi/bin/ghostkey-api.staged + sudo setcap cap_net_bind_service+ep /home/gkapi/bin/ghostkey-api.staged + sudo getcap /home/gkapi/bin/ghostkey-api.staged + sudo mv /home/gkapi/bin/ghostkey-api.staged /home/gkapi/bin/ghostkey-api + sudo systemctl reset-failed gkapi && sudo systemctl restart gkapi +' +``` + +The `setcap` line is required on the way back too, and this is the part that is genuinely +easy to get wrong under pressure: copying a backup *into place* produces a file with no +capability, so the service carries on panicking and it reads as "the new build is broken" +rather than "the copy dropped a capability". If a rollback does not fix the panic, run +`sudo getcap` on the live binary before concluding anything about the build. + ## letsencrypt Verify that certificate was automatically renewed by root cron job on vega by looking at write times