appliance: a reference stack, a bootable image, and app-store packages by bkeroack · Pull Request #708 · epochbtc/satd

bkeroack · 2026-09-09T13:54:23Z

Three ways to run satd beyond a bare binary — a docker-compose reference
stack, a downloadable appliance image, and app-store package sources — plus
the two satd-side changes they needed.

Every bundled application here is a compatibility claim, so each one is
exercised rather than asserted. Everything below was verified by building
and booting the real artifacts, not by inspection.

What lands


`contrib/stack/`	compose: satd with RPC, Electrum, Esplora, metrics and optional MCP, each TLS-terminated by a certificate the install issues for itself. Overlays for LND (Neutrino), Core Lightning, RTL, a Cashu mint, BTCPay, an Ark server, and a Caddy proxy.
`contrib/appliance/`	a bootable VM (`core` headless, `desktop` with Sparrow/Electrum/Liana) and a live ISO, built from one provisioning tree.
`contrib/packaging/`	an Umbrel app; a requirements document for StartOS.
`sat-cli` / `sat-tui`	`-rpctls`, `-rpccacert`, `-rpcclientcert`/`-rpcclientkey`.
container image	ships `sat-tui`, a `HEALTHCHECK`, and the stack's first-run tooling.

signet is the default everywhere. There is no prune option anywhere:
Electrum and Esplora both require txindex, which excludes pruning.

Verification

core image — 32/32 boot checks; desktop image — 30/30. Booted
under QEMU, checked through the guest agent and through forwarded ports,
with every TLS surface verified from outside the guest against the CA the
install generated. Each certificate probe is paired with the negative
control that the same handshake without the CA must fail.
live ISO — boots and completes its whole first-boot sequence.
stack — regtest bring-up, every TLS listener probed externally, LND
synced to tip over Neutrino, RTL served through the proxy.
Ark — a live Ark server on regtest reporting synced: true, with a
real signer pubkey and forfeit address.
clippy, 34 new Rust tests, shellcheck at error level, all six overlays
parse, mdbook builds.

Two satd-side changes worth reviewing on their own

sat-cli/sat-tui could not speak to satd's own TLS listener. satd has
served TLS on -rpctlsbind for releases, but both clients formatted
http:// and had no CA option — so the deployment most in need of TLS was
the one that could not have it end to end. Two failures are deliberately
loud: TLS material without -rpctls is an error (ignoring it would send the
RPC credential in the clear and look like success), and a CA file containing
no certificates is an error (reqwest parses such a file into an empty list,
so a wrong path would add no anchor and fail later as a generic handshake
error). The tests run real handshakes against an acceptor built by this
crate's own server half; the positive case is paired with an untrusted
server, the wrong CA, and a name mismatch.

contrib/systemd/satd.service had a real bug, unrelated to this work
but found by it: ExecStartPost relaxed only ${SATD_DATADIR}/.cookie, so
on signet, testnet4 and regtest — where the cookie lives under the network's
subdirectory — members of the satd group could not run sat-cli at all.
Worth cherry-picking ahead of the rest if you'd rather not wait for this.

Design decisions that are mine, not the plan's

Built with mmdebstrap, not Packer. Packer's QEMU builder drives
Debian's installer through a preseed inside a VM: it needs KVM to finish in
sensible time, needs the netinst ISO, and fails in ways you diagnose from a
VNC console. Building the filesystem directly and installing GRUB onto a
loop device runs with no KVM, inside a container, in minutes — which is what
made the CI boot gate practical rather than aspirational. Reversible: only
build.sh would change.

satd runs natively on the appliance, with the overlays reaching it
through the docker network's gateway. That needed two changes in
contrib/stack: overlays address ${SATD_HOST:-satd}, and they no longer
depends_on a satd service that does not exist in the appliance's project.

The reverse proxy uses one port per app, not one path per app. RTL and
BTCPay build absolute URLs from their own root, so path-mounting them breaks
on their next release.

The internal RPC port is pinned to 8332 on every network, so overlays,
the proxy and the store packages address one fixed port. The cost, which is
documented: in-container sat-cli needs -rpcport on non-mainnet.

Not shipped, deliberately

The Cashu wallet CLI is included, but the Ark overlay is experimental
— every arkd setting in it was established by running the binary rather than
read from a specification, so expect it to need attention on a version bump.

The StartOS package is a requirements document, not code. A StartOS
package is a TypeScript project built with Start9's SDK, and the SDK's shape
has changed across versions; written against a guessed API it would review
well and not build.

Known gaps

The cashu install is proven via pip; the appliance path goes through
pipx --pip-args, which has not been exercised in a full image build.
The Ark overlay has no automated smoke coverage — it was verified by hand.
arm64 is structured for but neither built nor tested.
Object storage for published images is unresolved; sign-tarballs.sh --images signs a checksum manifest, but nothing publishes the images yet.

Review round 1

Five defects found and fixed in 30335559. Each was verified against the
built artifact before being fixed, not just read out of the source.

	Defect	Evidence
P1	RTL ignored the configured password and accepted the upstream default, in front of LND's admin macaroon	`RTL_PASSWORD` appears nowhere in the pinned image; logging in with `password` returned http 200
P1	`install-to-disk` aborted after formatting the target, leaving it unbootable	rsync excludes `/dev`, `/proc`, `/sys`; the chroot bind mounts then fail under `set -e`
P1	BTCPay published its login and payment configuration in clear text on every interface	a published port is DNAT'd through `forward` (policy accept), so the default-deny input chain never saw it
P2	Mainnet did not start: `--mainnet` is not a flag satd has, and it is the Umbrel default	`satd --mainnet` exits with "unexpected argument"; `--chain=main` starts a mainnet node
P2	`set-network` and `disable` could not parse the overlay files, orphaning containers	overlay secrets use `${VAR:?}`, a hard parse error; only `enable` sourced them

Both fixes with a security consequence are covered by tests that were
perturbation-checked — reverting the fix fails a named check:

contrib/stack/tests/compose-test.sh (new, no docker or network) asserts
all five statically; all ten perturbations are caught.
smoke.sh now asks RTL to reject its upstream default password as well as
accept the generated one. Reverting the RTL fix makes it report
expected http 401, got 200.

Two of these were found only because the checks were run against the running
system: the first version of the RTL login test passed for the wrong reason
(wrong path, and express's static catch-all answers 200 to anything), and the
port checker silently checked nothing on compose.yml because
compose.*.yml does not match it.

Also fixed the Compose files parse CI step, which exported
POSTGRES_PASSWORD only for the cashu invocation and so failed on btcpay.

CI cost

The reference-stack bring-up and the appliance image build each compile satd
from scratch, about twenty minutes apiece, so they no longer run on every
pull request. Both are release gates now: v* tags (the release workflow's
own trigger) and workflow_dispatch. They are skipped rather than removed on
PRs, so branch protection still gets a reported context.

To run them on a PR that changes this surface, add the appliance-ci
label — labeled is in the trigger list, so applying it starts the run with
no extra push. The static checks over the certificate script, the healthcheck
and the compose/appliance invariants still run on every PR that touches them,
and cost seconds.

Both heavy jobs passed on 30335559 before this gating landed, so the fixes
above are verified against a real stack bring-up and a real QEMU boot.

Supersedes

The paused local branch feature/umbrel-app (contrib/umbrel/). Note its
Dockerfile change added --bin sat-tui to the build but not to the
cargo-chef cook line, which would have silently busted the dependency
cache on every image build; this PR does both.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

The runtime image carried satd and sat-cli only, so `docker exec -it satd sat-tui` — the obvious way to watch a containerised node — did not work, and the Umbrel/StartOS packages that expose sat-tui as their terminal would each have had to bolt it on themselves. Cook and build now select the same three bins, which the cargo-chef cache requires. The HEALTHCHECK reports liveness by default rather than readiness. The probe cannot see the daemon's credentials (the CMD may set -rpcuser, and the cookie may be unreadable) or its network flags, so it treats any HTTP status line from the RPC listener — 401 included — as healthy, which is the strongest claim it can honestly make. Operators who want a real readiness gate, which is what `depends_on: condition: service_healthy` needs, point SATD_HEALTH_URL at /readyz; contrib/stack does exactly that. No curl or wget: the runtime image is deliberately thin, so the probe speaks HTTP over bash's /dev/tcp. That transport is easy to get subtly wrong, so it has a test — a bodyless request assembled with command substitution loses the blank line that terminates it, and a healthy node then reads as down until the probe's own timeout fires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

satd has served TLS on -rpctlsbind for several releases, but both shipped clients formatted `http://{host}:{port}/` and had no CA option. An operator who turned RPC TLS on therefore had to keep the plain listener bound purely so the project's own tooling could reach the node — which means the deployment most in need of TLS was the one that could not have it end to end. Four flags on each binary, spelled the same and accepted in Core's single-dash form: -rpctls, -rpccacert, -rpcclientcert, -rpcclientkey. They are additive; nothing changes for an invocation that does not pass them. The shared implementation lives in tls-config beside the acceptor the same operator configured, behind a `client` feature so satd does not build it. Two behaviours are deliberate: - TLS material without -rpctls is an error, not a warning. Ignoring it would send the RPC credential over plain HTTP and look like success. - A CA file containing no PEM certificates is an error. reqwest parses such a file into an empty list, so pointing -rpccacert at a private key or the wrong path would add no anchor and surface later as a generic handshake failure, indistinguishable from the node being down. The tests run a real handshake against an acceptor built by this crate's server half, because loading a PEM proves nothing about verification: the positive case is paired with an untrusted server, the wrong CA, and a name mismatch, all of which must fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

A docker-compose stack that runs satd with RPC, Electrum, Esplora, metrics and (optionally) MCP all on, each TLS-terminated by a certificate the install issues for itself. It is the shared substrate the rest of the appliance work builds on: the image runs this stack, and the Umbrel and StartOS packages are derived from compose.yml, so the satd service is defined once and cannot drift between them. tls/mkca.sh is the single certificate script all three use. It creates a CA for one install only and issues one leaf that every surface presents, so a client imports one thing and trusts everything. Reissue happens on expiry, on a change to the machine's names or addresses, or on --force; the CA is never rotated automatically, because that invalidates trust every client already established. Nothing key-like is in any image — the smoke test asserts that, and it is what makes the appliance image redistributable. Two decisions worth stating. Plain RPC/Electrum/Esplora/metrics listeners bind the compose network and are never published: they exist because the overlay containers cannot be taught to trust a private CA, and nothing unencrypted leaves the host. And the internal RPC port is pinned to 8332 on every network so overlays, the proxy and the app-store packages address one fixed port — the cost, documented, is that in-container sat-cli needs -rpcport on a non-mainnet stack. LND runs in Neutrino mode because satd implements no raw ZMQ topics and rejects them by design; Neutrino needs none, pulling BIP157/158 filters over P2P from the peerblockfilters listener the stack turns on. Core Lightning is unaffected and runs as a full-node client. The image now also carries openssl, mkca.sh, satd-init and the config template, so a deployment that cannot mount repository files gets identical first-run behaviour. smoke.sh brings the stack up on regtest and probes every TLS listener from outside the container against the generated CA with -verify_return_error, paired with the negative control that the same handshake without the CA must fail — a probe that would pass unverified proves nothing about the certificate. Verified locally: core stack, LND syncing to tip over Neutrino, and RTL served through the proxy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

A bootable image with satd, its tooling and — in the desktop flavour — Sparrow, Electrum, Liana and a Cashu wallet already pointed at the node's Electrum server. Signet by default, because it is the only network on which the whole thing is a one-evening exercise; `satd-appliance set-network mainnet` switches and refuses below 1.5 TB free. Built with mmdebstrap and a GRUB install onto a loop device rather than Packer. Packer's QEMU builder drives Debian's installer through a preseed inside a running VM: it needs KVM to finish in sensible time, needs a 700 MB ISO, and fails in ways you diagnose by watching a VNC console. This runs in a container with no root and no KVM on the host, takes minutes, and every failure is a shell command that exited non-zero. That is what made the boot gate practical rather than aspirational. Everything unique to an install is created on first boot — the console password, the CA and server certificate, the MCP token, the disk's real size — because an image that shipped any of them would be an image where every download shared them. 90-cleanup.sh asserts none of them are present and refuses to finish a build that would ship one. satd runs natively under systemd; the overlays run as containers from contrib/stack's files, used unmodified. Making that sharing real took two changes there, included here: the overlays address the node as ${SATD_HOST:-satd} so the same file works against a compose service and against the host, and they no longer depend on a satd service that does not exist in the appliance's project. Four defects found by booting the artifact rather than reasoning about it: - grub-mkconfig's UUID probe comes back empty in a build container, so 10_linux fell back to the BUILD HOST's loop device. The image booted, the kernel started, and the initramfs waited forever for a /dev/loopNpM that exists on no machine but the builder. Now rewritten to the UUID and asserted. - `tr -dc < /dev/urandom | head -c 18` killed first boot outright: head closes the pipe, tr dies of SIGPIPE, and pipefail aborts the unit. - `systemctl enable --now satd` from inside first boot deadlocked against its own Before= ordering — satd waited for first boot, first boot waited for satd. `--no-block` queues the job instead. - sat-cli has no signet selector and derives the cookie path from the chain, so satd-appliance could not authenticate on any network but mainnet. It now uses the stable rpc-cookie symlink satd-init maintains. The last of those exposed a real bug in the shipped systemd unit, fixed here: ExecStartPost relaxed only ${SATD_DATADIR}/.cookie, so on signet, testnet4 and regtest — where the cookie lives under the network's subdirectory — members of the satd group could not run sat-cli at all. boot-test.sh boots the artifact under QEMU and checks it through two channels: the guest agent for what is inside, and forwarded ports for the TLS surfaces, verified from outside against the CA the agent hands out. Checking a certificate from inside the guest proves much less than connecting to it the way a client will. KVM where there is one, TCG where there is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

Three tiers, in increasing cost, each gated on the paths it covers: - the certificate script and the container healthcheck probe: seconds, no docker, plus shellcheck at error level over every shell file in this work; - the reference stack on regtest, with every TLS listener probed from outside the container against the CA the stack generated, LND syncing over Neutrino and RTL served through the proxy; - the appliance image, built and then booted under QEMU. Hosted runners only. satd is public, and a `pull_request` job on a self-hosted runner would let a fork PR execute arbitrary code on a maintainer machine. Path gating is per-job rather than a workflow-level `paths` filter: a path-skipped workflow never reports its contexts and leaves a PR waiting on a status that never arrives, where a skipped job reports "skipped" and satisfies branch protection. `sign-tarballs.sh --images <dir>` signs appliance images. They are several GB each and exceed GitHub's 2 GB per-asset limit, so the images go to object storage and what lands on the release is a manifest of their SHA-256 sums with a minisign signature over it — which authenticates every image in the list just as well, next to the tarball signatures where anyone verifying already knows to look. The script deliberately does not upload the images themselves: it has no object-storage credentials and should not acquire any. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

Sources for the app-store packages, versioned with satd and published from their own repositories, which is how both stores consume them. Both contain satd and nothing else — the daemon, sat-cli, sat-tui and MCP. Umbrel and StartOS users compose Lightning, BTCPay and wallets from their own stores, and a package that bundled a second copy of software the store already offers would be worse than useless. The best-effort notice that applies to the appliance image therefore does not apply here: there is no third-party software to disclaim. The Umbrel app derives its satd service from contrib/stack/compose.yml and runs the same satd-init the reference stack does — which is why that script and mkca.sh are baked into the container image. A package that reimplemented first-run behaviour would drift from the stack within a release. exports.sh mirrors the official `bitcoin` app's variable names, so an app that already knows how to find Bitcoin Core finds satd: it speaks Core's JSON-RPC and cookie format, so nothing else has to change. startos/ is requirements, not a package. A StartOS package is a TypeScript project built with Start9's SDK, and that SDK's shape has changed across StartOS versions; written against a guessed API it would look right in review and not build. The README records what the package must declare — interfaces, health check, backup set, the config options that must NOT exist because txindex is not optional here — so writing it is mechanical once the target version is chosen. It also records the one open question: the certificate renews on container start there, with no scheduler to run the daily timer the appliance uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

A new chapter covering all three deliverables together, because they share one node configuration and one certificate scheme and are best understood that way: what each is, which ports are published and which deliberately are not, why there is no prune option anywhere, how the CA and certificate work and how to trust them from each kind of client, and what is actually checked in CI rather than asserted. The support policy appears verbatim here, in contrib/appliance/README.md, in contrib/stack/README.md and on the image's own welcome page, so that someone who meets the bundled software in any of those places meets the same statement about it. packaging.md gains a pointer to the chapter and an accurate description of what the container image now carries, since a packager building on that image inherits its first-run tooling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

Review of the appliance/stack branch turned up five problems, each verified against the artifact rather than the source before being fixed. RTL was reachable with a well-known password. RTL v0.15.x reads APP_PASSWORD and nothing else; the overlay passed RTL_PASSWORD, which the container ignores, so RTL fell back to the password in the config it generates for itself -- the literal string "password" -- in front of LND's admin macaroon, on a port the proxy publishes. Confirmed by grepping the pinned image (RTL_PASSWORD appears nowhere in it) and then by logging in. Now passed as APP_PASSWORD and required rather than defaulted. Installing to disk destroyed the target. rsync excludes /dev, /proc and /sys, so those directories do not exist on the new root; the chroot bind mounts that follow failed on the first one, under `set -e`, after the disk had been formatted and before GRUB ran. The mountpoints are now created (with /run, /tmp, /mnt and /media, which the installed system needs regardless), and the mounts are unwound by a trap so a later failure does not leave the target's filesystems held. Mainnet did not start at all. `NETWORK=mainnet` rendered `--mainnet`, and satd has bare flags for the test networks but none for mainnet, so the daemon exited on an unknown argument -- including in the Umbrel package, where mainnet is the default. Both now use `--chain=`, which accepts every name. Overlay lifecycle commands could not parse their own compose files. The overlays declare secrets as ${VAR:?}, a hard parse error rather than an empty string, and only `enable` sourced overlay.env: `set-network` left overlays on the old chain, and `disable` swallowed a failed teardown and then removed the marker, orphaning containers nothing could name again. Secrets are now loaded inside run_compose, the one place that runs compose, and a failed teardown keeps the marker. `enable ark` and `enable lightning` also generated no secret at all and so could never have started. BTCPay published its login and payment configuration in clear text on every interface, which the proxy neither replaced nor protected -- and a published container port is DNAT'd through the forward chain, so the appliance's default-deny input chain never saw it. It now binds loopback, and the proxy serves it over TLS on 49393. Tests: contrib/stack/tests/compose-test.sh asserts each of these statically, and smoke.sh now asks RTL to reject its upstream default password as well as accept the generated one. Both were perturbation-checked: reverting each fix fails a named test, and reverting the RTL fix specifically shows "password" logging in with http 200. Also fixes the Compose-parse CI step, which exported POSTGRES_PASSWORD only for the cashu invocation and so failed on the btcpay overlay. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

+    treating that as the end of the block silently checked nothing.
+    """
+    out, in_ports, blocks = [], False, 0
+    for line in open(path):


The reference-stack bring-up and the appliance image build each compile satd from scratch and take about twenty minutes apiece, so between them they cost roughly forty minutes of runner time on every pull request that touches contrib/stack or contrib/appliance. That is the wrong place to spend it: both are artifact-shaped gates whose result matters at release, not per commit. Both now run on `v*` tags -- the same trigger the release workflow uses -- and on workflow_dispatch. They are skipped, not removed, on pull requests, so branch protection still sees a reported context. The escape hatch is the `appliance-ci` label. `labeled` is in the trigger list, so applying it to an open PR starts the run without needing another push; a PR that genuinely changes this surface can still be gated on the full thing. On a labelled PR the image matrix stays core-only, as it already did. Unchanged: the static checks over the certificate script, the container healthcheck and the compose/appliance invariants. Those are seconds, need no docker or network, and keep running on every PR that touches them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9Nb3m8HuhNbjkJYp9Frmt

bkeroack force-pushed the feature/appliance-reference-stack branch from 15aa6c0 to 63a6ba5 Compare September 9, 2026 14:21

bkeroack and others added 7 commits September 9, 2026 09:17

bkeroack force-pushed the feature/appliance-reference-stack branch from 63a6ba5 to 3c50a86 Compare September 9, 2026 15:21

github-code-quality Bot found potential problems Sep 9, 2026

View reviewed changes

Comment thread contrib/stack/tests/published-ports.py

treating that as the end of the block silently checked nothing.

"""

out, in_ports, blocks = [], False, 0

for line in open(path):

bkeroack added the appliance-ci Run the heavy appliance CI jobs (stack bring-up + image build) on this PR label Sep 9, 2026

Navigation Menu

Uh oh!

appliance: a reference stack, a bootable image, and app-store packages - #708

appliance: a reference stack, a bootable image, and app-store packages#708
bkeroack wants to merge 9 commits into
masterfrom
feature/appliance-reference-stack

bkeroack commented Sep 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

bkeroack commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What lands

Verification

Two satd-side changes worth reviewing on their own

Design decisions that are mine, not the plan's

Not shipped, deliberately

Known gaps

Review round 1

CI cost

Supersedes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bkeroack commented Sep 9, 2026 •

edited

Loading