appliance: a reference stack, a bootable image, and app-store packages

.github/workflows/appliance.yml

@@ -0,0 +1,263 @@
+# Appliance CI — the reference stack and the downloadable VM image.
+#
+# Three things are gated here, in increasing cost. Only the first runs on an
+# ordinary pull request:
+#
+#   1. The static checks: the certificate script, the container healthcheck
+#      probe, and the compose/appliance invariants. Seconds, no docker, no
+#      network. Runs on any PR that touches them.
+#   2. The reference stack (contrib/stack): brought up on regtest, with every
+#      TLS listener probed from outside the container against the CA the
+#      stack generated, plus LND syncing to the node over Neutrino and RTL
+#      served through the proxy. This is where a claim like "Sparrow can talk
+#      to this" is actually checked. ~20 minutes.
+#   3. The appliance image: built with mmdebstrap, booted under QEMU, and
+#      inspected through the guest agent. This is the "the image is not
+#      broken" gate, and it boots the artifact itself rather than a
+#      test-only variant. ~20 minutes.
+#
+# 2 and 3 each compile satd from scratch, so together they cost around forty
+# minutes of runner time. They run on `v*` tags — the same trigger the release
+# workflow uses — and on workflow_dispatch, not on every pull request.
+#
+# To run them on a PR that actually changes this surface, add the
+# `appliance-ci` label. `labeled` is in the trigger list below, so applying it
+# starts the run; no push is needed.
+#
+# Everything runs on GitHub-hosted runners. satd is public, and a
+# `pull_request` job on a self-hosted runner would let a fork PR execute
+# arbitrary code on a maintainer machine.
+#
+# On a tag the same build runs for every published format and uploads the
+# artifacts; signing stays a manual post-tag step, as it is for tarballs.
+
+name: Appliance
+
+on:
+  pull_request:
+    # `labeled` so that adding `appliance-ci` to an open PR starts the heavy
+    # jobs without needing a fresh push.
+    types: [opened, synchronize, reopened, labeled]
+  push:
+    tags:
+      - 'v*'
+  workflow_dispatch:
+    inputs:
+      flavor:
+        description: "Which image flavour to build"
+        required: false
+        default: "core"
+        type: choice
+        options: ["core", "desktop", "both"]
+
+concurrency:
+  group: appliance-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  RUST_TOOLCHAIN: '1.93.0'
+
+jobs:
+  # Job-level path gating rather than a workflow-level `paths` filter: a
+  # path-skipped workflow never reports its contexts, which leaves a PR
+  # waiting on a status that will never arrive. A skipped *job* reports
+  # "skipped", which satisfies branch protection.
+  changes:
+    name: detect appliance changes
+    runs-on: ubuntu-24.04
+    timeout-minutes: 5
+    outputs:
+      stack: ${{ steps.filter.outputs.stack }}
+      appliance: ${{ steps.filter.outputs.appliance }}
+      scripts: ${{ steps.filter.outputs.scripts }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - id: filter
+        run: |
+          set -euo pipefail
+          if [[ "${{ github.event_name }}" != "pull_request" ]]; then
+            echo "stack=true"     >> "$GITHUB_OUTPUT"
+            echo "appliance=true" >> "$GITHUB_OUTPUT"
+            echo "scripts=true"   >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          base="${{ github.event.pull_request.base.sha }}"
+          files="$(git diff --name-only "$base"...HEAD)"
+          echo "changed files:"; echo "$files" | sed 's/^/  /'
+          match() { grep -qE "$1" <<< "$files" && echo true || echo false; }
+          # The stack job also covers the runtime image, since it ships
+          # mkca.sh, satd-init and the config template.
+          echo "stack=$(match '^(contrib/stack/|contrib/docker/|Dockerfile$)')" >> "$GITHUB_OUTPUT"
+          echo "appliance=$(match '^(contrib/appliance/|contrib/stack/|contrib/systemd/)')" >> "$GITHUB_OUTPUT"
+          # Widened past tls/tests: compose-test.sh asserts invariants over the
+          # compose files, the appliance CLI and the Umbrel package, so it has
+          # to run when any of those change.
+          echo "scripts=$(match '^contrib/(stack/|docker/|appliance/bin/|packaging/umbrel/)')" >> "$GITHUB_OUTPUT"
+
+  # Cheap and always worth running when touched: no docker, no network.
+  scripts:
+    name: stack + appliance scripts
+    needs: changes
+    if: needs.changes.outputs.scripts == 'true'
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - name: mkca.sh
+        run: contrib/stack/tests/mkca-test.sh
+      - name: satd-healthcheck
+        run: contrib/docker/tests/healthcheck-test.sh
+      - name: compose + appliance invariants
+        run: contrib/stack/tests/compose-test.sh
+      - name: shellcheck
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y -qq shellcheck
+          # `-S error` keeps this a correctness gate rather than a style one;
+          # the tree has its own conventions that shellcheck disagrees with.
+          shellcheck -S error \
+            contrib/stack/tls/mkca.sh \
+            contrib/stack/satd/satd-init \
+            contrib/docker/satd-healthcheck \
+            contrib/appliance/build.sh \
+            contrib/appliance/bin/satd-appliance \
+            contrib/appliance/firstboot/satd-appliance-firstboot \
+            contrib/appliance/provision/*.sh \
+            contrib/stack/tests/compose-test.sh
+
+  stack:
+    name: reference stack (regtest, TLS probed)
+    needs: changes
+    # Release trigger, dispatch, or an explicitly labelled PR. Roughly twenty
+    # minutes of runner time, most of it compiling satd, which is not worth
+    # spending on every push to every branch.
+    if: needs.changes.outputs.stack == 'true' && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'appliance-ci'))
+    runs-on: ubuntu-24.04
+    # Building the runtime image compiles satd from scratch on a cold cache,
+    # and two stack bring-ups follow it.
+    timeout-minutes: 90
+    steps:
+      - uses: actions/checkout@v4
+      - name: Free disk space
+        run: |
+          set -euo pipefail
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+          df -h /
+      - name: Build the runtime image
+        # The real Dockerfile, not a shortcut: the image under test has to be
+        # the one that ships mkca.sh, satd-init and the config template.
+        run: docker build -t satd:ci .
+      - name: Core stack
+        run: SATD_IMAGE=satd:ci contrib/stack/tests/smoke.sh
+      - name: With the Lightning and proxy overlays
+        run: |
+          SATD_IMAGE=satd:ci SATD_SMOKE_PORT_BASE=21600 \
+            contrib/stack/tests/smoke.sh --with lightning --with proxy
+      - name: Compose files parse
+        run: |
+          set -euo pipefail
+          # Every secret an overlay declares as ${VAR:?} has to be in the
+          # environment for the whole loop, not just for the overlay that
+          # prompted it: a missing one is a parse error, so the step fails
+          # on the first overlay that needs one it was not given.
+          export RTL_PASSWORD=ci
+          export MINT_PRIVATE_KEY=ci
+          export POSTGRES_PASSWORD=ci
+          export ARK_POSTGRES_PASSWORD=ci
+          for overlay in lightning cln btcpay ark proxy; do
+            docker compose -f contrib/stack/compose.yml \
+              -f "contrib/stack/compose.$overlay.yml" \
+              --env-file contrib/stack/.env.example \
+              config > /dev/null
+            echo "ok: $overlay"
+          done
+          # cashu extends the Lightning overlay, so it only parses with it.
+          docker compose \
+            -f contrib/stack/compose.yml \
+            -f contrib/stack/compose.lightning.yml \
+            -f contrib/stack/compose.cashu.yml \
+            --env-file contrib/stack/.env.example config > /dev/null
+          echo "ok: cashu"
+          # And all of them together, which is what the appliance can end up
+          # running and what no single-overlay parse would catch.
+          docker compose -f contrib/stack/compose.yml \
+            -f contrib/stack/compose.lightning.yml \
+            -f contrib/stack/compose.cashu.yml \
+            -f contrib/stack/compose.btcpay.yml \
+            -f contrib/stack/compose.ark.yml \
+            -f contrib/stack/compose.proxy.yml \
+            --env-file contrib/stack/.env.example config > /dev/null
+          echo "ok: all overlays together"
+
+  image:
+    name: build and boot the appliance image
+    needs: changes
+    # Same gate as `stack`: tags, dispatch, or the `appliance-ci` label.
+    if: needs.changes.outputs.appliance == 'true' && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'appliance-ci'))
+    runs-on: ubuntu-24.04
+    timeout-minutes: 90
+    strategy:
+      fail-fast: false
+      matrix:
+        flavor: ${{ github.event_name == 'pull_request' && fromJSON('["core"]') || fromJSON('["core","desktop"]') }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Free disk space
+        run: |
+          set -euo pipefail
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+          sudo docker image prune --all --force > /dev/null 2>&1 || true
+          df -h /
+      - name: Install build dependencies
+        # rocksdb-sys runs bindgen, which needs libclang; the native deps
+        # (rocksdb, zstd, lz4) need cmake and a compiler, and reqwest's TLS
+        # backend needs libssl. Every other Rust job in this repository
+        # installs the same set — without it the build dies in under a
+        # minute, long before anything interesting compiles.
+        run: |
+          set -euo pipefail
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            clang \
+            cmake \
+            libclang-dev \
+            libssl-dev \
+            pkg-config
+      - uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: ${{ env.RUST_TOOLCHAIN }}
+      - uses: Swatinem/rust-cache@v2
+      - name: Build the binaries the image installs
+        run: cargo build --release --locked --bin satd --bin sat-cli --bin sat-tui
+      - name: Build the image
+        run: |
+          contrib/appliance/build-in-docker.sh \
+            --flavor ${{ matrix.flavor }} \
+            --out "$GITHUB_WORKSPACE/appliance-out"
+          ls -lh "$GITHUB_WORKSPACE/appliance-out"
+      - name: Boot it and check every surface
+        run: |
+          set -euo pipefail
+          # /dev/kvm is present on hosted Linux runners, so this boots with
+          # hardware acceleration; the test falls back to TCG where it is not.
+          ls -l /dev/kvm || echo "no /dev/kvm; the boot test will use TCG"
+          image="$(ls "$GITHUB_WORKSPACE"/appliance-out/*.qcow2 | head -1)"
+          contrib/appliance/tests/boot-test.sh --image "$image" --in-docker
+      - name: Upload
+        if: github.event_name != 'pull_request'
+        uses: actions/upload-artifact@v4
+        with:
+          name: satd-appliance-${{ matrix.flavor }}
+          # The raw image is large and reconstructible from the qcow2; the
+          # checksums cover everything the build produced.
+          path: |
+            appliance-out/*.qcow2
+            appliance-out/*.ova
+            appliance-out/*.SHA256SUMS
+          retention-days: 14
+          compression-level: 0

.gitignore

@@ -36,3 +36,7 @@ result-*
 # Core's framework seeds a cached chain next to the tests; run.sh redirects it
 # out of the tree, but a hand-run test would otherwise drop it here.
 /contrib/core-functional/cache/
+
+# Built appliance images. Multi-GB disk images and ISOs; the build writes
+# here by default and nothing in the tree should ever carry one.
+contrib/appliance/out/

CHANGELOG.md

@@ -58,6 +58,22 @@ item below is (or will be) written up in full in the in-development
 
 ### Added
 
+- A **reference stack** (`contrib/stack/`): docker-compose running satd with
+  RPC, Electrum, Esplora, metrics and optional MCP, each TLS-terminated by a
+  certificate the install issues for itself, plus best-effort overlays for
+  LND (Neutrino), Core Lightning, Ride The Lightning, a Cashu mint and
+  BTCPay Server.
+- A **downloadable appliance image** (`contrib/appliance/`): a bootable VM
+  with satd, its tooling and — in the desktop flavour — Sparrow, Electrum
+  and Liana already pointed at the node. Signet by default;
+  `satd-appliance set-network mainnet` switches. Built with `mmdebstrap`,
+  and gated in CI by booting the artifact under QEMU.
+- `sat-cli` and `sat-tui` can reach a TLS-terminated RPC listener:
+  `-rpctls`, `-rpccacert`, and `-rpcclientcert` / `-rpcclientkey` for mTLS.
+  Previously an operator who enabled `-rpctlsbind` had to keep the plain
+  listener up for the project's own clients.
+- The container image ships `sat-tui` and a `HEALTHCHECK`, so `docker exec
+  -it satd sat-tui` works and `depends_on: service_healthy` means something.
 - `addconnection`, Bitcoin Core's hidden regtest-only RPC for opening an
   outbound connection of a chosen type (`outbound-full-relay`,
   `block-relay-only`, `addr-fetch`, `feeler`). `getpeerinfo` now reports the

CONTRIBUTING.md

@@ -31,6 +31,15 @@ means in practice for contributors.
 5. CI must be green for a PR to be merged. CI runs the same checks listed
    below plus `cargo-deny` on dep-graph-touching PRs.
 
+Two appliance jobs are release gates rather than per-PR ones, because each
+compiles satd from scratch and they cost around forty minutes of runner time
+between them: the reference-stack bring-up and the appliance image build.
+They run on release tags and on demand. If your change touches
+`contrib/stack/` or `contrib/appliance/`, add the **`appliance-ci`** label to
+your PR to run them there — applying the label starts the run, so no extra
+push is needed. The cheap static checks over those directories run on every
+PR regardless.
+
 Stacked PRs are fine. State the merge order in each PR description and
 land them in that order.