Native cli artifacts: stage binaries from the R2 registry + close the verify false-pass#42
Merged
Merged
Conversation
Add native-binary delivery for cli backends. A cli app may now declare
`assets` (per-OS/arch url + sha256 + exec_path, with install order/deps/args);
the generator emits:
- install.json — the staging spec the adapter reads at startup
- install.sh — a transparent direct-install path (0o755)
- internal/backend/stage.go (StageAssets) — at install the adapter selects
the host os/arch asset, fetches it, sha256-verifies it, stages a single
file or a tar.gz (extracted via host `tar` with a zip-slip name scan),
runs any ordered install args, and is idempotent via a .staged/<sha>
marker. main rewrites the runner's base command to the staged path, so
the host need not have the CLI pre-installed.
config.go gains the Asset struct, HasAssets/AssetName/AssetHosts/PrimaryExecPath,
ResolveAssets (Kahn topo-sort over deps with order+name tiebreak), and
validateAssets (cli-only; known os/arch; https url; 64-hex sha; relative
exec_path under $APP; unique order per platform). The manifest emits the
fs.read $APP/install.json, fs.write $APP, and per-host net.dial grants only
when assets are present. No-asset cli apps are unchanged.
Submission gains an `artifacts` array (SubArtifact, mirroring scaffold.Asset) with validateArtifacts() wired into Validate() — cli-only, known os/arch, https R2 url, 64-hex sha, relative exec_path, unique install order per platform — so a publisher gets server-authoritative errors before any build. ToConfig now maps artifacts → scaffold.Config.Assets. BuildBundle copies the generated install.json (and install.sh) into the shared bundle dir before the per-platform tar loop, so every platform tarball ships the staging spec. This is the root-cause fix for published cli bundles that silently lacked install.json: main's BuildBundle dropped the submission's artifacts entirely.
…ships no install.json
Close the false-pass gap: the catalogue review gate only checks the binary
sha/signature and is blind to install.json, so a submission that declared
`artifacts` but whose build dropped them still passed verify-submission — and
published a bundle with no binary to run.
After BuildBundle, when the submission has artifacts, verify-submission now
asserts per platform that the built tarball contains install.json (with at
least one asset) AND a manifest carrying the fs.write $APP staging grant —
proof the adapter is actually wired for delivery, not just that the spec file
rode along. Missing either fails the build.
Tests:
- cmd/pilot-app: a miren-shaped cli+artifacts submission builds a bundle
that passes checkStaging on every platform; a bundle with install.json
stripped FAILS (regression guard against reopening the gap).
- publish: TestCLIAssetsSubmissionBuildsAndVerifies — full pipeline build
ships install.json + the delivery grants.
- scaffold: install_test.go (install.json/install.sh rendering),
TestGeneratedCLIWithAssetsCompiles (asset-aware main + stage.go type-check),
r2_e2e_test.go (env-gated live R2 delivery e2e).
Add docs/R2-ARTIFACT-REGISTRY.md (the canonical, implemented design the code comments reference) and mark docs/NATIVE-APPS.md as superseded for the delivery model (by-reference URL → Pilot-hosted R2 bytes). Add scripts/e2e-smolvm.sh, which uploads a real artifact and sets the PILOT_E2E_ASSET_* env vars that the env-gated scaffold r2_e2e_test.go consumes.
So the staging PR doesn't reintroduce Go-cased changelog keys in metadata.json regardless of merge order with #41.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Brings native-cli artifact delivery to
main: a cli app can ship per-OS/arch binaries from the Pilot R2 artifact registry, and the generated adapter fetches → sha256-verifies → stages → execs them at install. Also closes the false-pass that let broken bundles publish:verify-submissionnow fails a submission that declaresartifactsbut whose built bundle lacksinstall.json/staging wiring.This is the feature whose absence on
mainmade published cli bundles broken (noinstall.json/StageAssets;main'sBuildBundlesilently dropped the submission'sartifacts), and whose absence letverify-submissionfalse-pass.How it works
config.go,install.go,scaffold.go, templates):assets(per-OS/arch url+sha256+exec_path, with installorder/deps/args) generateinstall.json+install.sh(0o755) andinternal/backend/stage.go(StageAssets). At install the adapter selects the host os/arch asset, fetches it, verifies its sha256, stages a single file or atar.gz(extracted via hosttarwith a zip-slip name scan), runs ordered install args, and is idempotent via a.staged/<sha>marker.mainrewrites the runner's base command to the staged path. Manifest emitsfs.read $APP/install.json,fs.write $APP, and per-hostnet.dialgrants only when assets are present.ResolveAssetsis a Kahn topo-sort overdeps.submission.go,build.go):Submission.Artifacts(+validateArtifacts, wired intoValidate) →ToConfigmaps them toscaffold.Asset.BuildBundlecopiesinstall.json/install.shinto the shared bundle dir before the per-platform tar loop, so every platform tarball ships the spec.cmd/pilot-app/main.go): afterBuildBundle, when the submission has artifacts, each built platform bundle must containinstall.json(≥1 asset) and a manifest with thefs.write $APPstaging grant, elseverify-submissionFAILS.Validation
go build ./... && go vet ./... && go test ./...all green (incl. cross-compiling e2e tests).pilot-app verify-submissionagainst the real miren submission (4 platforms, 4 tar.gz artifacts) now reportsnative-delivery (install.json): install.json + staging grant present (4 asset(s))per platform →VERIFY OK.Ported onto current main (no reverts)
Kept
#34publisher-pin,#37verify-submission,#38app_description/descOr. The verify gate is added to#37'scmdVerifySubmission, not a rewrite.Tests
cmd/pilot-app/verify_staging_test.go: a miren-shaped cli+artifacts submission builds a bundle that passescheckStagingon every platform; a bundle withinstall.jsonstripped fails (regression guard against reopening the gap).publish:TestCLIAssetsSubmissionBuildsAndVerifies— full pipeline build shipsinstall.json+ the delivery grants.scaffold:install_test.go,TestGeneratedCLIWithAssetsCompiles(asset-awaremain+stage.gotype-check),r2_e2e_test.go(env-gated live R2 delivery e2e).Deliberately out of scope
internal/publish/r2.gopresign client + thecmd/publish-serverArtifacts form step). This PR is the consume/stage/verify half — the half that fixes broken bundles + the false-pass. The upload UI is a separate, larger surface.scripts/ab_report.pyleft untouched (keeps#37's--cases; the branch removed it)..github/workflows/submission-validate.ymlleft asmain's — it already runsverify-submissionon rich submissions, so the new gate runs in CI automatically. (The branch's older version reverted that to bundle-pointerverify.)