fix(build): generate the license surface and make build-metrics staleness fail (TIN-161) - #270
Draft
Jesssullivan wants to merge 2 commits into
Draft
Jesssullivan wants to merge 2 commits into
Jesssullivan wants to merge 2 commits into
Conversation
…ness fail (TIN-161) TIN-161, the "ship truthful performance/license surfaces" half. Two static pages in this repo described the build and were allowed to go on describing a build that no longer existed, because nothing checked them. **`THIRD-PARTY-LICENSES.md` was hand-maintained and had drifted badly.** As of `origin/main` it listed `shiki 3.22.0` against a resolution of 4.2.0, `vite 6.4.1` against 8.0.14, `svelte 5.50.1` against 5.55.4, `sharp 0.33.5` against 0.34.5, carried `mermaid` as a RUNTIME dependency it has not been for some time, and published 18 of 42 direct dependencies. It is now generated. `scripts/truth-surfaces.mjs` derives every row from `package-lock.json`: the package and its declared range from `package.json`, the resolved version and the `license` string from the lockfile entry for that exact resolution. The npm lockfile is the right source here because it is current with `package.json` and the pnpm resolutions AND carries a per-package license field, which `pnpm-lock.yaml` does not. Regenerate with `node scripts/truth-surfaces.mjs --write`. A package whose lockfile entry declares no license prints as `UNDECLARED` and is never guessed — five resolved packages are in that state and the document now names them instead of quietly implying MIT. The page also publishes the license histogram for the full 1,150-package resolved tree. **`docs/build-metrics.md` was a February snapshot titled "Baseline".** `main` had already added a prose "Superseded" note (#264); this makes the staleness machine-checked instead of remembered. The page now declares a `Status`, a capture date, and the toolchain it was captured against — `vite ^6.4.1`, `svelte ^5.19.0`, `@sveltejs/kit ^2.16.0`, read from `9aac474:package.json`, the commit that added the file, so the comparison is declared-range against declared-range rather than range against resolution. The check fails in BOTH directions: a toolchain bump without a recapture must carry STALE, and a page that carries STALE once the toolchain matches again must drop it. No permanent warning that nobody reads. `scripts/truth-surfaces.test.mts` covers it, and half the cases exist to prove the guard can go red — a check only ever observed passing is an unfailable assertion. `BUILD.bazel`'s `vitest_srcs` and `scripts/run-vitest-bazel.mjs` gain the two documents so they reach the Vitest sandbox next to `package-lock.json`. Verification (plain node 22; `node_modules` is not installed on this machine so vitest itself was not run here — the test's fixtures and assertions were ported 1:1 to a harness and executed): $ node scripts/truth-surfaces.mjs truth surfaces OK 12/12 harness cases pass, including: a lockfile version bump with a stale document goes red; an undeclared license renders UNDECLARED and NOT MIT; a moved toolchain without STALE fails; an unjustified STALE marker fails; a metrics page that will not name its toolchain fails; and the real tree publishes a row for all 42 direct dependencies with none unresolved. **Out of scope here, and why.** The other TIN-161 bullets — mermaid prerender, `sharp`-unreadable images, Pagefind skipping `/` and `/consultancy.html`, and the large-chunk/post-import warnings — all need a real build to triage honestly, and this machine cannot run one. Two of them look to have moved since the audit and want re-derivation rather than a fix: `puppeteer` is now a direct devDependency (`^24.43.1`), `scripts/render-mermaid.mts` carries an explicit "Cannot find package 'puppeteer'" degradation path, and there is a `puppeteer_chromium_smoke` bazel target. Stated as observations, not as a closure. CI pending.
…ie build-metrics to its capture commit (TIN-161) Review of #270 found the guard did not run and the page it was meant to protect was untouched. Seven findings, all applied. BLOCKER — the guard never executed. `vitest.bazel.config.ts` is an exact-path allow-list and `scripts/truth-surfaces.test.mts` matched none of its five patterns, so `//:vitest_unit_tests` — the only test authority CI has — never loaded it. Grep found zero invocation sites anywhere in the tree. The test is now in the include list, and `.github/workflows/ci.yml` gains a dependency-free `node scripts/truth-surfaces.mjs` step, which is also the only place the new provenance check can run (the Vitest sandbox is a copied tree with no .git). BLOCKER — the published page was still lying. `/THIRD-PARTY-LICENSES` is prerendered and linked from the footer, and it was a hand-written duplicate carrying every defect the PR set out to fix: mermaid as a runtime dependency, shiki 3.22.0, vite 6.4.1, `@sveltejs/enhanced-img` which is not a dependency at all, and "All 243 production packages ... Apache-2.0 (8)" against a real runtime figure of 106. `truth-surfaces.mjs --write` now also emits `src/lib/data/third-party-licenses.json`, the route renders from it, and the byte-comparison covers it — so the page and the Markdown cannot disagree. MAJOR — the freshness check verified the label, not the measurement. Editing the three "Toolchain At Capture" cells and the Status line turned the page green without recapturing a number, which made falsifying provenance the cheapest path to green. `checkBuildMetricsProvenance` now validates that row against `<capture commit>:package.json`; forging it requires moving the capture commit, which is a claim the metrics were retaken. Verified red on the exact edit the review demonstrated. MAJOR — "1150 third-party packages" was an install-path count. npm records a package once per place it lands (`@opentelemetry/core` 21 times), and the filter also swept in the three `link: true` workspace entries — so 3 of the 5 names published as "declare no license" were this repository's own code. Tree figures now deduplicate on name@version and drop workspace links: 1150 installs, 1078 distinct third-party packages, 361 runtime, 2 genuinely undeclared. MINOR — the production-only breakdown was deleted, not replaced. Runtime and full histograms are now emitted separately and labelled, so LGPL-3.0-or-later and EPL-2.0 stop appearing on the attribution surface of a static site that ships neither. MINOR — `optionalDependencies`/`peerDependencies` were silently omitted, and the test computed its expected count from the same two keys, so it could never observe the omission. All four groups are read; the test derives the count from every `*ependencies` key in the manifest. MINOR — `resolved` was collected and never rendered. It is now a Source column, which is the only field that reveals a package that did not come from the registry (`@tummycrypt/tinyvectors` resolves to a GitHub tarball). NIT — "rendering twice produces identical bytes" asserted f(x) === f(x) over a pure module and stayed green with every `.sort()` deleted. It now renders from reversed manifest and lockfile key order and compares bytes. NIT — a missing surface threw ENOENT instead of reporting a violation. Guarded. 28 tests, all failable: every fix above was mutated to confirm it goes red, including unwiring the provenance check from `checkTruthSurfaces`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TIN-161, the "ship truthful performance/license surfaces" half. Static surfaces
in this repo described the build and were allowed to go on describing a build
that no longer existed, because nothing checked them.
THIRD-PARTY-LICENSES.mdwas hand-maintained and had drifted badly. As oforigin/mainit listedshiki 3.22.0against a resolution of 4.2.0,vite 6.4.1against 8.0.14,svelte 5.50.1against 5.55.4,sharp 0.33.5against 0.34.5, carried
mermaidas a RUNTIME dependency it has not been forsome time, and published 18 of 42 direct dependencies.
It is now generated.
scripts/truth-surfaces.mjsderives every row frompackage-lock.json: the package and its declared range frompackage.json,the resolved version, the
licensestring and the source URL from the lockfileentry for that exact resolution. The npm lockfile is the right source here
because it is current with
package.jsonand the pnpm resolutions AND carriesa per-package license field, which
pnpm-lock.yamldoes not. Regenerate withnode scripts/truth-surfaces.mjs --write.src/routes/THIRD-PARTY-LICENSES/+page.svelteis the copy the public reads,and it is generated too. It is prerendered and linked from the footer, so
regenerating only the Markdown would have left the published page publishing
mermaidand an "All 243 production packages … Apache-2.0 (8)" summary againsta real runtime figure of 106.
--writenow emitssrc/lib/data/third-party-licenses.jsonalongside the Markdown, the routerenders from it, and the byte-comparison covers both — the two surfaces are the
same data and cannot disagree.
Counts are packages, not lockfile rows.
lock.packagesis keyed by installpath, so a dependency npm cannot hoist is recorded once per place it lands
(
@opentelemetry/coreappears 21 times). The tree figures deduplicate onname@versionand drop the threelink: trueworkspace entries, which are thisrepository's own code: 1150 installs, 1078 distinct third-party
packages, 361 of them reachable outside
devDependencies, and 2genuinely undeclared (
@better-fetch/fetch,khroma). Runtime anddev-inclusive license histograms are published separately, because attribution
for a prerendered static site is governed by the runtime set — blending in 700-odd
dev-only packages put
LGPL-3.0-or-laterandEPL-2.0on an attribution pagefor a site that ships neither.
docs/build-metrics.mdwas a February snapshot titled "Baseline".mainhad already added a prose "Superseded" note (#264); this makes the staleness
machine-checked instead of remembered. The page declares a
Status, a capturedate, and the toolchain it was captured against, and the check fails in both
directions: a toolchain bump without a recapture must carry STALE, and a page
that carries STALE once the toolchain matches again must drop it.
The toolchain row is provenance, not a field. Freshness alone checks the
label rather than the measurement: after a Vite bump the cheapest path to green
was to retype three cells and delete the STALE marker, without recapturing a
single number — the guard would have rewarded falsifying the one part of the
page that records historical fact.
checkBuildMetricsProvenancevalidates thatrow against
<capture commit>:package.json(9aac474, the commit that addedthe file, which declares
vite ^6.4.1,svelte ^5.19.0,@sveltejs/kit ^2.16.0). Editing the row now requires moving the capture commit, and movingthe capture commit is a claim that the measurements were retaken.
The guard actually runs.
vitest.bazel.config.tsis an exact-pathallow-list;
scripts/truth-surfaces.test.mtsmatched none of its patterns, so//:vitest_unit_tests— the only test authority CI has — never loaded it, andan earlier revision of this PR was green for that reason alone. The test is now
in the include list.
.github/workflows/ci.ymlalso gains a dependency-freenode scripts/truth-surfaces.mjsstep, which is where the provenance checklives: the Vitest sandbox is a copied tree with no
.git, so history is onlyreachable from a real checkout.
scripts/truth-surfaces.test.mtshas 28 cases and most exist to prove the guardcan go red — a check only ever observed passing is an unfailable assertion.
Every behaviour above was mutated to confirm it fails: dropping the optional and
peer groups, counting install paths instead of packages, keeping workspace links
in the third-party figures, collapsing the runtime histogram into the full one,
dropping the Source column, removing the missing-surface guards, checking only
the Markdown and not the data the page renders, deleting every
.sort(),accepting any toolchain row, and unwiring the provenance check from
checkTruthSurfaces— ten mutations, ten targeted failures.BUILD.bazel'svitest_srcsandscripts/run-vitest-bazel.mjscarry the twodocuments so they reach the Vitest sandbox next to
package-lock.json; thegenerated JSON travels with the existing
src/**inputs.Verification (plain node 22;
node_modulesis not installed on this machine sovitest itself was not run here — the test file was executed unmodified against a
minimal describe/it/expect shim):
Out of scope here, and why. The other TIN-161 bullets — mermaid prerender,
sharp-unreadable images, Pagefind skipping/and/consultancy.html, andthe large-chunk/post-import warnings — all need a real build to triage
honestly, and this machine cannot run one. Two of them look to have moved since
the audit and want re-derivation rather than a fix:
puppeteeris now a directdevDependency (
^24.43.1),scripts/render-mermaid.mtscarries an explicit"Cannot find package 'puppeteer'" degradation path, and there is a
puppeteer_chromium_smokebazel target. Stated as observations, not as aclosure.