Skip to content

fix(build): generate the license surface and make build-metrics staleness fail (TIN-161) - #270

Draft
Jesssullivan wants to merge 2 commits into
mainfrom
tin-161-truth-surfaces
Draft

Jesssullivan wants to merge 2 commits into
mainfrom
tin-161-truth-surfaces

Conversation

@Jesssullivan

@Jesssullivan Jesssullivan commented Aug 29, 2026

Copy link
Copy Markdown
Owner

TIN-161, the "ship truthful performance/license surfaces" half. Static surfaces
in this repo described the build and were allowed to go on describing a build
that no longer existed, because nothing checked them.

THIRD-PARTY-LICENSES.md was hand-maintained and had drifted badly. As of
origin/main it listed shiki 3.22.0 against a resolution of 4.2.0,
vite 6.4.1 against 8.0.14, svelte 5.50.1 against 5.55.4, sharp 0.33.5
against 0.34.5, carried mermaid as a RUNTIME dependency it has not been for
some time, and published 18 of 42 direct dependencies.

It is now generated. scripts/truth-surfaces.mjs derives every row from
package-lock.json: the package and its declared range from package.json,
the resolved version, the license string and the source URL from the lockfile
entry for that exact resolution. The npm lockfile is the right source here
because it is current with package.json and the pnpm resolutions AND carries
a per-package license field, which pnpm-lock.yaml does not. Regenerate with
node scripts/truth-surfaces.mjs --write.

src/routes/THIRD-PARTY-LICENSES/+page.svelte is the copy the public reads,
and it is generated too.
It is prerendered and linked from the footer, so
regenerating only the Markdown would have left the published page publishing
mermaid and an "All 243 production packages … Apache-2.0 (8)" summary against
a real runtime figure of 106. --write now emits
src/lib/data/third-party-licenses.json alongside the Markdown, the route
renders from it, and the byte-comparison covers both — the two surfaces are the
same data and cannot disagree.

Counts are packages, not lockfile rows. lock.packages is keyed by install
path, so a dependency npm cannot hoist is recorded once per place it lands
(@opentelemetry/core appears 21 times). The tree figures deduplicate on
name@version and drop the three link: true workspace entries, which are this
repository's own code: 1150 installs, 1078 distinct third-party
packages, 361 of them reachable outside devDependencies, and 2
genuinely undeclared (@better-fetch/fetch, khroma). Runtime and
dev-inclusive license histograms are published separately, because attribution
for a prerendered static site is governed by the runtime set — blending in 700-odd
dev-only packages put LGPL-3.0-or-later and EPL-2.0 on an attribution page
for a site that ships neither.

docs/build-metrics.md was a February snapshot titled "Baseline". main
had already added a prose "Superseded" note (#264); this makes the staleness
machine-checked instead of remembered. The page declares a Status, a capture
date, and the toolchain it was captured against, and the check fails in both
directions: a toolchain bump without a recapture must carry STALE, and a page
that carries STALE once the toolchain matches again must drop it.

The toolchain row is provenance, not a field. Freshness alone checks the
label rather than the measurement: after a Vite bump the cheapest path to green
was to retype three cells and delete the STALE marker, without recapturing a
single number — the guard would have rewarded falsifying the one part of the
page that records historical fact. checkBuildMetricsProvenance validates that
row against <capture commit>:package.json (9aac474, the commit that added
the file, which declares vite ^6.4.1, svelte ^5.19.0, @sveltejs/kit ^2.16.0). Editing the row now requires moving the capture commit, and moving
the capture commit is a claim that the measurements were retaken.

The guard actually runs. vitest.bazel.config.ts is an exact-path
allow-list; scripts/truth-surfaces.test.mts matched none of its patterns, so
//:vitest_unit_tests — the only test authority CI has — never loaded it, and
an earlier revision of this PR was green for that reason alone. The test is now
in the include list. .github/workflows/ci.yml also gains a dependency-free
node scripts/truth-surfaces.mjs step, which is where the provenance check
lives: the Vitest sandbox is a copied tree with no .git, so history is only
reachable from a real checkout.

scripts/truth-surfaces.test.mts has 28 cases and most exist to prove the guard
can go red — a check only ever observed passing is an unfailable assertion.
Every behaviour above was mutated to confirm it fails: dropping the optional and
peer groups, counting install paths instead of packages, keeping workspace links
in the third-party figures, collapsing the runtime histogram into the full one,
dropping the Source column, removing the missing-surface guards, checking only
the Markdown and not the data the page renders, deleting every .sort(),
accepting any toolchain row, and unwiring the provenance check from
checkTruthSurfaces — ten mutations, ten targeted failures.

BUILD.bazel's vitest_srcs and scripts/run-vitest-bazel.mjs carry the two
documents so they reach the Vitest sandbox next to package-lock.json; the
generated JSON travels with the existing src/** inputs.

Verification (plain node 22; node_modules is not installed on this machine so
vitest itself was not run here — the test file was executed unmodified against a
minimal describe/it/expect shim):

$ node scripts/truth-surfaces.mjs
truth surfaces OK

28/28 cases pass. Also verified against the real tree: forging the
toolchain row to `^8.0.14` / `^5.55.4` / `^2.61.1` and flipping Status to
CURRENT — the exact escape hatch this closes — now exits 1 with three
provenance violations, and adding an `optionalDependencies` block to
`package.json` now goes red instead of silently omitting the package.

$ bash scripts/check-bazel-graph-hygiene.sh   # passed
$ node scripts/test-workflow-authority.mjs    # passed

Out of scope here, and why. The other TIN-161 bullets — mermaid prerender,
sharp-unreadable images, Pagefind skipping / and /consultancy.html, and
the large-chunk/post-import warnings — all need a real build to triage
honestly, and this machine cannot run one. Two of them look to have moved since
the audit and want re-derivation rather than a fix: puppeteer is now a direct
devDependency (^24.43.1), scripts/render-mermaid.mts carries an explicit
"Cannot find package 'puppeteer'" degradation path, and there is a
puppeteer_chromium_smoke bazel target. Stated as observations, not as a
closure.

…ness fail (TIN-161)

TIN-161, the "ship truthful performance/license surfaces" half. Two static
pages in this repo described the build and were allowed to go on describing a
build that no longer existed, because nothing checked them.

**`THIRD-PARTY-LICENSES.md` was hand-maintained and had drifted badly.** As of
`origin/main` it listed `shiki 3.22.0` against a resolution of 4.2.0,
`vite 6.4.1` against 8.0.14, `svelte 5.50.1` against 5.55.4, `sharp 0.33.5`
against 0.34.5, carried `mermaid` as a RUNTIME dependency it has not been for
some time, and published 18 of 42 direct dependencies.

It is now generated. `scripts/truth-surfaces.mjs` derives every row from
`package-lock.json`: the package and its declared range from `package.json`,
the resolved version and the `license` string from the lockfile entry for that
exact resolution. The npm lockfile is the right source here because it is
current with `package.json` and the pnpm resolutions AND carries a per-package
license field, which `pnpm-lock.yaml` does not. Regenerate with
`node scripts/truth-surfaces.mjs --write`.

A package whose lockfile entry declares no license prints as `UNDECLARED` and
is never guessed — five resolved packages are in that state and the document
now names them instead of quietly implying MIT. The page also publishes the
license histogram for the full 1,150-package resolved tree.

**`docs/build-metrics.md` was a February snapshot titled "Baseline".** `main`
had already added a prose "Superseded" note (#264); this makes the staleness
machine-checked instead of remembered. The page now declares a `Status`, a
capture date, and the toolchain it was captured against — `vite ^6.4.1`,
`svelte ^5.19.0`, `@sveltejs/kit ^2.16.0`, read from `9aac474:package.json`,
the commit that added the file, so the comparison is declared-range against
declared-range rather than range against resolution. The check fails in BOTH
directions: a toolchain bump without a recapture must carry STALE, and a page
that carries STALE once the toolchain matches again must drop it. No permanent
warning that nobody reads.

`scripts/truth-surfaces.test.mts` covers it, and half the cases exist to prove
the guard can go red — a check only ever observed passing is an unfailable
assertion. `BUILD.bazel`'s `vitest_srcs` and `scripts/run-vitest-bazel.mjs`
gain the two documents so they reach the Vitest sandbox next to
`package-lock.json`.

Verification (plain node 22; `node_modules` is not installed on this machine so
vitest itself was not run here — the test's fixtures and assertions were ported
1:1 to a harness and executed):

    $ node scripts/truth-surfaces.mjs
    truth surfaces OK

    12/12 harness cases pass, including: a lockfile version bump with a stale
    document goes red; an undeclared license renders UNDECLARED and NOT MIT;
    a moved toolchain without STALE fails; an unjustified STALE marker fails;
    a metrics page that will not name its toolchain fails; and the real tree
    publishes a row for all 42 direct dependencies with none unresolved.

**Out of scope here, and why.** The other TIN-161 bullets — mermaid prerender,
`sharp`-unreadable images, Pagefind skipping `/` and `/consultancy.html`, and
the large-chunk/post-import warnings — all need a real build to triage
honestly, and this machine cannot run one. Two of them look to have moved since
the audit and want re-derivation rather than a fix: `puppeteer` is now a direct
devDependency (`^24.43.1`), `scripts/render-mermaid.mts` carries an explicit
"Cannot find package 'puppeteer'" degradation path, and there is a
`puppeteer_chromium_smoke` bazel target. Stated as observations, not as a
closure. CI pending.
…ie build-metrics to its capture commit (TIN-161)

Review of #270 found the guard did not run and the page it was meant to
protect was untouched. Seven findings, all applied.

BLOCKER — the guard never executed. `vitest.bazel.config.ts` is an exact-path
allow-list and `scripts/truth-surfaces.test.mts` matched none of its five
patterns, so `//:vitest_unit_tests` — the only test authority CI has — never
loaded it. Grep found zero invocation sites anywhere in the tree. The test is
now in the include list, and `.github/workflows/ci.yml` gains a dependency-free
`node scripts/truth-surfaces.mjs` step, which is also the only place the new
provenance check can run (the Vitest sandbox is a copied tree with no .git).

BLOCKER — the published page was still lying. `/THIRD-PARTY-LICENSES` is
prerendered and linked from the footer, and it was a hand-written duplicate
carrying every defect the PR set out to fix: mermaid as a runtime dependency,
shiki 3.22.0, vite 6.4.1, `@sveltejs/enhanced-img` which is not a dependency at
all, and "All 243 production packages ... Apache-2.0 (8)" against a real runtime
figure of 106. `truth-surfaces.mjs --write` now also emits
`src/lib/data/third-party-licenses.json`, the route renders from it, and the
byte-comparison covers it — so the page and the Markdown cannot disagree.

MAJOR — the freshness check verified the label, not the measurement. Editing the
three "Toolchain At Capture" cells and the Status line turned the page green
without recapturing a number, which made falsifying provenance the cheapest path
to green. `checkBuildMetricsProvenance` now validates that row against
`<capture commit>:package.json`; forging it requires moving the capture commit,
which is a claim the metrics were retaken. Verified red on the exact edit the
review demonstrated.

MAJOR — "1150 third-party packages" was an install-path count. npm records a
package once per place it lands (`@opentelemetry/core` 21 times), and the filter
also swept in the three `link: true` workspace entries — so 3 of the 5 names
published as "declare no license" were this repository's own code. Tree figures
now deduplicate on name@version and drop workspace links: 1150 installs, 1078
distinct third-party packages, 361 runtime, 2 genuinely undeclared.

MINOR — the production-only breakdown was deleted, not replaced. Runtime and
full histograms are now emitted separately and labelled, so LGPL-3.0-or-later
and EPL-2.0 stop appearing on the attribution surface of a static site that
ships neither.

MINOR — `optionalDependencies`/`peerDependencies` were silently omitted, and the
test computed its expected count from the same two keys, so it could never
observe the omission. All four groups are read; the test derives the count from
every `*ependencies` key in the manifest.

MINOR — `resolved` was collected and never rendered. It is now a Source column,
which is the only field that reveals a package that did not come from the
registry (`@tummycrypt/tinyvectors` resolves to a GitHub tarball).

NIT — "rendering twice produces identical bytes" asserted f(x) === f(x) over a
pure module and stayed green with every `.sort()` deleted. It now renders from
reversed manifest and lockfile key order and compares bytes.

NIT — a missing surface threw ENOENT instead of reporting a violation. Guarded.

28 tests, all failable: every fix above was mutated to confirm it goes red,
including unwiring the provenance check from `checkTruthSurfaces`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant