Skip to content

feat(bench): add JS/TS Tier B scenarios, go/js/both selector - #367

Draft
raphaelvigee wants to merge 3 commits into
raphaelvigee/feat-plugin-js-tsfrom
raphaelvigee/feat-plugin-js-m7-bench
Draft

feat(bench): add JS/TS Tier B scenarios, go/js/both selector#367
raphaelvigee wants to merge 3 commits into
raphaelvigee/feat-plugin-js-tsfrom
raphaelvigee/feat-plugin-js-m7-bench

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 5, 2026

Copy link
Copy Markdown
Member

Part 7/7 of the stack — extends heph-bench to measure the new plugin-js the same way it already measures plugin-go, generalizes Tier B (dist.rs) across languages instead of duplicating the go-specific path, and publishes the js plugin as a real, generally-installable artifact — same as go/gha.

What's here

  • crates/bench-corpus: generate_js_tree produces a synthetic, network-free pnpm workspace (zero third-party deps by design — no js_install fetch, no lockfile, fully hermetic) with the same layered-DAG shape the bash and go generators already use. Wired via js_packages/js_max_depth, mirroring go_packages/go_max_depth exactly — 0 (the default) is a no-op, so existing callers get byte-identical output.
  • crates/bench: dist.rs rewritten around a Lang abstraction (GO/JS consts: name, provider-option fragment, package-count accessor, incrementalize fn) instead of two near-duplicate go/js code paths. run dist --lang go|js|both (default go, preserving existing-caller behavior exactly — byte-identical .hephconfig/scenario names to the pre-existing code path). both reports each language as its own distinct ScenarioResult (<scenario>-go/<scenario>-js) — never silently summed/averaged.
  • The corpus CLI subcommand was missing --js-packages/--js-max-depth entirely (bench-corpus's library supported them, main.rs never exposed them) — added, mirroring --go-packages/--go-max-depth.

A real pre-existing bug, found while mirroring the go path for js

Tier B's heph r build //<lang>/... invocation is the two-positional-arg form, which parses as label("build") && //<lang>/... — no BUILD file in the corpus sets that label, so it has always matched zero targets and exited 0, silently measuring an empty build. This affected the existing go benchmark too, not just the new js one — it predates this PR entirely, just surfaced while faithfully mirroring the path. Fixed to the query form (heph r -e '//<lang>/...'), verified against a real binary (built a fixture, confirmed the old form reports "matched 0 targets" and the new one actually executes targets), with a regression test that fails if the matcher regresses back to the broken form.

Artifact pipeline — js is now published like go/gha

plugin-js-cdylib is built and published by heph.yml's build job alongside plugin-go-cdylib/plugin-gha-cdylib, in the same pre-release artifact bundle, and upload_artifacts now generates a public heph-js-plugin.json manifest via tools/pluginmanifest, exactly the same way go/gha do (per-os/arch checksum entries, URL-base pointing at the release, a .sha256 sidecar for the manifest itself). A real workspace can now opt in the same way as go/gha:

plugins:
  - path: .heph3/heph-js-plugin.json   # or a url: pointing at the published release asset

crates/bench/src/dist.rs's write_dist_config is unaffected — it still builds its own local manifest straight from the downloaded raw dylib and never reads the published one; the two are independent consumers of the same underlying cdylib asset.

devenv.nix's e2e script still deliberately does not build plugin-js-cdylib locally — there is no shipped_js_cdylib_loads bin-e2e test yet to justify the build cost, mirroring the existing lint-but-don't-e2e-test asymmetry already in place for this crate. That's a narrower, still-open gap than manifest publishing and unaffected by this update.

Known, disclosed gap

No oxlint/esbuild toolchain is provisioned on any CI runner yet, so perf.yml's new JS Tier B comparison step will report "no results" until one is. The target-matcher fix above makes that failure mode honest (a real attempt, a real failure) rather than the previous silent zero-targets false success. Provisioning a JS toolchain for CI (host vs. hermetic, which versions) is a separate follow-up decision, not made here.

Test plan

  • cargo build -p bench -p bench-corpus
  • cargo test -p bench -p bench-corpus (14 + 8 passed, 2 pre-existing network-gated ignored)
  • cargo clippy -p bench -p bench-corpus --all-targets -- -D warnings (clean)
  • cargo fmt --check -p bench -p bench-corpus
  • cargo build --release -p plugin-js-cdylib (the artifact this whole change assumes exists)
  • Workflow YAML (heph.yml, perf.yml) parses
  • CI (tst, lint) on this PR

Stack created with GitHub Stacks CLIGive Feedback 💬

raphaelvigee and others added 3 commits August 5, 2026 10:42
Extends heph-bench to measure the new plugin-js the same way it already
measures plugin-go, and generalizes Tier B (dist.rs) across languages
instead of duplicating the go-specific path.

- crates/bench-corpus: generate_js_tree produces a synthetic, network-free
  pnpm workspace (zero third-party deps by design — no js_install fetch,
  no lockfile, fully hermetic) with the same layered-DAG shape the bash
  and go generators already use. Wired via js_packages/js_max_depth,
  mirroring go_packages/go_max_depth exactly; 0 (the default) is a no-op,
  so existing callers get byte-identical output.
- crates/bench: dist.rs rewritten around a Lang abstraction (GO/JS
  consts: name, provider-option fragment, package-count accessor,
  incrementalize fn) instead of two near-duplicate go/js code paths.
  `run dist --lang go|js|both` (default go, preserving existing-caller
  behavior exactly). `both` reports each language as its own distinct
  ScenarioResult ("<scenario>-go"/"<scenario>-js") — RunResults already
  carried a Vec of scenarios for exactly this, so results are never
  silently summed/averaged across languages.
- The `corpus` CLI subcommand was missing --js-packages/--js-max-changed
  entirely (bench-corpus's library supported them, main.rs never exposed
  them) — added, mirroring --go-packages/--go-max-depth.

Found and fixed a real, pre-existing bug while mirroring the go path for
js: Tier B's `heph r build //<lang>/...` invocation is the two-positional
form, which parses as `label("build") && //<lang>/...` — no BUILD file in
the corpus sets that label, so it has always matched zero targets and
exited 0, silently measuring an empty build. This affected the existing
go benchmark too, not just the new js one. Fixed to the query form
(`heph r -e '//<lang>/...'`), verified against a real binary (built a
fixture, confirmed the old form reports "matched 0 targets" and the new
one actually executes targets), with a regression test that fails if the
matcher regresses back to the broken form.

Artifact pipeline (a real decision, not a side effect): plugin-js-cdylib
is now built and published by heph.yml's `build` job alongside
plugin-go-cdylib/plugin-gha-cdylib, in the same pre-release artifact
bundle. No public heph-js-plugin.json manifest is generated (unlike
go/gha) — bench builds its own local manifest from the raw dylib, same as
it always has for go; declaring the plugin generally installable is a
separate, larger decision left unmade. devenv.nix's `e2e` script
deliberately does NOT build plugin-js-cdylib — there is no
shipped_js_cdylib_loads bin-e2e test yet to justify the build cost,
mirroring the existing lint-but-don't-e2e-test asymmetry.

perf.yml wires --lang js into the perfbench job's Tier B step, gated
best-effort on the baseline release actually having a js-plugin asset
(older releases won't). Known, disclosed gap: no oxlint/esbuild toolchain
is provisioned on any CI runner yet, so the JS Tier B comparison reports
"no results" until one is — the target-matcher fix above makes that
failure mode honest (real attempt, real failure) rather than the
previous silent zero-targets false success.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3wZfyPsG8stfRQuybLRjN
Makes the js plugin generally installable the same way go/gha already
are: plugins: - { identifier: { url: https://.../heph-js-plugin.json } }
now resolves for any real workspace, not just this bench pipeline's own
locally-built manifest (crates/bench/src/dist.rs's write_dist_config,
unaffected — it still builds its own manifest straight from the raw
dylib and never reads this one).

Generated the same way as go/gha: tools/pluginmanifest against the
per-os/arch heph-js-plugin_<os>_<arch>.<ext> release assets the `build`
job already publishes (wired in the prior commit on this PR), with the
same URL-base/checksum-sidecar shape. perf.yml is unaffected — it
downloads only the raw dylib, not this manifest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3wZfyPsG8stfRQuybLRjN
Removed a 9-line comment on PLUGIN_JS_NAME and a redundant one above the
"Upload js plugin artifact" step -- the sibling PLUGIN_GO_NAME/GHA_NAME
vars and their upload steps carry none, and the decision they restated
is already recorded in the manifest-generation step's comment and the
commit messages. Also dropped two comments in a test helper that just
restated the .pop() calls immediately above them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3wZfyPsG8stfRQuybLRjN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant