feat(bench): add JS/TS Tier B scenarios, go/js/both selector - #367
Draft
raphaelvigee wants to merge 3 commits into
Draft
feat(bench): add JS/TS Tier B scenarios, go/js/both selector#367raphaelvigee wants to merge 3 commits into
raphaelvigee wants to merge 3 commits into
Conversation
Extends heph-bench to measure the new plugin-js the same way it already
measures plugin-go, and generalizes Tier B (dist.rs) across languages
instead of duplicating the go-specific path.
- crates/bench-corpus: generate_js_tree produces a synthetic, network-free
pnpm workspace (zero third-party deps by design — no js_install fetch,
no lockfile, fully hermetic) with the same layered-DAG shape the bash
and go generators already use. Wired via js_packages/js_max_depth,
mirroring go_packages/go_max_depth exactly; 0 (the default) is a no-op,
so existing callers get byte-identical output.
- crates/bench: dist.rs rewritten around a Lang abstraction (GO/JS
consts: name, provider-option fragment, package-count accessor,
incrementalize fn) instead of two near-duplicate go/js code paths.
`run dist --lang go|js|both` (default go, preserving existing-caller
behavior exactly). `both` reports each language as its own distinct
ScenarioResult ("<scenario>-go"/"<scenario>-js") — RunResults already
carried a Vec of scenarios for exactly this, so results are never
silently summed/averaged across languages.
- The `corpus` CLI subcommand was missing --js-packages/--js-max-changed
entirely (bench-corpus's library supported them, main.rs never exposed
them) — added, mirroring --go-packages/--go-max-depth.
Found and fixed a real, pre-existing bug while mirroring the go path for
js: Tier B's `heph r build //<lang>/...` invocation is the two-positional
form, which parses as `label("build") && //<lang>/...` — no BUILD file in
the corpus sets that label, so it has always matched zero targets and
exited 0, silently measuring an empty build. This affected the existing
go benchmark too, not just the new js one. Fixed to the query form
(`heph r -e '//<lang>/...'`), verified against a real binary (built a
fixture, confirmed the old form reports "matched 0 targets" and the new
one actually executes targets), with a regression test that fails if the
matcher regresses back to the broken form.
Artifact pipeline (a real decision, not a side effect): plugin-js-cdylib
is now built and published by heph.yml's `build` job alongside
plugin-go-cdylib/plugin-gha-cdylib, in the same pre-release artifact
bundle. No public heph-js-plugin.json manifest is generated (unlike
go/gha) — bench builds its own local manifest from the raw dylib, same as
it always has for go; declaring the plugin generally installable is a
separate, larger decision left unmade. devenv.nix's `e2e` script
deliberately does NOT build plugin-js-cdylib — there is no
shipped_js_cdylib_loads bin-e2e test yet to justify the build cost,
mirroring the existing lint-but-don't-e2e-test asymmetry.
perf.yml wires --lang js into the perfbench job's Tier B step, gated
best-effort on the baseline release actually having a js-plugin asset
(older releases won't). Known, disclosed gap: no oxlint/esbuild toolchain
is provisioned on any CI runner yet, so the JS Tier B comparison reports
"no results" until one is — the target-matcher fix above makes that
failure mode honest (real attempt, real failure) rather than the
previous silent zero-targets false success.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3wZfyPsG8stfRQuybLRjN
Makes the js plugin generally installable the same way go/gha already
are: plugins: - { identifier: { url: https://.../heph-js-plugin.json } }
now resolves for any real workspace, not just this bench pipeline's own
locally-built manifest (crates/bench/src/dist.rs's write_dist_config,
unaffected — it still builds its own manifest straight from the raw
dylib and never reads this one).
Generated the same way as go/gha: tools/pluginmanifest against the
per-os/arch heph-js-plugin_<os>_<arch>.<ext> release assets the `build`
job already publishes (wired in the prior commit on this PR), with the
same URL-base/checksum-sidecar shape. perf.yml is unaffected — it
downloads only the raw dylib, not this manifest.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3wZfyPsG8stfRQuybLRjN
Removed a 9-line comment on PLUGIN_JS_NAME and a redundant one above the "Upload js plugin artifact" step -- the sibling PLUGIN_GO_NAME/GHA_NAME vars and their upload steps carry none, and the decision they restated is already recorded in the manifest-generation step's comment and the commit messages. Also dropped two comments in a test helper that just restated the .pop() calls immediately above them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M3wZfyPsG8stfRQuybLRjN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 7/7 of the stack — extends
heph-benchto measure the new plugin-js the same way it already measures plugin-go, generalizes Tier B (dist.rs) across languages instead of duplicating the go-specific path, and publishes the js plugin as a real, generally-installable artifact — same as go/gha.What's here
crates/bench-corpus:generate_js_treeproduces a synthetic, network-free pnpm workspace (zero third-party deps by design — no js_install fetch, no lockfile, fully hermetic) with the same layered-DAG shape the bash and go generators already use. Wired viajs_packages/js_max_depth, mirroringgo_packages/go_max_depthexactly —0(the default) is a no-op, so existing callers get byte-identical output.crates/bench:dist.rsrewritten around aLangabstraction (GO/JS consts: name, provider-option fragment, package-count accessor, incrementalize fn) instead of two near-duplicate go/js code paths.run dist --lang go|js|both(defaultgo, preserving existing-caller behavior exactly — byte-identical.hephconfig/scenario names to the pre-existing code path).bothreports each language as its own distinctScenarioResult(<scenario>-go/<scenario>-js) — never silently summed/averaged.corpusCLI subcommand was missing--js-packages/--js-max-depthentirely (bench-corpus's library supported them, main.rs never exposed them) — added, mirroring--go-packages/--go-max-depth.A real pre-existing bug, found while mirroring the go path for js
Tier B's
heph r build //<lang>/...invocation is the two-positional-arg form, which parses aslabel("build") && //<lang>/...— no BUILD file in the corpus sets that label, so it has always matched zero targets and exited 0, silently measuring an empty build. This affected the existing go benchmark too, not just the new js one — it predates this PR entirely, just surfaced while faithfully mirroring the path. Fixed to the query form (heph r -e '//<lang>/...'), verified against a real binary (built a fixture, confirmed the old form reports "matched 0 targets" and the new one actually executes targets), with a regression test that fails if the matcher regresses back to the broken form.Artifact pipeline — js is now published like go/gha
plugin-js-cdylibis built and published byheph.yml'sbuildjob alongsideplugin-go-cdylib/plugin-gha-cdylib, in the same pre-release artifact bundle, andupload_artifactsnow generates a publicheph-js-plugin.jsonmanifest viatools/pluginmanifest, exactly the same way go/gha do (per-os/arch checksum entries, URL-base pointing at the release, a.sha256sidecar for the manifest itself). A real workspace can now opt in the same way as go/gha:crates/bench/src/dist.rs'swrite_dist_configis unaffected — it still builds its own local manifest straight from the downloaded raw dylib and never reads the published one; the two are independent consumers of the same underlying cdylib asset.devenv.nix'se2escript still deliberately does not buildplugin-js-cdyliblocally — there is noshipped_js_cdylib_loadsbin-e2e test yet to justify the build cost, mirroring the existing lint-but-don't-e2e-test asymmetry already in place for this crate. That's a narrower, still-open gap than manifest publishing and unaffected by this update.Known, disclosed gap
No
oxlint/esbuildtoolchain is provisioned on any CI runner yet, soperf.yml's new JS Tier B comparison step will report "no results" until one is. The target-matcher fix above makes that failure mode honest (a real attempt, a real failure) rather than the previous silent zero-targets false success. Provisioning a JS toolchain for CI (host vs. hermetic, which versions) is a separate follow-up decision, not made here.Test plan
cargo build -p bench -p bench-corpuscargo test -p bench -p bench-corpus(14 + 8 passed, 2 pre-existing network-gated ignored)cargo clippy -p bench -p bench-corpus --all-targets -- -D warnings(clean)cargo fmt --check -p bench -p bench-corpuscargo build --release -p plugin-js-cdylib(the artifact this whole change assumes exists)heph.yml,perf.yml) parsestst,lint) on this PRStack created with GitHub Stacks CLI • Give Feedback 💬