perf: benchmarks, and a number for the slot-scopes question - #74
Merged
Merged
Conversation
The characterization suite is the arbiter of meaning and there was nothing playing the same role for cost. Aria was never trying to be fast, and docs/compatibility.md puts performance outside what a version promises, so these are not targets to hit. They are for noticing when a change costs something nobody meant to spend. Go's testing.B rather than a harness of our own: benchstat, -count and -benchmem for free, and it runs under `go test` like everything else. Three phases separately, because a regression should point at a stage rather than at "the number moved". The macro workloads are examples rather than fixtures written for the benchmark, so check-examples.sh already keeps them from rotting into something that measures nothing. The standard library had to be handled before any of this meant anything. Eval loads it on every call, at 1.3ms and 11,005 allocations against 308 for an empty program, so a workload measured through Eval is ninety-seven percent stdlib loading and three percent workload. It gets its own benchmark, being a real cost every `aria run` pays, and everything else builds it with the timer stopped. Checked that StopTimer excludes allocations and not merely time: a trivial program through the same path reports 0 allocs/op, so the numbers are the workload. Read allocs/op, not ns/op. Every collection operation returns a new value, so allocation is the design's dominant cost, and a deterministic program allocates the same number of times on a laptop and on a shared runner. Wall-clock does not, which is why CI runs these at -benchtime=1x to prove they still run and compares nothing. Gating on a number would need a stored baseline, which goes stale and cries wolf. The last Known Gap now has evidence. name-lookup-deep and name-lookup-shallow run the same loop and the same arithmetic, differing only in whether the names are three scopes up or beside the loop, so the gap between them is chain-walking alone. It costs about a fifth more time and ten more allocations out of eleven thousand: pointer chasing, not garbage. That is the ceiling on what slot indices could win, measured in a case built to make the chain as expensive as it gets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fadion
added a commit
that referenced
this pull request
Aug 27, 2026
…75) #74 said the deep-versus-shallow name lookup difference was "the ceiling on what slots could win". It is not, and the overclaim is the kind that gets read later as a settled answer. Both benchmarks pay exactly one successful map lookup. What differs between them is the failed lookups at the scopes in between, so the fifth-more-time figure bounds chain-walking and says nothing about the other half of the change: replacing that final map lookup with an array index. No benchmark in the file can see that half, and none of them was built to. Two things worth writing down while correcting it. The cheaper half stands alone. Ref.Hops already records how far up a binding lives, so the evaluator could walk exactly that many parents and do one map lookup instead of up to Hops + 1. That collects the entire measured difference with `vars` still a map, no scope sizes plumbed anywhere, and define and assign untouched. And either version is a bigger step than swapping a data structure, because the evaluator has no runtime dependency on resolution at all: i.info is assigned in two places and read in none. Both would introduce one. The paths that would still need names are real -- the REPL's :vars walks globals.vars, an aliased import builds its module from scope.vars[name], and four New(file, nil) call sites have no Info to consult at all. The benchmark's own comment said the same loose thing and now says which half it measures, since that is where somebody reads it. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The characterization suite is the arbiter of meaning and nothing played the same role for cost. Aria was never trying to be fast, and
docs/compatibility.mdputs performance outside what a version promises, so these are not targets to hit — they are for noticing when a change costs something nobody meant to spend.What changed
internal/interp/bench_test.go: Go'stesting.Brather than a harness of our own, sobenchstat,-countand-benchmemcome free.check-examples.shalready stops them rotting into something that measures nothing.-benchtime=1x, which measures nothing and proves they still run.The standard library had to be handled first
Evalloads it on every call: 1.3ms and 11,005 allocations, against 308 for an empty program. A workload measured throughEvalis 97% stdlib loading. So it gets its own benchmark, being a real cost everyaria runpays, and everything else is built with the timer stopped.I checked that
StopTimerexcludes allocations and not merely time — a trivial program through the same path reports0 allocs/op, so the numbers are the workload rather than the setup.Read allocs/op, not ns/op
Every collection operation returns a new value, so allocation is the dominant cost, and a deterministic program allocates identically on a laptop and a shared runner. Wall-clock does not, which is why nothing gates on it — that would need a stored baseline that goes stale and cries wolf.
The last Known Gap now has evidence
name-lookup-deepandname-lookup-shallowrun the same loop and the same arithmetic, differing only in whether the names are three scopes up or beside the loop, so the gap is chain-walking alone: about a fifth more time and ten more allocations out of eleven thousand. Pointer chasing, not garbage. That is the ceiling on what slot indices could win, in a case built to make the chain as expensive as possible.