Skip to content

perf: benchmarks, and a number for the slot-scopes question - #74

Merged
fadion merged 1 commit into
masterfrom
perf/benchmarks
Aug 27, 2026
Merged

fadion merged 1 commit into
masterfrom
perf/benchmarks

Conversation

@fadion

@fadion fadion commented Aug 27, 2026

Copy link
Copy Markdown
Owner

The characterization suite is the arbiter of meaning and nothing played the same role for cost. Aria was never trying to be fast, and docs/compatibility.md puts performance outside what a version promises, so these are not targets to hit — they are for noticing when a change costs something nobody meant to spend.

What changed

  • internal/interp/bench_test.go: Go's testing.B rather than a harness of our own, so benchstat, -count and -benchmem come free.
  • Three phases separately — parse, resolve, eval — because a regression should point at a stage rather than at "the number moved".
  • Macro workloads are examples, not fixtures written for the benchmark, so check-examples.sh already stops them rotting into something that measures nothing.
  • Micro workloads for what the examples mix together: name lookup at depth, call overhead, and building an immutable collection.
  • CI runs them at -benchtime=1x, which measures nothing and proves they still run.

The standard library had to be handled first

Eval loads it on every call: 1.3ms and 11,005 allocations, against 308 for an empty program. A workload measured through Eval is 97% stdlib loading. So it gets its own benchmark, being a real cost every aria run pays, and everything else is built with the timer stopped.

I checked that StopTimer excludes allocations and not merely time — a trivial program through the same path reports 0 allocs/op, so the numbers are the workload rather than the setup.

Read allocs/op, not ns/op

Every collection operation returns a new value, so allocation is the dominant cost, and a deterministic program allocates identically on a laptop and a shared runner. Wall-clock does not, which is why nothing gates on it — that would need a stored baseline that goes stale and cries wolf.

The last Known Gap now has evidence

name-lookup-deep and name-lookup-shallow run the same loop and the same arithmetic, differing only in whether the names are three scopes up or beside the loop, so the gap is chain-walking alone: about a fifth more time and ten more allocations out of eleven thousand. Pointer chasing, not garbage. That is the ceiling on what slot indices could win, in a case built to make the chain as expensive as possible.

The characterization suite is the arbiter of meaning and there was nothing
playing the same role for cost. Aria was never trying to be fast, and
docs/compatibility.md puts performance outside what a version promises, so
these are not targets to hit. They are for noticing when a change costs
something nobody meant to spend.

Go's testing.B rather than a harness of our own: benchstat, -count and
-benchmem for free, and it runs under `go test` like everything else.

Three phases separately, because a regression should point at a stage
rather than at "the number moved". The macro workloads are examples rather
than fixtures written for the benchmark, so check-examples.sh already keeps
them from rotting into something that measures nothing.

The standard library had to be handled before any of this meant anything.
Eval loads it on every call, at 1.3ms and 11,005 allocations against 308
for an empty program, so a workload measured through Eval is ninety-seven
percent stdlib loading and three percent workload. It gets its own
benchmark, being a real cost every `aria run` pays, and everything else
builds it with the timer stopped. Checked that StopTimer excludes
allocations and not merely time: a trivial program through the same path
reports 0 allocs/op, so the numbers are the workload.

Read allocs/op, not ns/op. Every collection operation returns a new value,
so allocation is the design's dominant cost, and a deterministic program
allocates the same number of times on a laptop and on a shared runner.
Wall-clock does not, which is why CI runs these at -benchtime=1x to prove
they still run and compares nothing. Gating on a number would need a
stored baseline, which goes stale and cries wolf.

The last Known Gap now has evidence. name-lookup-deep and
name-lookup-shallow run the same loop and the same arithmetic, differing
only in whether the names are three scopes up or beside the loop, so the
gap between them is chain-walking alone. It costs about a fifth more time
and ten more allocations out of eleven thousand: pointer chasing, not
garbage. That is the ceiling on what slot indices could win, measured in a
case built to make the chain as expensive as it gets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fadion
fadion merged commit a2084b2 into master Aug 27, 2026
8 checks passed
@fadion
fadion deleted the perf/benchmarks branch August 27, 2026 23:02
fadion added a commit that referenced this pull request Aug 27, 2026
…75)

#74 said the deep-versus-shallow name lookup difference was "the ceiling on
what slots could win". It is not, and the overclaim is the kind that gets
read later as a settled answer.

Both benchmarks pay exactly one successful map lookup. What differs between
them is the failed lookups at the scopes in between, so the fifth-more-time
figure bounds chain-walking and says nothing about the other half of the
change: replacing that final map lookup with an array index. No benchmark
in the file can see that half, and none of them was built to.

Two things worth writing down while correcting it.

The cheaper half stands alone. Ref.Hops already records how far up a
binding lives, so the evaluator could walk exactly that many parents and do
one map lookup instead of up to Hops + 1. That collects the entire measured
difference with `vars` still a map, no scope sizes plumbed anywhere, and
define and assign untouched.

And either version is a bigger step than swapping a data structure, because
the evaluator has no runtime dependency on resolution at all: i.info is
assigned in two places and read in none. Both would introduce one. The
paths that would still need names are real -- the REPL's :vars walks
globals.vars, an aliased import builds its module from scope.vars[name],
and four New(file, nil) call sites have no Info to consult at all.

The benchmark's own comment said the same loose thing and now says which
half it measures, since that is where somebody reads it.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant