The economics of code reuse: benchmark evidence pack

This repo is the full evidence pack for the benchmark described in The economics of code reuse in AI development: raw telemetry, prompts, run outputs, verification documents, and the harness that produced them — including the runs Stellify lost.

What was measured

One claim, stated falsifiably: once functionality exists in Stellify's corpus, reusing it into a new build should cost a roughly flat number of generated tokens regardless of module size, while building the same module from scratch should scale with the code.

Both arms performed the identical task to completion — deliver a working module of a given size into a project. One arm was Claude Code working natively on files (Read/Edit, prompt caching enabled). The other arm referenced the module already present in Stellify via MCP. Same pinned model (claude-opus-4-8) in every run; token counts from Claude Code's own OpenTelemetry output; billed dollars recorded alongside raw tokens.

The result

Module	Reuse: output tokens	From scratch: output tokens	Advantage	Reuse: cost	From scratch: cost
Report, ~130 lines	1,536	3,067	2.0×	$0.38	$0.22 — reuse loses
Payroll, ~641 lines	1,196–1,316	13,209	~10×	$0.29–0.38	$0.59 — up to 2× cheaper
Ledger, ~1,174 lines	1,249–1,545	22,292	~14×	$0.50–0.53	$0.92 — ~1.8× cheaper

The reuse cost is flat (~1,200–1,550 output tokens in all nine reuse runs, under both mechanisms); generation scales at roughly 19–24 output tokens per line. The blended-cost crossover sits around 400–500 lines (interpolated — below it a plain file agent is cheaper).

The runs reuse didn't win are here too: a single small greenfield build (file agent $0.66 vs Stellify MCP $0.93) and ten sequential edits to an existing file ($0.99 vs $3.87). If you reproduce the cheapest, smallest case, you will get the opposite of the headline result. That is expected, and documented before you start.

Where to look

• evidence/METHODOLOGY.md — full methodology: arms, model pinning, telemetry basis, per-run session IDs, verification procedure.
• evidence/ — forensic verification per reuse run: per-message output-token breakdowns proving the reuse arm emitted no module code; byte-for-byte assembled-output comparisons (*-assembled.diff, *-assembled.php) proving the consumer's module is identical to the corpus original and the corpus was unchanged after every consumer build; transcript excerpts (*-transcript-excerpt.jsonl).
• evidence/corpus-integrity-incident.md — during verification, the harness surfaced a corpus-integrity issue caused by cross-project row sharing; it was repaired and re-verified. Published because a benchmark that hides its accidents isn't one you should trust.
• RESULTS-module-scaling.md — the headline three-point curve.
• RESULTS-reuse-vs-cold-vs-files.md, RESULTS-reuse-vs-cold.md, RESULTS-4arm.md — earlier runs, including the greenfield and iterative-edit tests Stellify lost.
• HARNESS.md — the measurement harness (Claude Code OpenTelemetry → local OTLP collector), as used internally.
• out/ — raw OTel metrics/logs, run manifests, prompts, and run logs. python3 analyze.py out/metrics.jsonl reproduces the per-session token/cost tables.
• scenarios/ — the specs and prompts each arm was given.

Reproduction scope (honest version)

• The file-agent arm is reproducible by anyone: the specs and prompts are in scenarios/ and out/; you need Claude Code and the OTel harness (env.sh + run-collector.sh).
• The Stellify arm requires a Stellify account with MCP access (the corpus the reuse runs reference lives on the platform), plus a project whose .mcp.json loads the stellify MCP server (run-arm.sh reads BENCH_PROJECT_DIR). reset-project.sh is the internal between-runs reset and assumes platform database access — it is published for transparency, not something an external reproducer can run.
• Costs are computed from the pricing constants in analyze.py — verify them against current Anthropic pricing before quoting dollars.

Open caveats

Carried verbatim from the write-up, so they travel with the data:

• Reuse arms are n=2 per module under each of two mechanisms (mutually confirming); the file-agent arms are single runs — separations of 10–14× against measured rep noise of about ±5%.
• The ~400–500-line cost crossover is interpolated, not measured.
• One module family per size, one model; whether the pattern holds for e.g. frontend components is untested.
• At small module sizes reuse loses on cost — the fixed MCP overhead outweighs the generation it avoids.
• Repeated edits and small greenfield builds through the current MCP tooling are more expensive than Claude Code's native tools (see the appendix runs); statement-level edit tooling is on the roadmap.
• References are not yet version-pinned: locking, pinned versions and controlled upgrade propagation are the next stage of the architecture, not something claimed today.

Redactions

The raw OpenTelemetry files in out/ are published as captured, with two mechanical changes: NUL padding left by the collector's file writer was stripped, and five operator-identity attribute values (user.email, user.id, user.account_uuid, user.account_id, organization.id) were replaced with [REDACTED-*] placeholders. Session IDs, timestamps, models, token counts and tool names are untouched — the numbers are exactly as recorded.