Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/reference/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -1432,6 +1432,7 @@ The render page at `packages/eval/render/index.html` loads `tokens.css`, generat
- Scripts calculate every report number from `runs/*.jsonl`. Thresholds come from `eval/thresholds.json` as `{ "s1": 0.9, "s2": 0.95, "s3": 0.05 }`. Numeric literals other than 0, 1, and 2 in report code fail the scoring gate.
- `combinations` in Sample size and limitations counts distinct run-line targets, not rows. A later sent row removes the same target's earlier skipped row from aggregation. Never delete historical rows from append-only JSONL.
- A rescore appends `rescored` with `at` and `reason` and replaces the earlier equal `target`, `repeat`, `promptHash`, and `invocation` row in aggregation. Two unmarked sent rows remain separate runs, and source rows remain in the file.
- A rescore reads only the row's own embedded `artifact.output`; there is no path that ingests an external `runs/out/` raw file back into a row, so rescoring is forward-only. The original one-shot raw outputs `runs/out/2026-09-03` and `runs/out/2026-09-05` are confirmed absent and unrecoverable, so the historical one-shot rows from those two dates are permanently non-rescorable. A row written before `artifact.output` embedding carries no stored response and is refused by this ordinary rule with no date or era special case, the same way section 9.5 treats a missing artifact as incomparable.
- When `sent < combinations`, the difference contains `skipped: MAX_INPUT_TOKENS` rows. What does not work includes their combination names and estimated token counts.
- `--dry-run` returns run count, estimated input tokens, and estimated pricing without calls. `estCostUsd` charges each distinct prompt once at the cache-write rate, then charges per-call session prefixes and repeated prompts at the cache-read rate. Derive prompt-token coefficients, overhead, and prefix size from committed cache-write rows using least squares and p50. Return the hash, row count, coefficients, overhead, and prefix size in `costBasis`. Without enough rows, input and total cost are `null` while token totals remain valid.
- Estimate output tokens by multiplying sendable combinations by the mean output tokens of committed, sent rows with the same model and prompt hash. If no hash matches, use same-model rows and disclose their hash and count. Without any basis row, output and total estimates are `null`. Always read the baseline from repository `runs/`, independent of `TOKENLOOM_RUNS_DIR`.
Expand Down
Loading