From 6630ee79f074807bf744a388f8eca29d7da44020 Mon Sep 17 00:00:00 2001 From: hsskey Date: Thu, 17 Sep 2026 23:32:58 +0900 Subject: [PATCH] docs(eval): record forward-only rescore posture in spec 9.6 The one-shot raw outputs runs/out/2026-09-03 and runs/out/2026-09-05 are confirmed absent and unrecoverable, so the historical one-shot rows from those dates are permanently non-rescorable. State that a rescore reads only the row's own embedded artifact.output, that there is no external runs/out/ ingestion path, and that a pre-embedding row is refused by the ordinary rule with no date or era special case. --- docs/reference/spec.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/spec.md b/docs/reference/spec.md index 9d713a0..1b93a07 100644 --- a/docs/reference/spec.md +++ b/docs/reference/spec.md @@ -1432,6 +1432,7 @@ The render page at `packages/eval/render/index.html` loads `tokens.css`, generat - Scripts calculate every report number from `runs/*.jsonl`. Thresholds come from `eval/thresholds.json` as `{ "s1": 0.9, "s2": 0.95, "s3": 0.05 }`. Numeric literals other than 0, 1, and 2 in report code fail the scoring gate. - `combinations` in Sample size and limitations counts distinct run-line targets, not rows. A later sent row removes the same target's earlier skipped row from aggregation. Never delete historical rows from append-only JSONL. - A rescore appends `rescored` with `at` and `reason` and replaces the earlier equal `target`, `repeat`, `promptHash`, and `invocation` row in aggregation. Two unmarked sent rows remain separate runs, and source rows remain in the file. +- A rescore reads only the row's own embedded `artifact.output`; there is no path that ingests an external `runs/out/` raw file back into a row, so rescoring is forward-only. The original one-shot raw outputs `runs/out/2026-09-03` and `runs/out/2026-09-05` are confirmed absent and unrecoverable, so the historical one-shot rows from those two dates are permanently non-rescorable. A row written before `artifact.output` embedding carries no stored response and is refused by this ordinary rule with no date or era special case, the same way section 9.5 treats a missing artifact as incomparable. - When `sent < combinations`, the difference contains `skipped: MAX_INPUT_TOKENS` rows. What does not work includes their combination names and estimated token counts. - `--dry-run` returns run count, estimated input tokens, and estimated pricing without calls. `estCostUsd` charges each distinct prompt once at the cache-write rate, then charges per-call session prefixes and repeated prompts at the cache-read rate. Derive prompt-token coefficients, overhead, and prefix size from committed cache-write rows using least squares and p50. Return the hash, row count, coefficients, overhead, and prefix size in `costBasis`. Without enough rows, input and total cost are `null` while token totals remain valid. - Estimate output tokens by multiplying sendable combinations by the mean output tokens of committed, sent rows with the same model and prompt hash. If no hash matches, use same-model rows and disclose their hash and count. Without any basis row, output and total estimates are `null`. Always read the baseline from repository `runs/`, independent of `TOKENLOOM_RUNS_DIR`.