tokenomics

Token-efficiency skills library — the Tokenomics pillar of the Integrate.AI Harness.

Agents are assembled per client from the Harness: bespoke in selection, boilerplate in parts. This repo is the parts bin for token efficiency — self-contained skills that make a deployed agent read PDFs, spreadsheets, documents, and tool results at a fraction of the token cost, each with measured evidence behind its savings claim.

• Why this exists: NORTH_STAR.md
• What's being built and how: SPEC.md
• Build state: BUILDLOG.md

Catalog

Generated from catalog.json (entries sorted by name), which is itself derived from each skill's manifest.yaml + its committed meter evidence. Savings are measured on this repo's fixture corpus, not claimed — see the linked evidence.

Skill	Category	When to equip	Measured savings	Evidence
financial-model-ingest	ingestion	Agent will read spreadsheet financial models (DCFs, budgets, forecasts) where the formulas, not the values, are the content.	64.5% fewer input tokens (879 saved · estimate; grows with model size)	report · json
image-ingest	ingestion	Agent will read images regularly (screenshots, UI captures, scanned docs, diagrams).	87.0% fewer input tokens (2,713 saved · estimate · OCR)	report · json
log-json-discipline	ingestion	Agent will read logs, JSONL/NDJSON, or CI/test output (any file >200 lines or >20KB about to enter context).	99.2% fewer input tokens (105,294 saved · estimate)	report · json
media-transcript-first	media	Agent will ingest video or audio (recordings, webinars, calls, screen captures, lectures, voice memos, or hosted URLs).	98.6% fewer input tokens (24,104 saved · estimate)	report · json
office-to-markdown	ingestion	Agent will read Office or web documents (docx, pptx, xlsx, html, epub).	86.8% fewer input tokens (1,959 saved · estimate)	report · json
pdf-ingest	ingestion	Agent will read PDFs regularly (reports, invoices, contracts).	66.5% fewer input tokens (6,190 saved · estimate · real OCR)	report · json
tabular-discipline	ingestion	Agent works with CSVs, spreadsheets, or data files of any size.	77.8% fewer input tokens (3,390 saved · estimate)	report · json
web-ingest	ingestion	Agent will read web pages or docs sites (articles, API docs, product/forum/listing pages).	80.5% fewer input tokens (5,246 saved · estimate)	report · json

Numbers are estimate-mode (the meter's offline byte/geometry estimator) against deliberately conservative baselines — page/raster images priced with Anthropic's standard-tier patch geometry (⌈w/28⌉×⌈h/28⌉) for PDFs and images, raw OOXML for Office files, full-CSV text for tables, the full raw HTTP response body (tags + inline script/style, no JS) for web pages, and for logs/JSONL the whole file verbatim / the minified JSON payload (never pretty-printed, which would inflate the saving). pdf-ingest is measured with the real --ocr route (recovered text is credited; pure-figure pages honestly keep image tokens); image-ingest with real tesseract OCR, and its via-skill cost counts the ~400px layout thumbnail, not just the text; web-ingest with real trafilatura extraction, and its via-skill credits the raw-HTML passthrough of rowspan/colspan tables (the larger, honest number); log-json-discipline's via-skill counts the full drill-down session (profile + drain3 templates for the log; full-scan schema + one capped aggregate for the JSONL), not a single cheapest call. media-transcript-first is measured on its two offline routes — the cleaned caption transcript and real scene-detect + tesseract-OCR keyframes — against a naive 1fps frame dump priced at the conservative Claude-512px per-frame figure (~209 tok, never the ~1,200-token native-720p worst case); its local-ASR route is guidance-only (a model download away from determinism) and contributes nothing to the measured number. financial-model-ingest is measured against a vanilla cell dump (every non-blank cell as Sheet!Addr: value-or-formula, SpreadsheetLLM's own baseline); its via-skill is one bounded analyst session — workbook map + assumption ledger + one scoped dependency trace — whose cost does not grow with the workbook, so 64.5% on the small committed fixture is the floor (a real 10k-cell model ≈ 75k tokens dumped vs the same few-hundred-token session, 95%+). The API-backed --live counts only move these figures at the margins. Where competing extraction methods existed, the shipped method carries a gauntlet receipt (meter/results/gauntlet/) showing why it won (e.g. web-ingest → trafilatura over readability-lxml / html-to-markdown / raw; log-json-discipline → drain3 templates over grep-with-context / tail-200 / raw — with the recorded caveat that token-cheapest ≠ diagnostic-best: tail -200 beats grep on tokens but misses app.log's mid-file ERROR burst, so the skill ships the hybrid grep→templates ladder; media-transcript-first → scene-detect keyframes OCR'd to text over keyframes-as-images over the raw 1fps dump, 1,881 → 627 → 24 tokens on tiny.mp4; financial-model-ingest → the layered session over a SheetCompressor-style modules-1+2 inverted index over the vanilla dump, 1,363 → 954 → 484 tokens on dcf_model.xlsx — the receipt for why the graph, not grid compression, is the right representation for a formula-dense model).

Equipping a skill

Each skill is a self-contained folder in the Anthropic Agent Skills layout. To equip one in an agent build:

1. Copy the skill folder into the build's skills directory, e.g. cp -r skills/pdf-ingest <agent-build>/.claude/skills/ (or install via the plugin marketplace once published).
2. Install its dependencies — check the skill's manifest.yaml requires block for the Python packages and any system binaries (e.g. pdf-ingest needs ocrmypdf + tesseract for the scanned/OCR route only).
3. The skill's SKILL.md description is its trigger — the agent reaches for it when its when_to_equip condition is met. No skill depends on any other skill being installed.

Regenerating the evidence

Evidence artifacts under meter/results/ are committed inputs (tests read them offline; they are not regenerated during the test run). To reproduce them from the fixtures, run the meter with the fixed timestamp so output is byte-identical:

# pdf-ingest MUST use --ocr (measures the real OCR route, not a zero-credit estimate)
uv run meter report fixtures/text_report.pdf fixtures/scanned_report.pdf fixtures/figure_report.pdf \
  --skill pdf-ingest --ocr --generated-at 2026-07-02T00:00:00Z --out meter/results/pdf-ingest.json

uv run meter report fixtures/sample.docx fixtures/sample.pptx fixtures/sales_small.xlsx \
  --skill office-to-markdown --generated-at 2026-07-02T00:00:00Z --out meter/results/office-to-markdown.json

uv run meter report fixtures/sales.csv fixtures/sales.parquet fixtures/sales_big.xlsx \
  --skill tabular-discipline --generated-at 2026-07-02T00:00:00Z --out meter/results/tabular-discipline.json

# image-ingest needs tesseract on PATH (brew install tesseract)
uv run meter report fixtures/screenshot.png fixtures/screenshot_log.png \
  --skill image-ingest --generated-at 2026-07-02T00:00:00Z --out meter/results/image-ingest.json

# web-ingest over the WCXB non-SPA corpus (the SPA shell is detect-and-refuse, not measured)
uv run meter report fixtures/article.html fixtures/docs.html fixtures/product.html \
  fixtures/forum.html fixtures/listing.html fixtures/table_heavy.html fixtures/code_heavy.html \
  --skill web-ingest --generated-at 2026-07-02T00:00:00Z --out meter/results/web-ingest.json

# log-json-discipline: whole-log + minified-JSONL baselines vs the drill-down session
uv run meter report fixtures/app.log fixtures/events.jsonl \
  --skill log-json-discipline --generated-at 2026-07-02T00:00:00Z --out meter/results/log-json-discipline.json

# media-transcript-first: 1fps-frame-dump baseline vs cleaned captions / OCR'd keyframes
# (the .mp4 route needs ffmpeg + tesseract on PATH; a captions-only corpus needs neither)
uv run meter report fixtures/fake_captions.vtt fixtures/tiny.mp4 \
  --skill media-transcript-first --generated-at 2026-07-02T00:00:00Z --out meter/results/media-transcript-first.json

# financial-model-ingest: vanilla cell-dump baseline vs the layered analyst session
uv run meter report fixtures/dcf_model.xlsx \
  --skill financial-model-ingest --generated-at 2026-07-02T00:00:00Z --out meter/results/financial-model-ingest.json

# the gauntlet receipts (which extractor/engine won, and why):
uv run meter gauntlet fixtures/screenshot.png fixtures/screenshot_log.png \
  --skill image-ingest --methods raw-image-tokens,ocr-tesseract,ocr-apple-vision \
  --generated-at 2026-07-02T00:00:00Z
uv run meter gauntlet fixtures/article.html fixtures/docs.html fixtures/product.html \
  fixtures/forum.html fixtures/listing.html fixtures/table_heavy.html fixtures/code_heavy.html \
  --skill web-ingest --methods raw-html-tokens,trafilatura,readability-lxml,html-to-markdown \
  --generated-at 2026-07-02T00:00:00Z
uv run meter gauntlet fixtures/app.log \
  --skill log-json-discipline --methods raw-file-tokens,grep-errors,drain3-templates,tail-200 \
  --generated-at 2026-07-02T00:00:00Z
uv run meter gauntlet fixtures/tiny.mp4 \
  --skill media-transcript-first \
  --methods raw-1fps-frames,scene-detect-keyframes-as-images,scene-detect-keyframes-ocr \
  --generated-at 2026-07-02T00:00:00Z
uv run meter gauntlet fixtures/dcf_model.xlsx \
  --skill financial-model-ingest --methods vanilla-dump,sheetcompressor-12,layered-l0-l3 \
  --generated-at 2026-07-02T00:00:00Z

Each command writes the JSON artifact plus a sibling .md client summary. Then rebuild the catalog (and the numbers this README quotes) from those artifacts:

uv run python scripts/gen_catalog.py          # rewrite catalog.json
uv run python scripts/gen_catalog.py --check  # verify catalog.json is up to date (CI gate)

Validating the library

scripts/validate.py is the repo gate: it fails (non-zero, with a terse failure list) if any skill is missing its SKILL.md/manifest.yaml, has invalid frontmatter, a name that disagrees across folder/frontmatter/manifest, a SKILL.md body over 150 lines, a description over the 1024-char context budget, missing manifest keys, an evidence path that isn't on disk, evidence that fails the meter's schema, or an evidence skill name that doesn't match its manifest.

uv run python scripts/validate.py    # "OK: 8 skills valid", or a FAIL list + exit 1
uv run pytest -q                     # full offline suite

How it fits the Harness

The Harness sells agents that are bespoke to each client yet built from proven, reused parts — the only way to fit one client exactly without rebuilding ingestion, memory, and governance every time. Tokenomics is the pillar that keeps those assembled agents cheap to run: the catalog above is the menu, a client build is the order (heavy PDF flow → equip pdf-ingest; Excel-driven finance → tabular-discipline), and every line item comes with a metered receipt. That is the client-facing promise made concrete — your agents, at a fraction of the token bill, and we can prove it. See NORTH_STAR.md for the full thesis.