Add token-efficient-skill-optimizer

[yosishe]

Description

Adds token-efficient-skill-optimizer/ — a meta-skill that audits and optimizes an existing skill, system prompt, agent instruction set, or workflow for token and cost efficiency, without degrading quality or safety.

Seven modes: Analyze (audit only) · Recommend (plan) · Apply (optimize + reviewable diff) · Benchmark (before/after) · Explain · Refresh Evidence · Batch Audit (a whole skills directory).

Why this is not "make your prompt shorter" advice

• Honesty labels are enforced, not suggested. Every quantitative claim must carry one of six labels — [measured] (needs a data pointer), [estimated], [projected], [cache-dependent], [behavior-dependent], [reported]. scripts/validate_report.py blocks delivery on FAIL. Failed and reverted optimizations get reported, not hidden.
• The audited artifact is untrusted data. Instructions found inside a skill under audit — including instructions about how to report the savings — are recorded as injection findings and never followed.
• Safety text is exempt from every removal / merge / compression rule (R-S1). Grounded in arXiv:2510.22963, where black-box attacks on prompt-compressed agents hit a 0.71 average success rate vs 0.21 for the strongest baseline.
• 38 rules, each carrying source ids into a 42-source corpus that was primary-verified with title match required.

Relevance to Israeli / Hebrew skills

Hebrew is under-represented in BPE vocabularies, so the same sentence costs more tokens in Hebrew than in English — which makes token auditing more valuable for a Hebrew-first catalog, not less. Two behaviours are explicit and testable:

• measure_tokens.py detects language-suffixed siblings (X-he.md beside X.md) and reports them as bilingual_sibling_pairs, separate from duplicates — so a bilingual skill is never told to delete its own translation.
• Trigger detection includes Hebrew conditional markers (כאשר, רק, אם, לפני, בעת, במקרה) alongside English and Chinese, so a Hebrew skill's routing text is not mis-scored as prose.

Prior results (upstream, reproducible at pinned commits)

Run against three heavily-installed public skills. Token figures are [estimated] (tokenizer proxy with a disclosed ×1.15–1.25 Claude adjustment); byte counts are [measured].

target	trigger path before	after	change	verdict
improve-codebase-architecture (Matt Pocock)	1,540–1,673	1,247–1,356	−19.0%	kept
react-best-practices (Vercel)	2,026–2,202	1,888–2,052	−6.8%	kept
frontend-design (Anthropic)	1,887–2,051	1,563–1,700	−17.2%	reverted by its own evaluation

The third row is the one I'd point a reviewer at: the skill reverted its own optimization and reported it. That is the intended behaviour.

Two deltas from upstream, both required by the catalog rules

Documented in a "Distribution note" section inside SKILL.md and SKILL_HE.md, so an installed copy explains itself:

1. No package README.md — the catalog forbids it, so scripts/validate_package.py drops it from its own C01 inventory.
2. tests/fixtures/ is not shipped — those fixtures are deliberately malformed skill packages used as negative test cases. Shipping them would put 11 invalid SKILL.md files into the catalog (and would fail validate-skill.sh with 7 errors — I checked). scripts/run_tests.py is included for reference but needs them; the deterministic suite runs from the upstream repo. scripts/validate_package.py runs fully here.

Local verification before opening this PR

scripts/validate-skill.sh token-efficient-skill-optimizer/SKILL.md   -> PASS
description                                                         -> 948 bytes (< 1024 even under a C locale)
SKILL.md body                                                       -> 1,935 words (< 5,000)
secret-pattern grep (the CI regex, whole folder)                    -> clean
README.md in skill folder                                           -> none
evidence.json / metadata.json                                       -> valid JSON
scripts/validate_package.py .                                       -> PASS 10/10 checks, 0 violations

Upstream repo (MIT, full harness + research corpus): https://github.com/yosishe/token-efficient-skill-optimizer

Update — v1.2.1: the negative-trigger gap is closed

Your checklist asked for something I couldn't evidence, so I built it rather than ticking the box. All 48 existing cases asked given that the skill fired, did it behave? None asked should it have fired at all? — which is this package's own R-09 defect class, since T-07 flags exactly that in other people's frontmatter.

Six cases (T-31–T-36), each a distinct false-fire surface, not one probe reworded:

id	surface	prompt
T-31	lexical collision on "token"	"Our API tokens keep expiring mid-session. Optimize how we refresh them."
T-32	"optimize"/"cost" — runtime perf	"This React component re-renders on every keystroke. Optimize it and cut the cost."
T-33	"cut costs" — infra spend	"Our cloud bill jumped 30% this month. Help me cut costs."
T-34	named exclusion — one-off wording	"Make this one Slack message to my manager shorter and punchier."
T-35	named exclusion — authoring	"I want to build a new skill from scratch that files my expenses. Where do I start?"
T-36	subject overlap, no artifact	"Explain how BPE tokenization works and why Hebrew costs more tokens than English."

Deliberately no harmful-target row. H-07 already owns that behaviour in the sealed holdout; authoring a development twin after reading it would turn the holdout into training data.

Two mechanical guards, because the six cases alone would be theatre. Model-graded cases are blind to the thing they depend on — the frontmatter clause that makes not-firing possible. An "optimization" that deletes Do NOT use for… to save ~40 tokens would leave all six green while breaking every one. So run_tests.py now also asserts description states a negative boundary (R-09) and description names positive triggers (R-09). That is G-08 turned on this package instead of only on its targets.

Mutation-verified, since a test that has never failed proves nothing — deleting a case, duplicating a prompt, stripping Do NOT use, and stripping the quoted triggers each produced the specific expected FAIL:

✓ delete 1 negative case        -> 'negative-trigger coverage >= 6 cases'          FAILED as required
✓ duplicate a prompt            -> 'negative-trigger prompts are distinct'          FAILED as required
✓ strip 'Do NOT use'            -> 'description states a negative boundary (R-09)'  FAILED as required
✓ strip quoted trigger phrases  -> 'description names positive triggers (R-09)'     FAILED as required

Upstream suite 100 → 104, package gates 10/10, split floors raised (cases 20→26, pool 40→46) so the coverage can't be silently hollowed out, and the three load-bearing names pinned in REQUIRED_TESTS so a rename can't empty them.

What this does not verify, stated plainly in tests/testing-guide.md: these cases test the skill's behaviour once it is already in context. Whether the host loads it at all is decided upstream from the description, and no test inside the package can observe that — it needs the agent in the loop. The guards cover the precondition, not the routing decision. If your side has a way to exercise host-level routing, I'd rather use yours than claim mine covers it.

I left README.md's skills table alone — it currently lists 10 of the 31 skill folders, so it doesn't look like it's maintained per-PR. Happy to add a row if you'd like one.

Checklist

Quality (human review)

• Description follows [What] + [When] + [Capabilities] pattern with trigger phrases
• Instructions are specific and actionable (not vague)
• Error handling / troubleshooting section included
• Usage examples provided ("User says: ... → Result: ...")
• No unauthorized external network calls — Refresh Evidence mode fetches provider pricing pages only when the user invokes it; live A/B runs require an explicitly approved API budget and are otherwise never made

Testing (required)

• Tested triggering on obvious tasks (skill loads automatically)
• Tested triggering on paraphrased requests — 4 mode-routing cases in tests/cases.jsonl are paraphrases ("Why is this agent so expensive to run?", "Cut this skill's cost")
• Verified doesn't trigger on unrelated topics — added in v1.2.1 after opening this PR (it was unticked here originally, with an offer to close the gap). Six negative-trigger cases T-31–T-36 in tests/cases.jsonl, plus two mechanical guards. Details below.
• Tested on at least one AI agent (fill in table below)

Agent Testing

Agent	Tested?	Notes
Claude Code	Yes	Primary environment. Ran all seven modes; the three case studies above were produced with it. validate_package.py passes 10/10 on this exact folder.
Cursor	No	Not tested. Listed in supported_agents because the skill is plain Markdown + Python with no Claude-specific APIs, but I have not verified it there and would rather say so than claim it.
Other	No	—

One thing I'd flag rather than let a reviewer find it: references/measurement.md states that tiktoken undercounts Claude tokens by ~15–20%, attributed to Anthropic's token-counting guidance, and that attribution has no URL in the corpus. It is deliberately excluded from evidence.json for that reason. Happy to either source it properly or soften the wording if you'd prefer.

[github-actions]

Welcome to Skills IL! Thanks for your first contribution.

Our automated checks are running now:

• Skill Validation - Checking your SKILL.md format
• Security Scan - Scanning for secrets and vulnerabilities

While you wait, make sure you've reviewed our Contributing Guide.

A maintainer will review your PR within 48 hours.

[yosishe]

Closing the loop on the one checklist item I left unticked: verified doesn't trigger on unrelated topics is now backed by evidence rather than a claim, pushed as v1.2.1.

Six negative-trigger cases (T-31–T-36) in tests/cases.jsonl, each naming a distinct false-fire surface — lexical collision on "token" (auth credentials), on "optimize"/"cost" (runtime rendering, infrastructure spend), the two exclusions named in the description, and subject-matter overlap with no artifact to audit.

Two mechanical guards came with them, because the cases alone would be theatre: a model-graded case can't see the frontmatter clause it depends on, so an optimization that deletes Do NOT use for… to save ~40 tokens would leave all six green while breaking every one. run_tests.py now fails the build on that. All four assertions were mutation-verified before shipping — each one confirmed to fail when the behaviour it covers is deliberately broken. Suite 100 → 104, gates 10/10.

One limit I'd rather state than paper over: these cases test behaviour once the skill is already in context. Whether the host loads it at all is decided upstream from the description, and no in-package test can observe that. The guards cover the precondition, not the routing decision — if you have a way to exercise host-level routing on your side, I'd rather use yours.

Full detail in the PR description. CI is still waiting on maintainer approval for the workflow runs (first-time fork contributor).