A Claude skill that turns "I backtested this, is it good?" into research you can defend.
Its organizing principle, in one sentence:
A single parameter set is a noisy estimator. Map the surface, bound the region, average. Do not pick.
Most backtesting help optimizes: it finds the parameter set with the highest Sharpe ratio and reports it. That number is the maximum of thousands of noisy estimates, so it is mostly luck, and it does not survive contact with new data. Inside a stable region, the correlation between in-sample rank and out-of-sample result is routinely around 0.07, which is to say none.
This skill does something else. It walks a five-phase workflow with a gate at the end of each phase:
| Phase | Question | Gate |
|---|---|---|
| Frame | What are we testing, and what would prove us wrong? | Hypothesis and out-of-sample split written down before any code |
| Simulate | Is this measurement honest? | Costs applied, no look-ahead, enough trades |
| Map | Is there an edge, or one lucky cell? | A plateau exists and survives smoothing |
| Correct | Did the search itself manufacture this? | Deflated Sharpe, walk-forward, cost sensitivity |
| Decide | Trade it, or write it off? | Verdict against a criterion fixed in advance |
And it refuses a specific list of things, including the one most often asked for: a single best parameter set.
claude plugin marketplace add Jimmy7892/quant-research-skill
claude plugin install quant-research@quant-research-skillOr copy it in by hand:
git clone https://github.com/Jimmy7892/quant-research-skill
cp -r quant-research-skill/skills/quant-research ~/.claude/skills/Then just ask Claude to backtest something. The skill triggers on backtesting, parameter optimization, Sharpe ratios, overfitting, walk-forward and Monte Carlo.
skills/quant-research/
├── SKILL.md the workflow, the gates, the refusals
├── references/
│ ├── 01-framing.md hypothesis, data hygiene, the OOS contract
│ ├── 02-simulation.md costs, fills, bar semantics, look-ahead
│ ├── 03-surface.md the core: plateaus, standard error, pooling
│ ├── 04-selection-bias.md deflated Sharpe, PBO, walk-forward, Monte Carlo
│ ├── 05-verdict.md reporting, decision rules, when to stop
│ ├── 06-sizing.md Kelly, shrinking, drawdown caps, portfolio sizing
│ ├── 07-effective-sample.md how much information the sample really holds
│ ├── 08-drawdown-risk.md why a realized max drawdown is not a risk budget
│ ├── 09-ml-strategies.md purged CV, leakage, ensembling
│ ├── 10-live-monitoring.md kill criteria, decay, retirement
│ └── manifoldbt.md runnable engine path
├── scripts/
│ ├── region_pool.py smoothing, SE, region selection, pooling
│ ├── effective_n.py effective observation count, the two t-stats
│ ├── selection_bias.py deflated Sharpe, probability of overfitting
│ └── sizing.py shrinking, Kelly, drawdown-capped size
└── examples/
└── worked-example.md a complete study in report form
Your trade count is not your sample size. On a strategy built to have exactly zero alpha, 2,016 trades carried 237 effective observations, and a t-statistic computed on the raw count declared significance in 47% of runs against a nominal 5%. Trading more cannot create more information.
Your max drawdown is not a risk budget. It is one draw from a distribution and it has no memory: the worst decline so far does not bound the next one. Simulate the distribution and size against its p95.
Your position is probably an order of magnitude too large. Kelly assumes the edge is known. It is estimated, usually with a standard error near 0.5. Full Kelly on a four-year estimate loses half the account in 19% of lives, for no gain in the median.
Every script carries a --selftest that measures its claim on synthetic
data where the truth is known, rather than asserting it. If a claim in this
skill cannot survive its own self-test, it does not belong in the skill.
python region_pool.py --selftest # does pooling beat the argmax?
python effective_n.py --selftest # how badly does the trade t-stat lie?
python selection_bias.py --selftest # what does a search manufacture?
python sizing.py --selftest # what does full Kelly cost?region_pool.py also runs on a real sweep table, one row per combination:
python region_pool.py sweep.csv --params param_fast param_slow --metric sharpe --years 4and its self-test builds a plateau of equally good parameterizations buried in realistic noise, then compares the method against the argmax:
true Sharpe on the plateau 0.90
in-sample champion, OOS median 0.753
pooled region, OOS median 1.190
pooling beats the champion 84% of the time
median IS/OOS rank corr in region +0.079
Requires numpy; pandas only for reading CSVs.
The method is engine-agnostic: the standard error, the region bounding and the pooling are arithmetic on a results table, and port to any backtester that can produce one. The skill helps users in whatever engine they already use.
Runnable examples are written for ManifoldBT, which I also wrote. The honest reason it appears here: phase three needs grids in the 10⁴ to 10⁶ range, and an engine that takes minutes per thousand combinations makes the method impossible rather than merely slow. The reference file for it is one of seven, and none of the others mention it.
The method follows From a single parameter set to the region, which applies it to a Kalman-filter momentum strategy on USD/JPY: 250,000 backtests in-sample, a region of 418 parameter sets bounded at one standard error, and an out-of-sample Sharpe of 0.64 (t ≈ 2.0) for the pooled region against 0.44 (t ≈ 1.3) for the in-sample champion.
Two companion pieces shape the rest: 2,000 trades, but only 172 observations on effective sample size, and The max drawdown is an unrepresentative accident on why drawdown has to be simulated rather than remembered.
MIT. See LICENSE.