Add a model-vs-baseline benchmark script

[shivamlalakiya]

Summary

Every estimator in this library gets its own accuracy number, but nothing checked whether that number actually beats the simple rule a fundraising shop already uses instead. scripts/benchmark_models_vs_baselines.py pairs each model with its naive baseline (rank by last year's total, predict last gift, mail everyone) and evaluates both on held-out, walk-forward splits: the synthetic donor panel (five seeds, mean and min-max range) by default, and the real KDD Cup 1998 file when you opt in.

Metrics: top-1/5/10 percent hit rate and lift, ROC-AUC, average precision, and a decile calibration gap for classifiers; MAE and share within 25 percent for amount predictions; net revenue and ROI for cost-aware mail selection. A verdict column (wins, modest, loses) is computed from one ratio rule documented in the module docstring, so the table reads without cross-referencing code.

This is separate from the existing scripts/benchmark_models.py, which is locked to a golden-file test and measures per-model accuracy alone. This one is a baseline comparison and does not touch that script, its test, or its golden file.

How to run it

python scripts/benchmark_models_vs_baselines.py                     # both datasets
python scripts/benchmark_models_vs_baselines.py --skip-kdd98         # synthetic only, no download
python scripts/benchmark_models_vs_baselines.py --fast --skip-kdd98  # one-seed smoke run
python scripts/benchmark_models_vs_baselines.py --out results/bench  # also writes bench.json / bench.csv

The KDD98 section downloads cup98lrn.zip once to ~/philanthropy_data; --skip-kdd98 stays fully offline. A tiny smoke test (tests/test_benchmark_models_vs_baselines_smoke.py) runs the --fast --skip-kdd98 path as a subprocess to prove it works offline; it does not touch KDD98 and is not a golden-file lock.

What a full run found

Full run: five synthetic seeds plus the full 95,412-donor KDD98 file. Most models beat their baseline: KDD98 MajorGiftClassifier response wins over the RFM rule, LapsePredictor beats "gave nothing last period" on both datasets, cost-aware mail selection raises more net revenue than mailing everyone ($4,382 vs $3,149) with a better ROI (0.35 vs 0.16), and the synthetic upgrade model beats "highest current-year total."

Some lose, stated plainly:

• On the synthetic panel, both DonorPropensityModel and MajorGiftClassifier lose to the plain "rank by cumulative giving" rule on every metric (hit rate and ROC-AUC). With only three engineered features, the naive rule already captures most of the signal.
• AskAmountRecommender beats "last gift" on the synthetic panel but loses to a flat median baseline there, and on KDD98 it loses to "last gift" (though it beats "average gift").
• KDD98 DonorPropensityModel response ROC-AUC (0.508) loses to the RFM rule (0.525).

One caveat on the KDD98 upgrade-model row: its top-1 and top-5 percent hit rates come out as 0 with an "n/a" verdict. Building that model's gift log from individual response dates and cutting it into calendar fiscal years leaves the walk-forward test fold landing on a fiscal year with an almost-zero rate of crossing the (rescaled) threshold, because very little giving is recorded in this file after mid-1997 outside the held-out mailing itself. docs/explanation/real_data_replication.md already recommends promotion-index periods over calendar fiscal years on this file for the same reason. The script only includes fiscal years that are fully resolved (mirroring the rule score_upgrade_prospects already uses internally), which fixed a real bug where an unresolved stub year was being scored; the remaining sparsity is a property of this file's tail, not something this PR forces a fix for.

Test plan

• make ci passes
• make riskcov passes
• flake8 and mypy clean on the new script
• python scripts/benchmark_models_vs_baselines.py --fast --skip-kdd98 runs offline and completes
• Full run (five seeds, full KDD98) completed and is summarized above
• CHANGELOG entry added under Unreleased