Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
## [Unreleased]

### Added
- `scripts/benchmark_models_vs_baselines.py`: pairs every estimator with the simple domain rule it is meant to replace (rank by last year's total, predict last gift, mail everyone) and evaluates both on held-out, walk-forward splits: top-1%/5%/10% hit rate and lift, ROC-AUC, average precision and decile calibration for classifiers; MAE and within-25% for amount predictions; net revenue/ROI for cost-aware mail selection. Runs on the synthetic donor panel (five seeds, mean and min-max) by default; `--skip-kdd98` stays fully offline, `--fast` gives a one-seed smoke run, and `--out` writes the results table to JSON and CSV. Separate from the existing `scripts/benchmark_models.py` per-model accuracy table, which this does not replace or touch.
- `philanthropy.ingest.map_columns(df, mapping, *, required=...)`: renames a
user-supplied export's headers to canonical names and raises one
`ValueError` listing every still-missing required column, for callers
Expand Down
7 changes: 7 additions & 0 deletions docs/explanation/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ script:
python scripts/benchmark_models.py
```

Each model's accuracy above is measured alone. For whether a model actually
earns its complexity over the simple rule it would replace (rank by last
year's total, predict last gift, mail everyone), see
`scripts/benchmark_models_vs_baselines.py`, a separate script pairing every
estimator with its naive baseline on held-out, walk-forward splits of both
the synthetic donor panel and KDD Cup 1998.

For each of **five seeds** (42–46) the script builds a 4,000-row synthetic pool,
takes a stratified 75/25 train/test split, and fits every applicable binary
classifier in `philanthropy.models` on the documented feature set
Expand Down
Loading
Loading