Skip to content

Add ranking metrics to philanthropy validate - #250

Merged
shivamlalakiya merged 5 commits into
mainfrom
f8-validate-report
Sep 26, 2026
Merged

shivamlalakiya merged 5 commits into
mainfrom
f8-validate-report

Conversation

@shivamlalakiya

Copy link
Copy Markdown
Contributor

What

validate now reports, alongside the existing precision/recall/F1/ROC-AUC:

  • average precision
  • the holdout's base rate
  • a 10-row decile table (n, positives, hit rate, lift over the base rate), sorted by predicted probability descending
  • a top-N hit rate and share-of-positives-captured line, controlled by a new --top-n flag that takes a row count or a percentage like 10% (default: top 10% of rows)

Precision, recall, and F1 are now labelled "at threshold 0.5" so they read as one view of the model rather than the whole picture.

Also fixed train, score, and validate scoring on a bare numpy array after the model was fit on named columns, which made every run of score or validate print an sklearn "X does not have valid feature names" warning. All three now pass the feature DataFrame through consistently.

Why

On a holdout where only about 9% of donors were the target outcome, the old report showed recall of 0.107 at the fixed 0.5 threshold, which reads as a broken model. The same model had a decile-1 hit rate of 0.420 (4.5x lift over the base rate) and an average precision of 0.459, meaning it ranked donors usefully even though it looked broken under a single fixed threshold. The old report had no way to see that. This mirrors how a real prospect list gets used: an analyst mails or calls the top N donors by score, not everyone above 0.5, so hit rate and lift in that top slice is the number that actually matters.

Example output

On a holdout with a 12.4% positive rate:

precision (at threshold 0.5) 1.000
recall    (at threshold 0.5) 1.000
f1        (at threshold 0.5) 1.000
roc_auc                      1.000
average_precision            1.000
base_rate                    0.124

decile  n     positives  hit_rate  lift
     1  146   146        1.000     8.05x
     2  146   35         0.240     1.93x
     3  146   0          0.000     0.00x
     4  146   0          0.000     0.00x
     5  146   0          0.000     0.00x
     6  146   0          0.000     0.00x
     7  146   0          0.000     0.00x
     8  145   0          0.000     0.00x
     9  145   0          0.000     0.00x
    10  145   0          0.000     0.00x

top 146 of 1457: hit_rate 1.000, captures 0.807 of all positives

How tested

  • Added test_cli_validate_reports_ranking_metrics, test_cli_validate_top_n_accepts_count_and_percentage, and test_cli_train_score_validate_do_not_warn_on_feature_names to tests/test_cli.py.
  • make ci: 2198 passed, 30 skipped, required coverage 92%, actual 98.36%.
  • make riskcov: risk-tier floor 93%, actual 98%.
  • Updated docs/how-to/use_the_cli.md and docs/reference/cli.md with the new flag and example output, and added a CHANGELOG entry under Unreleased.

Precision/recall/F1 at a fixed 0.5 threshold hid a usable model whenever
the target was rare: a holdout with a 9% positive rate could show a
recall near zero while the same model still ranked donors well. validate
now also reports average precision, the base rate, a 10-row decile table
(hit rate and lift over the base rate per decile), and a top-N hit
rate/capture line via a new --top-n flag (count or percentage, default
top 10%). The old metrics are now labelled "at threshold 0.5" so they
read as one view rather than the whole picture.

Also fixed train, score, and validate scoring on a bare numpy array after
fitting on named columns, which made every run of score or validate print
an sklearn "does not have valid feature names" warning.
@shivamlalakiya
shivamlalakiya merged commit 53e2314 into main Sep 26, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant