Skip to content

Fix score_upgrade_prospects crashes and its fixed top-N report - #247

Merged
shivamlalakiya merged 3 commits into
mainfrom
f5-upgrade-fixes
Sep 26, 2026
Merged

shivamlalakiya merged 3 commits into
mainfrom
f5-upgrade-fixes

Conversation

@shivamlalakiya

Copy link
Copy Markdown
Contributor

What

Four fixes to score_upgrade_prospects found in review:

  • A donor population with no historical upgraders at all (or none left below the leadership threshold) crashed with an opaque IndexError deep inside predict_proba. A thin training history (too few rows for the model's internal 5-fold calibration) crashed the same way inside sklearn's CalibratedClassifierCV. Both now raise a clear, documented ValueError instead.
  • fiscal_year was leaking in as a model feature even though it isn't donor-specific, and the scored row's fiscal year always sits outside the range the model was trained on. It's still returned as an output column, just no longer used to train the model or picked for top_reasons.
  • The validation report used a fixed top_n=10, which read as a perfect 1.0 lift on a small fold and an uninformative one on a large real fiscal year. top_n is now a parameter that defaults to ~10% of the held-out fold, and the report gained a per-decile breakdown, roc_auc, average_precision, and a second naive baseline ("gave >= X last FY") alongside the existing "top N by FY total" one.
  • The CLI/direct-call parity test compared indices with check_index_type=False; replaced with explicit str casts on both sides so the intent is visible instead of papered over.

Why

These are correctness and honesty issues, not style: a shop whose upgrade candidates never (or always) crossed the threshold in history should get a clear error, not a stack trace from inside sklearn. A shop with a real, large validation fold deserves an honest top-1%-style lift number instead of one computed against a hardcoded 10 rows.

One note on the two floors that now exist together: the new 5-row floor is only the point at which CalibratedClassifierCV's internal cross-validation literally cannot split and crashes - it is not a recommendation. The existing ~500-row low_data_warning is still the real guidance for when scores and the validation report are trustworthy; the 5-row floor just turns an unusable crash into a clear error instead of silently accepting data far below the point where any of this is meaningful.

How tested

  • Added regression tests for each fix (single-class history, too-few-rows, fiscal_year excluded from top_reasons, top_n scaling with fold size, decile/roc_auc/average_precision shape, both named baselines). All fail against the old code and pass against the new.
  • make ci: 2206 passed, 30 skipped, coverage 98.39% (floor 92%); _upgrade.py itself at 100% line/branch coverage.
  • make riskcov: risk-tier subtree at 98% (floor 93%).
  • Ran the example from make_donor_panel(random_state=0) end to end:
    • n_training_rows=5819, n_validation_rows=1089, top_n=109 (scaling with fold size instead of a fixed 10)
    • roc_auc=0.625, average_precision=0.182
    • baseline_topn_fy_total_upgrade_rate=0.128, lift_topn_fy_total=1.86
    • baseline_giving_threshold=500.0, baseline_gave_threshold_upgrade_rate=0.153, lift_over_gave_threshold=1.56
    • deciles ranks correctly: decile 1 (top-scored 10%) actual upgrade rate 0.241, decile 10 (bottom) 0.055

Changelog / docs

Added a ### Fixed entry under ## [Unreleased] in CHANGELOG.md and updated the existing score_upgrade_prospects ### Added bullet to match the new signature and report shape. Docstring's Parameters/Returns/Raises/Notes sections rewritten to describe the new top_n/baseline_giving_threshold parameters and report keys.

…port

A donor population with no historical upgrades at all (or none left
below the leadership threshold) crashed with an opaque IndexError deep
inside predict_proba, and a thin training history crashed the same way
inside sklearn's internal cross-validated calibration. Both now raise a
clear, documented ValueError instead.

fiscal_year was also leaking in as a model feature even though it is
not donor-specific and always sits outside the training range at
scoring time; it stays as an output column but is no longer used to
train the model or picked for top_reasons.

The validation report's top-N lift used a fixed top_n=10, which read
as a perfect 1.0 lift on a small validation fold and an uninformative
one on a large real fiscal year. top_n now scales with the fold size
(configurable), and the report adds a per-decile breakdown, roc_auc,
average_precision, and a second naive baseline ("gave >= X last FY")
alongside the existing "top N by FY total" one.
@shivamlalakiya
shivamlalakiya merged commit f7597b8 into main Sep 26, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant