Fix score_upgrade_prospects crashes and its fixed top-N report - #247
Merged
Merged
Conversation
…port
A donor population with no historical upgrades at all (or none left
below the leadership threshold) crashed with an opaque IndexError deep
inside predict_proba, and a thin training history crashed the same way
inside sklearn's internal cross-validated calibration. Both now raise a
clear, documented ValueError instead.
fiscal_year was also leaking in as a model feature even though it is
not donor-specific and always sits outside the training range at
scoring time; it stays as an output column but is no longer used to
train the model or picked for top_reasons.
The validation report's top-N lift used a fixed top_n=10, which read
as a perfect 1.0 lift on a small validation fold and an uninformative
one on a large real fiscal year. top_n now scales with the fold size
(configurable), and the report adds a per-decile breakdown, roc_auc,
average_precision, and a second naive baseline ("gave >= X last FY")
alongside the existing "top N by FY total" one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Four fixes to
score_upgrade_prospectsfound in review:IndexErrordeep insidepredict_proba. A thin training history (too few rows for the model's internal 5-fold calibration) crashed the same way inside sklearn'sCalibratedClassifierCV. Both now raise a clear, documentedValueErrorinstead.fiscal_yearwas leaking in as a model feature even though it isn't donor-specific, and the scored row's fiscal year always sits outside the range the model was trained on. It's still returned as an output column, just no longer used to train the model or picked fortop_reasons.top_n=10, which read as a perfect 1.0 lift on a small fold and an uninformative one on a large real fiscal year.top_nis now a parameter that defaults to ~10% of the held-out fold, and the report gained a per-decile breakdown,roc_auc,average_precision, and a second naive baseline ("gave >= X last FY") alongside the existing "top N by FY total" one.check_index_type=False; replaced with explicitstrcasts on both sides so the intent is visible instead of papered over.Why
These are correctness and honesty issues, not style: a shop whose upgrade candidates never (or always) crossed the threshold in history should get a clear error, not a stack trace from inside sklearn. A shop with a real, large validation fold deserves an honest top-1%-style lift number instead of one computed against a hardcoded 10 rows.
One note on the two floors that now exist together: the new 5-row floor is only the point at which
CalibratedClassifierCV's internal cross-validation literally cannot split and crashes - it is not a recommendation. The existing ~500-rowlow_data_warningis still the real guidance for when scores and the validation report are trustworthy; the 5-row floor just turns an unusable crash into a clear error instead of silently accepting data far below the point where any of this is meaningful.How tested
fiscal_yearexcluded fromtop_reasons,top_nscaling with fold size, decile/roc_auc/average_precision shape, both named baselines). All fail against the old code and pass against the new.make ci: 2206 passed, 30 skipped, coverage 98.39% (floor 92%);_upgrade.pyitself at 100% line/branch coverage.make riskcov: risk-tier subtree at 98% (floor 93%).make_donor_panel(random_state=0)end to end:n_training_rows=5819,n_validation_rows=1089,top_n=109(scaling with fold size instead of a fixed 10)roc_auc=0.625,average_precision=0.182baseline_topn_fy_total_upgrade_rate=0.128,lift_topn_fy_total=1.86baseline_giving_threshold=500.0,baseline_gave_threshold_upgrade_rate=0.153,lift_over_gave_threshold=1.56decilesranks correctly: decile 1 (top-scored 10%) actual upgrade rate 0.241, decile 10 (bottom) 0.055Changelog / docs
Added a
### Fixedentry under## [Unreleased]in CHANGELOG.md and updated the existingscore_upgrade_prospects### Addedbullet to match the new signature and report shape. Docstring's Parameters/Returns/Raises/Notes sections rewritten to describe the newtop_n/baseline_giving_thresholdparameters and report keys.