Add a plain-language Results section to the docs - #255
Open
shivamlalakiya wants to merge 2 commits into
Open
shivamlalakiya wants to merge 2 commits into
shivamlalakiya wants to merge 2 commits into
Conversation
Fundraisers keep asking the same question about every model: does it actually beat what we already do without it? These six pages answer that in donor counts instead of statistics, one per model ($1K upgrade, response/major gift, lapse, suggested ask, planned giving, who to mail), each comparing the model's picks against a simple rule and against random and giving a plain verdict. scripts/make_results_pages.py generates every number and chart the pages cite by running the existing benchmark_models_vs_baselines.py plus one worked example of score_upgrade_prospects, so nothing here is typed by hand; its output is committed under docs/assets/results/.
Match the upgrade page's rule figure to its chart (18, not 17), say what the decile chart actually shows (only the model's top slice stands out), mark the planned-giving score as untested on real bequest data rather than as a win, and title the response page by the question it answers.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fundraisers keep asking the same question about every model in this
library: does it actually beat what we already do without it? This adds a
"Results" section to the docs with one short page per model ($1K upgrade,
response/major gift, lapse, suggested ask, planned giving, who to mail),
each showing the model's picks against a simple rule and against random
picking in donor counts, with a plain verdict ("beats the rule", "about
the same", or "does not beat the rule; use the rule instead").
docs/results/index.mdplus one page per model, an index table with aone-line verdict each, and a short "how we tested" note.
scripts/make_results_pages.pygenerates every number and chart thepages cite, running
scripts/benchmark_models_vs_baselines.pyplus oneworked example of
score_upgrade_prospectson a seeded sample panel.Its committed output (
docs/assets/results/*.png,results.json) iswhat the pages reference; nothing is typed by hand.
[Unreleased].Two losing/mixed results are called out plainly rather than smoothed
over: the response/major-gift model does not beat ranking by past giving
on the sample data, and the suggested-ask model loses to "ask what they
gave last time" on the real KDD Cup 1998 file. The lapse page also flags
that on a file where almost every donor is about to lapse (KDD Cup 1998),
a 96% hit rate is about the same as guessing, since the base rate is
already 95%.
Test plan
make ci(2229 passed, 30 skipped, 98.42% coverage, floor 92%)make riskcov(98% over the risk tier, floor 93%)mkdocs build --strictpassespython scripts/make_results_pages.py --with-kdd98regenerates thecommitted JSON/PNGs from scratch