adding HDP based coverage test - #19
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The two_tailed_p semantic change conflicts with existing tests/docs and the new HDP additions need small correctness/doc fixes (example code + input handling) before merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR adds an HDP-based coverage test (hdp_coverage_test) intended to provide a more sensitive calibration check when posterior density values are available, alongside documentation and test coverage.
Changes:
- Add
hdp_coverage_testutility and export it from the package. - Add unit test coverage for
hdp_coverage_test. - Document the new HDP workflow in the README.
File summaries
| File | Description |
|---|---|
tests/test_utils.py |
Adds a new test for hdp_coverage_test and updates utils imports. |
src/pted/utils.py |
Reworks two_tailed_p and introduces hdp_coverage_test. |
src/pted/__init__.py |
Exposes hdp_coverage_test at the package level. |
README.md |
Adds usage docs for hdp_coverage_test and updates footnotes. |
Review details
Suppressed comments (2)
src/pted/utils.py:319
- The leftover commented-out block should be removed entirely instead of kept inline (it’s dead code after the
two_tailed_prewrite, and makes the surrounding logic harder to scan).
# assert df > 2, "Degrees of freedom must be greater than 2 for two-tailed p-value calculation."
# alpha = chi2_dist.pdf(chi2, df)
# mode = df - 2
# if np.isclose(chi2, mode):
src/pted/utils.py:497
- The docstring describes “higher density than the ground truth”, but the code uses
>=(ties counted as higher). Also, input shapes aren’t validated, and importingchi2_distinside the function is redundant since it’s already imported at module scope.
Consider normalizing inputs with np.asarray, validating shapes, using strict > to match the wording, and handling the Nsim==1 case explicitly to avoid an ill-defined chi² combination step.
from scipy.stats import chi2 as chi2_dist
Nsamp, Nsim = posterior_samples.shape
q = np.sum(posterior_samples >= ground_truth[None], axis=0)
chi2_hdp = -2 * np.sum(np.log((q + 1) / (Nsamp + 1)))
- Files reviewed: 4/4 changed files
- Comments generated: 4
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #19 +/- ##
==========================================
- Coverage 97.66% 97.65% -0.01%
==========================================
Files 4 4
Lines 342 341 -1
==========================================
- Hits 334 333 -1
Misses 8 8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
If you have a posterior density function, that is probably more sensitive than pted. We now provide a consistent implementation for the sake of comparison.