Skip to content

Fix pre-submission imputation and evaluation correctness - #219

Draft
juaristi22 wants to merge 5 commits into
mainfrom
fix/publication-correctness
Draft

Fix pre-submission imputation and evaluation correctness#219
juaristi22 wants to merge 5 commits into
mainfrom
fix/publication-correctness

Conversation

@juaristi22

@juaristi22 juaristi22 commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The pre-submission audit in #201 identified silent errors in preprocessing, conditional distributions, sampling, and method comparison. This change preserves donor-fitted transformations, computes probability-based scores, and prevents unsupported distribution forecasts from entering model selection.

Correctness changes

  • Reuse donor preprocessing on receivers and future predictions from returned fitted models; fit CV transformations on each training fold and score in original units.
  • Invert zero-inflated mixture distributions, including signed components and zero masses; apply survey weights to gates and component models.
  • Use QRF defaults that retain within-leaf variation, honor survey weights in the empirical leaf distributions, and offer independent-target marginal quantiles for distributional comparison. Sequential mode continues to support stochastic joint draws.
  • Fit requested QuantReg percentiles lazily, preserve regression intercepts for single-row/homogeneous predictions, and reject non-finite model inputs.
  • Draw OLS shocks per row with persistent independent draws and make WLS quantiles invariant to common survey-weight rescaling.
  • Score actual class probabilities, retain numeric counts as regression targets, support explicit target declarations and custom quantile grids, and handle constant or previously unseen evaluation classes.
  • Honor train_size and model seeds; select final tuned parameters through fresh training-data tuning after fold scoring.
  • Preserve Matching weights and RNG state, prune failed tuning trials, and expose failed-record counts. Matching supports donor draws; unsupported quantiles/probabilities fail explicitly and are excluded from distributional comparison.
  • Normalize mutual information using the same paired discretization and natural-log entropies; validate duplicate/overlapping columns and incompatible predictor types.
  • Run core autoimpute tests without optional R/MDN dependencies.

Fixes #202, fixes #203, fixes #204, fixes #205, fixes #206, fixes #208, fixes #209, fixes #212. Addresses the validation portion of #213.

This branch includes the existing fixes from #214 (#207), #215 (#210), and #218 (pre-existing lint failures), retaining their authorship. Those PRs can merge separately before rebasing this branch; this does not replace the JOSS paper PR.

Compatibility and publication

Numeric-coded categorical targets now require a categorical dtype or target_types={"variable": "categorical"}. Use QRF(sequential=False) for marginal quantiles of multiple targets. Quantiles of a sequential multivariate chain require a different estimator and now fail explicitly. Zero-inflated models offer independent mode for the same reason.

These corrections change imputed values and model rankings. Regenerate paper benchmarks and affected downstream datasets before submission. QRF retaining all leaf samples can increase memory use. This PR does not resolve the separate license, authorship, metadata, release/DOI, and documentation requirements tracked by #201.

Adversarial review fixes

The follow-up review reproduced five edge-case failures; all five are repaired with regression tests:

  • Promote numeric operands before subtraction in quantile loss and Matching tuning. The unsigned-count reproduction now reports loss 1.0 rather than 64.0, and tuning selects the independently verified lower-error distance.
  • Honor explicit numeric declarations for native and nullable boolean targets across fitting, comparison and cross-validation, while retaining default boolean classification.
  • Keep raw survey weights aligned and separate when the same column is transformed as a predictor, including folds, tuning, final fits and returned-model predictions.
  • Preserve constant categorical predictions with quantiles=None and attach exact point-mass probabilities using the documented result container.
  • Convert nullable numeric OLS prediction designs to native floating values while retaining intercepts and row indices.

Validation

All scheduled checks passed on repair commit b2c7ce3d8336669bc1ee1c3dfbb3a75569c6db67. The Python 3.14 CI run completed with 484 passed, 2 skipped in 239.53 seconds, including all 33 new regression cases and all 15 real-R Matching tests. The end-to-end pipeline, Python 3.12 smoke checks, lint, changelog, documentation build, and Vercel checks also passed. Formatting passed for 113 files; the post-format affected test run passed 118 tests.

The new regression suite reproduced 25 failures with 8 passing controls before repair. All 33 new cases now pass. Independent verification passed 31 checks and found all five planned repairs complete. The affected suite passed 425 tests, 1 skipped; the complete local Python 3.13.14 suite passed 469 tests, 3 skipped. The skips are the optional R and MDN runtime modules.

On a controlled normal-noise QRF fixture (3,000 training and 2,000 test observations), nominal q10/q50/q90 coverage changed from 27.45%/48.8%/70.25% to 11.15%/50.9%/88.3%. This is a regression fixture, not a universal calibration guarantee.

Optional MDN validation remains a gap because the existing workflow skips its dependency installation for this diff. This PR remains a draft; the separate submission requirements above are still open.

The CI suite reported 207 warnings, chiefly rpy2 deprecations plus solver/dependency warnings. Coverage XML was generated, but Codecov rejected the upload because a protected branch requires a token. The workflow treats that upload failure as non-blocking.

vahid-ahmadi and others added 4 commits September 16, 2026 14:30
@vahid-ahmadi @juaristi22
Give each per-variable QRF model its own seed
a297d63 
Every per-variable model builds its own generator from the seed it is
given, and all of them were handed self.seed. They therefore drew the
same random quantiles in the same row order, so variables imputed
together came out rank-comonotonic whatever their dependence in the
donor: three targets with nil conditional dependence reproduced at 0.71
Spearman, 0.11 after this change.

The offset follows the convention already used for the subsampling seed
in _apply_max_train_samples. QRF also now accepts a seed argument; there
was previously no way for a caller to vary the draws.

Fixes #207
@vahid-ahmadi @juaristi22
Prune failed Matching trials instead of scoring the training mean
4dd7caf 
Both tuning handlers caught every exception and substituted the training
mean, with no log and no counter, and that score went straight into the
Optuna objective. A mean-predictor is not a neutral score - on a
low-signal target it can beat a genuine matching fit on quantile loss -
so a parameter set under which matching always failed could be selected
as best and reported as the winning method.

Both now log the exception and raise TrialPruned. The predict path keeps
its NaN fill, which is the right behaviour there, but now reports the
total number of unmatched records rather than leaving silent NaN blocks.

Fixes #210
@vahid-ahmadi @juaristi22
Fix the Lint job, which is failing on main
2862e6f 
The Lint job installs ruff>=0.9.0 with no upper bound. ruff 0.16.7
formats Python inside markdown code blocks, which earlier versions left
alone, so five documentation files under docs/ became unformatted
without anyone changing them. make check-format fails on an untouched
checkout of main, and therefore on every open pull request.

Reformats the five files and gives the constraint an upper bound, so a
future ruff release changes the lint result only when someone chooses to
move the pin.
@juaristi22
Fix pre-submission imputation and evaluation correctness
eabe40b
@vercel

vercel Bot commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
microimpute-dashboard Ready Ready Preview Sep 17, 2026 9:39am UTC

This was referenced Sep 17, 2026
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment