Per-variable QRF models share one seed, making jointly imputed variables comonotonic

[vahid-ahmadi]

Every per-variable QRF model is seeded identically, so when several variables are imputed together each row receives the same random quantile for all of them. The imputed variables come out rank-comonotonic, and their joint distribution is an artefact of the shared RNG rather than something estimated from the donor.

This is on the default path, and policyengine-uk-data hits it in production.

Mechanism

_create_model_for_variable passes the same integer seed to every target's model:

# microimpute/models/qrf.py:711
return _QRFModel(seed=self.seed, ...)

and each model builds its own generator from it:

# microimpute/models/qrf.py:172
self._rng = np.random.default_rng(self.seed)

Two independently constructed models therefore emit identical draws:

from microimpute.models.qrf import _QRFModel
import numpy as np, logging
a, b = _QRFModel(42, logging.getLogger("x")), _QRFModel(42, logging.getLogger("x"))
print(np.allclose(a._rng.beta(1, 1, 8), b._rng.beta(1, 1, 8)))   # True

A bare predict(X) — no explicit quantiles= — falls through to the stochastic mean_quantile beta-sampling branch (qrf.py:273, reached via qrf.py:422, :524-533), so this is the default, not an opt-in.

Reproduction

Three targets with genuinely independent conditional shocks, imputed together, using the same call shape as policyengine-uk-data:

import numpy as np, pandas as pd
from microimpute.models import QRF

rng = np.random.default_rng(7); n = 1500
X = pd.DataFrame({"inc": rng.normal(30, 8, n), "age": rng.normal(45, 12, n)})
d = X.copy()
for v in ["savings", "property_wealth", "corporate_wealth"]:
    d[v] = 0.5 * X["inc"] + rng.normal(0, 10, n)      # independent shocks

te = pd.DataFrame({"inc": rng.normal(30, 8, 600), "age": rng.normal(45, 12, 600)})
p = QRF().fit(d, ["inc", "age"], ["savings", "property_wealth", "corporate_wealth"]).predict(te)
print(p.rank().corr().round(3))

                  savings  property_wealth  corporate_wealth
savings             1.000            0.706             0.708
property_wealth     0.706            1.000             0.705
corporate_wealth    0.708            0.705             1.000

True conditional rank correlation is ~0. Isolating the mechanism from sequential imputation — two separate single-target imputers, so no target can be a predictor for another — gives Spearman 0.85 on a truth of −0.019. Desynchronising the two generators by burning one extra predict() call collapses it to 0.018, which confirms the shared seed is the cause.

Production exposure

policyengine-uk-data/policyengine_uk_data/utils/qrf.py:68 calls self.model.predict(X) with no quantiles, and datasets/imputations/wealth.py:48 imputes ten variables in one call — owned_land, property_wealth, corporate_wealth, gross_financial_wealth, net_financial_wealth, main_residence_value, other_residential_property_value, non_residential_property_value, savings, num_vehicles. consumption.py imputes eleven the same way. The dependence structure among imputed UK wealth and consumption components is therefore induced by the RNG, which matters for any analysis keying on concentration or on the joint tail.

Worth noting alongside: that wrapper's docstring says "Predictions at the 0.5 quantile (median)", but a bare predict() returns a random draw, not the median.

Suggested fix

Derive a distinct child seed per target at qrf.py:711, e.g. np.random.SeedSequence(self.seed).spawn(n_targets), or pass a single shared Generator advanced across variables rather than an int. Also expose a seed argument on QRF.__init__ — there is currently no way for a caller to vary the draws (QRF(seed=1234) raises TypeError).

Origin

qrf.py:172 comes from #177 (541e9d3, 17 Apr 2026), which made the RNG persistent per instance to fix a variance-collapse bug across repeated calls. That fix was correct as far as it went; it did not consider that sibling per-variable models all seed from the same self.seed.

Found during a pre-JOSS-submission audit (#201), reproduced independently twice.