compute_predictor_correlations documents a "normalized mutual information matrix" that is "comparable to correlation coefficients" (predictor_analysis.py:72), and hardcodes the diagonal to 1.0 (:154-155). The values are not normalized: mutual information in nats is divided by an entropy in bits, and for a continuous column that entropy is a plug-in discrete estimate that reduces to log2(n_distinct) — a measure of sample size, not information.
Reproduction
Two identical columns, where the correct normalized value is exactly 1.0:
import numpy as np, pandas as pd
from microimpute import compute_predictor_correlations
x = np.random.default_rng(3).normal(size=1000)
df = pd.DataFrame({"a": x, "b": x.copy()})
print(compute_predictor_correlations(df, ["a", "b"], method="mutual_info")["mutual_info"].loc["a", "b"])n=200 0.5285
n=1000 0.5671
n=5000 0.5909
Perfectly dependent variables score around 0.55, and the value drifts upward with sample size. At n=1000: MI is 5.651 nats while _compute_entropy returns 9.9658 bits, which is exactly log2(1000).
Two errors compound. Dividing by H·ln2 to fix the units gives 0.818, still not 1.0, because the log2(n_distinct) proxy is not the entropy of a continuous variable. Because genuinely categorical columns get a real entropy while continuous ones get log2(n), values are not comparable across variable types — and predictor_target_mi (:221-225) is used to rank predictors for imputation.
Locations
predictor_analysis.py:163-171 — predictor-predictor normalisationpredictor_analysis.py:221-225 — predictor-target normalisation_compute_mutual_information at :529-550 — sklearn, returns nats_compute_entropy at :553-568 — log2, returns bits
Suggested fix
Convert units consistently, and use a normaliser valid for continuous variables — a proper NMI estimator, or normalising by min(H(X), H(Y)) computed on the same discretisation used for the MI estimate. Failing that, drop the normalisation, document the units as raw nats, and stop hardcoding the diagonal to 1.0, which currently hides the defect from inspection.
Found during a pre-JOSS-submission audit (#201), reproduced independently by a second reviewer.
compute_predictor_correlationsdocuments a "normalized mutual information matrix" that is "comparable to correlation coefficients" (predictor_analysis.py:72), and hardcodes the diagonal to 1.0 (:154-155). The values are not normalized: mutual information in nats is divided by an entropy in bits, and for a continuous column that entropy is a plug-in discrete estimate that reduces tolog2(n_distinct)— a measure of sample size, not information.Reproduction
Two identical columns, where the correct normalized value is exactly 1.0:
Perfectly dependent variables score around 0.55, and the value drifts upward with sample size. At n=1000: MI is 5.651 nats while
_compute_entropyreturns 9.9658 bits, which is exactlylog2(1000).Two errors compound. Dividing by
H·ln2to fix the units gives 0.818, still not 1.0, because thelog2(n_distinct)proxy is not the entropy of a continuous variable. Because genuinely categorical columns get a real entropy while continuous ones getlog2(n), values are not comparable across variable types — andpredictor_target_mi(:221-225) is used to rank predictors for imputation.Locations
predictor_analysis.py:163-171— predictor-predictor normalisationpredictor_analysis.py:221-225— predictor-target normalisation_compute_mutual_informationat:529-550— sklearn, returns nats_compute_entropyat:553-568—log2, returns bitsSuggested fix
Convert units consistently, and use a normaliser valid for continuous variables — a proper NMI estimator, or normalising by
min(H(X), H(Y))computed on the same discretisation used for the MI estimate. Failing that, drop the normalisation, document the units as raw nats, and stop hardcoding the diagonal to 1.0, which currently hides the defect from inspection.Found during a pre-JOSS-submission audit (#201), reproduced independently by a second reviewer.