compute_loss(..., "log_loss") is called with class labels rather than probabilities, and substitutes fabricated 0.99/0.01 probabilities for them. The number it returns is an affine function of accuracy, not a proper scoring rule — and it disagrees with the log loss that cross_validate_model reports for the same model on the same data.
A second defect feeds into the first: integer targets with fewer than ten distinct values are silently reclassified as categorical, which switches a regression to a classifier and routes its output into this same path.
Both are reachable from the documented public API.
1. Log loss from class labels
Call site microimpute/comparisons/metrics.py:343-347 (in _compute_method_losses, def at :245); substitution at metrics.py:97-137.
import numpy as np
from microimpute import compute_loss
y = np.random.default_rng(0).integers(0, 2, 4000)
p = np.random.default_rng(1).integers(0, 2, 4000)
print(compute_loss(y, p, "log_loss", labels=np.unique(y))[1])
Returns 2.2525, which is exactly -(acc·ln0.99 + (1-acc)·ln0.01) for the realised accuracy of 0.512. The honest log loss of a calibrated 0.5/0.5 forecast is 0.6931. A model that memorises labels scores far better than a perfectly calibrated one, which inverts the ranking the metric exists to produce.
cross_validate_model does this correctly — it requests return_probs=True and uses predict_proba (cross_validation.py:290-320). So the two public evaluation paths report incomparable numbers. End to end on a coin-flip target with QRF, n=3000:
compare_metrics log_loss (q=0.5): 2.1376
cross_validate_model mean_test log_loss: 0.7899
truth, log(2): 0.6931
predictor_analysis.py:635-640 (_compute_losses_from_predictions, def at :610) has the same defect and passes no labels= at all. The other log-loss path in that file (:169-190) is correct.
2. Integer counts silently become classification
utils/type_handling.py:49-66 treats any numeric column with nunique() < 10 and equally spaced levels as categorical (max_unique=10, hardcoded), and metrics.py:33-52 then selects log loss for it.
A Poisson(2) count clipped to 0–8 qualifies. Running get_imputations([OLS], ...) then compare_metrics returns one value, 4.9359, repeated across all 19 quantiles — a metric completely insensitive to model quality. The cause is that the imputer switches OLS to a classifier: predictions at q=0.05 and q=0.95 come back byte-identical, in {1.0, 2.0}. Those labels then hit the substitution above.
Where a genuine regression output does reach log loss, it hard-fails on a legitimate prediction:
from microimpute import compute_loss
compute_loss(np.array([0,1,2,1,0]), np.array([1.5,1.2,0.4,0.9,1.1]), "log_loss", labels=np.array([0,1,2]))
# RuntimeError: Failed to compute log loss: y_prob contains values greater than 1: 1.5
The detection is also sample-size dependent, so the same variable changes type between folds: with a single rare tenth level and KFold(5, shuffle=True, random_state=42), folds 0, 1, 2 and 4 treat it as numeric and fold 3 treats it as categorical. The same variable is categorical at n=50 and n=200, numeric at n=2000.
Suggested fix
- Have
_compute_method_losses and _compute_losses_from_predictions consume the "probabilities" entry that models already return under return_probs=True, as cross_validation._compute_fold_loss_by_metric does. Where probabilities genuinely are unavailable, report accuracy or Brier rather than a fabricated log loss.
- Select the metric from the model's declared target type rather than re-detecting it from the test column, so type cannot vary across folds.
- Refuse to compute log loss when the input is neither valid probabilities nor class labels, instead of returning a constant.
- Reconsider the
max_unique=10 heuristic for count data, and make it overridable.
Related, minor
kl_divergence (metrics.py:571-586) returns an arbitrary finite constant of about 22.3 for disjoint categorical distributions, which is ≈ log(1/ε) for the hardcoded epsilon = 1e-10 at metrics.py:578. Since epsilon is a local with no parameter, a caller cannot change it. Returning inf, or a bounded symmetric divergence such as Jensen–Shannon, would be more defensible than a constant that lands in the same column as other variables' values.
Found during a pre-JOSS-submission audit (#201); claims independently reproduced by a second reviewer, who corrected the mechanism in part 2 and downgraded the KL point to a note.
compute_loss(..., "log_loss")is called with class labels rather than probabilities, and substitutes fabricated 0.99/0.01 probabilities for them. The number it returns is an affine function of accuracy, not a proper scoring rule — and it disagrees with the log loss thatcross_validate_modelreports for the same model on the same data.A second defect feeds into the first: integer targets with fewer than ten distinct values are silently reclassified as categorical, which switches a regression to a classifier and routes its output into this same path.
Both are reachable from the documented public API.
1. Log loss from class labels
Call site
microimpute/comparisons/metrics.py:343-347(in_compute_method_losses, def at :245); substitution atmetrics.py:97-137.Returns 2.2525, which is exactly
-(acc·ln0.99 + (1-acc)·ln0.01)for the realised accuracy of 0.512. The honest log loss of a calibrated 0.5/0.5 forecast is 0.6931. A model that memorises labels scores far better than a perfectly calibrated one, which inverts the ranking the metric exists to produce.cross_validate_modeldoes this correctly — it requestsreturn_probs=Trueand usespredict_proba(cross_validation.py:290-320). So the two public evaluation paths report incomparable numbers. End to end on a coin-flip target with QRF, n=3000:predictor_analysis.py:635-640(_compute_losses_from_predictions, def at :610) has the same defect and passes nolabels=at all. The other log-loss path in that file (:169-190) is correct.2. Integer counts silently become classification
utils/type_handling.py:49-66treats any numeric column withnunique() < 10and equally spaced levels as categorical (max_unique=10, hardcoded), andmetrics.py:33-52then selects log loss for it.A Poisson(2) count clipped to 0–8 qualifies. Running
get_imputations([OLS], ...)thencompare_metricsreturns one value, 4.9359, repeated across all 19 quantiles — a metric completely insensitive to model quality. The cause is that the imputer switches OLS to a classifier: predictions at q=0.05 and q=0.95 come back byte-identical, in {1.0, 2.0}. Those labels then hit the substitution above.Where a genuine regression output does reach log loss, it hard-fails on a legitimate prediction:
The detection is also sample-size dependent, so the same variable changes type between folds: with a single rare tenth level and
KFold(5, shuffle=True, random_state=42), folds 0, 1, 2 and 4 treat it as numeric and fold 3 treats it as categorical. The same variable is categorical at n=50 and n=200, numeric at n=2000.Suggested fix
_compute_method_lossesand_compute_losses_from_predictionsconsume the"probabilities"entry that models already return underreturn_probs=True, ascross_validation._compute_fold_loss_by_metricdoes. Where probabilities genuinely are unavailable, report accuracy or Brier rather than a fabricated log loss.max_unique=10heuristic for count data, and make it overridable.Related, minor
kl_divergence(metrics.py:571-586) returns an arbitrary finite constant of about 22.3 for disjoint categorical distributions, which is ≈log(1/ε)for the hardcodedepsilon = 1e-10atmetrics.py:578. Since epsilon is a local with no parameter, a caller cannot change it. Returninginf, or a bounded symmetric divergence such as Jensen–Shannon, would be more defensible than a constant that lands in the same column as other variables' values.Found during a pre-JOSS-submission audit (#201); claims independently reproduced by a second reviewer, who corrected the mechanism in part 2 and downgraded the KL point to a note.