fix: use validation data and support N-forecaster ensembles in WeightsCombiner

[majidkhoshrou]

What does this PR do?

Fixes three bugs in WeightsCombiner (learned-weights ensemble combiner) and one in
StackingCombiner that limit the ensemble machinery to small (2-model), always-trainable
ensembles, even though EnsembleForecastingModel.forecasters is already a generic
dict[str, Forecaster] intended to support any number of base forecasters:

1. WeightsCombiner.fit() ignored data_val. The method accepted a data_val
   parameter but never referenced it, so the per-quantile "who wins" classifier was always
   trained on in-sample training predictions. In-sample predictions from boosted/tree
   forecasters are systematically over-optimistic, biasing the combiner toward whichever
   forecaster overfits the training window hardest. fit() now trains on data_val when
   supplied, falling back to data otherwise.

2. The degenerate-label guard didn't generalize past 2 forecasters.
   _validate_labels() only fell back to a DummyClassifier when exactly one forecaster
   ever won (the only degenerate case possible with 2 base models). With 3+ forecasters
   it's common for a subset to never win, which crashed predict_proba with a
   ValueError (column count mismatch) instead of falling back gracefully. The guard now
   triggers whenever fewer forecasters win than are registered, so ensembles of arbitrary
   size no longer crash.

3. The DummyClassifier fallback silently discarded its own answer.
   _predict_weights() hardcoded weight 1.0 on self._label_encoder.classes_[0] — the
   alphabetically-first registered forecaster name — instead of asking the fitted
   DummyClassifier(strategy="most_frequent") which forecaster actually wins most often.
   This is easy to miss with 2 models (right about half the time by chance) but becomes a
   frequent, silent mis-attribution with 3+ models. The fallback now uses the recorded
   majority winner.

4. StackingCombiner.fit() also discarded data_val. It accepted data_val but
   always passed data_val=None to the underlying meta-forecaster's fit(). It now passes
   validation data through so meta-forecasters that support early stopping (e.g. LGBM,
   XGBoost) can use it.

Together these make the ensemble machinery robust for ensembles of arbitrary size (not
just the built-in 2-model lgbm/gblinear preset default) and make the learned-weights
combiner actually use held-out validation data as its name/docstring already implied.

No public API changes — all fixes are internal to fit()/_validate_labels()/
_predict_weights() behavior.

Closes #

Type of change

• Bug fix
• New feature
• Breaking change (see checklist below)
• Documentation
• Refactor / chore / CI

Breaking changes checklist

N/A — internal fitting behavior only; no public API, config schema, or serialized object
changes.

AI disclosure

• No AI assistance was used (beyond grammar/spelling)
• AI assistance was used — tool(s): GitHub Copilot (Claude Sonnet 4.5)
  • I have reviewed, understand, and can explain all AI-generated code in this PR
  • This is disclosed in a commit message (Assisted-by: GitHub Copilot (Claude Sonnet 4.5))

Checklist

• poe all --check passes locally
• Tests added/updated for the change
• Documentation updated (docstrings, user guide, examples) if needed
• Commits are signed off per our DCO (git commit -s)
• PR title follows Conventional Commits

[Copilot]

Pull request overview

This PR fixes internal fitting/prediction issues in the ensemble forecast combiners so they correctly leverage held-out validation predictions and behave robustly with ensembles containing more than two base forecasters.

Changes:

• Update WeightsCombiner.fit() to train the “best-forecaster” classifier on data_val when provided, rather than always using in-sample training predictions.
• Generalize WeightsCombiner’s degenerate-label handling for N-forecaster ensembles and ensure the dummy fallback uses the true majority winner.
• Fix StackingCombiner.fit() to pass validation data through to the meta-forecaster (for early stopping / evaluation).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
packages/openstef-meta/tests/unit/models/forecast_combiners/test_learned_weights_combiner.py	Adds regression tests for N-forecaster edge cases, dummy fallback behavior, and data_val training.
packages/openstef-meta/src/openstef_meta/models/forecast_combiners/stacking_combiner.py	Passes prepared validation data to meta-forecaster during stacking fit.
packages/openstef-meta/src/openstef_meta/models/forecast_combiners/learned_weights_combiner.py	Trains learned weights on validation predictions when available and improves N-forecaster robustness.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.