Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions docs/source/user_guide/concepts/beam.rst
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,10 @@ three dimensions. Each dimension answers a different operational question:
**Configurable metrics.** Metrics are supplied as provider objects. BEAM includes
providers such as ``RMAEProvider`` (Relative Mean Absolute Error) and ``RCRPSProvider``
(Relative Continuous Ranked Probability Score) for probabilistic evaluation. Interval
metrics such as ``RCSProvider`` and
``RIQDProvider`` help assess quantile calibration and sharpness for symmetric
quantile ranges. You can implement custom providers to add domain-specific metrics.
metrics such as ``RCSProvider`` (Regression Coverage Score, measuring the fraction of
observations inside predicted quantile bounds) and ``RIQDProvider`` (Relative Inter-Quantile
Distance) help assess quantile calibration and sharpness for symmetric quantile ranges.
You can implement custom providers to add domain-specific metrics.

.. code-block:: python

Expand Down Expand Up @@ -249,4 +250,4 @@ benchmarks (hundreds of targets, multiple models) remain tractable.
- :ref:`concept_models` for the forecasting models that BEAM evaluates.
- :ref:`concept_metalearning` for how BEAM results inform model selection decisions.
- :doc:`/user_guide/guides/backtesting_tutorial` for a hands-on walkthrough of setting up and running a backtest.
- :doc:`/api/beam` for the full openstef-beam API reference.
- :doc:`/api/beam` for the full openstef-beam API reference.
7 changes: 4 additions & 3 deletions docs/source/user_guide/guides/probabilistic_forecasting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -253,15 +253,16 @@ Calibration quality can be assessed by comparing expected vs. observed quantile
Key metrics for probabilistic forecast quality include:

- **Calibration error**: the difference between expected and observed coverage per quantile
- **Regression Coverage Score (RCS)**: the fraction of actual values inside a prediction interval such as P10-P90
- **Regression Coverage Score (RCS)**: the fraction of actual values falling within a symmetric prediction interval (e.g., P10-P90). A calibrated 90% interval should yield an RCS close to 0.9. Available in BEAM evaluation pipelines via :class:`~openstef_beam.evaluation.metric_providers.RCSProvider`.
- **Sharpness**: the width of prediction intervals (narrower is better, given proper calibration)
- **Pinball loss**: the proper scoring rule for quantile forecasts, penalizing both miscalibration and lack of sharpness

See :doc:`/user_guide/guides/backtesting_tutorial` for how to evaluate forecast quality on historical data.
See :doc:`/user_guide/guides/backtesting` for how to evaluate forecast quality on historical data.

.. seealso::

- :doc:`/user_guide/guides/forecasting` for the overall forecasting workflow (fitting, predicting, model selection).
- :doc:`/user_guide/concepts/models` for understanding how different model types compare.
- :doc:`/user_guide/guides/backtesting_tutorial` for evaluating forecast performance systematically.
- :doc:`/user_guide/guides/backtesting` for evaluating forecast performance systematically.
- :doc:`/user_guide/guides/reliability_fallback` for operational concerns like fallback behavior when data is missing.
```
Loading