diff --git a/docs/source/user_guide/concepts/beam.rst b/docs/source/user_guide/concepts/beam.rst index 761f72a3e..41422d1d2 100644 --- a/docs/source/user_guide/concepts/beam.rst +++ b/docs/source/user_guide/concepts/beam.rst @@ -139,9 +139,10 @@ three dimensions. Each dimension answers a different operational question: **Configurable metrics.** Metrics are supplied as provider objects. BEAM includes providers such as ``RMAEProvider`` (Relative Mean Absolute Error) and ``RCRPSProvider`` (Relative Continuous Ranked Probability Score) for probabilistic evaluation. Interval -metrics such as ``RCSProvider`` and -``RIQDProvider`` help assess quantile calibration and sharpness for symmetric -quantile ranges. You can implement custom providers to add domain-specific metrics. +metrics such as ``RCSProvider`` (Regression Coverage Score, measuring the fraction of +observations inside predicted quantile bounds) and ``RIQDProvider`` (Relative Inter-Quantile +Distance) help assess quantile calibration and sharpness for symmetric quantile ranges. +You can implement custom providers to add domain-specific metrics. .. code-block:: python @@ -249,4 +250,4 @@ benchmarks (hundreds of targets, multiple models) remain tractable. - :ref:`concept_models` for the forecasting models that BEAM evaluates. - :ref:`concept_metalearning` for how BEAM results inform model selection decisions. - :doc:`/user_guide/guides/backtesting_tutorial` for a hands-on walkthrough of setting up and running a backtest. - - :doc:`/api/beam` for the full openstef-beam API reference. + - :doc:`/api/beam` for the full openstef-beam API reference. \ No newline at end of file diff --git a/docs/source/user_guide/guides/probabilistic_forecasting.rst b/docs/source/user_guide/guides/probabilistic_forecasting.rst index 584abadf2..f6813511d 100644 --- a/docs/source/user_guide/guides/probabilistic_forecasting.rst +++ b/docs/source/user_guide/guides/probabilistic_forecasting.rst @@ -253,15 +253,16 @@ Calibration quality can be assessed by comparing expected vs. observed quantile Key metrics for probabilistic forecast quality include: - **Calibration error**: the difference between expected and observed coverage per quantile -- **Regression Coverage Score (RCS)**: the fraction of actual values inside a prediction interval such as P10-P90 +- **Regression Coverage Score (RCS)**: the fraction of actual values falling within a symmetric prediction interval (e.g., P10-P90). A calibrated 90% interval should yield an RCS close to 0.9. Available in BEAM evaluation pipelines via :class:`~openstef_beam.evaluation.metric_providers.RCSProvider`. - **Sharpness**: the width of prediction intervals (narrower is better, given proper calibration) - **Pinball loss**: the proper scoring rule for quantile forecasts, penalizing both miscalibration and lack of sharpness -See :doc:`/user_guide/guides/backtesting_tutorial` for how to evaluate forecast quality on historical data. +See :doc:`/user_guide/guides/backtesting` for how to evaluate forecast quality on historical data. .. seealso:: - :doc:`/user_guide/guides/forecasting` for the overall forecasting workflow (fitting, predicting, model selection). - :doc:`/user_guide/concepts/models` for understanding how different model types compare. - - :doc:`/user_guide/guides/backtesting_tutorial` for evaluating forecast performance systematically. + - :doc:`/user_guide/guides/backtesting` for evaluating forecast performance systematically. - :doc:`/user_guide/guides/reliability_fallback` for operational concerns like fallback behavior when data is missing. +``` \ No newline at end of file