Skip to content

feat: add regression coverage score metric - #1068

Merged
egordm merged 7 commits into
OpenSTEF:mainfrom
MaxPolak97:feature/1064-add-rcs-metric
Aug 25, 2026
Merged

feat: add regression coverage score metric#1068
egordm merged 7 commits into
OpenSTEF:mainfrom
MaxPolak97:feature/1064-add-rcs-metric

Conversation

@MaxPolak97

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the Regression Coverage Score (RCS) metric to openstef-beam for evaluating probabilistic forecast interval calibration.

RCS measures the fraction of observed values that fall within a predicted quantile interval, such as P10-P90. The metric is implemented as rcs(...) in the deterministic metrics module, exported through openstef_beam.metrics, and exposed through RCSProvider for evaluation pipelines. The provider follows the same symmetric quantile-pair logic as RIQDProvider.

Closes #1064

Type of change

  • Bug fix
  • New feature
  • Breaking change (see checklist below)
  • Documentation
  • Refactor / chore / CI

Breaking changes checklist

  • Public API, config schema, or serialized/pickled objects changed in a way that affects existing users
  • If yes: see the breaking changes guide for the pickle/_migrate_state migration pattern

Migration path for existing users (e.g. "old pickled XScaler objects auto-migrate on load", "users must now pass X explicitly"):

AI disclosure

  • No AI assistance was used (beyond grammar/spelling)
  • AI assistance was used — tool(s):
    • I have reviewed, understand, and can explain all AI-generated code in this PR
    • This is disclosed in a commit message (e.g. Assisted-by: <tool name>)

Checklist

  • poe all --check passes locally
  • Tests added/updated for the change
  • Documentation updated (docstrings, user guide, examples) if needed
  • Commits are signed off per our DCO (git commit -s)
  • PR title follows Conventional Commits (e.g. feat: ..., fix: ..., feat!: ... for breaking changes)

Signed-off-by: Max Polak <maxpolak97@gmail.com>
Signed-off-by: Max Polak <maxpolak97@gmail.com>
Signed-off-by: Max Polak <maxpolak97@gmail.com>
Signed-off-by: Max Polak <maxpolak97@gmail.com>
@MaxPolak97
MaxPolak97 requested a review from a team August 19, 2026 13:38
@github-actions github-actions Bot added the feature New feature or request label Aug 19, 2026
Comment thread packages/openstef-beam/src/openstef_beam/evaluation/metric_providers.py Outdated
…quantile metrics into a helper function

Signed-off-by: Max Polak <maxpolak97@gmail.com>

@egordm egordm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks great now! Thanks for your contribution.
Merging.

@lschilders lschilders left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement, LGTM!

@egordm
egordm merged commit 7e69e6d into OpenSTEF:main Aug 25, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add RCS for Probabilistic Forecast Evaluation

3 participants