feat: add precipitation forecast backtest tool for Adlershof, Berlin

[kaihowl]

Summary

Adds a standalone Python CLI tool (scripts/precip_backtest.py) that backtests precipitation forecast accuracy for Adlershof, Berlin (52.43°N, 13.53°E) across weather models and lead times, unrelated to the Rust git-perf/cli_types crates.

Type of Change

• feat: New feature
• fix: Bug fix
• docs: Documentation update
• refactor: Code refactoring (no functional changes)
• test: Test additions or improvements
• chore: Maintenance tasks (dependencies, build, etc.)
• perf: Performance improvement
• ci: CI/CD changes

Changes

• scripts/precip_backtest.py — CLI tool (run via uv run scripts/precip_backtest.py, PEP 723 inline deps: requests, pandas, matplotlib) that:
  • Fetches per-model, per-lead-time (1-7 days) forecast data from Open-Meteo's Previous Runs API for icon_seamless, gfs_seamless, ecmwf_ifs025, meteofrance_arome_france, and ukmo_seamless.
  • Fetches ERA5 reanalysis daily precipitation from Open-Meteo's Historical Weather API as ground truth.
  • Caches all fetched data in a local SQLite database, so re-running only fetches missing days — cheap to run incrementally (e.g. daily via cron).
  • Computes MAE, bias, CSI, and ETS (at >0.1mm / >1mm / >5mm thresholds) per model × lead time, written to metrics.csv.
  • Renders a skill-degradation line chart (skill_degradation.png, MAE and CSI(>1mm) vs. lead time, one line per model).
• scripts/precip_backtest_README.md — usage notes.
• .gitignore — ignore the tool's default cache DB and output directory.

Related Issues

N/A — ad hoc tooling request, not tied to a tracked issue.

Testing

• Manual testing performed (see below)
• All tests pass: cargo nextest run -- --skip slow — N/A, no Rust code changed
• Mutation testing passed — N/A, this PR only adds a standalone Python script; no Rust code in cli_types/git_perf was touched

Manual test results:

Ran the script end-to-end against the live Open-Meteo APIs for a small date range (2025-06-01..2025-06-15, two models, three lead times):

• Verified the SQLite cache is created and populated correctly.
• Verified a second run with an extended --end-date only fetched the new days (confirmed via log output), leaving previously cached days untouched.
• Verified --skip-fetch recomputes metrics.csv and skill_degradation.png from the cache with no network access.
• Verified metrics.csv contents and inspected the generated chart visually.
• Linted with ruff check / formatted with ruff format (clean).

Documentation

• Updated relevant documentation (scripts/precip_backtest_README.md)
• Regenerated manpages — N/A, no CLI changes to the Rust git-perf binary
• Added code comments for complex logic — kept to a couple of comments explaining non-obvious cache/incremental-fetch behavior
• No documentation changes needed

Pre-Submission Checklist

• Code formatted with cargo fmt — N/A, Python only; formatted with ruff format instead
• Linting passes: cargo clippy — N/A, Python only; ruff check passes instead
• All tests pass: cargo nextest run -- --skip slow — N/A, no Rust code changed
• Mutation testing passes — N/A, no Rust code changed
• Commit messages follow Conventional Commits
• PR title follows Conventional Commits format
• Branch is up to date with base branch
• No merge conflicts

Additional Context

Out of scope per the original request: UI/dashboard (CLI + CSV/PNG output only).

---

Generated by Claude Code

[github-actions]

Performance Report

⏱ Performance Results

Audit Results

Auditing measurement "add-benchmark" (os=ubuntu-22.04/rust=beta):
  ⚠️  WARNING: Change points detected in current epoch for 'add-benchmark':
     commit 6e3baa3 (-35.5%)
     commit ee82970 (+63.3%)
     commit 5533710 (-39.0%)
     commit 8ef210c (+53.1%)
     commit 3541c6a (-38.1%)
     commit dc90cee (+47.8%)
     Historical z-score comparison may be unreliable due to regime shift.
     Consider bumping epoch or investigating the change.
  ✅ 'add-benchmark'
  Aggregation: min
  z-score (mad): ↑ 1.54
  Head: μ: 24.8ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 22.2ms σ: 3.9ms MAD: 1.6ms n: 33
   [-38.48% – +9.45%] ▅▇█▇▇█▇▇█▆██▇█▇▂▂█▂▇▇█▆▇█▆▁▁▁▇▆▄▁▇

Auditing measurement "add-benchmark" (os=ubuntu-22.04/rust=stable):
  ⚠️  WARNING: Change points detected in current epoch for 'add-benchmark':
     commit 4493382 (+45.7%)
     commit b194e1c (-63.3%)
     commit c2dc7a7 (+187.3%)
     Historical z-score comparison may be unreliable due to regime shift.
     Consider bumping epoch or investigating the change.
  ✅ 'add-benchmark'
  Aggregation: min
  z-score (mad): ↑ 7.18
  Head: μ: 26.2ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 22.1ms σ: 4.3ms MAD: 565.8µs n: 30
   [-65.09% – +8.70%] ▇▇▇▇▇▄▇▇▄█▇▁▇█▇▇█▇▇▆▇█▄▇▄▇▇▇▃▆█
  Note: Passed due to relative deviation (8.7%) being below threshold (10.0%)

Auditing measurement "bench::add_measurements/add_measurement/50::median" (os=ubuntu-22.04/rust=beta):
  ✅ 'bench::add_measurements/add_measurement/50::median'
  Aggregation: min
  z-score (mad): ↑ 12.55
  Head: μ: 889.4ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 869ms σ: 33.5ms MAD: 1.6ms n: 5
   [-4.00% – +6.75%] ▁▄█▄▃▅
  Note: Passed due to relative deviation (2.9%) being below threshold (12.0%)

Auditing measurement "bench::add_measurements/add_measurement/50::median" (os=ubuntu-22.04/rust=stable):
  ✅ 'bench::add_measurements/add_measurement/50::median'
  Aggregation: min
  z-score (mad): ↑ 37.54
  Head: μ: 934.3ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 874.1ms σ: 7.7ms MAD: 1.6ms n: 5
   [-1.76% – +6.34%] ▃▃▃▂▁█
  Note: Passed due to relative deviation (6.3%) being below threshold (12.0%)

Auditing measurement "bench::report/report_generation/10::median" (os=ubuntu-22.04/rust=beta):
  ✅ 'bench::report/report_generation/10::median'
  Aggregation: median
  z-score (mad): ↑ 0.09
  Head: μ: 22.4ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 22.3ms σ: 898.5µs MAD: 682.7µs n: 12
   [-4.64% – +7.35%] ▁▃▆▃▃█▃▄█▃▇▇▅

Auditing measurement "bench::report/report_generation/10::median" (os=ubuntu-22.04/rust=stable):
  ✅ 'bench::report/report_generation/10::median'
  Aggregation: median
  z-score (mad): ↑ 8.01
  Head: μ: 23.9ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 19.7ms σ: 5ms MAD: 518.1µs n: 12
   [-67.14% – +8.51%] ▇▇▇▇▇▄▇█▄█▇▁█
  Note: Passed due to relative deviation (8.5%) being below threshold (10.0%)

Auditing measurement "release-binary-size" (os=ubuntu-22.04/rust=stable):
  ✅ 'release-binary-size'
  Aggregation: min
  z-score (stddev): ↑ 0.93
  Head: μ: 8.3MB σ: N/A MAD: 0B n: 1
  Tail: μ: 8.3MB σ: 61.3kB MAD: 28.9kB n: 36
   [-2.81% – +0.40%] █████████▇▇▇▇▇▇▇▇▇▇▇▇▇▇▆▆▆▆▅▅▅▅▅▅▅▁▂█

Auditing measurement "report" (os=ubuntu-22.04/rust=beta):
  ⚠️  WARNING: Change points detected in current epoch for 'report':
     commit 6e3baa3 (-35.6%)
     commit ee82970 (+61.2%)
     commit 5533710 (-37.7%)
     commit 8ef210c (+53.8%)
     commit dc90cee (+46.5%)
     Historical z-score comparison may be unreliable due to regime shift.
     Consider bumping epoch or investigating the change.
  ✅ 'report'
  Aggregation: min
  z-score (stddev): ↑ 0.61
  Head: μ: 28.8ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 26.1ms σ: 4.3ms MAD: 1.9ms n: 33
   [-34.31% – +11.60%] ▅▆▇▆▆█▆▇█▆█▇▆▇▆▁▁▇▁▆▇▇▇▆▇▆▁▁▁▇▆▅▁▇

Auditing measurement "report" (os=ubuntu-22.04/rust=stable):
  ⚠️  WARNING: Change points detected in current epoch for 'report':
     commit b194e1c (-63.8%)
     commit c2dc7a7 (+189.9%)
     Historical z-score comparison may be unreliable due to regime shift.
     Consider bumping epoch or investigating the change.
  ✅ 'report'
  Aggregation: min
  z-score (stddev): ↑ 0.86
  Head: μ: 30.1ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 25.9ms σ: 4.9ms MAD: 976.7µs n: 30
   [-65.19% – +8.23%] ▇▇▇▇▇▄█▇▄█▇▁▇█▇██▇▇▆▇█▄▇▄█▇█▄▆█

Auditing measurement "report-benchmark" (os=ubuntu-22.04/rust=beta):
  ⚠️  WARNING: Change points detected in current epoch for 'report-benchmark':
     commit 6e3baa3 (-34.8%)
     commit ee82970 (+59.6%)
     commit 5533710 (-37.9%)
     commit 8ef210c (+53.9%)
     commit dc90cee (+46.6%)
     Historical z-score comparison may be unreliable due to regime shift.
     Consider bumping epoch or investigating the change.
  ✅ 'report-benchmark'
  Aggregation: min
  z-score (stddev): ↑ 0.52
  Head: μ: 28.4ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 26.2ms σ: 4.2ms MAD: 1.8ms n: 33
   [-33.85% – +10.91%] ▅▆▇▆▇█▆▇█▆▇▇▆▇▆▁▁▇▁▆▇▇▆▆▇▆▁▁▁▇▆▅▁▇

Auditing measurement "report-benchmark" (os=ubuntu-22.04/rust=stable):
  ⚠️  WARNING: Change points detected in current epoch for 'report-benchmark':
     commit b194e1c (-64.1%)
     commit c2dc7a7 (+190.0%)
     Historical z-score comparison may be unreliable due to regime shift.
     Consider bumping epoch or investigating the change.
  ✅ 'report-benchmark'
  Aggregation: min
  z-score (stddev): ↑ 0.88
  Head: μ: 30.2ms σ: N/A MAD: 0ns n: 1
  Tail: μ: 25.9ms σ: 4.9ms MAD: 1ms n: 30
   [-65.26% – +9.68%] ▇▇▇▇▇▄▇▇▄█▇▁▇█▇▇█▇▇▆▇█▄▇▄█▇█▄▆█

Auditing measurement "report-size" (os=ubuntu-22.04/rust=beta):
  ✅ 'report-size'
  Aggregation: min
  z-score (stddev): →
  Head: μ: 20.3kB σ: N/A MAD: 0B n: 1
  Tail: μ: 20.3kB σ: 0B MAD: 0B n: 33
   [+0.00% – +0.00%] ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅

Auditing measurement "report-size" (os=ubuntu-22.04/rust=stable):
  ✅ 'report-size'
  Aggregation: min
  z-score (stddev): →
  Head: μ: 20.3kB σ: N/A MAD: 0B n: 1
  Tail: μ: 20.3kB σ: 0B MAD: 0B n: 30
   [+0.00% – +0.00%] ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅

Auditing measurement "report-size-benchmark" (os=ubuntu-22.04/rust=beta):
  ✅ 'report-size-benchmark'
  Aggregation: median
  z-score (stddev): →
  Head: μ: 20.3kB σ: N/A MAD: 0B n: 1
  Tail: μ: 20.3kB σ: 0B MAD: 0B n: 33
   [+0.00% – +0.00%] ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅

Auditing measurement "report-size-benchmark" (os=ubuntu-22.04/rust=stable):
  ✅ 'report-size-benchmark'
  Aggregation: median
  z-score (stddev): →
  Head: μ: 20.3kB σ: N/A MAD: 0B n: 1
  Tail: μ: 20.3kB σ: 0B MAD: 0B n: 30
   [+0.00% – +0.00%] ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅

Overall: PASSED (15/15 groups passed)

Measurement Storage Size

Live Measurement Size Report
============================

⚠️  Shallow clone detected - measurement counts may be incomplete (see FAQ)

Number of commits with measurements: 342
Total measurement data size (on-disk (compressed)): 541.7kB

Repository Statistics (for context):
-------------------------------------
  Loose objects: 0 (0B)
  Packed objects: 2975 (2.1MB)
  Total repository size: 2.1MB

Created by git-perf