Skip to content

fix(weather): wire forecast cache + weight-aware throttle + variable trim#66

Closed
zax0rz wants to merge 2 commits into
mostlyrightmd:mainfrom
zax0rz:fix/64-open-meteo-rate-limiting-v2
Closed

fix(weather): wire forecast cache + weight-aware throttle + variable trim#66
zax0rz wants to merge 2 commits into
mostlyrightmd:mainfrom
zax0rz:fix/64-open-meteo-rate-limiting-v2

Conversation

@zax0rz

@zax0rz zax0rz commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #64. Three compounding issues cause Open-Meteo free-tier rate-limit exhaustion during backtests: uncached forecasts re-fetched on every research() call, a politeness throttle blind to weighted call costs, and over-fetching 18 variables when only 3 are consumed by the pairs join.

This PR wires the existing Phase 20 forecast cache into _fetch_open_meteo_range (research.py), adds weight-aware client-side date chunking for Previous Runs requests exceeding 14 days, and trims the variable set on the research() path to the 3 columns the pairs join actually uses.

Test plan

  • test_open_meteo_cache_wiring.py (247 lines) — cache hit, cache miss, partial cache, stale cache, NaT-to-None cleanup before Parquet, duplicate-row guard on non-contiguous cache miss
  • test_open_meteo_variables_param.py (152 lines) — variable subset validation, unknown variable rejection, endpoint _previous_day1 suffix handling with subset, full-set passthrough
  • test_open_meteo_window_chunking.py (143 lines) — single-chunk passthrough, multi-chunk boundaries, weight-aware delay calculation, Single Runs bypasses chunking
  • All upstream tests pass (ruff + ruff-format + pytest pre-push clean)
  • Full suite confirmation in upstream CI

Cross-SDK Sync (see .planning/CROSS-SDK-SYNC.md)

python_only: true — all changes are in the Python weather fetcher and core research module; no parity-trigger paths touched

@helloiamvu

Copy link
Copy Markdown
Member

Codex review (gpt-5.5, medium reasoning) — 1 blocking finding (P2):

  • [P2] Partial-month forecast cache treated as completepackages/core/src/mostlyright/research.py:1431-1432. When the first request for a month covers only a subrange (e.g. 2024-06-01..2024-06-02), only that subrange is written to the monthly partition, but a later request for a different June window sees hit is not None and skips the network fetch — silently dropping Open-Meteo forecasts for the uncached days in that month. Needs coverage metadata, or the writer must populate full-month partitions before the read path treats a partition as complete.

CI is green and the diff is solid otherwise; this is the one correctness issue to resolve before merge. (Reviewed as part of the 2026-06-06 autonomous triage run — this is feature-scope, recommend landing in a 1.6.0 minor rather than the v1.5.3 patch.)

…mplete

Addresses P2 review finding on PR mostlyrightmd#66. A cache hit for June 1-2 would
silently satisfy a June 15-20 request with zero matching rows.

Changes:
- cache.py: read_forecast_cache accepts need_from/need_to params,
  returns None if stored range doesn't cover requested range or if
  range metadata is absent (backwards-compat path)
- cache.py: write_forecast_cache accepts from_date/to_date, embeds
  ISO-8601 range as parquet table metadata
- research.py: passes full month boundaries as need_from/need_to;
  missing months are fetched in full for accurate partition metadata
- 4 new tests: partial coverage re-fetch, full-month hit skip,
  backwards-compat miss, current month skip preserved

All 2947 tests passing.
helloiamvu added a commit that referenced this pull request Jun 6, 2026
helloiamvu added a commit that referenced this pull request Jun 6, 2026
, codex P2)

Codex P2 on #66: _fetch_open_meteo_range clamped the fetch span to the
request's [start, end], so a subrange request wrote an incomplete monthly
forecast partition; a later same-month window then read it as a hit and
silently dropped the uncached days. Fetch on full-month boundaries (clamped
only to today to avoid future dates) so every written elapsed-month partition
is complete. Current UTC month is never read-served nor written, so its
partialness stays harmless. +1 regression test.
helloiamvu added a commit that referenced this pull request Jun 6, 2026
…-join fix (#67) + docs (#52)

Minor release (features added -> 1.6.0, not a patch). Bump all three
packages 1.5.3 -> 1.6.0; regenerate uv.lock (codex P2 on #71: lock was
stale at 1.5.2/1.5.3). CHANGELOG: fold the never-published 1.5.3 entry into
1.6.0 and add the #63 NWP-fields and #64 OM-cache/throttle feature entries.

Bundles PRs #67/#52 (already in merged-vision) + #66/#64 + #68/#63, each
with its codex P2 resolved on this integration branch.
@helloiamvu

Copy link
Copy Markdown
Member

Incorporated into the v1.6.0 release via the integration branch merged-visionmain (#71, merged as 4c0bd19). The branch content was merged plus a follow-up fix for the Codex P2 (full-month cache partitions) and three further P2s found during integration review (chunk-concat attrs, Single-Runs delay scaling, seamless chunking, weather>=1.6.0 floor). Published to PyPI as mostlyrightmd/-weather/-markets 1.6.0. Thanks @zax0rz!

@helloiamvu helloiamvu closed this Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(weather): Open-Meteo forecast path uncached + weight-blind throttle → users hit free-tier rate limits in backtests

2 participants