Harden RTSW NOAA client for WAF 202 and empty bodies - #8
Conversation
Treat AWS WAF challenge and empty JSON responses as failures so resilience can retry instead of returning an empty feed. Co-authored-by: Cursor <cursoragent@cursor.com>
NOAA emits bare NaN in rtsw_wind_1m.json which breaks System.Text.Json and stops the import job. Replace non-standard literals with null in RtswClient and bump NoaaClient to 1.2.2. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR hardens the NOAA RTSW client’s HTTP handling so AWS WAF “challenge” responses and successful-but-empty bodies are treated as failures (enabling retries), and introduces a JSON sanitizer to replace non-standard NaN/Infinity literals with null before deserialization.
Changes:
- Add
NoaaJsonSanitizerto rewrite non-standard NOAA numeric literals (NaN,Infinity,-Infinity) tonull. - Update
RtswClientto detect WAF202challenge responses and empty-body success responses and throwHttpRequestException. - Add unit tests covering WAF/empty-body failure paths and NaN sanitization behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/UnitTests/NoaaClient/Utilities/NoaaJsonSanitizerTests.cs | Adds focused unit coverage for sanitizer replacement rules and no-op behavior. |
| tests/UnitTests/NoaaClient/Rtsw/RtswClientTests.cs | Adds tests for WAF 202 challenge handling, empty-body handling, and NaN deserialization. |
| src/NoaaClient/Utilities/NoaaJsonSanitizer.cs | Introduces byte-level sanitizer for non-RFC JSON numeric literals. |
| src/NoaaClient/Rtsw/RtswClient.cs | Switches to manual HTTP handling + sanitation, adds WAF/empty-body failure logic and logging. |
| src/NoaaClient/NoaaClient.csproj | Adds logging abstractions dependency for ILogger<RtswClient>. |
| Directory.Packages.props | Pins Microsoft.Extensions.Logging.Abstractions version for central package management. |
| Directory.Build.props | Updates package base version (currently mismatched vs PR description). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…r, empty body check. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…erelease restore. Dual packageSourceMapping lets exact stable pins (e.g. Framework 10.0.7) resolve from nuget.org while prereleases remain available from GH Packages. Also bump Microsoft.Extensions.* to 10.0.10 for Framework 10.0.7. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (5)
src/NoaaClient/Ace/AceClient.cs:42
- This method will treat an empty-body HTTP 2xx response as a successful call and pass an empty string into the parser. For the hardening described in this PR, empty bodies should throw a
HttpRequestException(viaThrowIfEmptyBody) to avoid silently returning incorrect “empty” data.
var url = new Uri(_baseUrl, "text/ace-swepam.txt");
using var response = await _httpClient.GetAsync(url, cancellationToken).ConfigureAwait(false);
await response.EnsureNoaaSuccessAsync(url, cancellationToken).ConfigureAwait(false);
var text = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
return SolarWindPlasmaDataParser.Parse(text);
}
src/NoaaClient/WsaEnlil/WsaEnlilClient.cs:106
FetchManifestAsynctreats an HTTP 200/OK with an empty body as a non-error (returns null), which then becomes an emptyMemoryStreamresult. This contradicts the PR goal of treating empty bodies as failures (to trigger retries instead of silently producing an empty animation). UseThrowIfEmptyBodyon the downloaded bytes and let the exception propagate.
var rawBytes = await response.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false);
if (rawBytes.Length == 0)
{
return null;
}
return JsonSerializer.Deserialize<IReadOnlyCollection<WsaEnlilManifestEntry>>(rawBytes, s_jsonOptions);
src/NoaaClient/KpIndex/KpIndexClient.cs:48
GetStringOrDefaultAsynccurrently returns an empty string on an HTTP 2xx response with an empty body, and the callers convert that into an empty result set. For the hardening described in this PR, empty bodies should be treated as failures (including chunked responses withoutContent-Length) so higher-level retry logic can kick in.
using var response = await _client.GetAsync(url, cancellationToken).ConfigureAwait(false);
await response.EnsureNoaaSuccessAsync(url, cancellationToken).ConfigureAwait(false);
return await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
CHANGELOG.md:16
- This changelog entry claims empty-body failures are handled “via
EnsureNoaaSuccessAsync”, butEnsureNoaaSuccessAsynccurrently only checks for WAF challenge + success status and does not validate body emptiness. Either update the changelog wording, or move the empty-body behavior into the helper API (or consistently callThrowIfEmptyBodyin all clients).
#### NoaaClient Package
- Treat AWS WAF `202` challenges (`x-amzn-waf-action: challenge`) and empty response bodies as failures via `EnsureNoaaSuccessAsync` for all NOAA clients (`Rtsw`, `Ace`, `KpIndex`, `WsaEnlil`)
- Sanitize bare/quoted `NaN`/`Infinity` literals in RTSW JSON before deserialization (lazy allocation when the payload is clean)
src/NoaaClient/Ace/AceClient.cs:31
- This method will happily parse an HTTP 2xx response with an empty body, which can mask NOAA/WAF edge cases as “valid but empty” data. To match the PR’s empty-body hardening intent, read the payload bytes and call
ThrowIfEmptyBodybefore parsing.
This issue also appears on line 36 of the same file.
var url = new Uri(_baseUrl, "text/ace-magnetometer.txt");
using var response = await _httpClient.GetAsync(url, cancellationToken).ConfigureAwait(false);
await response.EnsureNoaaSuccessAsync(url, cancellationToken).ConfigureAwait(false);
var text = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
return MagnetometerDataParser.Parse(text);
Summary
EnsureNoaaSuccessAsync: treat AWS WAF202+x-amzn-waf-action: challengeand empty bodies (including chunked withoutContent-Length) as failures — applied to all NOAA clients (Rtsw,Ace,KpIndex,WsaEnlil).NaN/Infinityin RTSW JSON before deserialization (lazy allocation when payload is clean).AuroraScienceHub.Framework.Http/Utilities10.0.5 → 10.0.7, andMicrosoft.Extensions.*pins 10.0.7 → 10.0.10 (required by Framework 10.0.7).AuroraScienceHub.*to nuget.org inNuGet.Configso stable Framework packages restore from nuget.org.[1.2.2] - 2026-08-11.Context
Companion to SWeather-core
fix/rtsw-import-plasma-gaps/ SWeather-core#199. Without WAF/empty-body handling, challenges look like a successful empty tick. Without NaN sanitization, bareNaNinrtsw_wind_1m.jsoncrashes JSON deserialization.Test plan
RtswClientTests,NoaaJsonSanitizerTestsdotnet restore/dotnet buildwith Framework 10.0.7