Skip to content

Harden RTSW NOAA client for WAF 202 and empty bodies - #8

Merged
Demosfen merged 5 commits into
mainfrom
fix/rtsw-import-plasma-gaps
Aug 11, 2026
Merged

Harden RTSW NOAA client for WAF 202 and empty bodies#8
Demosfen merged 5 commits into
mainfrom
fix/rtsw-import-plasma-gaps

Conversation

@Demosfen

@Demosfen Demosfen commented Aug 10, 2026

Copy link
Copy Markdown
Member

Summary

  • Harden NOAA HTTP handling via EnsureNoaaSuccessAsync: treat AWS WAF 202 + x-amzn-waf-action: challenge and empty bodies (including chunked without Content-Length) as failures — applied to all NOAA clients (Rtsw, Ace, KpIndex, WsaEnlil).
  • Sanitize bare/quoted NaN/Infinity in RTSW JSON before deserialization (lazy allocation when payload is clean).
  • Version: 1.2.2 (patch; WAF/empty-body hardening + NaN fix + review refactor).
  • Bump AuroraScienceHub.Framework.Http / Utilities 10.0.5 → 10.0.7, and Microsoft.Extensions.* pins 10.0.7 → 10.0.10 (required by Framework 10.0.7).
  • Dual-map AuroraScienceHub.* to nuget.org in NuGet.Config so stable Framework packages restore from nuget.org.
  • CHANGELOG entry for [1.2.2] - 2026-08-11.

Context

Companion to SWeather-core fix/rtsw-import-plasma-gaps / SWeather-core#199. Without WAF/empty-body handling, challenges look like a successful empty tick. Without NaN sanitization, bare NaN in rtsw_wind_1m.json crashes JSON deserialization.

Test plan

  • RtswClientTests, NoaaJsonSanitizerTests
  • Local dotnet restore / dotnet build with Framework 10.0.7
  • CI green on this PR

Александр Николаев and others added 2 commits August 10, 2026 19:23
Treat AWS WAF challenge and empty JSON responses as failures so resilience can retry instead of returning an empty feed.

Co-authored-by: Cursor <cursoragent@cursor.com>
NOAA emits bare NaN in rtsw_wind_1m.json which breaks System.Text.Json and stops the import job. Replace non-standard literals with null in RtswClient and bump NoaaClient to 1.2.2.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Demosfen Demosfen self-assigned this Aug 11, 2026
@Demosfen
Demosfen requested a review from alex1ozr August 11, 2026 06:57
@alex1ozr
alex1ozr requested a lite review from Copilot August 11, 2026 06:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the NOAA RTSW client’s HTTP handling so AWS WAF “challenge” responses and successful-but-empty bodies are treated as failures (enabling retries), and introduces a JSON sanitizer to replace non-standard NaN/Infinity literals with null before deserialization.

Changes:

  • Add NoaaJsonSanitizer to rewrite non-standard NOAA numeric literals (NaN, Infinity, -Infinity) to null.
  • Update RtswClient to detect WAF 202 challenge responses and empty-body success responses and throw HttpRequestException.
  • Add unit tests covering WAF/empty-body failure paths and NaN sanitization behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/UnitTests/NoaaClient/Utilities/NoaaJsonSanitizerTests.cs Adds focused unit coverage for sanitizer replacement rules and no-op behavior.
tests/UnitTests/NoaaClient/Rtsw/RtswClientTests.cs Adds tests for WAF 202 challenge handling, empty-body handling, and NaN deserialization.
src/NoaaClient/Utilities/NoaaJsonSanitizer.cs Introduces byte-level sanitizer for non-RFC JSON numeric literals.
src/NoaaClient/Rtsw/RtswClient.cs Switches to manual HTTP handling + sanitation, adds WAF/empty-body failure logic and logging.
src/NoaaClient/NoaaClient.csproj Adds logging abstractions dependency for ILogger<RtswClient>.
Directory.Packages.props Pins Microsoft.Extensions.Logging.Abstractions version for central package management.
Directory.Build.props Updates package base version (currently mismatched vs PR description).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/NoaaClient/Rtsw/RtswClient.cs
Comment thread Directory.Build.props
Comment thread src/NoaaClient/Utilities/NoaaJsonSanitizer.cs
Comment thread src/NoaaClient/Rtsw/RtswClient.cs Outdated
Comment thread src/NoaaClient/Rtsw/RtswClient.cs Outdated
Comment thread src/NoaaClient/Utilities/NoaaJsonSanitizer.cs
…r, empty body check.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Demosfen
Demosfen requested a review from alex1ozr August 11, 2026 14:52
Co-authored-by: Cursor <cursoragent@cursor.com>
@alex1ozr
alex1ozr requested a lite review from Copilot August 11, 2026 14:56
…erelease restore.

Dual packageSourceMapping lets exact stable pins (e.g. Framework 10.0.7) resolve from nuget.org while prereleases remain available from GH Packages. Also bump Microsoft.Extensions.* to 10.0.10 for Framework 10.0.7.

Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (5)

src/NoaaClient/Ace/AceClient.cs:42

  • This method will treat an empty-body HTTP 2xx response as a successful call and pass an empty string into the parser. For the hardening described in this PR, empty bodies should throw a HttpRequestException (via ThrowIfEmptyBody) to avoid silently returning incorrect “empty” data.
        var url = new Uri(_baseUrl, "text/ace-swepam.txt");
        using var response = await _httpClient.GetAsync(url, cancellationToken).ConfigureAwait(false);
        await response.EnsureNoaaSuccessAsync(url, cancellationToken).ConfigureAwait(false);
        var text = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);

        return SolarWindPlasmaDataParser.Parse(text);
    }

src/NoaaClient/WsaEnlil/WsaEnlilClient.cs:106

  • FetchManifestAsync treats an HTTP 200/OK with an empty body as a non-error (returns null), which then becomes an empty MemoryStream result. This contradicts the PR goal of treating empty bodies as failures (to trigger retries instead of silently producing an empty animation). Use ThrowIfEmptyBody on the downloaded bytes and let the exception propagate.
        var rawBytes = await response.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false);
        if (rawBytes.Length == 0)
        {
            return null;
        }

        return JsonSerializer.Deserialize<IReadOnlyCollection<WsaEnlilManifestEntry>>(rawBytes, s_jsonOptions);

src/NoaaClient/KpIndex/KpIndexClient.cs:48

  • GetStringOrDefaultAsync currently returns an empty string on an HTTP 2xx response with an empty body, and the callers convert that into an empty result set. For the hardening described in this PR, empty bodies should be treated as failures (including chunked responses without Content-Length) so higher-level retry logic can kick in.
        using var response = await _client.GetAsync(url, cancellationToken).ConfigureAwait(false);
        await response.EnsureNoaaSuccessAsync(url, cancellationToken).ConfigureAwait(false);

        return await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);

CHANGELOG.md:16

  • This changelog entry claims empty-body failures are handled “via EnsureNoaaSuccessAsync”, but EnsureNoaaSuccessAsync currently only checks for WAF challenge + success status and does not validate body emptiness. Either update the changelog wording, or move the empty-body behavior into the helper API (or consistently call ThrowIfEmptyBody in all clients).
#### NoaaClient Package
- Treat AWS WAF `202` challenges (`x-amzn-waf-action: challenge`) and empty response bodies as failures via `EnsureNoaaSuccessAsync` for all NOAA clients (`Rtsw`, `Ace`, `KpIndex`, `WsaEnlil`)
- Sanitize bare/quoted `NaN`/`Infinity` literals in RTSW JSON before deserialization (lazy allocation when the payload is clean)

src/NoaaClient/Ace/AceClient.cs:31

  • This method will happily parse an HTTP 2xx response with an empty body, which can mask NOAA/WAF edge cases as “valid but empty” data. To match the PR’s empty-body hardening intent, read the payload bytes and call ThrowIfEmptyBody before parsing.

This issue also appears on line 36 of the same file.

        var url = new Uri(_baseUrl, "text/ace-magnetometer.txt");
        using var response = await _httpClient.GetAsync(url, cancellationToken).ConfigureAwait(false);
        await response.EnsureNoaaSuccessAsync(url, cancellationToken).ConfigureAwait(false);
        var text = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);

        return MagnetometerDataParser.Parse(text);

@Demosfen
Demosfen requested a review from alex1ozr August 11, 2026 15:20
@Demosfen
Demosfen merged commit ce6d15c into main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants