Skip to content

fix(rss): bypass WAF bot blocking with browser User-Agent and fix search feed hang - #1322

Open
heberjeur wants to merge 1 commit into
ReadYouApp:mainfrom
heberjeur:fix/issue-1164-feed-import
Open

fix(rss): bypass WAF bot blocking with browser User-Agent and fix search feed hang#1322
heberjeur wants to merge 1 commit into
ReadYouApp:mainfrom
heberjeur:fix/issue-1164-feed-import

Conversation

@heberjeur

Copy link
Copy Markdown

What & Why

Fixes #1164
Related to #13

When subscribing to feeds protected by Cloudflare WAF / Bot Fight Mode (such as https://www.phoronix.com/rss.php), the request was blocked with HTTP 403 Forbidden and a Cloudflare HTML challenge page, throwing:
Invalid XML: Error on line 1: At line 1, column 766: not well-formed (invalid token)

Additionally:

  1. OkHttp's BridgeInterceptor was injecting its default User-Agent: okhttp/..., which was not replaced when checking header("User-Agent") == null.
  2. BestIconFinder was downloading up to 15 icon candidate variants sequentially in cleartext http://, causing the subscribe search dialog to hang in the "Searching..." state for 30+ seconds.
  3. In AbstractRssRepository.kt, searchedFeed.title.decodeHTML()!! had a potential NPE risk.
  4. In SubscribeViewModel.kt, searchFeed aborted if groupsFlow was initially empty.

Fixes Applied

  • Browser User-Agent (OkHttpClientModule.kt): Set standard Android browser User-Agent (Mozilla/5.0 ...) at both application and network interceptor levels, filtering out OkHttp's internal default User-Agent.
  • Fast & Safe Icon Finder (BestIconFinder.kt): Default to https://, limit icon probing to a max of 4 candidates, and wrap all icon fetching in runCatching so icon resolution never blocks or fails feed addition.
  • Robust RSS Search & Sync (RssHelper.kt): Check response.commonIsSuccessful before attempting to parse bodies as XML, logging clear HTTP error codes rather than passing HTML error pages into ROME's XmlReader.
  • Safe Subscribe & Null-Safety (AbstractRssRepository.kt & SubscribeViewModel.kt): Safe fallback for feed title decoding and empty group handling.
  • Unit Tests: Added unit and real network integration tests in OkHttpClientModuleTest and RssHelperTest.

Tested

  • Tested feed search with direct feed URL (https://www.phoronix.com/rss.php) and website URL (https://www.phoronix.com).
  • Ran unit tests: ./gradlew testGithubReleaseUnitTest -> All passed.
  • Built release APK: ./gradlew assembleGithubRelease -> BUILD SUCCESSFUL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can't import rss feed from phoronix

1 participant