fix(rss): properly detect charset and decode French and non-UTF8 full content - #1323
Open
heberjeur wants to merge 1 commit into
Open
fix(rss): properly detect charset and decode French and non-UTF8 full content#1323heberjeur wants to merge 1 commit into
heberjeur wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
Fixes #1094
When fetching and parsing full article content (e.g. from
developpez.comor other sites declaring ISO-8859-1 / Windows-1252):é,è,à,ç,ê,î,ô,ù, etc.) were displayed as corrupted replacement symbols (`` or?)."Perplexity a lanc son nouvel abonnement Max, factur 200 dollars..."Root Cause
Content-TypeHeader with Comma: Some legacy/PHP servers returnContent-Type: text/html, charset=iso-8859-1with a comma rather than a semicolon. OkHttp'sMediaType.parse()fails on this format and returnsnullforresponseBody.contentType()?.charset().<meta charset="...">Detection: Onlymeta[http-equiv=content-type]was inspected, completely ignoring HTML5<meta charset="iso-8859-1">.it.peek().readString(Charsets.UTF_8)converted raw ISO-8859-1 bytes (such as0xE9foré) directly into\uFFFD, causing irreversible character loss if meta charset wasn't found in time.Fixes Applied
detectHtmlCharset) inRssHelper.kt:Content-Typeheader (handling both,and;delimiters as well as quotes).<head>for<meta charset="...">(HTML5) and<meta http-equiv="content-type" content="...">.UTF-8.parseFullContent(): Reads the raw response bytes and decodes them directly with the detected charset, preventing lossy intermediate UTF-8 conversions.toHttpContentType(): Normalizes commas to semicolons when reading feed XML content types.RssHelperTest.kt): Added comprehensive tests verifying charset detection from comma-separated headers, HTML5 meta tags, and accurate decoding of French accented text.Tested
./gradlew testGithubReleaseUnitTest->BUILD SUCCESSFUL../gradlew assembleGithubRelease->BUILD SUCCESSFUL.