webhttrack passes the engine form-charset argv instead of UTF-8#635
Merged
Conversation
The httrack CLI and the WinHTTrack GUI both hand the engine UTF-8 argv, the contract htsname and htscache now assume. webhttrack did not: its back_launch_cmd splits the raw HTTP POST body into argv and calls hts_main2 directly, and that body is in the form's declared LANGUAGE_CHARSET (ISO-8859-1, windows-125x, BIG5, ... per language), never UTF-8. A non-ASCII output path or URL from the web UI reached the engine as raw form-charset bytes and was then decoded as UTF-8 downstream, landing the mirror in a mojibake directory. Convert the command line from the current LANGUAGE_CHARSET to UTF-8 before launching, at the one point where that charset is known (LANGSEL, right before webhttrack_main). ASCII and already-UTF-8 forms pass through unchanged. webhttrack's own argv also gets the Windows hts_argv_utf8 treatment the CLI has. This exports hts_convertStringToUTF8 (an additive ABI change: new symbol, soname unchanged), alongside the already-exported hts_convertStringSystemToUTF8, so the server binary can reach it. Closes #629 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Resolve tests/Makefile.am: keep 66_engine-port80-strip (#632) alongside this branch's test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Drop the origin server: only the decoded -O directory name is under test, and htsserver creates it from the POST'd path before any fetch, so a dead crawl target (port 1) exercises the same #629 path without a second server and its poll loop. One htsserver, one poll, modelled on the sibling webhttrack tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Resolve tests/Makefile.am: keep 66/67/69 (merged) alongside this branch's 68. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
webhttrack splits the raw HTTP POST body into argv and calls hts_main2 directly, but that body is in the web form's declared charset (LANGUAGE_CHARSET: ISO-8859-1, windows-125x, BIG5, gb2312, shift-jis, depending on the language), not UTF-8. The httrack CLI and WinHTTrack both hand the engine UTF-8, which htsname's path budget and htscache's format detection now assume, so a non-ASCII output path or URL from the web UI reached the engine as raw form-charset bytes and put the mirror in a mojibake directory instead of the one the user named.
The command line is now converted from the current LANGUAGE_CHARSET to UTF-8 before the crawl starts, in htsserver.c right before webhttrack_main() where that charset is known via LANGSEL. ASCII and already-UTF-8 input pass through untouched. webhttrack's own argv also gets the Windows hts_argv_utf8 treatment the CLI already has.
This exports hts_convertStringToUTF8 so htsserver (which links the shared library) can reach it: an additive ABI change, new symbol, soname unchanged, alongside the already-exported hts_convertStringSystemToUTF8. Flagging it since it touches the public export set.
Test 68 drives the real htsserver over HTTP, posts a start command whose -O dir is café in ISO-8859-1, and checks the mirror lands under the UTF-8 café directory, not the ISO-8859-1 twin. It fails on master and passes with the fix.
Closes #629