Skip to content

Expose post-3.49-2 engine options in the Android GUI (SOCKS5, cookies-file, pause, keep-*, strip-query, UA) #6

Description

@xroche

Context

The engine gained several user-facing crawl options between 3.49-3 and 3.49-12 that the
GUIs were frozen before. The Windows GUI now exposes them in xroche/httrack-windows#31
(green, reviewed). The Android app should expose the same set so the front-ends stay in
parity. This issue carries everything needed to do it independently.

The engine is vendored here under app/src/main/jni/httrack/, so it already contains these
options once the JNI copy is current — this is purely about surfacing them in the Android
options UI.

Where this lands in this repo

The options UI mirrors the desktop tabs:

  • app/src/main/java/com/httrack/android/OptionsMapper.java — maps UI fields ⇄ engine options (the seam to extend).
  • app/src/main/java/com/httrack/android/OptionsActivity.java — options screens.
  • app/src/main/res/layout/activity_options_proxy.xml, …_spider.xml, …_flowcontrol.xml, …_limits.xml, …_build.xml, …_logindexcache.xml — per-panel layouts.
  • UI strings go in app/src/main/res/values/strings.xml (+ per-locale values-*/).

Options to expose

Emitted to the engine as the CLI flag shown. Short forms verified in the engine's
src/htsalias.c; long forms are the documented public names.

Feature CLI (long / short) Kind Where (panel) Semantics
SOCKS5 proxy -P socks5://[user:pass@]host[:port] proxy string proxy Scheme prefixed onto the single -P. socks5://socks5h:// (remote DNS). Default port 1080 vs 8080. CONNECT-for-HTTPS and SOCKS tunneling are automatic. Reuse the single proxy field; add a protocol selector.
Preload cookies --cookies-file <path> / -%K value spider Load a Netscape cookies.txt before crawling. A file picker fits Android well.
Random pause --pause MIN[:MAX] / -%G value (seconds) flow control Random delay between downloads. Distinct from -G (pause after N bytes).
Strip query keys --strip-query <[host/pattern=]k1,k2,…> / -%g value limits/expert Drop query keys from dedup naming. Simple sid,utm_source; scoped www.example.com/*=sid,ref.
Keep www. prefix --keep-www-prefix / -%j toggle links Opt out of one part of the -%u URL hack.
Keep double slashes --keep-double-slashes / -%o toggle links Opt out of -%u.
Keep query order --keep-query-order / -%y toggle links Opt out of -%u.

Value options take a separate argument; the three keep-* are bare toggles that only matter
when -%u is on.

Stale default to check

The engine (3.49-11) defaults the User-Agent to
Mozilla/5.0 (compatible; HTTrack/<ver>; +https://www.httrack.com/). Make sure the Android
app doesn't ship the old Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) default. Also
check the default max transfer rate (engine default 100000 B/s, cap 10000000 B/s).

No UI needed (automatic)

deflate, robots Allow/wildcards, <source>/<track>, full srcset, WHATWG entities,
modern MIME types, HTTPS-over-HTTP-proxy CONNECT, SOCKS tunneling.

Deferred

--why / -%Y (filter-verdict diagnostic, diagnose-and-exit) — leave out or make a "test a
URL" action, matching the Windows decision.

Reference implementation (WinHTTrack, xroche/httrack-windows#31)

Commit-by-commit, same options, done:

  1. Modernize the default User-Agent.
  2. SOCKS5 — proxy-type (HTTP/SOCKS5) selector prepending socks5:// to -P, reusing the single proxy field.
  3. --cookies-file field.
  4. --pause field (seconds, MIN:MAX).
  5. Three --keep-* checkboxes.
  6. --strip-query field.
  7. Polish: file picker for the cookies path; clearer labels.

OptionsMapper.java is the Android analogue of WinHTTrack's Shell.cpp command-builder —
add the same flag emissions there and the matching widgets in the layout XML + strings.xml.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions