From 50d30e974e8ee5a3fc00d369ddba00a4d3cfc163 Mon Sep 17 00:00:00 2001 From: Aleksandr Cupacenko Date: Wed, 26 Aug 2026 20:08:27 +0300 Subject: [PATCH] docs: tighten README around core use case --- README.md | 339 +++++++++++++++++++++--------------------------------- 1 file changed, 131 insertions(+), 208 deletions(-) diff --git a/README.md b/README.md index 9f2229a..0f229b0 100644 --- a/README.md +++ b/README.md @@ -2,25 +2,23 @@ Fetch one file from a remote ZIP archive without downloading the complete archive. -`zget` is a command-line tool and C library for listing and extracting -individual files from remote ZIP and ZIP64 archives using precise HTTP Range -requests. If a server cannot provide the required ranges, zget fails rather -than silently downloading the complete archive. It streams archive metadata -with memory usage independent of entry count, so the same design works for -ordinary archives and scales to very large ones. +`zget` is a command-line tool and C library for selective access to remote ZIP +and ZIP64 archives over HTTP Range requests. It can extract one exact member or +stream the archive listing. If the server cannot provide the required ranges, +zget fails rather than silently downloading the complete archive. ```sh zget https://example.com/archive.zip README.txt zget -o report.pdf https://example.com/documents.zip path/to/report.pdf +zget https://example.com/data.zip config.json | jq . zget -l https://example.com/archive.zip zget -1 https://example.com/archive.zip -zget https://example.com/data.zip config.json | jq . ``` ## Installation -Install the required runtime dependencies (`libcurl` and `zlib`) first. Then -download the archive for your platform from the +Install the runtime dependencies (`libcurl` and `zlib`), then download the +archive for your platform from the [latest release](https://github.com/unitmatrix/zget/releases/latest), extract it, and put `zget` somewhere on your `$PATH`. @@ -45,91 +43,147 @@ zget -l URL zget -1 URL ``` -## Familiar by design +`zget URL MEMBER` streams the exact member to standard output. `-o FILE` writes +it directly to the requested path using normal file-open semantics; `-o -` +selects standard output. `-l` prints an `unzip`-style whole-archive listing and +`-1` prints only member names. -Archive operations follow `unzip` where practical, while output conventions -follow `curl`. `zget URL MEMBER` therefore streams to standard output, while -`-o FILE` streams directly to a local output using normal file-open semantics: -an existing file is truncated and overwritten, and `-o -` selects standard -output. -`-l` requests an `unzip`-style listing; the compact `-1` form follows `zipinfo` -and emits only names. +Extraction requires both a URL and an exact, case-sensitive member path. A URL +without a member is a usage error; use `-l` or `-1` explicitly to list the +archive. -Extraction requires both a URL and an exact member name. A URL without a member -is a usage error rather than an implicit request to list the archive; use `-l` -or `-1` explicitly to stream the complete Central Directory listing. +## Library -Listings use the familiar `unzip -l` columns: uncompressed length, modification -date and time in UTC, and member name. Timestamps are resolved from NTFS, -Extended Timestamp, or packed DOS metadata in that order. Listings scan the -Central Directory once and retain only the current entry. Control characters -and backslashes are escaped so every member stays on one safe output line. +`libzget` exposes two synchronous one-shot operations: -For a compact listing, `-1` writes only member names, one per line, with no -header or totals, using the same safe escaping as `-l`. +```c +int zget_get(const char *archive_url, const char *member_name, + zget_write_cb write_cb, void *userdata); -## Why zget? +int zget_list(const char *archive_url, zget_list_cb list_cb, + void *userdata); +``` -Remote ZIP access is not unique to zget. Python projects such as -[`remotezip`](https://github.com/gtsystem/python-remotezip) and -[`unzip-http`](https://github.com/saulpw/unzip-http) already list and extract -members over HTTP without downloading the whole archive when Range requests are -available. `zget` is a native C library and CLI for efficient remote ZIP/ZIP64 -access. Its streaming, bounded-memory design works for ordinary archives and -remains practical as archive sizes and entry counts grow, with a strict and -predictable Range-access contract. +The CLI is the reference consumer of this same public API. Its extraction path +uses the public call directly: -`zget` streams the Central Directory, discards metadata for non-matching entries -immediately, and can stop scanning as soon as the requested entry is found. It -does not build an in-memory index of the entire archive. +```c +static int write_file(void *opaque, const void *data, size_t size) +{ + return write_stream(opaque, data, size); +} -```text -Central Directory HTTP stream - ↓ -parse one entry - ↓ -compare exact name - ├─ no match → discard metadata → next entry - └─ target found → stop the HTTP transfer +rc = zget_get(url, member, write_file, &output); ``` -Memory use is O(1) with respect to archive entry count. This property matters -for archives containing hundreds of thousands, millions, or tens of millions -of entries; scan time and transferred metadata still depend on the target's -position in the Central Directory. +Listing follows the same pattern: -A successful lookup uses semantically precise byte ranges: +```c +rc = zget_list(url, list_member, &listing); +``` -```text -tail -> central directory -> target local header -> target payload +Callbacks run inline. Output buffers and listing records are borrowed only for +the duration of the callback. Extraction may emit data before a later +compression, size, or CRC failure, so only `ZGET_OK` guarantees complete +validated output. Applications that require atomic publication should write to +temporary storage and publish it only after success. + +Each `zget_member_info` contains a valid UTF-8 name, compressed and uncompressed +sizes, CRC32, numeric ZIP compression method, and modification time as Unix UTC +seconds. Names are resolved from the ZIP UTF-8 flag, a usable Info-ZIP Unicode +Path field, or CP437 in that order. + +Link with pkg-config: + +```sh +cc -o example example.c $(pkg-config --cflags --libs libzget) ``` -If the server ignores a required Range request or otherwise cannot provide a -valid partial response, zget fails with a Range or HTTP error. It never silently -turns selective member retrieval into a complete archive download. +Or use the installed CMake target `Zget::libzget` after +`find_package(Zget 0.6 REQUIRED)`. + +The library is pre-1.0. API and ABI may change between minor releases; patch +releases preserve the ABI of their `0.MINOR` line. Compile-time version macros +are available in ``, and `zget_version()` reports the linked +library version at runtime. + +## Why zget? + +Remote ZIP access exists in other ecosystems, including Python projects such as +[`remotezip`](https://github.com/gtsystem/python-remotezip) and +[`unzip-http`](https://github.com/saulpw/unzip-http). zget provides the same core +capability as a small native C library and CLI with a strict selective-access +contract. -The first exact Central Directory name match wins. Extraction is streamed -through STORE or raw-DEFLATE decoding and checked against the entry CRC32. +For exact lookup, zget streams the Central Directory one entry at a time, +discards non-matching metadata immediately, and can stop the HTTP transfer on +the first matching entry. It does not build an archive-wide in-memory index. + +```text +archive tail + ↓ +Central Directory stream + ↓ +first exact name match + ↓ +target local header + ↓ +target payload +``` + +Memory use is O(1) with respect to archive entry count. Scan time and transferred +Central Directory metadata still depend on where the requested member appears. +The payload is streamed through STORE or raw-DEFLATE decoding and checked against +its expected size and CRC32. ## Architecture -The CLI is a thin frontend over `libzget`. Internally, format code requests -semantic byte ranges from a source instead of depending on a transport: +The CLI is a thin frontend over `libzget`. Internally, ZIP code requests byte +ranges from a private source abstraction rather than depending on libcurl +itself: ```text zget CLI | public libzget API | -format engine -- semantic ranges --> source - | | -ZIP / zlib HTTP / libcurl +format engine -- ranges --> source + | | +ZIP / zlib HTTP / libcurl ``` -The HTTP source knows nothing about ZIP layout, and the ZIP engine knows -nothing about libcurl. This small internal boundary exists for clear ownership -and independent testing; it is not a public extension point, registry, or -dynamic plugin system. +The HTTP source knows nothing about ZIP layout, and the ZIP engine knows nothing +about libcurl. This boundary exists for ownership, testing, and fuzzing; it is +not a public plugin or transport API. + +## HTTP contract + +Archive data is accepted only from valid partial responses. A successful Range +request must return `206 Partial Content` with a matching `Content-Range`, the +expected body length, and the identity representation. + +The initial archive tail normally uses a suffix range. Some servers reject +suffix syntax while accepting explicit ranges; in that case zget performs a +one-byte explicit probe to learn the object size and retries the equivalent tail +interval. + +A server that ignores a required Range request and returns HTTP 200 is rejected. +Object size must remain stable between requests, and a strong ETag from the +first accepted response is reused with `If-Match` when available. HTTPS +redirects cannot downgrade to HTTP. + +## Scope + +- HTTP(S) remote sources. +- Single-volume ZIP32 and ZIP64, including data descriptors. +- Exact, case-sensitive member paths; no path or Unicode normalization and no + globbing. +- Compression methods STORE (0) and DEFLATE (8). +- Valid UTF-8 semantic names resolved from UTF-8, Info-ZIP Unicode Path, or + CP437 metadata. +- No encryption, split archives, resume, or random seeks within a DEFLATE + member. +- HTTP Range support is required; there is no complete-download fallback. ## Build @@ -141,9 +195,11 @@ cmake --build build ctest --test-dir build --output-on-failure ``` -CMake uses installed system copies of libcurl and zlib; production builds do -not vendor or download dependencies. Tests are self-contained and use a local -HTTP fixture rather than the public Internet. +CMake uses installed system copies of libcurl and zlib. Tests use local fixtures +rather than the public Internet. Optional build switches include +`ZGET_ENABLE_SANITIZERS=ON`, `ZGET_BUILD_FUZZERS=ON`, and +`ZGET_BUILD_LARGE_TESTS=ON`; setting `ZGET_MILLION_ENTRY_TEST=1` while running +CTest enables the generated one-million-entry case. For a staged distro-style install: @@ -155,142 +211,9 @@ cmake --build build DESTDIR="$PWD/stage" cmake --install build ``` -Install destinations follow `GNUInstallDirs`, including multiarch or `lib64` -layouts selected by the toolchain or packager. The install includes the CLI, -shared and static libraries, public header, CMake and pkg-config metadata, the -`zget(1)` man page, and the license under `share/licenses/zget`. - -Set `ZGET_ENABLE_SANITIZERS=ON` for ASan and UBSan. Clang users can set -`ZGET_BUILD_FUZZERS=ON` to build the libFuzzer parser targets. -`ZGET_BUILD_LARGE_TESTS=ON` adds a generated 100k-entry integration test; set -`ZGET_MILLION_ENTRY_TEST=1` when running CTest to include one million entries. - -The build produces shared and static `libzget` libraries. The `zget` executable -links the project library statically, so it does not require a separately -installed `libzget`; curl, zlib, TLS, and the platform C runtime remain dynamic. - -## Library - -`libzget` is the reusable implementation behind the `zget` CLI. Its installed -header is ``. The public API consists of two synchronous one-shot -operations, callbacks, member records, and zget error codes; it exposes no -contexts, global lifecycle, or transport handles. libcurl and zlib remain -private implementation dependencies. Current library support is HTTP(S), -single-volume ZIP/ZIP64, STORE, and DEFLATE. - -The library is pre-1.0. Its API and ABI may change between minor releases while -the design is refined; patch releases preserve the ABI within one `0.MINOR` -line. Compile-time version macros are available in ``, and -`zget_version()` reports the linked library version at runtime. - -Link with pkg-config: - -```sh -cc -o example example.c $(pkg-config --cflags --libs libzget) -``` - -Or use the installed CMake target `Zget::libzget` after -`find_package(Zget 0.6 REQUIRED)`. - -### Minimal extraction example - -```c -#include - -#include - -static int write_stdout(void *userdata, const void *data, size_t size) -{ - FILE *output = userdata; - return fwrite(data, 1, size, output) == size ? 0 : 1; -} - -int main(int argc, char **argv) -{ - int rc; - - if (argc != 3) { - fprintf(stderr, "usage: %s URL MEMBER\n", argv[0]); - return 2; - } - rc = zget_get(argv[1], argv[2], write_stdout, stdout); - if (rc != ZGET_OK) - fprintf(stderr, "zget: %s\n", zget_error_string(rc)); - return rc == ZGET_OK ? 0 : 1; -} -``` - -The output callback borrows each buffer only for that callback invocation. -Extraction may emit data before a later decompression, size, or CRC failure, so -only a `ZGET_OK` return guarantees complete validated output. Applications that -need atomic publication should stream to temporary storage and publish it only -after success. The CLI deliberately gives `-o` ordinary curl-style streaming -semantics instead. - -### Listing - -`zget_list()` streams every Central Directory entry as one borrowed record and -does not construct an archive-wide index: - -```c -static int print_member(void *userdata, const zget_member_info *member) -{ - FILE *output = userdata; - - if (fwrite(member->name, 1, member->name_length, output) != - member->name_length || fputc('\n', output) == EOF) - return 1; - return 0; -} - -rc = zget_list(url, print_member, stdout); -``` - -The record and its NUL-terminated name become invalid when the callback -returns. Every name is valid UTF-8. Resolution uses the ZIP UTF-8 flag first, -then a valid Info-ZIP Unicode Path (`0x7075`) field, then CP437 conversion. -`mtime` is signed Unix time in UTC seconds. The record also supplies compressed -and uncompressed sizes, CRC32, and the numeric ZIP compression method. - -Both public operations manage their own resources and libcurl lifecycle. They -are synchronous and callbacks run inline. Independent calls share no archive -state and may be made from different threads on supported libcurl builds. - -## HTTP invariants - -Range-capable servers must return each accepted archive-data response as a -matching `206 Partial Content` response with the correct body length. The -initial tail normally uses a suffix range. If a server rejects or ignores that -syntax but supports explicit ranges, zget uses a one-byte size probe and retries -the tail as an explicit interval. - -If a server ignores the required Range request entirely and returns HTTP 200 -with the complete representation, zget rejects the response with -`ZGET_ERANGE`. Supplying `MEMBER` requests selective retrieval; zget never -silently replaces it with a complete archive download. - -An inconsistent `Content-Range`, changed object size, non-identity -`Content-Encoding`, or failed `If-Match` aborts the Range operation. HTTPS -redirects cannot downgrade to HTTP. A strong ETag from the first accepted -response is used for later `If-Match` requests. Without one, consistency is -best-effort and still checks the object size. - -`-o FILE` opens the requested path normally and streams member data into it. -Existing regular files are truncated, and paths such as symlinks, FIFOs, and -devices follow the platform's normal open behavior. If extraction fails after -writing begins, the partial output remains. `-o -` and the default output both -write to stdout, which likewise cannot be rolled back after a late error. - -## Scope - -- Exact, case-sensitive full member paths; no normalization or globbing. -- Single-volume ZIP32 and ZIP64, including data descriptors. -- Compression methods STORE (0) and DEFLATE (8) only. -- Names are resolved to valid UTF-8 using the UTF-8 flag, a valid Info-ZIP - Unicode Path field, or CP437. Embedded NUL bytes are malformed. -- No encryption, split archives, resume, or random seeks within a DEFLATE - member. -- HTTP Range support is required. Servers that ignore required Range requests - fail with a Range error instead of triggering a complete download. +Installation follows `GNUInstallDirs` and includes the CLI, shared and static +libraries, public headers, CMake and pkg-config metadata, the `zget(1)` man page, +and the license. Release CLI binaries link project code statically; curl, zlib, +TLS, and the platform C runtime remain dynamic dependencies. This project is MIT licensed.