Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ find_package(ZLIB REQUIRED)

set(ZGET_LIBRARY_SOURCES
src/zget.c src/error.c src/util.c src/source/source.c
src/source/http-parse.c src/source/http.c src/source/download.c
src/source/file.c src/format/format.c src/format/zip/zip.c
src/source/http-parse.c src/source/http.c
src/format/format.c src/format/zip/zip.c
src/format/zip/zip-parse.c src/format/zip/extract.c)

add_library(libzget SHARED ${ZGET_LIBRARY_SOURCES})
Expand Down Expand Up @@ -147,13 +147,12 @@ if(BUILD_TESTING)
COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/integration.py
$<TARGET_FILE:zget>)
set_tests_properties(integration_reference PROPERTIES TIMEOUT 90)
# Keep the no-Range behavior isolated: this small fixture makes the
# fallback contract obvious without coupling it to the large oracle.
add_test(NAME integration_no_range_fallback
# Keep the strict Range contract isolated from the large oracle.
add_test(NAME integration_strict_range
COMMAND ${Python3_EXECUTABLE}
${CMAKE_CURRENT_SOURCE_DIR}/tests/no_range_fallback.py
${CMAKE_CURRENT_SOURCE_DIR}/tests/strict_range.py
$<TARGET_FILE:zget>)
set_tests_properties(integration_no_range_fallback PROPERTIES TIMEOUT 30)
set_tests_properties(integration_strict_range PROPERTIES TIMEOUT 30)
endif()
if(Python3_Interpreter_FOUND AND ZGET_BUILD_LARGE_TESTS)
add_test(NAME integration_100k_entries
Expand Down
38 changes: 17 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# zget

Fetch one file from a remote ZIP archive while downloading only the bytes needed when the server supports HTTP Range requests.
Fetch one file from a remote ZIP archive without downloading the complete archive.

`zget` is a command-line tool and C library for listing and extracting
individual files from remote ZIP and ZIP64 archives. It prefers precise HTTP
Range requests and transparently falls back to a complete temporary download
when a server ignores Range requests entirely. It streams archive metadata with
memory usage independent of entry count, so the same design works for ordinary
archives and scales to very large ones.
individual files from remote ZIP and ZIP64 archives using precise HTTP Range
requests. If a server cannot provide the required ranges, zget fails rather
than silently downloading the complete archive. It streams archive metadata
with memory usage independent of entry count, so the same design works for
ordinary archives and scales to very large ones.

```sh
zget https://example.com/archive.zip README.txt
Expand Down Expand Up @@ -79,8 +79,8 @@ Remote ZIP access is not unique to zget. Python projects such as
members over HTTP without downloading the whole archive when Range requests are
available. `zget` is a native C library and CLI for efficient remote ZIP/ZIP64
access. Its streaming, bounded-memory design works for ordinary archives and
remains practical as archive sizes and entry counts grow, with predictable
Range behavior and a compatibility fallback for servers that ignore Range.
remains practical as archive sizes and entry counts grow, with a strict and
predictable Range-access contract.

`zget` streams the Central Directory, discards metadata for non-matching entries
immediately, and can stop scanning as soon as the requested entry is found. It
Expand All @@ -101,18 +101,15 @@ for archives containing hundreds of thousands, millions, or tens of millions
of entries; scan time and transferred metadata still depend on the target's
position in the Central Directory.

When the server supports Range requests, a successful lookup uses semantically
precise byte ranges:
A successful lookup uses semantically precise byte ranges:

```text
tail -> central directory -> target local header -> target payload
```

If the server ignores Range requests entirely and returns the complete object,
zget downloads the archive once into anonymous temporary storage and continues
through the same local source and ZIP implementation. The operation remains
transparent to callers, but it necessarily transfers the whole archive and
therefore loses the bandwidth advantage of Range-based access.
If the server ignores a required Range request or otherwise cannot provide a
valid partial response, zget fails with a Range or HTTP error. It never silently
turns selective member retrieval into a complete archive download.

The first exact Central Directory name match wins. Extraction is streamed
through STORE or raw-DEFLATE decoding and checked against the entry CRC32.
Expand Down Expand Up @@ -323,10 +320,9 @@ the tail as an explicit interval. These extra requests count toward
`max_http_requests`.

If a server ignores the required Range request entirely and returns HTTP 200
with the complete representation, zget falls back to one complete download in
anonymous temporary storage and continues through the local-file source. This
fallback preserves extraction and listing behavior, but it transfers the entire
archive and does not provide Range-based bandwidth savings.
with the complete representation, zget rejects the response with
`ZGET_ERANGE`. Supplying `MEMBER` requests selective retrieval; zget never
silently replaces it with a complete archive download.

An inconsistent `Content-Range`, changed object size, non-identity
`Content-Encoding`, or failed `If-Match` aborts the Range operation. HTTPS
Expand All @@ -349,7 +345,7 @@ write to stdout, which likewise cannot be rolled back after a late error.
matching only when entirely ASCII; CP437 conversion is intentionally absent.
- No encryption, split archives, resume, or random seeks within a DEFLATE
member.
- HTTP Range is preferred for efficient remote access. Servers that ignore
Range entirely are supported through a transparent complete-download fallback.
- HTTP Range support is required. Servers that ignore required Range requests
fail with a Range error instead of triggering a complete download.

This project is MIT licensed.
21 changes: 10 additions & 11 deletions docs/zget.1.in
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,9 @@ zget \- fetch one member from a remote ZIP archive
.SH DESCRIPTION
.B zget
fetches one exact, case-sensitive member from a remote ZIP or ZIP64 archive.
It prefers HTTP Range requests so only the required archive bytes are
transferred. If a server ignores Range requests entirely and returns the
complete representation, zget transparently downloads the archive into
anonymous temporary storage and continues locally. That compatibility fallback
necessarily transfers the entire archive.
It uses HTTP Range requests so only the required archive bytes are transferred.
If a server cannot provide the required ranges, zget fails rather than silently
downloading the complete archive.
.PP
The Central Directory is parsed as a stream. Metadata for each non-matching
entry is discarded immediately, so memory use is independent of the archive
Expand Down Expand Up @@ -103,10 +101,11 @@ probe and retries the tail as an explicit interval. These extra requests count
toward the configured HTTP request limit.
.PP
If the server ignores a required Range request entirely and returns HTTP 200
with the complete representation, zget performs one complete download into an
anonymous temporary file and continues through its local-file source. This
preserves normal extraction and listing semantics but forfeits the bandwidth
savings of Range-based access.
with the complete representation, zget rejects the response with a Range error.
Supplying
.I MEMBER
requests selective retrieval; zget never silently replaces it with a complete
archive download.
.PP
HTTPS redirects may not downgrade to HTTP. A strong ETag, when supplied by the
server, is used with subsequent Range requests to ensure that all ranges refer
Expand Down Expand Up @@ -161,8 +160,8 @@ zget -1 https://example.com/archive.zip
Only single-volume ZIP32 and ZIP64 archives are supported. Member names are
exact full paths; there is no path normalization or globbing. Compression
methods are limited to STORE and DEFLATE. Encryption and split archives are
not supported. Servers that ignore HTTP Range are supported by a complete
fallback download, so such operations may transfer the entire archive.
not supported. Servers must provide valid HTTP Range responses; zget does not
fall back to downloading the complete archive.
.SH SEE ALSO
.BR curl (1),
.BR unzip (1),
Expand Down
136 changes: 0 additions & 136 deletions src/source/download.c

This file was deleted.

15 changes: 0 additions & 15 deletions src/source/download.h

This file was deleted.

Loading