feat(api): Add batched chunk retrieval endpoint (e.g., POST /chunks/batch)

[martinconic]

Summary

Currently, Bee only supports retrieving chunks one at a time via GET /chunks/{address}. While /chunks/stream exists, it is strictly an inbound WebSocket for uploading chunks (chunkUploadStreamHandler). There is no mechanism in Bee for bulk or streamed chunk retrieval.

We propose adding a batched chunk retrieval API (e.g., POST /chunks/batch or a bidirectional /chunks/stream download mode) allowing clients to request multiple chunk references in a single call and receive them as a streamed or framed response.

Motivation

Today, any client that needs multiple chunks must fire individual HTTP GET /chunks/{address} requests for each chunk. This introduces high latency and HTTP overhead for several major use cases:

1. Video & Audio Streaming (Buffer Lookahead & Pre-caching):

   • Media players (HLS/DASH or raw chunk streams) maintain a lookahead buffer (e.g., 10–30 seconds of playback).
   • A single high-latency Kademlia lookup on one chunk can cause playback buffer underrun and video freezing.
   • Knowing the sequence of chunk addresses in advance, the player or gateway can request the next 30–50 chunks in a single batched call. Bee can retrieve them concurrently from neighborhood peers in the background, smoothing out network jitter.

2. Decentralized Databases & Analytical Engines:

   • SQLite over Swarm: SQLite's default page size is 4096 bytes, matching Swarm’s chunk size (ChunkSize = 4096). Implementing a remote SQLite VFS requires reading ranges of B-tree pages efficiently.
   • Data Lakes & Parquet (DuckDB): Analytical queries frequently read multiple contiguous column chunks across row groups.
   • Trie / Manifest Traversal: Traversing Mantaray nodes or single-owner feeds incurs $O(\log N)$ serial round-trips if pointer-chased one by one.

3. HTTP Connection Overhead & Node Protection:

   • Web browsers enforce limits on concurrent connections per host (typically 6). Blasting a local Bee node with 50+ concurrent HTTP requests causes socket churn, context-switching overhead, and potential connection drops.
   • A batched endpoint allows Bee to control internal concurrency using a bounded worker pool rather than exposing the node to uncontrolled incoming HTTP request storms.

Implementation

A straightforward approach could be:

• Endpoint: POST /chunks/batch
• Request Body:

{
  "addresses": [
    "7a2b...",
    "8f1c...",
    "3d4e..."
  ]
}

(A batch size limit, e.g. max 100 or 256 chunks per request, can be enforced).

• Response Format Options:

  • Framed Binary Stream: A lightweight binary stream where each chunk is prefixed by a header: [32-byte chunk address][4-byte uint32 payload length][payload data]. If a chunk is not found or fails, a status byte or length 0 indicates a miss.
  • Multipart / Mixed: Standard HTTP multipart/mixed response where each part contains headers (Swarm-Address, Content-Length) and chunk payload bytes.
  • WebSocket: Allow clients to send chunk request frames over a WebSocket and receive chunks as they arrive.

• Internal Flow:

  • In pkg/api/chunk.go, leverage s.storer.Download(cache).Get(ctx, address) across a worker pool (e.g., using errgroup with a concurrency limit of 8 or 16).
  • Stream chunks to the HTTP response writer as soon as each chunk resolves to avoid buffering large batches in memory.
  • Handle partial successes gracefully (if 49 chunks succeed and 1 returns storage.ErrNotFound, return the 49 without failing the entire batch).

Drawbacks

• Memory usage if responses were to be buffered entirely before sending (can be mitigated by streaming chunks directly to the client as they are retrieved from storage).
• Potential for clients to request arbitrarily huge batches (can be mitigated with a sensible batch size limit, e.g., 100 chunks per request).

AI Disclosure

• This issue contains suggestions and text generated by an LLM.
• I have reviewed the AI generated content thoroughly.
• I possess the technical expertise to responsibly review the AI generated content mentioned in this issue.