docs(xet): correct shard/xorb format specs and stale client defaults

[rajatarya]

Aligns the public Xet specs with xet-core v1.6.0, which is the ground truth for every change here. Five findings from the xet doc-drift audit (open drift issues) — tracked as #1045, #1152, #1176, #1247, #1264.

Each change is described below, so the PR is self-contained for review.

docs/xet/shard.md (#1045)

The involved one — two layout-level drifts, both field-labeling rather than size changes, so no byte offsets shift.

• CASChunkSequenceEntry: the trailing 8-byte _unused is actually flags: u32 + _unused: u32. flags is load-bearing — it carries MDB_CHUNK_WITH_GLOBAL_DEDUP_FLAG (1 << 31), which marks a chunk eligible for global dedup. Adds a bitfield table and a rule that readers mask undefined bits rather than compare for equality. Entry stays 48 bytes.
• MDBShardFileFooter: documented 9 fields; source has 17. The two _buffer spans masked six lookup offset/count fields (bytes 24–72) and three accounting totals (bytes 168–192). Footer stays 200 bytes.
• cas_info_offset → xorb_info_offset to match source, with a note recording the old name. The section it points at is still called the CAS Info Section here.
• New: the three lookup tables. Keys are truncated hashes (first 8 bytes as u64) and values are indices, not byte offsets — so a match is inconclusive and must be confirmed against the full hash. Also documents per-entry sizes and that a 0 count means the table is absent.
• Added the tables to both layout diagrams, and corrected two consequences: the CAS info section ends at its bookend (not footer_offset, now that tables may sit between), and footer deserialization can't read every field as u64 (the HMAC key is 32 B, _buffer is 48 B).

docs/xet/xorb.md (#1176, #1264)

• Improve search on papers page #1176 — replaces "no explicit limit on the number of chunks" with the 8192-chunk cap (MAX_XORB_CHUNKS), scoped precisely: it's producer-side cut logic in the reference client, not in the wire format and not server-enforced. This is the reconciliation the issue asks for — internal docs presented 8192 as a hard maximum while this doc said there was no limit; neither was quite right.
• Update model-card-guidebook.md #1264 — documents the XorbObjectInfo footer and the trailing info_length: u32, previously absent entirely, which left readers no documented way to locate the footer (the old diagram's trailing ... read as "more chunks"). Adds the reader entry point (end - 4, length excludes itself), the fixed-size tail layout, and the uniqueness nonce — leading 4 bytes of the 16-byte buffer, excluded from the xorb hash, so identical content need not serialize to identical bytes. Pre-nonce xorbs carry an all-zero buffer and stay valid.

docs/xet/api.md (#1247)

One-line fix: /v2/shards falls back to /v1/shards on a 404 or 501, not 404 alone. A client built strictly from the old text would treat 501 as fatal, and 501 is classed as permanent so no retry papers over it. The sibling reconstruction note already said "404 or 501", so the two notes in this file disagreed. Also notes that the reference client caches the detected version per session.

docs/hub/xet/using-xet-storage.md (#1152)

Three stale defaults:

Variable	Was	Now
HF_XET_CLIENT_AC_INITIAL_UPLOAD_CONCURRENCY	1	2
HF_XET_CLIENT_AC_INITIAL_DOWNLOAD_CONCURRENCY	1	4
HF_XET_CLIENT_READ_TIMEOUT	120s	300s

Verification

• Every claim re-checked against xet-core v1.6.0 source rather than taken from the issue text. Two issues were stale and corrected in the process: Update model-card-guidebook.md #1264 calls the footer CasObjectInfo, a name no longer in source (XorbObjectInfoV1), and Improve search on papers page #1176's original "debug-assert only" framing understates the cap, which is real release-path cut logic.
• Byte-layout diagrams were generated programmatically, and offset rulers verified to align with column borders. Footer arithmetic checked to sum to exactly 200 bytes and the chunk entry to 48.
• Diagram alignment before/after: shard.md had 9 diagrams whose offset rulers didn't line up with their borders on main; this branch has 6. The 3 fixed are the ones regenerated here; none were introduced.

Deliberately out of scope

Broader CAS* → Xorb* terminology alignment across shard.md (section headings, CASChunkSequenceHeader, bookend prose). Only cas_info_offset is renamed, because it's a field name clients bind to; the rest is prose and a larger public-terminology decision.

---

Note

Low Risk
Changes are documentation only; they correct spec drift and env-var defaults without altering runtime code or deployed behavior.

Overview
Documentation-only update to match xet-core v1.6.0 and fix doc drift across Hub env vars and Xet wire-format specs.

shard.md expands the shard layout: optional lookup tables after the CAS info section; footer documents xorb_info_offset (renamed from cas_info_offset), six lookup offset/count fields, and accounting totals; CASChunkSequenceEntry now exposes flags (including global-dedup) instead of an 8-byte _unused field; parsing notes clarify CAS info ends at the bookend and footer field types.

xorb.md adds the 8192-chunk producer cap (not wire/server enforced), the XorbObjectInfo footer with trailing info_length, and the uniqueness nonce behavior relative to the xorb hash.

api.md states clients should fall back from /v2/shards to /v1/shards on 404 or 501, with per-session version caching.

using-xet-storage.md updates documented defaults: initial upload concurrency 2, download 4, read timeout 300s.

^{Reviewed by Cursor Bugbot for commit bc810d6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[rajatarya]

cc @sirahd for review alongside @seanses — GitHub won't accept a formal review request here since you're not a collaborator on this repo, so flagging it this way instead.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.