PDF: legacy CJK CMaps — code → CID → Unicode for composite fonts

CMakeLists.txt

@@ -183,6 +183,7 @@ set(ODR_SOURCE_FILES
         "src/odr/internal/pdf/pdf_afm.cpp"
         "src/odr/internal/pdf/pdf_afm_data.cpp"
         "src/odr/internal/pdf/pdf_cid.cpp"
+        "src/odr/internal/pdf/pdf_cid_data.cpp"
         "src/odr/internal/pdf/pdf_cmap.cpp"
         "src/odr/internal/pdf/pdf_cmap_parser.cpp"
         "src/odr/internal/pdf/pdf_color.cpp"

docs/design/README.md

@@ -1,5 +1,11 @@
 # Design
 
+## Diagrams
+
+- [PDF CJK text: the `code → CID → Unicode` path](pdf-cjk-code-cid-unicode.html)
+  — why composite (Type0) fonts need the legacy-CMap tables and how the
+  `pdf_cid_data` lookup works (open in a browser).
+
 ## Motivation
 
 - non-intrusive read/write access to documents

src/odr/internal/pdf/AGENTS.md

@@ -118,12 +118,18 @@ inline SVG per page. Experimental and not production-quality.
   mixes a 1-byte code (e.g. a space) among 2-byte CIDs stays aligned and selects
   the right glyph/advance. Extraction is driven by the `/ToUnicode` CMap (the
   common case — every Type0 font in the corpus carries one). When a composite
-  font has no `/ToUnicode`, a **predefined Unicode `/Encoding`** — the
-  `Uni*-UCS2/UTF16/UTF32` CMaps — is decoded directly (`pdf_cid`), since those
-  character codes already are Unicode (big-endian); any
-  other case (`Identity-H/V`, or the legacy CJK code→CID CMaps) yields "no
-  Unicode" (not byte-garbage) until the legacy CID → Unicode tables land or —
-  for an embedded font program — the reverse map below recovers it.
+  font has no `/ToUnicode`, the named `/Encoding` still resolves the text
+  (`pdf_cid`, backed by the generated `pdf_cid_data` tables): a **predefined
+  Unicode `/Encoding`** — the `Uni*-UCS2/UTF16/UTF32` CMaps — is decoded directly
+  (those codes already are Unicode, big-endian), and a **legacy CJK CMap**
+  (`90ms-RKSJ-H`, `GBK-EUC-H`, `B5pc-H`, `KSC-EUC-H`, …) maps code → CID through
+  its own codespace + range tables and then CID → Unicode through its character
+  collection's table. When `/Encoding` is `Identity-H/V` (code == CID) or an
+  embedded CMap stream (`Font::codes()` already yields the CIDs), the collection
+  named by the descendant CIDFont's `/CIDSystemInfo` supplies the CID → Unicode
+  step directly. Only a genuinely unmapped code (a CMap we ship no tables for, an
+  `Identity` ordering, an unlisted CID) falls through to the embedded font's
+  reverse map below, and failing that yields "no Unicode" (not byte-garbage).
 - **Glyph metrics**: a font's advance widths are parsed —
   `/FirstChar` + `/Widths` + `/FontDescriptor` `/MissingWidth` for simple fonts,
   `/W` + `/DW` (the descendant CIDFont, both `c [w…]` and `c_first c_last w`
@@ -360,9 +366,10 @@ automated perceptual-diff gate is deferred; see gaps.)
 | `pdf_document.hpp`                     | `Document`: arena of `Element`s + `catalog` pointer |
 | `pdf_document_element.hpp`             | Element structs: `Catalog`, `Pages`, `Page`, `Annotation`, `Resources` (font + XObject + `/Properties` tables), `XObject` (Form/Image subtype, `/Matrix`, decoded content, own `/Resources`), `Font` (incl. the `composite`/`cid_registry`/`cid_ordering` Type0 facts, the `/Widths`-`/W`/`/DW` glyph metrics + `advance_width`, the `embedded_font`/`cid_to_gid` + `glyph_for_code`, and `to_unicode` with its embedded reverse-map fallback) |
 | `pdf_cmap.{hpp,cpp}`                   | `CMap`: 1-byte glyph → UTF-16 `bfchar` map + string translation |
-| `pdf_cmap_parser.{hpp,cpp}`            | `ToUnicode` CMap stream parser (`begincodespacerange`/`beginbfchar`/`beginbfrange`; only `bfchar` applied) |
+| `pdf_cmap_parser.{hpp,cpp}`            | `ToUnicode` CMap stream parser (`begincodespacerange` sets code widths; `beginbfchar` and both `beginbfrange` forms applied) |
 | `pdf_encoding.{hpp,cpp}`               | Simple-font `/Encoding` → Unicode: `BaseEncoding` tables, `/Differences` overlay (`Encoding`), glyph-name → Unicode via AGL + `uniXXXX`/`uXXXXXX` |
-| `pdf_cid.{hpp,cpp}`                    | Composite-font predefined `/Encoding` → Unicode: the `Uni*-UCS2/UTF16/UTF32` CMaps decoded directly (no data tables); legacy CJK CMaps deferred (see `tools/pdf/generate_cid_data.py`) |
+| `pdf_cid.{hpp,cpp}`                    | Composite-font predefined `/Encoding` → Unicode: `Uni*-UCS2/UTF16/UTF32` decoded directly; legacy CJK CMaps mapped code → CID (interned-range binary search) → Unicode (collection bitmap rank); plus `cid_to_unicode(registry, ordering, cid)` for the `Identity-H/V` + embedded-stream path |
+| `pdf_cid_data.{hpp,cpp}`              | **Generated** (`tools/pdf/generate_cid_data.py`): the S3-packed legacy-CMap tables — interned code → CID ranges + per-CMap `uint16` index lists, and a per-collection CID → Unicode presence bitmap + rank index + `uint16` value array (astral escape). ~0.58 MB compiled |
 | `pdf_encoding_data.{hpp,cpp}`          | **Generated** (`tools/pdf/generate_encoding_data.py`): base-encoding tables + the Adobe Glyph List as a name-sorted array |
 | `util/math_util.hpp`                   | `util::math::Transform2D`: 2-D affine transform (PDF row-vector convention) — compose, point-apply, translation/scaling factories |
 | `pdf_graphics_operator.hpp`            | `GraphicsOperatorType` enum (full operator set) + `GraphicsOperator` (type + `Object` arguments) |
@@ -463,8 +470,9 @@ surprises **throw** `std::runtime_error` (missing `obj`/`endobj`/`stream`/
 element type, stream exhaustion). A missing or unresolvable `/Length` is
 tolerated — the stream extent is recovered by scanning to `endstream`. Unknown
 **content** is tolerated: unrecognized operators logged and skipped, unmodelled
-operators ignored by `execute`, annotations keep their raw dictionary, CMap
-`codespacerange`/`bfrange` parsed past without effect. References to free/absent
+operators ignored by `execute`, annotations keep their raw dictionary. An
+unmapped CMap code passes through as its numeric value (identity for single
+bytes). References to free/absent
 objects resolve to null with a warning; unknown xref-stream entry types treated
 as absent (7.5.8.3). A structural throw in the cross-reference layer is not
 fatal, though: it is caught once and the file is forward-scanned to rebuild the
@@ -478,15 +486,16 @@ in the module — new diagnostics should follow the same pattern.
 no rasterized output: glyphs render via the embedded font (`@font-face`) and
 vector content via SVG — the browser draws everything. Because display
 is driven by the *glyph*, not the extracted Unicode, **text-extraction gaps
-degrade only selectability and search — never display.** Every deferred
-code → Unicode case (legacy CJK CID → Unicode tables, `Identity-H` without
-`/ToUnicode`) still renders correctly: all glyphs are re-encoded to the Private
-Use Area for display (the uniform re-encode, decision 2026-06-19), with the
-extracted Unicode carried separately to drive selection/search; runs with no
-recoverable Unicode are additionally marked non-extractable (`user-select: none`,
-`aria-hidden`). So the deferred CJK/legacy-CMap work is a *selectability* gap, not
-a rendering risk — such PDFs look right, their text just isn't selectable until
-the tables land.
+degrade only selectability and search — never display.** A residual
+code → Unicode case (a CMap we ship no tables for, an unlisted CID) still renders
+correctly: all glyphs are re-encoded to the Private Use Area for display (the
+uniform re-encode, decision 2026-06-19), with the extracted Unicode carried
+separately to drive selection/search; runs with no recoverable Unicode are
+additionally marked non-extractable (`user-select: none`, `aria-hidden`). So even
+an unshipped-CMap CJK PDF looks right — only its text selectability is affected.
+The common legacy-CJK selectability gap is now closed: the `pdf_cid_data` tables
+resolve the predefined legacy CMaps and the `/CIDSystemInfo` collections (see
+*Fonts / text mapping*).
 
 **Font handling is in-house, facts-vs-glyphs split.** No FontForge (pdf.js proves
 it unnecessary; it is a heavy build and no read-only library can inject the PUA
@@ -562,10 +571,16 @@ glyph, with the extracted Unicode carried separately for selection/search (see
   algorithmic `uniXXXX`/`uXXXXXX` forms, `Encoding::translate_string` with a base
   encoding, the Latin-1 upper half (WinAnsi/MacRoman), a `/Differences` override,
   and the WinAnsi-vs-Standard `0x27` divergence.
-- `test/src/internal/pdf/pdf_cid.cpp` — **assertion-based**, no fixtures:
+- `test/src/internal/pdf/pdf_cid.cpp` — **assertion-based**, no fixtures (the
+  vectors are derived from the Adobe CMap data by `tools/pdf` and frozen inline):
   `translate_predefined_cmap` over the predefined Unicode CMaps — `UCS2`/`UTF16`
-  (incl. a surrogate pair) and `UTF32` decoding, a `-V` writing-mode variant, and
-  the `nullopt` for `Identity-H` and the legacy CJK CMaps.
+  (incl. a surrogate pair) and `UTF32` decoding, a `-V` writing-mode variant — and
+  over the legacy CJK CMaps — a Shift-JIS (`90ms-RKSJ-H`, Adobe-Japan1), an EUC
+  (`GBK-EUC-H`, Adobe-GB1) and a Hojo (`Hojo-H`, deprecated Adobe-Japan2) `code →
+  CID → Unicode`, plus a mixed 1-/2-byte codespace staying aligned; `cid_to_unicode`
+  by `/CIDSystemInfo` collection (`Adobe`/`Japan1`, `Manga1` and `Japan2`, incl.
+  an astral escape and the unknown-collection/CID `nullopt`); and the `nullopt`
+  for `Identity-H` and an unshipped name.
 - `test/src/internal/util/math_util_test.cpp` — **assertion-based**, no fixtures:
   `Transform2D` point-apply (identity/translation/scaling), the ordered
   (row-vector) composition, and compose-then-apply ≡ sequential apply.
@@ -674,12 +689,6 @@ The next feature cluster — needs destinations from the page tree, little else.
   unindexed. Both are edge cases beyond the corpus seen so far.
 - **Linearized files** are not handled specially (the tail-first read usually
   still works, but hint streams are ignored).
-- **CMap coverage** — still open: the legacy CJK code→CID CMaps
-  (RKSJ/EUC/Big5/GBK/KSC) and their CID → Unicode tables (large external data;
-  the generator scaffolding in `tools/pdf/generate_cid_data.py` is landed, the
-  storage decision and lookup remain). Everything else is handled (`ToUnicode`,
-  simple `/Encoding` → AGL, composite via `/ToUnicode` / predefined Unicode CMaps
-  / embedded reverse map — see *What works*).
 - **Bidi & vertical writing** (deferred): RTL run reordering for the
   layout/selection order, and vertical writing mode (`Identity-V`/CJK — the
   `/W2`/`/DW2` vertical metrics and a perpendicular pen advance, which the

src/odr/internal/pdf/pdf_cid.cpp

@@ -1,20 +1,26 @@
 #include <odr/internal/pdf/pdf_cid.hpp>
 
+#include <odr/internal/pdf/pdf_cid_data.hpp>
 #include <odr/internal/util/string_util.hpp>
 
+#include <algorithm>
+#include <bit>
 #include <cstddef>
 #include <cstdint>
 #include <string>
+#include <utility>
 
 namespace odr::internal::pdf {
 
 namespace {
 
-// The Adobe predefined Unicode CMaps name their flavour in the middle segment:
-// `Uni<Collection>-UCS2-H`, `-UTF16-H`, `-UTF32-H` (optionally with a
-// `HW`/`Pro` or writing-mode suffix). UCS-2 and UTF-16 are both 2-byte
-// big-endian code units (UTF-16 additionally pairs surrogates); UTF-32 is
-// 4-byte big-endian.
+// --- predefined Unicode CMaps (Uni*): the codes already are Unicode ---------
+
+/// The Adobe predefined Unicode CMaps name their flavour in the middle segment:
+/// `Uni<Collection>-UCS2-H`, `-UTF16-H`, `-UTF32-H` (optionally with a
+/// `HW`/`Pro` or writing-mode suffix). UCS-2 and UTF-16 are both 2-byte
+/// big-endian code units (UTF-16 additionally pairs surrogates); UTF-32 is
+/// 4-byte big-endian.
 enum class UnicodeCodec { utf16be, utf32be };
 
 std::optional<UnicodeCodec> classify(const std::string_view name) {
@@ -53,17 +59,186 @@ std::string decode_utf32be(const std::string &codes) {
   return result;
 }
 
-} // namespace
+// --- legacy CJK CMaps: code -> CID -> Unicode via the generated tables -------
 
-std::optional<std::string>
-translate_predefined_cmap(const std::string_view name,
-                          const std::string &codes) {
-  const std::optional<UnicodeCodec> codec = classify(name);
-  if (!codec.has_value()) {
+const cid_data::PredefinedCMap *
+find_predefined_cmap(const std::string_view name) {
+  const auto *const begin = cid_data::predefined_cmaps.data();
+  const auto *const end = begin + cid_data::predefined_cmaps.size();
+  const auto *const it = std::lower_bound(
+      begin, end, name,
+      [](const cid_data::PredefinedCMap &cmap, const std::string_view target) {
+        return cmap.name < target;
+      });
+  return it != end && it->name == name ? it : nullptr;
+}
+
+/// Byte width of the code at the front of `bytes`, matched *per byte* against
+/// the codespace ranges (ISO 32000-1 9.7.6.2). Full-byte matching (not just the
+/// leading byte) is required where ranges share a leading byte but diverge
+/// later: GB18030's 2-byte (`0x8140`-`0xfefe`) and 4-byte
+/// (`0x81308130`-`0xfe39fe39`) ranges both lead with `0x81`-`0xfe`, and the
+/// second byte (`0x40`-`0xfe` vs `0x30`-`0x39`) decides. Ranges are stored
+/// width-ascending, so the first full match is the shortest valid code. Falls
+/// back to a single byte when none matches (or the tail is a partial code).
+std::size_t code_width(const cid_data::PredefinedCMap &cmap,
+                       const std::string_view bytes) {
+  for (std::uint32_t i = 0; i < cmap.codespace_count; ++i) {
+    const cid_data::CodespaceRange &range = cmap.codespace[i];
+    if (bytes.size() < range.width) {
+      continue;
+    }
+    bool matched = true;
+    for (unsigned b = 0; b < range.width; ++b) {
+      const auto shift = static_cast<unsigned>(8 * (range.width - 1 - b));
+      const auto low = static_cast<std::uint8_t>(range.low >> shift);
+      const auto high = static_cast<std::uint8_t>(range.high >> shift);
+      const auto value = static_cast<std::uint8_t>(bytes[b]);
+      if (value < low || value > high) {
+        matched = false;
+        break;
+      }
+    }
+    if (matched) {
+      return range.width;
+    }
+  }
+  return 1;
+}
+
+/// The CID a `width`-byte code with numeric value `value` selects, via the
+/// CMap's interned range index list (sorted by `(width, code_low)`); `nullopt`
+/// for an unmapped code.
+std::optional<std::uint32_t> code_to_cid(const cid_data::PredefinedCMap &cmap,
+                                         const std::uint32_t value,
+                                         const std::uint8_t width) {
+  const std::pair key{width, value};
+  std::uint32_t low = 0;
+  std::uint32_t high = cmap.range_count;
+  while (low < high) {
+    const std::uint32_t mid = low + (high - low) / 2;
+    const cid_data::CidRange &range =
+        cid_data::cid_range_pool[cmap.ranges[mid]];
+    if (std::pair{range.width, range.code_low} <= key) {
+      low = mid + 1;
+    } else {
+      high = mid;
+    }
+  }
+  if (low == 0) {
     return std::nullopt;
   }
-  return *codec == UnicodeCodec::utf32be ? decode_utf32be(codes)
-                                         : decode_utf16be(codes);
+  const cid_data::CidRange &range =
+      cid_data::cid_range_pool[cmap.ranges[low - 1]];
+  if (range.width == width && value >= range.code_low &&
+      value <= range.code_low + range.run_len) {
+    return static_cast<std::uint32_t>(range.cid_low) + (value - range.code_low);
+  }
+  return std::nullopt;
+}
+
+const cid_data::Collection *find_collection(const std::string_view registry,
+                                            const std::string_view ordering) {
+  for (const cid_data::Collection &collection : cid_data::collections) {
+    if (collection.registry == registry && collection.ordering == ordering) {
+      return &collection;
+    }
+  }
+  return nullptr;
+}
+
+/// CID -> Unicode within a collection: the presence bitmap gates the CID, then
+/// `rank(cid)` (words before + popcount within the word) indexes the dense
+/// value array; a `0xffff` sentinel escapes to the astral side table.
+std::optional<char32_t>
+collection_cid_to_unicode(const cid_data::Collection &collection,
+                          const std::uint32_t cid) {
+  if (cid > collection.max_cid) {
+    return std::nullopt;
+  }
+  const std::uint32_t word = cid >> 6;
+  const std::uint64_t bit = std::uint64_t{1} << (cid & 63);
+  if ((collection.bitmap[word] & bit) == 0) {
+    return std::nullopt;
+  }
+  const auto index = collection.rank[word] +
+                     static_cast<std::uint32_t>(
+                         std::popcount(collection.bitmap[word] & (bit - 1)));
+  const std::uint16_t value = collection.values[index];
+  if (value != 0xffff) {
+    return static_cast<char32_t>(value);
+  }
+  const auto *const begin = collection.astral;
+  const auto *const end = begin + collection.astral_count;
+  const auto *const it = std::lower_bound(
+      begin, end, index,
+      [](const cid_data::AstralEntry &entry, const std::uint32_t target) {
+        return entry.value_index < target;
+      });
+  return it != end && it->value_index == index ? std::optional{it->codepoint}
+                                               : std::nullopt;
 }
 
+/// Split a code string through a named legacy CMap and translate each code
+/// code -> CID -> Unicode; unmapped codes contribute nothing.
+std::string translate_legacy_cmap(const cid_data::PredefinedCMap &cmap,
+                                  const std::string &codes) {
+  const cid_data::Collection &collection =
+      cid_data::collections[cmap.collection];
+  std::string result;
+  std::size_t pos = 0;
+  while (pos < codes.size()) {
+    const std::size_t width =
+        code_width(cmap, std::string_view(codes).substr(pos));
+    std::uint32_t value = 0;
+    for (std::size_t k = 0; k < width; ++k) {
+      value = (value << 8) | byte(codes, pos + k);
+    }
+    pos += width;
+
+    if (const std::optional<std::uint32_t> cid =
+            code_to_cid(cmap, value, static_cast<std::uint8_t>(width));
+        cid.has_value()) {
+      if (const std::optional<char32_t> unicode =
+              collection_cid_to_unicode(collection, *cid);
+          unicode.has_value()) {
+        util::string::append_c32(*unicode, result);
+      }
+    }
+  }
+  return result;
+}
+
+} // namespace
+
 } // namespace odr::internal::pdf
+
+namespace odr::internal {
+
+std::optional<std::string>
+pdf::translate_predefined_cmap(const std::string_view name,
+                               const std::string &codes) {
+  if (const std::optional<UnicodeCodec> codec = classify(name);
+      codec.has_value()) {
+    return *codec == UnicodeCodec::utf32be ? decode_utf32be(codes)
+                                           : decode_utf16be(codes);
+  }
+  if (const cid_data::PredefinedCMap *const cmap = find_predefined_cmap(name);
+      cmap != nullptr) {
+    return translate_legacy_cmap(*cmap, codes);
+  }
+  return std::nullopt;
+}
+
+std::optional<char32_t> pdf::cid_to_unicode(const std::string_view registry,
+                                            const std::string_view ordering,
+                                            const std::uint32_t cid) {
+  if (const cid_data::Collection *const collection =
+          find_collection(registry, ordering);
+      collection != nullptr) {
+    return collection_cid_to_unicode(*collection, cid);
+  }
+  return std::nullopt;
+}
+
+} // namespace odr::internal