PDF: legacy CJK CMaps — code → CID → Unicode for composite fonts#593
Merged
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2285c936d0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
andiwand
force-pushed
the
pdf-link-annotations
branch
from
5346eef to
803c25d
Compare
andiwand
force-pushed
the
pdf-cjk-legacy-cmaps
branch
3 times, most recently
from
636da54 to
dabc445
Compare
A composite (Type0) font with no /ToUnicode whose /Encoding is a legacy
CJK CMap (90ms-RKSJ-H, GBK-EUC-H, B5pc-H, KSC-EUC-H, …) previously
yielded "no Unicode" — the glyphs rendered (PUA re-encode) but the text
was not selectable. This closes that gap.
generate_cid_data.py now emits pdf_cid_data.{hpp,cpp} (the deferred
tables): interned code → CID ranges with per-CMap uint16 index lists, and
a per-collection CID → Unicode presence bitmap + rank index + uint16 value
array (astral escape). This is the "S3" packing from the storage analysis
— ~0.58 MB compiled, directly indexable, no new dependency. Data is
Adobe's cmap-resources (BSD-3-Clause), fetched by the script, not vendored.
pdf_cid gains the lookup: translate_predefined_cmap handles the named
legacy CMaps (split by codespace → code → CID by binary search over the
interned ranges → CID → Unicode by bitmap rank), and cid_to_unicode maps a
CID through the /CIDSystemInfo collection for the Identity-H/V and
embedded-CMap-stream cases (code == CID, collection from the descendant
CIDFont). Font::to_unicode wires both in, guarding the identity path so a
named CMap we lack tables for is never misread as identity CIDs.
Codespace splitting matches per byte (ISO 32000-1 9.7.6.2), not just the
leading byte, so GB18030's overlapping 2- and 4-byte ranges (both leading
0x81-0xFE) pick the right width — e.g. GBK2K-H `81 30 84 36` → CID 22354
→ U+00A5 rather than two mis-split 2-byte codes.
Generated arrays are std::array, and the exported definitions are
qualified (cid_data::…) in the parent namespace so a header rename fails to
compile rather than defining a stray symbol — matching pdf_encoding_data.
Tests (assertion-based, vectors derived from the Adobe data and frozen
inline): Shift-JIS and EUC code → CID → Unicode, a mixed 1-/2-byte
codespace, the GB18030 four-byte case, cid_to_unicode by collection incl.
an astral escape, and the nullopt for Identity-H / an unshipped name.
A visual walkthrough of the path and tables is added at
docs/design/pdf-cjk-code-cid-unicode.html.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
andiwand
force-pushed
the
pdf-cjk-legacy-cmaps
branch
from
dabc445 to
a9f6723
Compare
Both collections ship a Uni* CMap, so they invert to CID -> Unicode tables like the existing five; add them to generate_cid_data.py's _COLLECTIONS and regenerate (7 collections, 136 predefined CMaps, ~+54 KB payload). Hojo also brings legacy code -> CID CMaps (Hojo-H/V, Hojo-EUC-H/V). Adobe-Identity-0 stays out by necessity: it has no Uni* CMap to invert. No runtime changes: lookups are by name / (registry, ordering) against the generated tables. Adds pdf_cid tests for the Hojo legacy path and the Manga1/Japan2 collection lookups. Drops the now-resolved "CMap coverage" gap from AGENTS.md. Also tidies the generator (drop the always-true std_array flag) and Font::to_unicode (use !result.empty() instead of a separate mapped flag). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Generated with Claude Code
Stacked on #591. Closes the deferred legacy CJK CMap selectability gap for the in-house PDF engine.
Problem
A composite (Type0) font with no
/ToUnicodewhose/Encodingis a legacy CJK CMap (90ms-RKSJ-H,GBK-EUC-H,B5pc-H,KSC-EUC-H, …) — orIdentity-H/Vwith a real/CIDSystemInfo— previously yielded "no Unicode". The glyphs rendered correctly (PUA re-encode), but the text was not selectable or searchable. This was the last open item in CMap coverage.What lands
pdf_cid_data.{hpp,cpp}, fromgenerate_cid_data.py).130 legacy CMaps across Adobe-Japan1/-GB1/-CNS1/-Korea1/-KR. Packed as the "S3" tier from the storage analysis (0.58 MB compiled, directly indexable, no new dependency):code → CIDranges (73% cross-CMap dedup) + per-CMapuint16index lists;CID → Unicodeas a presence bitmap + rank index +uint16value array (astral escape for the ~1% > U+FFFF).cmap-resources(BSD-3-Clause) — fetched by the script, not vendored; already attributed inTHIRD_PARTY_LICENSES.md.pdf_cid):translate_predefined_cmapnow resolves named legacy CMaps (codespace split →code → CIDbinary search over the interned ranges →CID → Unicodebitmap rank); newcid_to_unicode(registry, ordering, cid)covers theIdentity-H/V+ embedded-CMap-stream case (code == CID, collection from/CIDSystemInfo).Font::to_unicode): tries the named CMap, then the/CIDSystemInfocollection — guarded so a named CMap we lack tables for is never misread as identity CIDs. Falls through to the embedded reverse map as before.std::array, exported definitions qualified (cid_data::…) in the parent namespace so a header rename fails to compile — matchingpdf_encoding_data.Display was never affected (glyphs already render via the font); this is purely a selectability/search improvement.
Tests
pdf_cid.cpp(assertion-based, vectors derived from the Adobe data and frozen inline): Shift-JIS + EUCcode → CID → Unicode, a mixed 1-/2-byte codespace,cid_to_unicodeby collection incl. an astral escape, andnulloptforIdentity-H/ an unshipped name. Full suite green (425 unit + 49 PDF HTML reference-output); no corpus fixture hits the path, so no snapshot regen.