diff --git a/AGENTS.md b/AGENTS.md index 53a530a5..f7ed8404 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,17 +1,14 @@ # AGENTS.md — OpenDocument.core -Orientation for AI agents working in this repo. Summarises the architecture, the -conventions, and where to find things. For user-facing docs see -[`README.md`](README.md) and [`docs/`](docs/README.md). +Orientation for AI agents. Architecture, conventions, and where things live. For +user-facing docs see [`README.md`](README.md) and [`docs/`](docs/README.md). ## What this is `odr` (a.k.a. `odrcore`) is a **C++20 library that decodes documents and renders -them to HTML**. It reads many formats (ODF, OOXML, legacy MS, PDF, CSV, …) behind -one abstract document model and a generic HTML renderer. It is the backend for -OpenDocument.droid / .ios. - -Build system: **CMake + Conan**. Language standard: **C++20** (`CMakeLists.txt`). +them to HTML**. It reads many formats (ODF, OOXML, legacy MS binary, PDF, CSV, …) +behind one abstract document model and a generic HTML renderer. It is the backend +for OpenDocument.droid / .ios. Build: **CMake + Conan**; standard: **C++20**. ## Big picture: how a file becomes HTML @@ -21,162 +18,144 @@ bytes ─▶ magic/open_strategy ─▶ DecodedFile ─▶ Document ─▶ Eleme DecoderEngine) format) elements) walks public API) ``` -1. **Detection** — `internal/magic.cpp` (+ `internal/libmagic`) sniffs the file; - `internal/open_strategy.cpp` picks a `FileType` and a `DecoderEngine` and - constructs the matching `abstract::DecodedFile`. -2. **Decode** — a document file yields an `abstract::Document` (the engine's - subclass of `internal::Document`). +1. **Detect** — `internal/magic.cpp` (+ `internal/libmagic`) sniffs the file; + `internal/open_strategy.cpp` picks a `FileType` + `DecoderEngine` and builds + the matching `abstract::DecodedFile`. +2. **Decode** — a document file yields an `abstract::Document`. 3. **Element tree** — a `Document` exposes a root `ElementIdentifier` plus an - `abstract::ElementAdapter`. The public, value-semantics handles - (`Element`, `Slide`, `Paragraph`, `Text`, `Frame`, …) in - `src/odr/document_element.hpp` are thin wrappers that delegate to the adapter. -4. **Render** — `internal/html/` walks that public element API and writes HTML. - Entry point: `odr::html::translate(...)` → `HtmlService` (paginated fragments; + `abstract::ElementAdapter`. Public value-semantics handles (`Element`, `Slide`, + `Paragraph`, `Text`, `Frame`, …) in `src/odr/document_element.hpp` are thin + wrappers that delegate to the adapter. +4. **Render** — `internal/html/` walks the public element API and writes HTML. + Entry: `odr::html::translate(...)` → `HtmlService` (paginated fragments; `bring_offline` materialises files). -### The element-adapter pattern (every document engine follows it) - -Two pieces per engine: +### The element-adapter pattern (every engine follows it) -- An **`ElementRegistry`**: a flat `std::vector` (id = index + 1) where - each `Element` holds `parent`/`first_child`/`last_child`/`prev`/`next` ids and a - `type`, plus side `unordered_map`s for per-type payloads (text strings, frame - anchors, …). Builders are `create_element` / `create_*_element` / - `append_child`. See `oldms/text/doc_element_registry.*` or - `oldms/presentation/ppt_element_registry.*` for the minimal version. -- An **`ElementAdapter`**: one class that implements `abstract::ElementAdapter` - (tree navigation by id) and, by multiple inheritance, the per-element-type - adapters it supports (`SlideAdapter`, `ParagraphAdapter`, `TextAdapter`, - `FrameAdapter`, …). The `*_adapter(id)` methods return `this` when the element - is of that type, else `nullptr`. See `oldms/presentation/ppt_document.cpp` for a - compact example. +- An **`ElementRegistry`**: a flat `std::vector` (id = index + 1); each + `Element` holds `parent`/`first_child`/`last_child`/`prev`/`next` ids and a + `type`, plus side maps for per-type payloads. Builders: `create_element` / + `create_*_element` / `append_child`. Minimal example: + `oldms/presentation/ppt_element_registry.*`. +- An **`ElementAdapter`**: one class implementing `abstract::ElementAdapter` (tree + navigation by id) and, via multiple inheritance, the per-type adapters it + supports (`SlideAdapter`, `ParagraphAdapter`, …). Each `*_adapter(id)` returns + `this` when the element is that type, else `nullptr`. Compact example: + `oldms/presentation/ppt_document.cpp`. -`ElementType` is the shared enum in `src/odr/document_element.hpp` (`root`, -`slide`, `paragraph`, `text`, `line_break`, `frame`, `table*`, `sheet*`, …). +`ElementType` is the shared enum in `src/odr/document_element.hpp`. ## Directory map | Path | What | |------|------| -| `src/odr/*.hpp` | **Public API**: `file.hpp`, `document.hpp`, `document_element.hpp`, `html.hpp`, `style.hpp`, `quantity.hpp` (`Measure`), `odr.hpp`. | -| `src/odr/internal/abstract/` | Core interfaces: `File`/`DecodedFile`, `Document` + `ElementAdapter` (and all per-element adapters), `Filesystem`, `Archive`, `HtmlService`. | +| `src/odr/*.hpp` | **Public API**: `file`, `document`, `document_element`, `html`, `style`, `quantity` (`Measure`), `odr`. | +| `src/odr/internal/abstract/` | Core interfaces: `File`/`DecodedFile`, `Document` + `ElementAdapter`, `Filesystem`, `Archive`, `HtmlService`. | | `src/odr/internal/common/` | Reusable impls: `Path`/`AbsPath`, base `Document`, filesystem, `style`, table cursor/range, temp files. | -| `src/odr/internal/util/` | Helpers: `byte_stream_util` (POD reads), `string_util` (`split`, `u16string_to_string`), `stream_util`, `document_util`, `xml_util`. | -| `src/odr/internal/magic.*`, `open_strategy.*` | File-type detection and the open/dispatch logic. | -| `src/odr/internal/html/` | Generic HTML renderer (`document.cpp`, `document_element.cpp`, `document_style.cpp`). | -| `src/odr/internal/cfb/`, `zip/` | Container formats (Compound File Binary, ZIP). | +| `src/odr/internal/util/` | Helpers: `byte_stream_util`, `string_util`, `stream_util`, `document_util`, `xml_util`. | +| `src/odr/internal/magic.*`, `open_strategy.*` | File-type detection + open/dispatch. | +| `src/odr/internal/html/` | Generic HTML renderer. | +| `src/odr/internal/cfb/`, `zip/` | Container formats (CFB, ZIP). | | `src/odr/internal/odf/` | OpenDocument (odt/ods/odp/odg). | -| `src/odr/internal/ooxml/` | OOXML (docx/pptx/xlsx); subdirs `text`/`presentation`/`spreadsheet`. | -| `src/odr/internal/oldms/` | **Legacy MS binary** (.doc/.ppt/.xls); subdirs `text`/`presentation`/`spreadsheet`. | +| `src/odr/internal/ooxml/` | OOXML (docx/pptx/xlsx). | +| `src/odr/internal/oldms/` | **Legacy MS binary** (.doc/.ppt/.xls). | | `src/odr/internal/oldms_wvware/` | Alternative .doc decoder via wvWare. | | `src/odr/internal/pdf/`, `pdf_poppler/` | PDF (own parser + poppler/pdf2htmlEX path). | | `src/odr/internal/{csv,json,text,svm}/` | Smaller formats. | | `cli/src/` | CLI tools: `translate`, `back_translate`, `meta`, `server`. | -| `tools/pdf/` | Developer tooling (not built): generators for the PDF engine's committed encoding data, see [`tools/pdf/README.md`](tools/pdf/README.md). | -| `test/src/` | GoogleTest suites; data in `test/data` (git submodules, see below). | -| `offline/documentation/MS-*/` | Vendored Microsoft spec text (PDF + extracted markdown), see [Specs](#specs). | +| `tools/pdf/` | Dev tooling (not built): PDF encoding-data generators, see `tools/pdf/README.md`. | +| `test/src/` | GoogleTest suites; data in `test/data` (git submodules). | +| `offline/documentation/MS-*/` | Vendored Microsoft spec text (see [Specs](#specs)). | | `docs/design/README.md` | High-level design rationale. | ## Build & test -A configured build dir already exists (`cmake-build-relwithdebinfo`, also `…-debug`, -`…-release`). Typical loop: +Configured build dirs already exist. Typical loop: ```bash -# library -cmake --build cmake-build-relwithdebinfo --target odr -# tests (the ODR_TEST option is on in this build dir) -cmake --build cmake-build-relwithdebinfo --target odr_test -# run from inside the build dir so any test output stays out of the repo tree +cmake --build cmake-build-relwithdebinfo --target odr # library +cmake --build cmake-build-relwithdebinfo --target odr_test # tests (ODR_TEST on) (cd cmake-build-relwithdebinfo && ./test/odr_test --gtest_filter='OldMs.*') -# CLI (renders a file to a directory of HTML) -cmake --build cmake-build-relwithdebinfo --target translate +cmake --build cmake-build-relwithdebinfo --target translate # CLI: file → HTML dir ``` -- **Use `cmake-build-relwithdebinfo`** as the default build/test directory (not - `cmake-build-debug`). -- **Only run the full suite when really necessary** — it takes a while. Default - to a targeted `--gtest_filter` for the area you touched. -- **For debugging, prefer the `translate` CLI** over the test suite — build and - run it on a single file to reproduce/inspect behaviour quickly. -- **Run the test binary from the build directory** (as above) so any files it - writes land there, not in the repo working tree. - -Notable CMake options (`CMakeLists.txt`): `ODR_TEST`, `ODR_CLI`, -`ODR_WITH_PDF2HTMLEX`, `ODR_WITH_WVWARE`, `ODR_WITH_LIBMAGIC`, `ODR_CLANG_TIDY`. -A new `.cpp` must be added to the `ODR_SOURCE_FILES` list in `CMakeLists.txt`. - -**Test data lives in git submodules** under `test/data/input/odr-public`, -`…/odr-private`, and `test/data/reference-output/*`. +- **Default to `cmake-build-relwithdebinfo`** (not `…-debug`). +- **Run only a targeted `--gtest_filter`**; the full suite is slow — run it only + when really necessary. +- **Run the test binary from the build dir** so output stays out of the repo tree. +- **For debugging, prefer the `translate` CLI** on a single file over the suite. +- CMake options (`CMakeLists.txt`): `ODR_TEST`, `ODR_CLI`, `ODR_WITH_PDF2HTMLEX`, + `ODR_WITH_WVWARE`, `ODR_WITH_LIBMAGIC`, `ODR_CLANG_TIDY`. A new `.cpp` must be + added to `ODR_SOURCE_FILES`. +- **Test data lives in git submodules** under `test/data/`. ## Conventions -- **Formatting**: clang-format, LLVM-based (`.clang-format`); run `scripts/format` - (or rely on the git hook from `scripts/setup`). `clang-tidy` config in - `.clang-tidy`; CI enforces both (`.github/workflows/format.yml`, `tidy.yml`). -- **Error handling — fail fast**: where the spec/format dictates what to expect, - **throw** on unexpected input (`std::runtime_error`, or the typed exceptions in - `src/odr/exceptions.hpp`) rather than silently degrading. Only **pass through** - (return empty / skip) values that are genuinely *optional* or *not yet - modelled*. -- **Public API**: value semantics; immutable handles; iterators only for - immutable traversal (`docs/design/README.md`). -- **Byte parsing**: read POD structs via `util::byte_stream::read`; this assumes - host byte order matches the file's (little-endian) — big-endian is a known - not-yet-handled gap in the binary engines. -- Match the **surrounding file's** style, includes, and idioms; mirror a sibling - engine when adding a format (the `oldms/text` `.doc` impl is the reference the - `.ppt` impl was modelled on). -- **Comments — keep them minimal**: a function/struct doc comment is at most a - couple of terse lines stating the key point (what it does, stream/ownership - preconditions, the spec section it implements, e.g. `[MS-PPT] 2.3.2`). Don't - restate the code or spell out every case; cite the spec instead of paraphrasing - it. The detailed design rationale belongs in the per-module `AGENTS.md`, not in - source comments. -- **Doc-comment markers**: document functions, classes, structs and enums with - `///` doc comments; use a trailing `///<` for the short note on the same line - (e.g. an enumerator or member). Keep them terse per the rule above. -- **Fixed-width integer types**: prefer `` types (`std::uint8_t`, - `std::uint32_t`, …) over `unsigned char` / `unsigned` / `long` and friends - whenever the width matters (byte buffers, on-disk/wire fields, bit math). +- **Formatting**: clang-format (LLVM-based, `.clang-format`); run `scripts/format` + or use the `scripts/setup` git hook. `clang-tidy` per `.clang-tidy`. CI enforces + both. +- **Fail fast**: where the spec dictates what to expect, **throw** on unexpected + input (`std::runtime_error` or the typed exceptions in `src/odr/exceptions.hpp`) + rather than silently degrading. Only pass through (return empty / skip) values + that are genuinely *optional* or *not yet modelled*. +- **Fixed-width integer types — always**: prefer `` types (`std::int32_t`, + `std::uint8_t`, …) over `int` / `unsigned` / `long` / `unsigned char`. Reserve + the built-in types for genuinely index/size-like values (`std::size_t`) or where + an API forces them. +- **Prefer ranges**: use `std::ranges` algorithms and range-based overloads over + iterator pairs (`std::ranges::find_if(v, …)` not `std::find_if(v.begin(), …)`), + and prefer range views/`for (auto &x : range)` over manual iterator loops. Fall + back to iterator pairs only where a range overload genuinely doesn't fit + (e.g. reverse-iterator `.base()` tricks). +- **Bind free-function definitions to a namespace**: in a source file, define a + header-declared free function (or util struct's static method) with its + **qualified** name — the `Ret ns::fn(...)` / `Ret Struct::fn(...)` form, inside + the reopened `namespace` — never as a bare unqualified redeclaration. A qualified + out-of-line definition must match an existing declaration, so a signature that + drifts from the header fails at **compile** time instead of becoming an obscure + linker error. (The `util` helpers use the `struct string { static … }` idiom for + exactly this.) Keep translation-unit-local helpers in an **anonymous namespace**. +- **Public API**: value semantics; immutable handles; iterators only for immutable + traversal (`docs/design/README.md`). +- **Byte parsing**: read POD structs via `util::byte_stream::read`; assumes host + byte order matches the file's (little-endian) — big-endian is a known gap. +- **Match the surrounding file** and mirror a sibling engine when adding a format + (the `oldms/text` `.doc` impl is the reference the `.ppt` impl was modelled on). +- **Comments — minimal**: a doc comment is at most a couple of terse lines stating + the key point (what it does, stream/ownership preconditions, spec section, e.g. + `[MS-PPT] 2.3.2`). Don't restate the code; cite the spec, don't paraphrase it. + Detailed rationale belongs in the per-module `AGENTS.md`. +- **Doc-comment markers**: `///` for functions/classes/structs/enums; trailing + `///<` for the short note on the same line (enumerator/member). Keep terse. - **Pull requests**: put the `🤖 Generated with [Claude Code](https://claude.com/claude-code)` line **at the top** of the PR body. ## Adding / extending a document format -1. Detection: extend `magic`/`open_strategy` to map the bytes to a `FileType` +1. Detection: extend `magic`/`open_strategy` to map bytes → `FileType` (+ `DecoderEngine`) and construct your `DecodedFile`. 2. For documents: subclass `internal::Document`; in its constructor build an - `ElementRegistry` and an `ElementAdapter` (see the pattern above). + `ElementRegistry` and an `ElementAdapter` (pattern above). 3. Implement the per-element adapters you can populate; the **generic HTML renderer then works for free**. -4. Register the format's factory (e.g. `oldms_file.cpp::document()` switches on - `file_type()`), add sources to `CMakeLists.txt`, and add a GoogleTest. +4. Register the factory (e.g. `oldms_file.cpp::document()` switches on + `file_type()`), add sources to `CMakeLists.txt`, add a GoogleTest. ## Legacy Microsoft binary formats (`oldms`) -Container handling (CFB) already exists; each format is a small module under -`oldms/` mirroring `oldms/text` (`.doc`). Spec references in -`src/odr/internal/oldms/README.md`. - -- **`.doc`** (`oldms/text`): working, visible-text extraction. -- **`.ppt`** (`oldms/presentation`): implemented — slides resolved via the - persist directory (the only spec-defined read path), each slide's text boxes - modelled as positioned `frame`s. **Read its docs before touching it**: - [`oldms/presentation/AGENTS.md`](src/odr/internal/oldms/presentation/AGENTS.md) - — what's implemented and **why** (persist-directory resolution, no scan - fallback, sequential `ChildCursor` reading without `tellg`, fail-fast error - handling, the two-text-locations finding, endianness), the open work (frame - refinements, smaller shortcomings), and the verified `[MS-PPT]`/`[MS-ODRAW]` - drawing-tree map. -- **`.xls`** (`oldms/spreadsheet`): working, visible cell-text extraction - (BIFF8). See [`oldms/spreadsheet/AGENTS.md`](src/odr/internal/oldms/spreadsheet/AGENTS.md). +CFB container handling exists; each format is a small module under `oldms/` +mirroring `oldms/text` (`.doc`), all doing visible-text extraction only. Shared +conventions + the endianness analysis are in +[`oldms/AGENTS.md`](src/odr/internal/oldms/AGENTS.md); spec refs in +`src/odr/internal/oldms/README.md`. **Read the module's own `AGENTS.md` before +touching it** — each carries the design rationale, spec-record maps, and open work: +`text/` (`.doc`), `presentation/` (`.ppt`, BIFF-style drawing tree), +`spreadsheet/` (`.xls`, BIFF8). ## Specs -Vendored Microsoft Open Specifications live under -`offline/documentation//-/`, both as `original.pdf` and an -extracted `docling-from-docx.md` (grep-friendly). Available: **MS-PPT**, -**MS-ODRAW** (Office Art / Escher drawing records), **MS-DOC**, **MS-XLS**, -**MS-CFB** (container), **MS-OFFCRYPTO** (encryption). Cite section numbers from -these when implementing binary parsing. +Vendored Microsoft Open Specifications under +`offline/documentation//-/` as `original.pdf` + an extracted +`docling-from-docx.md` (grep-friendly). Available: **MS-PPT**, **MS-ODRAW**, +**MS-DOC**, **MS-XLS**, **MS-CFB**, **MS-OFFCRYPTO**. Cite section numbers when +implementing binary parsing. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..7e721c60 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,10 @@ +# CLAUDE.md + +**Read [`AGENTS.md`](AGENTS.md) first, before doing anything else in this repo.** +It is the single source of truth for the architecture, directory layout, build & +test loop, and the coding conventions you are required to follow here. Do not +propose changes or write code until you have read it. + +Per-module `AGENTS.md` files (e.g. `src/odr/internal/oldms/presentation/AGENTS.md`) +carry deeper, area-specific rules — read the relevant one before touching that +module, as `AGENTS.md` instructs. diff --git a/src/odr/internal/oldms/presentation/AGENTS.md b/src/odr/internal/oldms/presentation/AGENTS.md index 223c3f8a..1c77c036 100644 --- a/src/odr/internal/oldms/presentation/AGENTS.md +++ b/src/odr/internal/oldms/presentation/AGENTS.md @@ -1,330 +1,133 @@ -# `.ppt` (PowerPoint) support — status, design & open work +# `.ppt` (PowerPoint) support — design & open work -What the `oldms/presentation/` module does **today**, the **design decisions** -behind it, and the **open work**. Shared `oldms/` conventions are in -[`../AGENTS.md`](../AGENTS.md). +The **why** and the roadmap; what is concretely done lives in the code. Shared +`oldms/` conventions are in [`../AGENTS.md`](../AGENTS.md). **Scope.** Extract the **visible text of each slide, positioned in its text -boxes**, and expose it through the abstract document model so the generic HTML -renderer lays each slide out as positioned frames. No character/paragraph -styles, master/notes pages, images, charts, tables, or animations. +boxes**, through the abstract model so the generic HTML renderer lays each slide +out as positioned frames. No character/paragraph styles, master/notes pages, +images, charts, tables, or animations. -**Specs.** `offline/documentation/MS-PPT/` (the PowerPoint stream) and -`MS-ODRAW/` (the Office Art / Escher drawing records). CFB container handling -already existed in the engine. - ---- - -## What works - -- `.ppt` is detected and decoded to a `Document` (presentation), one `slide` - per presentation slide **in presentation order**. -- Each slide's on-slide **text boxes** become positioned `frame`s; their text is - split into paragraphs / line breaks. -- A text box that stores no inline text but an `OutlineTextRefAtom` (the common - PowerPoint placeholder representation) is resolved against the slide's text in - the `SlideListWithTextContainer`, so placeholder/body text is not lost. -- The generic HTML renderer produces one page per slide with each text box - absolutely positioned (verified: `position:absolute;left:…;top:…` with the - decoded coordinates). +**Specs.** `[MS-PPT]` (the PowerPoint stream) + `[MS-ODRAW]` (the Office Art / +Escher drawing records) + `[MS-CFB]` container. Section numbers cited inline in +code and in the drawing-tree map below. ## Module layout (mirrors `../text`) -| File (`oldms/presentation/`) | Role | -|----------------------------------|-----------------------------------------------------| -| `ppt_structs.hpp` | `#pragma pack(1)` PODs (`RecordHeader`, atom bodies, `Anchor`) + `static_assert` sizes + the `RecordType` / `SlideListInstance` enums | -| `ppt_io.{hpp,cpp}` | `read(...)` helpers over `std::istream` (text atoms, the anchor rect, fixed structs) | -| `ppt_parser.{hpp,cpp}` | `parse_tree(registry, files)` → walks the stream and builds the element tree | -| `ppt_element_registry.{hpp,cpp}` | Flat element store (copy of `doc_element_registry`) + text & frame side-payloads | -| `ppt_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | - -`ElementRegistry` is a `vector` (id = index) with parent/child/sibling -ids and side maps for the text and frame payloads; `create_element` / -`create_text_element` / `create_frame_element` / `append_child` are the only -builders. - -## Pipeline: how a `.ppt` becomes the element tree - -1. **Wiring.** `LegacyMicrosoftFile` already detected `.ppt` (the `/PowerPoint - Document` stream → `FileType::legacy_powerpoint_presentation`, - `DocumentType::presentation`) and `open_strategy` routed it here; the - `legacy_powerpoint_presentation` case in `LegacyMicrosoftFile::document()` - returns `presentation::Document`. -2. **Resolve slides (persist directory).** `parse_tree` opens both required - streams and hands them to `collect_slides(current_user, document)`, following - the `[MS-PPT]` reading algorithm: read `CurrentUserAtom` from `/Current User` - → walk the `UserEditAtom` chain newest→oldest, building the persist object - directory (newest offset per id wins) → resolve the **live** - `DocumentContainer` via `docPersistIdRef` → walk the slide list's - `SlidePersistAtom`s **in presentation order**, resolving each `persistIdRef` - to its `SlideContainer`. See *Design decisions* for why this is the only read - path. -3. **Read text boxes per slide.** For each `SlideContainer` the parser descends - the drawing and reads its text boxes (with positions) — see [Text boxes - (frames)](#text-boxes-frames). -4. **Build the tree.** `parse_tree` makes one `slide`, one `frame` per text box - (storing its anchor), and `build_paragraphs` hangs the box's text off the - frame: - - ``` - root (ElementType::root) - └── slide (ElementType::slide) one per slide, in order - └── frame (ElementType::frame) one per on-slide text box - └── paragraph (ElementType::paragraph) split on 0x0D - ├── text (ElementType::text) - └── line_break (ElementType::line_break) for 0x0B in a paragraph - ``` -5. **Render.** HTML works through the generic renderer via the public `Slide` / - `Frame` / `Paragraph` / `Text` API and our adapters. - -## Text boxes (frames) - -A `.ppt` slide is a *drawing of shapes*; each text box / placeholder is a shape -with its own position. `collect_slides` returns, per slide, the on-slide text -boxes in shape (z) order, each becoming a `frame`. - -Per slide the parser descends `SlideContainer → DrawingContainer (0x040C) → -OfficeArtDgContainer (0xF002) → OfficeArtSpgrContainer (0xF003)` and walks the -`OfficeArtSpContainer` (0xF004) shapes. For each shape it reads: -- the **optional** `OfficeArtClientAnchor` (0xF010) → `read_client_anchor` - (`SmallRectStruct`/`RectStruct`, master units = 1/576 inch), and -- the text in its `OfficeArtClientTextbox` (0xF00D). - -Shapes with no text are dropped, so the group shape and pictures disappear. -`FrameAdapter` returns `anchor_type = at_page` and `x/y/width/height` as Measures -(master units / 576 → inches); a shape without an anchor yields a frame with no -position. - -**First cut (current):** only **top-level** shapes — direct children of the root -`OfficeArtSpgrContainer`, whose anchors are already in the slide's master-unit -system. Nested-group coordinate transforms, non-grouped shapes, and -master-placeholder geometry inheritance are deferred — see [open -work](#1-frame-refinements). The verified record map of the drawing tree is in -[Reference](#reference-the-drawing-tree). - -## Adapters - -`ppt_document.cpp` implements the generic `ElementAdapter` (tree navigation, -copied from `doc_document.cpp`) plus `SlideAdapter` / `FrameAdapter` / -`ParagraphAdapter` / `TextAdapter` / `LineBreakAdapter`: -- `FrameAdapter`: `anchor_type = at_page`; `x/y/width/height` from the frame's - anchor (or empty when absent); `z_index` / `style` empty. -- `SlideAdapter`: `slide_page_layout` → hardcoded 10"×7.5" (4:3); `slide_name` → - empty; `slide_master_page` → `null_element_id`. -- `paragraph_text_style` / `text_style` set `font_size = 11pt` so empty - paragraphs still have height. -- `Document::is_editable()` → `false`; `save(...)` → throws - `UnsupportedOperation`. - -## Binary format reference - -Every record starts with an 8-byte `RecordHeader`: - -``` -RecordHeader { - uint16 recVer : 4 ; // 0xF marks a container - uint16 recInstance : 12 ; - uint16 recType ; - uint32 recLen ; // bytes of body that follow the header -} -``` - -`recVer == 0xF` marks a **container** (body is a sequence of records); otherwise -it's an **atom** with `recLen` bytes of payload. - -| Record | Type | Kind | Purpose | -|------------------------|--------|-----------|------------------------------------------| -| `CurrentUserAtom` | 0x0FF6 | atom | in `/Current User`; newest edit offset | -| `UserEditAtom` | 0x0FF5 | atom | edit chain + persist directory offset | -| `PersistDirectoryAtom` | 0x1772 | atom | persist id → stream offset | -| `DocumentContainer` | 0x03E8 | container | top-level document | -| `SlideListWithText` | 0x0FF0 | container | per-list slide refs (+ optional outline) | -| `SlidePersistAtom` | 0x03F3 | atom | one per slide; `persistIdRef` + order | -| `SlideContainer` | 0x03EE | container | a slide (drawing + placeholders) | -| `MainMaster` | 0x03F8 | container | master slide (skipped) | -| `Notes` | 0x03F0 | container | notes page (skipped) | -| `TextHeaderAtom` | 0x0F9F | atom | type of the text block that follows | -| `TextCharsAtom` | 0x0FA0 | atom | UTF-16 text (two bytes per code unit) | -| `TextBytesAtom` | 0x0FA8 | atom | "compressed" text: one byte per char | - -The Office Art drawing records (`RT_Drawing` 0x040C and `0xF00*`/`0xF010`) used -for text boxes are listed with the full drawing-tree map in -[Reference](#reference-the-drawing-tree). - -### Text decoding - -- `TextCharsAtom`: `recLen / 2` UTF-16 code units → `u16string_to_string`. -- `TextBytesAtom`: each byte is one character value (0x00–0xFF). -- In-text control characters: `0x0D` = paragraph break, `0x0B` = vertical tab = - manual line break — split on these like `doc_parser`. `0x09` (tab) kept; other - control characters dropped (`clean_text`). +| File (`oldms/presentation/`) | Role | +|---|---| +| `ppt_structs.hpp` | `#pragma pack(1)` PODs (`RecordHeader`, atom bodies, `Anchor`) + `static_assert` sizes + `RecordType` / `SlideListInstance` enums | +| `ppt_io.{hpp,cpp}` | `read(...)` helpers over `std::istream` | +| `ppt_parser.{hpp,cpp}` | `parse_tree(registry, files)` → walks the stream, builds the element tree | +| `ppt_element_registry.{hpp,cpp}` | Flat element store + text & frame side-payloads | +| `ppt_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | ---- +Element tree shape: `root → slide → frame (one per text box) → paragraph (split on +0x0D) → text / line_break (0x0B)`. ## Design decisions -**Slide resolution is persist-directory based (the single spec path).** The +**Slide resolution is persist-directory based — the single spec path.** The persist directory gives correct slide **ordering** for incrementally-saved files -(where stream order ≠ presentation order) and picks the **live** -`DocumentContainer` rather than the first one in the stream. Verified on -`slides.ppt`: `/Current User` → `offsetToCurrentEdit=11646` → `UserEditAtom` -(`docPersistIdRef=1`, `offsetPersistDirectory=11606`, `offsetLastEdit=0`) → 2 -slides in order with correct text. - -**No scan/heuristic fallback — spec-justified.** Both `/Current User` (§2.1.1) -and `/PowerPoint Document` (§2.1.2) are *required* streams, every conformant -file has at least one `UserEditAtom` + `PersistDirectoryAtom`, and the reading -algorithm has no alternative branch. An earlier draft kept a stream-scan -fallback (first `DocumentContainer`, every `SlideContainer` in stream order, -plus an outline-vs-container "more text wins" heuristic); it was **removed** — -unreachable for conformant files and able to silently serve *wrong* results (a -stale `DocumentContainer`, wrong slide order). `collect_slides` returns an empty -presentation only for the one *optional* structure: a document with no -presentation slide list (§2.4.1). Every mandatory structure that can't be -resolved — empty edit chain, unresolved `docPersistIdRef`, a slide -`persistIdRef` not in the directory — **throws**. +(stream order ≠ presentation order) and picks the **live** `DocumentContainer` +rather than the first in the stream. The read follows the `[MS-PPT]` algorithm: +`CurrentUserAtom` (`/Current User`) → walk the `UserEditAtom` chain newest→oldest +building the persist directory (newest offset per id wins) → resolve the live +`DocumentContainer` via `docPersistIdRef` → walk the slide list's +`SlidePersistAtom`s in presentation order. + +**No scan/heuristic fallback — spec-justified.** Both `/Current User` (§2.1.1) and +`/PowerPoint Document` (§2.1.2) are *required* streams, every conformant file has +a `UserEditAtom` + `PersistDirectoryAtom`, and the reading algorithm has no +alternative branch. An earlier stream-scan fallback (first `DocumentContainer`, +every `SlideContainer` in stream order, plus an outline-vs-container "more text +wins" heuristic) was **removed** — unreachable for conformant files and able to +silently serve *wrong* results (stale `DocumentContainer`, wrong order). +`collect_slides` returns empty only for the one *optional* structure: no +presentation slide list (§2.4.1). Every mandatory structure that can't be resolved +**throws**. **Two places hold slide text — and they are not equivalent.** - The **outline** (`SlideListWithTextContainer`, §2.4.14.3) is **optional** - (`DocumentContainer.slideList`, §2.4.1). When present it carries, per slide, - the title/body **placeholder** text only — free text boxes are *never* in it. -- The **`SlideContainer`** (§2.5.1) is the authoritative source: on-slide text - lives in the drawing's `ClientTextbox` records. - -In LibreOffice-exported `.ppt` the outline is **empty** (verified on -`slides.ppt`: the `0x0FF0` lists hold zero text atoms), so there we read each -slide's text from its `SlideContainer`. But PowerPoint-authored placeholders -commonly carry **no inline text** in the `SlideContainer` and instead an -`OutlineTextRefAtom` (§2.9.78) pointing, by index, at the *i*-th `TextHeaderAtom` -block of that slide in the `SlideListWithTextContainer`. So we read the outline -too: `read_slide_list_text` collects, per slide (keyed by `persistIdRef`), the -ordered list of its `TextHeaderAtom` texts, and `gather_text` resolves an -`OutlineTextRefAtom` box against it. On-slide `ClientTextbox` text still wins -when present. - -**`RT_SlideListWithText` recInstance disambiguates three lists.** -`MasterListWithTextContainer` (§2.4.14.1), `SlideListWithTextContainer` -(§2.4.14.3) and `NotesListWithTextContainer` (§2.4.14.6) share `recType = -RT_SlideListWithText` (0x0FF0); only `recInstance` tells them apart: - -| recInstance | container | meaning | -|-------------|-------------------------------|---------------------| -| `0x000` | `SlideListWithTextContainer` | presentation slides | -| `0x001` | `MasterListWithTextContainer` | masters | -| `0x002` | `NotesListWithTextContainer` | notes | - -An early draft had Slides/Master swapped, making the lookup read the *master* -list; fixed in `ppt_structs.hpp` (`SlideListInstance`). + (§2.4.1). When present it carries, per slide, the **placeholder** text only — + free text boxes are *never* in it. +- The **`SlideContainer`** (§2.5.1) is authoritative: on-slide text lives in the + drawing's `ClientTextbox` records. + +LibreOffice-exported `.ppt` leaves the outline **empty**, so there we read from the +`SlideContainer`. But PowerPoint-authored placeholders commonly carry *no* inline +text and instead an `OutlineTextRefAtom` (§2.9.78) pointing by index at the *i*-th +`TextHeaderAtom` block in the `SlideListWithTextContainer`. So we read the outline +too and resolve `OutlineTextRefAtom` boxes against it; on-slide `ClientTextbox` +text still wins when present. + +**`RT_SlideListWithText` recInstance disambiguates three lists** that share +`recType` 0x0FF0: `0x000` = slides (`SlideListWithTextContainer`), `0x001` = +masters, `0x002` = notes. An early draft had slides/master swapped (read the +*master* list); fixed in `ppt_structs.hpp` (`SlideListInstance`). **Sequential reading, no `tellg`.** The CFB-backed stream's `tellg()` returns -bogus values (it broke an early offset-tracking `read_children`). The parser -never depends on `tellg`: the caller `seekg`s to known offsets (from the persist -directory or a parent record), and child records are walked **forward** with a -`ChildCursor` — `read` header → `read`/recurse/`ignore` body — tracking the -bytes left in the container. A record that overruns its container throws, -keeping nested containers in sync or failing loudly. - -**Fail early on malformed input.** Where the spec dictates what to expect, -unexpected input **throws** (matches the sibling `.doc` parser). We **throw** on: -a missing required stream; a wrong record type (`read_header` — so a truncated -read, whose garbage type won't match, also throws); a record that overruns its -container (`ChildCursor`); a missing **mandatory** child record — the -`DrawingContainer` / `OfficeArtDgContainer` / `OfficeArtSpgrContainer` of a -slide (`require_child`); an `OfficeArtClientAnchor` whose `recLen` is neither 8 -nor 16; a non-decreasing (looping) `UserEditAtom` chain, an empty chain, an -unresolved `docPersistIdRef`, or a slide `persistIdRef` not in the persist -directory. We **pass through** (no throw) for values we don't model or that are -optional: an absent presentation slide list (0 slides), a shape with no -`OfficeArtClientAnchor` (unpositioned frame), nested groups and non-`Sp` records -in a group, and any non-text / unrecognised child record. - -**Endianness.** Host byte order / LSB-first bit-fields assumed; shared `oldms/` -assumption, see [`../AGENTS.md`](../AGENTS.md). For `.ppt`: every record field is -read in host byte order (see the note in `ppt_io.hpp`), and the `RecordHeader` -recVer/recInstance bit-fields assume LSB-first allocation. - -## Tests - -- `ppt_empty` — `odr-public/ppt/empty.ppt`: 1 slide. -- `ppt_slides` — `odr-public/ppt/slides.ppt`: 2 slides, 2 positioned frames each - (all `at_page` with `x/y/width/height`), distinct vertical positions, exact - per-box text. +bogus values. The parser never depends on it: the caller `seekg`s to known offsets +(persist directory / parent record), and child records are walked **forward** with +a `ChildCursor` (`read` header → body → recurse/ignore) tracking the bytes left in +the container. A record that overruns its container throws, keeping nested +containers in sync or failing loudly. + +**Fail early on malformed input** (matches the sibling `.doc` parser). **Throw** +on: a missing required stream; a wrong/truncated record type (`read_header`); a +record overrunning its container (`ChildCursor`); a missing **mandatory** child +(the `DrawingContainer`/`OfficeArtDgContainer`/`OfficeArtSpgrContainer`, +`require_child`); an `OfficeArtClientAnchor` whose `recLen` is neither 8 nor 16; +a looping/empty `UserEditAtom` chain, an unresolved `docPersistIdRef`, or a slide +`persistIdRef` not in the directory. **Pass through** (no throw) for the +unmodelled/optional: an absent slide list (0 slides), a shape with no anchor +(unpositioned frame), nested groups and non-`Sp` records in a group, any +unrecognised child. + +**First cut: top-level shapes only.** Only direct children of the root +`OfficeArtSpgrContainer`, whose anchors are already in the slide's master-unit +system (master units = 1/576 inch → inches via `/576`). Nested-group transforms, +non-grouped shapes, and master-placeholder geometry inheritance are deferred +(open work §1). Shapes with no text are dropped, so the group shape and pictures +disappear. -The non-empty fixture `slides.ppt` and reference-output HTML wiring are open -items (see below). +**Slide size hardcoded 10"×7.5"** (`slide_page_layout`) and the `font_size = 11pt` +in the adapters is a placeholder hack (same as `.doc`/`.xls`) — both open work. +`Document::is_editable()` → `false`; `save` throws. -## Out of scope +**Endianness.** Host byte order / LSB-first bit-fields assumed; shared `oldms/` +assumption, see [`../AGENTS.md`](../AGENTS.md). -Character/paragraph styles, fonts and colours; master and notes slides; -images/charts/tables and non-text shapes; animations/transitions; and -encrypted/obfuscated presentations. +### Text decoding ---- +`TextCharsAtom` = UTF-16 (`recLen/2` code units); `TextBytesAtom` = 1 byte/char +(0x00–0xFF). In-text controls: `0x0D` = paragraph break, `0x0B` = manual line +break (split like `doc_parser`); `0x09` tab kept; other controls dropped +(`clean_text`). -# Open work +## Tests -## 1. Frame refinements +- `ppt_empty` (`empty.ppt`): 1 slide. +- `ppt_slides` (`slides.ppt`): 2 slides, 2 positioned frames each (all `at_page` + with `x/y/width/height`), distinct positions, exact per-box text. -The first cut reads only **top-level** shapes — direct children of the root -`OfficeArtSpgrContainer` — whose anchors are already in the slide's master-unit -coordinate system. The refinements below raise fidelity; each is optional and -independent. - -- **1.1 Nested groups.** A shape nested inside a sub-group has its anchor - expressed in **that group's** coordinate system, defined by the group's - `OfficeArtFSPGR` (0xF009, `recVer 0x1`, `recLen 16`: `xLeft, yTop, xRight, - yBottom`), not in slide units. To support it: recurse into nested - `OfficeArtSpgrContainer` (0xF003), and for each descendant map its anchor from - the group's `[xLeft..xRight] × [yTop..yBottom]` onto the group shape's own - anchor rect in the parent, composing transforms down the nesting, before the - `/576` conversion. -- **1.2 Non-grouped shapes.** `OfficeArtDgContainer` (0xF002) also has an - optional direct `shape` (`OfficeArtSpContainer`, §2.2.13) for a shape not in a - group — the current walk only iterates the `OfficeArtSpgrContainer`. Rare in - real files, but read that child too for completeness. -- **1.3 Optional / inherited anchor.** A shape without an - `OfficeArtClientAnchor` (0xF010) currently yields a frame with no position. - PowerPoint placeholders often omit the anchor and inherit geometry from the - matching placeholder shape on the **master slide** (resolve via - `OfficeArtClientData.placeholderAtom` → the master's placeholder). -- **1.4 Origin / sign sanity check.** Field order and units are spec-confirmed - (top/left/right/bottom; master units = 1/576 inch) and verified on - `slides.ppt`. Still worth confirming the origin (top-left of the slide) and - non-negative values on a second, independently produced real file. +Fixture-commit / reference-HTML wiring / `OutlineTextRefAtom` fixture are open +(§2 below). -## 2. Smaller shortcomings +## Drawing-tree reference -- **2.1 Slide size is hardcoded.** `slide_page_layout` returns a fixed 10"×7.5" - (`ppt_document.cpp`). The real size is `DocumentAtom.slideSize` - (`RT_DocumentAtom` 0x03E9, the first child of the `DocumentContainer`) — a - `PointStruct` in master units (`/576` → inches). Read it and feed the page - layout; fall back to 10"×7.5" only if absent. -- **2.2 Reference-output HTML not wired.** `html_output_test` has no `ppt` case. - Add reference HTML under - `test/data/reference-output/odr-public/output/ppt/...` and wire it in (needs - the `OpenDocument.test.output` submodule). -- **2.3 Fixture not committed.** `test/data/input/odr-public/ppt/slides.ppt` - exists only in the local `odr-public` submodule working tree. It must be - committed/pushed to the `OpenDocument.test` repo and the submodule pointer - bumped, or CI can't see it (so `ppt_slides` would fail there). -- **2.4 No `OutlineTextRefAtom` fixture.** `OutlineTextRefAtom` resolution is - implemented but **unexercised by any committed fixture** — all three current - `.ppt` files are LibreOffice-authored with an empty outline (`grep` for the - `00 00 9E 0F 04 00 00 00` header finds none). A PowerPoint-authored `.ppt` - whose placeholders use the outline indirection is needed to regression-test - the path. Pairs with §2.3. -- **2.5 Auto-field metacharacters dropped.** Slide-number / date / header / - footer placeholders are separate records (`RT_*MetaCharAtom`) interleaved with - the text; we ignore them, so e.g. a slide-number placeholder yields nothing. - Low priority for "visible text only". -- **2.6 `slide_name` is empty.** Could return `"Slide N"` (index-based) so the - HTML page/tab has a label, matching how other formats name pages. -- **2.7 Endianness** — shared `oldms/` shortcoming; see [`../AGENTS.md`](../AGENTS.md). +Every record starts with an 8-byte `RecordHeader` (`recVer:4`, `recInstance:12`, +`recType:u16`, `recLen:u32`). `recVer == 0xF` marks a **container** (body is a +sequence of records); otherwise an **atom** with `recLen` payload bytes. -## Reference: the drawing tree +Key records: `CurrentUserAtom` 0x0FF6, `UserEditAtom` 0x0FF5, +`PersistDirectoryAtom` 0x1772, `DocumentContainer` 0x03E8, `SlideListWithText` +0x0FF0, `SlidePersistAtom` 0x03F3, `SlideContainer` 0x03EE, `MainMaster` 0x03F8 +(skipped), `Notes` 0x03F0 (skipped), `TextHeaderAtom` 0x0F9F, `TextCharsAtom` +0x0FA0, `TextBytesAtom` 0x0FA8. -Inside each `SlideContainer` (0x03EE) is the Office Art (Escher) drawing that -holds the slide's text boxes: +Inside each `SlideContainer` is the Office Art (Escher) drawing that holds the +text boxes: ``` SlideContainer (0x03EE) [MS-PPT] 2.5.1 @@ -343,21 +146,61 @@ SlideContainer (0x03EE) [MS-PPT] 2.5.1 └─ OfficeArtSpContainer (0xF004) shape #2 … ``` -- The `OfficeArt*` container/shape records are `[MS-ODRAW]`; the - `DrawingContainer` and the *client* records (`0xF00D` textbox, `0xF010` - anchor, `0xF011` data) are `[MS-PPT]`. `[MS-ODRAW]` §2.2.14 defers - `clientAnchor`/`clientData`/`clientTextbox` to the host app. -- **`OfficeArtSpContainer` (0xF004) child order** per `[MS-ODRAW]` §2.2.14: - `shapeGroup?` (`OfficeArtFSPGR`, group shapes only), `shapeProp` - (`OfficeArtFSP`, 16 B), `shapePrimaryOptions?` (`OfficeArtFOPT`), …, - **`clientAnchor?`**, `clientData?`, `clientTextbox?`. The parser matches by - recType, so order only documents what to expect. -- **Anchor body** (`OfficeArtClientAnchor`, atom, `recLen == 8` or `16`), field - order **top, left, right, bottom** (y, x, x, y): - - `recLen == 8` → `SmallRectStruct` (`[MS-PPT]` 2.12.8): four **signed 2-byte**. - - `recLen == 16` → `RectStruct` (`[MS-PPT]` 2.12.7): four **signed 4-byte**. - - `width = right - left`, `height = bottom - top`; master units → inches = `/576`. +- `OfficeArt*` records are `[MS-ODRAW]`; the *client* records (`0xF00D` textbox, + `0xF010` anchor, `0xF011` data) and `DrawingContainer` are `[MS-PPT]` + (`[MS-ODRAW]` §2.2.14 defers `clientAnchor`/`clientData`/`clientTextbox` to the + host app). The parser matches by recType, so child order only documents what to + expect. +- **Anchor body** (`OfficeArtClientAnchor`, atom), field order **top, left, right, + bottom**: `recLen == 8` → `SmallRectStruct` (§2.12.8, four signed 2-byte); + `recLen == 16` → `RectStruct` (§2.12.7, four signed 4-byte). `width = right − + left`, `height = bottom − top`; master units → inches = `/576`. - The first child `OfficeArtSpContainer` of the root spgr is the **group shape** - itself (holds the `OfficeArtFSPGR`, has no `clientTextbox`); the parser drops - it implicitly because it has no text. + itself (holds `OfficeArtFSPGR`, no `clientTextbox`) — dropped implicitly because + it has no text. + +--- + +# Open work + +## 1. Frame refinements + +The first cut reads only top-level shapes. Each refinement below raises fidelity +and is independent: + +- **1.1 Nested groups.** A shape in a sub-group has its anchor in *that group's* + coordinate system (the group's `OfficeArtFSPGR` 0xF009: `xLeft, yTop, xRight, + yBottom`), not slide units. Recurse into nested `OfficeArtSpgrContainer` (0xF003) + and map each descendant's anchor from `[xLeft..xRight]×[yTop..yBottom]` onto the + group shape's own anchor rect in the parent, composing transforms down the + nesting, before the `/576` conversion. +- **1.2 Non-grouped shapes.** `OfficeArtDgContainer` (0xF002) also has an optional + direct `shape` (§2.2.13) for a shape not in a group — the current walk only + iterates the `OfficeArtSpgrContainer`. Rare; read that child too. +- **1.3 Optional / inherited anchor.** A shape without an `OfficeArtClientAnchor` + currently yields a frame with no position. PowerPoint placeholders often omit + it and inherit geometry from the matching placeholder on the **master slide** + (resolve via `OfficeArtClientData.placeholderAtom`). +- **1.4 Origin / sign sanity check.** Field order and units are spec-confirmed and + verified on `slides.ppt`; still worth confirming origin (top-left) and + non-negative values on a second, independently produced real file. + +## 2. Smaller shortcomings + +- **2.1 Slide size hardcoded.** Real size is `DocumentAtom.slideSize` + (`RT_DocumentAtom` 0x03E9, first child of `DocumentContainer`) — a `PointStruct` + in master units. Read it; fall back to 10"×7.5" only if absent. +- **2.2 Reference-output HTML not wired.** `html_output_test` has no `ppt` case; + add reference HTML under `test/data/reference-output/…/output/ppt/` (needs the + `OpenDocument.test.output` submodule). +- **2.3 Fixture not committed.** `slides.ppt` exists only in the local + `odr-public` working tree; must be committed/pushed and the submodule pointer + bumped, or CI can't see it. +- **2.4 No `OutlineTextRefAtom` fixture.** The resolution path is implemented but + unexercised — all current `.ppt` files are LibreOffice-authored (empty outline). + A PowerPoint-authored file using the outline indirection is needed. Pairs with + §2.3. +- **2.5 Auto-field metacharacters dropped.** Slide-number/date/header/footer + placeholders (`RT_*MetaCharAtom`) are ignored. Low priority. +- **2.6 `slide_name` is empty.** Could return `"Slide N"` for the HTML page label. +- **2.7 Endianness** — shared `oldms/` shortcoming; see [`../AGENTS.md`](../AGENTS.md). diff --git a/src/odr/internal/oldms/spreadsheet/AGENTS.md b/src/odr/internal/oldms/spreadsheet/AGENTS.md index 25b1349b..8f1a3e9e 100644 --- a/src/odr/internal/oldms/spreadsheet/AGENTS.md +++ b/src/odr/internal/oldms/spreadsheet/AGENTS.md @@ -1,138 +1,89 @@ -# `.xls` (Excel / BIFF8) support — status, design & open work +# `.xls` (Excel / BIFF8) support — design & open work -What the `oldms/spreadsheet/` module does **today**, the **design decisions** -behind it, and the **open work**. Shared `oldms/` conventions are in -[`../AGENTS.md`](../AGENTS.md). +The **why** and the roadmap; what is concretely done lives in the code. Shared +`oldms/` conventions are in [`../AGENTS.md`](../AGENTS.md). -**Scope.** Extract the **visible cell text** of every worksheet and expose it -through the abstract document model so the generic HTML renderer produces a plain -table per sheet. Every cell value is rendered as a *string* — no styles, -number/date formats, merged cells, drawings, or charts. +**Scope.** Extract the **visible cell text** of every worksheet through the +abstract model so the generic HTML renderer produces a plain table per sheet. +Every cell value is rendered as a *string* — no styles, number/date formats, +merged cells, drawings, or charts. -**Specs.** `[MS-XLS]` (the record stream, the SST, the cell records) and -`[MS-CFB]` for the container. Section numbers are cited inline below and in code. - ---- - -## What works - -- `.xls` is detected (`/Workbook` stream) and decoded to a `Document` - (spreadsheet): one `sheet` element per worksheet, with `sheet_cell` → - `paragraph` → `text` elements for every non-empty cell. -- **All BIFF8 cell value kinds** become display text: SST strings (`LabelSst`), - inline strings (`Label`), numbers (`RK`, `MulRk`, `Number`), booleans/errors - (`BoolErr`), and **cached formula results** (`Formula` + `String` for string - results; numeric/boolean/error results from the `FormulaValue`). -- **SST `CONTINUE` splitting** is handled, including a split *mid-string* where - the continuation re-declares the character encoding (§2.5.293). -- Sheet `dimensions` come from the `Dimensions` record; `content` is the tight - extent of the non-empty cells (what the HTML renderer uses by default). -- The generic HTML renderer produces one table per sheet - (`html::translate_sheet`), with column letters and row numbers. - -Verified against `[MS-XLS]`: the record stream (§2.1.4), BOF/substream layout -(§2.4.21), `BoundSheet8` (§2.4.28), `SST`/`Continue` (§2.4.265/.58), -`XLUnicodeRichExtendedString` (§2.5.293), `RkNumber` (§2.5.217: bit 0 = `fX100`, -bit 1 = `fInt`), `FormulaValue` (§2.5.133), `Dimensions` (§2.4.90). +**Specs.** `[MS-XLS]` (record stream, SST, cell records) + `[MS-CFB]` container. +Section numbers cited inline in code. ## Module layout (sibling of `../text`, `../presentation`) -| File (`oldms/spreadsheet/`) | Role | -|------------------------------------|---------------------------------------------------| -| `xls_structs.hpp` | `#pragma pack(1)` PODs for the record bodies + `static_assert` sizes + record type enum | -| `xls_io.{hpp,cpp}` | `BiffReader` (record walker with transparent `CONTINUE` hopping; the `[MS-XLS]` string readers and `expect_bof` are methods), RK decoding, number formatting | -| `xls_parser.{hpp,cpp}` | `parse_tree(registry, files)` → globals (BoundSheet8 + SST) then one pass per sheet substream | -| `xls_element_registry.{hpp,cpp}` | Flat element store + `Sheet` (name, dimensions, cell position map) and `SheetCell` (position) payloads | -| `xls_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | - -## Pipeline: how a `.xls` becomes the element tree - -1. **Wiring.** `LegacyMicrosoftFile::parse_meta` detects the `/Workbook` stream - → `FileType::legacy_excel_worksheets`, `DocumentType::spreadsheet`, and - `document()` returns `spreadsheet::Document`. -2. **Globals substream.** `/Workbook` is a flat sequence of `(u16 type, u16 - size, body)` records. The first substream (after its `BOF`, which must - declare BIFF8 = `vers 0x0600`) holds, per sheet, a `BoundSheet8` (name + - absolute offset of the sheet's `BOF`; only `dt == worksheet` is kept) and the - `SST` — all shared string constants, deduplicated. -3. **SST / CONTINUE.** A record body is capped at 8224 bytes; the SST payload - spills into `Continue` records, and the split can fall *inside* a string. - `BiffReader`'s body accessors hop into a following `CONTINUE` transparently - (throwing if the next record is anything else); character data additionally - re-reads a fresh flags byte at each hop, since the continuation re-declares - compressed (1 byte/char) vs UTF-16 for the remainder. Formatting runs - (`cRun`·4 bytes) and phonetic data (`cbExtRst` bytes) are read and skipped. -4. **Sheet substreams.** For each kept `BoundSheet8`, seek to its `BOF` and scan - records until `EOF`: `Dimensions` → sheet extents; `LabelSst` / `Label` / - `RK` / `MulRk` / `Number` / `BoolErr` → one cell each; `Formula` → the cached - result in its `FormulaValue` (an Xnum double unless `fExprO == 0xFFFF`, then - string/bool/error/blank — a string result follows in a `String` record, - matched via a pending-cell marker). `Blank` / `MulBlank` carry no text and - are ignored. -5. **Tree.** Each non-empty cell becomes `sheet_cell → paragraph → text` (the - cell's rendered string). Cells hang off their sheet by `parent_id` only — - they are *not* in the sibling chain (mirrors `ooxml/spreadsheet`); lookup goes - through the sheet's `(column,row) → id` map, which also tracks the tight - `content` extent. -6. **Render.** `html::translate_sheet` walks the sheet purely through the public - `Sheet` / `SheetCell` API, which delegates to our adapter. - -### Value formatting - -- **RK numbers** (§2.5.217): low 2 bits are flags — bit 0 `fX100` (divide by - 100), bit 1 `fInt` (30-bit signed integer vs the *high 30 bits* of an IEEE - double, rest zero). -- Numbers are formatted with `%.15g` (≈ Excel's "General": up to 15 significant - digits, no trailing zeros, integers without a decimal point). -- Booleans → `TRUE`/`FALSE`; error codes (BErr, §2.5.10) → `#DIV/0!`, `#VALUE!`, - `#REF!`, `#NAME?`, `#NUM!`, `#N/A`, `#NULL!`. -- Dates are **not** decoded: a date cell shows its raw serial number unless the - file stored it as a string (number-format handling is open work). - -## Adapters - -`xls_document.cpp` implements the generic `ElementAdapter` plus `SheetAdapter` / -`SheetCellAdapter` / `ParagraphAdapter` / `TextAdapter`: -- `sheet_name` / `sheet_dimensions` → from the registry payload; - `sheet_content(range)` → the tight content extent, clamped to `range`. -- `sheet_cell(col,row)` → map lookup, `null_element_id` for empties; - `sheet_first_shape` → none. -- All `*_style(...)` → `{}`; `sheet_cell_value_type` → `ValueType::string` - (every value is pre-rendered text); `sheet_cell_span` → `{1,1}`. -- `paragraph_text_style` / `text_style` set `font_size = 11pt` so empty - paragraphs have height (same hack as the `.doc`/`.ppt` modules). -- `Document::is_editable()` → `false`; `save(...)` → `UnsupportedOperation`. +| File (`oldms/spreadsheet/`) | Role | +|---|---| +| `xls_structs.hpp` | `#pragma pack(1)` PODs for record bodies + `static_assert` sizes + record-type enum | +| `xls_io.{hpp,cpp}` | `BiffReader` (record walker with transparent `CONTINUE` hopping; the `[MS-XLS]` string readers + `expect_bof`), RK decoding, number formatting | +| `xls_parser.{hpp,cpp}` | `parse_tree` → globals (BoundSheet8 + SST) then one pass per sheet substream | +| `xls_element_registry.{hpp,cpp}` | Flat element store + `Sheet` (name, dimensions, cell-position map) and `SheetCell` payloads | +| `xls_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | + +Tree shape: `sheet → sheet_cell → paragraph → text`, one `sheet_cell` per +non-empty cell. ## Design decisions -- **Fail early on malformed input** (matches the sibling modules): missing or - non-BIFF8 `BOF`, a non-`CONTINUE` record where a body continuation is - required, an out-of-range SST index, a malformed `MulRk` body, an unknown - `FormulaValue` type, and truncated streams all **throw**. Records that are - merely *not modelled* are skipped. -- **Pre-rendered text instead of typed values.** Cell values are converted to - display strings at parse time; the model exposes `ValueType::string` only. - Typed values would require XF/number-format plumbing — deliberately deferred. -- **Endianness/bit order**: bytes are copied straight into native - integers/doubles and bit-field structs (`RkNumber`, `UnicodeStringFlags`, flag - fields of `BoundSheet8Fixed`/`FormulaFixed`) — little-endian, LSB-first hosts - only; shared `oldms/` assumption, see [`../AGENTS.md`](../AGENTS.md). +**Pre-rendered text instead of typed values.** Cell values are converted to +display strings at parse time; the model exposes `ValueType::string` only. All +BIFF8 value kinds become text: SST strings (`LabelSst`), inline (`Label`), numbers +(`RK`/`MulRk`/`Number`), booleans/errors (`BoolErr`), and cached formula results +(`Formula` + a following `String` record for string results, matched via a +pending-cell marker). Typed values would require XF/number-format plumbing — +deliberately deferred (open work §1). + +**Cells hang off their sheet by `parent_id` only** — they are *not* in the sibling +chain (mirrors `ooxml/spreadsheet`); lookup goes through the sheet's `(column,row) +→ id` map, which also tracks the tight `content` extent (what the HTML renderer +uses by default). `sheet_dimensions` comes from the `Dimensions` record. + +**SST `CONTINUE` splitting is the subtle part.** A record body is capped at 8224 +bytes, so the SST spills into `Continue` records and the split can fall *inside* a +string. `BiffReader`'s body accessors hop into a following `CONTINUE` transparently +(throwing if the next record is anything else); character data additionally +re-reads a fresh flags byte at each hop, since the continuation re-declares +compressed (1 B/char) vs UTF-16 for the remainder (§2.5.293). Formatting runs +(`cRun`·4 B) and phonetic data (`cbExtRst` B) are read and skipped. + +**Fail early on malformed input** (matches the siblings): missing/non-BIFF8 `BOF` +(`vers != 0x0600`), a non-`CONTINUE` record where a body continuation is required, +an out-of-range SST index, a malformed `MulRk` body, an unknown `FormulaValue` +type, and truncated streams all **throw**. Records merely *not modelled* are +skipped. + +**Endianness/bit order.** Bytes copied straight into native integers/doubles and +bit-field structs (`RkNumber`, `UnicodeStringFlags`, flags of +`BoundSheet8Fixed`/`FormulaFixed`) — little-endian, LSB-first hosts only; shared +`oldms/` assumption, see [`../AGENTS.md`](../AGENTS.md). + +**Adapters** expose `ValueType::string` for every cell, `sheet_cell_span` → +`{1,1}`, all `*_style` → `{}`, and the `font_size = 11pt` placeholder hack (same as +`.doc`/`.ppt`). `Document::is_editable()` → `false`; `save` throws. + +### Value formatting (the non-obvious bits) + +- **RK numbers** (§2.5.217): low 2 bits are flags — bit 0 `fX100` (÷100), bit 1 + `fInt` (30-bit signed int vs the *high 30 bits* of an IEEE double, rest zero). +- Numbers use `%.15g` (≈ Excel "General"). Booleans → `TRUE`/`FALSE`; errors + (§2.5.10) → `#DIV/0!`, `#VALUE!`, `#REF!`, `#NAME?`, `#NUM!`, `#N/A`, `#NULL!`. +- **Dates are not decoded**: a date cell shows its raw serial number unless the + file stored it as a string (open work §1). ## Tests -- `xls_string_split_across_continue` — a string split mid-character-data with an - encoding switch at the boundary. -- `xls_rich_string_runs_across_continue` — formatting-run skip across a - `CONTINUE` (no flags byte there) + correct position for the next string. -- `xls_decode_rk` — all four RK flag combinations + number formatting; the - inputs are raw on-disk encodings, so it also pins the `RkNumber` bit-field - layout. -- `xls_empty` / `xls_file_example_10` / `xls_file_example_5000` — real fixtures: - sheet names, dimensions, content extents, string/number cells; the 5000-row - file exercises SST `CONTINUE` handling on real data. -- HTML output: `html_output_test` no longer skips `legacy_excel_worksheets`; - reference output lives under - `test/data/reference-output/{odr-public,odr-private}/output/xls/`. +- `xls_string_split_across_continue` — split mid-character-data with an encoding + switch at the boundary. +- `xls_rich_string_runs_across_continue` — formatting-run skip across `CONTINUE` + (no flags byte there) + correct next-string position. +- `xls_decode_rk` — all four RK flag combinations + number formatting (raw on-disk + encodings, so it also pins the `RkNumber` bit-field layout). +- `xls_empty` / `xls_file_example_10` / `xls_file_example_5000` — real fixtures + (names, dimensions, extents, string/number cells; the 5000-row file exercises + SST `CONTINUE` on real data). +- HTML output wired (`html_output_test` no longer skips `legacy_excel_worksheets`; + reference under `test/data/reference-output/…/output/xls/`). --- @@ -142,42 +93,38 @@ Roughly ordered by value. ## 1. Number & date formatting (the biggest visible gap) -Cells currently show raw values: a date cell renders as its serial number (e.g. -`43023` instead of `15/10/2017`) and numbers ignore their format codes. Fix by -following the format chain: -- Each cell record carries an `ixfe` (currently discarded — the parser already - reads it). It indexes the `XF` records (0x00E0) in the globals substream; - `XF.ifmt` picks a number format: a built-in id (0–163, the table is in - [MS-XLS] 2.4.126 `Format`) or a `Format` record (0x041E) with a format string. -- MVP: keep `ixfe` per cell, parse `XF`/`Format`, and special-case the date/time - formats (built-in ids 14–22, 45–47 + anything containing `y/m/d/h`) to convert - the serial date (days since 1899-12-31, fractional part = time; mind the - workbook's 1904 flag in `Date1904`, 0x0022) into a sensible string. Full - custom-format rendering is a rabbit hole; approximate first. +A date cell renders as its serial number (e.g. `43023` not `15/10/2017`); numbers +ignore their format codes. Fix by following the format chain: +- Each cell carries an `ixfe` (already read, currently discarded) indexing the + `XF` records (0x00E0) in globals; `XF.ifmt` picks a number format: a built-in id + (0–163, table in [MS-XLS] 2.4.126) or a `Format` record (0x041E) string. +- MVP: keep `ixfe` per cell, parse `XF`/`Format`, special-case the date/time + formats (built-in ids 14–22, 45–47 + anything with `y/m/d/h`) to convert the + serial date (days since 1899-12-31, fractional = time; mind the workbook 1904 + flag in `Date1904` 0x0022). Full custom-format rendering is a rabbit hole; + approximate first. ## 2. Coverage gaps -- **Merged cells**: `MergeCells` record (0x00E5) → `sheet_cell_span` / - `sheet_cell_is_covered` (the adapter stubs are in place). -- **Styles**: fonts (`Font`, 0x0031), fills/borders from `XF` → - `sheet_cell_style` / `text_style`; column widths (`ColInfo`, 0x007D) and row - heights (`Row`, 0x0208) → `sheet_column_style` / `sheet_row_style`. +- **Merged cells**: `MergeCells` (0x00E5) → `sheet_cell_span`/ + `sheet_cell_is_covered` (adapter stubs in place). +- **Styles**: fonts (`Font` 0x0031), fills/borders from `XF`; column widths + (`ColInfo` 0x007D) and row heights (`Row` 0x0208). - **Hidden rows/columns** (`Row.fDyZero`, `ColInfo.fHidden`). - **Typed cell values**: expose numeric/bool/date `ValueType`s instead of - pre-rendered strings (needed for anything smarter than HTML text). -- **Encrypted workbooks**: a `FilePass` record (0x002F) in the globals substream - means the rest of the stream is encrypted ([MS-OFFCRYPTO]) — currently it - parses as garbage or throws; should report password-protected. -- **BIFF5/BIFF7** (`BOF.vers != 0x0600`): currently throws; older `.xls` files - exist in the wild (no SST — `Label` records carry the strings inline). -- **Drawings/charts/images** (`MsoDrawing`/`Obj`/chart substreams) — likely - never worth it for text extraction. + pre-rendered strings. +- **Encrypted workbooks**: a `FilePass` (0x002F) in globals means the rest is + encrypted ([MS-OFFCRYPTO]); currently parses as garbage or throws — should report + password-protected. +- **BIFF5/BIFF7** (`vers != 0x0600`): currently throws; older files exist in the + wild (no SST — `Label` records carry strings inline). +- **Drawings/charts/images** — likely never worth it for text extraction. ## 3. Smaller shortcomings - **Endianness/bit order** — shared `oldms/` shortcoming, see [`../AGENTS.md`](../AGENTS.md). -- `RString` (0x00D6, rich inline string cell) is rare and currently skipped. +- `RString` (0x00D6, rich inline string cell) is rare and skipped. - A `Formula` string result is matched to the *immediately following* `String` - record via a pending-cell marker; an intervening `SharedFmla`/`Array`/`Table` - record is tolerated only because unknown records are skipped — not validated. + record; an intervening `SharedFmla`/`Array`/`Table` is tolerated only because + unknown records are skipped — not validated. diff --git a/src/odr/internal/oldms/text/AGENTS.md b/src/odr/internal/oldms/text/AGENTS.md index a934eab7..ab513d25 100644 --- a/src/odr/internal/oldms/text/AGENTS.md +++ b/src/odr/internal/oldms/text/AGENTS.md @@ -1,335 +1,110 @@ -# `.doc` (Word) support — status, design & open work +# `.doc` (Word) support — design & open work -What the `oldms/text/` module does **today**, the **design decisions** behind -it, and the **open work**. Shared `oldms/` conventions are in -[`../AGENTS.md`](../AGENTS.md). +The **why** and the roadmap; what is concretely done lives in the code. Shared +`oldms/` conventions are in [`../AGENTS.md`](../AGENTS.md). **Scope.** Extract the **visible text of the main document body**, split into -paragraphs and manual line breaks, and expose it through the abstract document -model so the generic HTML renderer lays it out as a flat run of paragraphs. No -character/paragraph styles, no headers/footers/footnotes/endnotes/annotations, -no tables, frames, images, or fields beyond showing their result text. +paragraphs and manual line breaks, through the abstract model so the generic HTML +renderer lays it out as a flat run of paragraphs. No character/paragraph styles, +headers/footers/notes/annotations, tables, frames, images, or fields beyond their +result text. -**Specs.** `[MS-DOC]` (the FIB, the Clx / piece table, text decoding) and -`[MS-CFB]` for the container. Section numbers are cited inline below. - ---- - -## What works - -- `.doc` is detected (`/WordDocument` stream) and decoded to a `Document` - (text), one flat element tree under the root. -- The **main document body** (the first `ccpText` characters) is read from the - piece table, decoded (compressed 8-bit *or* UTF-16), split into paragraphs / - manual line breaks, with a `page_break` element at each end-of-section / - manual page break (`0x0C`). -- Field codes are resolved to their **result** text (the instruction part is - hidden); anchor/control characters are stripped. -- The generic HTML renderer produces the body as a sequence of paragraphs. - -Verified against `[MS-DOC]`: the read path matches *Retrieving Text* (§2.4.1, -steps 1–6), the FIB version map (§2.5.1), the Clx / Pcdt / Prc lead bytes -(§2.9.38/.178/.209), `FcCompressed` incl. the `0x82–0x9F` byte map (§2.9.73), -and the field characters (§2.8.25). +**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding) + `[MS-CFB]` +container. The implemented read path matches *Retrieving Text* (§2.4.1 steps 1–6); +section numbers are cited inline in code and in the read-path map below. ## Module layout (sibling of `../presentation`) -| File (`oldms/text/`) | Role | -|-----------------------------------|-----------------------------------------------------| -| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/2002/2003/2007` chain, `Sprm`, `FcCompressed`, `Pcd`) + `static_assert` sizes + the `PlcPcdMap` piece-table view + `ParsedFib` | -| `doc_io.{hpp,cpp}` | `read(...)` helpers over `std::istream`: the variable-length FIB, the Clx walk, string decoding (compressed / UTF-16) | -| `doc_helper.{hpp,cpp}` | `CharacterIndex` (the decoded piece table) + `read_character_index` | -| `doc_parser.{hpp,cpp}` | `parse_tree(registry, files)` → reads the body text and builds the element tree, incl. `clean_text` (field & control-char handling) | -| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + a text side-payload | -| `doc_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | - -`ElementRegistry` is a `vector` (id = index) with parent/child/sibling -ids and a side map for the text payload; `create_element` / `create_text_element` -/ `append_child` are the only builders. - -## Pipeline: how a `.doc` becomes the element tree - -1. **Wiring.** `LegacyMicrosoftFile::parse_meta` detects the `/WordDocument` - stream → `FileType::legacy_word_document`, `DocumentType::text`, and - `document()` returns `text::Document`. -2. **Read the FIB.** `parse_tree` opens `/WordDocument` and reads the **File - Information Block** (§2.5.1). The FIB is variable-length and self-describing: - a fixed `FibBase` (32 B) followed by four counted arrays — `csw`·uint16 - (`fibRgW`), `cslw`·uint32 (`fibRgLw`), `cbRgFcLcb`·`FcLcb` (`fibRgFcLcb`), - `cswNew`·uint16 (`fibRgCswNew`). `read(ParsedFib&)` reads each count, - validates it covers the struct we model, then `ignore`s any surplus. -3. **Pick the FIB version.** The effective `nFib` is `fibRgCswNew.nFibNew` when - `cswNew > 0`, else `FibBase.nFib`. `type_dispatch_FibRgFcLcb` maps it - (`nFib97 … nFib2007`) to the right `FibRgFcLcb*` layout and `memcpy`s the raw - `fibRgFcLcb` bytes into it. We only read `clx` out of it, but the whole - versioned struct is modelled so the offset is correct. -4. **Locate & read the Clx (piece table).** The table stream is `/1Table` or - `/0Table` per `FibBase.fWhichTblStm`. The Clx (§2.9.38) lives at - `fibRgFcLcb->clx.fc`. `read_Clx` walks it: leading `Prc` entries (lead - `0x01`) are skipped, then the `Pcdt` (lead `0x02`) carries the `PlcPcd` — the - piece table mapping CP ranges to byte offsets in `/WordDocument`. - `read_character_index` turns it into a `CharacterIndex`. -5. **Concatenate the body text.** Pieces come in ascending CP order; - `parse_tree` clamps each to the remaining `ccpText` budget (so only the main - body is taken), seeks to each piece's `data_offset`, decodes it. -6. **Build the tree.** Split the body on `0x0D` (paragraph mark) — dropping the - trailing empty paragraph from the body's guard mark — then each paragraph on - `0x0C` (end-of-section / manual page break) and each segment on `0x0B` - (manual line break): - - ``` - root (ElementType::root) - ├── paragraph (ElementType::paragraph) split on 0x0D, then 0x0C - │ ├── text (ElementType::text) clean_text(...) of the run - │ └── line_break (ElementType::line_break) for 0x0B in a paragraph - └── page_break (ElementType::page_break) one per 0x0C boundary - ``` -7. **Render.** HTML works through the generic renderer via the public - `Paragraph` / `Text` / `LineBreak` API and our adapters. - -## The piece table (`CharacterIndex`) - -A `.doc` stores text in **pieces** rather than one contiguous run: the `PlcPcd` -is `n+1` ascending CP boundaries (`aCP`) followed by `n` `Pcd` structures -(`aData`). `PlcPcdMap` is a zero-copy view over the raw `plcPcd` bytes computing -`n = (cb - 4) / (4 + sizeof(Pcd))`, exposing `aCP(i)` / `aData(i)`. - -Each `Pcd` holds an `FcCompressed`: -- `fCompressed == 0` → **UTF-16**, `data_offset = fc`, 2 bytes per CP. -- `fCompressed == 1` → **compressed** (one byte per CP), `data_offset = fc / 2`. - -`read_character_index` records, per piece, `(start_cp, length_cp, data_offset, -is_compressed)`; `CharacterIndex::Iterator` derives `length_cp` from adjacent CP -boundaries and `data_length` from the compression flag. `append` enforces -ascending CP order (throws otherwise). - -### Text decoding - -- **Uncompressed**: `length_cp` UTF-16 code units → `u16string_to_string`. -- **Compressed**: each byte is one code point (§2.9.73 / §2.4.1 step 6). Bytes - `0x82–0x9F` are remapped via `uncompress_char` (the Windows-1252 "smart - quotes" block — e.g. `0x92 → U+2019`, `0x96 → U+2013`); every other byte `b` - is code point `U+00b` and UTF-8-encoded, so `0xA0–0xFF` round-trip (e.g. - `0xE9 → "é"`). -- **In-text control characters** (`clean_text`): - - `0x0D` paragraph mark, `0x0C` end-of-section / manual page break, `0x0B` - manual line break are consumed by the caller's splits and never reach - `clean_text`. A `0x0C` boundary emits a `page_break` (§2.8.26). - - `0x13`/`0x14`/`0x15` delimit a **field**: instruction (begin→separator) - hidden, result (separator→end) shown. The separator `0x14` is optional - (§2.8.25); a separator-less field is hidden up to its `0x15` end. Nesting is - tracked with a per-field stack. - - `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; `0x1F` optional hyphen - dropped; all other control characters `< 0x20` (picture/OLE `0x01`, footnote - ref `0x02`, cell mark `0x07`, …) dropped. - -## Adapters - -`doc_document.cpp` implements the generic `ElementAdapter` plus -`TextRootAdapter` / `ParagraphAdapter` / `SpanAdapter` / `TextAdapter` / -`LineBreakAdapter`: -- `text_root_page_layout` / `text_root_first_master_page` → empty. -- `paragraph_style` / `span_style` / `line_break_style` → empty (`TODO`). -- `paragraph_text_style` / `text_style` set `font_size = 11pt` so empty - paragraphs still have height (same hack as the PPT module; removed when - character formatting lands — see open work). -- `Document::is_editable()` → `true` and `is_savable(encrypted)` → - `!encrypted`, but `save(...)` and `text_set_content(...)` throw - `UnsupportedOperation` — read-only in practice. - -## Binary format reference (FIB) - -The FIB is the root of every `.doc`, at offset 0 of `/WordDocument`: - -``` -FibBase 32 B fixed (wIdent, nFib, flags incl. fWhichTblStm/fEncrypted, …) -csw uint16 count of the following uint16 array -fibRgW csw·uint16 -cslw uint16 count of the following uint32 array -fibRgLw cslw·uint32 (holds ccpText at uint16 indices 6–7) -cbRgFcLcb uint16 count of the following FcLcb (8-byte) array -fibRgFcLcb cbRgFcLcb·FcLcb (holds clx → the piece table) -cswNew uint16 count of the following uint16 array -fibRgCswNew cswNew·uint16 (nFibNew overrides FibBase.nFib when present) -``` - -`ccpText` (count of CPs in the main body) is read out of `fibRgLw` as a -little-endian uint32 spanning indices 6–7; it is signed and MUST be ≥ 0, so a -value with the sign bit set **throws** (§2.5.5). `nFib` values handled: `nFib97` -(0x00C1), `nFib2000` (0x00D9), `nFib2002` (0x0101), `nFib2003` (0x010C), -`nFib2007` (0x0112). A value **above** `nFib2007` falls back to the -`FibRgFcLcb2007` layout; a value below `nFib97` **throws**. - ---- +| File (`oldms/text/`) | Role | +|---|---| +| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`) + `static_assert` sizes + `PlcPcdMap` + `ParsedFib` | +| `doc_io.{hpp,cpp}` | `read(...)` over `std::istream`: variable-length FIB, Clx walk, string decoding | +| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index` | +| `doc_parser.{hpp,cpp}` | `parse_tree` → body text + tree; `clean_text` (field & control-char handling) | +| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload | +| `doc_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | ## Design decisions -**Main body only, via the `ccpText` budget.** `/WordDocument` interleaves the -body with headers, footnotes, annotations, etc.; the FIB's `ccp*` counts -partition the CP space. We take only the first `ccpText` CPs by clamping each -piece to the remaining budget and stopping when exhausted. +**Main body only, via the `ccpText` budget.** `/WordDocument` interleaves the body +with headers, footnotes, annotations, etc.; the FIB's `ccp*` counts partition the +CP space. We take only the first `ccpText` CPs by clamping each piece to the +remaining budget and stopping when exhausted. **Self-describing FIB read — forward-compatible.** `read(ParsedFib&)` trusts the -on-disk counts rather than a fixed layout: it reads what we model and `ignore`s -the surplus. A FIB from a newer Word that appends fields still parses — the -version dispatch picks the matching `FibRgFcLcb*` (or `FibRgFcLcb2007` for a -newer-than-2007 `nFib`), and the `FcLcb` block is copied **clamped** to -`min(sizeof(layout), cbRgFcLcb·8)`, so extra trailing entries are ignored and a -shorter block leaves the remainder zero (the `clx`/`fcClx` we need lives in the -`FibRgFcLcb97` base, always covered). The `csw`/`cslw` counts must still cover -the arrays we read, else they throw. - -**Fail early on malformed input** (matches the sibling `.ppt` parser). We -**throw** on: an `nFib` below `nFib97` or an unknown `nFibNew` (newer-than-2007 -`nFib` does **not** throw — it uses the 2007 layout); a `ccpText` with the sign -bit set (§2.5.5); a `csw`/`cslw` count too small to cover the array we read; an -unexpected lead byte while walking the Clx (anything other than `0x01`/`0x02`); -a piece table whose CP boundaries are not ascending; a compressed byte outside -`0x00–0xFF` or an early EOF while decoding. We **pass through** for things we -don't model: text after the main body, the `Prc` formatting runs, and every -control/field character `clean_text` drops. +on-disk counts, not a fixed layout: it reads what we model and `ignore`s the +surplus. A FIB from a newer Word still parses — version dispatch picks the matching +`FibRgFcLcb*` (or `FibRgFcLcb2007` for a newer-than-2007 `nFib`), and the `FcLcb` +block is copied **clamped** to `min(sizeof(layout), cbRgFcLcb·8)`, so extra +trailing entries are ignored and a shorter block leaves the remainder zero (the +`clx`/`fcClx` we need is in the `FibRgFcLcb97` base, always covered). The +`csw`/`cslw` counts must still cover the arrays we read, else they throw. + +**Piece table, not one contiguous run.** A `.doc` stores text in **pieces**; the +`PlcPcd` is `n+1` ascending CP boundaries followed by `n` `Pcd`s. `PlcPcdMap` is a +zero-copy view over the raw bytes (`n = (cb − 4)/(4 + sizeof(Pcd))`). Each `Pcd`'s +`FcCompressed`: `fCompressed == 0` → UTF-16 at `fc` (2 B/CP); `== 1` → compressed +at `fc/2` (1 B/CP, `0x82–0x9F` remapped via `uncompress_char` — the Windows-1252 +smart-quotes block; every other byte `b` → `U+00b`, so `0xA0–0xFF` round-trip). +`append` enforces ascending CP order (throws otherwise). + +**Fail early on malformed input** (matches the sibling `.ppt` parser). **Throw** +on: `nFib` below `nFib97` or an unknown `nFibNew` (newer-than-2007 does *not* +throw — uses the 2007 layout); `ccpText` with the sign bit set (§2.5.5, it's +signed and MUST be ≥ 0); a `csw`/`cslw` count too small to cover an array we read; +an unexpected Clx lead byte (not `0x01`/`0x02`); non-ascending CP boundaries; a bad +compressed byte or early EOF. **Pass through** what we don't model: text after the +main body, `Prc` formatting runs, and every control/field char `clean_text` drops. **Endianness.** Host byte order / LSB-first bit-fields assumed; shared `oldms/` -assumption, analysis and fix plan in [`../AGENTS.md`](../AGENTS.md). +assumption + fix plan in [`../AGENTS.md`](../AGENTS.md). -## Tests - -- `OldMs.doc_read_string_compressed` — the compressed (1-byte-per-CP) decoder - against the §2.9.73 byte map: ASCII passthrough, the `0x82–0x9F` remap, the - `0xA0–0xFF` UTF-8 round-trip. +**The `font_size = 11pt` in the adapters is a placeholder hack** so empty +paragraphs still have height (same as the `.ppt`/`.xls` modules); removed when +character formatting lands (open work §1). `Document::is_editable()` → `true` and +`is_savable(!encrypted)`, but `save`/`text_set_content` throw — read-only in +practice. -The FIB-robustness behaviours (negative `ccpText` rejected, newer-than-2007 -`nFib` falling back to the 2007 layout) and the `0x0C` page-break emission are -**not yet unit-tested**; there is also **no assertion-based render test** over a -real `.doc` fixture (unlike the `.ppt` cases). +**`clean_text` control-character handling** (the non-obvious cases): `0x0D`/`0x0C`/ +`0x0B` are consumed by the caller's paragraph/page/line splits and never reach +`clean_text`; `0x13`/`0x14`/`0x15` delimit a **field** — instruction hidden, +result shown, the `0x14` separator optional (§2.8.25), nesting tracked with a +per-field stack; `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; `0x1F` +optional hyphen dropped; every other control char < `0x20` dropped. ---- - -# Open work - -## 1. Character (font) formatting → the IR (the next feature) +## Tests -**Goal.** Extract per-run character properties (font name, size, bold, italic, -underline, strikethrough, colour, highlight) and surface them through the -abstract model's `TextStyle`, so the HTML renderer styles text instead of -emitting one flat 11pt run. This replaces the `font_size = 11pt` placeholder in -`doc_document.cpp`. - -`TextStyle` (`src/odr/style.hpp`) maps almost 1:1 onto the `.doc` character -SPRMs: - -| `TextStyle` field | SPRM (opcode) | operand → value | -|---------------------|--------------------------|------------------------------------------------------------| -| `font_size` | `sprmCHps` (0x4A43) | u16 **half-points** → `Measure(hps/2.0, pt)` (default 20 = 10pt) | -| `font_weight` | `sprmCFBold` (0x0835) | `ToggleOperand` → `FontWeight::bold` when on | -| `font_style` | `sprmCFItalic` (0x0836) | `ToggleOperand` → `FontStyle::italic` when on | -| `font_underline` | `sprmCKul` (0x2A3E) | `Kul` value, `0x00` = none → `bool` | -| `font_line_through` | `sprmCFStrike` (0x0837) | `ToggleOperand` → `bool` | -| `font_color` | `sprmCCv` (0x6870) | `COLORREF` → `Color`; legacy `sprmCIco` (0x2A42) is a palette index | -| `background_color` | `sprmCHighlight` (0x2A0C)| `Ico` highlight index → `Color` | -| `font_name` | `sprmCRgFtc0` (0x4A4F) | s16 index into `SttbfFfn` → font name (intern it; see below) | - -`font_name` is a `const char *`, so the resolved name needs stable storage — -intern it in the `ElementRegistry` (e.g. a `std::deque` whose -elements never move) and hand out the pointer. - -**How `[MS-DOC]` stores & retrieves character properties** — the authoritative -algorithm is **Direct Character Formatting** (§2.4.6.2), which reuses the -*Retrieving Text* walk we already have: -1. For a character at `cp`, run *Retrieving Text* (§2.4.1) to get its byte - offset `fc` in `/WordDocument` and the owning `Pcd` (we already compute both). -2. Read the **`PlcBteChpx`** (§2.8.5) at `fcPlcfBteChpx`/`lcbPlcfBteChpx` in the - table stream — a PLC keyed by **stream offset**: `aFC[n+1]` boundaries + - `aPnBteChpx[n]` (`PnFkpChpx`, 4 bytes each). -3. Find the largest `i` with `aFC[i] ≤ fc`; read a **`ChpxFkp`** (§2.9.33) at - `aPnBteChpx[i].pn * 512` in `/WordDocument` (a fixed 512-byte page: `rgfc` - run boundaries, parallel `rgb` offsets, `crun` in the last byte). -4. Find the largest `j` with `rgfc[j] ≤ fc`; the `Chpx` (§2.9.32) lives at - `rgb[j] * 2` within the page. `Chpx.grpprl` is an array of **`Prl`** = `Sprm` - (2 bytes) + operand. -5. Append the `Pcd.Prm` modifications (§2.9.214–216): a `Prm0` (inline) or - `Prm1` (index) carrying extra SPRMs for this run. - -`Prl`/`Sprm` is already modelled in `doc_structs.hpp` (`Sprm` with -`ispmd/fSpec/sgc/spra` and `operand_size()`); a **character** property is a SPRM -with `sgc == 2`. Walk each `Chpx.grpprl` by reading a 2-byte `Sprm` then -`operand_size()` operand bytes (note `spra == 6` is length-prefixed/variable), -keeping only the opcodes above. +- `OldMs.doc_read_string_compressed` — the compressed decoder against the §2.9.73 + byte map (ASCII passthrough, `0x82–0x9F` remap, `0xA0–0xFF` round-trip). -**First cut — direct formatting only.** Implement §2.4.6.2 (`Chpx.grpprl` + -`Pcd.Prm`) and map the table's SPRMs. Captures the common case: bold/italic/ -size/font/colour applied directly to runs. Resolve `sprmCRgFtc0` by reading -**`SttbfFfn`** (§2.9.286) once at `fcSttbfFfn`/`lcbSttbfFfn` (an STTB of `FFN` -records; `FFN.xszFfn` is the UTF-16 font name) and indexing it. Drop the -hardcoded 11pt; use 10pt (the `sprmCHps` default of 20 half-points). - -**Full fidelity — styles (later).** *Determining Formatting Properties* -(§2.4.6.6) layers, in order: document defaults → `STSH` (§2.4.6.5, -`fcStshf`/`lcbStshf`) paragraph- and character-style `grpprl`s resolved via the -paragraph's `istd` → table-style props → direct paragraph → direct character. -The first cut skips the STSH layer, so style-dependent props fall back to -defaults; wiring the STSH closes that gap. - -**Wiring to the abstract model.** Today `parse_tree` concatenates all body -pieces into one `body_text` and emits one `text` per paragraph. Per-run styling -needs run boundaries, expressed in `/WordDocument` byte offsets -(`ChpxFkp.rgfc`, `PlcBteChpx.aFC`) — so: -1. **Keep the FC↔text mapping.** While concatenating, retain each piece's - `data_offset` and compression so any character's source `fc` is recoverable - (the `CharacterIndex` already holds this; thread it through instead of - discarding it after building `body_text`). -2. **Split paragraphs into runs.** Within a paragraph, cut at every `ChpxFkp` - run boundary inside it, resolve each run's `TextStyle` once, and emit a - **`span`** (`ElementType::span`, already wired via `SpanAdapter`) per run, - with the `text` element(s) as its children. Paragraph/line-break splitting - stays as-is. -3. **Store the style.** Add a `TextStyle` side-map to `ElementRegistry` keyed by - span id (mirror the text side-payload, and the frame-payload pattern in - `presentation`) plus the font-name intern store. `SpanAdapter::span_style` - returns the stored style; `text_style` / `paragraph_text_style` then return - `{}` (or the paragraph mark's run style) instead of the 11pt hack. +**Not yet tested:** FIB robustness (negative `ccpText`, newer-than-2007 fallback), +`0x0C` page-break emission, and there is **no assertion-based render test** over a +real `.doc` fixture (unlike `.ppt`). -## 2. Coverage gaps +## Binary format reference (FIB) -- **Only the main document body.** `parse_tree` stops at the `ccpText` budget, - so headers/footers, footnotes, endnotes, comments/annotations, and text boxes - — each its own CP range after the body (`ccpFtn`, `ccpHdd`, `ccpAtn`, … in - FibRgLw97, located via the matching `plcf*` in the table stream) — are - dropped. Extending coverage means walking the later CP ranges and their - `Plcf*` structures. -- **Tables.** Cell text renders as plain paragraphs: the end-of-cell mark `0x07` - is dropped by `clean_text` and row/cell structure (§2.4.3, `sprmPFInTable` / - `sprmPTtp` / the `TC`/`TAP` tables) is unmodelled. Reconstruct table structure - from the paragraph properties to emit real `table`/`row`/`cell` elements. - Paragraph-level formatting (alignment, indent, spacing) via `PlcBtePapx` → - `PapxFkp` belongs here too, alongside the character work. -- **Fields show only the cached result.** `clean_text` keeps the field *result* - and drops the *instruction* (§2.8.25); page numbers, dates, refs show their - last-saved value and are never evaluated. Acceptable for "visible text". -- **Images / OLE / drawn objects.** The anchor characters (`0x01` inline - picture, `0x08` floating picture, OLE) are dropped. No image extraction; would - require `PlcfSpa` / the Office Art (`dggInfo`) drawing data. -- **Encrypted / obfuscated documents.** `FibBase.fEncrypted` / `fObfuscated` are - parsed but not acted on; `decrypt` throws `UnsupportedOperation`. - XOR-obfuscated and `[MS-OFFCRYPTO]`-encrypted `.doc` are unsupported. +Root of every `.doc`, at offset 0 of `/WordDocument`. Variable-length, +self-describing — each counted array follows its count: -## 3. Smaller shortcomings +``` +FibBase 32 B fixed (wIdent, nFib, flags incl. fWhichTblStm/fEncrypted, …) +csw u16 → fibRgW csw·u16 +cslw u16 → fibRgLw cslw·u32 (ccpText at u16 idx 6–7, §2.5.5) +cbRgFcLcb u16 → fibRgFcLcb cbRgFcLcb·FcLcb (clx → piece table; version by nFib) +cswNew u16 → fibRgCswNew cswNew·u16 (nFibNew overrides nFib when present) +``` -- **Endianness.** Shared `oldms/` shortcoming — see [`../AGENTS.md`](../AGENTS.md). - For `.doc`: every field is read in host byte order, and the - `FibBase`/`Sprm`/`FcCompressed` bit-fields in `doc_structs.hpp` assume - LSB-first allocation. +`nFib` handled: `nFib97` 0x00C1, `nFib2000` 0x00D9, `nFib2002` 0x0101, `nFib2003` +0x010C, `nFib2007` 0x0112. Above 2007 → `FibRgFcLcb2007` layout; below 97 throws. -## Reference: the read path +### Read path ``` WordDocument stream └─ FIB @ 0 [MS-DOC] §2.5.1 ├─ FibBase (32 B): fWhichTblStm, fEncrypted, nFib - ├─ csw·u16 fibRgW ├─ cslw·u32 fibRgLw → ccpText (idx 6–7) §2.5.5 - ├─ cbRgFcLcb·FcLcb fibRgFcLcb → clx.fc §2.5.7 (version by nFib) - └─ cswNew·u16 fibRgCswNew → nFibNew + └─ cbRgFcLcb·FcLcb fibRgFcLcb → clx.fc §2.5.7 (version by nFib) Table stream (/1Table or /0Table per fWhichTblStm) §1.4 └─ Clx @ clx.fc §2.9.38 @@ -340,30 +115,107 @@ Table stream (/1Table or /0Table per fWhichTblStm) §1.4 ├─ fCompressed=0 → UTF-16 @ fc └─ fCompressed=1 → 8-bit @ fc/2 (+ 0x82–0x9F map) -Retrieving Text algorithm: §2.4.1 (steps 1–6, matches parse_tree) -Field characters 0x13/0x14/0x15: §2.8.25 +Retrieving Text: §2.4.1 (steps 1–6) Field chars 0x13/0x14/0x15: §2.8.25 ``` -Character-formatting path (open work §1), keyed by `/WordDocument` byte offset -`fc`: +--- + +# Open work + +## 1. Character (font) formatting → the IR (the next feature) + +**Goal.** Extract per-run character properties (font name, size, bold, italic, +underline, strike, colour, highlight) and surface them through `TextStyle`, so the +renderer styles text instead of emitting one flat 11pt run. Replaces the +`font_size = 11pt` placeholder. + +`TextStyle` (`src/odr/style.hpp`) maps almost 1:1 onto the `.doc` character SPRMs: + +| `TextStyle` field | SPRM (opcode) | operand → value | +|---|---|---| +| `font_size` | `sprmCHps` (0x4A43) | u16 **half-points** → `Measure(hps/2, pt)` (default 20 = 10pt) | +| `font_weight` | `sprmCFBold` (0x0835) | `ToggleOperand` → `bold` | +| `font_style` | `sprmCFItalic` (0x0836) | `ToggleOperand` → `italic` | +| `font_underline` | `sprmCKul` (0x2A3E) | `Kul`, `0x00` = none → `bool` | +| `font_line_through` | `sprmCFStrike` (0x0837) | `ToggleOperand` → `bool` | +| `font_color` | `sprmCCv` (0x6870) | `COLORREF` → `Color` (legacy `sprmCIco` 0x2A42 is a palette index) | +| `background_color` | `sprmCHighlight` (0x2A0C) | `Ico` highlight index → `Color` | +| `font_name` | `sprmCRgFtc0` (0x4A4F) | s16 index into `SttbfFfn` → font name | + +`font_name` is a `const char *`, so the resolved name needs stable storage — intern +it in the `ElementRegistry` (e.g. a `std::deque` whose elements never +move) and hand out the pointer. + +**How `[MS-DOC]` retrieves character properties** — Direct Character Formatting +(§2.4.6.2) reuses the *Retrieving Text* walk we already have: +1. For `cp`, run §2.4.1 to get its byte offset `fc` and owning `Pcd` (already + computed). +2. Read **`PlcBteChpx`** (§2.8.5) at `fcPlcfBteChpx` in the table stream — a PLC + keyed by stream offset: `aFC[n+1]` + `aPnBteChpx[n]` (`PnFkpChpx`, 4 B). +3. Largest `i` with `aFC[i] ≤ fc` → read a **`ChpxFkp`** (§2.9.33) at + `aPnBteChpx[i].pn * 512` (fixed 512-byte page: `rgfc` boundaries, parallel + `rgb`, `crun` in the last byte). +4. Largest `j` with `rgfc[j] ≤ fc` → the `Chpx` (§2.9.32) at `rgb[j] * 2` within + the page. `Chpx.grpprl` is an array of `Prl` = `Sprm` (2 B) + operand. +5. Append `Pcd.Prm` mods (§2.9.214–216): `Prm0` (inline) or `Prm1` (index). + +`Prl`/`Sprm` is already modelled (`Sprm` with `ispmd/fSpec/sgc/spra` + +`operand_size()`); a **character** property is a SPRM with `sgc == 2` (note +`spra == 6` is length-prefixed/variable). + +**First cut — direct formatting only.** Implement §2.4.6.2 (`Chpx.grpprl` + +`Pcd.Prm`) and map the SPRMs above (bold/italic/size/font/colour applied directly +to runs). Resolve `sprmCRgFtc0` by reading **`SttbfFfn`** (§2.9.286) once and +indexing it. Drop the 11pt; use 10pt (the `sprmCHps` default). + +**Full fidelity — styles (later).** *Determining Formatting Properties* (§2.4.6.6) +layers: document defaults → `STSH` (§2.4.6.5) para/char-style `grpprl`s via the +paragraph's `istd` → table-style → direct paragraph → direct character. The first +cut skips STSH (style-dependent props fall back to defaults). + +**Wiring to the abstract model.** Per-run styling needs run boundaries in +`/WordDocument` byte offsets, so: +1. **Keep the FC↔text mapping** while concatenating (thread the `CharacterIndex` + through instead of discarding it after `body_text`). +2. **Split paragraphs into runs** at every `ChpxFkp` boundary, resolve each run's + `TextStyle`, emit a **`span`** (already wired via `SpanAdapter`) per run. +3. **Store the style** in a `TextStyle` side-map keyed by span id (mirror the text + side-payload + the `presentation` frame-payload) plus the font-name intern + store. `SpanAdapter::span_style` returns it; `text_style`/ + `paragraph_text_style` then return `{}` instead of the 11pt hack. + +Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: ``` Table stream -├─ PlcBteChpx @ fcPlcfBteChpx §2.8.5 -│ └─ aFC[n+1] (stream offsets) + aPnBteChpx[n] (PnFkpChpx, 4 B) -├─ SttbfFfn @ fcSttbfFfn (font names, FFN.xszFfn) §2.9.286 -└─ STSH @ fcStshf (styles — full fidelity only) §2.4.6.5 +├─ PlcBteChpx @ fcPlcfBteChpx §2.8.5 aFC[n+1] + aPnBteChpx[n] (PnFkpChpx, 4 B) +├─ SttbfFfn @ fcSttbfFfn (font names, FFN.xszFfn) §2.9.286 +└─ STSH @ fcStshf (styles — full fidelity only) §2.4.6.5 WordDocument stream └─ ChpxFkp @ aPnBteChpx[i].pn * 512 (512-byte page) §2.9.33 - ├─ rgfc[crun+1] run boundaries (stream offsets) - ├─ rgb[crun] → Chpx @ rgb[j]*2 within page - └─ crun (last byte) - └─ Chpx = cb + grpprl(Prl[]) §2.9.32 - └─ Prl = Sprm (2 B) + operand §2.2.x - └─ character SPRMs have sgc == 2; + Pcd.Prm §2.9.214–216 - -Direct Character Formatting: §2.4.6.2 (Determining Formatting Properties: §2.4.6.6) -Font SPRMs: CHps 0x4A43, CFBold 0x0835, CFItalic 0x0836, CKul 0x2A3E, - CFStrike 0x0837, CCv 0x6870, CHighlight 0x2A0C, CRgFtc0 0x4A4F + ├─ rgfc[crun+1] boundaries (stream offsets), rgb[crun] → Chpx @ rgb[j]*2, crun (last byte) + └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 ``` + +## 2. Coverage gaps + +- **Only the main body.** Headers/footers, footnotes, endnotes, comments, text + boxes — each its own CP range after the body (`ccpFtn`/`ccpHdd`/`ccpAtn`/… in + FibRgLw97, via the matching `plcf*`) — are dropped. +- **Tables.** Cell text renders as plain paragraphs (`0x07` end-of-cell dropped; + `TC`/`TAP` unmodelled). Reconstruct from paragraph properties (`sprmPFInTable`/ + `sprmPTtp`) to emit real `table`/`row`/`cell`. Paragraph-level formatting + (`PlcBtePapx` → `PapxFkp`) belongs here too. +- **Fields show only the cached result** (§2.8.25) — never evaluated. Acceptable + for "visible text". +- **Images / OLE / drawn objects** — anchor chars dropped; would need `PlcfSpa` / + Office Art (`dggInfo`). +- **Encrypted / obfuscated** — `fEncrypted`/`fObfuscated` parsed but not acted on; + `decrypt` throws. + +## 3. Smaller shortcomings + +- **Endianness** — shared `oldms/` shortcoming, see [`../AGENTS.md`](../AGENTS.md). + For `.doc`: fields read in host byte order; `FibBase`/`Sprm`/`FcCompressed` + bit-fields assume LSB-first. diff --git a/src/odr/internal/pdf/AGENTS.md b/src/odr/internal/pdf/AGENTS.md index 6ad8ccff..9074ee32 100644 --- a/src/odr/internal/pdf/AGENTS.md +++ b/src/odr/internal/pdf/AGENTS.md @@ -1,731 +1,312 @@ -# In-house PDF support (`pdf/`) — status, design & roadmap +# In-house PDF support (`pdf/`) — design & roadmap -What the `pdf/` module does **today**, the **design decisions** behind it, and -the **roadmap** for turning it into a faithful renderer. Reference links -(web resources; offline spec docs are planned) live in [`README.md`](README.md). +The **why** behind the `pdf/` module and the roadmap. What is concretely +implemented is in the code; this file keeps the rationale, the non-obvious +invariants, and where things live. Reference links live in [`README.md`](README.md). -This is the `DecoderEngine::odr` path for PDF; the sibling `../pdf_poppler/` +This is the `DecoderEngine::odr` path for PDF. The sibling `../pdf_poppler/` module (poppler / pdf2htmlEX, behind `ODR_WITH_PDF2HTMLEX`) is the -production-quality alternative engine. - -**Scope today.** Parse the PDF object/file structure (classic cross-reference -tables, cross-reference streams, object streams, hybrid files, with a -forward-scan recovery path for broken cross-references), build the page -tree with fonts and annotations, tokenize page content streams into graphics -operators, and emit an **HTML rendering**: absolutely positioned -text spans, one per shown segment, placed by the full text transform (CTM × text -matrix) and advanced by the parsed glyph widths, recursing into form XObjects, -with text render modes and `/ActualText` honoured and omitted spaces inferred -from glyph gaps, pages sized from `MediaBox`. Encrypted files are decrypted (RC4, -AES-128, AES-256). Embedded font programs (TrueType, CFF, Type1) render through -`@font-face`; non-embedded fonts are substituted to CSS `font-family` stacks -(with standard-14 AFM widths); and vector graphics, images, shadings, tiling -patterns, Type3 glyphs and transparency (constant alpha + blend modes) render as -inline SVG per page. Experimental and not production-quality. +production-quality alternative engine; this one is experimental. ---- - -## What works - -- `.pdf` is detected by file magic and opened as `PdfFile` - (`DecoderEngine::odr`); `is_decodable()` returns `false`. Page-tree/content - parsing is lazy (on HTML request), but `file_meta()` carries a best-effort - `document_meta`: the page count (root `/Pages /Count`, a single lookup) and - the `/Info` document-information strings (Title/Author/Subject/Keywords/ - Creator/Producer/CreationDate/ModDate, decoded from UTF-16BE-BOM or - PDFDocEncoding via `decode_text_string`), read once at construction (an - owner-locked file is unlocked with the empty password first so its `/Info` - decrypts). A malformed structure simply leaves `document_meta` minimal. XMP - metadata is not yet parsed (see *Other known gaps*). -- **Object syntax**: null, booleans, integers/reals, names (incl. `#xx` - escapes), literal strings (the `\n \r \t \b \f` control escapes, `\ddd` - octal, escaped delimiters, and `\`-before-EOL line continuation — Table 3), - hex strings, arrays, dictionaries, indirect references (`n g R`) — - standalone and nested. -- **File structure**: header, `n g obj … endobj`, `stream` payloads (via - `/Length`, with a scan-to-`endstream` fallback), classic `xref` tables, - `trailer`, `startxref`, `%%EOF`; both sequential reading (`read_entry`) and - random access via the xref table. **Incremental updates**: `startxref` found - by scanning the file tail, then the `Prev` chain is followed (cycle-guarded), - merging xref tables so the newest entry for each object wins. -- **Cross-reference streams, object streams, hybrid files** (PDF 1.5+): each - trailer-chain section may be a classic table or a cross-reference stream - (`/W`/`/Index`/`Size`, decoded via the filter framework, entry types 0/1/2; - unknown types treated as absent). Xref entries are a tagged union - (`FreeEntry`/`UsedEntry`/`CompressedEntry`); compressed objects are read from - their object stream (`/N`/`/First` header, decoded once and cached per - stream). Hybrid files follow the `XRefStm`-before-`Prev` lookup order. - Lenient where the wild demands: `/Type /XRef` only warns, references to free - or absent objects resolve to null with a `Logger` warning, `n g obj` need not - end with a newline. -- **Cross-reference recovery**: when the trailer-chain walk throws (missing or - garbage `startxref`, a broken `Prev` chain) or the document fails to build - (no `/Root`, offsets pointing at the wrong objects), the whole file is - forward-scanned for `n g obj` starts, rebuilding a synthetic xref (last - definition of an id wins). `trailer` dictionaries are collected for `/Root`, - `/Encrypt`, `/ID`; recovered `/Type /ObjStm` members are indexed as - compressed entries; and, when no trailer supplied a `/Root`, a `/Type - /Catalog` object is searched. Handles e.g. an HTTP response saved as `.pdf` - (every offset shifted by the header). -- **Page tree**: `Catalog` → `Pages` (recursive) → `Page` with per-page - `Resources` (fonts only) and `Annots` (raw dictionary only). Objects cached by - reference (`DocumentParser::m_objects`). -- **Inherited page attributes**: the inheritable set per spec Table 30 — - `Resources`, `MediaBox`, `CropBox`, `Rotate` — resolved by threading an - accumulator down the `Pages` recursion (no `Parent` walk). Each `Page` carries - the resolved `media_box`/`crop_box`/`rotate` and its resolved `resources`. - Lenience: `CropBox` defaults to `MediaBox`, `Rotate` normalized to - {0,90,180,270}, a `MediaBox` missing everywhere falls back to US Letter, a - missing `Resources` to an empty dict — all with a `Logger` warning. -- **Stream filters** (`pdf_filter`): `/Filter` and `/DecodeParms` honoured, - including chains and the inline-image abbreviations — FlateDecode and - LZWDecode (both with TIFF and PNG predictors), ASCIIHexDecode, ASCII85Decode, - RunLengthDecode. Image codecs (DCTDecode, JPXDecode, CCITTFaxDecode, - JBIG2Decode) are deliberately not decoded: `decode()` stops and hands back the - still-encoded payload for the image path; `read_decoded_stream` treats them as an - error. The `Crypt` filter passes through only as `Identity`. -- **Encryption** (`pdf_encryption`): the standard security handler. An - `Authenticator` parses `/Encrypt` and authenticates the password (user then - owner; the empty password is tried first, so owner-locked files open - transparently), producing a `Decryptor` that decrypts object strings and - streams. RC4 (V 1/2, R 2/3, 40–128 bit), - AES-128 crypt filters (V 4, R 4 — `StdCF` with `V2`/`AESV2`, `Identity`, - honouring `StmF`/`StrF`) and AES-256 (V 5, R 6, AESV3) are all supported, - including owner-only files and `EncryptMetadata false`. Streams are decrypted - before `/Filter` decoding; cross-reference streams and object-stream members - are left untouched. The user password is never retained: once `authenticate` - succeeds, the derived key lives only inside the `Decryptor` (no accessor), and - `PdfFile` carries the whole authenticated `Decryptor` forward — from the - encryption probe to the render parse — so the HTML service unlocks the - document without re-deriving the key. Permission bits (`/P`) are recorded, not - enforced. -- **Fonts / text mapping**: a font's `ToUnicode` CMap stream is decoded and - parsed. The `CMap` is multi-byte aware: `codespacerange` declares the code - widths, `bfchar` and both `bfrange` forms (destination increment and explicit - array) are applied, and destinations may be multi-character (ligatures). - `translate_string` splits a string into codes of the codespace-declared width; - an unmapped code passes through as its numeric value (identity for single - bytes). When a simple font carries no `ToUnicode` CMap, `Font::to_unicode` - falls back to its `/Encoding` — a base encoding (Standard/WinAnsi/MacRoman) - overlaid with `/Differences`, each code → glyph name → Unicode via the Adobe - Glyph List (incl. the `uniXXXX`/`uXXXXXX` forms). **Composite - (Type0) fonts** are recognized: the descendant CIDFont's - `/CIDSystemInfo` `/Registry`/`/Ordering` is recorded on the `Font`, and the - Type0 `/Encoding` (a code → CID CMap such as `Identity-H`) is kept out of the - simple-font encoding path. An *embedded* `/Encoding` CMap stream is parsed - (`cidchar`/`cidrange` → `Font::cid_encoding`) so `Font::codes()` yields CIDs - through it — this also carries the authoritative codespace, so a producer that - mixes a 1-byte code (e.g. a space) among 2-byte CIDs stays aligned and selects - the right glyph/advance. Extraction is driven by the `/ToUnicode` CMap (the - common case — every Type0 font in the corpus carries one). When a composite - font has no `/ToUnicode`, the named `/Encoding` still resolves the text - (`pdf_cid`, backed by the generated `pdf_cid_data` tables): a **predefined - Unicode `/Encoding`** — the `Uni*-UCS2/UTF16/UTF32` CMaps — is decoded directly - (those codes already are Unicode, big-endian), and a **legacy CJK CMap** - (`90ms-RKSJ-H`, `GBK-EUC-H`, `B5pc-H`, `KSC-EUC-H`, …) maps code → CID through - its own codespace + range tables and then CID → Unicode through its character - collection's table. When `/Encoding` is `Identity-H/V` (code == CID) or an - embedded CMap stream (`Font::codes()` already yields the CIDs), the collection - named by the descendant CIDFont's `/CIDSystemInfo` supplies the CID → Unicode - step directly. Only a genuinely unmapped code (a CMap we ship no tables for, an - `Identity` ordering, an unlisted CID) falls through to the embedded font's - reverse map below, and failing that yields "no Unicode" (not byte-garbage). -- **Glyph metrics**: a font's advance widths are parsed — - `/FirstChar` + `/Widths` + `/FontDescriptor` `/MissingWidth` for simple fonts, - `/W` + `/DW` (the descendant CIDFont, both `c [w…]` and `c_first c_last w` - forms) for composite fonts. `Font::advance_width(code)` returns the advance in - text-space units (glyph-space / 1000), falling back to `/MissingWidth` or `/DW`. - Codes outside the corpus are interpreted as CIDs for composite fonts (identity). - A non-embedded standard-14 font (which ships no `/Widths`) falls back to a - **generated AFM width table** (see *Graphics, images & transparency*). -- **Embedded font programs**: every embedded flavor is decoded into an - `abstract::Font` held on `Font::embedded_font` — `/FontFile2` (TrueType, simple - or composite `CIDFontType2`) → `SfntFont`; `/FontFile3` (CFF / `Type1C` / - `CIDFontType0C`) → a bare `CffFont`, or an `SfntFont` when the program is a full - OpenType SFNT; `/FontFile` (Type1) → translated to a CFF (`type1::to_cff`) and - read as a `CffFont`, so it reuses the whole CFF path. An explicit `/CIDToGIDMap` - stream is read into `Font::cid_to_gid` (empty = `Identity`). A malformed program - is logged and leaves `embedded_font` null (fallback path). `Font::glyph_for_code(code)` - maps a character code to a glyph id — composite: CID → GID via `/CIDToGIDMap`; - simple TrueType: the embedded `cmap`, else the code's Unicode, else the code as a - GID (best effort, ISO 32000-1 9.6.6.4). Two consumers read it: the HTML layer - renders the actual glyphs (see *HTML*), and `Font::to_unicode` gains an - **embedded-font reverse map** as its final fallback (code → glyph → - `code_point_for_glyph`), so a font with neither a `/ToUnicode` CMap nor a usable - `/Encoding` becomes selectable; a partially mapped run recovers what it can. -- **Content streams**: the full graphics-operator vocabulary is tokenized; - `GraphicsState` executes a subset (state stack `q`/`Q`, matrices `cm`/`Tm`, - line parameters, text state `Tc`/`Tw`/`Tz`/`TL`/`Tf`/`Tr`/`Ts`, glyph metrics - `d0`/`d1`, grey/RGB/CMYK colors). The CTM **concatenates** on `cm` (ISO 32000-1 - 8.4.4); the text matrix `Tm` and text line matrix `Tlm` are tracked as 2-D - affine `Transform2D` values (`util/math_util.hpp`), with `BT` resetting them, `Td`/`TD` - /`T*` (and the line-move half of `'`/`"`) advancing `Tlm` → `Tm`. Unknown - operators are logged to stderr and skipped. -- **Text layout** (`pdf_page_text`): `extract_text` runs the - operator parser + `GraphicsState` over a page's content and emits a - renderer-agnostic `TextElement` per shown *segment* (one `Tj`/`'`/`"`, or one - string of a `TJ` array) — its text-space → user-space transform (CTM × `Tm`, - with horizontal scaling and rise folded in, font size kept separate), the - resolved font, size, spacing parameters, raw codes, the CMap-translated - Unicode, and the segment's per-code advances plus their total. Font lookup is - lenient (unknown ref → warn, raw codes). **Glyph advances are applied**: after - each segment the text matrix `Tm` advances by the glyph widths × font size plus - char/word spacing (× horizontal scaling), and a `TJ` number translates `Tm` by - `−n/1000 × Tfs × Th` — so segments, `TJ` kerning and lines land in the right - place. The element carries the per-code advances directly, so a renderer wanting - per-glyph placement need not re-derive them from `font->advance_width`. Each - element also carries its text render mode (`Tr`) and a `no_unicode` flag — set - when the font's code → Unicode chain yields nothing (a composite font with no - `/ToUnicode` or usable predefined encoding), so `text` is empty and the run is - knowingly non-extractable (its glyphs still render via the embedded program's - PUA re-encode; see *HTML*). - Marked content is tracked (`BMC`/`BDC … EMC`, balanced per stream and reset - across form invocation): a sequence carrying `/ActualText` (inline property - dictionary, or a name resolved through the `/Properties` resource) overrides the - per-glyph Unicode of its enclosed shows — the decoded text (UTF-16BE BOM or - PDFDocEncoding) emitted once and the rest of the sequence suppressed. - **Space inference**: a running pen (the user-space origin after - each segment, with the writing-line direction and em scale) is threaded through - the executor, and a single space is prepended to a segment's `text` when the - gap past the previous pen exceeds ~0.2 em along the line (a word break) or - ~0.5 em perpendicular (a new line) — recovering the inter-word/-line spaces the - producer omitted, in `text` only (codes/advances/placement untouched). Still - deferred: vertical writing-mode advances (the deferred bidi & vertical writing - work, see *Other known gaps*). -- **Form XObjects**: a resource dictionary's `/XObject` - subdictionary is parsed into `Resources::x_object`; each `/Subtype /Form` is an - `XObject` element carrying its `/Matrix` (default identity), its decoded - content stream (read eagerly at parse time, so text extraction needs no parser - handle), and its own parsed `/Resources` (or `nullptr` to inherit the invoking - scope). `Do` in `extract_text` saves the state, concatenates the form `/Matrix` - onto the CTM, runs the form content against the form's resources (falling back - to the enclosing scope), then restores — so text inside forms is placed - correctly. Image XObjects (`/Subtype /Image`) are decoded and emitted (see - *Graphics, images & transparency*); unknown subtypes are inexecutable. The - parser memoizes XObjects by - reference (`ObjectReference -> XObject*`), so a form shared across pages is - parsed once and a cyclic form reference resolves to the existing element — the - in-memory graph mirrors the file, cycles included. A render-time active-set - guard cuts cyclic invocation (the spec forbids it, 8.10.1, but real files - contain it). `/BBox` clipping is deferred. -- **HTML**: one `document.html` view; each page is a `div` sized from `MediaBox` - (points → inches). Each `TextElement` becomes an absolutely positioned `span` - carrying a CSS `transform` matrix (the placement transform mapped from PDF user - space — y-up, MediaBox origin — into the page box in CSS pixels, the glyphs - un-mirrored so text stays upright), `font-size` from the text state, and the - Unicode text. Invisible render modes (`Tr` 3/7) keep the span but paint it - transparent (the `.i` class) so OCR-over-scan text stays selectable. - **Embedded fonts**: a run whose font carries a usable program is - emitted as a **dual layer** — a visible glyph layer (PUA code points in the - font's `@font-face`, `aria-hidden` + `user-select:none`) overlaid by a - transparent selectable layer carrying the real Unicode, so display is - glyph-exact while copy/search read the Unicode (and the PUA code points never - reach the clipboard). Each embedded program is re-encoded to the PUA once and - wrapped into an OTF — after extraction, so the in-place re-encode can't disturb - the reverse-map / glyph lookups that read the original `cmap` — and served as an - inline `@font-face`; - too many glyphs for the BMP PUA falls back to the default font. A run with no - embedded program keeps the single-span fallback path; one with neither a - program nor extractable text (a `no_unicode` run or an `/ActualText`-suppressed - show) still emits no span. -- **Baseline placement.** PDF's text origin is the glyph *baseline*, but a CSS - span anchors its box *top*, which sits one ascent above the baseline — so left - uncorrected every run renders ~one ascent too low (highlights/underlines, which - are painted correctly from user space, then float a line above their text). - Each run is therefore raised by one font ascent so the baseline lands on the - origin: in the uniform branch the shift comes off `top` - (`top = (m.f - ascent_em·m.a·size)·pt_to_px`), in the general branch it goes - *through* the matrix, subtracted along the local y axis `(c, d)` from the - translation (it cannot be applied to `m.f` after the fact). The dual layer - shares one shift: the nested PUA glyph layer is positioned relative to its - parent, so placing the parent moves both. - - **Why `line-height:1`.** The browser puts the first baseline at - `top + half_leading + ascent`, with `ascent` read from the *rendering* font's - `hhea`/`OS/2` and `half_leading` from `line-height`. `line-height:1` (on both - `.t` and the nested-glyph `placement` constant) zeroes the half-leading band - so the box-top→baseline distance is just the ascent; default `normal` would - add an unknown offset. This is exact when ascent + descent ≈ 1 em, near so - otherwise. We control the rendering metric: the re-encode synthesizes - `OS/2`/`hhea` (`font/cff_transform.cpp` `serialize_os2`/`serialize_hhea`; the - SFNT path passes the originals through), so our ascent can match the - browser's. - - **Non-embedded substitutes** render in a *local* system font whose - `hhea`/`OS/2` we do **not** control, so the box-top→baseline distance would - be that font's ascent, not our `ascent_em` — dropping e.g. a 120pt Times - title well below its intended baseline. `SubstituteFontFaces` (in - `pdf_file.cpp`) closes this by routing each substitute through a generated - `@font-face` (`'odr-sN'`, `src: local(...)` of the family stack) carrying - `ascent-override:ascent_em`, `descent-override:1−ascent_em`, - `line-gap-override:0` — so the browser positions the baseline from *our* - metric. Faces are deduped by (family, style, ascent); the family stack is - kept after `'odr-sN'` as a fallback for the rare unresolved local. - - **`ascent_em`** (in `pdf_file.cpp`): FontDescriptor `/Ascent`, else the - embedded font's `bounding_box().y_max / units_per_em()`, else `0.8` em (which - matches `serialize_os2`'s degenerate 0.8/0.2 fallback, so the fallback font - and our math agree); clamped to `[0.5, 1.2]`. It is **font-format agnostic** - — only `abstract::Font` virtuals (`bounding_box`, `units_per_em`) plus the - PDF descriptor — so SFNT, bare-CFF, and Type1 (→CFF) all work with no - per-subclass code. - - **Refinements left open.** The bounding-box fallback is coarser than the - font's designed ascender; an SFNT-specific `OS/2.sTypoAscender`/`hhea` read - would match the browser better, but `/Ascent` is present and preferred in - almost all real PDFs so the fallback rarely fires. When `/Ascent` and the - embedded `OS/2` disagree, which to prefer (descriptor states intent, embedded - matches rendering) is unsettled — currently `/Ascent` wins. The ascent is - per-font, not per-CID. No focused unit test yet (verified visually on - `style-various-1.pdf`); the reference-output snapshot test is the gate. -- **Link annotations** (`html/pdf_file.cpp`): a page's `/Link` annotations - (ISO 32000-1 12.5.6.5) become absolutely-positioned `` overlays. A `/URI` - action is an external link (attribute-escaped `href`, opened in `_blank` via - the document target); a `/GoTo` action or a direct `/Dest` is an internal - `#pN` link — each page `div` carries a matching `id="pN"`. Destinations - resolve a page *reference* (the first element of a `[page …]` array) to its - index, including named destinations via the catalog `/Dests` dictionary and - the `/Names /Dests` name tree (depth-guarded). The `/Rect` is mapped through - the page transform to page-box points. Resolved entirely in the HTML layer - from the parser's raw annotation dictionaries — no parser/IR change. Both - render modes emit them. Deferred: remote/launch actions (`/GoToR`, `/Launch`), - the destination's scroll position/zoom (only the target page is used), and - non-`/Link` annotation appearances (see *Interaction & navigation*). - -## Graphics, images & transparency - -Vector and raster content renders as **inline SVG per page**, layered under the -text spans — serialization, not rasterization (**decision, 2026-06**: no native -renderer; pdf.js proves the full PDF graphics model needs none, and the PDF and -SVG imaging models are close cousins, so the mapping is mostly mechanical). The -rasterized-background fallback is rejected: it reintroduces the exact renderer -dependency the `odr` engine exists to avoid. Fidelity is bounded by operator -coverage rather than by a renderer's long tail; the reference-output test guards -it. - -- **Paths**: `extract_page` produces a page-element IR (`PathElement`, - `ImageElement`, `ShadingElement`, `TextElement`) in paint order; the executor - resolves geometry through the CTM at construction, so the renderer maps it - through the page transform alone. Fills (nonzero / even-odd) and strokes (width, - caps, joins, miter limit, dash) → `` with the matching SVG attributes; a - zero-width stroke floors to a hairline. -- **Clipping** (`W`/`W*`): the current clip (an intersection of path regions) is - snapshotted onto each painted element and installed as a per-page `` - referenced by `clip-path` (deduplicated by geometry). Text is not yet clipped - (see *Other known gaps*). -- **Colour spaces & functions**: DeviceGray/RGB/CMYK directly; CIE-based, - ICCBased, Indexed, Separation/DeviceN and Lab resolve to RGB at emission time by - sampling the tint `/Function` (types 0/2/3/4, evaluated by `pdf_function`). CMYK - uses a naive (no-ICC) conversion; overprint is ignored. -- **Images**: `DCTDecode` JPEG pass-through → ``; Flate/LZW rasters - re-encoded as PNG; inline images (`BI`/`ID`/`EI`); `/ImageMask` stencils painted - in the current fill colour; `/SMask` and `/Mask` (stencil + colour-key) - composited into RGBA on the raster path (a mask on a JPEG base is left alone — - decoding the JPEG to composite is out of scope). An image's `/ColorSpace` may - be a device space, an inline array (Indexed/ICCBased/…), or a *name* resolved - against the enclosing `/Resources /ColorSpace` table (the table is parsed - before the `/XObject` table and threaded into image parsing). -- **Shadings** (axial type 2, radial type 3): `parse_shading` pre-samples the - tint function into sRGB stops, so the renderer needs no evaluator. `sh` floods - the clip (a `` filled with the gradient); a `/PatternType 2` shading - pattern named by `scn` fills a path. Both emit - ``/`` with `gradientUnits="userSpaceOnUse"`. - `/Extend`/`/Background`/`/BBox` are parsed but not yet honoured (see gaps). -- **Tiling patterns** (`/PatternType 1`): the cell content stream runs as a mini - page and becomes an SVG `` tile repeated every `/XStep`/`/YStep`, with - `patternTransform` placing the lattice; coloured (`/PaintType 1`) cells carry - their own colours, uncoloured (`/PaintType 2`) cells paint in the current fill - colour. Each cell is clipped to its `/BBox` (overlapping lattices and nested - tile content are gaps; see gaps). -- **Non-embedded fonts**: a font with no `/FontFile*` is *substituted*, not - rendered — `/BaseFont` + `/FontDescriptor` flags map to a CSS `font-family` - fallback stack (serif/sans/mono, bold/italic), and the standard-14 metrics - (which ship no `/Widths`) come from a **generated AFM width table** - (`tools/pdf/generate_afm_data.py`, pinned to PDFBox). Glyph shapes are the - browser's fallback font. -- **Type3 fonts**: the glyphs are `/CharProcs` content streams drawn in glyph - space and mapped by `/FontMatrix`. Each glyph runs through the same graphics - executor at `/FontMatrix × size × Tm × CTM`, so it paints as ordinary - path/image elements; the shown run stays selectable (Unicode from the - code → Unicode chain) but paints no visible text of its own (`render_as_graphics`, - like an invisible `Tr 3` run). Recursion is depth-guarded. -- **Transparency**: `/ExtGState` constant alpha (`ca`/`CA`) → - `fill-opacity`/`stroke-opacity`/`opacity`, blend modes (`/BM`) → - `mix-blend-mode` (the 16 separable/non-separable modes map 1:1; unmappable names - render normal), and soft masks (`/SMask`) → an SVG ``. All are part of the - saved graphics state, so `q`/`Q` scope them like the CTM. A soft mask's - transparency group `/G` is rendered by the extractor into a `SoftMask` (its - content in user space); the HTML `MaskRegistry` serializes that into a - `` (luminosity → the SVG luminance default, `/Alpha` → `mask-type`, - `/BC` → a backdrop rect), and the painted element is wrapped in a masked ``. - A transparency group (`/Group` with `/S /Transparency`) is treated as a unit: - the constant alpha, blend mode and soft mask in force when it is invoked fold - into the group's output even when the group's own content resets them — the - drop-shadow / faded-reflection idiom. Text under a soft mask is not yet masked - (see gaps), and per-element folding approximates true group compositing where - interior paints overlap. - -The **reference-output snapshot test** (`test/data/reference-output/`) is the -graphics oracle: each change regenerates it and the diff is reviewed. (An -automated perceptual-diff gate is deferred; see gaps.) The snapshot is now the -**odr engine's own output only** — the `pdf2htmlEX`/poppler (`*.pdf-poppler`) and -wvWare (`*.doc-wvware`) cross-engine reference snapshots were removed once the odr -engine was ready to replace them, so the test no longer runs those engines. - -## Module layout +**Goal.** Faithful read-only HTML for common real-world PDFs through a +pure-serialization pipeline (no native renderer), so the poppler engine becomes +optional rather than required. The file-format, text-extraction, font and +graphics foundations are in place; what remains is interaction & navigation plus +a tail of known gaps (see *Roadmap*). -| File (`pdf/`) | Role | -|----------------------------------------|-------------------------------------------------------| -| `pdf_object.{hpp,cpp}` | Object model: `Object` (`std::any`-based variant), `Array`, `Dictionary`, `Name`, `StandardString`/`HexString`, `ObjectReference`; `to_stream`/`to_string` dumping | -| `pdf_object_parser.{hpp,cpp}` | Tokenizer over `std::streambuf`: whitespace/lines, numbers, names, strings, arrays, dictionaries, references | -| `pdf_file_object.{hpp,cpp}` | File-structure entries: `Header`, `IndirectObject`, `Trailer`, `Xref` (tagged-union entries, `append`/`merge_hybrid`), `StartXref`, `Eof`, the `Entry` any-holder; `parse_xref_stream_table` and the `ObjectStream` payload wrapper | -| `pdf_file_parser.{hpp,cpp}` | File-level reads on top of `ObjectParser`: indirect objects, xref, trailer, startxref, stream payloads, `seek_start_xref` | -| `pdf_filter.{hpp,cpp}` | Stream filter framework: `decode()` over the `/Filter`/`/DecodeParms` chain; ASCIIHex/ASCII85/LZW/Flate/RunLength decoders, TIFF/PNG predictors; image codecs returned undecoded (`DecodeResult::stopped_at_filter`) | -| `pdf_document_parser.{hpp,cpp}` | `parse_document()`: xref/trailer chain → catalog → page tree; lazy object reads with cache; (deep) reference resolution; resources incl. the `/XObject` table, with an `ObjectReference → XObject*` cache that dedups shared forms and breaks cyclic form references; loads each font's embedded program (`/FontFile2` → `SfntFont`, `/FontFile3` → `CffFont`/`SfntFont`, `/FontFile` Type1 → `CffFont` via `type1::to_cff`) and `/CIDToGIDMap` | -| `pdf_encryption.{hpp,cpp}` | Standard security handler: `Authenticator` (parse `/Encrypt`, authenticate password → `Decryptor`) and `Decryptor` (decrypt strings/streams; RC4, AES-128, AES-256), plus a `standard_security` namespace of pure key/password algorithms for known-answer tests | -| `pdf_document.hpp` | `Document`: arena of `Element`s + `catalog` pointer | -| `pdf_document_element.hpp` | Element structs: `Catalog`, `Pages`, `Page`, `Annotation`, `Resources` (font + XObject + `/Properties` tables), `XObject` (Form/Image subtype, `/Matrix`, decoded content, own `/Resources`), `Font` (incl. the `composite`/`cid_registry`/`cid_ordering` Type0 facts, the `/Widths`-`/W`/`/DW` glyph metrics + `advance_width`, the `embedded_font`/`cid_to_gid` + `glyph_for_code`, and `to_unicode` with its embedded reverse-map fallback) | -| `pdf_cmap.{hpp,cpp}` | `CMap`: 1-byte glyph → UTF-16 `bfchar` map + string translation | -| `pdf_cmap_parser.{hpp,cpp}` | `ToUnicode` CMap stream parser (`begincodespacerange` sets code widths; `beginbfchar` and both `beginbfrange` forms applied) | -| `pdf_encoding.{hpp,cpp}` | Simple-font `/Encoding` → Unicode: `BaseEncoding` tables, `/Differences` overlay (`Encoding`), glyph-name → Unicode via AGL + `uniXXXX`/`uXXXXXX` | -| `pdf_cid.{hpp,cpp}` | Composite-font predefined `/Encoding` → Unicode: `Uni*-UCS2/UTF16/UTF32` decoded directly; legacy CJK CMaps mapped code → CID (interned-range binary search) → Unicode (collection bitmap rank); plus `cid_to_unicode(registry, ordering, cid)` for the `Identity-H/V` + embedded-stream path | -| `pdf_cid_data.{hpp,cpp}` | **Generated** (`tools/pdf/generate_cid_data.py`): the S3-packed legacy-CMap tables — interned code → CID ranges + per-CMap `uint16` index lists, and a per-collection CID → Unicode presence bitmap + rank index + `uint16` value array (astral escape). ~0.58 MB compiled | -| `pdf_encoding_data.{hpp,cpp}` | **Generated** (`tools/pdf/generate_encoding_data.py`): base-encoding tables + the Adobe Glyph List as a name-sorted array | -| `util/math_util.hpp` | `util::math::Transform2D`: 2-D affine transform (PDF row-vector convention) — compose, point-apply, translation/scaling factories | -| `pdf_graphics_operator.hpp` | `GraphicsOperatorType` enum (full operator set) + `GraphicsOperator` (type + `Object` arguments) | -| `pdf_graphics_operator_parser.{hpp,cpp}` | Content-stream tokenizer: arguments then operator name | -| `pdf_graphics_state.{hpp,cpp}` | `GraphicsState`: stack of `State` (general/path/text/color), `execute(op)` for the modelled subset; CTM/`Tm`/`Tlm` as `Transform2D`, `text_placement_matrix()` for the text rendering transform sans font size, `advance_text()` for the post-glyph `Tm` advance, `save()`/`restore()`/`concat_matrix()` reused by `q`/`Q`/`cm` and by form-XObject invocation | -| `pdf_page_text.{hpp,cpp}` | `extract_text`: run the content stream through `GraphicsState`, emit a `TextElement` (placed transform + font/size/spacing + codes + Unicode + per-code advances + total advance + render mode + `no_unicode` flag) per shown segment, advancing `Tm` by the glyph widths and `TJ` adjustments; `Do` recurses into a form XObject (state save / `/Matrix` concat / scoped resources / restore) with an active-set cycle guard; a marked-content stack applies `/ActualText` and the no-Unicode marking; a running pen infers omitted inter-word/-line spaces | -| `pdf_file.{hpp,cpp}` | `abstract::PdfFile` wrapper; probes encryption at construction and implements `password_encrypted()`/`decrypt()`, carrying the authenticated `Decryptor` (not the password) so rendering needs no re-derivation | - -Consumers outside the module: `open_strategy.cpp` (detection / engine -selection) and `html/pdf_file.cpp` (`create_pdf_service`; also the per-font PUA -re-encode + OTF wrap + `@font-face` and the dual-layer glyph/Unicode span -emission, using the `font/` module — `sfnt_*`, `cff_*`, `type1_*`). - -## Pipeline: how a `.pdf` becomes HTML - -1. **Wiring.** `open_strategy` maps `FileType::portable_document_format` to - `PdfFile`; `DecoderEngine::poppler` (or the unknown-file-type fallback) can - yield a `PopplerPdfFile` instead when built with `ODR_WITH_PDF2HTMLEX`. - `html::translate(PdfFile)` picks the matching HTML service. -2. **Locate the xref.** `seek_start_xref` seeks to `EOF − 64`, scans for - `startxref`; `read_start_xref` yields the most recent xref offset. - (`read_header` exists but `parse_document` does not call it — the `%PDF-` - header is only checked by magic detection earlier.) -3. **Walk the trailer chain.** `read_xref_section` dispatches: a classic table - (`read_xref` + `read_trailer`) or a cross-reference stream (an indirect - object whose dictionary doubles as the trailer dict; payload decoded via the - filter framework, entries via `parse_xref_stream_table`). A trailer `XRefStm` - (hybrid file) is read next and fills entries the classic table lacks or marks - free (`merge_hybrid`). Sections merge into the accumulated table - (`std::map::insert` keeps the first/newest entry), then `Prev` is followed - (cycle-guarded). The first/newest trailer provides `Root`. -4. **Build the page tree.** `parse_catalog` → `parse_pages` recurses over - `Kids` (dispatching on `Type`). Each `Page` keeps its raw dictionary, its - `Contents` reference(s), parsed `Resources` (the `Font` table — each font's - `ToUnicode` CMap parsed if present — and the `/XObject` table, with form - XObjects' content/`/Matrix`/nested `/Resources` read eagerly and memoized by - reference) and `Annots` (raw). `read_object` - dispatches on the xref entry kind: used → seek + `read_indirect_object`; - compressed → owning object stream decoded once, cached, member parsed from - the cached payload; free/absent → null with a warning. Parsed objects cached - by reference. -5. **Decode content.** Per page (depth-first), the `Contents` streams are read, - decoded through their `/Filter` chain (`read_decoded_stream`), concatenated - with a newline between streams. -6. **Lay out and emit.** `extract_text` runs `GraphicsOperatorParser` + - `GraphicsState` over the content and returns a `TextElement` per shown - segment, each placed by `text_placement_matrix()` (CTM × `Tm`, with horizontal - scaling and rise folded in), its glyphs translated through the font's CMap. - After each segment `Tm` is advanced by the glyph widths (`advance_width`) plus - char/word spacing, and `TJ` numbers translate `Tm` directly, so segments and - lines land correctly. `Do` recurses into a form XObject — state saved, the - form `/Matrix` concatenated onto the CTM, the form content run against its own - resources (or the enclosing scope), state restored — guarded against cyclic - invocation. The HTML layer maps each element to a positioned `span` - with a CSS `transform` (PDF user space → the page box in CSS pixels) and - `font-size` from the text state. An embedded font program renders the real - glyphs via `@font-face` (dual layer); a non-embedded font is substituted to a - CSS `font-family` stack, and non-text content (paths, images, shadings, - patterns, transparency) emits inline SVG. +**Scope in one line.** Parse the PDF object/file structure (xref tables, xref +streams, object streams, hybrid files, forward-scan recovery), decrypt (RC4, +AES-128, AES-256), build the page tree, tokenize content streams, and emit HTML: +absolutely-positioned text spans placed by the full text transform, with vector +graphics / images / shadings / patterns / transparency as inline SVG per page, +embedded fonts via `@font-face` and non-embedded fonts substituted to CSS stacks. --- -## Design decisions +## Design decisions (the load-bearing part) -**Stream-based parsing with seeks, lazy object access.** Everything is parsed -off a `std::istream`/`std::streambuf` — no full-file buffer. Random access -(xref lookups, stream payloads) seeks; sequential tokenizing uses -single-character peek/bump (`geti`/`getc`/`bumpc`). Objects are parsed only when -referenced, and parsed `IndirectObject`s are cached by reference, so shared -objects are read once. Positions are `std::uint32_t` (files ≥ 4 GiB are out of -scope). +**Rendering is deferred to the browser; display and text are decoupled.** We emit +no rasterized output — glyphs render via the embedded font (`@font-face`), vector +content via SVG. Because display is driven by the *glyph*, not the extracted +Unicode, **text-extraction gaps degrade only selectability/search — never +display.** Every glyph is re-encoded to the Private Use Area (`U+E000 + glyph +index`, the uniform re-encode, decision 2026-06-19) for display, with the +extracted Unicode carried separately to drive selection/search; runs with no +recoverable Unicode are additionally marked non-extractable +(`user-select:none`, `aria-hidden`). So even an unshipped-CMap CJK PDF *looks* +right; only its selectability is affected. + +**No native renderer — SVG serialization, not rasterization (decision, 2026-06).** +pdf.js proves the full PDF graphics model needs none, and the PDF and SVG imaging +models are close cousins, so the mapping is mostly mechanical. The rasterized- +background fallback is rejected: it reintroduces the exact renderer dependency the +`odr` engine exists to avoid. Fidelity is bounded by operator coverage, not a +renderer's long tail; the reference-output snapshot test guards it. + +**Font handling is in-house, facts-vs-glyphs split.** No FontForge (pdf.js proves +it unnecessary; heavy build, and no read-only library can inject the PUA mappings +we need). A thin `abstract::Font` exposes the *facts* every consumer needs (glyph +count, glyph→Unicode, advance widths, units-per-em, name, bbox, symbolic flag) +while glyph outlines pass through **byte-for-byte** (no decompile/recompile). +Every embedded flavor feeds the one path: SFNT (`/FontFile2`), bare CFF +(`/FontFile3`), Type1 (`/FontFile` → CFF via `type1::to_cff`). + +**Text mapping is a fallback chain, not one method.** code → Unicode is tried in +order: `/ToUnicode` CMap → simple-font `/Encoding` (base + `/Differences` → glyph +name → AGL) → composite predefined `/Encoding` (`Uni*` Unicode CMaps decoded +directly; legacy CJK CMaps via `pdf_cid`/`pdf_cid_data`: code → CID → Unicode) → +`/CIDSystemInfo` collection for `Identity-H`/embedded-CMap → embedded-font reverse +map (code → glyph → `code_point_for_glyph`). Only a genuinely unmapped code yields +"no Unicode" — never byte-garbage. The point of the chain is that each link +recovers a class of real-world PDF the previous one misses. **`std::any`-based object model.** `Object` holds its value in `std::any` with typed `is_*`/`as_*` accessors (mirrors `oldms/`'s `Entry`). Pro: one value type -throughout parser, document elements, and operator arguments. Con: no exhaustive -matching, RTTI lookups, and accidental copies are easy — `resolve_object_copy` -exists because rvalue access proved fiddly (see the `TODO why rvalue not -working?` in `pdf_document_parser.cpp`). - -**References are recognized by lookahead.** `n g R` reads as plain integers -until the `R` appears, so a standalone `read_object` returns the *id* integer of -a reference; the enclosing context folds it in. `read_array` reacts when the `R` -token itself shows up (array elements may be bare adjacent integers, so it -cannot promote earlier); `read_dictionary` values and indirect-object bodies use -`promote_indirect_reference`, where a digit after the value can only be a `gen` -(the next token is otherwise a key, `>>`, or `endobj`). All rewind-free. +throughout parser, document elements, operator arguments. Con: no exhaustive +matching, RTTI lookups, accidental copies easy — `resolve_object_copy` exists +because rvalue access proved fiddly (see the `TODO why rvalue not working?` in +`pdf_document_parser.cpp`). + +**References recognized by lookahead, rewind-free.** `n g R` reads as plain +integers until `R` appears, so standalone `read_object` returns the *id* integer +and the enclosing context folds it in. `read_array` reacts at the `R` token +(array elements may be bare adjacent integers, so it can't promote earlier); +`read_dictionary`/indirect-object bodies use `promote_indirect_reference` (a digit +after a value can only be a `gen`, since the next token is otherwise a key, `>>`, +or `endobj`). **Element tree as an arena.** `Document` owns all elements (`vector>`); `Catalog`/`Pages`/`Page`/… hold raw non-owning pointers plus their original dictionary (`Element::object`), so unmodelled keys -stay inspectable. Navigation is by typed `is_()`/`as_()` accessors over -`kids` — thin `dynamic_cast` wrappers mirroring `Object`'s `is_*`/`as_*` -surface (the former `Type` tag enum was dropped in favour of RTTI). +stay inspectable. Navigation is typed `is_()`/`as_()` `dynamic_cast` +wrappers over `kids` (the former `Type` tag enum was dropped for RTTI). + +**Stream-based parsing, lazy objects.** Everything parses off a +`std::istream`/`std::streambuf` — no full-file buffer. Random access seeks; +sequential tokenizing uses single-char peek/bump. Objects parse only when +referenced and cache by reference (shared objects read once). Positions are +`std::uint32_t` — **files ≥ 4 GiB are out of scope.** **Fail early on malformed structure, tolerate unknown content.** Structural surprises **throw** `std::runtime_error` (missing `obj`/`endobj`/`stream`/ -`endstream`/`xref`/`startxref`, unexpected characters, an unknown page-tree -element type, stream exhaustion). A missing or unresolvable `/Length` is -tolerated — the stream extent is recovered by scanning to `endstream`. Unknown -**content** is tolerated: unrecognized operators logged and skipped, unmodelled -operators ignored by `execute`, annotations keep their raw dictionary. An -unmapped CMap code passes through as its numeric value (identity for single -bytes). References to free/absent -objects resolve to null with a warning; unknown xref-stream entry types treated -as absent (7.5.8.3). A structural throw in the cross-reference layer is not -fatal, though: it is caught once and the file is forward-scanned to rebuild the -table (*Cross-reference recovery* above) before giving up. - -**Diagnostics route through `Logger`.** `DocumentParser` and `extract_text` take -a `Logger &` (default `Logger::null()`); no stray `stdout`/`stderr` prints remain -in the module — new diagnostics should follow the same pattern. +`endstream`/`xref`/`startxref`, unexpected chars, unknown page-tree element type, +stream exhaustion). A missing/unresolvable `/Length` is tolerated (scan to +`endstream`). Unknown *content* is tolerated: unrecognized operators logged and +skipped, annotations keep their raw dictionary, unmapped CMap codes pass through +as their numeric value, free/absent object refs resolve to null with a warning. +**Crucially, a structural throw in the cross-reference layer is not fatal** — it +is caught once and the file is forward-scanned to rebuild a synthetic xref +(handles e.g. an HTTP response saved as `.pdf`, every offset shifted by the +header) before giving up. + +**Diagnostics route through `Logger`.** `DocumentParser` and `extract_text` take a +`Logger &` (default `Logger::null()`); no stray `stdout`/`stderr` remains — new +diagnostics follow the same pattern. + +**Encryption: the derived key is never retained.** The empty password is tried +first (user then owner), so owner-locked files open transparently. Once +`authenticate` succeeds the key lives only inside the `Decryptor` (no accessor); +`PdfFile` carries the authenticated `Decryptor` forward from the encryption probe +to the render parse, so the HTML service unlocks without re-deriving. Permission +bits (`/P`) are recorded, not enforced. + +### Baseline placement (why the CSS is the way it is) + +PDF's text origin is the glyph *baseline*; a CSS span anchors its box *top*, one +ascent above. Left uncorrected, every run renders ~one ascent too low +(highlights/underlines, painted correctly from user space, then float a line +above their text). So each run is raised by one font ascent: + +- **Uniform branch**: shift comes off `top` + (`top = (m.f − ascent_em·m.a·size)·pt_to_px`). **General branch**: it goes + *through* the matrix, subtracted along the local y axis `(c,d)` (it can't be + applied to `m.f` after the fact). The dual layer shares one shift (the nested + PUA glyph layer is positioned relative to its parent). +- **`line-height:1`** (on `.t` and the nested-glyph placement) zeroes the + half-leading band so box-top→baseline is just the ascent; default `normal` + would add an unknown offset. Exact when ascent+descent ≈ 1 em. We control our + rendering metric: the re-encode synthesizes `OS/2`/`hhea` + (`font/cff_transform.cpp`), so our ascent matches the browser's. +- **Non-embedded substitutes** render in a *local* font whose metrics we don't + control. `SubstituteFontFaces` (`pdf_file.cpp`) routes each through a generated + `@font-face` (`'odr-sN'`, `src: local(...)`) carrying `ascent-override` / + `descent-override` / `line-gap-override:0`, so the browser positions the + baseline from *our* metric. Deduped by (family, style, ascent). +- **`ascent_em`** (`pdf_file.cpp`): FontDescriptor `/Ascent`, else embedded + `bbox.y_max / units_per_em`, else `0.8` em; clamped `[0.5, 1.2]`. Font-format + agnostic (only `abstract::Font` virtuals + the descriptor). Open: `/Ascent` vs + embedded `OS/2` disagreement (currently `/Ascent` wins); per-font not per-CID; + no focused unit test yet (guarded by the reference snapshot). -**Rendering is deferred to the browser; display and text are decoupled.** We emit -no rasterized output: glyphs render via the embedded font (`@font-face`) and -vector content via SVG — the browser draws everything. Because display -is driven by the *glyph*, not the extracted Unicode, **text-extraction gaps -degrade only selectability and search — never display.** A residual -code → Unicode case (a CMap we ship no tables for, an unlisted CID) still renders -correctly: all glyphs are re-encoded to the Private Use Area for display (the -uniform re-encode, decision 2026-06-19), with the extracted Unicode carried -separately to drive selection/search; runs with no recoverable Unicode are -additionally marked non-extractable (`user-select: none`, `aria-hidden`). So even -an unshipped-CMap CJK PDF looks right — only its text selectability is affected. -The common legacy-CJK selectability gap is now closed: the `pdf_cid_data` tables -resolve the predefined legacy CMaps and the `/CIDSystemInfo` collections (see -*Fonts / text mapping*). +--- -**Font handling is in-house, facts-vs-glyphs split.** No FontForge (pdf.js proves -it unnecessary; it is a heavy build and no read-only library can inject the PUA -mappings we need). A thin `abstract::Font` interface exposes the *facts* every -consumer needs (glyph count, glyph → Unicode, advance widths, units-per-em, name, -bbox, symbolic flag) while glyph outlines pass through **byte-for-byte** (no -decompile/recompile); every embedded flavor — SFNT (`/FontFile2`), bare CFF -(`/FontFile3`), Type1 (`/FontFile` → CFF via `type1::to_cff`) — feeds the one -path. Display uses a **uniform PUA re-encode** (`U+E000 + glyph index`) over every -glyph, with the extracted Unicode carried separately for selection/search (see -*Rendering is deferred* above). +## Non-obvious facts worth knowing + +Things the code won't shout at you: + +- **`is_decodable()` returns `false`** for PDF; page-tree/content parsing is lazy + (on HTML request). But `file_meta()` still carries a best-effort `document_meta` + (page count + `/Info` strings), read once at construction (an owner-locked file + is unlocked with the empty password first so `/Info` decrypts). XMP is not + parsed — `document_meta` is `/Info`-only. +- **Image codecs are deliberately not decoded** in the filter framework + (DCTDecode/JPXDecode/CCITTFaxDecode/JBIG2Decode): `decode()` stops and hands + back the still-encoded payload for the image path; `read_decoded_stream` treats + them as an error. `Crypt` passes through only as `Identity`. +- **Inherited page attributes** (`Resources`/`MediaBox`/`CropBox`/`Rotate`, Table + 30) are resolved by threading an accumulator down the `Pages` recursion — *not* + by a `Parent` walk. Lenience (all with a `Logger` warning): `CropBox` ← + `MediaBox`, `Rotate` normalized to {0,90,180,270}, missing `MediaBox` → US + Letter, missing `Resources` → empty dict. +- **Form XObjects are memoized by reference**, so a form shared across pages is + parsed once and a cyclic form reference resolves to the existing element (the + in-memory graph mirrors the file, cycles included). A render-time active-set + guard cuts cyclic *invocation* (spec forbids it, 8.10.1, but real files do it). + `/BBox` clipping is deferred. +- **Space inference**: producers omit inter-word/-line spaces. A running pen + threaded through the executor prepends a single space when the gap exceeds + ~0.2 em along the line or ~0.5 em perpendicular — in `text` only + (codes/advances/placement untouched). +- **`/ActualText`** (marked content, balanced per stream, reset across form + invocation) overrides the per-glyph Unicode of its enclosed shows: emitted once, + rest of the sequence suppressed. +- **`no_unicode` runs** (composite font, no `/ToUnicode` or usable encoding) emit + no selectable span but still render via the embedded program's PUA re-encode. +- **Type3 fonts** paint as ordinary path/image elements (`/CharProcs` run through + the graphics executor at `/FontMatrix × size × Tm × CTM`) but contribute no + visible text of their own (`render_as_graphics`, like an invisible `Tr 3` run); + the run stays selectable. Depth-guarded. +- **Link annotations** resolve entirely in the HTML layer from raw annotation + dictionaries (no parser/IR change): `/URI` → external ``, `/GoTo`/`/Dest` → + internal `#pN` (each page `div` carries `id="pN"`), named dests via `/Dests` + + the `/Names` name tree (depth-guarded). +- **CMYK is naive (no ICC); overprint ignored.** CIE/ICCBased/Indexed/Separation/ + DeviceN/Lab resolve to RGB at emission by sampling the tint `/Function` (types + 0/2/3/4). + +--- + +## Module layout + +| File (`pdf/`) | Role | +|---|---| +| `pdf_object.{hpp,cpp}` | Object model: `Object` (`std::any` variant), `Array`, `Dictionary`, `Name`, strings, `ObjectReference`; dumping | +| `pdf_object_parser.{hpp,cpp}` | Tokenizer over `std::streambuf` | +| `pdf_file_object.{hpp,cpp}` | File-structure entries (`Header`/`IndirectObject`/`Trailer`/`Xref` tagged-union/`StartXref`/`Eof`); `parse_xref_stream_table`, `ObjectStream` | +| `pdf_file_parser.{hpp,cpp}` | File-level reads on `ObjectParser`: indirect objects, xref, trailer, startxref, stream payloads, `seek_start_xref` | +| `pdf_filter.{hpp,cpp}` | `/Filter`/`/DecodeParms` chain: ASCIIHex/ASCII85/LZW/Flate/RunLength + TIFF/PNG predictors; image codecs returned undecoded | +| `pdf_document_parser.{hpp,cpp}` | `parse_document()`: xref/trailer chain → catalog → page tree; lazy cached object reads; resources incl. `/XObject` table with the dedup/cycle-breaking cache; loads embedded font programs + `/CIDToGIDMap` | +| `pdf_encryption.{hpp,cpp}` | Standard security handler: `Authenticator` → `Decryptor` (RC4, AES-128/256) + a `standard_security` namespace of pure algorithms for known-answer tests | +| `pdf_document.hpp` | `Document`: arena of `Element`s + `catalog` pointer | +| `pdf_document_element.hpp` | Element structs: `Catalog`/`Pages`/`Page`/`Annotation`/`Resources`/`XObject`/`Font` (incl. Type0 facts, glyph metrics, `embedded_font`/`glyph_for_code`, `to_unicode` fallback) | +| `pdf_cmap*.{hpp,cpp}` | `CMap` + the `ToUnicode` CMap stream parser | +| `pdf_encoding.{hpp,cpp}` | Simple-font `/Encoding` → Unicode: base tables, `/Differences`, AGL | +| `pdf_cid.{hpp,cpp}` | Composite predefined `/Encoding` → Unicode (Unicode CMaps direct; legacy CJK code→CID→Unicode); `cid_to_unicode(registry, ordering, cid)` | +| `pdf_cid_data.{hpp,cpp}` | **Generated** (`tools/pdf/generate_cid_data.py`): S3-packed legacy-CMap + collection tables. ~0.58 MB compiled | +| `pdf_encoding_data.{hpp,cpp}` | **Generated** (`tools/pdf/generate_encoding_data.py`): base encodings + AGL | +| `util/math_util.hpp` | `util::math::Transform2D`: 2-D affine (PDF row-vector convention) | +| `pdf_graphics_operator*.{hpp,cpp}` | Operator enum + `GraphicsOperator`; content-stream tokenizer | +| `pdf_graphics_state.{hpp,cpp}` | `GraphicsState`: state stack, `execute(op)` for the modelled subset; CTM/`Tm`/`Tlm`; `text_placement_matrix()`, `advance_text()`; `save`/`restore`/`concat_matrix` reused by `q`/`Q`/`cm` and form invocation | +| `pdf_page_text.{hpp,cpp}` | `extract_text`: content → `TextElement` per shown segment; `Do` recursion; marked-content/`ActualText`; pen-based space inference | +| `pdf_file.{hpp,cpp}` | `abstract::PdfFile`; probes encryption at construction, carries the authenticated `Decryptor` forward | + +Consumers outside the module: `open_strategy.cpp` (detection/engine selection) and +`html/pdf_file.cpp` (`create_pdf_service`; the per-font PUA re-encode + OTF wrap + +`@font-face`, and the dual-layer glyph/Unicode span emission, using `font/` — +`sfnt_*`, `cff_*`, `type1_*`). + +The **reference-output snapshot test** (`test/data/reference-output/`) is the +graphics oracle: each change regenerates it and the diff is reviewed. It is now +the **odr engine's own output only** — the `*.pdf-poppler` / `*.doc-wvware` +cross-engine snapshots were removed once the odr engine could replace them. --- ## Tests -- `test/src/internal/pdf/pdf_filter.cpp` — **assertion-based**, all inputs - inline strings: every decoder, predictors, chains, image-codec stop, +`test/src/internal/pdf/` — all **assertion-based**, inline strings or +test-builder mini-PDFs; a handful of end-to-end fixtures. What is covered: + +- **`pdf_filter`** — every decoder, predictors, chains, image-codec stop, `Crypt`/unknown-filter errors. -- `test/src/internal/pdf/pdf_file_object.cpp` — **assertion-based**, inline - only: cross-reference-stream entry decoding (field widths incl. 0, type - default, big-endian fields, subsections, unknown types, error paths), - `ObjectStream` header parsing and member lookup, `Xref::append` / - `Xref::merge_hybrid` precedence. -- `test/src/internal/pdf/pdf_encryption.cpp` — **assertion-based**, inline - vectors only: the standard security handler across R 2 (RC4-40), R 3 - (RC4-128), R 4 (AES-128/AESV2, incl. `EncryptMetadata false` and an - owner-locked file) and R 6 (AES-256). Vectors come from the real fixtures and - from `qpdf --encrypt` output frozen as literals — decrypting back to a known - marker, so no test is circular and no fixture file ships. - `crypto_util_test.cpp` covers the new MD5/RC4/SHA-384/512 primitives against - public standard vectors. -- `test/src/internal/pdf/pdf_document_parser.cpp` — **assertion-based** - whole-file tests over mini-PDFs assembled by the test-only - `pdf_test_file_builder.{hpp,cpp}` (computes xref offsets/`startxref`, so tests - show only the dictionaries; classic-table and uncompressed-xref-stream - variants), plus inherited-page-attribute coverage (a multi-level `Pages` tree: - per-page resolved `MediaBox`/`CropBox`/`Rotate`/`Resources`, override vs. - inheritance, the `CropBox` ← `MediaBox` default, the missing-`MediaBox` - US-Letter lenience), plus cross-reference-recovery coverage (inline broken - mini-PDFs: garbage prepended, a bad `startxref`, no trailer at all → catalog - scan, a duplicate id → last definition wins, a page tree living in an object - stream), plus composite-font coverage (a Type0 font over an `Identity-H` - descendant `CIDFontType2`: `composite`/`/CIDSystemInfo` recorded, 2-byte - `/ToUnicode` extraction, the no-`/ToUnicode` "no Unicode" fallback, and a - predefined `Uni*-UCS2-H` `/Encoding` extracting without a `/ToUnicode`), plus - glyph-metric coverage (the composite `/W`+`/DW` and a simple - `/FirstChar`/`/Widths`/`/MissingWidth` font, asserted through `advance_width`), - plus form-XObject coverage: a cyclic `/Resources` - reference (Fm0 → Fm1 → Fm0) terminating via the XObject cache and represented - faithfully (the back-edge points at the same cached element), and a form shared - by two pages parsed once. End-to-end: the classic fixture - `odr-public/pdf/style-various-1.pdf`, plus decryption of - `odr-public/pdf/Casio_WVA-M650-7AJF.pdf` (RC4, empty password) and - `odr-private/pdf/encrypted_fontfile3_opentype.pdf` (AES-256; skipped when the - private submodule is absent), and recovery of the real - `odr-private/pdf/order-EK52VKL0.pdf` (an HTTP response saved as `.pdf`; - likewise skipped when absent). The `odr-private` xref-stream/objstm/hybrid - fixtures (`basic_text.pdf`, `geneve_1564.pdf`, `test_fail.pdf`, `Kayla….pdf`, - `svg_background…issue402.pdf`, `Core_v5.1.pdf`, `onepage.pdf`) were verified - manually but are not pinned in unit tests. Also still contains the original - print-everything smoke test. -- `test/src/internal/pdf/pdf_file_parser.cpp` — sequential `read_entry` walk - (smoke) + assertion-based xref/trailer/root navigation over - `style-various-1.pdf`. -- `test/src/internal/pdf/pdf_cmap.cpp` — **assertion-based**, inline CMap - strings parsed through `CMapParser`: single- and two-byte `bfchar`, both - `bfrange` forms, multi-character (ligature) targets, the identity fallback for - unmapped codes, and mixed code widths driven by `codespacerange`. -- `test/src/internal/pdf/pdf_encoding.cpp` — **assertion-based**, no fixtures: - `base_encoding_from_name`, glyph-name → Unicode via the AGL (the `fi` ligature, - a multi-code-point decomposition, and the `name.suffix` form) and the - algorithmic `uniXXXX`/`uXXXXXX` forms, `Encoding::translate_string` with a base - encoding, the Latin-1 upper half (WinAnsi/MacRoman), a `/Differences` override, - and the WinAnsi-vs-Standard `0x27` divergence. -- `test/src/internal/pdf/pdf_cid.cpp` — **assertion-based**, no fixtures (the - vectors are derived from the Adobe CMap data by `tools/pdf` and frozen inline): - `translate_predefined_cmap` over the predefined Unicode CMaps — `UCS2`/`UTF16` - (incl. a surrogate pair) and `UTF32` decoding, a `-V` writing-mode variant — and - over the legacy CJK CMaps — a Shift-JIS (`90ms-RKSJ-H`, Adobe-Japan1), an EUC - (`GBK-EUC-H`, Adobe-GB1) and a Hojo (`Hojo-H`, deprecated Adobe-Japan2) `code → - CID → Unicode`, plus a mixed 1-/2-byte codespace staying aligned; `cid_to_unicode` - by `/CIDSystemInfo` collection (`Adobe`/`Japan1`, `Manga1` and `Japan2`, incl. - an astral escape and the unknown-collection/CID `nullopt`); and the `nullopt` - for `Identity-H` and an unshipped name. -- `test/src/internal/util/math_util_test.cpp` — **assertion-based**, no fixtures: - `Transform2D` point-apply (identity/translation/scaling), the ordered - (row-vector) composition, and compose-then-apply ≡ sequential apply. -- `test/src/internal/pdf/pdf_page_text.cpp` — **assertion-based**, inline content - streams through `extract_text`: `Td` translation, `Tm` scaling, `cm` CTM - concatenation under `Tm`, horizontal scaling and rise in the transform, and the - `T*`/`'`/`"` line moves with their leading and spacing; plus - glyph-advance coverage with hand-built `Font`s — simple `/Widths` advancing a - following show, `TJ` emitting per string with the numeric adjustment applied, - char spacing, word spacing on the single-byte space, the composite 2-byte `/DW` - advance, and the `advance_width` fallbacks; plus form-XObject - coverage with hand-built `XObject`s — invocation via `Do`, the - `/Matrix` placement, state restoration after the form, the form's own - `/Resources` scope, nested forms, image/unknown XObjects ignored, and a - self-referential form terminating at render time via the active-set guard; - plus render-mode and extraction-refinement coverage — `Tr` - propagation, a composite font with no `/ToUnicode` marked `no_unicode` with - empty text, and `/ActualText` overriding a segment (literal and UTF-16BE), - emitted once then suppressed across the sequence, resolved via a named - `/Properties` entry, the no-`/ActualText` passthrough, and a tolerated stray - `EMC`; plus space-inference coverage — a word-gap space, no space - when segments abut, the `TJ`-kern threshold (sub- vs. supra-threshold), a - new-line space, and the no-double-space after a trailing space. -- `test/src/internal/pdf/pdf_font_program.cpp` — **assertion-based**, no - fixtures: hand-built `Font`s over tiny in-memory `SfntFont`s (a compact SFNT - builder mapping a code run to glyph ids). `glyph_for_code` — composite - `Identity`, an explicit `/CIDToGIDMap` (incl. an out-of-range CID → `.notdef`), - a simple-font code reaching its glyph through the embedded `(3,1)` `cmap`, and - the no-program zero — and the embedded reverse-map `to_unicode`: recovery when - there is no `/ToUnicode` (with an unreachable glyph staying unmapped) and a - `/ToUnicode` CMap taking precedence over the reverse map. - -The tokenizer's string parsing is covered (`PdfObjectParser`: literal-string -control/octal/delimiter/line-continuation escapes and hex strings); references -and the HTML output itself (the span emission / CSS transform mapping, incl. the -dual-layer glyph/Unicode emission) are not yet asserted. +- **`pdf_file_object`** — xref-stream entry decoding, `ObjectStream`, + `Xref::append`/`merge_hybrid` precedence. +- **`pdf_encryption`** + `crypto_util_test` — the standard handler across R 2/3/4/6 + (vectors from real fixtures and frozen `qpdf --encrypt` output — no fixture + ships, no test is circular); MD5/RC4/SHA-384/512 primitives vs public vectors. +- **`pdf_document_parser`** — whole-file mini-PDFs via `pdf_test_file_builder` + (classic + xref-stream), inherited-page-attributes, cross-reference recovery, + composite fonts, glyph metrics, form-XObject cycles/sharing. End-to-end: + `style-various-1.pdf`, RC4 + AES-256 decryption, HTTP-saved-as-PDF recovery + (private fixtures skipped when the submodule is absent). +- **`pdf_file_parser`** — sequential walk + xref/trailer/root navigation. +- **`pdf_cmap` / `pdf_encoding` / `pdf_cid`** — the code→Unicode chain per link. +- **`pdf_page_text`** — `extract_text` placement, glyph advances, `TJ`, form + XObjects, render modes, `ActualText`, space inference. +- **`pdf_font_program`** — `glyph_for_code` + embedded reverse-map `to_unicode`. +- **`math_util_test`** — `Transform2D`. + +**Not yet asserted:** the HTML output itself (span emission / CSS transform +mapping, dual-layer glyph/Unicode). Several `odr-private` xref/objstm/hybrid +fixtures are verified manually but not pinned. --- # Roadmap -Goal: faithful read-only HTML for common real-world PDFs through the odr engine, -so the poppler/pdf2htmlEX engine becomes optional rather than required. The -file-format, text-extraction, font and graphics foundations are in place (see -*What works* and *Graphics, images & transparency*); what remains is interaction -& navigation plus a tail of known gaps. Each remaining item gets its own detailed -design before implementation. +The next feature cluster is **interaction & navigation**; the rest is a tail of +known gaps. Each remaining item gets its own detailed design before +implementation. Grow the corpus alongside (odr-public fixtures + the PDF101 +"nasty files" collection linked in `README.md`; assertion tests per feature). ## Interaction & navigation -The next feature cluster — needs destinations from the page tree, little else. -Link annotations (`/URI` + internal `/GoTo`) already land — see *What works*. +Link annotations (`/URI` + internal `/GoTo`) already land. Remaining: -- **Annotation appearances**: render `/AP` appearance streams (form XObjects - again) for highlights, stamps, form-field appearances; AcroForm - *interactivity* stays out of scope (read-only). -- **Remote/launch link actions** (`/GoToR`, `/Launch`) and destination scroll +- **Annotation appearances**: render `/AP` streams (form XObjects again) for + highlights/stamps/form-field appearances; AcroForm *interactivity* stays out of + scope (read-only). +- **Remote/launch actions** (`/GoToR`, `/Launch`) and destination scroll position/zoom (the internal-link handler uses only the target page). -- **Link overlays vs. text selection**: the `` overlays sit above the `.sel` - text, so a mousedown over a link starts navigation instead of a selection — - links are clickable but text under them cannot be selected, and there is no - hover cursor issue since native `:hover` applies. A JS workaround existed - (overlays `pointer-events:none` so mousedown falls through to `.sel`; a - once-per-document click handler re-enabled hit-testing via an `.lkhit` class, - `elementFromPoint`, and forwarded the click; a rAF-throttled `mousemove` - restored the pointer cursor via an `.lkc` class) but was rolled back as too - involved (reverted commit `5cfa8a09`, reachable for later). Options to - revisit: (a) reinstate that JS approach; (b) a CSS-only route if one exists; - (c) leave overlays clickable-only and accept no under-link selection. -- **Document outline** (`/Outlines`) → navigation anchors/sidebar. -- **Optional content groups** (layers): honor default visibility; no toggle UI. -- **Output scaling**: monolithic HTML vs. per-page lazy loading for large - documents (check what odr's HTML service model already provides first). - -## Cross-cutting (any time) - -- Grow a corpus: `odr-public` fixtures, the PDF101 "nasty files" collection - linked in `README.md`; assertion-based tests per feature. -- Spec docs offline under `offline/documentation/PDF/` (ISO 32000-1:2008, ISO - 32000-2:2020, Adobe PDF Reference 1.7, with markdown conversions); still to - do: fold them into `README.md` in place of the web links. - -## Other known gaps - -- **Graphics deferrals** (from stage 4, the *Graphics, images & transparency* - tail): - - **Soft masks** (`/SMask` in `/ExtGState` → SVG ``): done for - luminosity and alpha masks on graphic content (see *Transparency* above). - Remaining gaps: a soft mask (or group opacity/blend) on *text* is not applied - — the visible glyphs live in the HTML text layer, not the SVG, so the mask - would need a CSS `mask` on the line block or the run promoted into SVG (the - masked content in practice is paths/images, not text). Isolated vs. - non-isolated and knockout group semantics (`/I`, `/K`) are not distinguished; - a soft mask's own group is rendered with its default (black) backdrop unless - `/BC` is a plain device colour. - - **Mesh & function-based shadings** (types 1, 4–7): only axial (2) and radial - (3) are emitted; the rest would tessellate into flat polygons (pdf.js's - approach) — not done. - - **Non-extended shadings**: `/Extend`/`/Background`/`/BBox` are parsed onto - `Shading` but not honoured — the renderer always uses SVG's `pad` spread, so a - non-extended shading over-paints past its interval (needs the fill clipped to - the gradient band/annulus). - - **Overlapping tiling lattices** (a `/PatternType 1` step smaller than the - `/BBox`) can't be expressed as one SVG `` and are not reproduced; - nested text/shadings/patterns inside a tile are skipped (rare). - - **Text clipping**: the clip is not applied to text runs (paths and images are - clipped); text clip render modes (`Tr` 4–7) add nothing to the clip. - - **Perceptual-diff oracle**: the reference-output snapshot test (odr output - only) is the current graphics gate; an automated pdf.js screenshot-diff is - not built. -- **Encryption edge cases** (deferred from stage 0 until a real file needs - them): per-stream `/Crypt` filter `Name` overrides, the `EncryptMetadata - false` metadata-stream `Identity` special case, and `Perms` (Algorithm 13) - validation. The public-key security handler and revision 5 are out of scope. -- **Recovery limitations** (deferred from stage 0): when several `/Type - /Catalog` objects survive, the first in id order is picked rather than the - newest; the constructor-triggered recovery path cannot decode object streams - in an *encrypted* broken file (no decryptor yet), so such members go - unindexed. Both are edge cases beyond the corpus seen so far. -- **Linearized files** are not handled specially (the tail-first read usually - still works, but hint streams are ignored). -- **Bidi & vertical writing** (deferred): RTL run reordering for the - layout/selection order, and vertical writing mode (`Identity-V`/CJK — the - `/W2`/`/DW2` vertical metrics and a perpendicular pen advance, which the - horizontal-only `extract_text` and space inference assume away). No corpus - fixture needs either yet; revisit when one does. -- **Annotations**: `/Link` annotations render as `` overlays (see *What - works*); other annotation types are collected but their appearance streams are - not interpreted (*Interaction & navigation*). -- **XMP metadata** (`/Metadata` XML packet) is not parsed; `file_meta()`'s - `document_meta` comes from the `/Info` dictionary only. XMP would supplement - or override `/Info` for producers that write it. +- **Link overlays vs. text selection**: `` overlays sit above `.sel` text, so + text under a link can't be selected. A JS workaround (overlays + `pointer-events:none`, a click handler re-hit-tests via `elementFromPoint`) was + built then **reverted as too involved** (commit `5cfa8a09`, reachable for + later). Options: (a) reinstate it; (b) a CSS-only route if one exists; + (c) accept clickable-only. +- **Document outline** (`/Outlines`) → nav anchors/sidebar. +- **Optional content groups** (layers): honor default visibility, no toggle UI. +- **Output scaling**: monolithic HTML vs. per-page lazy loading (check what odr's + HTML service model already provides first). + +## Known gaps + +- **Graphics deferrals**: + - **Soft masks on *text*** are not applied (the visible glyphs live in the HTML + text layer, not the SVG; graphic content *is* masked). Isolated/knockout group + semantics (`/I`, `/K`) not distinguished; a soft mask's own group renders with + its default (black) backdrop unless `/BC` is a plain device colour. + - **Mesh & function-based shadings** (types 1, 4–7): only axial (2) / radial (3) + are emitted; the rest would tessellate into flat polygons — not done. + - **`/Extend`/`/Background`/`/BBox`** on shadings parsed but not honoured (always + SVG `pad` spread), so a non-extended shading over-paints past its interval. + - **Overlapping tiling lattices** (`/PatternType 1` step < `/BBox`) can't be one + SVG `` and are not reproduced; nested content inside a tile skipped. + - **Text clipping** (`Tr` 4–7 / clip paths on text) not applied. + - **Perceptual-diff oracle**: only the odr-output snapshot test gates graphics; + an automated pdf.js screenshot-diff is not built. +- **Encryption edge cases**: per-stream `/Crypt` `Name` overrides, the + `EncryptMetadata false` metadata-stream `Identity` special case, `Perms` + (Algorithm 13) validation. Public-key handler and revision 5 out of scope. +- **Recovery limits**: with several surviving `/Type /Catalog` objects the first + in id order wins (not the newest); the constructor-triggered recovery path + can't decode object streams in an *encrypted* broken file (no decryptor yet). +- **Linearized files** not handled specially (tail-first read usually still works; + hint streams ignored). +- **Bidi & vertical writing**: RTL reordering and vertical mode (`Identity-V`/CJK, + the `/W2`/`/DW2` metrics + perpendicular pen advance) deferred — `extract_text` + and space inference assume horizontal. No corpus fixture needs either yet. +- **XMP metadata** (`/Metadata` packet) not parsed; would supplement/override + `/Info`. +- **Spec docs offline** under `offline/documentation/PDF/` (ISO 32000-1/-2, Adobe + 1.7) planned, to replace the web links in `README.md`.