Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 103 additions & 124 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
# AGENTS.md β€” OpenDocument.core

Orientation for AI agents working in this repo. Summarises the architecture, the
conventions, and where to find things. For user-facing docs see
[`README.md`](README.md) and [`docs/`](docs/README.md).
Orientation for AI agents. Architecture, conventions, and where things live. For
user-facing docs see [`README.md`](README.md) and [`docs/`](docs/README.md).

## What this is

`odr` (a.k.a. `odrcore`) is a **C++20 library that decodes documents and renders
them to HTML**. It reads many formats (ODF, OOXML, legacy MS, PDF, CSV, …) behind
one abstract document model and a generic HTML renderer. It is the backend for
OpenDocument.droid / .ios.

Build system: **CMake + Conan**. Language standard: **C++20** (`CMakeLists.txt`).
them to HTML**. It reads many formats (ODF, OOXML, legacy MS binary, PDF, CSV, …)
behind one abstract document model and a generic HTML renderer. It is the backend
for OpenDocument.droid / .ios. Build: **CMake + Conan**; standard: **C++20**.

## Big picture: how a file becomes HTML

Expand All @@ -21,162 +18,144 @@ bytes ─▢ magic/open_strategy ─▢ DecodedFile ─▢ Document ─▢ Eleme
DecoderEngine) format) elements) walks public API)
```

1. **Detection** β€” `internal/magic.cpp` (+ `internal/libmagic`) sniffs the file;
`internal/open_strategy.cpp` picks a `FileType` and a `DecoderEngine` and
constructs the matching `abstract::DecodedFile`.
2. **Decode** β€” a document file yields an `abstract::Document` (the engine's
subclass of `internal::Document`).
1. **Detect** β€” `internal/magic.cpp` (+ `internal/libmagic`) sniffs the file;
`internal/open_strategy.cpp` picks a `FileType` + `DecoderEngine` and builds
the matching `abstract::DecodedFile`.
2. **Decode** β€” a document file yields an `abstract::Document`.
3. **Element tree** β€” a `Document` exposes a root `ElementIdentifier` plus an
`abstract::ElementAdapter`. The public, value-semantics handles
(`Element`, `Slide`, `Paragraph`, `Text`, `Frame`, …) in
`src/odr/document_element.hpp` are thin wrappers that delegate to the adapter.
4. **Render** β€” `internal/html/` walks that public element API and writes HTML.
Entry point: `odr::html::translate(...)` β†’ `HtmlService` (paginated fragments;
`abstract::ElementAdapter`. Public value-semantics handles (`Element`, `Slide`,
`Paragraph`, `Text`, `Frame`, …) in `src/odr/document_element.hpp` are thin
wrappers that delegate to the adapter.
4. **Render** β€” `internal/html/` walks the public element API and writes HTML.
Entry: `odr::html::translate(...)` β†’ `HtmlService` (paginated fragments;
`bring_offline` materialises files).

### The element-adapter pattern (every document engine follows it)

Two pieces per engine:
### The element-adapter pattern (every engine follows it)

- An **`ElementRegistry`**: a flat `std::vector<Element>` (id = index + 1) where
each `Element` holds `parent`/`first_child`/`last_child`/`prev`/`next` ids and a
`type`, plus side `unordered_map`s for per-type payloads (text strings, frame
anchors, …). Builders are `create_element` / `create_*_element` /
`append_child`. See `oldms/text/doc_element_registry.*` or
`oldms/presentation/ppt_element_registry.*` for the minimal version.
- An **`ElementAdapter`**: one class that implements `abstract::ElementAdapter`
(tree navigation by id) and, by multiple inheritance, the per-element-type
adapters it supports (`SlideAdapter`, `ParagraphAdapter`, `TextAdapter`,
`FrameAdapter`, …). The `*_adapter(id)` methods return `this` when the element
is of that type, else `nullptr`. See `oldms/presentation/ppt_document.cpp` for a
compact example.
- An **`ElementRegistry`**: a flat `std::vector<Element>` (id = index + 1); each
`Element` holds `parent`/`first_child`/`last_child`/`prev`/`next` ids and a
`type`, plus side maps for per-type payloads. Builders: `create_element` /
`create_*_element` / `append_child`. Minimal example:
`oldms/presentation/ppt_element_registry.*`.
- An **`ElementAdapter`**: one class implementing `abstract::ElementAdapter` (tree
navigation by id) and, via multiple inheritance, the per-type adapters it
supports (`SlideAdapter`, `ParagraphAdapter`, …). Each `*_adapter(id)` returns
`this` when the element is that type, else `nullptr`. Compact example:
`oldms/presentation/ppt_document.cpp`.

`ElementType` is the shared enum in `src/odr/document_element.hpp` (`root`,
`slide`, `paragraph`, `text`, `line_break`, `frame`, `table*`, `sheet*`, …).
`ElementType` is the shared enum in `src/odr/document_element.hpp`.

## Directory map

| Path | What |
|------|------|
| `src/odr/*.hpp` | **Public API**: `file.hpp`, `document.hpp`, `document_element.hpp`, `html.hpp`, `style.hpp`, `quantity.hpp` (`Measure`), `odr.hpp`. |
| `src/odr/internal/abstract/` | Core interfaces: `File`/`DecodedFile`, `Document` + `ElementAdapter` (and all per-element adapters), `Filesystem`, `Archive`, `HtmlService`. |
| `src/odr/*.hpp` | **Public API**: `file`, `document`, `document_element`, `html`, `style`, `quantity` (`Measure`), `odr`. |
| `src/odr/internal/abstract/` | Core interfaces: `File`/`DecodedFile`, `Document` + `ElementAdapter`, `Filesystem`, `Archive`, `HtmlService`. |
| `src/odr/internal/common/` | Reusable impls: `Path`/`AbsPath`, base `Document`, filesystem, `style`, table cursor/range, temp files. |
| `src/odr/internal/util/` | Helpers: `byte_stream_util` (POD reads), `string_util` (`split`, `u16string_to_string`), `stream_util`, `document_util`, `xml_util`. |
| `src/odr/internal/magic.*`, `open_strategy.*` | File-type detection and the open/dispatch logic. |
| `src/odr/internal/html/` | Generic HTML renderer (`document.cpp`, `document_element.cpp`, `document_style.cpp`). |
| `src/odr/internal/cfb/`, `zip/` | Container formats (Compound File Binary, ZIP). |
| `src/odr/internal/util/` | Helpers: `byte_stream_util`, `string_util`, `stream_util`, `document_util`, `xml_util`. |
| `src/odr/internal/magic.*`, `open_strategy.*` | File-type detection + open/dispatch. |
| `src/odr/internal/html/` | Generic HTML renderer. |
| `src/odr/internal/cfb/`, `zip/` | Container formats (CFB, ZIP). |
| `src/odr/internal/odf/` | OpenDocument (odt/ods/odp/odg). |
| `src/odr/internal/ooxml/` | OOXML (docx/pptx/xlsx); subdirs `text`/`presentation`/`spreadsheet`. |
| `src/odr/internal/oldms/` | **Legacy MS binary** (.doc/.ppt/.xls); subdirs `text`/`presentation`/`spreadsheet`. |
| `src/odr/internal/ooxml/` | OOXML (docx/pptx/xlsx). |
| `src/odr/internal/oldms/` | **Legacy MS binary** (.doc/.ppt/.xls). |
| `src/odr/internal/oldms_wvware/` | Alternative .doc decoder via wvWare. |
| `src/odr/internal/pdf/`, `pdf_poppler/` | PDF (own parser + poppler/pdf2htmlEX path). |
| `src/odr/internal/{csv,json,text,svm}/` | Smaller formats. |
| `cli/src/` | CLI tools: `translate`, `back_translate`, `meta`, `server`. |
| `tools/pdf/` | Developer tooling (not built): generators for the PDF engine's committed encoding data, see [`tools/pdf/README.md`](tools/pdf/README.md). |
| `test/src/` | GoogleTest suites; data in `test/data` (git submodules, see below). |
| `offline/documentation/MS-*/` | Vendored Microsoft spec text (PDF + extracted markdown), see [Specs](#specs). |
| `tools/pdf/` | Dev tooling (not built): PDF encoding-data generators, see `tools/pdf/README.md`. |
| `test/src/` | GoogleTest suites; data in `test/data` (git submodules). |
| `offline/documentation/MS-*/` | Vendored Microsoft spec text (see [Specs](#specs)). |
| `docs/design/README.md` | High-level design rationale. |

## Build & test

A configured build dir already exists (`cmake-build-relwithdebinfo`, also `…-debug`,
`…-release`). Typical loop:
Configured build dirs already exist. Typical loop:

```bash
# library
cmake --build cmake-build-relwithdebinfo --target odr
# tests (the ODR_TEST option is on in this build dir)
cmake --build cmake-build-relwithdebinfo --target odr_test
# run from inside the build dir so any test output stays out of the repo tree
cmake --build cmake-build-relwithdebinfo --target odr # library
cmake --build cmake-build-relwithdebinfo --target odr_test # tests (ODR_TEST on)
(cd cmake-build-relwithdebinfo && ./test/odr_test --gtest_filter='OldMs.*')
# CLI (renders a file to a directory of HTML)
cmake --build cmake-build-relwithdebinfo --target translate
cmake --build cmake-build-relwithdebinfo --target translate # CLI: file β†’ HTML dir
```

- **Use `cmake-build-relwithdebinfo`** as the default build/test directory (not
`cmake-build-debug`).
- **Only run the full suite when really necessary** β€” it takes a while. Default
to a targeted `--gtest_filter` for the area you touched.
- **For debugging, prefer the `translate` CLI** over the test suite β€” build and
run it on a single file to reproduce/inspect behaviour quickly.
- **Run the test binary from the build directory** (as above) so any files it
writes land there, not in the repo working tree.

Notable CMake options (`CMakeLists.txt`): `ODR_TEST`, `ODR_CLI`,
`ODR_WITH_PDF2HTMLEX`, `ODR_WITH_WVWARE`, `ODR_WITH_LIBMAGIC`, `ODR_CLANG_TIDY`.
A new `.cpp` must be added to the `ODR_SOURCE_FILES` list in `CMakeLists.txt`.

**Test data lives in git submodules** under `test/data/input/odr-public`,
`…/odr-private`, and `test/data/reference-output/*`.
- **Default to `cmake-build-relwithdebinfo`** (not `…-debug`).
- **Run only a targeted `--gtest_filter`**; the full suite is slow β€” run it only
when really necessary.
- **Run the test binary from the build dir** so output stays out of the repo tree.
- **For debugging, prefer the `translate` CLI** on a single file over the suite.
- CMake options (`CMakeLists.txt`): `ODR_TEST`, `ODR_CLI`, `ODR_WITH_PDF2HTMLEX`,
`ODR_WITH_WVWARE`, `ODR_WITH_LIBMAGIC`, `ODR_CLANG_TIDY`. A new `.cpp` must be
added to `ODR_SOURCE_FILES`.
- **Test data lives in git submodules** under `test/data/`.

## Conventions

- **Formatting**: clang-format, LLVM-based (`.clang-format`); run `scripts/format`
(or rely on the git hook from `scripts/setup`). `clang-tidy` config in
`.clang-tidy`; CI enforces both (`.github/workflows/format.yml`, `tidy.yml`).
- **Error handling β€” fail fast**: where the spec/format dictates what to expect,
**throw** on unexpected input (`std::runtime_error`, or the typed exceptions in
`src/odr/exceptions.hpp`) rather than silently degrading. Only **pass through**
(return empty / skip) values that are genuinely *optional* or *not yet
modelled*.
- **Public API**: value semantics; immutable handles; iterators only for
immutable traversal (`docs/design/README.md`).
- **Byte parsing**: read POD structs via `util::byte_stream::read`; this assumes
host byte order matches the file's (little-endian) β€” big-endian is a known
not-yet-handled gap in the binary engines.
- Match the **surrounding file's** style, includes, and idioms; mirror a sibling
engine when adding a format (the `oldms/text` `.doc` impl is the reference the
`.ppt` impl was modelled on).
- **Comments β€” keep them minimal**: a function/struct doc comment is at most a
couple of terse lines stating the key point (what it does, stream/ownership
preconditions, the spec section it implements, e.g. `[MS-PPT] 2.3.2`). Don't
restate the code or spell out every case; cite the spec instead of paraphrasing
it. The detailed design rationale belongs in the per-module `AGENTS.md`, not in
source comments.
- **Doc-comment markers**: document functions, classes, structs and enums with
`///` doc comments; use a trailing `///<` for the short note on the same line
(e.g. an enumerator or member). Keep them terse per the rule above.
- **Fixed-width integer types**: prefer `<cstdint>` types (`std::uint8_t`,
`std::uint32_t`, …) over `unsigned char` / `unsigned` / `long` and friends
whenever the width matters (byte buffers, on-disk/wire fields, bit math).
- **Formatting**: clang-format (LLVM-based, `.clang-format`); run `scripts/format`
or use the `scripts/setup` git hook. `clang-tidy` per `.clang-tidy`. CI enforces
both.
- **Fail fast**: where the spec dictates what to expect, **throw** on unexpected
input (`std::runtime_error` or the typed exceptions in `src/odr/exceptions.hpp`)
rather than silently degrading. Only pass through (return empty / skip) values
that are genuinely *optional* or *not yet modelled*.
- **Fixed-width integer types β€” always**: prefer `<cstdint>` types (`std::int32_t`,
`std::uint8_t`, …) over `int` / `unsigned` / `long` / `unsigned char`. Reserve
the built-in types for genuinely index/size-like values (`std::size_t`) or where
an API forces them.
- **Prefer ranges**: use `std::ranges` algorithms and range-based overloads over
iterator pairs (`std::ranges::find_if(v, …)` not `std::find_if(v.begin(), …)`),
and prefer range views/`for (auto &x : range)` over manual iterator loops. Fall
back to iterator pairs only where a range overload genuinely doesn't fit
(e.g. reverse-iterator `.base()` tricks).
- **Bind free-function definitions to a namespace**: in a source file, define a
header-declared free function (or util struct's static method) with its
**qualified** name β€” the `Ret ns::fn(...)` / `Ret Struct::fn(...)` form, inside
the reopened `namespace` β€” never as a bare unqualified redeclaration. A qualified
out-of-line definition must match an existing declaration, so a signature that
drifts from the header fails at **compile** time instead of becoming an obscure
linker error. (The `util` helpers use the `struct string { static … }` idiom for
exactly this.) Keep translation-unit-local helpers in an **anonymous namespace**.
- **Public API**: value semantics; immutable handles; iterators only for immutable
traversal (`docs/design/README.md`).
- **Byte parsing**: read POD structs via `util::byte_stream::read`; assumes host
byte order matches the file's (little-endian) β€” big-endian is a known gap.
- **Match the surrounding file** and mirror a sibling engine when adding a format
(the `oldms/text` `.doc` impl is the reference the `.ppt` impl was modelled on).
- **Comments β€” minimal**: a doc comment is at most a couple of terse lines stating
the key point (what it does, stream/ownership preconditions, spec section, e.g.
`[MS-PPT] 2.3.2`). Don't restate the code; cite the spec, don't paraphrase it.
Detailed rationale belongs in the per-module `AGENTS.md`.
- **Doc-comment markers**: `///` for functions/classes/structs/enums; trailing
`///<` for the short note on the same line (enumerator/member). Keep terse.
- **Pull requests**: put the `πŸ€– Generated with [Claude Code](https://claude.com/claude-code)`
line **at the top** of the PR body.

## Adding / extending a document format

1. Detection: extend `magic`/`open_strategy` to map the bytes to a `FileType`
1. Detection: extend `magic`/`open_strategy` to map bytes β†’ `FileType`
(+ `DecoderEngine`) and construct your `DecodedFile`.
2. For documents: subclass `internal::Document`; in its constructor build an
`ElementRegistry` and an `ElementAdapter` (see the pattern above).
`ElementRegistry` and an `ElementAdapter` (pattern above).
3. Implement the per-element adapters you can populate; the **generic HTML
renderer then works for free**.
4. Register the format's factory (e.g. `oldms_file.cpp::document()` switches on
`file_type()`), add sources to `CMakeLists.txt`, and add a GoogleTest.
4. Register the factory (e.g. `oldms_file.cpp::document()` switches on
`file_type()`), add sources to `CMakeLists.txt`, add a GoogleTest.

## Legacy Microsoft binary formats (`oldms`)

Container handling (CFB) already exists; each format is a small module under
`oldms/` mirroring `oldms/text` (`.doc`). Spec references in
`src/odr/internal/oldms/README.md`.

- **`.doc`** (`oldms/text`): working, visible-text extraction.
- **`.ppt`** (`oldms/presentation`): implemented β€” slides resolved via the
persist directory (the only spec-defined read path), each slide's text boxes
modelled as positioned `frame`s. **Read its docs before touching it**:
[`oldms/presentation/AGENTS.md`](src/odr/internal/oldms/presentation/AGENTS.md)
β€” what's implemented and **why** (persist-directory resolution, no scan
fallback, sequential `ChildCursor` reading without `tellg`, fail-fast error
handling, the two-text-locations finding, endianness), the open work (frame
refinements, smaller shortcomings), and the verified `[MS-PPT]`/`[MS-ODRAW]`
drawing-tree map.
- **`.xls`** (`oldms/spreadsheet`): working, visible cell-text extraction
(BIFF8). See [`oldms/spreadsheet/AGENTS.md`](src/odr/internal/oldms/spreadsheet/AGENTS.md).
CFB container handling exists; each format is a small module under `oldms/`
mirroring `oldms/text` (`.doc`), all doing visible-text extraction only. Shared
conventions + the endianness analysis are in
[`oldms/AGENTS.md`](src/odr/internal/oldms/AGENTS.md); spec refs in
`src/odr/internal/oldms/README.md`. **Read the module's own `AGENTS.md` before
touching it** β€” each carries the design rationale, spec-record maps, and open work:
`text/` (`.doc`), `presentation/` (`.ppt`, BIFF-style drawing tree),
`spreadsheet/` (`.xls`, BIFF8).

## Specs

Vendored Microsoft Open Specifications live under
`offline/documentation/<NAME>/<NAME>-<date>/`, both as `original.pdf` and an
extracted `docling-from-docx.md` (grep-friendly). Available: **MS-PPT**,
**MS-ODRAW** (Office Art / Escher drawing records), **MS-DOC**, **MS-XLS**,
**MS-CFB** (container), **MS-OFFCRYPTO** (encryption). Cite section numbers from
these when implementing binary parsing.
Vendored Microsoft Open Specifications under
`offline/documentation/<NAME>/<NAME>-<date>/` as `original.pdf` + an extracted
`docling-from-docx.md` (grep-friendly). Available: **MS-PPT**, **MS-ODRAW**,
**MS-DOC**, **MS-XLS**, **MS-CFB**, **MS-OFFCRYPTO**. Cite section numbers when
implementing binary parsing.
10 changes: 10 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# CLAUDE.md

**Read [`AGENTS.md`](AGENTS.md) first, before doing anything else in this repo.**
It is the single source of truth for the architecture, directory layout, build &
test loop, and the coding conventions you are required to follow here. Do not
propose changes or write code until you have read it.

Per-module `AGENTS.md` files (e.g. `src/odr/internal/oldms/presentation/AGENTS.md`)
carry deeper, area-specific rules β€” read the relevant one before touching that
module, as `AGENTS.md` instructs.
Loading
Loading