From a73f1b0479ba635eff3efc893dda97573b968a32 Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Thu, 16 Jul 2026 19:33:01 +0200 Subject: [PATCH 1/7] feat(doc): per-run character formatting from CHPX Implement Direct Character Formatting ([MS-DOC] 2.4.6.2): read the PlcBteChpx and its ChpxFkp pages, split the decoded piece text at every formatting-run boundary, and emit styled span elements carrying bold, italic, underline, strike, font size, font name (SttbfFfn), font color (sprmCCv/sprmCIco) and highlight (sprmCHighlight). Replaces the flat 11pt placeholder; unformatted text now defaults to 10pt (the sprmCHps default). Paragraphs store the style of their first run so empty paragraphs keep their height. Equal Chpx byte sequences share one resolved style and adjacent equal-style runs merge. Field-code hiding is now stateful across runs and paragraphs. Pcd.Prm modifications and STSH style layering remain open (documented in AGENTS.md). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R9Z3kyUaHNkJacAEggevHH --- src/odr/internal/oldms/text/AGENTS.md | 152 ++++----- src/odr/internal/oldms/text/doc_document.cpp | 26 +- .../oldms/text/doc_element_registry.cpp | 23 ++ .../oldms/text/doc_element_registry.hpp | 14 + src/odr/internal/oldms/text/doc_helper.cpp | 310 ++++++++++++++++++ src/odr/internal/oldms/text/doc_helper.hpp | 56 ++++ src/odr/internal/oldms/text/doc_parser.cpp | 279 ++++++++++------ src/odr/internal/oldms/text/doc_structs.hpp | 49 +++ test/src/internal/oldms/doc_test.cpp | 275 ++++++++++++++++ 9 files changed, 990 insertions(+), 194 deletions(-) diff --git a/src/odr/internal/oldms/text/AGENTS.md b/src/odr/internal/oldms/text/AGENTS.md index ab513d25..7cf20fcc 100644 --- a/src/odr/internal/oldms/text/AGENTS.md +++ b/src/odr/internal/oldms/text/AGENTS.md @@ -4,24 +4,26 @@ The **why** and the roadmap; what is concretely done lives in the code. Shared `oldms/` conventions are in [`../AGENTS.md`](../AGENTS.md). **Scope.** Extract the **visible text of the main document body**, split into -paragraphs and manual line breaks, through the abstract model so the generic HTML -renderer lays it out as a flat run of paragraphs. No character/paragraph styles, -headers/footers/notes/annotations, tables, frames, images, or fields beyond their -result text. +paragraphs and manual line breaks, plus **direct character formatting** (font, +size, bold, italic, underline, strike, color, highlight) as styled spans, +through the abstract model. No paragraph styles or style-sheet (STSH) +inheritance, headers/footers/notes/annotations, tables, frames, images, or +fields beyond their result text. -**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding) + `[MS-CFB]` -container. The implemented read path matches *Retrieving Text* (§2.4.1 steps 1–6); -section numbers are cited inline in code and in the read-path map below. +**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding, CHPX) + +`[MS-CFB]` container. The implemented read path matches *Retrieving Text* +(§2.4.1 steps 1–6) and *Direct Character Formatting* (§2.4.6.2); section +numbers are cited inline in code and in the read-path maps below. ## Module layout (sibling of `../presentation`) | File (`oldms/text/`) | Role | |---|---| -| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`) + `static_assert` sizes + `PlcPcdMap` + `ParsedFib` | +| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`, `PnFkpChpx`, `FfnFixed`) + `static_assert` sizes + `PlcPcdMap`/`PlcBteChpxMap` + `ParsedFib` + the character SPRM opcodes | | `doc_io.{hpp,cpp}` | `read(...)` over `std::istream`: variable-length FIB, Clx walk, string decoding | -| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index` | -| `doc_parser.{hpp,cpp}` | `parse_tree` → body text + tree; `clean_text` (field & control-char handling) | -| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload | +| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index`; `CharacterStyles` (fc-keyed style runs) + `read_character_styles` (PlcBteChpx/ChpxFkp walk), `apply_character_sprms`, `read_font_names` (SttbfFfn) | +| `doc_parser.{hpp,cpp}` | `parse_tree` → styled runs + tree; `TextCleaner` (field & control-char handling) | +| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload + per-element `TextStyle` map + font-name intern store | | `doc_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | ## Design decisions @@ -59,23 +61,47 @@ main body, `Prc` formatting runs, and every control/field char `clean_text` drop **Endianness.** Host byte order / LSB-first bit-fields assumed; shared `oldms/` assumption + fix plan in [`../AGENTS.md`](../AGENTS.md). -**The `font_size = 11pt` in the adapters is a placeholder hack** so empty -paragraphs still have height (same as the `.ppt`/`.xls` modules); removed when -character formatting lands (open work §1). `Document::is_editable()` → `true` and -`is_savable(!encrypted)`, but `save`/`text_set_content` throw — read-only in -practice. - -**`clean_text` control-character handling** (the non-obvious cases): `0x0D`/`0x0C`/ -`0x0B` are consumed by the caller's paragraph/page/line splits and never reach -`clean_text`; `0x13`/`0x14`/`0x15` delimit a **field** — instruction hidden, -result shown, the `0x14` separator optional (§2.8.25), nesting tracked with a -per-field stack; `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; `0x1F` -optional hyphen dropped; every other control char < `0x20` dropped. +**Direct character formatting only, resolved to styled spans.** The tree is +`root → paragraph → span → text`; each span (and each paragraph, for +empty-paragraph height) stores a resolved `TextStyle` in a registry side-map. +The §2.4.6.2 walk: `PlcBteChpx` (table stream) → 512-byte `ChpxFkp` pages +(WordDocument stream) → per-run `Chpx.grpprl`, applied over a default of 10pt +(the `sprmCHps` default of 20 half-points; the old flat `11pt` placeholder is +gone). The non-obvious bits: +- **Runs are keyed by stream offset (fc), not CP**, so the piece decode walks + each piece in style-uniform chunks (`CharacterStyles::chunk_end`); a + boundary inside a 2-byte CP is pushed past it. +- **Equal `Chpx` bytes share one resolved style**, and adjacent equal-style + runs merge, so span count stays low. `rgb[j] == 0` means default properties. +- **`ToggleOperand`** (§2.9.327) values `0x80`/`0x81` refer to the (unmodelled) + style value, which defaults to off → `0x80` = off, `0x81` = on. +- **Colors**: `sprmCCv` is a `COLORREF` (`fAuto` → unset); the legacy + `sprmCIco`/`sprmCHighlight` use the `Ico` palette (§2.9.119) — the spec's + extracted table repeats `0x0C` for `0x0D`, which is dark red (`0x800000`). +- **`Pcd.Prm` modifications and STSH styles are not applied** (open work §1). +- Font names from `SttbfFfn` are interned in a registry `std::deque` so + `TextStyle::font_name` (`const char *`) stays valid. + +`Document::is_editable()` → `false`; `save`/`text_set_content` throw. + +**`TextCleaner` control-character handling** (the non-obvious cases): `0x0D`/ +`0x0C`/`0x0B` are consumed by the caller's paragraph/page/line splits and never +reach the cleaner; `0x13`/`0x14`/`0x15` delimit a **field** — instruction +hidden, result shown, the `0x14` separator optional (§2.8.25), nesting tracked +with a per-field stack that now **persists across style runs and paragraphs** +(a field can span both); `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; +`0x1F` optional hyphen dropped; every other control char < `0x20` dropped. ## Tests - `OldMs.doc_read_string_compressed` — the compressed decoder against the §2.9.73 byte map (ASCII passthrough, `0x82–0x9F` remap, `0xA0–0xFF` round-trip). +- `OldMs.doc_apply_character_sprms` — every modelled SPRM, toggle semantics, + cvAuto reset, unknown fixed/variable SPRM skipping, malformed-grpprl throws. +- `OldMs.doc_read_font_names` — synthetic SttbfFfn. +- `OldMs.doc_character_formatting` — end-to-end over a synthetic `.doc` + (inline bytes: FIB + ChpxFkp + piece table): span splitting at run + boundaries, default 10pt, bold + font-name run, paragraph style. **Not yet tested:** FIB robustness (negative `ccpText`, newer-than-2007 fallback), `0x0C` page-break emission, and there is **no assertion-based render test** over a @@ -122,67 +148,21 @@ Retrieving Text: §2.4.1 (steps 1–6) Field chars 0x13/0x14/0x15: §2.8.25 # Open work -## 1. Character (font) formatting → the IR (the next feature) - -**Goal.** Extract per-run character properties (font name, size, bold, italic, -underline, strike, colour, highlight) and surface them through `TextStyle`, so the -renderer styles text instead of emitting one flat 11pt run. Replaces the -`font_size = 11pt` placeholder. - -`TextStyle` (`src/odr/style.hpp`) maps almost 1:1 onto the `.doc` character SPRMs: - -| `TextStyle` field | SPRM (opcode) | operand → value | -|---|---|---| -| `font_size` | `sprmCHps` (0x4A43) | u16 **half-points** → `Measure(hps/2, pt)` (default 20 = 10pt) | -| `font_weight` | `sprmCFBold` (0x0835) | `ToggleOperand` → `bold` | -| `font_style` | `sprmCFItalic` (0x0836) | `ToggleOperand` → `italic` | -| `font_underline` | `sprmCKul` (0x2A3E) | `Kul`, `0x00` = none → `bool` | -| `font_line_through` | `sprmCFStrike` (0x0837) | `ToggleOperand` → `bool` | -| `font_color` | `sprmCCv` (0x6870) | `COLORREF` → `Color` (legacy `sprmCIco` 0x2A42 is a palette index) | -| `background_color` | `sprmCHighlight` (0x2A0C) | `Ico` highlight index → `Color` | -| `font_name` | `sprmCRgFtc0` (0x4A4F) | s16 index into `SttbfFfn` → font name | - -`font_name` is a `const char *`, so the resolved name needs stable storage — intern -it in the `ElementRegistry` (e.g. a `std::deque` whose elements never -move) and hand out the pointer. - -**How `[MS-DOC]` retrieves character properties** — Direct Character Formatting -(§2.4.6.2) reuses the *Retrieving Text* walk we already have: -1. For `cp`, run §2.4.1 to get its byte offset `fc` and owning `Pcd` (already - computed). -2. Read **`PlcBteChpx`** (§2.8.5) at `fcPlcfBteChpx` in the table stream — a PLC - keyed by stream offset: `aFC[n+1]` + `aPnBteChpx[n]` (`PnFkpChpx`, 4 B). -3. Largest `i` with `aFC[i] ≤ fc` → read a **`ChpxFkp`** (§2.9.33) at - `aPnBteChpx[i].pn * 512` (fixed 512-byte page: `rgfc` boundaries, parallel - `rgb`, `crun` in the last byte). -4. Largest `j` with `rgfc[j] ≤ fc` → the `Chpx` (§2.9.32) at `rgb[j] * 2` within - the page. `Chpx.grpprl` is an array of `Prl` = `Sprm` (2 B) + operand. -5. Append `Pcd.Prm` mods (§2.9.214–216): `Prm0` (inline) or `Prm1` (index). - -`Prl`/`Sprm` is already modelled (`Sprm` with `ispmd/fSpec/sgc/spra` + -`operand_size()`); a **character** property is a SPRM with `sgc == 2` (note -`spra == 6` is length-prefixed/variable). - -**First cut — direct formatting only.** Implement §2.4.6.2 (`Chpx.grpprl` + -`Pcd.Prm`) and map the SPRMs above (bold/italic/size/font/colour applied directly -to runs). Resolve `sprmCRgFtc0` by reading **`SttbfFfn`** (§2.9.286) once and -indexing it. Drop the 11pt; use 10pt (the `sprmCHps` default). - -**Full fidelity — styles (later).** *Determining Formatting Properties* (§2.4.6.6) -layers: document defaults → `STSH` (§2.4.6.5) para/char-style `grpprl`s via the -paragraph's `istd` → table-style → direct paragraph → direct character. The first -cut skips STSH (style-dependent props fall back to defaults). - -**Wiring to the abstract model.** Per-run styling needs run boundaries in -`/WordDocument` byte offsets, so: -1. **Keep the FC↔text mapping** while concatenating (thread the `CharacterIndex` - through instead of discarding it after `body_text`). -2. **Split paragraphs into runs** at every `ChpxFkp` boundary, resolve each run's - `TextStyle`, emit a **`span`** (already wired via `SpanAdapter`) per run. -3. **Store the style** in a `TextStyle` side-map keyed by span id (mirror the text - side-payload + the `presentation` frame-payload) plus the font-name intern - store. `SpanAdapter::span_style` returns it; `text_style`/ - `paragraph_text_style` then return `{}` instead of the 11pt hack. +## 1. Character formatting fidelity + +Direct formatting (§2.4.6.2) is implemented — see the design notes above and +the read-path map below. Remaining layers of *Determining Formatting +Properties* (§2.4.6.6): + +- **`Pcd.Prm` modifications** (§2.9.214–216): per-piece property diffs — + `Prm0` (one inline SPRM) or `Prm1` (index into the Clx's `Prc` array, which + `read_character_index` currently skips). Rare for character properties, but + incremental saves can carry them. +- **STSH style layering** (§2.4.6.5): document defaults → paragraph/character + style `grpprl`s via the paragraph's `istd` → direct formatting. Without it, + style-derived properties (e.g. heading fonts defined only in the style + sheet, the usual Times New Roman default font) fall back to defaults; + `ToggleOperand` `0x80`/`0x81` resolve against "off". Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: @@ -190,12 +170,12 @@ Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: Table stream ├─ PlcBteChpx @ fcPlcfBteChpx §2.8.5 aFC[n+1] + aPnBteChpx[n] (PnFkpChpx, 4 B) ├─ SttbfFfn @ fcSttbfFfn (font names, FFN.xszFfn) §2.9.286 -└─ STSH @ fcStshf (styles — full fidelity only) §2.4.6.5 +└─ STSH @ fcStshf (styles — not read) §2.4.6.5 WordDocument stream └─ ChpxFkp @ aPnBteChpx[i].pn * 512 (512-byte page) §2.9.33 ├─ rgfc[crun+1] boundaries (stream offsets), rgb[crun] → Chpx @ rgb[j]*2, crun (last byte) - └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 + └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 (not read) ``` ## 2. Coverage gaps diff --git a/src/odr/internal/oldms/text/doc_document.cpp b/src/odr/internal/oldms/text/doc_document.cpp index 5aec89d2..b9719ddd 100644 --- a/src/odr/internal/oldms/text/doc_document.cpp +++ b/src/odr/internal/oldms/text/doc_document.cpp @@ -157,19 +157,12 @@ class ElementAdapter final : public abstract::ElementAdapter, } [[nodiscard]] TextStyle paragraph_text_style(const ElementIdentifier element_id) const override { - (void)element_id; - // TODO setting font size otherwise the text will be invisible. upstream - // this is used to make empty paragraphs works correctly - TextStyle style{}; - // TODO - style.font_size = Measure("11pt"); - return style; + return stored_style(element_id); } [[nodiscard]] TextStyle span_style(const ElementIdentifier element_id) const override { - (void)element_id; - return {}; // TODO + return stored_style(element_id); } [[nodiscard]] std::string @@ -184,13 +177,9 @@ class ElementAdapter final : public abstract::ElementAdapter, } [[nodiscard]] TextStyle text_style(const ElementIdentifier element_id) const override { + // The enclosing span carries the character style. (void)element_id; - // TODO setting font size otherwise the text will be invisible. upstream - // this is used to make empty paragraphs works correctly - TextStyle style{}; - // TODO - style.font_size = Measure("11pt"); - return style; + return {}; } private: @@ -198,6 +187,13 @@ class ElementAdapter final : public abstract::ElementAdapter, [[maybe_unused]] const Document *m_document{nullptr}; ElementRegistry *m_registry{nullptr}; + + /// The character style stored for a paragraph or span element. + [[nodiscard]] TextStyle + stored_style(const ElementIdentifier element_id) const { + const TextStyle *style = m_registry->element_style(element_id); + return style != nullptr ? *style : TextStyle{}; + } }; std::unique_ptr diff --git a/src/odr/internal/oldms/text/doc_element_registry.cpp b/src/odr/internal/oldms/text/doc_element_registry.cpp index 772fd677..f8ce1336 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.cpp +++ b/src/odr/internal/oldms/text/doc_element_registry.cpp @@ -1,5 +1,6 @@ #include +#include #include namespace odr::internal::oldms::text { @@ -7,6 +8,8 @@ namespace odr::internal::oldms::text { void ElementRegistry::clear() noexcept { m_elements.clear(); m_texts.clear(); + m_styles.clear(); + m_font_names.clear(); } [[nodiscard]] std::size_t ElementRegistry::size() const noexcept { @@ -76,6 +79,26 @@ void ElementRegistry::append_child(const ElementIdentifier parent_id, element_at(parent_id).last_child_id = child_id; } +const char *ElementRegistry::intern_font_name(const std::string &name) { + if (const auto it = std::ranges::find(m_font_names, name); + it != m_font_names.end()) { + return it->c_str(); + } + return m_font_names.emplace_back(name).c_str(); +} + +void ElementRegistry::set_element_style(const ElementIdentifier id, + TextStyle style) { + check_element_id(id); + m_styles[id] = std::move(style); +} + +const TextStyle * +ElementRegistry::element_style(const ElementIdentifier id) const { + const auto it = m_styles.find(id); + return it != m_styles.end() ? &it->second : nullptr; +} + void ElementRegistry::check_element_id(const ElementIdentifier id) const { if (id == null_element_id) { throw std::out_of_range( diff --git a/src/odr/internal/oldms/text/doc_element_registry.hpp b/src/odr/internal/oldms/text/doc_element_registry.hpp index 41bcaea4..48fbe8d1 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.hpp +++ b/src/odr/internal/oldms/text/doc_element_registry.hpp @@ -2,8 +2,11 @@ #include #include +#include +#include #include +#include #include #include @@ -39,9 +42,20 @@ class ElementRegistry final { void append_child(ElementIdentifier parent_id, ElementIdentifier child_id); + /// Stores a font name and returns a pointer that stays valid for the + /// registry's lifetime (`TextStyle::font_name` is a `const char *`). + const char *intern_font_name(const std::string &name); + + /// Character style of a span or paragraph element. + void set_element_style(ElementIdentifier id, TextStyle style); + [[nodiscard]] const TextStyle *element_style(ElementIdentifier id) const; + private: std::vector m_elements; std::unordered_map m_texts; + std::unordered_map m_styles; + /// Deque: elements never move, so interned `c_str()`s stay valid. + std::deque m_font_names; void check_element_id(ElementIdentifier id) const; void check_text_id(ElementIdentifier id) const; diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 0cd437b1..468d56ad 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -3,8 +3,59 @@ #include #include #include +#include +#include +#include +#include #include +#include +#include + +namespace odr::internal::oldms::text { + +namespace { + +/// The Ico palette ([MS-DOC] 2.9.119); index 0 is cvAuto (no explicit color). +constexpr std::array ico_colors = { + 0x000000, 0x000000, 0x0000FF, 0x00FFFF, 0x00FF00, 0xFF00FF, + 0xFF0000, 0xFFFF00, 0xFFFFFF, 0x000080, 0x008080, 0x008000, + 0x800080, 0x800000, 0x808000, 0x808080, 0xC0C0C0}; + +std::optional ico_color(const std::uint8_t ico) { + if (ico >= ico_colors.size()) { + throw std::runtime_error("doc: Ico value out of range"); + } + if (ico == 0) { + return std::nullopt; // cvAuto + } + return Color(ico_colors[ico]); +} + +/// ToggleOperand ([MS-DOC] 2.9.327) against the (unmodelled) style value: +/// 0x80 matches the style, 0x81 inverts it; styles default to off. +bool toggle_on(const std::uint8_t value) { + switch (value) { + case 0x00: + case 0x80: + return false; + case 0x01: + case 0x81: + return true; + default: + throw std::runtime_error("doc: unexpected ToggleOperand value"); + } +} + +std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { + std::uint16_t value; + std::memcpy(&value, bytes.data() + at, sizeof(value)); + return value; +} + +} // namespace + +} // namespace odr::internal::oldms::text namespace odr::internal::oldms { @@ -32,4 +83,263 @@ text::CharacterIndex text::read_character_index(std::istream &in) { return result; } +text::CharacterStyles::CharacterStyles(TextStyle default_style) { + m_styles.push_back(std::move(default_style)); +} + +std::uint32_t text::CharacterStyles::add_style(TextStyle style) { + m_styles.push_back(std::move(style)); + return static_cast(m_styles.size() - 1); +} + +void text::CharacterStyles::append_run(const std::uint32_t begin_fc, + const std::uint32_t end_fc, + const std::uint32_t style_index) { + if (begin_fc >= end_fc || + (!m_runs.empty() && begin_fc < m_runs.back().end_fc)) { + throw std::runtime_error("doc: character runs must be ascending"); + } + if (style_index >= m_styles.size()) { + throw std::out_of_range("doc: style index out of range"); + } + if (!m_runs.empty() && m_runs.back().end_fc == begin_fc && + m_runs.back().style_index == style_index) { + m_runs.back().end_fc = end_fc; + return; + } + m_runs.push_back({begin_fc, end_fc, style_index}); +} + +std::uint32_t text::CharacterStyles::index_at(const std::uint32_t fc) const { + const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); + if (it == m_runs.begin()) { + return 0; + } + const Run &run = *std::prev(it); + return fc < run.end_fc ? run.style_index : 0; +} + +std::uint32_t text::CharacterStyles::chunk_end(const std::uint32_t fc) const { + const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); + if (it != m_runs.begin() && fc < std::prev(it)->end_fc) { + return std::prev(it)->end_fc; + } + if (it != m_runs.end()) { + return it->begin_fc; + } + return std::numeric_limits::max(); +} + +const TextStyle &text::CharacterStyles::style(const std::uint32_t index) const { + return m_styles.at(index); +} + +TextStyle +text::apply_character_sprms(TextStyle style, const std::string_view grpprl, + const std::vector &font_names) { + std::size_t at = 0; + while (at + sizeof(Sprm) <= grpprl.size()) { + Sprm sprm; + std::memcpy(&sprm, grpprl.data() + at, sizeof(sprm)); + const std::uint16_t opcode = std::bit_cast(sprm); + at += sizeof(sprm); + + std::size_t operand_size; + if (const int fixed_size = sprm.operand_size(); fixed_size >= 0) { + operand_size = static_cast(fixed_size); + } else { + // spra == 6: the first operand byte is the size of the remainder. The + // two SPRMs encoded differently (sprmTDefTable, sprmPChgTabs with + // cb == 0xFF) are not character properties and must not appear here. + if (at >= grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + operand_size = static_cast(grpprl[at]); + ++at; + if (opcode == 0xD608 || (opcode == 0xC615 && operand_size == 0xFF)) { + throw std::runtime_error("doc: unexpected table SPRM in a Chpx"); + } + } + if (at + operand_size > grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + const std::string_view operand = grpprl.substr(at, operand_size); + at += operand_size; + + switch (opcode) { + case sprmCFBold: + style.font_weight = toggle_on(static_cast(operand[0])) + ? FontWeight::bold + : FontWeight::normal; + break; + case sprmCFItalic: + style.font_style = toggle_on(static_cast(operand[0])) + ? FontStyle::italic + : FontStyle::normal; + break; + case sprmCFStrike: + style.font_line_through = + toggle_on(static_cast(operand[0])); + break; + case sprmCHighlight: + style.background_color = ico_color(static_cast(operand[0])); + break; + case sprmCKul: + style.font_underline = operand[0] != 0; + break; + case sprmCIco: + style.font_color = ico_color(static_cast(operand[0])); + break; + case sprmCHps: { + const std::uint16_t half_points = read_u16(operand, 0); + if (half_points < 2 || half_points > 3276) { + throw std::runtime_error("doc: sprmCHps value out of range"); + } + style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); + } break; + case sprmCRgFtc0: { + const auto ftc = static_cast(read_u16(operand, 0)); + if (ftc < 0 || static_cast(ftc) >= font_names.size()) { + throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); + } + style.font_name = font_names[static_cast(ftc)]; + } break; + case sprmCCv: { + // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. + if (static_cast(operand[3]) == 0xFF) { + style.font_color = std::nullopt; // cvAuto + } else { + style.font_color = Color(static_cast(operand[0]), + static_cast(operand[1]), + static_cast(operand[2])); + } + } break; + default: + break; // not modelled + } + } + if (at != grpprl.size()) { + throw std::runtime_error("doc: grpprl does not hold whole Prls"); + } + + return style; +} + +std::vector text::read_font_names(std::istream &table_stream, + const FcLcb sttbf_ffn) { + std::vector result; + if (sttbf_ffn.lcb == 0) { + return result; + } + table_stream.seekg(sttbf_ffn.fc); + + // SttbfFfn is a non-extended STTB ([MS-DOC] 2.2.4, 2.9.286): u16 cData, + // u16 cbExtra (0), then per entry a u8 byte count and an FFN. + const auto c_data = util::byte_stream::read(table_stream); + if (c_data == 0xFFFF) { + throw std::runtime_error("doc: unexpected extended SttbfFfn"); + } + const auto cb_extra = util::byte_stream::read(table_stream); + + result.reserve(c_data); + for (std::uint16_t i = 0; i < c_data; ++i) { + const auto cch_data = util::byte_stream::read(table_stream); + if (cch_data < sizeof(FfnFixed)) { + throw std::runtime_error("doc: FFN too short"); + } + const auto ffn = util::byte_stream::read(table_stream); + (void)ffn; + + // xszFfn: null-terminated UTF-16 within the remaining bytes; an + // alternative name (xszAlt) may follow the terminator. + const std::size_t name_units = (cch_data - sizeof(FfnFixed)) / 2; + std::u16string name = read_string_uncompressed(table_stream, name_units); + if ((cch_data - sizeof(FfnFixed)) % 2 != 0) { + table_stream.ignore(1); + } + if (const std::size_t nul = name.find(u'\0'); nul != std::u16string::npos) { + name.resize(nul); + } + result.push_back(util::string::u16string_to_string(name)); + + table_stream.ignore(cb_extra); + } + + return result; +} + +text::CharacterStyles text::read_character_styles( + std::istream &document_stream, std::istream &table_stream, + const FcLcb plcf_bte_chpx, const TextStyle &default_style, + const std::vector &font_names) { + CharacterStyles result(default_style); + if (plcf_bte_chpx.lcb == 0) { + return result; + } + + table_stream.seekg(plcf_bte_chpx.fc); + std::string plc_bytes = util::stream::read(table_stream, plcf_bte_chpx.lcb); + const PlcBteChpxMap plc(plc_bytes.data(), plc_bytes.size()); + + // One resolved style per distinct Chpx byte sequence. + std::unordered_map style_cache; + const auto resolve = [&](const std::string_view grpprl) -> std::uint32_t { + if (grpprl.empty()) { + return 0; + } + const auto it = style_cache.find(std::string(grpprl)); + if (it != style_cache.end()) { + return it->second; + } + const std::uint32_t index = result.add_style( + apply_character_sprms(default_style, grpprl, font_names)); + style_cache.emplace(std::string(grpprl), index); + return index; + }; + + for (std::uint32_t i = 0; i < plc.n(); ++i) { + // A 512-byte ChpxFkp page ([MS-DOC] 2.9.33) at pn * 512: crun in the last + // byte, crun+1 rgfc offsets, crun rgb bytes; rgb[j] * 2 locates the Chpx + // within the page, 0 means default properties. + std::array page; + document_stream.seekg(static_cast(plc.aData(i).pn) * 512); + document_stream.read(page.data(), page.size()); + if (!document_stream) { + throw std::runtime_error("doc: truncated ChpxFkp"); + } + + const auto crun = static_cast(page[511]); + if (crun < 0x01 || crun > 0x65) { + throw std::runtime_error("doc: ChpxFkp crun out of range"); + } + + for (std::uint8_t j = 0; j < crun; ++j) { + std::uint32_t begin_fc; + std::uint32_t end_fc; + std::memcpy(&begin_fc, page.data() + static_cast(j) * 4, 4); + std::memcpy(&end_fc, page.data() + static_cast(j + 1) * 4, + 4); + + const auto rgb = + static_cast(page[(crun + 1) * 4 + std::size_t{j}]); + std::string_view grpprl; + if (rgb != 0) { + const std::size_t chpx_at = static_cast(rgb) * 2; + if (chpx_at >= page.size() - 1) { + throw std::runtime_error("doc: Chpx offset out of range"); + } + const auto cb = static_cast(page[chpx_at]); + if (chpx_at + 1 + cb > page.size() - 1) { + throw std::runtime_error("doc: Chpx overruns its ChpxFkp"); + } + grpprl = std::string_view(page.data() + chpx_at + 1, cb); + } + + result.append_run(begin_fc, end_fc, resolve(grpprl)); + } + } + + return result; +} + } // namespace odr::internal::oldms diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index 62e346cd..537d4941 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -1,9 +1,15 @@ #pragma once +#include + +#include + #include #include #include #include +#include +#include #include namespace odr::internal::oldms::text { @@ -106,4 +112,54 @@ class CharacterIndex { CharacterIndex read_character_index(std::istream &in); +/// The document's character-formatting runs, keyed by WordDocument-stream +/// offset (fc). Style index 0 is the default style; offsets outside any run +/// resolve to it. +class CharacterStyles final { +public: + explicit CharacterStyles(TextStyle default_style); + + std::uint32_t add_style(TextStyle style); + /// Appends a run; runs must be ascending and non-overlapping. Adjacent runs + /// with the same style index are merged. + void append_run(std::uint32_t begin_fc, std::uint32_t end_fc, + std::uint32_t style_index); + + /// The style index of the run containing `fc` (0 if none). + [[nodiscard]] std::uint32_t index_at(std::uint32_t fc) const; + /// End offset of the run containing `fc`, or the next run's begin, so text + /// can be walked in style-uniform chunks; UINT32_MAX past the last run. + [[nodiscard]] std::uint32_t chunk_end(std::uint32_t fc) const; + [[nodiscard]] const TextStyle &style(std::uint32_t index) const; + +private: + struct Run { + std::uint32_t begin_fc; + std::uint32_t end_fc; + std::uint32_t style_index; + }; + + std::vector m_styles; + std::vector m_runs; +}; + +/// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of +/// `style`; non-character SPRMs are skipped via their operand size. +/// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). +TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, + const std::vector &font_names); + +/// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at +/// `fc` in the table stream; empty for lcb == 0. +std::vector read_font_names(std::istream &table_stream, + FcLcb sttbf_ffn); + +/// Reads the PlcBteChpx at `fc` in the table stream and every referenced +/// ChpxFkp page ([MS-DOC] 2.9.33) from the WordDocument stream, resolving +/// each run's Chpx against `default_style`. Equal Chpx bytes share one style. +CharacterStyles +read_character_styles(std::istream &document_stream, std::istream &table_stream, + FcLcb plcf_bte_chpx, const TextStyle &default_style, + const std::vector &font_names); + } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_parser.cpp b/src/odr/internal/oldms/text/doc_parser.cpp index c2f106cd..25f150f2 100644 --- a/src/odr/internal/oldms/text/doc_parser.cpp +++ b/src/odr/internal/oldms/text/doc_parser.cpp @@ -8,10 +8,13 @@ #include #include #include -#include #include +#include +#include +#include #include +#include #include namespace odr::internal::oldms { @@ -29,63 +32,125 @@ constexpr char page_break_mark = '\x0C'; // visible text. A field is 0x13 begin ... [0x14 separator] ... 0x15 end // ([MS-DOC] 2.8.25): the instruction (begin..separator) is hidden, the result // (separator..end) shown. A separator-less field is hidden entirely. -std::string clean_text(const std::string &in) { - std::string out; - out.reserve(in.size()); +// Stateful: a field can span style runs and paragraphs, so one cleaner +// processes the whole body in order. +class TextCleaner { +public: + std::string clean(const std::string_view in) { + std::string out; + out.reserve(in.size()); - // Open fields; entry is true while still in that field's instruction part. - // instruction_depth counts those; text is hidden whenever it is > 0. - std::vector field_in_instruction; - int instruction_depth = 0; - - for (const char c : in) { - switch (c) { - case '\x13': // field begin: a new field opens in its instruction part - field_in_instruction.push_back(true); - ++instruction_depth; - continue; - case '\x14': // field separator: the innermost field's instruction ends - if (!field_in_instruction.empty() && field_in_instruction.back()) { - field_in_instruction.back() = false; - --instruction_depth; + for (const char c : in) { + switch (c) { + case '\x13': // field begin: a new field opens in its instruction part + m_field_in_instruction.push_back(true); + ++m_instruction_depth; + continue; + case '\x14': // field separator: the innermost field's instruction ends + if (!m_field_in_instruction.empty() && m_field_in_instruction.back()) { + m_field_in_instruction.back() = false; + --m_instruction_depth; + } + continue; + case '\x15': // field end: close the innermost field + if (!m_field_in_instruction.empty()) { + if (m_field_in_instruction.back()) { + // separator-less field: its instruction part ends here, not + // earlier + --m_instruction_depth; + } + m_field_in_instruction.pop_back(); + } + continue; + default: + break; } - continue; - case '\x15': // field end: close the innermost field - if (!field_in_instruction.empty()) { - if (field_in_instruction.back()) { - // separator-less field: its instruction part ends here, not earlier - --instruction_depth; + + if (m_instruction_depth > 0) { + continue; // drop the hidden field instruction + } + + switch (c) { + case '\x09': // tab: keep as real text + out.push_back(c); + break; + case '\x1E': // non-breaking hyphen + out.push_back('-'); + break; + case '\x1F': // optional hyphen: drop + break; + default: + // Drop remaining control/anchor characters (< 0x20); keep the rest. + if (static_cast(c) >= 0x20) { + out.push_back(c); } - field_in_instruction.pop_back(); + break; } - continue; - default: - break; } - if (instruction_depth > 0) { - continue; // drop the hidden field instruction - } + return out; + } - switch (c) { - case '\x09': // tab: keep as real text - out.push_back(c); - break; - case '\x1E': // non-breaking hyphen - out.push_back('-'); - break; - case '\x1F': // optional hyphen: drop +private: + // Open fields; entry is true while still in that field's instruction part. + // m_instruction_depth counts those; text is hidden whenever it is > 0. + std::vector m_field_in_instruction; + int m_instruction_depth{0}; +}; + +/// A maximal piece of body text with uniform character formatting. +struct StyledRun { + std::string text; + std::uint32_t style_index{0}; +}; + +/// Decodes the main-body pieces (the first `ccp_text` CPs), split further at +/// every character-formatting boundary. +std::vector decode_styled_runs( + std::istream &document_stream, const text::CharacterIndex &character_index, + const text::CharacterStyles &styles, const std::size_t ccp_text) { + std::vector runs; + std::size_t consumed_cp = 0; + + for (const auto &entry : character_index) { + if (consumed_cp >= ccp_text) { break; - default: - // Drop remaining control/anchor characters (< 0x20); keep the rest. - if (static_cast(c) >= 0x20) { - out.push_back(c); + } + const std::size_t take = std::min(entry.length_cp, ccp_text - consumed_cp); + const std::size_t bytes_per_cp = entry.is_compressed ? 1 : 2; + + std::size_t cp = 0; + while (cp < take) { + const auto fc = + static_cast(entry.data_offset + cp * bytes_per_cp); + const std::uint32_t style_index = styles.index_at(fc); + + // CPs of this piece still in the same style run; a boundary that falls + // inside a 2-byte CP is pushed past it. + const std::uint32_t chunk_end_fc = styles.chunk_end(fc); + std::size_t chunk_cp = take - cp; + if (chunk_end_fc != std::numeric_limits::max()) { + chunk_cp = std::min( + chunk_cp, + std::max(1, (chunk_end_fc - fc + bytes_per_cp - 1) / + bytes_per_cp)); } - break; + + document_stream.seekg(fc); + std::string chunk_text = + text::read_string(document_stream, chunk_cp, entry.is_compressed); + if (!runs.empty() && runs.back().style_index == style_index) { + runs.back().text += chunk_text; + } else { + runs.push_back({std::move(chunk_text), style_index}); + } + cp += chunk_cp; } + + consumed_cp += take; } - return out; + return runs; } } // namespace @@ -100,63 +165,91 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, const std::string tableStreamPath = fib.base.fWhichTblStm == 1 ? "/1Table" : "/0Table"; - const auto table_stream = files.open(AbsPath(tableStreamPath))->stream(); - table_stream->ignore(fib.fibRgFcLcb->clx.fc); + // Font table, interned so TextStyle::font_name pointers stay valid. + std::vector font_names; + for (const std::string &name : + read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)) { + font_names.push_back(registry.intern_font_name(name)); + } + + // Direct character formatting ([MS-DOC] 2.4.6.2); Pcd.Prm modifications are + // not modelled. Without sprmCHps the size defaults to 20 half-points. + TextStyle default_style; + default_style.font_size = Measure(10.0, DynamicUnit("pt")); + const CharacterStyles styles = read_character_styles( + *document_stream, *table_stream, fib.fibRgFcLcb->plcfBteChpx, + default_style, font_names); + + table_stream->seekg(fib.fibRgFcLcb->clx.fc); const CharacterIndex character_index = read_character_index(*table_stream); - // Concatenate the decoded pieces of the main document body only (the first - // ccpText CPs). Pieces are in ascending CP order, so clamp each piece to the - // remaining budget and stop once it is exhausted. - const std::size_t ccp_text = fib.ccpText(); - std::size_t consumed_cp = 0; - std::string body_text; - for (const auto &entry : character_index) { - if (consumed_cp >= ccp_text) { - break; - } - const std::size_t take = std::min(entry.length_cp, ccp_text - consumed_cp); - document_stream->seekg(entry.data_offset); - body_text += read_string(*document_stream, take, entry.is_compressed); - consumed_cp += take; - } + const auto ccp_text = static_cast(fib.ccpText()); + const std::vector runs = + decode_styled_runs(*document_stream, character_index, styles, ccp_text); - // Split into paragraphs. The main body always ends with a guard paragraph - // mark, so drop the resulting trailing empty paragraph. - auto paragraphs = - util::string::split(body_text, std::string(1, paragraph_mark)); - if (!paragraphs.empty() && paragraphs.back().empty()) { - paragraphs.pop_back(); - } + // Build the tree: paragraphs are opened lazily so the trailing guard + // paragraph mark does not produce an extra empty paragraph. Each paragraph + // and span stores its style; empty paragraphs keep their height through the + // paragraph style. + TextCleaner cleaner; + ElementIdentifier paragraph_id = null_element_id; - for (const auto ¶graph : paragraphs) { - // A paragraph may contain end-of-section / manual page-break characters - // (0x0C): split on them and mark each boundary with a page_break element. - const auto pages = - util::string::split(paragraph, std::string(1, page_break_mark)); - for (std::size_t page_i = 0; page_i < pages.size(); ++page_i) { - if (page_i > 0) { - auto [page_break_id, _] = - registry.create_element(ElementType::page_break); - registry.append_child(root_id, page_break_id); - } + const auto ensure_paragraph = [&](const std::uint32_t style_index) { + if (paragraph_id == null_element_id) { + auto [id, paragraph] = registry.create_element(ElementType::paragraph); + registry.append_child(root_id, id); + registry.set_element_style(id, styles.style(style_index)); + paragraph_id = id; + } + }; - auto [paragraph_id, _] = registry.create_element(ElementType::paragraph); - registry.append_child(root_id, paragraph_id); + for (const StyledRun &run : runs) { + std::size_t at = 0; + while (at <= run.text.size()) { + const std::size_t control = run.text.find_first_of("\r\x0B\x0C", at); + const std::size_t segment_end = + control == std::string::npos ? run.text.size() : control; - const auto lines = - util::string::split(pages[page_i], std::string(1, line_break_mark)); - for (std::uint32_t line_i = 0; line_i < lines.size(); ++line_i) { - if (line_i > 0) { - auto [line_id, _] = registry.create_element(ElementType::line_break); - registry.append_child(paragraph_id, line_id); - } + if (std::string cleaned = cleaner.clean( + std::string_view(run.text).substr(at, segment_end - at)); + !cleaned.empty()) { + ensure_paragraph(run.style_index); + auto [span_id, span] = registry.create_element(ElementType::span); + registry.set_element_style(span_id, styles.style(run.style_index)); + registry.append_child(paragraph_id, span_id); + + auto [text_id, text_element, text_entry] = + registry.create_text_element(); + text_entry.text = std::move(cleaned); + registry.append_child(span_id, text_id); + } - auto [text_id, _, text_element] = registry.create_text_element(); - text_element.text = clean_text(lines[line_i]); - registry.append_child(paragraph_id, text_id); + if (control == std::string::npos) { + break; + } + switch (run.text[control]) { + case paragraph_mark: + ensure_paragraph(run.style_index); + paragraph_id = null_element_id; + break; + case page_break_mark: { + ensure_paragraph(run.style_index); + paragraph_id = null_element_id; + auto [page_break_id, page_break] = + registry.create_element(ElementType::page_break); + registry.append_child(root_id, page_break_id); + } break; + case line_break_mark: { + ensure_paragraph(run.style_index); + auto [line_id, line] = registry.create_element(ElementType::line_break); + registry.append_child(paragraph_id, line_id); + } break; + default: + break; } + at = control + 1; } } diff --git a/src/odr/internal/oldms/text/doc_structs.hpp b/src/odr/internal/oldms/text/doc_structs.hpp index 1cb3b4a3..8731f822 100644 --- a/src/odr/internal/oldms/text/doc_structs.hpp +++ b/src/odr/internal/oldms/text/doc_structs.hpp @@ -288,6 +288,20 @@ struct FibRgCswNewData2007 : FibRgCswNewData2000 { static_assert(sizeof(FibRgCswNewData2007) == 8, "FibRgCswNewData2007 should be 8 bytes"); +/// The character SPRMs mapped to TextStyle ([MS-DOC] 2.6.1); everything else +/// in a Chpx is skipped via Sprm::operand_size. +enum CharacterSprms : std::uint16_t { + sprmCFBold = 0x0835, //< ToggleOperand + sprmCFItalic = 0x0836, //< ToggleOperand + sprmCFStrike = 0x0837, //< ToggleOperand + sprmCHighlight = 0x2A0C, //< Ico + sprmCKul = 0x2A3E, //< Kul; 0x00 = none + sprmCIco = 0x2A42, //< Ico (legacy palette color) + sprmCHps = 0x4A43, //< u16 half-points (default 20) + sprmCRgFtc0 = 0x4A4F, //< s16 index into SttbfFfn + sprmCCv = 0x6870, //< COLORREF +}; + struct Sprm { std::uint16_t ispmd : 9; std::uint16_t fSpec : 1; @@ -333,6 +347,26 @@ struct Pcd { }; static_assert(sizeof(Pcd) == 8, "Pcd should be 8 bytes"); +/// Location of a ChpxFkp in the WordDocument stream, at pn * 512 +/// ([MS-DOC] 2.9.206). +struct PnFkpChpx { + std::uint32_t pn : 22; + std::uint32_t unused : 10; +}; +static_assert(sizeof(PnFkpChpx) == 4, "PnFkpChpx should be 4 bytes"); + +/// Fixed head of an FFN ([MS-DOC] 2.9.82); the null-terminated UTF-16 font +/// name (xszFfn) follows. +struct FfnFixed { + std::uint8_t ffid; + std::int16_t wWeight; + std::uint8_t chs; + std::uint8_t ixchSzAlt; + std::array panose; + std::array fs; +}; +static_assert(sizeof(FfnFixed) == 39, "FfnFixed should be 39 bytes"); + #pragma pack(pop) struct ParsedFibRgCswNew { @@ -409,4 +443,19 @@ class PlcPcdMap : public PlcPcdBase { std::size_t m_cbPlc{0}; }; +/// PlcBteChpx ([MS-DOC] 2.8.5): a PLC keyed by WordDocument-stream offsets +/// (fc, not CP) whose data elements locate the ChpxFkp pages. +class PlcBteChpxMap : public PlcBase { +public: + PlcBteChpxMap(char *data, const std::size_t cbPlc) + : m_data(data), m_cbPlc(cbPlc) {} + + [[nodiscard]] char *data() const { return m_data; } + [[nodiscard]] std::size_t cbPlc() const { return m_cbPlc; } + +private: + char *m_data{nullptr}; + std::size_t m_cbPlc{0}; +}; + } // namespace odr::internal::oldms::text diff --git a/test/src/internal/oldms/doc_test.cpp b/test/src/internal/oldms/doc_test.cpp index 37b1a212..c45cbed1 100644 --- a/test/src/internal/oldms/doc_test.cpp +++ b/test/src/internal/oldms/doc_test.cpp @@ -1,11 +1,26 @@ #include +#include +#include +#include + +#include +#include +#include +#include +#include #include +#include + +#include +#include #include #include +#include using namespace odr; +using odr::test::oldms::collect_text; // A compressed (1-byte-per-CP) piece is "an array of 8-bit Unicode characters" // ([MS-DOC] 2.9.73 / 2.4.1 step 6): each byte is a code point, except the @@ -41,3 +56,263 @@ TEST(OldMs, doc_read_string_compressed) { EXPECT_EQ(read_string_compressed(in, 4), "caf\xC3\xA9"); } } + +namespace { + +void append_doc_u16(std::string &out, const std::uint16_t value) { + out.push_back(static_cast(value & 0xFF)); + out.push_back(static_cast(value >> 8)); +} + +void append_doc_u32(std::string &out, const std::uint32_t value) { + append_doc_u16(out, static_cast(value & 0xFFFF)); + append_doc_u16(out, static_cast(value >> 16)); +} + +std::string doc_prl(const std::uint16_t opcode, const std::string &operand) { + std::string prl; + append_doc_u16(prl, opcode); + prl += operand; + return prl; +} + +/// A minimal FIB ([MS-DOC] 2.5.1): nFib97, table stream /1Table, with the +/// three FcLcbs the parser reads (clx, plcfBteChpx, sttbfFfn). +std::string make_fib(const std::uint32_t ccp_text, + const internal::oldms::text::FcLcb clx, + const internal::oldms::text::FcLcb plcf_bte_chpx, + const internal::oldms::text::FcLcb sttbf_ffn) { + std::string fib; + // FibBase (32 bytes) + append_doc_u16(fib, 0xA5EC); // wIdent + append_doc_u16(fib, 0x00C1); // nFib97 + append_doc_u16(fib, 0); // unused + append_doc_u16(fib, 0x0409); // lid + append_doc_u16(fib, 0); // pnNext + append_doc_u16(fib, 0x0200); // flags: fWhichTblStm = 1 + append_doc_u16(fib, 0x00C1); // nFibBack + append_doc_u32(fib, 0); // lKey + fib += std::string(2, '\0'); // envr, flags + fib += std::string(12, '\0'); // reserved3-6 + // csw + fibRgW + append_doc_u16(fib, 14); + fib += std::string(28, '\0'); + // cslw + fibRgLw; ccpText is the 4th 32-bit field + append_doc_u16(fib, 22); + std::string rg_lw(88, '\0'); + std::string ccp; + append_doc_u32(ccp, ccp_text); + rg_lw.replace(12, 4, ccp); + fib += rg_lw; + // cbRgFcLcb + FibRgFcLcb97: plcfBteChpx is FcLcb index 12, sttbfFfn 15, + // clx 33 + append_doc_u16(fib, 93); + std::string rg(744, '\0'); + const auto put_fclcb = [&rg](const std::size_t index, + const internal::oldms::text::FcLcb value) { + std::string bytes; + append_doc_u32(bytes, value.fc); + append_doc_u32(bytes, value.lcb); + rg.replace(index * 8, 8, bytes); + }; + put_fclcb(12, plcf_bte_chpx); + put_fclcb(15, sttbf_ffn); + put_fclcb(33, clx); + fib += rg; + // cswNew + append_doc_u16(fib, 0); + return fib; +} + +/// An SttbfFfn ([MS-DOC] 2.9.286) with the given font names. +std::string make_sttbf_ffn(const std::vector &names) { + std::string sttb; + append_doc_u16(sttb, static_cast(names.size())); // cData + append_doc_u16(sttb, 0); // cbExtra + for (const std::string &name : names) { + std::string ffn(39, '\0'); // FfnFixed + for (const char c : name) { + ffn.push_back(c); + ffn.push_back('\0'); + } + ffn += std::string(2, '\0'); // terminator + sttb.push_back(static_cast(ffn.size())); + sttb += ffn; + } + return sttb; +} + +} // namespace + +// Character SPRMs ([MS-DOC] 2.6.1) map onto TextStyle; unknown SPRMs are +// skipped via their operand size, malformed grpprls throw. +TEST(OldMs, doc_apply_character_sprms) { + using internal::oldms::text::apply_character_sprms; + + TextStyle base; + base.font_size = Measure("10pt"); + const std::vector fonts = {"Arial", "Courier New"}; + + { + std::string grpprl; + grpprl += doc_prl(0x0835, "\x01"); // bold: on + grpprl += doc_prl(0x0836, "\x81"); // italic: invert style default -> on + grpprl += doc_prl(0x0837, "\x80"); // strike: match style default -> off + grpprl += doc_prl(0x2A3E, "\x01"); // single underline + grpprl += doc_prl(0x4A43, std::string("\x20\x00", 2)); // 32 half-points + grpprl += doc_prl(0x4A4F, std::string("\x01\x00", 2)); // font index 1 + grpprl += doc_prl(0x6870, std::string("\x11\x22\x33\x00", 4)); // COLORREF + grpprl += doc_prl(0x2A0C, "\x07"); // highlight: yellow + const TextStyle style = apply_character_sprms(base, grpprl, fonts); + EXPECT_EQ(style.font_weight, FontWeight::bold); + EXPECT_EQ(style.font_style, FontStyle::italic); + EXPECT_EQ(style.font_line_through, false); + EXPECT_EQ(style.font_underline, true); + EXPECT_EQ(style.font_size, Measure("16pt")); + EXPECT_STREQ(style.font_name, "Courier New"); + ASSERT_TRUE(style.font_color.has_value()); + EXPECT_EQ(style.font_color->rgb(), 0x112233u); + ASSERT_TRUE(style.background_color.has_value()); + EXPECT_EQ(style.background_color->rgb(), 0xFFFF00u); + } + + { + // Legacy palette color; unknown fixed-size and variable-size SPRMs are + // skipped without disturbing later Prls. + std::string grpprl; + grpprl += doc_prl(0x2A42, "\x06"); // ico: red + grpprl += doc_prl(0x0838, "\x01"); // unknown, 1 B + grpprl += doc_prl(0xC875, std::string("\x02\xAA\xBB", 3)); // unknown, var + grpprl += doc_prl(0x0835, "\x01"); // bold: on + const TextStyle style = apply_character_sprms(base, grpprl, fonts); + ASSERT_TRUE(style.font_color.has_value()); + EXPECT_EQ(style.font_color->rgb(), 0xFF0000u); + EXPECT_EQ(style.font_weight, FontWeight::bold); + } + + { + // cvAuto ([MS-DOC] 2.9.43) resets an earlier explicit color. + TextStyle colored = base; + colored.font_color = Color(std::uint32_t{0x123456}); + const TextStyle style = apply_character_sprms( + colored, doc_prl(0x6870, std::string("\x00\x00\x00\xFF", 4)), fonts); + EXPECT_FALSE(style.font_color.has_value()); + } + + // Truncated operand, half a Sprm, and an out-of-range sprmCHps all throw. + EXPECT_THROW(apply_character_sprms(base, doc_prl(0x4A43, "\x01"), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms(base, std::string("\x35", 1), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A43, std::string("\x01\x00", 2)), fonts), + std::runtime_error); +} + +TEST(OldMs, doc_read_font_names) { + const std::string sttb = make_sttbf_ffn({"Arial", "Times New Roman"}); + std::istringstream in(sttb); + const std::vector names = internal::oldms::text::read_font_names( + in, {0, static_cast(sttb.size())}); + ASSERT_EQ(names.size(), 2); + EXPECT_EQ(names[0], "Arial"); + EXPECT_EQ(names[1], "Times New Roman"); +} + +// End-to-end over a synthetic .doc: the PlcBteChpx/ChpxFkp run boundaries +// split the text into styled spans ([MS-DOC] 2.4.6.2), the default run keeps +// the 10pt default, and the paragraph carries a style for empty-height. +TEST(OldMs, doc_character_formatting) { + const std::string body = "plain bold\r"; + constexpr std::uint32_t text_fc = 1536; + + // ChpxFkp page ([MS-DOC] 2.9.33) at pn 2: run "plain " -> no Chpx (rgb 0), + // run "bold\r" -> bold + font 0. + std::string fkp(512, '\0'); + { + std::string head; + append_doc_u32(head, text_fc); + append_doc_u32(head, text_fc + 6); + append_doc_u32(head, text_fc + 11); + head.push_back('\0'); // rgb[0]: default properties + head.push_back(static_cast(240)); // rgb[1]: Chpx at byte 480 + fkp.replace(0, head.size(), head); + + std::string chpx; + chpx += doc_prl(0x0835, "\x01"); // bold + chpx += doc_prl(0x4A4F, std::string("\x00\x00", 2)); // font index 0 + fkp[480] = static_cast(chpx.size()); + fkp.replace(481, chpx.size(), chpx); + fkp[511] = 2; // crun + } + + // Table stream: Clx at 0 (one compressed piece), SttbfFfn at 64, + // PlcBteChpx at 128. + std::string plc_pcd; + append_doc_u32(plc_pcd, 0); + append_doc_u32(plc_pcd, 11); + append_doc_u16(plc_pcd, 0); // Pcd flags + append_doc_u32(plc_pcd, (1u << 30) | (text_fc * 2u)); // fCompressed | fc + append_doc_u16(plc_pcd, 0); // prm + std::string clx; + clx.push_back('\x02'); + append_doc_u32(clx, static_cast(plc_pcd.size())); + clx += plc_pcd; + + const std::string sttbf_ffn = make_sttbf_ffn({"Arial"}); + + std::string plc_bte; + append_doc_u32(plc_bte, text_fc); + append_doc_u32(plc_bte, text_fc + 11); + append_doc_u32(plc_bte, 2); // PnFkpChpx: page at 2 * 512 + + std::string table(256, '\0'); + table.replace(0, clx.size(), clx); + table.replace(64, sttbf_ffn.size(), sttbf_ffn); + table.replace(128, plc_bte.size(), plc_bte); + + std::string word_document = + make_fib(11, {0, static_cast(clx.size())}, + {128, static_cast(plc_bte.size())}, + {64, static_cast(sttbf_ffn.size())}); + word_document.resize(1024, '\0'); + word_document += fkp; + word_document += body; + + auto files = std::make_shared(); + files->copy(std::make_shared(word_document), + internal::AbsPath("/WordDocument")); + files->copy(std::make_shared(table), + internal::AbsPath("/1Table")); + const Document document( + std::make_shared(files)); + + std::vector paragraphs; + for (const Element child : document.root_element().children()) { + paragraphs.push_back(child); + } + ASSERT_EQ(paragraphs.size(), 1); + ASSERT_EQ(paragraphs[0].type(), ElementType::paragraph); + + std::vector spans; + for (const Element child : paragraphs[0].children()) { + ASSERT_EQ(child.type(), ElementType::span); + spans.push_back(child); + } + ASSERT_EQ(spans.size(), 2); + + EXPECT_EQ(collect_text(spans[0]), "plain "); + const TextStyle plain = spans[0].as_span().style(); + EXPECT_EQ(plain.font_size, Measure("10pt")); + EXPECT_FALSE(plain.font_weight.has_value()); + EXPECT_EQ(plain.font_name, nullptr); + + EXPECT_EQ(collect_text(spans[1]), "bold"); + const TextStyle bold = spans[1].as_span().style(); + EXPECT_EQ(bold.font_weight, FontWeight::bold); + EXPECT_STREQ(bold.font_name, "Arial"); + EXPECT_EQ(bold.font_size, Measure("10pt")); + + EXPECT_EQ(paragraphs[0].as_paragraph().text_style().font_size, + Measure("10pt")); +} From 88cbba175f4a5158e90223fa7fc64c0599c4dc05 Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Thu, 16 Jul 2026 21:26:21 +0200 Subject: [PATCH 2/7] fix(doc): allow empty font table for sprmCRgFtc0 Per [MS-DOC] 2.6.1 an sprmCRgFtc0 operand of 0 is valid when SttbfFfn has no entries; the style-sheet default font (unmodelled) applies, so leave the font unset instead of throwing. Genuinely out-of-range indexes still throw. Addresses Codex review feedback on PR #608. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AEvWWwe1EyeWH7ANHoAvPH --- src/odr/internal/oldms/text/doc_helper.cpp | 10 ++++++++-- test/src/internal/oldms/doc_test.cpp | 21 +++++++++++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 468d56ad..5bd278b5 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -198,11 +198,17 @@ text::apply_character_sprms(TextStyle style, const std::string_view grpprl, style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); } break; case sprmCRgFtc0: { + // SttbfFfn index ([MS-DOC] 2.6.1); with 0 entries the value MUST be 0 + // and the (unmodelled) style-sheet default font applies — leave unset. const auto ftc = static_cast(read_u16(operand, 0)); - if (ftc < 0 || static_cast(ftc) >= font_names.size()) { + if (ftc < 0 || (font_names.empty() ? ftc != 0 + : static_cast(ftc) >= + font_names.size())) { throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); } - style.font_name = font_names[static_cast(ftc)]; + if (!font_names.empty()) { + style.font_name = font_names[static_cast(ftc)]; + } } break; case sprmCCv: { // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. diff --git a/test/src/internal/oldms/doc_test.cpp b/test/src/internal/oldms/doc_test.cpp index c45cbed1..b3e95a0a 100644 --- a/test/src/internal/oldms/doc_test.cpp +++ b/test/src/internal/oldms/doc_test.cpp @@ -199,6 +199,27 @@ TEST(OldMs, doc_apply_character_sprms) { EXPECT_FALSE(style.font_color.has_value()); } + { + // sprmCRgFtc0 with an empty SttbfFfn: 0 is valid ([MS-DOC] 2.6.1) and + // leaves the font unset; anything else is out of range. + const std::vector no_fonts; + const TextStyle style = apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\x00\x00", 2)), no_fonts); + EXPECT_EQ(style.font_name, nullptr); + EXPECT_THROW( + apply_character_sprms(base, doc_prl(0x4A4F, std::string("\x01\x00", 2)), + no_fonts), + std::runtime_error); + } + + // Out-of-range and negative sprmCRgFtc0 indexes throw. + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\x02\x00", 2)), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\xFF\xFF", 2)), fonts), + std::runtime_error); + // Truncated operand, half a Sprm, and an out-of-range sprmCHps all throw. EXPECT_THROW(apply_character_sprms(base, doc_prl(0x4A43, "\x01"), fonts), std::runtime_error); From 9556aa2a58b7522234107a9ec67bc89c6317a00b Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Tue, 21 Jul 2026 20:27:44 +0200 Subject: [PATCH 3/7] refactor(doc): split element and style registry Mirror the xls module: a StyleRegistry owns the resolved character TextStyles (index 0 = default) and the interned font names, while the ElementRegistry stores only a per-element style index. CharacterStyles shrinks to a runs-only CharacterRuns; the SPRM/font-table resolution moves to doc_style.{hpp,cpp}. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PvbJYsLLST3hkCgkcHBoFb --- CMakeLists.txt | 1 + src/odr/internal/oldms/text/AGENTS.md | 21 +- src/odr/internal/oldms/text/doc_document.cpp | 28 ++- src/odr/internal/oldms/text/doc_document.hpp | 4 + .../oldms/text/doc_element_registry.cpp | 26 +- .../oldms/text/doc_element_registry.hpp | 19 +- src/odr/internal/oldms/text/doc_helper.cpp | 236 ++---------------- src/odr/internal/oldms/text/doc_helper.hpp | 40 +-- src/odr/internal/oldms/text/doc_parser.cpp | 27 +- src/odr/internal/oldms/text/doc_parser.hpp | 5 + src/odr/internal/oldms/text/doc_style.cpp | 219 ++++++++++++++++ src/odr/internal/oldms/text/doc_style.hpp | 46 ++++ test/src/internal/oldms/doc_test.cpp | 2 +- 13 files changed, 368 insertions(+), 306 deletions(-) create mode 100644 src/odr/internal/oldms/text/doc_style.cpp create mode 100644 src/odr/internal/oldms/text/doc_style.hpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 6f1aeb25..ef8ac0aa 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -153,6 +153,7 @@ set(ODR_SOURCE_FILES "src/odr/internal/oldms/text/doc_helper.cpp" "src/odr/internal/oldms/text/doc_io.cpp" "src/odr/internal/oldms/text/doc_parser.cpp" + "src/odr/internal/oldms/text/doc_style.cpp" "src/odr/internal/oldms/spreadsheet/xls_document.cpp" "src/odr/internal/oldms/spreadsheet/xls_element_registry.cpp" "src/odr/internal/oldms/spreadsheet/xls_io.cpp" diff --git a/src/odr/internal/oldms/text/AGENTS.md b/src/odr/internal/oldms/text/AGENTS.md index 7cf20fcc..d05ba159 100644 --- a/src/odr/internal/oldms/text/AGENTS.md +++ b/src/odr/internal/oldms/text/AGENTS.md @@ -21,9 +21,10 @@ numbers are cited inline in code and in the read-path maps below. |---|---| | `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`, `PnFkpChpx`, `FfnFixed`) + `static_assert` sizes + `PlcPcdMap`/`PlcBteChpxMap` + `ParsedFib` + the character SPRM opcodes | | `doc_io.{hpp,cpp}` | `read(...)` over `std::istream`: variable-length FIB, Clx walk, string decoding | -| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index`; `CharacterStyles` (fc-keyed style runs) + `read_character_styles` (PlcBteChpx/ChpxFkp walk), `apply_character_sprms`, `read_font_names` (SttbfFfn) | +| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index`; `CharacterRuns` (fc-keyed style-index runs) + `read_character_runs` (PlcBteChpx/ChpxFkp walk) | +| `doc_style.{hpp,cpp}` | `StyleRegistry` (resolved `TextStyle`s by index + font-name intern store), `apply_character_sprms`, `read_font_names` (SttbfFfn) | | `doc_parser.{hpp,cpp}` | `parse_tree` → styled runs + tree; `TextCleaner` (field & control-char handling) | -| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload + per-element `TextStyle` map + font-name intern store | +| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload + per-element style-index map | | `doc_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | ## Design decisions @@ -63,13 +64,15 @@ assumption + fix plan in [`../AGENTS.md`](../AGENTS.md). **Direct character formatting only, resolved to styled spans.** The tree is `root → paragraph → span → text`; each span (and each paragraph, for -empty-paragraph height) stores a resolved `TextStyle` in a registry side-map. -The §2.4.6.2 walk: `PlcBteChpx` (table stream) → 512-byte `ChpxFkp` pages -(WordDocument stream) → per-run `Chpx.grpprl`, applied over a default of 10pt -(the `sprmCHps` default of 20 half-points; the old flat `11pt` placeholder is -gone). The non-obvious bits: +empty-paragraph height) stores a **style index** in an element-registry +side-map, resolved through the document's `StyleRegistry` (element/style +registry split, mirroring `../spreadsheet`; index 0 is the 10pt default — +the `sprmCHps` default of 20 half-points; the old flat `11pt` placeholder is +gone). The §2.4.6.2 walk: `PlcBteChpx` (table stream) → 512-byte `ChpxFkp` +pages (WordDocument stream) → per-run `Chpx.grpprl`, applied over the default +style. The non-obvious bits: - **Runs are keyed by stream offset (fc), not CP**, so the piece decode walks - each piece in style-uniform chunks (`CharacterStyles::chunk_end`); a + each piece in style-uniform chunks (`CharacterRuns::chunk_end`); a boundary inside a 2-byte CP is pushed past it. - **Equal `Chpx` bytes share one resolved style**, and adjacent equal-style runs merge, so span count stays low. `rgb[j] == 0` means default properties. @@ -79,7 +82,7 @@ gone). The non-obvious bits: `sprmCIco`/`sprmCHighlight` use the `Ico` palette (§2.9.119) — the spec's extracted table repeats `0x0C` for `0x0D`, which is dark red (`0x800000`). - **`Pcd.Prm` modifications and STSH styles are not applied** (open work §1). -- Font names from `SttbfFfn` are interned in a registry `std::deque` so +- Font names from `SttbfFfn` are interned in a `StyleRegistry` `std::deque` so `TextStyle::font_name` (`const char *`) stays valid. `Document::is_editable()` → `false`; `save`/`text_set_content` throw. diff --git a/src/odr/internal/oldms/text/doc_document.cpp b/src/odr/internal/oldms/text/doc_document.cpp index b9719ddd..def73011 100644 --- a/src/odr/internal/oldms/text/doc_document.cpp +++ b/src/odr/internal/oldms/text/doc_document.cpp @@ -13,15 +13,17 @@ namespace odr::internal::oldms::text { namespace { std::unique_ptr -create_element_adapter(const Document &document, ElementRegistry ®istry); +create_element_adapter(const Document &document, ElementRegistry ®istry, + const StyleRegistry &style_registry); } Document::Document(std::shared_ptr files) : internal::Document(FileType::legacy_word_document, DocumentType::text, std::move(files)) { - m_root_element = parse_tree(m_element_registry, *m_files); + m_root_element = parse_tree(m_element_registry, m_style_registry, *m_files); - m_element_adapter = create_element_adapter(*this, m_element_registry); + m_element_adapter = + create_element_adapter(*this, m_element_registry, m_style_registry); } ElementRegistry &Document::element_registry() { return m_element_registry; } @@ -30,6 +32,10 @@ const ElementRegistry &Document::element_registry() const { return m_element_registry; } +const StyleRegistry &Document::style_registry() const { + return m_style_registry; +} + bool Document::is_editable() const noexcept { return false; } bool Document::is_savable(const bool encrypted) const noexcept { @@ -57,8 +63,10 @@ class ElementAdapter final : public abstract::ElementAdapter, public abstract::SpanAdapter, public abstract::TextAdapter { public: - ElementAdapter(const Document &document, ElementRegistry ®istry) - : m_document(&document), m_registry(®istry) {} + ElementAdapter(const Document &document, ElementRegistry ®istry, + const StyleRegistry &style_registry) + : m_document(&document), m_registry(®istry), + m_style_registry(&style_registry) {} [[nodiscard]] ElementType element_type(const ElementIdentifier element_id) const override { @@ -187,18 +195,20 @@ class ElementAdapter final : public abstract::ElementAdapter, [[maybe_unused]] const Document *m_document{nullptr}; ElementRegistry *m_registry{nullptr}; + const StyleRegistry *m_style_registry{nullptr}; /// The character style stored for a paragraph or span element. [[nodiscard]] TextStyle stored_style(const ElementIdentifier element_id) const { - const TextStyle *style = m_registry->element_style(element_id); - return style != nullptr ? *style : TextStyle{}; + return m_style_registry->text_style( + m_registry->element_style_index(element_id)); } }; std::unique_ptr -create_element_adapter(const Document &document, ElementRegistry ®istry) { - return std::make_unique(document, registry); +create_element_adapter(const Document &document, ElementRegistry ®istry, + const StyleRegistry &style_registry) { + return std::make_unique(document, registry, style_registry); } } // namespace diff --git a/src/odr/internal/oldms/text/doc_document.hpp b/src/odr/internal/oldms/text/doc_document.hpp index c8812706..66f65389 100644 --- a/src/odr/internal/oldms/text/doc_document.hpp +++ b/src/odr/internal/oldms/text/doc_document.hpp @@ -2,6 +2,7 @@ #include #include +#include #include @@ -15,6 +16,8 @@ class Document final : public internal::Document { [[nodiscard]] const ElementRegistry &element_registry() const; + [[nodiscard]] const StyleRegistry &style_registry() const; + [[nodiscard]] bool is_editable() const noexcept override; [[nodiscard]] bool is_savable(bool encrypted) const noexcept override; @@ -23,6 +26,7 @@ class Document final : public internal::Document { private: ElementRegistry m_element_registry; + StyleRegistry m_style_registry; }; } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_element_registry.cpp b/src/odr/internal/oldms/text/doc_element_registry.cpp index f8ce1336..fa1e4627 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.cpp +++ b/src/odr/internal/oldms/text/doc_element_registry.cpp @@ -1,6 +1,5 @@ #include -#include #include namespace odr::internal::oldms::text { @@ -8,8 +7,7 @@ namespace odr::internal::oldms::text { void ElementRegistry::clear() noexcept { m_elements.clear(); m_texts.clear(); - m_styles.clear(); - m_font_names.clear(); + m_style_indices.clear(); } [[nodiscard]] std::size_t ElementRegistry::size() const noexcept { @@ -79,24 +77,16 @@ void ElementRegistry::append_child(const ElementIdentifier parent_id, element_at(parent_id).last_child_id = child_id; } -const char *ElementRegistry::intern_font_name(const std::string &name) { - if (const auto it = std::ranges::find(m_font_names, name); - it != m_font_names.end()) { - return it->c_str(); - } - return m_font_names.emplace_back(name).c_str(); -} - -void ElementRegistry::set_element_style(const ElementIdentifier id, - TextStyle style) { +void ElementRegistry::set_element_style_index(const ElementIdentifier id, + const std::uint32_t index) { check_element_id(id); - m_styles[id] = std::move(style); + m_style_indices[id] = index; } -const TextStyle * -ElementRegistry::element_style(const ElementIdentifier id) const { - const auto it = m_styles.find(id); - return it != m_styles.end() ? &it->second : nullptr; +std::uint32_t +ElementRegistry::element_style_index(const ElementIdentifier id) const { + const auto it = m_style_indices.find(id); + return it != m_style_indices.end() ? it->second : 0; } void ElementRegistry::check_element_id(const ElementIdentifier id) const { diff --git a/src/odr/internal/oldms/text/doc_element_registry.hpp b/src/odr/internal/oldms/text/doc_element_registry.hpp index 48fbe8d1..ad28267b 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.hpp +++ b/src/odr/internal/oldms/text/doc_element_registry.hpp @@ -2,10 +2,8 @@ #include #include -#include -#include -#include +#include #include #include #include @@ -42,20 +40,15 @@ class ElementRegistry final { void append_child(ElementIdentifier parent_id, ElementIdentifier child_id); - /// Stores a font name and returns a pointer that stays valid for the - /// registry's lifetime (`TextStyle::font_name` is a `const char *`). - const char *intern_font_name(const std::string &name); - - /// Character style of a span or paragraph element. - void set_element_style(ElementIdentifier id, TextStyle style); - [[nodiscard]] const TextStyle *element_style(ElementIdentifier id) const; + /// Character style of a span or paragraph element, as an index into the + /// document's `StyleRegistry` (0 is the default style). + void set_element_style_index(ElementIdentifier id, std::uint32_t index); + [[nodiscard]] std::uint32_t element_style_index(ElementIdentifier id) const; private: std::vector m_elements; std::unordered_map m_texts; - std::unordered_map m_styles; - /// Deque: elements never move, so interned `c_str()`s stay valid. - std::deque m_font_names; + std::unordered_map m_style_indices; void check_element_id(ElementIdentifier id) const; void check_text_id(ElementIdentifier id) const; diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 5bd278b5..6878eff5 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -1,62 +1,18 @@ #include #include +#include #include #include -#include #include -#include #include #include #include +#include +#include #include -namespace odr::internal::oldms::text { - -namespace { - -/// The Ico palette ([MS-DOC] 2.9.119); index 0 is cvAuto (no explicit color). -constexpr std::array ico_colors = { - 0x000000, 0x000000, 0x0000FF, 0x00FFFF, 0x00FF00, 0xFF00FF, - 0xFF0000, 0xFFFF00, 0xFFFFFF, 0x000080, 0x008080, 0x008000, - 0x800080, 0x800000, 0x808000, 0x808080, 0xC0C0C0}; - -std::optional ico_color(const std::uint8_t ico) { - if (ico >= ico_colors.size()) { - throw std::runtime_error("doc: Ico value out of range"); - } - if (ico == 0) { - return std::nullopt; // cvAuto - } - return Color(ico_colors[ico]); -} - -/// ToggleOperand ([MS-DOC] 2.9.327) against the (unmodelled) style value: -/// 0x80 matches the style, 0x81 inverts it; styles default to off. -bool toggle_on(const std::uint8_t value) { - switch (value) { - case 0x00: - case 0x80: - return false; - case 0x01: - case 0x81: - return true; - default: - throw std::runtime_error("doc: unexpected ToggleOperand value"); - } -} - -std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { - std::uint16_t value; - std::memcpy(&value, bytes.data() + at, sizeof(value)); - return value; -} - -} // namespace - -} // namespace odr::internal::oldms::text - namespace odr::internal::oldms { text::CharacterIndex text::read_character_index(std::istream &in) { @@ -83,25 +39,13 @@ text::CharacterIndex text::read_character_index(std::istream &in) { return result; } -text::CharacterStyles::CharacterStyles(TextStyle default_style) { - m_styles.push_back(std::move(default_style)); -} - -std::uint32_t text::CharacterStyles::add_style(TextStyle style) { - m_styles.push_back(std::move(style)); - return static_cast(m_styles.size() - 1); -} - -void text::CharacterStyles::append_run(const std::uint32_t begin_fc, - const std::uint32_t end_fc, - const std::uint32_t style_index) { +void text::CharacterRuns::append_run(const std::uint32_t begin_fc, + const std::uint32_t end_fc, + const std::uint32_t style_index) { if (begin_fc >= end_fc || (!m_runs.empty() && begin_fc < m_runs.back().end_fc)) { throw std::runtime_error("doc: character runs must be ascending"); } - if (style_index >= m_styles.size()) { - throw std::out_of_range("doc: style index out of range"); - } if (!m_runs.empty() && m_runs.back().end_fc == begin_fc && m_runs.back().style_index == style_index) { m_runs.back().end_fc = end_fc; @@ -110,7 +54,7 @@ void text::CharacterStyles::append_run(const std::uint32_t begin_fc, m_runs.push_back({begin_fc, end_fc, style_index}); } -std::uint32_t text::CharacterStyles::index_at(const std::uint32_t fc) const { +std::uint32_t text::CharacterRuns::index_at(const std::uint32_t fc) const { const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); if (it == m_runs.begin()) { return 0; @@ -119,7 +63,7 @@ std::uint32_t text::CharacterStyles::index_at(const std::uint32_t fc) const { return fc < run.end_fc ? run.style_index : 0; } -std::uint32_t text::CharacterStyles::chunk_end(const std::uint32_t fc) const { +std::uint32_t text::CharacterRuns::chunk_end(const std::uint32_t fc) const { const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); if (it != m_runs.begin() && fc < std::prev(it)->end_fc) { return std::prev(it)->end_fc; @@ -130,159 +74,19 @@ std::uint32_t text::CharacterStyles::chunk_end(const std::uint32_t fc) const { return std::numeric_limits::max(); } -const TextStyle &text::CharacterStyles::style(const std::uint32_t index) const { - return m_styles.at(index); -} - -TextStyle -text::apply_character_sprms(TextStyle style, const std::string_view grpprl, - const std::vector &font_names) { - std::size_t at = 0; - while (at + sizeof(Sprm) <= grpprl.size()) { - Sprm sprm; - std::memcpy(&sprm, grpprl.data() + at, sizeof(sprm)); - const std::uint16_t opcode = std::bit_cast(sprm); - at += sizeof(sprm); - - std::size_t operand_size; - if (const int fixed_size = sprm.operand_size(); fixed_size >= 0) { - operand_size = static_cast(fixed_size); - } else { - // spra == 6: the first operand byte is the size of the remainder. The - // two SPRMs encoded differently (sprmTDefTable, sprmPChgTabs with - // cb == 0xFF) are not character properties and must not appear here. - if (at >= grpprl.size()) { - throw std::runtime_error("doc: Prl operand overruns grpprl"); - } - operand_size = static_cast(grpprl[at]); - ++at; - if (opcode == 0xD608 || (opcode == 0xC615 && operand_size == 0xFF)) { - throw std::runtime_error("doc: unexpected table SPRM in a Chpx"); - } - } - if (at + operand_size > grpprl.size()) { - throw std::runtime_error("doc: Prl operand overruns grpprl"); - } - const std::string_view operand = grpprl.substr(at, operand_size); - at += operand_size; - - switch (opcode) { - case sprmCFBold: - style.font_weight = toggle_on(static_cast(operand[0])) - ? FontWeight::bold - : FontWeight::normal; - break; - case sprmCFItalic: - style.font_style = toggle_on(static_cast(operand[0])) - ? FontStyle::italic - : FontStyle::normal; - break; - case sprmCFStrike: - style.font_line_through = - toggle_on(static_cast(operand[0])); - break; - case sprmCHighlight: - style.background_color = ico_color(static_cast(operand[0])); - break; - case sprmCKul: - style.font_underline = operand[0] != 0; - break; - case sprmCIco: - style.font_color = ico_color(static_cast(operand[0])); - break; - case sprmCHps: { - const std::uint16_t half_points = read_u16(operand, 0); - if (half_points < 2 || half_points > 3276) { - throw std::runtime_error("doc: sprmCHps value out of range"); - } - style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); - } break; - case sprmCRgFtc0: { - // SttbfFfn index ([MS-DOC] 2.6.1); with 0 entries the value MUST be 0 - // and the (unmodelled) style-sheet default font applies — leave unset. - const auto ftc = static_cast(read_u16(operand, 0)); - if (ftc < 0 || (font_names.empty() ? ftc != 0 - : static_cast(ftc) >= - font_names.size())) { - throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); - } - if (!font_names.empty()) { - style.font_name = font_names[static_cast(ftc)]; - } - } break; - case sprmCCv: { - // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. - if (static_cast(operand[3]) == 0xFF) { - style.font_color = std::nullopt; // cvAuto - } else { - style.font_color = Color(static_cast(operand[0]), - static_cast(operand[1]), - static_cast(operand[2])); - } - } break; - default: - break; // not modelled - } - } - if (at != grpprl.size()) { - throw std::runtime_error("doc: grpprl does not hold whole Prls"); - } - - return style; -} - -std::vector text::read_font_names(std::istream &table_stream, - const FcLcb sttbf_ffn) { - std::vector result; - if (sttbf_ffn.lcb == 0) { - return result; - } - table_stream.seekg(sttbf_ffn.fc); - - // SttbfFfn is a non-extended STTB ([MS-DOC] 2.2.4, 2.9.286): u16 cData, - // u16 cbExtra (0), then per entry a u8 byte count and an FFN. - const auto c_data = util::byte_stream::read(table_stream); - if (c_data == 0xFFFF) { - throw std::runtime_error("doc: unexpected extended SttbfFfn"); - } - const auto cb_extra = util::byte_stream::read(table_stream); - - result.reserve(c_data); - for (std::uint16_t i = 0; i < c_data; ++i) { - const auto cch_data = util::byte_stream::read(table_stream); - if (cch_data < sizeof(FfnFixed)) { - throw std::runtime_error("doc: FFN too short"); - } - const auto ffn = util::byte_stream::read(table_stream); - (void)ffn; - - // xszFfn: null-terminated UTF-16 within the remaining bytes; an - // alternative name (xszAlt) may follow the terminator. - const std::size_t name_units = (cch_data - sizeof(FfnFixed)) / 2; - std::u16string name = read_string_uncompressed(table_stream, name_units); - if ((cch_data - sizeof(FfnFixed)) % 2 != 0) { - table_stream.ignore(1); - } - if (const std::size_t nul = name.find(u'\0'); nul != std::u16string::npos) { - name.resize(nul); - } - result.push_back(util::string::u16string_to_string(name)); - - table_stream.ignore(cb_extra); - } - - return result; -} - -text::CharacterStyles text::read_character_styles( - std::istream &document_stream, std::istream &table_stream, - const FcLcb plcf_bte_chpx, const TextStyle &default_style, - const std::vector &font_names) { - CharacterStyles result(default_style); +text::CharacterRuns +text::read_character_runs(std::istream &document_stream, + std::istream &table_stream, const FcLcb plcf_bte_chpx, + StyleRegistry &style_registry, + const std::vector &font_names) { + CharacterRuns result; if (plcf_bte_chpx.lcb == 0) { return result; } + // Copy: `add_style` may reallocate the registry's style storage. + const TextStyle default_style = style_registry.text_style(0); + table_stream.seekg(plcf_bte_chpx.fc); std::string plc_bytes = util::stream::read(table_stream, plcf_bte_chpx.lcb); const PlcBteChpxMap plc(plc_bytes.data(), plc_bytes.size()); @@ -297,7 +101,7 @@ text::CharacterStyles text::read_character_styles( if (it != style_cache.end()) { return it->second; } - const std::uint32_t index = result.add_style( + const std::uint32_t index = style_registry.add_style( apply_character_sprms(default_style, grpprl, font_names)); style_cache.emplace(std::string(grpprl), index); return index; @@ -326,8 +130,8 @@ text::CharacterStyles text::read_character_styles( std::memcpy(&end_fc, page.data() + static_cast(j + 1) * 4, 4); - const auto rgb = - static_cast(page[(crun + 1) * 4 + std::size_t{j}]); + const auto rgb = static_cast( + page[static_cast(crun + 1) * 4 + std::size_t{j}]); std::string_view grpprl; if (rgb != 0) { const std::size_t chpx_at = static_cast(rgb) * 2; diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index 537d4941..6420d2ce 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -1,19 +1,17 @@ #pragma once -#include - #include #include #include #include #include -#include -#include #include namespace odr::internal::oldms::text { +class StyleRegistry; + class CharacterIndex { public: [[nodiscard]] bool empty() const { return m_entries.empty(); } @@ -113,13 +111,10 @@ class CharacterIndex { CharacterIndex read_character_index(std::istream &in); /// The document's character-formatting runs, keyed by WordDocument-stream -/// offset (fc). Style index 0 is the default style; offsets outside any run -/// resolve to it. -class CharacterStyles final { +/// offset (fc), each referencing a `StyleRegistry` style index. Index 0 is +/// the default style; offsets outside any run resolve to it. +class CharacterRuns final { public: - explicit CharacterStyles(TextStyle default_style); - - std::uint32_t add_style(TextStyle style); /// Appends a run; runs must be ascending and non-overlapping. Adjacent runs /// with the same style index are merged. void append_run(std::uint32_t begin_fc, std::uint32_t end_fc, @@ -130,7 +125,6 @@ class CharacterStyles final { /// End offset of the run containing `fc`, or the next run's begin, so text /// can be walked in style-uniform chunks; UINT32_MAX past the last run. [[nodiscard]] std::uint32_t chunk_end(std::uint32_t fc) const; - [[nodiscard]] const TextStyle &style(std::uint32_t index) const; private: struct Run { @@ -139,27 +133,17 @@ class CharacterStyles final { std::uint32_t style_index; }; - std::vector m_styles; std::vector m_runs; }; -/// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of -/// `style`; non-character SPRMs are skipped via their operand size. -/// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). -TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, - const std::vector &font_names); - -/// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at -/// `fc` in the table stream; empty for lcb == 0. -std::vector read_font_names(std::istream &table_stream, - FcLcb sttbf_ffn); - /// Reads the PlcBteChpx at `fc` in the table stream and every referenced /// ChpxFkp page ([MS-DOC] 2.9.33) from the WordDocument stream, resolving -/// each run's Chpx against `default_style`. Equal Chpx bytes share one style. -CharacterStyles -read_character_styles(std::istream &document_stream, std::istream &table_stream, - FcLcb plcf_bte_chpx, const TextStyle &default_style, - const std::vector &font_names); +/// each run's Chpx on top of the default style (index 0) into +/// `style_registry`. Equal Chpx bytes share one style. +CharacterRuns read_character_runs(std::istream &document_stream, + std::istream &table_stream, + FcLcb plcf_bte_chpx, + StyleRegistry &style_registry, + const std::vector &font_names); } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_parser.cpp b/src/odr/internal/oldms/text/doc_parser.cpp index 25f150f2..cfeeb7b9 100644 --- a/src/odr/internal/oldms/text/doc_parser.cpp +++ b/src/odr/internal/oldms/text/doc_parser.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -108,7 +109,7 @@ struct StyledRun { /// every character-formatting boundary. std::vector decode_styled_runs( std::istream &document_stream, const text::CharacterIndex &character_index, - const text::CharacterStyles &styles, const std::size_t ccp_text) { + const text::CharacterRuns &character_runs, const std::size_t ccp_text) { std::vector runs; std::size_t consumed_cp = 0; @@ -123,11 +124,11 @@ std::vector decode_styled_runs( while (cp < take) { const auto fc = static_cast(entry.data_offset + cp * bytes_per_cp); - const std::uint32_t style_index = styles.index_at(fc); + const std::uint32_t style_index = character_runs.index_at(fc); // CPs of this piece still in the same style run; a boundary that falls // inside a 2-byte CP is pushed past it. - const std::uint32_t chunk_end_fc = styles.chunk_end(fc); + const std::uint32_t chunk_end_fc = character_runs.chunk_end(fc); std::size_t chunk_cp = take - cp; if (chunk_end_fc != std::numeric_limits::max()) { chunk_cp = std::min( @@ -156,6 +157,7 @@ std::vector decode_styled_runs( } // namespace ElementIdentifier text::parse_tree(ElementRegistry ®istry, + StyleRegistry &style_registry, const abstract::ReadableFilesystem &files) { auto [root_id, _] = registry.create_element(ElementType::root); @@ -171,28 +173,29 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, std::vector font_names; for (const std::string &name : read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)) { - font_names.push_back(registry.intern_font_name(name)); + font_names.push_back(style_registry.intern_font_name(name)); } // Direct character formatting ([MS-DOC] 2.4.6.2); Pcd.Prm modifications are // not modelled. Without sprmCHps the size defaults to 20 half-points. TextStyle default_style; default_style.font_size = Measure(10.0, DynamicUnit("pt")); - const CharacterStyles styles = read_character_styles( + style_registry.add_style(std::move(default_style)); // index 0 + const CharacterRuns character_runs = read_character_runs( *document_stream, *table_stream, fib.fibRgFcLcb->plcfBteChpx, - default_style, font_names); + style_registry, font_names); table_stream->seekg(fib.fibRgFcLcb->clx.fc); const CharacterIndex character_index = read_character_index(*table_stream); const auto ccp_text = static_cast(fib.ccpText()); - const std::vector runs = - decode_styled_runs(*document_stream, character_index, styles, ccp_text); + const std::vector runs = decode_styled_runs( + *document_stream, character_index, character_runs, ccp_text); // Build the tree: paragraphs are opened lazily so the trailing guard // paragraph mark does not produce an extra empty paragraph. Each paragraph - // and span stores its style; empty paragraphs keep their height through the - // paragraph style. + // and span stores its style index; empty paragraphs keep their height + // through the paragraph style. TextCleaner cleaner; ElementIdentifier paragraph_id = null_element_id; @@ -200,7 +203,7 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, if (paragraph_id == null_element_id) { auto [id, paragraph] = registry.create_element(ElementType::paragraph); registry.append_child(root_id, id); - registry.set_element_style(id, styles.style(style_index)); + registry.set_element_style_index(id, style_index); paragraph_id = id; } }; @@ -217,7 +220,7 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, !cleaned.empty()) { ensure_paragraph(run.style_index); auto [span_id, span] = registry.create_element(ElementType::span); - registry.set_element_style(span_id, styles.style(run.style_index)); + registry.set_element_style_index(span_id, run.style_index); registry.append_child(paragraph_id, span_id); auto [text_id, text_element, text_entry] = diff --git a/src/odr/internal/oldms/text/doc_parser.hpp b/src/odr/internal/oldms/text/doc_parser.hpp index d510d6b4..24dfb413 100644 --- a/src/odr/internal/oldms/text/doc_parser.hpp +++ b/src/odr/internal/oldms/text/doc_parser.hpp @@ -8,8 +8,13 @@ class ReadableFilesystem; namespace odr::internal::oldms::text { class ElementRegistry; +class StyleRegistry; +/// Parses the `/WordDocument` stream into root → paragraph → span → text +/// elements; fills `style_registry` with the resolved character styles. +/// \return the root element id. ElementIdentifier parse_tree(ElementRegistry ®istry, + StyleRegistry &style_registry, const abstract::ReadableFilesystem &files); } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_style.cpp b/src/odr/internal/oldms/text/doc_style.cpp new file mode 100644 index 00000000..03d3ad93 --- /dev/null +++ b/src/odr/internal/oldms/text/doc_style.cpp @@ -0,0 +1,219 @@ +#include + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace odr::internal::oldms::text { + +namespace { + +/// The Ico palette ([MS-DOC] 2.9.119); index 0 is cvAuto (no explicit color). +constexpr std::array ico_colors = { + 0x000000, 0x000000, 0x0000FF, 0x00FFFF, 0x00FF00, 0xFF00FF, + 0xFF0000, 0xFFFF00, 0xFFFFFF, 0x000080, 0x008080, 0x008000, + 0x800080, 0x800000, 0x808000, 0x808080, 0xC0C0C0}; + +std::optional ico_color(const std::uint8_t ico) { + if (ico >= ico_colors.size()) { + throw std::runtime_error("doc: Ico value out of range"); + } + if (ico == 0) { + return std::nullopt; // cvAuto + } + return Color(ico_colors[ico]); +} + +/// ToggleOperand ([MS-DOC] 2.9.327) against the (unmodelled) style value: +/// 0x80 matches the style, 0x81 inverts it; styles default to off. +bool toggle_on(const std::uint8_t value) { + switch (value) { + case 0x00: + case 0x80: + return false; + case 0x01: + case 0x81: + return true; + default: + throw std::runtime_error("doc: unexpected ToggleOperand value"); + } +} + +std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { + std::uint16_t value; + std::memcpy(&value, bytes.data() + at, sizeof(value)); + return value; +} + +} // namespace + +} // namespace odr::internal::oldms::text + +namespace odr::internal::oldms { + +const char *text::StyleRegistry::intern_font_name(const std::string &name) { + if (const auto it = std::ranges::find(m_font_names, name); + it != m_font_names.end()) { + return it->c_str(); + } + return m_font_names.emplace_back(name).c_str(); +} + +std::uint32_t text::StyleRegistry::add_style(TextStyle style) { + m_styles.push_back(std::move(style)); + return static_cast(m_styles.size() - 1); +} + +const TextStyle & +text::StyleRegistry::text_style(const std::uint32_t index) const { + return m_styles.at(index); +} + +TextStyle +text::apply_character_sprms(TextStyle style, const std::string_view grpprl, + const std::vector &font_names) { + std::size_t at = 0; + while (at + sizeof(Sprm) <= grpprl.size()) { + Sprm sprm; + std::memcpy(&sprm, grpprl.data() + at, sizeof(sprm)); + const std::uint16_t opcode = std::bit_cast(sprm); + at += sizeof(sprm); + + std::size_t operand_size; + if (const int fixed_size = sprm.operand_size(); fixed_size >= 0) { + operand_size = static_cast(fixed_size); + } else { + // spra == 6: the first operand byte is the size of the remainder. The + // two SPRMs encoded differently (sprmTDefTable, sprmPChgTabs with + // cb == 0xFF) are not character properties and must not appear here. + if (at >= grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + operand_size = static_cast(grpprl[at]); + ++at; + if (opcode == 0xD608 || (opcode == 0xC615 && operand_size == 0xFF)) { + throw std::runtime_error("doc: unexpected table SPRM in a Chpx"); + } + } + if (at + operand_size > grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + const std::string_view operand = grpprl.substr(at, operand_size); + at += operand_size; + + switch (opcode) { + case sprmCFBold: + style.font_weight = toggle_on(static_cast(operand[0])) + ? FontWeight::bold + : FontWeight::normal; + break; + case sprmCFItalic: + style.font_style = toggle_on(static_cast(operand[0])) + ? FontStyle::italic + : FontStyle::normal; + break; + case sprmCFStrike: + style.font_line_through = + toggle_on(static_cast(operand[0])); + break; + case sprmCHighlight: + style.background_color = ico_color(static_cast(operand[0])); + break; + case sprmCKul: + style.font_underline = operand[0] != 0; + break; + case sprmCIco: + style.font_color = ico_color(static_cast(operand[0])); + break; + case sprmCHps: { + const std::uint16_t half_points = read_u16(operand, 0); + if (half_points < 2 || half_points > 3276) { + throw std::runtime_error("doc: sprmCHps value out of range"); + } + style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); + } break; + case sprmCRgFtc0: { + // SttbfFfn index ([MS-DOC] 2.6.1); with 0 entries the value MUST be 0 + // and the (unmodelled) style-sheet default font applies — leave unset. + const auto ftc = static_cast(read_u16(operand, 0)); + if (ftc < 0 || (font_names.empty() ? ftc != 0 + : static_cast(ftc) >= + font_names.size())) { + throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); + } + if (!font_names.empty()) { + style.font_name = font_names[static_cast(ftc)]; + } + } break; + case sprmCCv: { + // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. + if (static_cast(operand[3]) == 0xFF) { + style.font_color = std::nullopt; // cvAuto + } else { + style.font_color = Color(static_cast(operand[0]), + static_cast(operand[1]), + static_cast(operand[2])); + } + } break; + default: + break; // not modelled + } + } + if (at != grpprl.size()) { + throw std::runtime_error("doc: grpprl does not hold whole Prls"); + } + + return style; +} + +std::vector text::read_font_names(std::istream &table_stream, + const FcLcb sttbf_ffn) { + std::vector result; + if (sttbf_ffn.lcb == 0) { + return result; + } + table_stream.seekg(sttbf_ffn.fc); + + // SttbfFfn is a non-extended STTB ([MS-DOC] 2.2.4, 2.9.286): u16 cData, + // u16 cbExtra (0), then per entry a u8 byte count and an FFN. + const auto c_data = util::byte_stream::read(table_stream); + if (c_data == 0xFFFF) { + throw std::runtime_error("doc: unexpected extended SttbfFfn"); + } + const auto cb_extra = util::byte_stream::read(table_stream); + + result.reserve(c_data); + for (std::uint16_t i = 0; i < c_data; ++i) { + const auto cch_data = util::byte_stream::read(table_stream); + if (cch_data < sizeof(FfnFixed)) { + throw std::runtime_error("doc: FFN too short"); + } + const auto ffn = util::byte_stream::read(table_stream); + (void)ffn; + + // xszFfn: null-terminated UTF-16 within the remaining bytes; an + // alternative name (xszAlt) may follow the terminator. + const std::size_t name_units = (cch_data - sizeof(FfnFixed)) / 2; + std::u16string name = read_string_uncompressed(table_stream, name_units); + if ((cch_data - sizeof(FfnFixed)) % 2 != 0) { + table_stream.ignore(1); + } + if (const std::size_t nul = name.find(u'\0'); nul != std::u16string::npos) { + name.resize(nul); + } + result.push_back(util::string::u16string_to_string(name)); + + table_stream.ignore(cb_extra); + } + + return result; +} + +} // namespace odr::internal::oldms diff --git a/src/odr/internal/oldms/text/doc_style.hpp b/src/odr/internal/oldms/text/doc_style.hpp new file mode 100644 index 00000000..be1dd2da --- /dev/null +++ b/src/odr/internal/oldms/text/doc_style.hpp @@ -0,0 +1,46 @@ +#pragma once + +#include + +#include + +#include +#include +#include +#include +#include +#include + +namespace odr::internal::oldms::text { + +/// Owns the document's resolved character styles — indexed by the style index +/// stored on paragraph/span elements, 0 being the default style — and the +/// font names `TextStyle::font_name` points into. +class StyleRegistry final { +public: + /// Stores a font name and returns a pointer that stays valid for the + /// registry's lifetime (`TextStyle::font_name` is a `const char *`). + const char *intern_font_name(const std::string &name); + + std::uint32_t add_style(TextStyle style); + /// Throws if the index has no style. + [[nodiscard]] const TextStyle &text_style(std::uint32_t index) const; + +private: + /// Deque: elements never move, so interned `c_str()`s stay valid. + std::deque m_font_names; + std::vector m_styles; +}; + +/// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of +/// `style`; non-character SPRMs are skipped via their operand size. +/// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). +TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, + const std::vector &font_names); + +/// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at +/// `fc` in the table stream; empty for lcb == 0. +std::vector read_font_names(std::istream &table_stream, + FcLcb sttbf_ffn); + +} // namespace odr::internal::oldms::text diff --git a/test/src/internal/oldms/doc_test.cpp b/test/src/internal/oldms/doc_test.cpp index b3e95a0a..b4b8f3ce 100644 --- a/test/src/internal/oldms/doc_test.cpp +++ b/test/src/internal/oldms/doc_test.cpp @@ -8,8 +8,8 @@ #include #include #include -#include #include +#include #include From 46d3a07824aa9f3c5beafba2b8e5752cbf5e2dcd Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Tue, 21 Jul 2026 20:33:49 +0200 Subject: [PATCH 4/7] refactor(doc): pass font names as std::span Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PvbJYsLLST3hkCgkcHBoFb --- src/odr/internal/oldms/text/doc_helper.cpp | 2 +- src/odr/internal/oldms/text/doc_helper.hpp | 3 ++- src/odr/internal/oldms/text/doc_style.cpp | 17 ++++++++--------- src/odr/internal/oldms/text/doc_style.hpp | 3 ++- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 6878eff5..3bd4d0a6 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -78,7 +78,7 @@ text::CharacterRuns text::read_character_runs(std::istream &document_stream, std::istream &table_stream, const FcLcb plcf_bte_chpx, StyleRegistry &style_registry, - const std::vector &font_names) { + const std::span font_names) { CharacterRuns result; if (plcf_bte_chpx.lcb == 0) { return result; diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index 6420d2ce..da9bac90 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -144,6 +145,6 @@ CharacterRuns read_character_runs(std::istream &document_stream, std::istream &table_stream, FcLcb plcf_bte_chpx, StyleRegistry &style_registry, - const std::vector &font_names); + std::span font_names); } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_style.cpp b/src/odr/internal/oldms/text/doc_style.cpp index 03d3ad93..1cedd33b 100644 --- a/src/odr/internal/oldms/text/doc_style.cpp +++ b/src/odr/internal/oldms/text/doc_style.cpp @@ -54,11 +54,7 @@ std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { } // namespace -} // namespace odr::internal::oldms::text - -namespace odr::internal::oldms { - -const char *text::StyleRegistry::intern_font_name(const std::string &name) { +const char *StyleRegistry::intern_font_name(const std::string &name) { if (const auto it = std::ranges::find(m_font_names, name); it != m_font_names.end()) { return it->c_str(); @@ -66,19 +62,22 @@ const char *text::StyleRegistry::intern_font_name(const std::string &name) { return m_font_names.emplace_back(name).c_str(); } -std::uint32_t text::StyleRegistry::add_style(TextStyle style) { +std::uint32_t StyleRegistry::add_style(TextStyle style) { m_styles.push_back(std::move(style)); return static_cast(m_styles.size() - 1); } -const TextStyle & -text::StyleRegistry::text_style(const std::uint32_t index) const { +const TextStyle &StyleRegistry::text_style(const std::uint32_t index) const { return m_styles.at(index); } +} // namespace odr::internal::oldms::text + +namespace odr::internal::oldms { + TextStyle text::apply_character_sprms(TextStyle style, const std::string_view grpprl, - const std::vector &font_names) { + const std::span font_names) { std::size_t at = 0; while (at + sizeof(Sprm) <= grpprl.size()) { Sprm sprm; diff --git a/src/odr/internal/oldms/text/doc_style.hpp b/src/odr/internal/oldms/text/doc_style.hpp index be1dd2da..acfb0401 100644 --- a/src/odr/internal/oldms/text/doc_style.hpp +++ b/src/odr/internal/oldms/text/doc_style.hpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -36,7 +37,7 @@ class StyleRegistry final { /// `style`; non-character SPRMs are skipped via their operand size. /// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, - const std::vector &font_names); + std::span font_names); /// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at /// `fc` in the table stream; empty for lcb == 0. From f22459e1fea83b7b51ee39df15613040ad846c32 Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Tue, 21 Jul 2026 21:03:37 +0200 Subject: [PATCH 5/7] refactor(doc): store font names once, drop the interning deque Mirror the xls StyleRegistry: set_font_names stores the SttbfFfn table in a vector that is never modified afterwards, so the font-name strings cannot move; apply_character_sprms reads them as a span and read_character_runs takes the fonts from the registry it already gets. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PvbJYsLLST3hkCgkcHBoFb --- src/odr/internal/oldms/text/AGENTS.md | 5 +++-- src/odr/internal/oldms/text/doc_helper.cpp | 13 ++++++------- src/odr/internal/oldms/text/doc_helper.hpp | 7 +++---- src/odr/internal/oldms/text/doc_parser.cpp | 15 ++++++--------- src/odr/internal/oldms/text/doc_style.cpp | 18 +++++++++--------- src/odr/internal/oldms/text/doc_style.hpp | 19 +++++++++++-------- test/src/internal/oldms/doc_test.cpp | 4 ++-- 7 files changed, 40 insertions(+), 41 deletions(-) diff --git a/src/odr/internal/oldms/text/AGENTS.md b/src/odr/internal/oldms/text/AGENTS.md index d05ba159..af795665 100644 --- a/src/odr/internal/oldms/text/AGENTS.md +++ b/src/odr/internal/oldms/text/AGENTS.md @@ -82,8 +82,9 @@ style. The non-obvious bits: `sprmCIco`/`sprmCHighlight` use the `Ico` palette (§2.9.119) — the spec's extracted table repeats `0x0C` for `0x0D`, which is dark red (`0x800000`). - **`Pcd.Prm` modifications and STSH styles are not applied** (open work §1). -- Font names from `SttbfFfn` are interned in a `StyleRegistry` `std::deque` so - `TextStyle::font_name` (`const char *`) stays valid. +- Font names from `SttbfFfn` are stored once in the `StyleRegistry` (never + modified afterwards, like the xls module) so `TextStyle::font_name` + (`const char *`) stays valid. `Document::is_editable()` → `false`; `save`/`text_set_content` throw. diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 3bd4d0a6..22348a92 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -74,11 +74,10 @@ std::uint32_t text::CharacterRuns::chunk_end(const std::uint32_t fc) const { return std::numeric_limits::max(); } -text::CharacterRuns -text::read_character_runs(std::istream &document_stream, - std::istream &table_stream, const FcLcb plcf_bte_chpx, - StyleRegistry &style_registry, - const std::span font_names) { +text::CharacterRuns text::read_character_runs(std::istream &document_stream, + std::istream &table_stream, + const FcLcb plcf_bte_chpx, + StyleRegistry &style_registry) { CharacterRuns result; if (plcf_bte_chpx.lcb == 0) { return result; @@ -101,8 +100,8 @@ text::read_character_runs(std::istream &document_stream, if (it != style_cache.end()) { return it->second; } - const std::uint32_t index = style_registry.add_style( - apply_character_sprms(default_style, grpprl, font_names)); + const std::uint32_t index = style_registry.add_style(apply_character_sprms( + default_style, grpprl, style_registry.font_names())); style_cache.emplace(std::string(grpprl), index); return index; }; diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index da9bac90..29ca5a8d 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -5,7 +5,6 @@ #include #include #include -#include #include #include @@ -140,11 +139,11 @@ class CharacterRuns final { /// Reads the PlcBteChpx at `fc` in the table stream and every referenced /// ChpxFkp page ([MS-DOC] 2.9.33) from the WordDocument stream, resolving /// each run's Chpx on top of the default style (index 0) into -/// `style_registry`. Equal Chpx bytes share one style. +/// `style_registry` (fonts via its font names). Equal Chpx bytes share one +/// style. CharacterRuns read_character_runs(std::istream &document_stream, std::istream &table_stream, FcLcb plcf_bte_chpx, - StyleRegistry &style_registry, - std::span font_names); + StyleRegistry &style_registry); } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_parser.cpp b/src/odr/internal/oldms/text/doc_parser.cpp index cfeeb7b9..376cb83b 100644 --- a/src/odr/internal/oldms/text/doc_parser.cpp +++ b/src/odr/internal/oldms/text/doc_parser.cpp @@ -169,21 +169,18 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, fib.base.fWhichTblStm == 1 ? "/1Table" : "/0Table"; const auto table_stream = files.open(AbsPath(tableStreamPath))->stream(); - // Font table, interned so TextStyle::font_name pointers stay valid. - std::vector font_names; - for (const std::string &name : - read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)) { - font_names.push_back(style_registry.intern_font_name(name)); - } + // Font table; TextStyle::font_name points into the registry's copy. + style_registry.set_font_names( + read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)); // Direct character formatting ([MS-DOC] 2.4.6.2); Pcd.Prm modifications are // not modelled. Without sprmCHps the size defaults to 20 half-points. TextStyle default_style; default_style.font_size = Measure(10.0, DynamicUnit("pt")); style_registry.add_style(std::move(default_style)); // index 0 - const CharacterRuns character_runs = read_character_runs( - *document_stream, *table_stream, fib.fibRgFcLcb->plcfBteChpx, - style_registry, font_names); + const CharacterRuns character_runs = + read_character_runs(*document_stream, *table_stream, + fib.fibRgFcLcb->plcfBteChpx, style_registry); table_stream->seekg(fib.fibRgFcLcb->clx.fc); const CharacterIndex character_index = read_character_index(*table_stream); diff --git a/src/odr/internal/oldms/text/doc_style.cpp b/src/odr/internal/oldms/text/doc_style.cpp index 1cedd33b..f68d0188 100644 --- a/src/odr/internal/oldms/text/doc_style.cpp +++ b/src/odr/internal/oldms/text/doc_style.cpp @@ -4,12 +4,12 @@ #include #include -#include #include #include #include #include #include +#include namespace odr::internal::oldms::text { @@ -54,12 +54,12 @@ std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { } // namespace -const char *StyleRegistry::intern_font_name(const std::string &name) { - if (const auto it = std::ranges::find(m_font_names, name); - it != m_font_names.end()) { - return it->c_str(); - } - return m_font_names.emplace_back(name).c_str(); +void StyleRegistry::set_font_names(std::vector names) { + m_font_names = std::move(names); +} + +std::span StyleRegistry::font_names() const { + return m_font_names; } std::uint32_t StyleRegistry::add_style(TextStyle style) { @@ -77,7 +77,7 @@ namespace odr::internal::oldms { TextStyle text::apply_character_sprms(TextStyle style, const std::string_view grpprl, - const std::span font_names) { + const std::span font_names) { std::size_t at = 0; while (at + sizeof(Sprm) <= grpprl.size()) { Sprm sprm; @@ -148,7 +148,7 @@ text::apply_character_sprms(TextStyle style, const std::string_view grpprl, throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); } if (!font_names.empty()) { - style.font_name = font_names[static_cast(ftc)]; + style.font_name = font_names[static_cast(ftc)].c_str(); } } break; case sprmCCv: { diff --git a/src/odr/internal/oldms/text/doc_style.hpp b/src/odr/internal/oldms/text/doc_style.hpp index acfb0401..e1798e06 100644 --- a/src/odr/internal/oldms/text/doc_style.hpp +++ b/src/odr/internal/oldms/text/doc_style.hpp @@ -5,7 +5,6 @@ #include #include -#include #include #include #include @@ -19,25 +18,29 @@ namespace odr::internal::oldms::text { /// font names `TextStyle::font_name` points into. class StyleRegistry final { public: - /// Stores a font name and returns a pointer that stays valid for the - /// registry's lifetime (`TextStyle::font_name` is a `const char *`). - const char *intern_font_name(const std::string &name); + /// Stores the font table (SttbfFfn names); must be set before any style + /// referencing a font is added. + void set_font_names(std::vector names); + [[nodiscard]] std::span font_names() const; std::uint32_t add_style(TextStyle style); /// Throws if the index has no style. [[nodiscard]] const TextStyle &text_style(std::uint32_t index) const; private: - /// Deque: elements never move, so interned `c_str()`s stay valid. - std::deque m_font_names; + /// Owns the font names: `TextStyle::font_name` (`const char *`) points into + /// them. Never modified after `set_font_names` (moving the registry is fine + /// — the strings themselves do not move). + std::vector m_font_names; std::vector m_styles; }; /// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of /// `style`; non-character SPRMs are skipped via their operand size. -/// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). +/// `font_names` resolves sprmCRgFtc0 (the strings must outlive the style — +/// `TextStyle::font_name` points into them). TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, - std::span font_names); + std::span font_names); /// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at /// `fc` in the table stream; empty for lcb == 0. diff --git a/test/src/internal/oldms/doc_test.cpp b/test/src/internal/oldms/doc_test.cpp index b4b8f3ce..27ffc9bc 100644 --- a/test/src/internal/oldms/doc_test.cpp +++ b/test/src/internal/oldms/doc_test.cpp @@ -151,7 +151,7 @@ TEST(OldMs, doc_apply_character_sprms) { TextStyle base; base.font_size = Measure("10pt"); - const std::vector fonts = {"Arial", "Courier New"}; + const std::vector fonts = {"Arial", "Courier New"}; { std::string grpprl; @@ -202,7 +202,7 @@ TEST(OldMs, doc_apply_character_sprms) { { // sprmCRgFtc0 with an empty SttbfFfn: 0 is valid ([MS-DOC] 2.6.1) and // leaves the font unset; anything else is out of range. - const std::vector no_fonts; + const std::vector no_fonts; const TextStyle style = apply_character_sprms( base, doc_prl(0x4A4F, std::string("\x00\x00", 2)), no_fonts); EXPECT_EQ(style.font_name, nullptr); From a108cf5580efc00689c4321982012705982608ea Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Tue, 21 Jul 2026 23:44:09 +0200 Subject: [PATCH 6/7] refactor(doc): make the style registry immutable after construction StyleRegistry now takes its font names and resolved styles in the constructor (mirroring the xls registry); set_font_names/add_style are gone, read_character_runs collects styles into a plain vector and no longer depends on the registry, and the default style moved into a named default_character_style() factory. Also review cleanups across the branch: - emplace_back for aggregate push_backs (C++20 parenthesized init) - intern cache: single try_emplace instead of find + emplace with a second std::string(grpprl) allocation - fixed-size integer types instead of plain int - drop the local read_u16 in favor of util::byte::from_little_endian - shared append_u16/append_u32 test helpers in oldms_test_util.hpp - doc_helper.cpp: definitions moved into the namespace block Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01WJRDZcksAtKnXyhDTeMkMK --- .../internal/oldms/spreadsheet/xls_parser.cpp | 2 +- src/odr/internal/oldms/text/doc_helper.cpp | 92 ++++++++++--------- src/odr/internal/oldms/text/doc_helper.hpp | 19 ++-- src/odr/internal/oldms/text/doc_parser.cpp | 20 ++-- src/odr/internal/oldms/text/doc_structs.hpp | 2 +- src/odr/internal/oldms/text/doc_style.cpp | 37 ++++---- src/odr/internal/oldms/text/doc_style.hpp | 21 +++-- test/src/internal/oldms/doc_test.cpp | 72 +++++++-------- test/src/internal/oldms/oldms_test_util.hpp | 12 +++ test/src/internal/oldms/xls_test.cpp | 8 +- 10 files changed, 146 insertions(+), 139 deletions(-) diff --git a/src/odr/internal/oldms/spreadsheet/xls_parser.cpp b/src/odr/internal/oldms/spreadsheet/xls_parser.cpp index dd54ba3d..5e421202 100644 --- a/src/odr/internal/oldms/spreadsheet/xls_parser.cpp +++ b/src/odr/internal/oldms/spreadsheet/xls_parser.cpp @@ -67,7 +67,7 @@ void parse_globals(BiffReader &reader, std::vector &sheets, case biff_font: { const auto font = reader.read(); std::string name = reader.read_short_xl_unicode_string(); - styles.fonts.push_back({font, std::move(name)}); + styles.fonts.emplace_back(font, std::move(name)); } break; case biff_xf: { styles.xfs.push_back(reader.read()); diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 22348a92..f4668240 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -13,35 +13,11 @@ #include #include -namespace odr::internal::oldms { - -text::CharacterIndex text::read_character_index(std::istream &in) { - CharacterIndex result; - - read_Clx(in, skip_Prc, [&](std::istream &) { - if (const int c = in.get(); c != 0x2) { - throw std::runtime_error("Unexpected input: " + std::to_string(c)); - } - const std::uint32_t lcb = util::byte_stream::read(in); - std::string plcPcd = util::stream::read(in, lcb); - const PlcPcdMap plc_pcd_map(plcPcd.data(), plcPcd.size()); - - for (std::uint32_t i = 0; i < plc_pcd_map.n(); ++i) { - const bool is_compressed = plc_pcd_map.aData(i).fc.fCompressed != 0; - const std::size_t data_offset = is_compressed - ? plc_pcd_map.aData(i).fc.fc / 2 - : plc_pcd_map.aData(i).fc.fc; - const std::size_t length_cp = plc_pcd_map.aCP(i + 1) - plc_pcd_map.aCP(i); - result.append(plc_pcd_map.aCP(i), length_cp, data_offset, is_compressed); - } - }); - - return result; -} +namespace odr::internal::oldms::text { -void text::CharacterRuns::append_run(const std::uint32_t begin_fc, - const std::uint32_t end_fc, - const std::uint32_t style_index) { +void CharacterRuns::append_run(const std::uint32_t begin_fc, + const std::uint32_t end_fc, + const std::uint32_t style_index) { if (begin_fc >= end_fc || (!m_runs.empty() && begin_fc < m_runs.back().end_fc)) { throw std::runtime_error("doc: character runs must be ascending"); @@ -51,10 +27,10 @@ void text::CharacterRuns::append_run(const std::uint32_t begin_fc, m_runs.back().end_fc = end_fc; return; } - m_runs.push_back({begin_fc, end_fc, style_index}); + m_runs.emplace_back(begin_fc, end_fc, style_index); } -std::uint32_t text::CharacterRuns::index_at(const std::uint32_t fc) const { +std::uint32_t CharacterRuns::index_at(const std::uint32_t fc) const { const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); if (it == m_runs.begin()) { return 0; @@ -63,7 +39,7 @@ std::uint32_t text::CharacterRuns::index_at(const std::uint32_t fc) const { return fc < run.end_fc ? run.style_index : 0; } -std::uint32_t text::CharacterRuns::chunk_end(const std::uint32_t fc) const { +std::uint32_t CharacterRuns::chunk_end(const std::uint32_t fc) const { const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); if (it != m_runs.begin() && fc < std::prev(it)->end_fc) { return std::prev(it)->end_fc; @@ -74,17 +50,46 @@ std::uint32_t text::CharacterRuns::chunk_end(const std::uint32_t fc) const { return std::numeric_limits::max(); } -text::CharacterRuns text::read_character_runs(std::istream &document_stream, - std::istream &table_stream, - const FcLcb plcf_bte_chpx, - StyleRegistry &style_registry) { +} // namespace odr::internal::oldms::text + +namespace odr::internal::oldms { + +text::CharacterIndex text::read_character_index(std::istream &in) { + CharacterIndex result; + + read_Clx(in, skip_Prc, [&](std::istream &) { + if (const int c = in.get(); c != 0x2) { + throw std::runtime_error("Unexpected input: " + std::to_string(c)); + } + const std::uint32_t lcb = util::byte_stream::read(in); + std::string plcPcd = util::stream::read(in, lcb); + const PlcPcdMap plc_pcd_map(plcPcd.data(), plcPcd.size()); + + for (std::uint32_t i = 0; i < plc_pcd_map.n(); ++i) { + const bool is_compressed = plc_pcd_map.aData(i).fc.fCompressed != 0; + const std::size_t data_offset = is_compressed + ? plc_pcd_map.aData(i).fc.fc / 2 + : plc_pcd_map.aData(i).fc.fc; + const std::size_t length_cp = plc_pcd_map.aCP(i + 1) - plc_pcd_map.aCP(i); + result.append(plc_pcd_map.aCP(i), length_cp, data_offset, is_compressed); + } + }); + + return result; +} + +text::CharacterRuns +text::read_character_runs(std::istream &document_stream, + std::istream &table_stream, const FcLcb plcf_bte_chpx, + std::vector &styles, + const std::span font_names) { CharacterRuns result; if (plcf_bte_chpx.lcb == 0) { return result; } - // Copy: `add_style` may reallocate the registry's style storage. - const TextStyle default_style = style_registry.text_style(0); + // Copy: appending to `styles` may reallocate. + const TextStyle default_style = styles.at(0); table_stream.seekg(plcf_bte_chpx.fc); std::string plc_bytes = util::stream::read(table_stream, plcf_bte_chpx.lcb); @@ -96,14 +101,13 @@ text::CharacterRuns text::read_character_runs(std::istream &document_stream, if (grpprl.empty()) { return 0; } - const auto it = style_cache.find(std::string(grpprl)); - if (it != style_cache.end()) { - return it->second; + const auto [it, inserted] = style_cache.try_emplace(std::string(grpprl)); + if (inserted) { + styles.push_back( + apply_character_sprms(default_style, grpprl, font_names)); + it->second = static_cast(styles.size() - 1); } - const std::uint32_t index = style_registry.add_style(apply_character_sprms( - default_style, grpprl, style_registry.font_names())); - style_cache.emplace(std::string(grpprl), index); - return index; + return it->second; }; for (std::uint32_t i = 0; i < plc.n(); ++i) { diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index 29ca5a8d..9087c654 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -1,17 +1,19 @@ #pragma once +#include + #include #include #include #include +#include #include +#include #include namespace odr::internal::oldms::text { -class StyleRegistry; - class CharacterIndex { public: [[nodiscard]] bool empty() const { return m_entries.empty(); } @@ -111,8 +113,8 @@ class CharacterIndex { CharacterIndex read_character_index(std::istream &in); /// The document's character-formatting runs, keyed by WordDocument-stream -/// offset (fc), each referencing a `StyleRegistry` style index. Index 0 is -/// the default style; offsets outside any run resolve to it. +/// offset (fc), each referencing a style index. Index 0 is the default style; +/// offsets outside any run resolve to it. class CharacterRuns final { public: /// Appends a run; runs must be ascending and non-overlapping. Adjacent runs @@ -138,12 +140,13 @@ class CharacterRuns final { /// Reads the PlcBteChpx at `fc` in the table stream and every referenced /// ChpxFkp page ([MS-DOC] 2.9.33) from the WordDocument stream, resolving -/// each run's Chpx on top of the default style (index 0) into -/// `style_registry` (fonts via its font names). Equal Chpx bytes share one -/// style. +/// each run's Chpx on top of `styles[0]` (the default style) and appending +/// each distinct resolved style to `styles` (fonts via `font_names`). Equal +/// Chpx bytes share one style. CharacterRuns read_character_runs(std::istream &document_stream, std::istream &table_stream, FcLcb plcf_bte_chpx, - StyleRegistry &style_registry); + std::vector &styles, + std::span font_names); } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_parser.cpp b/src/odr/internal/oldms/text/doc_parser.cpp index 376cb83b..732f58d8 100644 --- a/src/odr/internal/oldms/text/doc_parser.cpp +++ b/src/odr/internal/oldms/text/doc_parser.cpp @@ -96,7 +96,7 @@ class TextCleaner { // Open fields; entry is true while still in that field's instruction part. // m_instruction_depth counts those; text is hidden whenever it is > 0. std::vector m_field_in_instruction; - int m_instruction_depth{0}; + std::int32_t m_instruction_depth{0}; }; /// A maximal piece of body text with uniform character formatting. @@ -143,7 +143,7 @@ std::vector decode_styled_runs( if (!runs.empty() && runs.back().style_index == style_index) { runs.back().text += chunk_text; } else { - runs.push_back({std::move(chunk_text), style_index}); + runs.emplace_back(std::move(chunk_text), style_index); } cp += chunk_cp; } @@ -169,18 +169,18 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, fib.base.fWhichTblStm == 1 ? "/1Table" : "/0Table"; const auto table_stream = files.open(AbsPath(tableStreamPath))->stream(); - // Font table; TextStyle::font_name points into the registry's copy. - style_registry.set_font_names( - read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)); + // Font table; TextStyle::font_name points into the strings, which keep + // their buffers when the vector is moved into the registry below. + std::vector font_names = + read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn); // Direct character formatting ([MS-DOC] 2.4.6.2); Pcd.Prm modifications are - // not modelled. Without sprmCHps the size defaults to 20 half-points. - TextStyle default_style; - default_style.font_size = Measure(10.0, DynamicUnit("pt")); - style_registry.add_style(std::move(default_style)); // index 0 + // not modelled. + std::vector styles{default_character_style()}; // index 0 const CharacterRuns character_runs = read_character_runs(*document_stream, *table_stream, - fib.fibRgFcLcb->plcfBteChpx, style_registry); + fib.fibRgFcLcb->plcfBteChpx, styles, font_names); + style_registry = StyleRegistry(std::move(font_names), std::move(styles)); table_stream->seekg(fib.fibRgFcLcb->clx.fc); const CharacterIndex character_index = read_character_index(*table_stream); diff --git a/src/odr/internal/oldms/text/doc_structs.hpp b/src/odr/internal/oldms/text/doc_structs.hpp index 8731f822..0f806924 100644 --- a/src/odr/internal/oldms/text/doc_structs.hpp +++ b/src/odr/internal/oldms/text/doc_structs.hpp @@ -308,7 +308,7 @@ struct Sprm { std::uint16_t sgc : 3; std::uint16_t spra : 3; - [[nodiscard]] int operand_size() const { + [[nodiscard]] std::int32_t operand_size() const { switch (spra) { case 0: case 1: diff --git a/src/odr/internal/oldms/text/doc_style.cpp b/src/odr/internal/oldms/text/doc_style.cpp index f68d0188..17799bda 100644 --- a/src/odr/internal/oldms/text/doc_style.cpp +++ b/src/odr/internal/oldms/text/doc_style.cpp @@ -2,10 +2,12 @@ #include #include +#include #include #include #include +#include #include #include #include @@ -46,31 +48,22 @@ bool toggle_on(const std::uint8_t value) { } } -std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { - std::uint16_t value; - std::memcpy(&value, bytes.data() + at, sizeof(value)); - return value; -} - } // namespace -void StyleRegistry::set_font_names(std::vector names) { - m_font_names = std::move(names); -} - -std::span StyleRegistry::font_names() const { - return m_font_names; -} - -std::uint32_t StyleRegistry::add_style(TextStyle style) { - m_styles.push_back(std::move(style)); - return static_cast(m_styles.size() - 1); -} +StyleRegistry::StyleRegistry(std::vector font_names, + std::vector styles) + : m_font_names(std::move(font_names)), m_styles(std::move(styles)) {} const TextStyle &StyleRegistry::text_style(const std::uint32_t index) const { return m_styles.at(index); } +TextStyle default_character_style() { + TextStyle style; + style.font_size = Measure(10.0, DynamicUnit("pt")); + return style; +} + } // namespace odr::internal::oldms::text namespace odr::internal::oldms { @@ -86,7 +79,7 @@ text::apply_character_sprms(TextStyle style, const std::string_view grpprl, at += sizeof(sprm); std::size_t operand_size; - if (const int fixed_size = sprm.operand_size(); fixed_size >= 0) { + if (const std::int32_t fixed_size = sprm.operand_size(); fixed_size >= 0) { operand_size = static_cast(fixed_size); } else { // spra == 6: the first operand byte is the size of the remainder. The @@ -132,7 +125,8 @@ text::apply_character_sprms(TextStyle style, const std::string_view grpprl, style.font_color = ico_color(static_cast(operand[0])); break; case sprmCHps: { - const std::uint16_t half_points = read_u16(operand, 0); + const auto half_points = + util::byte::from_little_endian(operand); if (half_points < 2 || half_points > 3276) { throw std::runtime_error("doc: sprmCHps value out of range"); } @@ -141,7 +135,8 @@ text::apply_character_sprms(TextStyle style, const std::string_view grpprl, case sprmCRgFtc0: { // SttbfFfn index ([MS-DOC] 2.6.1); with 0 entries the value MUST be 0 // and the (unmodelled) style-sheet default font applies — leave unset. - const auto ftc = static_cast(read_u16(operand, 0)); + const auto ftc = static_cast( + util::byte::from_little_endian(operand)); if (ftc < 0 || (font_names.empty() ? ftc != 0 : static_cast(ftc) >= font_names.size())) { diff --git a/src/odr/internal/oldms/text/doc_style.hpp b/src/odr/internal/oldms/text/doc_style.hpp index e1798e06..b898dd2a 100644 --- a/src/odr/internal/oldms/text/doc_style.hpp +++ b/src/odr/internal/oldms/text/doc_style.hpp @@ -15,26 +15,31 @@ namespace odr::internal::oldms::text { /// Owns the document's resolved character styles — indexed by the style index /// stored on paragraph/span elements, 0 being the default style — and the -/// font names `TextStyle::font_name` points into. +/// font names `TextStyle::font_name` points into. Immutable after +/// construction. class StyleRegistry final { public: - /// Stores the font table (SttbfFfn names); must be set before any style - /// referencing a font is added. - void set_font_names(std::vector names); - [[nodiscard]] std::span font_names() const; + StyleRegistry() = default; + /// `font_names` are the SttbfFfn names the styles' `font_name` point into; + /// `styles` are the resolved character styles, index 0 the default style. + StyleRegistry(std::vector font_names, + std::vector styles); - std::uint32_t add_style(TextStyle style); /// Throws if the index has no style. [[nodiscard]] const TextStyle &text_style(std::uint32_t index) const; private: /// Owns the font names: `TextStyle::font_name` (`const char *`) points into - /// them. Never modified after `set_font_names` (moving the registry is fine - /// — the strings themselves do not move). + /// them. Never modified after construction (moving the registry is fine — + /// the strings themselves do not move). std::vector m_font_names; std::vector m_styles; }; +/// The base character style Chpx grpprls apply on top of; without sprmCHps +/// the font size defaults to 20 half-points ([MS-DOC] 2.6.1 sprmCHps). +[[nodiscard]] TextStyle default_character_style(); + /// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of /// `style`; non-character SPRMs are skipped via their operand size. /// `font_names` resolves sprmCRgFtc0 (the strings must outlive the style — diff --git a/test/src/internal/oldms/doc_test.cpp b/test/src/internal/oldms/doc_test.cpp index 27ffc9bc..b5cb24e9 100644 --- a/test/src/internal/oldms/doc_test.cpp +++ b/test/src/internal/oldms/doc_test.cpp @@ -20,6 +20,8 @@ #include using namespace odr; +using odr::test::oldms::append_u16; +using odr::test::oldms::append_u32; using odr::test::oldms::collect_text; // A compressed (1-byte-per-CP) piece is "an array of 8-bit Unicode characters" @@ -59,19 +61,9 @@ TEST(OldMs, doc_read_string_compressed) { namespace { -void append_doc_u16(std::string &out, const std::uint16_t value) { - out.push_back(static_cast(value & 0xFF)); - out.push_back(static_cast(value >> 8)); -} - -void append_doc_u32(std::string &out, const std::uint32_t value) { - append_doc_u16(out, static_cast(value & 0xFFFF)); - append_doc_u16(out, static_cast(value >> 16)); -} - std::string doc_prl(const std::uint16_t opcode, const std::string &operand) { std::string prl; - append_doc_u16(prl, opcode); + append_u16(prl, opcode); prl += operand; return prl; } @@ -84,35 +76,35 @@ std::string make_fib(const std::uint32_t ccp_text, const internal::oldms::text::FcLcb sttbf_ffn) { std::string fib; // FibBase (32 bytes) - append_doc_u16(fib, 0xA5EC); // wIdent - append_doc_u16(fib, 0x00C1); // nFib97 - append_doc_u16(fib, 0); // unused - append_doc_u16(fib, 0x0409); // lid - append_doc_u16(fib, 0); // pnNext - append_doc_u16(fib, 0x0200); // flags: fWhichTblStm = 1 - append_doc_u16(fib, 0x00C1); // nFibBack - append_doc_u32(fib, 0); // lKey + append_u16(fib, 0xA5EC); // wIdent + append_u16(fib, 0x00C1); // nFib97 + append_u16(fib, 0); // unused + append_u16(fib, 0x0409); // lid + append_u16(fib, 0); // pnNext + append_u16(fib, 0x0200); // flags: fWhichTblStm = 1 + append_u16(fib, 0x00C1); // nFibBack + append_u32(fib, 0); // lKey fib += std::string(2, '\0'); // envr, flags fib += std::string(12, '\0'); // reserved3-6 // csw + fibRgW - append_doc_u16(fib, 14); + append_u16(fib, 14); fib += std::string(28, '\0'); // cslw + fibRgLw; ccpText is the 4th 32-bit field - append_doc_u16(fib, 22); + append_u16(fib, 22); std::string rg_lw(88, '\0'); std::string ccp; - append_doc_u32(ccp, ccp_text); + append_u32(ccp, ccp_text); rg_lw.replace(12, 4, ccp); fib += rg_lw; // cbRgFcLcb + FibRgFcLcb97: plcfBteChpx is FcLcb index 12, sttbfFfn 15, // clx 33 - append_doc_u16(fib, 93); + append_u16(fib, 93); std::string rg(744, '\0'); const auto put_fclcb = [&rg](const std::size_t index, const internal::oldms::text::FcLcb value) { std::string bytes; - append_doc_u32(bytes, value.fc); - append_doc_u32(bytes, value.lcb); + append_u32(bytes, value.fc); + append_u32(bytes, value.lcb); rg.replace(index * 8, 8, bytes); }; put_fclcb(12, plcf_bte_chpx); @@ -120,15 +112,15 @@ std::string make_fib(const std::uint32_t ccp_text, put_fclcb(33, clx); fib += rg; // cswNew - append_doc_u16(fib, 0); + append_u16(fib, 0); return fib; } /// An SttbfFfn ([MS-DOC] 2.9.286) with the given font names. std::string make_sttbf_ffn(const std::vector &names) { std::string sttb; - append_doc_u16(sttb, static_cast(names.size())); // cData - append_doc_u16(sttb, 0); // cbExtra + append_u16(sttb, static_cast(names.size())); // cData + append_u16(sttb, 0); // cbExtra for (const std::string &name : names) { std::string ffn(39, '\0'); // FfnFixed for (const char c : name) { @@ -252,9 +244,9 @@ TEST(OldMs, doc_character_formatting) { std::string fkp(512, '\0'); { std::string head; - append_doc_u32(head, text_fc); - append_doc_u32(head, text_fc + 6); - append_doc_u32(head, text_fc + 11); + append_u32(head, text_fc); + append_u32(head, text_fc + 6); + append_u32(head, text_fc + 11); head.push_back('\0'); // rgb[0]: default properties head.push_back(static_cast(240)); // rgb[1]: Chpx at byte 480 fkp.replace(0, head.size(), head); @@ -270,22 +262,22 @@ TEST(OldMs, doc_character_formatting) { // Table stream: Clx at 0 (one compressed piece), SttbfFfn at 64, // PlcBteChpx at 128. std::string plc_pcd; - append_doc_u32(plc_pcd, 0); - append_doc_u32(plc_pcd, 11); - append_doc_u16(plc_pcd, 0); // Pcd flags - append_doc_u32(plc_pcd, (1u << 30) | (text_fc * 2u)); // fCompressed | fc - append_doc_u16(plc_pcd, 0); // prm + append_u32(plc_pcd, 0); + append_u32(plc_pcd, 11); + append_u16(plc_pcd, 0); // Pcd flags + append_u32(plc_pcd, (1u << 30) | (text_fc * 2u)); // fCompressed | fc + append_u16(plc_pcd, 0); // prm std::string clx; clx.push_back('\x02'); - append_doc_u32(clx, static_cast(plc_pcd.size())); + append_u32(clx, static_cast(plc_pcd.size())); clx += plc_pcd; const std::string sttbf_ffn = make_sttbf_ffn({"Arial"}); std::string plc_bte; - append_doc_u32(plc_bte, text_fc); - append_doc_u32(plc_bte, text_fc + 11); - append_doc_u32(plc_bte, 2); // PnFkpChpx: page at 2 * 512 + append_u32(plc_bte, text_fc); + append_u32(plc_bte, text_fc + 11); + append_u32(plc_bte, 2); // PnFkpChpx: page at 2 * 512 std::string table(256, '\0'); table.replace(0, clx.size(), clx); diff --git a/test/src/internal/oldms/oldms_test_util.hpp b/test/src/internal/oldms/oldms_test_util.hpp index 4cdc69fa..611d3d9b 100644 --- a/test/src/internal/oldms/oldms_test_util.hpp +++ b/test/src/internal/oldms/oldms_test_util.hpp @@ -2,10 +2,22 @@ #include +#include #include namespace odr::test::oldms { +/// Appends `value` little-endian, as the legacy binary formats store integers. +inline void append_u16(std::string &out, const std::uint16_t value) { + out.push_back(static_cast(value & 0xFF)); + out.push_back(static_cast(value >> 8)); +} + +inline void append_u32(std::string &out, const std::uint32_t value) { + append_u16(out, static_cast(value & 0xFFFF)); + append_u16(out, static_cast(value >> 16)); +} + /// Collects the text content of an element subtree, joining paragraphs with /// newlines so the paragraph structure is observable. inline std::string collect_text(const Element element) { diff --git a/test/src/internal/oldms/xls_test.cpp b/test/src/internal/oldms/xls_test.cpp index f0b41ff5..6c4eaff1 100644 --- a/test/src/internal/oldms/xls_test.cpp +++ b/test/src/internal/oldms/xls_test.cpp @@ -26,15 +26,11 @@ using namespace odr; using namespace odr::test; +using odr::test::oldms::append_u16; using odr::test::oldms::collect_text; namespace { -void append_u16(std::string &out, const std::uint16_t value) { - out.push_back(static_cast(value & 0xFF)); - out.push_back(static_cast(value >> 8)); -} - void append_record(std::string &out, const std::uint16_t type, const std::string &body) { append_u16(out, type); @@ -240,7 +236,7 @@ TEST(OldMs, xls_cell_styles) { make_font(320, 0x000A, 0x0011, 700, 1, "Comic Sans MS"); std::string globals; - for (int i = 0; i < 4; ++i) { + for (std::uint32_t i = 0; i < 4; ++i) { append_record(globals, 0x0031 /* Font */, plain_font); } append_record(globals, 0x0031 /* Font */, fancy_font); // ifnt 5 From d3b146b8d69075409d67910015045f8abc059a12 Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Tue, 21 Jul 2026 23:48:04 +0200 Subject: [PATCH 7/7] update refs --- test/data/reference-output/odr-public | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/data/reference-output/odr-public b/test/data/reference-output/odr-public index ea5173fc..eafd85eb 160000 --- a/test/data/reference-output/odr-public +++ b/test/data/reference-output/odr-public @@ -1 +1 @@ -Subproject commit ea5173fcdd5613269c0b4dd849c12f7e95e203b3 +Subproject commit eafd85eb4aaf6ece1ce76a36f614f5db514fa21f