feat(pdf): add to_markdown_pages for per-page extraction

[HaoChiBao]

Summary

• Expose MarkdownPage plus to_markdown_pages / to_markdown_pages_bytes wrapping pdf-inspector's per-page extraction (page boundaries, OCR flags)
• Keep PDF to_document unsupported; leave existing single-blob to_markdown / to_markdown_bytes behavior unchanged
• Document the new API in the README Rust/PDF sections

Closes #62

Why not to_document for PDF

PDFs intentionally bypass the document model (pdf-inspector emits Markdown directly). A thin per-page API is the low-risk path from the issue and does not race the OCR work in #61.

Test plan

• cargo fmt --all
• cargo clippy --locked --all-targets -- -D warnings
• cargo test --locked (fixture tests/fixtures/pdf/text.pdf)
• Optional follow-up: Node/Python/wasm bindings for the new API

---

Summary by cubic

Adds per-page Markdown extraction for PDFs so callers can keep page boundaries and detect OCR needs. Previously we only returned a single Markdown blob and errored on fully scanned PDFs; now to_markdown_pages/to_markdown_pages_bytes return one item per page with a needs_ocr flag, while single-blob APIs remain unchanged.

• Exposes MarkdownPage { page, markdown, needs_ocr, ocr_reason } and exports it from the crate.

• Keeps to_markdown/to_markdown_bytes behavior: still one string; still errors on image-only PDFs and logs when some pages need OCR.

• Adds to_markdown_pages/to_markdown_pages_bytes: never fail solely due to OCR; pages that need OCR have empty markdown and needs_ocr = true and may include ocr_reason. Logs a warn when any pages need OCR.

• Leaves to_document unsupported for Format::Pdf; updates the Unsupported message to mention to_markdown_pages.

• Updates README and adds a basic per-page test.

• Migration (optional): If you need page boundaries or to route OCR per page, switch PDF callers to to_markdown_pages/to_markdown_pages_bytes. Otherwise, keep using to_markdown/to_markdown_bytes.

^{Written for commit 9853fac. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/formats/pdf.rs">

<violation number="1" location="src/formats/pdf.rs:67">
P3: The OCR warning block added here duplicates the one already in `to_markdown` in the same file (same message and count structure, differing only in the denominator: `result.pages.len()` here vs `result.page_count` there). Extract a small shared helper, e.g. `fn warn_ocr_pages(need_ocr: usize, total: usize)`, and call it from both functions so the message and the OCR-recovery policy stay consistent in one place.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The OCR warning block added here duplicates the one already in to_markdown in the same file (same message and count structure, differing only in the denominator: result.pages.len() here vs result.page_count there). Extract a small shared helper, e.g. fn warn_ocr_pages(need_ocr: usize, total: usize), and call it from both functions so the message and the OCR-recovery policy stay consistent in one place.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/formats/pdf.rs, line 67:

<comment>The OCR warning block added here duplicates the one already in `to_markdown` in the same file (same message and count structure, differing only in the denominator: `result.pages.len()` here vs `result.page_count` there). Extract a small shared helper, e.g. `fn warn_ocr_pages(need_ocr: usize, total: usize)`, and call it from both functions so the message and the OCR-recovery policy stay consistent in one place.</comment>

<file context>
@@ -37,6 +56,33 @@ pub fn to_markdown(bytes: &[u8]) -> Result<String, ConvertError> {
+/// with an external OCR path.
+pub fn to_markdown_pages(bytes: &[u8]) -> Result<Vec<MarkdownPage>, ConvertError> {
+    let result = pdf_inspector::extract_pages_markdown_mem(bytes, None).map_err(map_error)?;
+    if !result.pages_needing_ocr.is_empty() {
+        log::warn!(
+            "{} of {} pages need OCR and were not extracted",
</file context>