WIP: feat(ocr): optional local OCR for scanned PDFs and image documents

[massimodeluisa]

WIP: this needs firecrawl/pdf-inspector#280 merged first, see the note at the bottom.

Hi! This is the follow-up of firecrawl/pdf-inspector#280. Now that the renderer exists over there, anydoc can finally do something with the pages pdf-inspector flags as needing OCR. Same premise as that PR, I'm still quite new to Rust, so if something looks wrong to you please point it out.

Why

Today a scanned PDF just errors with "unsupported input: PDF has no extractable text", and an image file is not a format anydoc accepts at all. But people throw these at a converter every day, and the frustrating part is that anydoc already knows page 2 is a scan, it just had no way to read it. With this PR you give the converter the two ocrs model files and it reads scanned and mixed PDFs, and PNG/JPEG/WebP/TIFF/BMP files become documents too. All local, you bring the models, the library never downloads anything.

For a concrete case I used 018-base64-image/base64image.pdf from the py-pdf corpus, a page that is just a screenshot. On main it errors. With an OCR converter it returns the text of the screenshot, and native and wasm return exactly the same characters.

Ps. It's the most common question on X reguarding this amazing library 😂.

What

An opt-in ocr cargo feature and a reusable Converter you build once from the model bytes and use for every conversion after that. The same converter exists in Rust, in Node (async, off the event loop) and in the browser wasm package (from Uint8Array). The free functions don't change and the default build doesn't even compile the OCR dependencies.

For PDFs I render and recognize only the pages that need it, one page at a time, through the new pdf-inspector render feature. A page with real text keeps the exact extraction output it had before, so a text PDF converts byte for byte the same as today. A mixed PDF today comes back partial (the scanned pages are just missing); now the recognized pages get merged with the extracted ones in source order. The recognized text stays plain text, I don't invent any structure out of it.

I wired the OCR in the demo page as well. Drop a scanned PDF or a photo of a page on it and the text shows up in the same preview a .docx does, in your browser. The models load only when a document actually needs them (12 MB, CC-BY-SA-4.0, fetched pinned by hash when the site deploys), so who converts a Word file never downloads them, and they never touch the crate.

Numbers

All numbers from my Apple M3 Pro, release profile. Engine init is ~21 ms cold and ~7 ms warm, and the 018 conversion above takes ~155 ms end to end. Peak RSS for one Letter page at 200 DPI is ~234 MB (that includes the RTen inference workspace), which fits a browser tab but not with a huge margin... for big documents is better to run the conversion in a Worker, the wasm README shows the pattern.

The published wasm bundle does not change (~2.95 MB gzip); an OCR build is ~5.45 MB gzip. The demo build turns on also wasm SIMD, that alone cuts the 018 conversion in wasm from ~1.8 s to ~0.9 s (I measured on Node, which runs the same V8 of Chrome) and costs ~11 KB of gzip more. The native bindings grow by ~4.75 MB when OCR is compiled in. The base crate keeps Rust 1.88; the ocr feature needs 1.92, that is hayro's floor and it comes in through the render feature.

Tests

cargo test --locked is 224 tests; with --features ocr it is 251. One snapshot changed and honestly it is not OCR: text.pdf converts now through the pinned pdf-inspector revision, which brings the upstream strikeout and footnote fixes, so the expected output moved with it.

For the OCR pipeline I scripted a deterministic fake engine, so I can test the routing, the merge order, which text wins on a page and the failure fallbacks without any model file. Then there are the tests with the real models. They cover the init with the standard models, a synthetic scanned page, and the 018 case above asserting the recognized phrase. I marked them as ignored, they run only when the model paths are set.

To run them (from the repo root):

git clone https://github.com/py-pdf/sample-files.git ../sample-files
git -C ../sample-files checkout 89039b6078fd0c9f98bf3d6fcb5583fac6b0ecaf
curl -LO https://ocrs-models.s3-accelerate.amazonaws.com/text-detection.rten
curl -LO https://ocrs-models.s3-accelerate.amazonaws.com/text-recognition.rten

ANYDOC_OCR_DETECTION_MODEL=text-detection.rten ANYDOC_OCR_RECOGNITION_MODEL=text-recognition.rten ANYDOC_PDF_SAMPLE_FILES=../sample-files cargo test --features ocr --test ocr -- --ignored --nocapture

The Node binding runs also its whole suite with the real models (17 passed locally), and the wasm package has Node tests for both builds plus the same 018 case through wasm.

Known limitations

The current ocrs models read Latin script only and the output is plain text, no structure. Probably some structure could be inferred from the line geometry (paragraphs, maybe headings) and I would like to try in a follow-up, but here I preferred to keep the recognized text plain on purpose.

On photos with no text the models can still produce short garbage fragments; in the corpus tests I assert the cases where nothing may be invented, rather than pretending it never happens. Overlapping text (a watermark over a paragraph) can garble the merged line. EXIF orientation is not applied to images yet. The models are CC-BY-SA-4.0 and are never bundled, the caller downloads them and passes the bytes.

Important notes

• I did not expose OCR in the Python binding on purpose: I personally don't like very much nor use Python, so I preferred to not ship a converter there that I cannot maintain. The only Python changes are the new image format name in the registry and a couple of refreshed docstrings, so the binding keeps compiling. The plumbing is all in the core crate, adding the converter later should be mechanical if you want it.
• Format::Image is a new variant on an exhaustive enum, so an exhaustive match downstream will need a new arm. I did not mark the enum non_exhaustive because that changes the ergonomics for everyone; your call if you prefer it.
• The release matrix for the native bindings has never compiled rten and hayro across all 7 napi targets, so please keep an eye on the first release with OCR in.
• This PR needs feat(render): add optional selected-page rasterization pdf-inspector#280 merged first. Right now pdf-inspector comes from a git pin on my fork's PR branch, so every lane runs from a clean clone, and I will switch it to the released crate version once #280 ships. That is also why this is a WIP.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}