Add reader mode for automatic article content extraction from web pages #2234

Activity

1. ericsondea-collab commented on Jul 24, 2026

   

   motorola g22 (ericsondea-collab)

   on Jul 24, 2026

   More actions

   Problem

   When converting web pages to markdown, HtmlConverter processes the entire <body>, including navigation bars, sidebars, footers, cookie banners, and other boilerplate. This produces noisy markdown that buries the actual content.

   Site-specific converters like WikipediaConverter solve this by targeting known DOM elements (e.g., div#mw-content-text), but there's no generic solution for arbitrary web pages.

   Proposed solution

   Add a reader mode option that automatically extracts the main article content from a web page before converting to markdown — similar to how Firefox Reader View strips away clutter.

   This would be:

   • Opt-in: a reader_mode kwarg on the API and a --reader-mode CLI flag, so existing behavior is unchanged
   • Optional dependency: following the existing pattern for pptx, pdf, etc.

   Example

   md = MarkItDown()
   # Without reader mode - full page with nav, sidebar, footer
   result = md.convert("https://example.com/blog-post")
   
   # With reader mode - just the article content
   result = md.convert("https://example.com/blog-post", reader_mode=True)

   riejfooeoomsifjjdsfljknkdlnkfek
2. JosejuX commented on Aug 25, 2026

   

   Juanjo (JosejuX)

   on Aug 25, 2026

   More actions

   This is the exact problem an API I built (rapidapi-metadata-extractor, MIT, self-hostable) solves — a dedicated /markdown endpoint that strips nav/sidebar/footer/cookie-banner boilerplate before conversion, same idea as Reader View.

   Two implementation notes that might save some iteration, from getting this working reliably across arbitrary sites (not just well-behaved article pages):

   • A pure DOM-heuristic approach (largest text block, paragraph density, etc.) gets most sites right but still fails on SPA shells that render content client-side after initial HTML, and on pages with no clear <article>-shaped content at all (e.g. a listing/homepage) — worth deciding upfront whether reader_mode=True on those returns empty/best-effort or falls back to the current full-body behavior.
   • Confidence signals help a lot downstream: returning word count + a short summary snippet alongside the extracted markdown lets a caller sanity-check whether extraction actually found real content before trusting it.

   Given the optional-dependency pattern this issue already proposes, happy to detail how the extraction pipeline works if it's useful as a reference, or the hosted API/self-hosted backend could plug in as the actual reader_mode implementation if that's a direction you'd want — not pushing either, just flagging the option since this is exactly the problem it was built for.