Skip to content

feat(rag): add ViDoRe V3 agentic RAG evaluation with reasoning-based page selectors#43

Draft
ceberam wants to merge 5 commits into
mainfrom
dev/vidore-eval
Draft

feat(rag): add ViDoRe V3 agentic RAG evaluation with reasoning-based page selectors#43
ceberam wants to merge 5 commits into
mainfrom
dev/vidore-eval

Conversation

@ceberam

@ceberam ceberam commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

Introduces an end-to-end evaluation pipeline for a novel chunkless, reasoning-driven RAG approach on the
ViDoRe V3 benchmark. Rather than splitting documents into fixed-size chunks and relying on vector similarity, the approach preserves the DoclingDocument structure, enriches it with AI-generated summaries, and uses an LLM to navigate the document hierarchy and select the most relevant pages per query.


What's new

Enricher — dual summarisation styles

DoclingEnrichingAgent._summarize_pages gains a style parameter ("sentences" / "keyphrases"). When "keyphrases" is chosen, page-level enrichment stores keyphrase lists in meta.keywords instead of prose summaries in meta.summary. Resume logic and the skip-if-already-enriched guard both respect the chosen style.

Two page-selector strategies

  • ReasoningBasedPageSelector — evaluates pages in iterative, sliding-window batches; works with both page-level and element-level enrichment. Includes multi-document support via select_relevant_documents, per-document select_pages, and cross-document rerank_across_documents.
  • TreeGuidedPageSelector — traverses the heading hierarchy top-down, drilling into the most promising sections at each step; requires element-level enrichment.

Both selectors are exported from docling_agent.agents.

perfs/ evaluation harness

A four-step pipeline script and YAML config template cover:

  1. PDF → DoclingDocument conversion
  2. Heading-level normalisation via DoclingEditingAgent
  3. Document enrichment (element-level or page-level, sentences or keyphrases)
  4. NDCG@10 scoring against ViDoRe ground truth using ranx

The pipeline is fully resumable — Step 3 checkpoints after every page and Step 4 flushes results after every query. The perfs/ README is updated to document both the existing extraction-quality evaluator and the new RAG evaluator side by side.

ceberam added 5 commits July 1, 2026 13:39
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

DCO Check Passed

Thanks @ceberam, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant