feat(rag): add ViDoRe V3 agentic RAG evaluation with reasoning-based page selectors#43
Draft
ceberam wants to merge 5 commits into
Draft
feat(rag): add ViDoRe V3 agentic RAG evaluation with reasoning-based page selectors#43ceberam wants to merge 5 commits into
ceberam wants to merge 5 commits into
Conversation
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Signed-off-by: Cesar Berrospi Ramis <ceb@zurich.ibm.com>
Contributor
|
✅ DCO Check Passed Thanks @ceberam, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces an end-to-end evaluation pipeline for a novel chunkless, reasoning-driven RAG approach on the
ViDoRe V3 benchmark. Rather than splitting documents into fixed-size chunks and relying on vector similarity, the approach preserves the
DoclingDocumentstructure, enriches it with AI-generated summaries, and uses an LLM to navigate the document hierarchy and select the most relevant pages per query.What's new
Enricher — dual summarisation styles
DoclingEnrichingAgent._summarize_pagesgains astyleparameter ("sentences"/"keyphrases"). When"keyphrases"is chosen, page-level enrichment stores keyphrase lists inmeta.keywordsinstead of prose summaries inmeta.summary. Resume logic and the skip-if-already-enriched guard both respect the chosen style.Two page-selector strategies
ReasoningBasedPageSelector— evaluates pages in iterative, sliding-window batches; works with both page-level and element-level enrichment. Includes multi-document support viaselect_relevant_documents, per-documentselect_pages, and cross-documentrerank_across_documents.TreeGuidedPageSelector— traverses the heading hierarchy top-down, drilling into the most promising sections at each step; requires element-level enrichment.Both selectors are exported from
docling_agent.agents.perfs/evaluation harnessA four-step pipeline script and YAML config template cover:
DoclingEditingAgentranxThe pipeline is fully resumable — Step 3 checkpoints after every page and Step 4 flushes results after every query. The
perfs/README is updated to document both the existing extraction-quality evaluator and the new RAG evaluator side by side.