CNS cell-entity extraction evaluation

[puja-trivedi]

Add CNS cell-entity extraction evaluation

Adds an evaluation harness that measures how many ground-truth cell entities StructSense actually extracts from the CNS/MeSH corpus.

What it does

evaluation/cns_cell/evaluate_extraction.py compares StructSense's per-paper JSON output against the ground-truth BioC XML (NLM-CellLink_CNS_MeSH_train_val.xml). For every annotation row in the ground truth, it checks whether StructSense extracted that entity — comparing entity text only (annotation labels are ignored, since StructSense uses a large open label vocabulary that doesn't map cleanly to the 3 ground-truth cell types).

Matching rules

• Entity text: normalized whole-string equality (case-insensitive, collapsed whitespace, stripped surrounding punctuation).
• Duplicates: every annotation row is scored independently.
• Two recall metrics reported side by side:
  • SAME-SENTENCE — entity extracted in the same sentence the ground truth annotated it in. Because the ground truth is clean curated text while StructSense parses PDFs (which inject reference numbers mid-sentence, merge captions, drop apostrophes, etc.), sentence correspondence uses token overlap (default ≥0.6) rather than exact substring.
  • ANYWHERE — entity extracted anywhere in the paper (location ignored).

Results (current output)
The evaluation now covers only the 90 papers that have structsense output.

Metric	Extracted	Recall
Same-sentence	224/869	25.8%
Anywhere	478/869	55.0%

Breakdowns by entity type (cell_phenotype, cell_vague, cell_hetero) and per-PMID are included in the summary.

Files

• evaluate_extraction.py — the script (CLI-configurable: --sentence-mode, --threshold, --min-token-len, --detail-csv, --summary-out).
• evaluation_summary.txt — shareable summary report.
• evaluation_detail.csv — per-annotation results with matched_anywhere / matched_same_sentence flags.

The ground-truth XML, source PDFs, and StructSense JSON outputs are intentionally not committed (data artifacts).

Notes / caveats for reviewers

• Only 103 of 208 ground-truth PMIDs have StructSense output, so the "covered PMIDs" denominator is the fairer read of extraction quality.
• The same-sentence vs anywhere gap reflects that StructSense often extracts an entity but tags a different mention than the annotated one, plus PDF-vs-curated text differences.

How to run

python3 evaluation/cns_cell/evaluate_extraction.py --detail-csv evaluation/cns_cell/evaluation_detail.csv

[gemini-code-assist]

Code Review

This pull request introduces an evaluation script (evaluate_extraction.py) and its corresponding summary report (evaluation_summary.txt) to assess structsense entity extraction against CNS/MeSH ground-truth XML. The code review feedback highlights a critical bug where only the first passage of each document is parsed instead of all passages, and recommends using context managers with explicit UTF-8 encoding when opening files to prevent resource leaks and encoding issues.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The current implementation only processes the first <passage> element of each <document> because it uses doc.find("passage"). In BioC XML format, a document typically contains multiple passages (e.g., title, abstract, paragraphs). Using doc.findall("passage") in a nested loop ensures all passages and their annotations are evaluated.

    for doc in root.findall("document"):
        for passage in doc.findall("passage"):
            pmid = passage_id = None
            for infon in passage.findall("infon"):
                key = infon.get("key")
                if key == "article-id_pmid":
                    pmid = infon.text
                elif key == "passage_id":
                    passage_id = infon.text
            passage_text = passage.findtext("text") or ""
            for ann in passage.findall("annotation"):
                loc = ann.find("location")
                if loc is None:
                    continue
                atype = None
                for infon in ann.findall("infon"):
                    if infon.get("key") == "type":
                        atype = infon.text
                rows.append(
                    GTAnnotation(
                        pmid=pmid or "",
                        passage_id=passage_id or (doc.findtext("id") or ""),
                        entity_text=ann.findtext("text") or "",
                        entity_type=atype or "",
                        offset=int(loc.get("offset", 0)),
                        length=int(loc.get("length", 0)),
                        passage_text=passage_text,
                    )
                )