Bug: Custom entity_labels silently drop PII (non-deterministic)

[memadi-nv]

Priority Level

Medium (Annoying but has workaround)

Describe the bug

Summary: When entity_labels don't match the canonical vocab in constants.py, detected PII is silently dropped from final_entities — ~2 entities/record, varying run-to-run.

Root cause:

1. GLiNER-pii ignores custom labels and always emits canonical ones (first_name, not givenname).
2. The only bridge to the custom vocab is the stochastic LLM validator reclass step.
3. When it returns keep instead of reclass, the exact-match filter in _materialize_final_entities (detection_workflow.py:414) drops the entity — silently.

Evidence (OpenPII quasi-25, custom vocab): GLiNER emitted 544 entities/run, 72% out-of-vocab; validator dropped 47–51 of them (last_name, first_name, phone_number, ssn, date_of_birth…). Not caused by strict_entity_protection — it has no code path into detection.

Fix:

• Either 1. Deterministic alias map (first_name↔givenname, postcode↔zipcode, …) at the filter boundary — don't rely on the LLM validator. or 2. Find possible ways of using the custom entity labels within Gliner in the first place.
• Log dropped out-of-vocab labels instead of silently discarding.
  Severity: High — silent, non-deterministic PII leakage.

Steps/Code to reproduce bug

1. Run detection with Detect(entity_labels=[...]) using a custom vocab (e.g. OpenPII: givenname, surname, zipcode…) on the OpenPII quasi-25 sample. The entities should preferably be the ones not in constant.py/
2. Inspect the trace: _raw_detected_entities shows GLiNER emitting canonical labels (first_name, not givenname); _validated_entities shows a mix of reclass and keep.
3. Any entity with a keep + out-of-vocab label is absent from final_entities — and the count varies run-to-run.

Expected behavior

When a user configures entity_labels, entities of those types should be reliably and deterministically detected and labeled in final_entities — regardless of which internal label GLiNER emits.

Additional context

No response