Base install cannot ingest HTML/TXT because tokenizer dependency is only in local extra

[jioffe502]

Summary

The documented CPU/hosted-inference installation (pip install nemo-retriever) discovers HTML and TXT inputs but produces zero rows because their chunking path imports transformers.AutoTokenizer, while transformers and tokenizers are available only through the GPU-oriented [local] extra.

This is a packaging-contract gap, not a missing HTML route: retriever ingest, GraphIngestor, ServiceIngestor, and the graph extraction operators all route HTML/TXT correctly.

Expected behavior

A base installation should support the documented remote-NIM workflow:

pip install nemo-retriever
export NVIDIA_API_KEY=nvapi-...
retriever ingest ./mixed-text-folder

For a directory containing one short .html, one short .txt, and one unsupported .md, ingest should skip Markdown and write two LanceDB rows without installing Torch or local model inference dependencies.

The README currently states that the base package is sufficient for remote NIM inference, and the Retriever skill states that the base installation covers HTML and TXT.

Actual behavior

A clean base-wheel installation has neither transformers nor Torch. HTML/TXT extraction calls transformers.AutoTokenizer; the extraction actors suppress that exception and return empty data. Root ingest consequently reports zero rows.

Installing [local] is not an appropriate CPU workaround: it brings in Torch, CUDA, vLLM, torchvision, Nemotron model packages, and other local-GPU dependencies.

Root cause

HTML/TXT token chunking was introduced when transformers, tokenizers, and Torch were core dependencies.

PR #1830 (feat(install): slim install for remote/NIM-only inference on Mac/Windows) moved transformers and tokenizers into [local] with the GPU inference stack. The HTML/TXT chunker remained in core and continued importing transformers.AutoTokenizer.

The mismatch remained hidden because:

• unit and Docker CI install [all,dev];
• TXT/HTML tests use pytest.importorskip("transformers");
• slim CI checks imports and absence of Triton but does not execute TXT/HTML extraction;
• library-mode CI installs the base package but only runs retriever --help;
• per-file extraction catches broad exceptions, obscuring the missing import.

Reproduction evidence

Validated with a wheel built from the current repository:

• Base wheel: transformers absent, Torch absent, two supported files produced zero rows.
• Base wheel + only transformers: Torch remained absent; hosted embedding succeeded; one HTML and one TXT input produced exactly two LanceDB rows; Markdown was skipped; both rows had nonempty text and 2048-dimensional embeddings.
• Base wheel + [local]: installed the full local-GPU stack and selected local vLLM, demonstrating that this is not a pragmatic CPU-hosted installation path.

No API keys or credential values are included in this issue.

Proposed scope

1. Move these lightweight tokenizer dependencies from [local] to core dependencies:
   • transformers>=4.57.6,<5
   • tokenizers>=0.21.1
2. Keep Torch, torchvision, CUDA, vLLM, accelerate, Nemotron model packages, and other local inference dependencies in [local].
3. Regenerate nemo_retriever/uv.lock.
4. Remove pytest.importorskip("transformers") from TXT/HTML conversion tests once it is a required dependency.
5. Strengthen the slim-install contract to verify:
   • transformers and tokenizers import;
   • torch and tritonclient remain absent;
   • focused TXT/HTML conversion tests run without downloading model weights.

Non-goals

• No tokenizer implementation or fallback tokenizer.
• No chunking behavior change.
• No Torch dependency in the base package.
• No CLI, retrieval-default, or skill changes.
• No local model inference changes.

Acceptance criteria

• A freshly built wheel installed with no extras includes transformers and tokenizers.
• The same installation does not include Torch or Triton.
• pip check passes.
• With a valid NVIDIA_API_KEY, retriever ingest on one short HTML file, one short TXT file, and one Markdown file:
  • exits successfully;
  • skips Markdown;
  • writes exactly two LanceDB rows;
  • writes nonempty text and valid hosted embeddings.
• Focused TXT/HTML and slim-install tests pass.