Skip to content

fix(language): make the parser setting real, and Chinese use jieba (#278) - #281

Open
HugoFara wants to merge 1 commit into
developfrom
fix/278-wire-parser-registry
Open

fix(language): make the parser setting real, and Chinese use jieba (#278)#281
HugoFara wants to merge 1 commit into
developfrom
fix/278-wire-parser-registry

Conversation

@HugoFara

Copy link
Copy Markdown
Owner

Fixes #278.

The symptom, reproduced

A Chinese language created from the built-in preset parses a Chinese text to:

sentences=1  words=0

No clickable words at all — nothing to look up, nothing to track. That is the screenshot in the issue.

Three gaps stacked up to it

1. The parser setting was decorative. LgParserType was written by the language form and read by nothing. TextParsing::tokenize() branches only on the legacy MECAB magic word and otherwise goes straight to StandardTextParser. Measured — the value makes no difference at all, including a value that isn't a parser:

LgParserType result
NULL sentences=2 words=13
jieba sentences=2 words=13
regex sentences=2 words=13
mecab sentences=2 words=13
totally-bogus sentences=2 words=13

ParserRegistry had exactly two callers, both just filling the dropdown. CharacterParser, MecabParser and ExternalParser were unreachable from the parsing path.

2. External parsers never reached the dropdown either. ParserRegistry::__construct() takes an optional loader; both direct call sites passed none, so registerExternalParsers() returned immediately. jieba is installed in the published Docker image (Dockerfile:59-65 builds the venv and copies parsers/, matching config/parsers.php) and works — it was simply never listed.

3. The presets could not express a parser. langdefs.json has carried parserType for the CJK languages all along, but LanguagePresets::loadFromJson() flattened it into an eight-slot tuple that dropped the field. That is also why the Japanese preset, declaring mecab, silently resolved to character parsing even with MeCab installed.

What changed

  • TextParsing consults the registry before the built-in pipeline and adapts TokenParsedToken (the shapes differ only in sentence numbering and token ordering). The check-text preview uses the same parser as the save.
  • ParserRegistry builds its own ExternalParserLoader when given none, so jieba and MeCab Python appear on an install that has them.
  • LanguagePresets carries parserType as slot 8; the API exposes it; both preset appliers (the Alpine store and the wizard's DOM applier) set it.
  • Chinese (Simplified) and (Traditional) ask for jieba, keeping makeCharacterWord so the fallback stays meaningful.
  • The form gained an Automatic option, so "infer from the flags" stays expressible. New locale key across all nine languages.

Opting in is deliberate; the fallback is safe

Only an explicit, non-default LgParserType routes to the registry. The legacy signals resolveParserTypeFromRow() also understands — the MECAB magic word and LgSplitEachChar — are deliberately ignored there, so every language that exists today parses byte-identically; none of them stores a parser type.

A parser the server cannot run falls back to the built-in pipeline, never to the regex parser — which for a CJK language is precisely the zero-word text this issue is about.

Verified end to end, real database, jieba installed

preset (jieba)        LgParserType='jieba'     sentences=2 words=7
no parser type        LgParserType=NULL        sentences=2 words=13
unavailable parser    LgParserType='sudachi…'  sentences=2 words=13

7 is jieba word segmentation (我 | 喜欢 | 学习 | 中文 | 这是 | 一个 | 句子); 13 is one token per character.

Checks

Psalm 0 errors · PHPCS 0 errors, 0 warnings · PHPUnit 9101 pass (11 new, covering opt-in resolution, the legacy signals staying inert, the unavailable-parser fallback, and the preset slot) · Vitest 4326 pass · tsc and ESLint clean · assets rebuilt.

Not in this PR

  • The "0 words" warning. Nothing tells you a language's rules match nothing in your text. Worth doing separately as the backstop for every other misconfiguration.
  • The dead NLP /parse/ route. services/nlp/ has a second, complete jieba — JiebaParser, a live endpoint, and a PHP client in NlpServiceHandler::parse() / getAvailableParsers(). Both have zero callers. Two implementations, one a decoy; worth either wiring up or deleting.

@HugoFara HugoFara added enhancement Develop an existing feature ui Any problem related to the User Interface new-feature A new feature labels Aug 23, 2026
@HugoFara HugoFara linked an issue Aug 23, 2026 that may be closed by this pull request
)

A Chinese language created from the built-in preset produced a text with no
clickable words at all: sentences=1, words=0. Nothing to look up, nothing to
track. Three separate gaps stacked up to that.

The parser setting was decorative. LgParserType was written by the language
form and read by nothing — TextParsing::tokenize() branched only on the legacy
MECAB magic word and otherwise went straight to StandardTextParser, so setting
a language to jieba, mecab or even a bogus value parsed identically. The whole
ParserRegistry existed only to populate the dropdown. It is now consulted
before the built-in pipeline, and its tokens are adapted to ParsedToken; the
two shapes differ only in sentence numbering and token ordering.

External parsers never reached the dropdown either. ParserRegistry's
constructor took an optional loader, and both direct call sites passed none,
so registerExternalParsers() returned immediately: jieba was installed in the
Docker image, working, and unlistable. It now builds a loader when given none.

The presets could not express a parser. langdefs.json has carried parserType
for the CJK languages all along, but LanguagePresets flattened it into an
eight-slot tuple that dropped the field — which is why the Japanese preset,
declaring mecab, silently resolved to character parsing. The tuple carries it
as slot 8, the API exposes it, and both preset appliers set it.

Chinese (Simplified) and (Traditional) now ask for jieba, keeping
makeCharacterWord so the fallback stays meaningful.

Opting in is deliberate and the fallback is safe. Only an explicit,
non-default LgParserType routes to the registry; the legacy signals it also
understands are ignored, so every language that exists today — all of which
store no parser type — parses byte-identically. A parser the server cannot run
falls back to the built-in pipeline, never to the regex parser, which for a
CJK language yields zero words. The form gained an "Automatic" option so that
"infer from the flags" stays expressible.

Verified end to end against a real database with jieba installed:

  preset (jieba)        LgParserType='jieba'    sentences=2 words=7
  no parser type        LgParserType=NULL       sentences=2 words=13
  unavailable parser    LgParserType='sudachi'  sentences=2 words=13

where 7 is jieba word segmentation and 13 is one token per character.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Develop an existing feature new-feature A new feature ui Any problem related to the User Interface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Simplified Chinese doesn't split words or allow splitting words

1 participant