fix(language): make the parser setting real, and Chinese use jieba (#278) - #281
Open
HugoFara wants to merge 1 commit into
Open
fix(language): make the parser setting real, and Chinese use jieba (#278)#281HugoFara wants to merge 1 commit into
HugoFara wants to merge 1 commit into
Conversation
) A Chinese language created from the built-in preset produced a text with no clickable words at all: sentences=1, words=0. Nothing to look up, nothing to track. Three separate gaps stacked up to that. The parser setting was decorative. LgParserType was written by the language form and read by nothing — TextParsing::tokenize() branched only on the legacy MECAB magic word and otherwise went straight to StandardTextParser, so setting a language to jieba, mecab or even a bogus value parsed identically. The whole ParserRegistry existed only to populate the dropdown. It is now consulted before the built-in pipeline, and its tokens are adapted to ParsedToken; the two shapes differ only in sentence numbering and token ordering. External parsers never reached the dropdown either. ParserRegistry's constructor took an optional loader, and both direct call sites passed none, so registerExternalParsers() returned immediately: jieba was installed in the Docker image, working, and unlistable. It now builds a loader when given none. The presets could not express a parser. langdefs.json has carried parserType for the CJK languages all along, but LanguagePresets flattened it into an eight-slot tuple that dropped the field — which is why the Japanese preset, declaring mecab, silently resolved to character parsing. The tuple carries it as slot 8, the API exposes it, and both preset appliers set it. Chinese (Simplified) and (Traditional) now ask for jieba, keeping makeCharacterWord so the fallback stays meaningful. Opting in is deliberate and the fallback is safe. Only an explicit, non-default LgParserType routes to the registry; the legacy signals it also understands are ignored, so every language that exists today — all of which store no parser type — parses byte-identically. A parser the server cannot run falls back to the built-in pipeline, never to the regex parser, which for a CJK language yields zero words. The form gained an "Automatic" option so that "infer from the flags" stays expressible. Verified end to end against a real database with jieba installed: preset (jieba) LgParserType='jieba' sentences=2 words=7 no parser type LgParserType=NULL sentences=2 words=13 unavailable parser LgParserType='sudachi' sentences=2 words=13 where 7 is jieba word segmentation and 13 is one token per character.
HugoFara
force-pushed
the
fix/278-wire-parser-registry
branch
from
August 23, 2026 11:41
fb0bf29 to
4c78c01
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #278.
The symptom, reproduced
A Chinese language created from the built-in preset parses a Chinese text to:
No clickable words at all — nothing to look up, nothing to track. That is the screenshot in the issue.
Three gaps stacked up to it
1. The parser setting was decorative.
LgParserTypewas written by the language form and read by nothing.TextParsing::tokenize()branches only on the legacyMECABmagic word and otherwise goes straight toStandardTextParser. Measured — the value makes no difference at all, including a value that isn't a parser:LgParserTypeNULLjiebaregexmecabtotally-bogusParserRegistryhad exactly two callers, both just filling the dropdown.CharacterParser,MecabParserandExternalParserwere unreachable from the parsing path.2. External parsers never reached the dropdown either.
ParserRegistry::__construct()takes an optional loader; both direct call sites passed none, soregisterExternalParsers()returned immediately. jieba is installed in the published Docker image (Dockerfile:59-65builds the venv and copiesparsers/, matchingconfig/parsers.php) and works — it was simply never listed.3. The presets could not express a parser.
langdefs.jsonhas carriedparserTypefor the CJK languages all along, butLanguagePresets::loadFromJson()flattened it into an eight-slot tuple that dropped the field. That is also why the Japanese preset, declaringmecab, silently resolved to character parsing even with MeCab installed.What changed
TextParsingconsults the registry before the built-in pipeline and adaptsToken→ParsedToken(the shapes differ only in sentence numbering and token ordering). The check-text preview uses the same parser as the save.ParserRegistrybuilds its ownExternalParserLoaderwhen given none, so jieba and MeCab Python appear on an install that has them.LanguagePresetscarriesparserTypeas slot 8; the API exposes it; both preset appliers (the Alpine store and the wizard's DOM applier) set it.makeCharacterWordso the fallback stays meaningful.Opting in is deliberate; the fallback is safe
Only an explicit, non-default
LgParserTyperoutes to the registry. The legacy signalsresolveParserTypeFromRow()also understands — theMECABmagic word andLgSplitEachChar— are deliberately ignored there, so every language that exists today parses byte-identically; none of them stores a parser type.A parser the server cannot run falls back to the built-in pipeline, never to the regex parser — which for a CJK language is precisely the zero-word text this issue is about.
Verified end to end, real database, jieba installed
7 is jieba word segmentation (
我 | 喜欢 | 学习 | 中文 | 这是 | 一个 | 句子); 13 is one token per character.Checks
Psalm 0 errors · PHPCS 0 errors, 0 warnings · PHPUnit 9101 pass (11 new, covering opt-in resolution, the legacy signals staying inert, the unavailable-parser fallback, and the preset slot) · Vitest 4326 pass · tsc and ESLint clean · assets rebuilt.
Not in this PR
/parse/route.services/nlp/has a second, complete jieba —JiebaParser, a live endpoint, and a PHP client inNlpServiceHandler::parse()/getAvailableParsers(). Both have zero callers. Two implementations, one a decoy; worth either wiring up or deleting.