Fixes #396: auto-detect CJK text in NgramSampleEvaluator scoring

[zrh805]

Fixes #396

Problem

In the default language='en' mode, NgramSampleEvaluator._score_func splits text via str.split(). Chinese text has no spaces, so the whole sentence becomes a single token; with ngrams=5 the token count is below the threshold and the sample always scores 0.0, causing all Chinese samples to be filtered out — contradicting the documented example output.

Fix

In dataflow/operators/general_text/eval/ngram_sample_evaluator.py, the splitting condition now also checks for CJK characters (re.search(r'[\u4e00-\u9fff]', content)): when CJK text is detected, it falls back to character-level splitting even in en mode.

• Pure English text: behavior unchanged (whitespace splitting)
• Explicit language='zh': behavior unchanged (character-level splitting)
• Chinese text in default en mode: now correctly scored instead of 0.0

Updated get_desc (zh/en) to document the auto-detection.

[SunnyHaze]

Thanks for identifying this use case and contributing the initial implementation! I pushed a follow-up commit that preserves the intent of your proposal while avoiding an implicit behavior change.

The evaluator now supports an explicit language='auto' mode:

• language='en' always uses whitespace-based word tokenization.
• language='zh' always uses character-level tokenization.
• language='auto' selects the strategy per sample based on the presence of Han characters.

This keeps existing behavior backward-compatible and predictable, while still supporting datasets containing both English and Chinese text. I also expanded Han character coverage, updated the evaluator/filter descriptions, and added regression tests for English, Chinese, mixed-language text, and extended Han characters.

Since #396 was previously addressed through explicit language='zh' support in #397, this PR is now better treated as an opt-in enhancement rather than a replacement for the existing language configuration.