fix(KeywordSearch): keep punctuation handling consistent in case-sensitive mode

[kz930]

What changes were proposed in this PR?

Fixes an unintended side effect of the KeywordSearch "case sensitive" option.

KeywordSearchOpExec selects an analyzer based on isCaseSensitive:

• off (default) → StandardAnalyzer, whose StandardTokenizer splits on Unicode word boundaries and strips punctuation, so "perfect." tokenizes to perfect.
• on → CaseSensitiveAnalyzer, which used a bare WhitespaceTokenizer. That splits only on whitespace, so punctuation stays glued to the token: "...absolutely perfect." tokenizes to [absolutely, perfect.].

The intent of CaseSensitiveAnalyzer was only to skip lowercasing. But swapping the tokenizer also dropped Unicode word-boundary tokenization, so enabling "case sensitive" silently changed punctuation handling too — a side effect the option name does not imply. A user who turns on case sensitivity to match Trump exactly finds that perfect no longer matches perfect., USA no longer matches USA., etc.

The fix makes CaseSensitiveAnalyzer mirror StandardAnalyzer's pipeline minus the LowerCaseFilter: it now uses StandardTokenizer (Unicode word-boundary tokenization, punctuation stripped) and omits lowercasing. Case sensitivity becomes the only behavioral difference from the default analyzer.

-import org.apache.lucene.analysis.core.WhitespaceTokenizer
+import org.apache.lucene.analysis.standard.StandardTokenizer
 ...
-    val tokenizer = new WhitespaceTokenizer()
+    val tokenizer = new StandardTokenizer()
     val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET)

Any related issues, documentation, discussions?

Closes #6761

Note: the standalone/Python translation of KeywordSearch (KeywordSearchOpDesc.generateStandaloneCode) does not implement case sensitivity at all — that is a separate gap and out of scope here.

How was this PR tested?

Added a regression test to KeywordSearchOpExecSpec asserting that in case-sensitive mode, keyword perfect matches the row everything was absolutely perfect. (a word adjacent to trailing punctuation). This test fails on main (the row is not matched) and passes with this change.

Ran the full suite locally:

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.keywordSearch.KeywordSearchOpExecSpec"
...
Tests: succeeded 18, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

The existing case-sensitive tests (twitter/Twitter) still pass, confirming case sensitivity itself is unchanged.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @SarahAsad23, @aglinxinyuan
  You can notify them by mentioning @SarahAsad23, @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.52%. Comparing base (bf1f589) to head (f245bb7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6801      +/-   ##
============================================
+ Coverage     75.41%   79.52%   +4.11%     
+ Complexity     3452      552    -2900     
============================================
  Files          1160      589     -571     
  Lines         45807    31968   -13839     
  Branches       5071     3516    -1555     
============================================
- Hits          34544    25423    -9121     
+ Misses         9657     5552    -4105     
+ Partials       1606      993     -613     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)
agent-service	76.76% <ø> (ø)		Carriedforward from a8ea882
amber	73.11% <ø> (+5.89%)	⬆️	Carriedforward from a8ea882
computing-unit-managing-service	20.49% <ø> (ø)		Carriedforward from a8ea882
config-service	66.66% <ø> (ø)
file-service	67.21% <ø> (ø)		Carriedforward from a8ea882
frontend	79.86% <ø> (ø)		Carriedforward from a8ea882
notebook-migration-service	78.94% <ø> (ø)
pyamber	92.15% <ø> (ø)		Carriedforward from a8ea882
workflow-compiling-service	55.14% <ø> (ø)		Carriedforward from a8ea882

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 2 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main bf1f589 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	419	0.256	22,426/34,970/34,970 us	🔴 +15.5% / 🔴 +117.0%
🟢	bs=100 sw=10 sl=64	865	0.528	116,806/123,389/123,389 us	🟢 -21.2% / 🔴 +15.1%
⚪	bs=1000 sw=10 sl=64	944	0.576	1,053,892/1,117,265/1,117,265 us	⚪ within ±5% / 🔴 +8.5%

Baseline details

Latest main bf1f589 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	419 tuples/sec	437 tuples/sec	760.2 tuples/sec	-4.1%	-44.9%
bs=10 sw=10 sl=64	MB/s	0.256 MB/s	0.267 MB/s	0.464 MB/s	-4.1%	-44.8%
bs=10 sw=10 sl=64	p50	22,426 us	23,406 us	12,781 us	-4.2%	+75.5%
bs=10 sw=10 sl=64	p95	34,970 us	30,284 us	16,118 us	+15.5%	+117.0%
bs=10 sw=10 sl=64	p99	34,970 us	30,284 us	19,074 us	+15.5%	+83.3%
bs=100 sw=10 sl=64	throughput	865 tuples/sec	853 tuples/sec	988.68 tuples/sec	+1.4%	-12.5%
bs=100 sw=10 sl=64	MB/s	0.528 MB/s	0.521 MB/s	0.603 MB/s	+1.3%	-12.5%
bs=100 sw=10 sl=64	p50	116,806 us	114,884 us	101,448 us	+1.7%	+15.1%
bs=100 sw=10 sl=64	p95	123,389 us	156,576 us	107,410 us	-21.2%	+14.9%
bs=100 sw=10 sl=64	p99	123,389 us	156,576 us	120,258 us	-21.2%	+2.6%
bs=1000 sw=10 sl=64	throughput	944 tuples/sec	937 tuples/sec	1,023 tuples/sec	+0.7%	-7.7%
bs=1000 sw=10 sl=64	MB/s	0.576 MB/s	0.572 MB/s	0.624 MB/s	+0.7%	-7.8%
bs=1000 sw=10 sl=64	p50	1,053,892 us	1,069,113 us	983,888 us	-1.4%	+7.1%
bs=1000 sw=10 sl=64	p95	1,117,265 us	1,096,657 us	1,030,100 us	+1.9%	+8.5%
bs=1000 sw=10 sl=64	p99	1,117,265 us	1,096,657 us	1,056,513 us	+1.9%	+5.8%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,476.92,200,128000,419,0.256,22426.33,34969.92,34969.92
1,100,10,64,20,2311.60,2000,1280000,865,0.528,116805.51,123388.97,123388.97
2,1000,10,64,20,21195.59,20000,12800000,944,0.576,1053891.62,1117264.96,1117264.96

[kz930]

@SarahAsad23 May you take a look at it please?

[aglinxinyuan]

Please fix the test case

[Copilot]

Pull request overview

This PR fixes an unintended behavioral change in the KeywordSearch operator when case-sensitive mode is enabled: tokenization now stays consistent with Lucene’s StandardAnalyzer punctuation/word-boundary handling while preserving case (i.e., the only difference becomes lowercasing vs. not lowercasing).

Changes:

• Update CaseSensitiveAnalyzer to use Lucene StandardTokenizer instead of WhitespaceTokenizer, aligning punctuation handling with the default analyzer.
• Add a regression test to ensure case-sensitive searches still match words adjacent to trailing punctuation (e.g., perfect matches perfect.).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzer.scala	Switch tokenizer to StandardTokenizer to keep punctuation/word-boundary behavior consistent while preserving case.
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/keywordSearch/KeywordSearchOpExecSpec.scala	Add regression coverage for case-sensitive matching when the target word is adjacent to punctuation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kz930]

Done fixing the test.

[aglinxinyuan]

LGTM!