[VLDB 2026][Task 4] Optimize fulltext index hot paths

[ShilongLiu4]

Task Description

VLDB 2026 training Task 4: improve full-text index construction, tokenization, and BM25 query hot paths.

Solution Description

• Reuse IK parser metadata and processors while resetting per-document scratch state.
• Add ASCII and Unicode classification fast paths for English and IK tokenization.
• Reduce stop-word and word-frequency lookup overhead.
• Reduce BM25/DAAT/BMW query hot-path datum, comparator, iterator, and debug-log overhead.

Passed Regressions

• make -j20 observer
• English and IK tokenization output is identical to the pre-change output.
• Query hit counts remain unchanged: 8001 / 11000 / 7332 / 20.
• Top-50 document ID ranking remains unchanged.
• git diff --check
• Local official-parameter scorer: 87.88 / 100 (local estimate only; official CI result is pending).

Upgrade Compatibility

No SQL syntax, public API, or on-disk format changes.

Other Information

This is a Task 4 scoring PR targeting vldb_2026. It is not the final team submission and should not be merged as a team submission.

Release Note

Improve full-text indexing, tokenization, and BM25 query performance without changing result semantics.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 0.00 / 100
load_file: 0 / 50
ai_split_document: 0 / 50

IK Custom Dictionary Score
==========================
score: 0.00 / 100
ik_custom_dict: 0 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 9.99 / 100
mean_improvement: 4.99%
full_score_improvement: 50.00%

build_improvement: 5.09%
  build_ik_all_sec: baseline=35.2836, current=32.96, improvement=6.59%
  build_ik_content_sec: baseline=28.3764, current=27.112, improvement=4.46%
  build_beng_en_sec: baseline=14.7578, current=14.133, improvement=4.23%
tokenize_improvement: -3.67%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.7766, improvement=-1.55%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4471, improvement=-5.79%
query_improvement: 13.56%
  query_cn_avg_ms: baseline=16.6628, current=14.5465, improvement=12.70%
  query_beng_avg_ms: baseline=24.3042, current=20.8786, improvement=14.09%
  query_mixed_avg_ms: baseline=17.5593, current=15.2757, improvement=13.00%
  query_limit_avg_ms: baseline=16.2334, current=13.8894, improvement=14.44%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-15 16:49:08
label:                  vldb-ci-29432073705-2
git_head:               7610a7f
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2655
select1_stdev_ms:       0.0015
raw_load_sec:           1.542
raw_load_rows_per_sec:  12970.2
build_ik_all_sec:       32.960
build_ik_content_sec:   27.112
build_beng_en_sec:      14.133
build_total_sec:        74.223
----------------------------------------
tokenize_ik_avg_ms:     0.7766
tokenize_ik_median_ms:  0.7772
tokenize_ik_stdev_ms:   0.0009
tokenize_beng_avg_ms:   0.4471
tokenize_beng_median_ms:0.4442
tokenize_beng_stdev_ms: 0.0042
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        14.5465
query_cn_stdev_ms:      0.0123
query_beng_hits:        11000
query_beng_avg_ms:      20.8786
query_beng_stdev_ms:    0.0097
query_mixed_hits:       7332
query_mixed_avg_ms:     15.2757
query_mixed_stdev_ms:   0.0448
query_limit_hits:       20
query_limit_avg_ms:     13.8894
query_limit_stdev_ms:   0.0023
========================================

Workflow run