perf: optimize full-text index build performance

[jbjvhvhh]

Task Description

Improve full-text index build performance while preserving existing tokenization and query behavior.

Solution Description

• Optimize tokenizer hot paths through parser reuse, reusable token containers, cached metadata, and reduced temporary memory allocation.
• Stream generated full-text index rows directly from the token map instead of materializing all rows for each document, reducing allocation and copying overhead.

Passed Regressions

• Full Debug observer build passed.
• ik_custom_dict.test passed.
• Basic TOKENIZE checks for ik, space, ngram, and ngram2 passed.
• Three rounds of fts_large_bench.sh completed successfully.
• All expected query hit counts remained correct:
  • query_cn_hits = 8001
  • query_beng_hits = 11000
  • query_mixed_hits = 7332
  • query_limit_hits = 20

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

jbjvhvhh seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 0.00 / 100
load_file: 0 / 50
ai_split_document: 0 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 0.00 / 100
mean_improvement: -0.18%
full_score_improvement: 50.00%

build_improvement: 4.30%
  build_ik_all_sec: baseline=35.2836, current=33.368, improvement=5.43%
  build_ik_content_sec: baseline=28.3764, current=27.779, improvement=2.11%
  build_beng_en_sec: baseline=14.7578, current=13.967, improvement=5.36%
tokenize_improvement: -4.51%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.8183, improvement=-7.00%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4312, improvement=-2.03%
query_improvement: -0.32%
  query_cn_avg_ms: baseline=16.6628, current=16.8137, improvement=-0.91%
  query_beng_avg_ms: baseline=24.3042, current=24.2664, improvement=0.16%
  query_mixed_avg_ms: baseline=17.5593, current=17.5968, improvement=-0.21%
  query_limit_avg_ms: baseline=16.2334, current=16.2849, improvement=-0.32%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-15 06:50:46
label:                  vldb-ci-29393891197-1
git_head:               40e63af
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2042
select1_stdev_ms:       0.0016
raw_load_sec:           1.947
raw_load_rows_per_sec:  10272.2
build_ik_all_sec:       33.368
build_ik_content_sec:   27.779
build_beng_en_sec:      13.967
build_total_sec:        75.131
----------------------------------------
tokenize_ik_avg_ms:     0.8183
tokenize_ik_median_ms:  0.8179
tokenize_ik_stdev_ms:   0.0023
tokenize_beng_avg_ms:   0.4312
tokenize_beng_median_ms:0.4148
tokenize_beng_stdev_ms: 0.0239
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.8137
query_cn_stdev_ms:      0.0275
query_beng_hits:        11000
query_beng_avg_ms:      24.2664
query_beng_stdev_ms:    0.0185
query_mixed_hits:       7332
query_mixed_avg_ms:     17.5968
query_mixed_stdev_ms:   0.0678
query_limit_hits:       20
query_limit_avg_ms:     16.2849
query_limit_stdev_ms:   0.0199
========================================

Workflow run