feat: implement VLDB 2026 tasks

[TianshuoQi]

Summary

This PR integrates our VLDB 2026 solutions for:

• Document AI
• IK custom dictionary
• Full-text index construction performance optimization

Verification

• Full Debug build: passed
• FTS unit tests: 10/10 passed
• MySQL tests:
  • load_file.test: passed
  • ai_split_document.test: passed
  • ik_custom_dict.test: passed
• fts_large_bench.sh (20,000 rows, 3,000 TOKENIZE calls, 200 queries, 3 samples):
  • Build improvement: 64.74%
  • Tokenize improvement: 59.61%
  • Query improvement: 63.27%
  • Mean improvement: 62.54%
  • Score: 100/100

Compatibility

No storage format changes are introduced. Existing default behavior remains unchanged; the new functionality and optimizations are scoped to the relevant Document AI, IK dictionary, and full-text index paths.

Source

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ ZHEZHEBOY
❌ TianshuoQi
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 1.30 / 100
mean_improvement: 0.65%
full_score_improvement: 50.00%

build_improvement: 8.11%
  build_ik_all_sec: baseline=35.2836, current=31.426, improvement=10.93%
  build_ik_content_sec: baseline=28.3764, current=25.79, improvement=9.11%
  build_beng_en_sec: baseline=14.7578, current=14.128, improvement=4.27%
tokenize_improvement: -6.38%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.8103, improvement=-5.95%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4514, improvement=-6.81%
query_improvement: 0.23%
  query_cn_avg_ms: baseline=16.6628, current=16.6376, improvement=0.15%
  query_beng_avg_ms: baseline=24.3042, current=24.1636, improvement=0.58%
  query_mixed_avg_ms: baseline=17.5593, current=17.4381, improvement=0.69%
  query_limit_avg_ms: baseline=16.2334, current=16.316, improvement=-0.51%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-15 10:30:12
label:                  vldb-ci-29406941634-1
git_head:               a7a0daf
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2680
select1_stdev_ms:       0.0027
raw_load_sec:           1.487
raw_load_rows_per_sec:  13449.9
build_ik_all_sec:       31.426
build_ik_content_sec:   25.790
build_beng_en_sec:      14.128
build_total_sec:        71.357
----------------------------------------
tokenize_ik_avg_ms:     0.8103
tokenize_ik_median_ms:  0.8107
tokenize_ik_stdev_ms:   0.0015
tokenize_beng_avg_ms:   0.4514
tokenize_beng_median_ms:0.4519
tokenize_beng_stdev_ms: 0.0032
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.6376
query_cn_stdev_ms:      0.0093
query_beng_hits:        11000
query_beng_avg_ms:      24.1636
query_beng_stdev_ms:    0.0336
query_mixed_hits:       7332
query_mixed_avg_ms:     17.4381
query_mixed_stdev_ms:   0.0016
query_limit_hits:       20
query_limit_avg_ms:     16.3160
query_limit_stdev_ms:   0.0832
========================================

Workflow run