feat: implement Document AI functions and IK custom dictionary support

[jbjvhvhh]

Task Description

Implement and integrate Document AI Functions and IK custom dictionary support for the VLDB 2026 tasks.

Solution Description

• Implement LOAD_FILE.
• Implement AI_SPLIT_DOCUMENT.
• Support IK custom dictionaries, stopword dictionaries and quantifier dictionaries.
• Resolve parser item type conflicts between the two implementations.
• Preserve the full-text tokenizer performance optimizations.

Test

• Full observer build passed.
• load_file: passed.
• ai_split_document: passed.
• ik_custom_dict: passed.
• Full-text benchmark query results are correct.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ bluejunhao
❌ jbjvhvhh

---

jbjvhvhh seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 0.00 / 100
mean_improvement: -0.16%
full_score_improvement: 50.00%

build_improvement: 2.09%
  build_ik_all_sec: baseline=35.2836, current=34.432, improvement=2.41%
  build_ik_content_sec: baseline=28.3764, current=28.432, improvement=-0.20%
  build_beng_en_sec: baseline=14.7578, current=14.161, improvement=4.04%
tokenize_improvement: -2.68%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.8089, improvement=-5.77%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4209, improvement=0.41%
query_improvement: 0.11%
  query_cn_avg_ms: baseline=16.6628, current=16.769, improvement=-0.64%
  query_beng_avg_ms: baseline=24.3042, current=24.2489, improvement=0.23%
  query_mixed_avg_ms: baseline=17.5593, current=17.5083, improvement=0.29%
  query_limit_avg_ms: baseline=16.2334, current=16.1402, improvement=0.57%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-14 16:04:05
label:                  vldb-ci-29345819260-1
git_head:               f6731b9
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2119
select1_stdev_ms:       0.0114
raw_load_sec:           1.471
raw_load_rows_per_sec:  13596.2
build_ik_all_sec:       34.432
build_ik_content_sec:   28.432
build_beng_en_sec:      14.161
build_total_sec:        77.037
----------------------------------------
tokenize_ik_avg_ms:     0.8089
tokenize_ik_median_ms:  0.8102
tokenize_ik_stdev_ms:   0.0030
tokenize_beng_avg_ms:   0.4209
tokenize_beng_median_ms:0.4100
tokenize_beng_stdev_ms: 0.0178
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.7690
query_cn_stdev_ms:      0.0094
query_beng_hits:        11000
query_beng_avg_ms:      24.2489
query_beng_stdev_ms:    0.0015
query_mixed_hits:       7332
query_mixed_avg_ms:     17.5083
query_mixed_stdev_ms:   0.0239
query_limit_hits:       20
query_limit_avg_ms:     16.1402
query_limit_stdev_ms:   0.0292
========================================

Workflow run