feat: implement document AI functions

[bluejunhao]

Task Description

Implement Document AI Functions for VLDB:

1. load_file(location_name, file_name) — read a local file via LOCATION and return BLOB
2. AI_SPLIT_DOCUMENT(content[, params_json]) — table function that splits text/markdown into chunk rows (chunk_id, chunk_offset, chunk_length, chunk_text)

Solution Description

• Add and register the load_file expression operator (parser item type, factory, eval functions, CMake)
• Implement LOCATION schema lookup, READ privilege check, file:// URI restriction, and safe file loading via storage adapter
• Add MySQL-mode grammar/keyword for AI_SPLIT_DOCUMENT as a FROM table function
• Resolve fixed output columns and execute via the multi-mode table-function framework (AiSplitDocumentTableFunc)
• Support params: type (text/markdown), by (word/sentence), max, overlap

Passed Regressions

• mysql_test/test_suite/ai_funcs/t/load_file.test
• mysql_test/test_suite/ai_funcs/t/ai_split_document.test

Upgrade Compatibility

No compatibility impact. New SQL functions/table function only; no storage format or system table changes.

Other Information

Local verification: seekdb built in debug mode on macOS; both Document AI mysqltests passed against a local instance on port 2881.

Release Note

Add load_file() and AI_SPLIT_DOCUMENT() for Document AI document loading and chunking.

Made with Cursor

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 0.00 / 100
ik_custom_dict: 0 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 0.73 / 100
mean_improvement: 0.37%
full_score_improvement: 50.00%

build_improvement: -0.48%
  build_ik_all_sec: baseline=35.2836, current=35.39, improvement=-0.30%
  build_ik_content_sec: baseline=28.3764, current=28.457, improvement=-0.28%
  build_beng_en_sec: baseline=14.7578, current=14.886, improvement=-0.87%
tokenize_improvement: 1.55%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.7569, improvement=1.03%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4139, improvement=2.06%
query_improvement: 0.04%
  query_cn_avg_ms: baseline=16.6628, current=16.6161, improvement=0.28%
  query_beng_avg_ms: baseline=24.3042, current=24.1304, improvement=0.72%
  query_mixed_avg_ms: baseline=17.5593, current=17.4405, improvement=0.68%
  query_limit_avg_ms: baseline=16.2334, current=16.4787, improvement=-1.51%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-14 11:49:40
label:                  vldb-ci-29328615547-1
git_head:               8ba6a09
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2100
select1_stdev_ms:       0.0048
raw_load_sec:           1.597
raw_load_rows_per_sec:  12523.5
build_ik_all_sec:       35.390
build_ik_content_sec:   28.457
build_beng_en_sec:      14.886
build_total_sec:        78.746
----------------------------------------
tokenize_ik_avg_ms:     0.7569
tokenize_ik_median_ms:  0.7569
tokenize_ik_stdev_ms:   0.0005
tokenize_beng_avg_ms:   0.4139
tokenize_beng_median_ms:0.4138
tokenize_beng_stdev_ms: 0.0003
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.6161
query_cn_stdev_ms:      0.0300
query_beng_hits:        11000
query_beng_avg_ms:      24.1304
query_beng_stdev_ms:    0.0271
query_mixed_hits:       7332
query_mixed_avg_ms:     17.4405
query_mixed_stdev_ms:   0.0057
query_limit_hits:       20
query_limit_avg_ms:     16.4787
query_limit_stdev_ms:   0.0126
========================================

Workflow run