feat: implement document AI functions

[sallyjinx]

Task Description

Solution Description

Passed Regressions

Upgrade Compatibility

Other Information

Release Note

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ sallyjinx
❌ zhanhongliu2001
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

This PR introduces new “document AI” SQL capabilities by adding the AI_SPLIT_DOCUMENT(...) table function (parser → resolver → execution → SQL printer) and a load_file(location, filename) system function for reading files from privileged LOCATION roots.

Changes:

• Add MySQL-mode parsing, resolving, and SQL re-printing for AI_SPLIT_DOCUMENT(content[, parameters_json]) as a table function in FROM.
• Implement runtime execution for AI_SPLIT_DOCUMENT using the existing multi-mode table-function framework, producing fixed output columns (chunk_id, chunk_offset, chunk_length, chunk_text).
• Add and register the load_file expression operator, including eval-function wiring and build integration.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/sql/resolver/dml/ob_dml_stmt.h	Adds a new multi-mode table type enum value for AI document splitting.
src/sql/resolver/dml/ob_dml_resolver.h	Declares resolver helpers for AI_SPLIT_DOCUMENT table items/columns.
src/sql/resolver/dml/ob_dml_resolver.cpp	Implements AI_SPLIT_DOCUMENT resolving, table item creation, and fixed output columns.
src/sql/printer/ob_dml_stmt_printer.cpp	Prints AI_SPLIT_DOCUMENT(...) back to SQL from the statement tree.
src/sql/parser/sql_parser_mysql_mode.y	Adds AI_SPLIT_DOCUMENT token and grammar for FROM usage with optional alias.
src/sql/parser/ob_item_type.h	Introduces new item types for AI_SPLIT_DOCUMENT and LOAD_FILE.
src/sql/parser/non_reserved_keywords_mysql_mode.c	Maps ai_split_document to the MySQL parser token.
src/sql/ob_sql_define.h	Adds a new JSON-table column type for AI document split outputs.
src/sql/engine/expr/ob_expr_operator_factory.cpp	Registers the new load_file expression operator.
src/sql/engine/expr/ob_expr_load_file.h	Declares the ObExprLoadFile operator.
src/sql/engine/expr/ob_expr_load_file.cpp	Implements load_file execution with LOCATION access checks and safe openat reads.
src/sql/engine/expr/ob_expr_eval_functions.cpp	Wires ObExprLoadFile::eval_load_file into the eval-function table.
src/sql/engine/basic/ob_json_table_op.h	Adds runtime type plumbing and declares AiSplitDocumentTableFunc.
src/sql/engine/basic/ob_json_table_op.cpp	Implements document splitting logic, runtime iteration, and spec (de)serialization support.
src/sql/CMakeLists.txt	Adds ob_expr_load_file.cpp to the SQL build.
src/objit/include/objit/common/ob_item_type.h	Adds T_AI_SPLIT_DOCUMENT_EXPRESSION mirror entry in the objit enum.
deps/oblib/src/lib/ob_name_def.h	Adds the canonical function name macro N_LOAD_FILE.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[LINxiansheng]

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 0.00 / 100
mean_improvement: -16.04%
full_score_improvement: 50.00%

build_improvement: -50.25%
  build_ik_all_sec: baseline=35.2836, current=52.302, improvement=-48.23%
  build_ik_content_sec: baseline=28.3764, current=43.39, improvement=-52.91%
  build_beng_en_sec: baseline=14.7578, current=22.081, improvement=-49.62%
tokenize_improvement: 2.23%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.7557, improvement=1.19%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4088, improvement=3.27%
query_improvement: -0.08%
  query_cn_avg_ms: baseline=16.6628, current=16.6826, improvement=-0.12%
  query_beng_avg_ms: baseline=24.3042, current=24.3587, improvement=-0.22%
  query_mixed_avg_ms: baseline=17.5593, current=17.5759, improvement=-0.09%
  query_limit_avg_ms: baseline=16.2334, current=16.2138, improvement=0.12%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-16 04:09:23
label:                  vldb-ci-29469401603-1
git_head:               7b18d19
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2066
select1_stdev_ms:       0.0033
raw_load_sec:           1.473
raw_load_rows_per_sec:  13577.7
build_ik_all_sec:       52.302
build_ik_content_sec:   43.390
build_beng_en_sec:      22.081
build_total_sec:        117.789
----------------------------------------
tokenize_ik_avg_ms:     0.7557
tokenize_ik_median_ms:  0.7560
tokenize_ik_stdev_ms:   0.0011
tokenize_beng_avg_ms:   0.4088
tokenize_beng_median_ms:0.4052
tokenize_beng_stdev_ms: 0.0064
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.6826
query_cn_stdev_ms:      0.0116
query_beng_hits:        11000
query_beng_avg_ms:      24.3587
query_beng_stdev_ms:    0.0436
query_mixed_hits:       7332
query_mixed_avg_ms:     17.5759
query_mixed_stdev_ms:   0.0184
query_limit_hits:       20
query_limit_avg_ms:     16.2138
query_limit_stdev_ms:   0.0104
========================================

Workflow run