Skip to content

feat: implement LOAD_FILE and AI_SPLIT_DOCUMENT#1066

Open
ZHEZHEBOY wants to merge 5 commits into
oceanbase:vldb_2026from
ZHEZHEBOY:document-ai
Open

feat: implement LOAD_FILE and AI_SPLIT_DOCUMENT#1066
ZHEZHEBOY wants to merge 5 commits into
oceanbase:vldb_2026from
ZHEZHEBOY:document-ai

Conversation

@ZHEZHEBOY

Copy link
Copy Markdown

Summary

Implement the two DBSchool 2026 Document AI functions requested for seekdb:

  • LOAD_FILE(location_name, filename)
  • AI_SPLIT_DOCUMENT(content[, params_json])

The change is limited to the parser/resolver/execution/serialization/registration/build wiring needed by those two functions. It does not modify any official test or result file.

Implementation

LOAD_FILE

  • Adds scalar expression registration and code generation for LOAD_FILE.
  • Resolves a named LOCATION, checks access, builds the object path, enforces document_ai_file_max_size, reads the file, and returns a BLOB.
  • Preserves NULL propagation and reports invalid names, missing files, empty files, access errors, and unsupported sizes through existing seekdb error paths.

Core files:

  • src/sql/engine/expr/ob_expr_ai/ob_expr_load_file.{h,cpp}
  • expression type, factory, evaluator, code-generation, and parameter registration files

AI_SPLIT_DOCUMENT

  • Adds MySQL grammar, resolver, statement printing, serialization, and JSON-table execution wiring for the table function.
  • Supports text and Markdown input, sentence/word splitting, max, and overlap.
  • Returns chunk_id, chunk_offset, chunk_length, and chunk_text.
  • Keeps JSON parsing in the SQL layer and passes plain parameter values to oblib, preserving module-layer boundaries and allocator lifetimes.

Core files:

  • deps/oblib/src/lib/ai_split_document/*
  • src/sql/engine/basic/ob_json_table_op.{h,cpp}
  • parser/resolver/printer/item-type/build registration files

Scope review

  • Final diff: 25 files, 1,876 insertions, 9 deletions.
  • Scope classification: EXPECTED.
  • No generated source, test/result changes, unrelated formatting, macOS compatibility code, build artifacts, logs, absolute local paths, credentials, or unrelated features are included.
  • The file count is consistent with the required end-to-end SQL plumbing and is smaller than the linked upstream Document AI reference implementation.

Validation

Environment: Ubuntu 22.04, 16 vCPU, targeted build with -j8.

PASS

  • bash build.sh init
  • bash build.sh debug -DOB_USE_CCACHE=ON -DNEED_PARSER_CACHE=OFF
  • make -j8 module_layer_check oblib_lib ob_sql
  • make -j8 observer
  • built binary revision: 17e03803463db1171f7c0682c16c6ad17972f88e
  • observer startup with explicit data-disk resource limits
  • test_utility: 1/1
  • official mysqltest ai_funcs/load_file: ok
  • official mysqltest ai_funcs/ai_split_document: ok
  • result-file SHA-256 values unchanged before and after mysqltest
  • supplementary runtime checks:
    • empty, short, and Unicode split input
    • invalid max=0 rejection
    • empty and missing-file LOAD_FILE error paths
  • git diff --check
  • clean tracked server worktree after exact removal of CMake-generated untracked files

FAIL

  • Generic test_sql_fast_parser result comparison failed.
  • The exact command and output hashes were reproduced on the untouched vldb_2026 baseline (a640cc3), so this is not caused by this PR. No expected result was modified.

NOT RUN

  • The complete repository-wide mysqltest/unit-test suite was not run because this PR is intentionally limited to the two Document AI functions and the requested workflow prioritizes targeted validation.
  • No test requiring unrelated services or modules was represented as passing.

Remaining validation

Official GitHub Actions are expected to run the repository-wide checks for the PR.

@CLAassistant

CLAassistant commented Jul 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@LINxiansheng

Copy link
Copy Markdown
Member

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 0.00 / 100
ik_custom_dict: 0 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 4.22 / 100
mean_improvement: 2.11%
full_score_improvement: 50.00%

build_improvement: 1.37%
  build_ik_all_sec: baseline=35.2836, current=35.097, improvement=0.53%
  build_ik_content_sec: baseline=28.3764, current=28.124, improvement=0.89%
  build_beng_en_sec: baseline=14.7578, current=14.362, improvement=2.68%
tokenize_improvement: 3.88%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.7437, improvement=2.76%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4015, improvement=5.00%
query_improvement: 1.08%
  query_cn_avg_ms: baseline=16.6628, current=16.4331, improvement=1.38%
  query_beng_avg_ms: baseline=24.3042, current=24.1097, improvement=0.80%
  query_mixed_avg_ms: baseline=17.5593, current=17.3522, improvement=1.18%
  query_limit_avg_ms: baseline=16.2334, current=16.0773, improvement=0.96%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-14 18:24:55
label:                  vldb-ci-29355666698-1
git_head:               708c302
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2072
select1_stdev_ms:       0.0020
raw_load_sec:           1.540
raw_load_rows_per_sec:  12987.0
build_ik_all_sec:       35.097
build_ik_content_sec:   28.124
build_beng_en_sec:      14.362
build_total_sec:        77.596
----------------------------------------
tokenize_ik_avg_ms:     0.7437
tokenize_ik_median_ms:  0.7419
tokenize_ik_stdev_ms:   0.0029
tokenize_beng_avg_ms:   0.4015
tokenize_beng_median_ms:0.4010
tokenize_beng_stdev_ms: 0.0009
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.4331
query_cn_stdev_ms:      0.0058
query_beng_hits:        11000
query_beng_avg_ms:      24.1097
query_beng_stdev_ms:    0.0312
query_mixed_hits:       7332
query_mixed_avg_ms:     17.3522
query_mixed_stdev_ms:   0.0486
query_limit_hits:       20
query_limit_avg_ms:     16.0773
query_limit_stdev_ms:   0.0098
========================================

Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants