Skip to content

[feat|WHT] add LOAD_FILE and AI_SPLIT_DOCUMENT#1061

Open
wht0not wants to merge 4 commits into
oceanbase:vldb_2026from
wht0not:vldb_2026-wht
Open

[feat|WHT] add LOAD_FILE and AI_SPLIT_DOCUMENT#1061
wht0not wants to merge 4 commits into
oceanbase:vldb_2026from
wht0not:vldb_2026-wht

Conversation

@wht0not

@wht0not wht0not commented Jul 14, 2026

Copy link
Copy Markdown

Description

Implement the two Document AI functions required by task 2:

  • Add LOAD_FILE(location_name, file_name) to read files from a local file:// LOCATION and return BLOB data.
  • Add AI_SPLIT_DOCUMENT(content[, parameters]) to split text or Markdown by word or sentence with configurable max and overlap.
  • Register the new expressions and integrate parser, resolver, runtime evaluation, and build configuration.

Testing

  • Successfully built seekdb.exe on Windows.
  • Verified both functions against the existing ai_funcs mysqltest expectations.
  • PyMySQL acceptance tests pass for file loading, sentence splitting, word overlap, Markdown headings, and max=2.

@LINxiansheng

LINxiansheng commented Jul 14, 2026

Copy link
Copy Markdown
Member

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 100.00 / 100
load_file: 50 / 50
ai_split_document: 50 / 50

IK Custom Dictionary Score
==========================
score: 100.00 / 100
ik_custom_dict: 100 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 0.00 / 100
mean_improvement: -2.42%
full_score_improvement: 50.00%

build_improvement: 4.51%
  build_ik_all_sec: baseline=35.2836, current=32.767, improvement=7.13%
  build_ik_content_sec: baseline=28.3764, current=26.832, improvement=5.44%
  build_beng_en_sec: baseline=14.7578, current=14.617, improvement=0.95%
tokenize_improvement: -12.51%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.8587, improvement=-12.28%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.4765, improvement=-12.75%
query_improvement: 0.75%
  query_cn_avg_ms: baseline=16.6628, current=16.5968, improvement=0.40%
  query_beng_avg_ms: baseline=24.3042, current=24.161, improvement=0.59%
  query_mixed_avg_ms: baseline=17.5593, current=17.4775, improvement=0.47%
  query_limit_avg_ms: baseline=16.2334, current=15.985, improvement=1.53%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-15 08:27:04
label:                  vldb-ci-29398698650-1
git_head:               b7ee6c7
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2732
select1_stdev_ms:       0.0035
raw_load_sec:           1.486
raw_load_rows_per_sec:  13459.0
build_ik_all_sec:       32.767
build_ik_content_sec:   26.832
build_beng_en_sec:      14.617
build_total_sec:        74.230
----------------------------------------
tokenize_ik_avg_ms:     0.8587
tokenize_ik_median_ms:  0.8596
tokenize_ik_stdev_ms:   0.0020
tokenize_beng_avg_ms:   0.4765
tokenize_beng_median_ms:0.4756
tokenize_beng_stdev_ms: 0.0016
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        16.5968
query_cn_stdev_ms:      0.0153
query_beng_hits:        11000
query_beng_avg_ms:      24.1610
query_beng_stdev_ms:    0.0212
query_mixed_hits:       7332
query_mixed_avg_ms:     17.4775
query_mixed_stdev_ms:   0.0729
query_limit_hits:       20
query_limit_avg_ms:     15.9850
query_limit_stdev_ms:   0.0137
========================================

Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants