Skip to content

feat: implement LOAD_FILE scalar function for Document AI#1071

Open
yanxf-git wants to merge 1 commit into
oceanbase:vldb_2026from
yanxf-git:feat/load-file
Open

feat: implement LOAD_FILE scalar function for Document AI#1071
yanxf-git wants to merge 1 commit into
oceanbase:vldb_2026from
yanxf-git:feat/load-file

Conversation

@yanxf-git

Copy link
Copy Markdown

Implements BLOB LOAD_FILE(location_name, file_name) that reads a local file via a file:// LOCATION. Registered as T_FUN_SYS_LOAD_FILE=2087.

Adapted from OceanBase commit 28c25842 with seekdb-specific changes:

  • Use POSIX open/pread instead of ObExternalDataAccessDriver (not in seekdb)
  • Use 1-arg get_tenant_schema_guard (seekdb signature)
  • Use 2-arg get_location_schema_by_name (seekdb signature)
  • Drop MultimodeAlloctor (not needed for simple file read)
  • Use ObString::prefix_match instead of 3-arg compare

Local verification: mysqltest ai_funcs/load_file.test PASS

Task Description

Solution Description

Passed Regressions

Upgrade Compatibility

Other Information

Release Note

Implements BLOB LOAD_FILE(location_name, file_name) that reads a local
file via a file:// LOCATION. Registered as T_FUN_SYS_LOAD_FILE=2087.

Adapted from OceanBase commit 28c25842 with seekdb-specific changes:
- Use POSIX open/pread instead of ObExternalDataAccessDriver (not in seekdb)
- Use 1-arg get_tenant_schema_guard (seekdb signature)
- Use 2-arg get_location_schema_by_name (seekdb signature)
- Drop MultimodeAlloctor (not needed for simple file read)
- Use ObString::prefix_match instead of 3-arg compare

Local verification: mysqltest ai_funcs/load_file.test PASS
@CLAassistant

CLAassistant commented Jul 15, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@LINxiansheng

Copy link
Copy Markdown
Member

Document AI & IK Custom Dictionary Score

Document AI Functions Score
===========================
score: 50.00 / 100
load_file: 50 / 50
ai_split_document: 0 / 50

IK Custom Dictionary Score
==========================
score: 0.00 / 100
ik_custom_dict: 0 / 100

FTS Large Benchmark Score

FTS Large Benchmark Score
=========================
score: 0.00 / 100
mean_improvement: -2.32%
full_score_improvement: 50.00%

build_improvement: -2.34%
  build_ik_all_sec: baseline=35.2836, current=36.057, improvement=-2.19%
  build_ik_content_sec: baseline=28.3764, current=28.903, improvement=-1.86%
  build_beng_en_sec: baseline=14.7578, current=15.198, improvement=-2.98%
tokenize_improvement: -0.97%
  tokenize_ik_avg_ms: baseline=0.76478, current=0.7554, improvement=1.23%
  tokenize_beng_avg_ms: baseline=0.42262, current=0.436, improvement=-3.17%
query_improvement: -3.64%
  query_cn_avg_ms: baseline=16.6628, current=17.3785, improvement=-4.30%
  query_beng_avg_ms: baseline=24.3042, current=25.1852, improvement=-3.62%
  query_mixed_avg_ms: baseline=17.5593, current=18.1195, improvement=-3.19%
  query_limit_avg_ms: baseline=16.2334, current=16.7947, improvement=-3.46%

FTS Large Benchmark Report

========================================
FTS Large Benchmark Report
========================================
timestamp:              2026-07-15 08:39:38
label:                  vldb-ci-29399798056-1
git_head:               8143909
git_dirty:              0
rows:                   20000
batch:                  500
rounds:                 3000
query_rounds:           200
samples:                3
warmup:                 30
skip_load:              0
----------------------------------------
select1_avg_ms:         0.2097
select1_stdev_ms:       0.0010
raw_load_sec:           1.610
raw_load_rows_per_sec:  12422.4
build_ik_all_sec:       36.057
build_ik_content_sec:   28.903
build_beng_en_sec:      15.198
build_total_sec:        80.170
----------------------------------------
tokenize_ik_avg_ms:     0.7554
tokenize_ik_median_ms:  0.7556
tokenize_ik_stdev_ms:   0.0008
tokenize_beng_avg_ms:   0.4360
tokenize_beng_median_ms:0.4270
tokenize_beng_stdev_ms: 0.0147
----------------------------------------
query_cn_hits:          8001
query_cn_avg_ms:        17.3785
query_cn_stdev_ms:      0.0148
query_beng_hits:        11000
query_beng_avg_ms:      25.1852
query_beng_stdev_ms:    0.0373
query_mixed_hits:       7332
query_mixed_avg_ms:     18.1195
query_mixed_stdev_ms:   0.0554
query_limit_hits:       20
query_limit_avg_ms:     16.7947
query_limit_stdev_ms:   0.2910
========================================

Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants