feat: implement LOAD_FILE and AI_SPLIT_DOCUMENT#1066
Open
ZHEZHEBOY wants to merge 5 commits into
Open
Conversation
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 0.00 / 100 ik_custom_dict: 0 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 4.22 / 100 mean_improvement: 2.11% full_score_improvement: 50.00% build_improvement: 1.37% build_ik_all_sec: baseline=35.2836, current=35.097, improvement=0.53% build_ik_content_sec: baseline=28.3764, current=28.124, improvement=0.89% build_beng_en_sec: baseline=14.7578, current=14.362, improvement=2.68% tokenize_improvement: 3.88% tokenize_ik_avg_ms: baseline=0.76478, current=0.7437, improvement=2.76% tokenize_beng_avg_ms: baseline=0.42262, current=0.4015, improvement=5.00% query_improvement: 1.08% query_cn_avg_ms: baseline=16.6628, current=16.4331, improvement=1.38% query_beng_avg_ms: baseline=24.3042, current=24.1097, improvement=0.80% query_mixed_avg_ms: baseline=17.5593, current=17.3522, improvement=1.18% query_limit_avg_ms: baseline=16.2334, current=16.0773, improvement=0.96% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-14 18:24:55 label: vldb-ci-29355666698-1 git_head: 708c302 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2072 select1_stdev_ms: 0.0020 raw_load_sec: 1.540 raw_load_rows_per_sec: 12987.0 build_ik_all_sec: 35.097 build_ik_content_sec: 28.124 build_beng_en_sec: 14.362 build_total_sec: 77.596 ---------------------------------------- tokenize_ik_avg_ms: 0.7437 tokenize_ik_median_ms: 0.7419 tokenize_ik_stdev_ms: 0.0029 tokenize_beng_avg_ms: 0.4015 tokenize_beng_median_ms:0.4010 tokenize_beng_stdev_ms: 0.0009 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 16.4331 query_cn_stdev_ms: 0.0058 query_beng_hits: 11000 query_beng_avg_ms: 24.1097 query_beng_stdev_ms: 0.0312 query_mixed_hits: 7332 query_mixed_avg_ms: 17.3522 query_mixed_stdev_ms: 0.0486 query_limit_hits: 20 query_limit_avg_ms: 16.0773 query_limit_stdev_ms: 0.0098 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement the two DBSchool 2026 Document AI functions requested for seekdb:
LOAD_FILE(location_name, filename)AI_SPLIT_DOCUMENT(content[, params_json])The change is limited to the parser/resolver/execution/serialization/registration/build wiring needed by those two functions. It does not modify any official test or result file.
Implementation
LOAD_FILE
LOAD_FILE.document_ai_file_max_size, reads the file, and returns a BLOB.Core files:
src/sql/engine/expr/ob_expr_ai/ob_expr_load_file.{h,cpp}AI_SPLIT_DOCUMENT
max, andoverlap.chunk_id,chunk_offset,chunk_length, andchunk_text.Core files:
deps/oblib/src/lib/ai_split_document/*src/sql/engine/basic/ob_json_table_op.{h,cpp}Scope review
Validation
Environment: Ubuntu 22.04, 16 vCPU, targeted build with
-j8.PASS
bash build.sh initbash build.sh debug -DOB_USE_CCACHE=ON -DNEED_PARSER_CACHE=OFFmake -j8 module_layer_check oblib_lib ob_sqlmake -j8 observer17e03803463db1171f7c0682c16c6ad17972f88etest_utility: 1/1ai_funcs/load_file:okai_funcs/ai_split_document:okmax=0rejectiongit diff --checkFAIL
test_sql_fast_parserresult comparison failed.vldb_2026baseline (a640cc3), so this is not caused by this PR. No expected result was modified.NOT RUN
Remaining validation
Official GitHub Actions are expected to run the repository-wide checks for the PR.