feat: add LOAD_FILE and AI_SPLIT_DOCUMENT Document AI functions#1059
Open
huangda99 wants to merge 3 commits into
Open
feat: add LOAD_FILE and AI_SPLIT_DOCUMENT Document AI functions#1059huangda99 wants to merge 3 commits into
huangda99 wants to merge 3 commits into
Conversation
Implement the two Document AI functions for the ai_funcs test suite: LOAD_FILE(location_name, file_name) -> BLOB - new scalar sys expr ObExprLoadFile (T_FUN_SYS_LOAD_FILE): looks up the LOCATION schema by name, joins its URL with the file name and reads the file through ObBackupIoAdapter, returning the content as a binary lob AI_SPLIT_DOCUMENT(content [, parameters]) table function - new non-reserved keyword + grammar rule producing T_AI_SPLIT_DOCUMENT_EXPRESSION, resolved as a JSON_TABLE-family table item (OB_AI_SPLIT_DOC_TABLE_TYPE) with four fixed output columns chunk_id / chunk_offset / chunk_length / chunk_text - AiSplitDocTableFunc splits the document per parameters json: type text|markdown (default markdown), by word|sentence (default word), max units per chunk (default 256), overlap for sliding windows; markdown mode sections the text at '#' heading lines and prefixes each chunk with its section heading - spec serialization, printer (view expansion) and rescan support Co-Authored-By: Claude Fable 5 <[email protected]>
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 3.96 / 100 mean_improvement: 1.98% full_score_improvement: 50.00% build_improvement: 2.32% build_ik_all_sec: baseline=35.2836, current=34.725, improvement=1.58% build_ik_content_sec: baseline=28.3764, current=27.218, improvement=4.08% build_beng_en_sec: baseline=14.7578, current=14.568, improvement=1.29% tokenize_improvement: 4.68% tokenize_ik_avg_ms: baseline=0.76478, current=0.7167, improvement=6.29% tokenize_beng_avg_ms: baseline=0.42262, current=0.4096, improvement=3.08% query_improvement: -1.06% query_cn_avg_ms: baseline=16.6628, current=16.5529, improvement=0.66% query_beng_avg_ms: baseline=24.3042, current=24.5542, improvement=-1.03% query_mixed_avg_ms: baseline=17.5593, current=17.666, improvement=-0.61% query_limit_avg_ms: baseline=16.2334, current=16.7657, improvement=-3.28% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-15 16:05:23 label: vldb-ci-29427344545-1 git_head: 27b5983 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2063 select1_stdev_ms: 0.0028 raw_load_sec: 1.474 raw_load_rows_per_sec: 13568.5 build_ik_all_sec: 34.725 build_ik_content_sec: 27.218 build_beng_en_sec: 14.568 build_total_sec: 76.523 ---------------------------------------- tokenize_ik_avg_ms: 0.7167 tokenize_ik_median_ms: 0.7173 tokenize_ik_stdev_ms: 0.0010 tokenize_beng_avg_ms: 0.4096 tokenize_beng_median_ms:0.4087 tokenize_beng_stdev_ms: 0.0013 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 16.5529 query_cn_stdev_ms: 0.0831 query_beng_hits: 11000 query_beng_avg_ms: 24.5542 query_beng_stdev_ms: 0.1386 query_mixed_hits: 7332 query_mixed_avg_ms: 17.6660 query_mixed_stdev_ms: 0.2234 query_limit_hits: 20 query_limit_avg_ms: 16.7657 query_limit_stdev_ms: 0.1729 ======================================== |
Implement the Custom Dictionary feature for the IK fulltext parser: - CREATE TABLE ... FULLTEXT_DICT='Y' table option: new FULLTEXT_DICT non-reserved keyword and table option, accepted for CREATE TABLE (value must be 'Y'/'N', rejected in ALTER TABLE like ORGANIZATION). - ALTER SYSTEM REFRESH FULLTEXT DICT [db.]table statement: new DICT keyword, T_REFRESH_FULLTEXT_DICT parse node, resolver, stmt and executor wired through the standard system-command chain. Accepts the unquoted db.table / table forms and the double-quoted string forms. Custom dictionaries are re-read from the dictionary table on every parser instantiation, so the refresh command has nothing to invalidate and newly written words become visible immediately. - Runtime: ObFTParserProperty now really extracts dict_table / stopword_table / quantifier_table values from the parser-properties JSON (deep-copied into owned buffers), ObFTParseHelper passes them into ObFTIKParam, and ObIKFTParser::init_dict loads a dictionary table (SELECT LOWER(word) ... ORDER BY LOWER(word) COLLATE utf8mb4_bin via inner SQL, then trie -> DAT -> ObFTCacheDict) when the configured table differs from the built-in one, REPLACING the corresponding built-in dictionary. An empty dictionary table maps to the new ObFTEmptyDict which never matches. Covers tools/deploy/mysql_test/test_suite/ai_funcs/t/ik_custom_dict.test: verified byte-exact against the recorded .result (custom words match, words missing from the custom dict degrade to single characters and no longer match, dynamic INSERT + REFRESH takes effect for new rows), and the built-in IK path (no dict_table property) is unchanged. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Optimize the IK/beng full-text tokenization paths that dominate fulltext index build, TOKENIZE() and MATCH query-token segmentation: - Share the three built-in IK dictionaries (main/quantifier/stopword) process-wide in ObFTParsePluginData: they are immutable, so building the range dict once and borrowing it from every parser instance removes the per-document KV-cache lookups and dictionary object rebuilds that used to run on every tokenization. Custom dictionary tables (FULLTEXT_DICT='Y') keep the per-instance build so REFRESH semantics are unchanged. - Classify each character with a single unicode decode in ObFTCharUtil::do_classify instead of re-decoding for every category check (up to 4 decodes per CJK char before). - Fetch char, length and type in one TokenizeContext::current_char_and_type call in the two per-character hot loops. - Store IK segmenters in a fixed array instead of an allocator-backed list. - Skip the per-token charset conversion (and its allocation) in ObStopWordChecker::check_stopword when source and stopword charsets match. All ai_funcs mysqltest cases still pass byte-exact and tokenization output is unchanged; local fts_large_bench shows build_ik_all -19%, tokenize_ik -37%, tokenize_beng -22% at identical hit counts. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement the two Document AI functions for the ai_funcs test suite:
LOAD_FILE(location_name, file_name) -> BLOB
AI_SPLIT_DOCUMENT(content [, parameters]) table function
team：大海好多水