feat: support custom dictionaries for IK parser#1054
Open
TianshuoQi wants to merge 1 commit into
Open
Conversation
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end support for user-defined (table-backed) IK dictionaries, including schema validation, dependency/drop protection, cache isolation, and a system command to refresh dictionary caches. This extends FTS parser configuration, DDL parsing/printing, and TOKENIZE() so custom dictionaries can be configured and used while keeping compatibility with existing IK properties.
Changes:
- Extend IK parser properties to persist dictionary table names + IDs (including legacy
quanitfier_tableread-compat), and propagate these properties through parser helper andTOKENIZE(). - Introduce
FULLTEXT_DICTtable option + schema validation, record dictionary dependencies for FTS indexes, and prevent dropping dictionary tables that are still referenced. - Add
ALTER SYSTEM REFRESH FULLTEXT DICTto force-refresh dictionary cache generations and isolate cache keys by tenant/table/generation/range.
Reviewed changes
Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| unittest/storage/test_fts_property.cpp | Extends unit coverage for parser-property JSON (table IDs) and SHOW output. |
| unittest/storage/fts/test_ft_parser.cpp | Updates cache-key tests and DAT hash map type rename usage. |
| src/storage/fts/ob_ik_ft_parser.cpp | Loads dictionary names/IDs into ObFTDictDesc and enables stopword matching. |
| src/storage/fts/ob_fts_plugin_helper.cpp | Propagates dictionary table names/IDs into parser params during segmentation. |
| src/storage/fts/ob_fts_parser_property.h | Adds dict table ID accessors + TOKENIZE helper overload and allocator plumbing. |
| src/storage/fts/ob_fts_parser_property.cpp | Implements dict table IDs, legacy quantifier spelling read, SHOW output, and TOKENIZE table resolution. |
| src/storage/fts/ob_fts_literal.h | Fixes quantifier_table spelling and adds legacy + table-id config literals. |
| src/storage/fts/dict/ob_ft_range_dict.cpp | Switches range-cache build/load to use tenant/table/generation keys; routes custom dicts to table iterator. |
| src/storage/fts/dict/ob_ft_dict_table_iter.h | Changes iterator init to accept ObFTDictDesc and tracks iteration end. |
| src/storage/fts/dict/ob_ft_dict_table_iter.cpp | Implements table-backed dictionary scanning with optional casedown + DISTINCT. |
| src/storage/fts/dict/ob_ft_dict_hub.h | Changes dict-info keying to tenant+table and adds refresh/build internal helper. |
| src/storage/fts/dict/ob_ft_dict_hub.cpp | Adds refresh support via cache generations and publishes latest generation/range count. |
| src/storage/fts/dict/ob_ft_dict_def.h | Expands ObFTDictDesc to carry tenant/table/generation/builtin/casedown fields. |
| src/storage/fts/dict/ob_ft_dat_dict.h | Renames internal fixed hash map type to ObFTArrayHashMap. |
| src/storage/fts/dict/ob_ft_dat_dict.cpp | Updates DAT builder/reader to use ObFTArrayHashMap. |
| src/storage/fts/dict/ob_ft_cache.h | Redefines cache key to include tenant/table/generation/range and updates hash/equals/deep-copy. |
| src/storage/fts/dict/ob_ft_cache_dict.cpp | Updates cache put/fetch to use the new cache key dimensions. |
| src/sql/resolver/ob_stmt_type.h | Adds statement type for REFRESH FULLTEXT DICT with privilege mapping. |
| src/sql/resolver/ob_resolver.cpp | Registers resolver for the new refresh statement. |
| src/sql/resolver/ddl/ob_fts_parser_resolver.h | Adds schema-aware resolution of dictionary table names/IDs into parser properties. |
| src/sql/resolver/ddl/ob_fts_parser_resolver.cpp | Resolves dict table name+id via schema guard; refactors table-config parsing. |
| src/sql/resolver/ddl/ob_fts_index_builder_util.h | Declares fulltext dict schema validation + dependency tracking + drop protection. |
| src/sql/resolver/ddl/ob_fts_index_builder_util.cpp | Implements dict table schema checks, dependency recording, and drop protection checks. |
| src/sql/resolver/ddl/ob_ddl_resolver.cpp | Parses FULLTEXT_DICT option; blocks index creation on dict tables; passes schema context to parser-property resolver. |
| src/sql/resolver/ddl/ob_alter_table_resolver.cpp | Blocks ALTER TABLE structure changes on fulltext dictionary tables. |
| src/sql/resolver/cmd/ob_alter_system_stmt.h | Adds ObRefreshFulltextDictStmt for the refresh system command. |
| src/sql/resolver/cmd/ob_alter_system_resolver.h | Declares resolver for REFRESH FULLTEXT DICT. |
| src/sql/resolver/cmd/ob_alter_system_resolver.cpp | Parses refresh target (db/table) and defaults database from session when omitted. |
| src/sql/printer/ob_schema_printer.cpp | Prints FULLTEXT_DICT='Y' in SHOW CREATE TABLE where applicable. |
| src/sql/parser/sql_parser_mysql_mode.y | Adds tokens/grammar for FULLTEXT_DICT and ALTER SYSTEM REFRESH FULLTEXT DICT. |
| src/sql/parser/ob_item_type.h | Adds new parse-node item types for FULLTEXT_DICT + refresh statement. |
| src/sql/parser/non_reserved_keywords_mysql_mode.c | Registers non-reserved keywords dict and fulltext_dict. |
| src/sql/executor/ob_cmd_executor.cpp | Wires execution dispatch for the refresh fulltext dict statement. |
| src/sql/engine/expr/ob_expr_tokenize.h | Threads database/tenant context into TOKENIZE parser-property parsing. |
| src/sql/engine/expr/ob_expr_tokenize.cpp | Resolves dictionary tables during TOKENIZE additional_args parsing. |
| src/sql/engine/cmd/ob_alter_system_executor.h | Declares executor for the refresh fulltext dict system command. |
| src/sql/engine/cmd/ob_alter_system_executor.cpp | Implements dictionary refresh by resolving table + invoking dict hub refresh. |
| src/share/schema/ob_table_schema.h | Adds schema flag setters/getters for fulltext dictionary tables. |
| src/share/schema/ob_schema_struct.h | Adds FULLTEXT_DICT_FLAG bit definition. |
| src/rootserver/parallel_ddl/ob_table_helper.cpp | Validates fulltext dictionary table schema during create-table flow. |
| src/rootserver/parallel_ddl/ob_drop_table_helper.cpp | Blocks dropping dict tables in use; deletes dependency rows when dropping FTS index aux. |
| src/rootserver/parallel_ddl/ob_create_table_helper.cpp | Records dictionary dependencies for newly created FTS index aux tables. |
| src/rootserver/ob_ddl_service.cpp | Records dictionary dependencies during create-tables-in-transaction flow. |
| src/rootserver/ob_ddl_operator.cpp | Records/deletes dictionary dependencies on index create/drop and blocks dict-table drop when referenced. |
| src/plugin/interface/ob_plugin_ftparser_intf.h | Extends IK params with dict IDs and adds tenant_id to parser param; resets fields correctly. |
| src/objit/include/objit/common/ob_item_type.h | Mirrors new parser item types for objit headers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1491
to
+1492
| const bool is_quantifier_table | ||
| = 0 == config_name.case_compare(ObFTSLiteral::CONFIG_NAME_QUANTIFIER_TABLE); |
| param.ik_param_.stopword_dict_id_ = property.stopword_table_id_; | ||
| param.min_ngram_size_ = property.min_ngram_token_size_; | ||
| param.max_ngram_size_ = property.max_ngram_token_size_; | ||
| param.tenant_id_ = common::OB_SERVER_TENANT_ID; |
Comment on lines
+234
to
+237
| ObSQLSessionInfo *session = ctx.exec_ctx_.get_my_session(); | ||
| const uint64_t tenant_id = common::OB_SERVER_TENANT_ID; | ||
| const ObString database_name = OB_ISNULL(session) | ||
| ? ObString() : session->get_database_name(); |
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 0.00 / 100 load_file: 0 / 50 ai_split_document: 0 / 50 IK Custom Dictionary Score ========================== score: 100.00 / 100 ik_custom_dict: 100 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 2.07 / 100 mean_improvement: 1.04% full_score_improvement: 50.00% build_improvement: 2.20% build_ik_all_sec: baseline=35.2836, current=33.943, improvement=3.80% build_ik_content_sec: baseline=28.3764, current=27.466, improvement=3.21% build_beng_en_sec: baseline=14.7578, current=14.819, improvement=-0.41% tokenize_improvement: 0.58% tokenize_ik_avg_ms: baseline=0.76478, current=0.7793, improvement=-1.90% tokenize_beng_avg_ms: baseline=0.42262, current=0.4097, improvement=3.06% query_improvement: 0.33% query_cn_avg_ms: baseline=16.6628, current=16.6642, improvement=-0.01% query_beng_avg_ms: baseline=24.3042, current=24.1105, improvement=0.80% query_mixed_avg_ms: baseline=17.5593, current=17.4534, improvement=0.60% query_limit_avg_ms: baseline=16.2334, current=16.2427, improvement=-0.06% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-14 04:47:03 label: vldb-ci-29305733474-1 git_head: d622c5e git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2187 select1_stdev_ms: 0.0219 raw_load_sec: 1.479 raw_load_rows_per_sec: 13522.7 build_ik_all_sec: 33.943 build_ik_content_sec: 27.466 build_beng_en_sec: 14.819 build_total_sec: 76.240 ---------------------------------------- tokenize_ik_avg_ms: 0.7793 tokenize_ik_median_ms: 0.7785 tokenize_ik_stdev_ms: 0.0028 tokenize_beng_avg_ms: 0.4097 tokenize_beng_median_ms:0.4105 tokenize_beng_stdev_ms: 0.0012 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 16.6642 query_cn_stdev_ms: 0.0120 query_beng_hits: 11000 query_beng_avg_ms: 24.1105 query_beng_stdev_ms: 0.0194 query_mixed_hits: 7332 query_mixed_avg_ms: 17.4534 query_mixed_stdev_ms: 0.0264 query_limit_hits: 20 query_limit_avg_ms: 16.2427 query_limit_stdev_ms: 0.0087 ======================================== |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Description
The IK parser only supports built-in dictionaries and cannot load or dynamically refresh user-defined dictionary tables.
Solution Description
FULLTEXT_DICTtable option and dictionary schema validation.ALTER SYSTEM REFRESH FULLTEXT DICT.TOKENIZE().Passed Regressions
test_fts_property: 4/4 passed.test_ft_parser: 8/8 passed.ai_funcs/ik_custom_dict.test: passed with outputok.git diff --check: passed.Upgrade Compatibility
Existing IK parser configurations continue to use the built-in dictionaries. The historical
quanitfier_tableproperty spelling remains read-compatible.Other Information
Release Note