OPENNLP-1850: Layered Term model — Term, TermAnalyzer (2b/7)#1111
Draft
krickert wants to merge 1 commit into
Draft
OPENNLP-1850: Layered Term model — Term, TermAnalyzer (2b/7)#1111krickert wants to merge 1 commit into
krickert wants to merge 1 commit into
Conversation
This was referenced Jun 23, 2026
krickert
force-pushed
the
OPENNLP-1850-2a-tokenizer
branch
from
a450069 to
dc02b9e
Compare
krickert
force-pushed
the
OPENNLP-1850-2b-term
branch
from
57e2b58 to
82cb041
Compare
krickert
force-pushed
the
OPENNLP-1850-2a-tokenizer
branch
from
dc02b9e to
dd1906d
Compare
krickert
force-pushed
the
OPENNLP-1850-2b-term
branch
from
82cb041 to
e35e859
Compare
The token analysis layer split out of the former tokenizer PR (#1104) on review request. A Term is one token projected through the ordered Dimension stack (original, NFC, NFKC, whitespace, dash, case fold, accent fold, confusable fold, stem, lemma), keeping its source Span and every intermediate form; TermAnalyzer segments with the UAX #29 WordTokenizer (from 2a) and applies the configured dimension prefix. Restores Dimension's {@link Term}/{@link TermAnalyzer} javadoc now that they exist. Builds on the tokenizer in 2a.
krickert
force-pushed
the
OPENNLP-1850-2a-tokenizer
branch
from
dd1906d to
3fae8aa
Compare
krickert
force-pushed
the
OPENNLP-1850-2b-term
branch
from
e35e859 to
55dbeb4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2b of the OPENNLP-1850 stack: the token-analysis layer, split out of the former tokenizer PR (#1104) on review request.
A
Termis one token projected through the orderedDimensionstack (original, NFC, NFKC, whitespace, dash, case fold, accent fold, confusable fold, stem, lemma), keeping its sourceSpanand every intermediate form.TermAnalyzersegments with the UAX #29WordTokenizer(from 2a) and applies the configured dimension prefix. RestoresDimension's{@link Term}/{@link TermAnalyzer}javadoc now that those types exist.Base:
OPENNLP-1850-2a-tokenizer(#1110). Stack: 1a → 1b → 2a → 2b (this) → 2c → DL → docs.