OPENNLP-1850: UAX #29 word tokenizer and the layered Term model (2/4) - #1104

Closed

krickert wants to merge 7 commits into

OPENNLP-1850-1b-alignmentfrom

OPENNLP-1850-2-tokenizer

Commits on Jun 23, 2026

OPENNLP-1850 UAX #29 word tokenizer and the layered Term model
krickert
committed
064d363
OPENNLP-1850 Address Copilot review on the UAX #29 tokenizer
krickert
committed
2ecf637
OPENNLP-1850 Address tokenizer review comments
krickert
committed
724a254
OPENNLP-1850 Clarify that Extended_Pictographic symbols are kept as emoji
krickert
committed
d064897
OPENNLP-1850 Fail fast on null public-entry arguments (review nits)
krickert
committed
2da2949
OPENNLP-1850 Address review: fail-loud TermAnalyzer default; harden WordBreakProperty
krickert
committed
ab038b4
OPENNLP-1850 Lazy, recoverable loading for WordBreakProperty and ExtendedPictographic
krickert
committed
0bf7f6c