fix(uts39): proper Unicode case folding and restriction levels#11
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b0dc478a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Base transform: NFKD → caseFold → NFD → applyConfusables. | ||
| s = uts15.NFKD(s) | ||
| s = caseFold(s) | ||
| s = uts15.NFD(s) |
There was a problem hiding this comment.
Fold confusable targets before returning skeletons
For inputs whose confusable mapping target is itself case-foldable, this now returns a non-final skeleton because the fixed-point loop only repeats NFD and confusable mapping, not case folding. For example, the checked-in data maps 0 to "O", so Skeleton("0") returns "O" while Skeleton("O") returns "o", making AreConfusable("0", "O") false and violating the idempotence promised by the new comment. The old lowercase-after-mapping path handled these targets; the new full fold needs to be applied after mappings or the data needs to be pre-folded.
Useful? React with 👍 / 👎.
Two correctness fixes in confusable detection / identifier safety.
First, Skeleton previously used strings.ToLower which is wrong per UTS #39; it now uses full Unicode case folding (generated casefold_data.go from CaseFolding.txt) with the correct transform order and a defensive fixed-point cap that panics on non-convergence instead of silently truncating.
Second, GetRestrictionLevel is fixed so the higher restriction levels are reachable (affects IsSafeIdentifier): the old code had a dead duplicate single-script branch and collapsed everything else to ModeratelyRestrictive, so HighlyRestrictive/ModeratelyRestrictive were not properly distinguished. It now implements the UAX #39 §5.2 Table 1 script-set rules (Latn+Jpan / Latn+Hanb / Latn+Kore as subset checks for HighlyRestrictive, Latin + one non-Cyrillic/Greek script for ModeratelyRestrictive).
Includes the case-fold generator and regression tests.