Skip to content

fix(uts39): proper Unicode case folding and restriction levels#11

Merged
SCKelemen merged 1 commit into
mainfrom
SCKelemen/uts39-case-folding
Jun 12, 2026
Merged

fix(uts39): proper Unicode case folding and restriction levels#11
SCKelemen merged 1 commit into
mainfrom
SCKelemen/uts39-case-folding

Conversation

@SCKelemen

Copy link
Copy Markdown
Owner

Two correctness fixes in confusable detection / identifier safety.

First, Skeleton previously used strings.ToLower which is wrong per UTS #39; it now uses full Unicode case folding (generated casefold_data.go from CaseFolding.txt) with the correct transform order and a defensive fixed-point cap that panics on non-convergence instead of silently truncating.

Second, GetRestrictionLevel is fixed so the higher restriction levels are reachable (affects IsSafeIdentifier): the old code had a dead duplicate single-script branch and collapsed everything else to ModeratelyRestrictive, so HighlyRestrictive/ModeratelyRestrictive were not properly distinguished. It now implements the UAX #39 §5.2 Table 1 script-set rules (Latn+Jpan / Latn+Hanb / Latn+Kore as subset checks for HighlyRestrictive, Latin + one non-Cyrillic/Greek script for ModeratelyRestrictive).

Includes the case-fold generator and regression tests.

@SCKelemen
SCKelemen merged commit c91b5f2 into main Jun 12, 2026
5 checks passed
@SCKelemen
SCKelemen deleted the SCKelemen/uts39-case-folding branch June 12, 2026 09:24

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b0dc478a6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread uts39/uts39.go
// Base transform: NFKD → caseFold → NFD → applyConfusables.
s = uts15.NFKD(s)
s = caseFold(s)
s = uts15.NFD(s)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fold confusable targets before returning skeletons

For inputs whose confusable mapping target is itself case-foldable, this now returns a non-final skeleton because the fixed-point loop only repeats NFD and confusable mapping, not case folding. For example, the checked-in data maps 0 to "O", so Skeleton("0") returns "O" while Skeleton("O") returns "o", making AreConfusable("0", "O") false and violating the idempotence promised by the new comment. The old lowercase-after-mapping path handled these targets; the new full fold needs to be applied after mappings or the data needs to be pre-folded.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant