Skip to content

Semantic tokens with per-client legend downgrade#11

Merged
gandarfh merged 1 commit into
mainfrom
feat/semantic-tokens
Jun 10, 2026
Merged

Semantic tokens with per-client legend downgrade#11
gandarfh merged 1 commit into
mainfrom
feat/semantic-tokens

Conversation

@gandarfh

Copy link
Copy Markdown
Member

What

The server now answers textDocument/semanticTokens/full for refs: alias declarations in fence info strings, ref names (block alias vs possible environment variable) and path segments, with the unresolved modifier marking refs whose alias is not in scope and declaration marking the alias=... site.

Legend mechanics

Token types are unique per range and clients silently ignore types outside their announced capabilities, so the fallback is a server-side downgrade computed at initialize: an httui.* kind is registered in the legend only when the client announced it via textDocument.semanticTokens.tokenTypes; otherwise its LSP-standard fallback (variable / property) takes the slot. Modifiers are intersected with the client's announced list the same way. This matches the canonical token-kinds spec in spec/token-kinds.md.

Implementation

  • lib/semantic_tokens.ml stays protocol-free: abstract tokens (byte ranges + kind + flags) sorted by position; the binary maps kinds to legend indices and emits the delta-encoded stride-5 data with UTF-16 lengths via Doc_position.
  • Refs.occurrence now exposes path segment ranges (needed for httui.ref_path tokens; also unlocks future per-segment diagnostics).

Tests

43 checks total. New ones: token extraction (count, declaration flag, unresolved ghost, env-var kind, ordering) plus two integration sessions against the real binary — a vanilla client whose legend downgrades to ["property"; "variable"], and a client announcing httui.* whose legend keeps the specific kinds and whose first data token decodes exactly to the req1 declaration with the declaration modifier bit.

@gandarfh
gandarfh merged commit 7053fc2 into main Jun 10, 2026
6 checks passed
@gandarfh
gandarfh deleted the feat/semantic-tokens branch June 10, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant