perf: highlight Markdown/MDX with tree-sitter (diff viewer, file viewer, content search)

[matej21]

Problem

Opening a git diff for some Markdown/MDX files was painfully slow — e.g. a 715-line .mdx with only a 3-line change took ~1.4 s (release) / ~3 s (dev) to render. The same slow highlighter also backs the file viewer and the content-search preview.

Profiling isolated the layer (it is not git, rendering, or diff size):

• syntect's Sublime Markdown grammar is pathologically slow — ~1 ms/line. On the same bytes: plain text ~0.3 ms, Rust grammar ~35 ms, Markdown ~700 ms (≈20× slower than Rust, ≈2000× slower than plain text). The diff viewer does it twice (old + new) ⇒ ~1.4 s.
• ~100 % of that cost is syntect's stateful parse, not styling. So "only style displayed lines" saves nothing — reaching a mid-file hunk still requires parsing the whole prefix line-by-line.

Fix

Highlight Markdown/MDX with tree-sitter (tree-sitter-md), which parses the whole document in ~30 ms and — because it parses the entire file cheaply — sidesteps the "carry parse state to a mid-file hunk" problem entirely.

Hybrid design so there's no regression on embedded code:

• tree-sitter handles the Markdown structure — headings, emphasis, links, inline code, fence delimiters (block + inline highlight queries → a per-byte colour buffer; smaller/more-specific captures win).
• Fenced code blocks keep going through the existing syntect path for their embedded language (graphql/json/ts/…). Those grammars are fast — only Markdown was the problem.

Unified across all viewers. A shared markdown_line_spans core feeds two thin adapters with the existing output shapes, so callers are unchanged:

• highlight_markdown_file → diff viewer (line -> spans map, all lines).
• highlight_markdown_content → file viewer & content-search preview (ordered HighlightedLines, capped at max_lines), wired in via syntax::highlight_content.

Non-Markdown files are completely unaffected (still syntect).

Result

Measured end-to-end on the same file (release), including embedded graphql/json highlighting:

	before (syntect)	after (tree-sitter)
single pass	~700 ms	~41 ms
diff old + new	~1.4 s	~82 ms

~17× faster — and now the file viewer / content-search preview get the same win.

Notes

• tree-sitter 0.26 and streaming-iterator are already in the workspace (via gpui-component), so this only adds the tree-sitter-md grammar crate.
• The grammar compiles C via the cc crate — fine on Linux/macOS; on Windows it needs the MSVC toolchain already used for this repo (x64 Native Tools).
• New module crates/okena-files/src/markdown_highlight.rs with unit tests (text reconstruction, heading colouring, embedded-language colouring, HighlightedLine shape + max_lines, path matching). No unwrap/expect in non-test code (crate lint).

Manual check suggested

Open a large .mdx/.md file (e.g. a Contember docs page with graphql code fences) in both the diff viewer and the file viewer; confirm it renders instantly with headings/links/code coloured.

🤖 Generated with Claude Code

https://claude.ai/code/session_019JZRfqVneyJnKjdrtSp3ro