Problem
Rylox currently assumes Python source files are UTF-8 (with or without a BOM). Valid Python files that declare another encoding via a PEP 263 encoding cookie (for example, latin-1) fail to decode and are skipped during indexing.
Supporting only UTF-8 is sufficient for most repositories, but Python officially supports source encoding declarations and Rylox should be able to index those files as well.
Proposed solution
Detect and respect PEP 263 encoding declarations when reading Python source files.
This should:
- Detect encoding cookies defined by PEP 263.
- Decode the file using the declared encoding.
- Fall back to the existing UTF-8 behavior when no encoding is specified.
- Continue treating genuinely unreadable files as malformed.
Alternatives considered
Continue supporting only UTF-8 source files.
This keeps the implementation simpler but prevents Rylox from indexing valid Python files that declare a different source encoding.
Additional context
No response
Problem
Rylox currently assumes Python source files are UTF-8 (with or without a BOM). Valid Python files that declare another encoding via a PEP 263 encoding cookie (for example,
latin-1) fail to decode and are skipped during indexing.Supporting only UTF-8 is sufficient for most repositories, but Python officially supports source encoding declarations and Rylox should be able to index those files as well.
Proposed solution
Detect and respect PEP 263 encoding declarations when reading Python source files.
This should:
Alternatives considered
Continue supporting only UTF-8 source files.
This keeps the implementation simpler but prevents Rylox from indexing valid Python files that declare a different source encoding.
Additional context
No response