Garbled Chinese characters (shown as �) appear in every session as long as there's enough text — which is basically always.
I traced the root cause to llm-rs/src/llm/sse.rs:57:
buffer.push_str(&String::from_utf8_lossy(&chunk));
SSE protocol: each data: line contains a complete JSON message, but multiple SSE messages can be coalesced into a single TCP chunk at the network layer. When that happens, a multi-byte UTF-8 character (e.g. 世 = E4 B8 96) can be split across two chunks. from_utf8_lossy decodes each chunk independently — the partial bytes get replaced with � (U+FFFD) and are permanently lost because buffer is a String and the raw bytes are already corrupted by the lossy decode.
The suggested fix (from an AI assistant, so it may not be entirely correct — take it as a reference):
Change buffer from String to Vec<u8>, accumulate raw bytes, and only decode to UTF-8 when a complete SSE message (delimited by \n) is extracted. This way partial multi-byte sequences crossing chunk boundaries remain as raw bytes in the buffer and naturally complete when the next chunk arrives, without losing any characters.
Garbled Chinese characters (shown as
�) appear in every session as long as there's enough text — which is basically always.I traced the root cause to
llm-rs/src/llm/sse.rs:57:SSE protocol: each
data:line contains a complete JSON message, but multiple SSE messages can be coalesced into a single TCP chunk at the network layer. When that happens, a multi-byte UTF-8 character (e.g.世=E4 B8 96) can be split across two chunks.from_utf8_lossydecodes each chunk independently — the partial bytes get replaced with�(U+FFFD) and are permanently lost becausebufferis aStringand the raw bytes are already corrupted by the lossy decode.The suggested fix (from an AI assistant, so it may not be entirely correct — take it as a reference):
Change
bufferfromStringtoVec<u8>, accumulate raw bytes, and only decode to UTF-8 when a complete SSE message (delimited by\n) is extracted. This way partial multi-byte sequences crossing chunk boundaries remain as raw bytes in the buffer and naturally complete when the next chunk arrives, without losing any characters.