fix: large-file corruption in snappy/lzss/lzs/xpress + stress harness - #116
Merged
Conversation
Streaming ~1-16 GiB through each codec (new examples/stress.rs, bounded memory) surfaced several corruption bugs that KB-scale unit tests and fuzzing never hit. None affect xz/lzma2/zstd/lz4/deflate/brotli/bzip2, which round-trip cleanly to 16 GiB (xz also verified against native `xz -d`). - snappy, lzss: the uncompressed length is written as a 32-bit field (`len as u32` / 4-byte LE header). At >= 4 GiB it silently truncated, making lzss decode the wrong length (MISMATCH) and snappy emit an undecodable stream. Reject over-length input with a clean `Error::Unsupported` instead of corrupting. - lzs, xpress: a fixed `SANITY_MATCH_LEN` bomb guard (1<<24 = 16 MiB for lzs, 1<<28 = 256 MiB for xpress) wrongly rejected VALID streams whose longest back-reference exceeded it — trivially reached on repetitive input (one match can span most of the output). Bound the guard by the declared total length instead. Their match finders also address positions with `u32`, which degrades/corrupts above 4 GiB, so the encoders now reject >= 4 GiB input with a clean error. Adds a fast regression test for the lzs guard (a ~20 MiB match that the old code rejected) and examples/stress.rs (generate -> encode -> decode -> 128-bit hash compare, plus an our-encode -> native `xz -d` mode). Full suite 1706 green; fmt + clippy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The previous commit bounded the tier-3 match-length guard by the declared `target`, but `target` is attacker-controlled (8-byte header), so a malformed stream declaring a huge target let `dw` reach `u32::MAX` and the `dw + 3` return overflowed (panic under debug assertions). Found by the decoder_xpress fuzz smoke job. Cap at `u32::MAX - 3` so the add can never overflow while still admitting legitimately long matches. Adds the libFuzzer crash input as a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This was referenced Jul 6, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streaming ~1–16 GiB through every codec with a new bounded-memory harness (
examples/stress.rs: generate → encode → decode → 128-bit hash compare) surfaced corruption bugs that KB-scale unit tests and fuzzing never reach.The good news first
xz — the reported concern — is clean at 16 GiB across all data kinds (text/runs/seq/mixed/random), verified both by self round-trip and by decoding our output with native
xz -d. So are zstd, lz4, lzma, lzma2, deflate/gzip/zlib, brotli, bzip2, and the secondary compressors/filters (lzo, lzw, lz5, adc, lznt1, deflate64, rle/rle90/packbits, mtf, delta, bcj, huffman, range).Bugs fixed (all niche block codecs)
len as u32→ truncatesError::UnsupportedSANITY_MATCH_LEN=1<<24falsely rejects valid long matchesSANITY_MATCH_LEN=1<<28, same classThe lzs/xpress ones are the most insidious: a single back-reference legitimately spans most of the output on repetitive data, so valid streams failed to decode at very ordinary sizes (16 MiB / 256 MiB), not just at the 4 GiB edge. Their
u32-addressed match finders also degrade/corrupt above 4 GiB, so the encoders now refuse ≥4 GiB with a clean error rather than risk silent corruption.Also included
examples/stress.rs— reusable large-file checker (self round-trip +our-encode → native xz -dmodes), bounded memory at any scale.Full suite 1706 green; fmt + clippy clean.
🤖 Generated with Claude Code