RAR4 (v29) decode: in-band filters, PPMd-II variant H, solid groups - #122
Open
jdlien wants to merge 10 commits into
Open
RAR4 (v29) decode: in-band filters, PPMd-II variant H, solid groups#122jdlien wants to merge 10 commits into
jdlien wants to merge 10 commits into
Conversation
The decode-direction address fixup flattened unrar's nested condition
into an else-if:
if (addr < 0) { if (addr + off >= 0) addr += FILE_SIZE; }
else { if (addr < FILE_SIZE) addr -= off; }
A negative operand that stays negative after adding the position (a
byte pattern the encoder never rewrote, e.g. a stray 0xE8 inside a
ModRM/displacement sequence followed by high bytes) fell through into
the (addr - FILE_SIZE) check, which any negative value passes, and got
the position wrongly subtracted.
Found by differential testing against WinRAR 7.23 + UnRAR: the first
32 KiB of notepad.exe compressed with -ma5 -m3 (filter type 1, X86Call)
decoded to the right length but 48 wrong bytes across 15 sites, each
the 4-byte operand after a false-positive 0xE8; every wrong value was
exactly (original - position). With the nested checks the whole
corpus decodes byte-identical to unrar. Regression test carries the
verbatim case from that archive plus both boundary companions.
`{:#02x}` sets a width of 2 that the `#` (0x) alternate form makes
inapplicable, so current stable clippy fails the --all-features test
build under -D warnings. Use `{:#04x}` for the intended two-digit form.
RAR3 (v29) in-band filter declarations (main symbol 257) are now decoded instead of refused. WinRAR's compressor only ever emits a fixed set of standard RarVM programs, so — like libarchive and unrar in practice — the decoder recognizes them by bytecode length + CRC-32 and runs native transforms; any other program still fails with Error::Unsupported, never wrong bytes: - Delta (channel de-interleave; channels in VM register 0) — new, shared with rar5 via the new crate-internal rar_filters module. This also closes the rar5 Delta gap: rar5 FilterKind::Delta now decodes instead of returning Unsupported. - x86 E8 / E8E9 call translation — reuses the rar5 transform (with the nested range checks from the recent corruption fix). The rar5 x86 filters also gain correct solid-archive semantics: unrar and libarchive compute E8 addresses relative to the *containing file*, not the solid stream (libarchive tracks this as solid_offset). A new Decoder::add_file_boundary lets the container register member offsets; without it single-file streams behave as before. Wire format and transform semantics were derived from public format documentation and validated bit-exact against archives produced by RARLAB rar 6.24, differentially compared byte-for-byte with UnRAR.exe 7.23 across a 100+ archive corpus (RAR4 m1-m5 matrix incl. dict sweeps, filter isolation archives, and RAR5 solid/nonsolid groups): 188 entries byte-identical, 0 mismatches. Delta covers more than bitmaps: rar 6.24 emits it for WAV (2ch) and even generic content at -m5 (12ch observed); filters are auto-applied at m2-m5, so this unblocks most real RAR4 archives, not just executables/images. Embedded regression fixtures carry real payloads from that corpus with the archive's own FILE_CRC as ground truth (tests/fixtures/rar3/). Clean-room: structure cross-checked against libarchive (BSD) only; no code from RARLAB's unRAR or The Unarchiver. Fingerprint constants were computed from archives our own tooling generated.
…ta fixtures Review findings on the in-band filter implementation, all three fixed by moving to incremental filter application (flush_completed_filters): - A slot-0 filter-table reset now applies the completed-window prefix and cancels everything still pending, mirroring unrar's InitFilters — a canceled filter must never rewrite output. Previously pending filters survived the reset and ran at finish. - The pending queue is capped at 8192 concurrent filters (unrar's MAX_UNPACK_FILTERS); when the cap is hit after draining completed windows the stream is rejected as corrupt. - A filter window the stream never finishes producing is now Corrupt instead of silently returning pre-filter bytes as success. (unrar writes the raw bytes and relies on the container CRC to flag the file; surfacing the error directly matches this crate's malformed-input policy.) Flushing is prefix-only — a completed window queued behind an incomplete one waits — so overlapping windows (filter chains) always apply in declaration order. Output is append-only and LZ back-references read the unfiltered window, so early application is equivalent to end-of-stream application. New coverage: - Declaration-level unit tests (reset/cancel semantics, the cap, slot-reuse remembered lengths, prefix ordering, truncated-window integration test) driving the parser with the real 29-byte Delta program. - rar5 end-to-end real-archive fixtures: a Delta-filtered member run and a whole six-member solid group exercising add_file_boundary against the archives' own data-CRCs, plus a regression guard showing exactly the x86 member corrupts when boundaries are not registered. - Both fuzz targets from the rar-fuzz-targets branch (KarpelesLab#120) merge-tested against this branch and run under ASan seeded with corpus payloads: decoder_rar3 328k runs / decoder_rar5 341k runs, no findings. Docs: README capability matrix updated for both formats; rar5 module docs now describe Delta support and the solid-group calling convention. Known gap, documented: the 57-byte E8E9 program has no real-archive fixture because no current archiver emits it (rar 6.24 uses the E8-only program); its fingerprint cites libarchive and the transform is unit-tested.
Port the rar2/rar5 segment-copy transformation (KarpelesLab#115) to rar3, which predated that pass: non-overlapping matches now bulk-copy disjoint window segments (extend_from_slice + copy_within) and distance-1 runs bulk-fill, both capped against ring wrap; genuinely overlapping matches keep the per-byte loop. Measured against UnRAR.exe 7.23 `t` on 32-61 MB RAR4 archives (Ryzen 9 9950X, best of 5, see the corpus repo's BENCH-RAR4.md): match-heavy text: 790 -> 1480 MB/s (+87%), 4.6x -> 1.9x vs unrar x86 + E8 filters: 631 -> 1145 MB/s (+81%), now at parity literal-heavy: unchanged (~122 MB/s, 1.4x) — path untouched Byte-identical: full differential corpus still 188 pass / 0 mismatch; decoder_rar3 fuzz re-run clean over the new path (129k runs, ASan).
Implement the full PPMII variant H model (Ppmd7) and wire it into both the standalone `.ppmd` framing decoder (7z range-coder flavour) and the RAR3/4 PPMd block path (RAR range-coder flavour, with the RAR literal/match/EOD escape layer). Replaces the earlier order-0 stub (arena.rs/model.rs removed). Model (src/ppmd/ppmd7.rs): information-inheritance context tree (CreateSuccessors/UpdateModel), binary-context fast path, masked-escape suffix walk, SEE, tree-wide Rescale, and the byte-arena suballocator with GlueFreeBlocks coalescing. OOB-safe arena accessors set an error flag rather than panicking. Derived from the public-domain LZMA SDK Ppmd7 and the RAR range-coder description in libarchive; no license-restricted code copied. Range decoder (src/ppmd/range_dec.rs): dual 7z/RAR carry-less range coder with a normalisation-step safety cap. Hardening: reject a declared output length beyond the buffer-then-decode ceiling up front and bail on range-coder overrun mid-loop — a high-probability PPMd symbol can decode repeatedly without consuming input, so `overran()` alone doesn't bound the known-length loop (fuzz-found OOM; regression test added). Fuzzing: add dedicated decoder_rar2/rar3/rar5 targets (input-prefix framing for unpack size / E8 flag / window selector). ASan campaigns over the standalone PPMd and RAR3 paths are clean after the OOM fix. Fixtures: real pyppmd Ppmd7 streams (order 6, 16 MiB) plus a RAR3 PPMd block. The encoder stays permanently Unsupported (RARLAB licence).
…h (codex review) Address a gpt-5.6-sol review of the PPMd-II var H work. Three silent-wrong- output paths are closed; a fourth (arena-exhaustion parity) is documented as a fail-closed limitation. - Standalone PPMd, unknown declared length: PPMd carries no in-band end-of-stream marker, so decoding a u64::MAX-framed stream until the range coder exhausts input appended finalisation-byte garbage (reframing the 1280-byte `repeat` fixture returned 1466 bytes). Refuse unknown length up front (Error::Unsupported) instead of guessing. - RAR3 PPMd block, truncated payload: the decode loop checked rc.err() but not rc.overran(), so a truncated payload kept fabricating symbols from zero-filled reads until it hit the declared unpacked size — returning StreamEnd with a wrong CRC. Fail with UnexpectedEnd once the range coder reads past the input. - RAR3 mixed blocks: a new-table boundary in the LZ path (symbol 256) that selects a PPMd block stored the header but let `expand` keep decoding the range-coded payload through the previous block's stale Huffman tables (garbage). Refuse the mid-stream switch into PPMd (Error::Unsupported), matching run_ppmd_block's own start-new-table stance. - Suballocator exhaustion (documented, not fixed): a high-order model over a large high-entropy payload in a small arena can restart the model at a different symbol than the encoder and desync, surfacing as Error::Corrupt. It fails closed — never wrong bytes, never OOB (verified: no panic under instrumented OOB checks). Full GlueFreeBlocks parity is Phase 3 hardening. Regression tests: unknown-length refusal, absurd-length rejection (existing), and RAR3 truncation-not-fabrication. Full suite 1733 green; ppmd/rar3 ASan fuzz campaigns clean.
…arry-over
RAR3 solid groups share one compression history across members while each
member's payload is its own byte-aligned stream. New API:
Decoder::with_solid() + begin_solid_member(n); the LZ window, code tables,
offset history, declared filter programs and any live PPMd model persist,
and per-member state (bit input, output, pending filter windows) resets.
Member-boundary framing was established empirically against the corpus
(probe payload heads + differential validation, no unrar source):
- The LZ end-of-block marker's second bit announces whether the next
member opens with its own table header or continues symbol decode
headerless (observed: photo.jpg, ramp.wav); an inline-table marker at
the exact boundary parses the tables from the current member's tail.
- PPMd members end with an explicit escape+2 marker decoded THROUGH the
model (the encoder's model saw it too — required for cross-member model
parity); a continuation header (no 0x20 reset flag) reuses the live
model and escape char with a freshly initialised range coder.
- Mid-member LZ<->PPMd switches (start-new-table in both domains) and
PPMd escape-code-3 filter declarations are now decoded; the previous
Unsupported arms for these are gone.
Fail-closed rules in solid mode: a short member is a hard error (the
shared history would silently desync every later member), and range-coder
state straddling a member boundary (markerless PPMd member end, or a PPMd
block starting exactly at the boundary via an inline announcement) stays
Unsupported — rar 6.24 emits neither.
Also aligns the in-band filter machinery with UnRAR 7.23 semantics per
review: block-length caps at VM_MEMSIZE (0x40000, delta half) and a filter
reset (slot 0) now cancels every scheduled filter without applying
completed-but-unflushed windows (unrar InitFilters30).
Validation: differential harness 206 pass / 0 mismatch (was 188) — all of
rar4_m{1,3,5}_solid + rar4_ppmd_solid byte-identical to UnRAR 7.23, plus
the 64-bit corpus-large case; whole solid groups embedded as fixtures
(m3 6-member group, PPMd pair) asserted against FILE_CRCs; decoder_rar3
fuzz target grew a solid-group mode with real-group seeds, corpus replay +
7-minute ASan campaign clean.
Claude-Session: https://claude.ai/code/session_01DYLfPTghh7DayY4M7YyeKY
Findings from the adversarially-verified multi-agent review of this branch, all confirmed against reference semantics: - rar_filters: the shared x86 E8/E8E9 transform gains a wrap_16m mode switch. RAR5 keeps the 16 MiB position-base mask (validated against real archives); RAR3's VM filter uses the unmasked 32-bit position (per libarchive's RAR3 reader) — the shared helper had silently applied RAR5 masking to RAR3, which would corrupt filter windows past 16 MiB of a large executable while reporting success. - ppmd7: decode_symbol now bounds num_stats (raw arena u16) before its `- 1` / `- num_masked` walks — a corrupt context tree could wrap the subtraction into a debug panic, a ps[256] out-of-bounds index, or a ~2^32-iteration stall. Fail closed with Error::Corrupt. Plus hot-path cleanups: char_mask is built only on the escape path, ps is hoisted out of the escape loop, and st_copy/st_swap move states with one range check instead of 12-24 byte-wise accessor calls. - rar5: raw_reset() clears file_boundaries — a decoder reused for an unrelated stream kept the previous solid group's boundaries and skewed every later x86 filter's position base (silent wrong bytes). - rar3 filters: Delta channel count capped at unrar's MAX3_UNPACK_CHANNELS (1024) and no longer rejected merely for exceeding the window length (well-defined, and unrar decodes it); out-of-range counts fail closed where unrar emits raw bytes with success. - ppmd decoder: the 11-byte standalone header is validated as soon as it arrives (fail-fast from decode(), sticky error) instead of after buffering an arbitrarily large payload. Differential harness unchanged at 206 pass / 0 mismatch; rar3 + ppmd fuzz corpora replay clean under ASan. Claude-Session: https://claude.ai/code/session_01DYLfPTghh7DayY4M7YyeKY
… path Two low-risk decode speedups, both byte-identical (differential harness 206 pass / 0 mismatch): - Preallocate `out` from the member's `unpack_size` (capped at 64 MiB so a hostile size can't drive a huge up-front allocation; the `u64::MAX` unknown-length sentinel reserves nothing). The output buffer previously grew from empty, reallocating ~log2(size) times mid-decode — ~26 grow-and-copy passes on a 61 MB member. This is the bulk of the win: match-heavy text 1496 -> 1879 MB/s (+26%), x86 1149 -> 1276 (+11%). - Add a check-free `BitReader::consume` for use right after a successful `peek` (the availability re-check in `drop_bits` is redundant there); route `read_bits` and the Huffman LUT fast path through it. After a 9-bit LUT miss the code is provably >9 bits, so the canonical slow-path scan starts at length 10. Small consistent x86 gain; literal/Huffman profile unchanged (it is not bit-decode-bound). Measured note for the record: directly skipping the sliding-window writes (a throwaway experiment) produced ~no speedup, so the out+window double write is not the bottleneck and an out-as-window refactor would not pay off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full RAR4 (v29 /
unpack29) decode support to therar3module —the format WinRAR 4.x and modern
rar(6.x,-ma4) write. The RAR3 LZ77 + Huffman core is extended with in-band standard filters, PPMd-II variant H, solid-archive support, and a shared filter/PPMd layer.The Delta transform is now shared with
rar5. x86 now uses the unmasked 32-bit position base for RAR3 while rar5 keeps its 16 MiB wrap, and solid x86 filters use file-relative address bases.Clean-room / licensing
Implemented from public specifications and the public-domain LZMA SDK
Ppmd7.{c,h}/Ppmd7Dec.c(followed closely for the PPMd model, as thecrate's policy allows), plus the algorithm/structure descriptions in BSD
libarchive and the LGPL unarr/The Unarchiver readers — no license-restricted
code was copied, and RARLAB
unRARwas not transliterated. This keeps thecontribution MIT + clean-room, consistent with
CONTRIBUTING.md.Design invariant
Every path is fail-closed: the decoder either produces the
correct bytes or returns an Error. Unsupported sub-features degrade to
Error::Unsupported; bad streams surfaceUnexpectedEnd/Corrupt.Validation
UnRAR.exe7.23 (47-archive corpus, incl. solidm{1,3,5}+PPMd groups and 64-bit sizes): 206 member-checks pass, 0 mismatch.
cargo test --all-features: 1740 passing; new rar2/3/5 + ppmd libFuzzertargets, ASan-clean;
fmtandclippy -D warningsclean.Performance
Output-buffer preallocation plus a check-free bit-consume path take the
match-heavy profile from 1.8× → 1.47× slower than unrar and the x86+filter
profile to parity, byte-identical. See
BENCH-RAR4.md.Known limitation (fail-closed)
A high-order PPMd stream that exhausts a small memory arena can return
Error::Corruptrather than decoding — never wrong bytes, never OOB. Fullallocator parity is tracked as follow-up.
Note
This branch is stacked on #121 (
rar5-x86-filter-fix): two of its commits(
1f28a46,5c47b0e) belong to that PR. Until #121 merges, a PR from here tomasterwill include them.