Size SSTable bloom filters by entry count#117
Open
mweiden wants to merge 1 commit into
Open
Conversation
Bloom filters were a fixed 1024 bits regardless of how many keys an SSTable held, so any table beyond a few hundred entries saturated the filter and may_contain returned true for nearly every absent key, defeating the filter's purpose of pruning disk reads. Filters are now sized at 10 bits per key (~3% false-positive rate with the two hash functions used), with a 1024-bit floor. The rebuild path in SsTable::load counts entries first and sizes the filter the same way. Also hardened the filter against empty bit vectors (e.g. decoded from a corrupt meta file): an empty filter now reports may_contain = true instead of panicking on a modulo-by-zero. https://claude.ai/code/session_01CJXWjB9ERGgV6B9mRqrdM9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bloom filters were a fixed 1024 bits regardless of how many keys an SSTable held. Any table beyond a few hundred entries saturated the filter, so
may_containreturned true for nearly every absent key — the filter stopped pruning disk reads entirely, which is its only job.Fix
BloomFilter::with_capacity(expected_keys): 10 bits per key (~3% false-positive rate with the two hash functions used), 1024-bit floor.SsTable::createsizes the filter from the entry count; the meta-less rebuild path inSsTable::loadcounts entries in a first pass and sizes identically.hashes(); it now reportsmay_contain = true(no information ⇒ must not rule keys out) andinsertis a no-op.new(0)is also guarded.Tests
sized_filter_keeps_false_positive_rate_low: 10k keys, no false negatives, <20% FP on absent keys (the old fixed filter is ~100% here).bloom_filter_scales_with_entry_count: same property at the SSTable level, both freshly created and rebuilt without a meta file.empty_filter_from_proto_does_not_panic,zero_size_filter_does_not_panic.Found by codebase audit (medium severity: bloom filter ineffective at scale + panic on corrupt meta).
https://claude.ai/code/session_01CJXWjB9ERGgV6B9mRqrdM9
Generated by Claude Code