Prune SSTable I/O in scan_ns with zone maps and the sparse index#118
Open
mweiden wants to merge 1 commit into
Open
Prune SSTable I/O in scan_ns with zone maps and the sparse index#118mweiden wants to merge 1 commit into
mweiden wants to merge 1 commit into
Conversation
scan_ns read every SSTable file in full from storage and walked every line for any namespace scan, making partial-key SELECTs and COUNTs O(total data) regardless of which table they touched. Add SsTable::scan_prefix: the zone map rules out tables whose key range cannot intersect the namespace (answered with zero I/O), the sparse index seeks near the first candidate line, and the scan stops at the first key sorting past the prefix. Local storage is read through a mmap instead of loading the file into memory. scan_ns now delegates to it; merge semantics (oldest table to newest, memtable last) are unchanged. https://claude.ai/code/session_01CJXWjB9ERGgV6B9mRqrdM9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Database::scan_nsread every SSTable file in full from storage and walked every line for any namespace scan. Partial-key SELECTs and COUNTs therefore cost O(total data across all tables) no matter which table they touched — and on the S3 backend that meant downloading every object per query.Fix
New
SsTable::scan_prefix(prefix, storage):ZoneMap::may_contain_prefixrules out tables whose[min, max]key range cannot intersect the prefix interval[prefix, succ(prefix))— answered with zero I/O.scan_nsdelegates to it; merge semantics (oldest table → newest, memtable overlay last) are unchanged.Tests
scan_prefix_returns_only_matching_entries(incl. theab:vsa:boundary).scan_prefix_skips_tables_outside_zone_map_without_io: a counting storage wrapper proves zone-map-excluded prefixes perform zero table reads.scan_ns_merges_sstables_and_memtable: end-to-end merge behavior across two flushed generations plus the memtable, with namespace isolation.Found by codebase audit (medium severity: O(total data) reads on the partial-key query path).
https://claude.ai/code/session_01CJXWjB9ERGgV6B9mRqrdM9
Generated by Claude Code