Fail fast on oversized non-pyramidal mask reads#160
Merged
Conversation
A mask whose nearest pyramid level to the requested segmentation spacing is still huge (because the mask lacks a coarse pyramid level) forces read_region to materialise a multi-GB raster, immediately followed by an 8x np.unique(int64) copy, OOM-killing the job. This was the root cause of a BEETLE dense-extraction SIGKILL: one single-level annotation mask read at full res (62407x43898 = 2740 Mpx) -> ~48 GB live. Add a size-gated guard in _read_discrete_mask_level (the single chokepoint both the precomputed-tissue and annotation read paths funnel through), raising an actionable ValueError before any read when the level exceeds MAX_MASK_READ_PX (256 Mpx). 256 Mpx sits cleanly in the empirical gap between healthy mask reads (<= 4.9 Mpx) and the fatal 2740 Mpx mask, with ~52x headroom over healthy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A mask whose nearest pyramid level to the requested segmentation spacing is still huge — because the mask lacks a coarse pyramid level — forces
read_regionto materialise a multi-GB raster, immediately followed by an 8×np.unique(int64)copy. On a real slide this OOM-killed the job.This was the root cause of a BEETLE dense-extraction SIGKILL: one single-level (non-pyramidal) annotation mask resolved to level 0 and was read at full resolution 62407×43898 = 2740 Mpx, blowing up to ~48 GB live (2.7 GB uint8 read + 21.9 GB int64
np.unique+ an 8 GB RGB read canvas) and exceeding the cgroup limit. Every other mask in the cohort has a pyramid and resolves to a ~5 Mpx level — only this one degenerates.Fix
Add a size-gated guard in
_read_discrete_mask_level— the single chokepoint both the precomputed-tissue path (_read_mask_level) and the annotation path (_read_label_mask_at_seg) funnel through. When the level about to be read exceedsMAX_MASK_READ_PX(256 Mpx), it raises an actionableValueErrorbefore anyread_region, naming the mask, the level, its dimensions/Mpx, the cap, and the remedy (regenerate as a multi-resolution pyramidal TIFF, or lowerseg_downsample).Threshold
MAX_MASK_READ_PX = 256_000_000. Measured across all 584 masks in the affected cohort: healthy masks read ≤ 4.9 Mpx (median 0.92), and nothing lands between 8 and 2000 Mpx — the one fatal mask reads 2740 Mpx. 256 Mpx sits cleanly in that gap: ~52× headroom over the largest healthy read (so a mask that's merely a pyramid step or two coarse still passes) and ~11× below the fatal one. Worst-case transient at the cap is ~256 MB uint8 + ~2 GB int64 — safe.Tests
New
tests/test_mask_read_size_guard.py(9 tests):read_regionasserts if invoked proves it's pre-allocation), with the informative message asserted.Full suite: 309 passed, 1 skipped (two pre-existing collection errors from a missing OpenSlide binary in the venv are unrelated to this change).