Skip to content

Fail fast on oversized non-pyramidal mask reads#160

Merged
clemsgrs merged 1 commit into
mainfrom
hs2p-mask-read-size-guard
Jun 26, 2026
Merged

Fail fast on oversized non-pyramidal mask reads#160
clemsgrs merged 1 commit into
mainfrom
hs2p-mask-read-size-guard

Conversation

@clemsgrs

Copy link
Copy Markdown
Owner

Problem

A mask whose nearest pyramid level to the requested segmentation spacing is still huge — because the mask lacks a coarse pyramid level — forces read_region to materialise a multi-GB raster, immediately followed by an 8× np.unique(int64) copy. On a real slide this OOM-killed the job.

This was the root cause of a BEETLE dense-extraction SIGKILL: one single-level (non-pyramidal) annotation mask resolved to level 0 and was read at full resolution 62407×43898 = 2740 Mpx, blowing up to ~48 GB live (2.7 GB uint8 read + 21.9 GB int64 np.unique + an 8 GB RGB read canvas) and exceeding the cgroup limit. Every other mask in the cohort has a pyramid and resolves to a ~5 Mpx level — only this one degenerates.

Fix

Add a size-gated guard in _read_discrete_mask_level — the single chokepoint both the precomputed-tissue path (_read_mask_level) and the annotation path (_read_label_mask_at_seg) funnel through. When the level about to be read exceeds MAX_MASK_READ_PX (256 Mpx), it raises an actionable ValueError before any read_region, naming the mask, the level, its dimensions/Mpx, the cap, and the remedy (regenerate as a multi-resolution pyramidal TIFF, or lower seg_downsample).

Threshold

MAX_MASK_READ_PX = 256_000_000. Measured across all 584 masks in the affected cohort: healthy masks read ≤ 4.9 Mpx (median 0.92), and nothing lands between 8 and 2000 Mpx — the one fatal mask reads 2740 Mpx. 256 Mpx sits cleanly in that gap: ~52× headroom over the largest healthy read (so a mask that's merely a pyramid step or two coarse still passes) and ~11× below the fatal one. Worst-case transient at the cap is ~256 MB uint8 + ~2 GB int64 — safe.

Tests

New tests/test_mask_read_size_guard.py (9 tests):

  • Guard fires before any read on both the annotation and precomputed-tissue paths (a fake slide whose read_region asserts if invoked proves it's pre-allocation), with the informative message asserted.
  • Under-cap masks read/validate unchanged through both paths.
  • Threshold value locked.

Full suite: 309 passed, 1 skipped (two pre-existing collection errors from a missing OpenSlide binary in the venv are unrelated to this change).

A mask whose nearest pyramid level to the requested segmentation spacing
is still huge (because the mask lacks a coarse pyramid level) forces
read_region to materialise a multi-GB raster, immediately followed by an
8x np.unique(int64) copy, OOM-killing the job. This was the root cause of
a BEETLE dense-extraction SIGKILL: one single-level annotation mask read
at full res (62407x43898 = 2740 Mpx) -> ~48 GB live.

Add a size-gated guard in _read_discrete_mask_level (the single chokepoint
both the precomputed-tissue and annotation read paths funnel through),
raising an actionable ValueError before any read when the level exceeds
MAX_MASK_READ_PX (256 Mpx). 256 Mpx sits cleanly in the empirical gap
between healthy mask reads (<= 4.9 Mpx) and the fatal 2740 Mpx mask, with
~52x headroom over healthy.
@clemsgrs
clemsgrs merged commit 0e8e0a1 into main Jun 26, 2026
1 check passed
@clemsgrs
clemsgrs deleted the hs2p-mask-read-size-guard branch June 26, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant