Skip to content

feat: make tiling the default input_prep, with ViT patch-grid safety guards#89

Merged
Dinghye merged 4 commits into
mainfrom
defaultinputprep
Jun 15, 2026
Merged

feat: make tiling the default input_prep, with ViT patch-grid safety guards#89
Dinghye merged 4 commits into
mainfrom
defaultinputprep

Conversation

@Dinghye

@Dinghye Dinghye commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Changes the default input_prep policy from "resize" to tiling, so that large on-the-fly inputs preserve native resolution by default instead of being silently downsampled to the model's image size.

  • New default. input_prep=None is now the "unset / use package default" sentinel and resolves to "tile". This default is applied consistently across get_embedding, get_embeddings_batch, Model(...), and ExportConfig. "resize" and "auto" remain available as explicit opt-ins.
  • Model-aware safety defaults. For image-level ViT patch-token-grid models (remoteclip, scalemae, satmae, satmaepp, satmaepp_s2_10b), an unset/"auto" input_prep resolves to "resize" instead of tile, because their tiled patch-token grids are not a seamless spatial mosaic and can show stitching seams. Requesting grid output on these models emits a warning; explicit "tile" still works but warns and records seam-risk metadata. The resolved embedding records the model policy in meta["input_prep"].
  • Soft + hard tile budgets. max_tiles default raised 9 → 64 and is now a soft threshold (requests above it still run but warn that they may be slow / memory-heavy). A new max_tiles_hard ceiling (default 1024, clamped >= max_tiles) raises to guard against runaway-area mistakes.
  • Graceful degradation. Because tiling is now the default, the tile path no longer hard-fails when a prerequisite is missing (no provider backend, no sensor, embedder doesn't accept prefetched CHW, or no derivable tile size). It degrades to a plain resize call and records resolved_mode="resize" in meta, so basic usage never breaks and the fallback stays auditable.
  • Reproducibility metadata. Every embedding now carries a versioned meta["input_prep"] block (INPUT_PREP_VERSION = 1) recording requested vs. resolved mode (and tiling layout / counts when tiled), regardless of which code path produced it.
  • remoteclip pooled/grid consistency fix. The single path now uses encode_image (canonical 512-d CLIP embedding) so pooled is identical across single, batch, and tiled paths instead of being a 768-d raw-token mean on the single path. grid extraction was rewritten to read open_clip's forward_intermediates patch grid. See docs/models/remoteclip.md.

Why: the previous "resize" default quietly discarded spatial resolution for larger ROIs, and tile mode was easy to trip into hard errors. This makes the high-fidelity behavior the default while guarding the cases (ViT patch grids, huge areas) where naive tiling misleads.

Testing

How did you verify the change?

  • I ran the relevant tests locally.
  • I updated docs for user-facing behavior changes.
  • I updated CHANGELOG.md for user-facing API, model, semantic, or installation changes.
  • This PR does not need a changelog entry.

New/updated coverage: tests/test_model_aware_input_prep.py (default resolves to resize with metadata for ViT-grid models; explicit tile allowed but marked experimental), plus updated test_input_prep_tiling.py, test_specs.py, test_api.py, test_batch_overrides_all_models.py, test_combined_flow.py, test_export_batch.py, and test_scalemae_preprocess_alignment.py.

Notes

  • Behavior change (not strictly backward compatible). Code relying on the old "resize" default will now tile by default; results for large ROIs change shape/values. Pin input_prep="resize" to keep prior behavior. Documented in CHANGELOG.md under Changed.
  • The ViT patch-grid models intentionally opt out of the new default and stay on resize — pooled output is the recommended path for those.
  • Docs updated: choosing_settings.md, concepts.md, api_embedding.md, api_specs.md, and the per-model pages for the affected ViT models.

@Dinghye Dinghye merged commit 8552f74 into main Jun 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant