Fix large-floodplain OOM in LULC transition + vectorize (terra-native, changes_only) — v0.4.0#35
Merged
NewGraphEnvironment merged 10 commits intoJul 9, 2026
Conversation
Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
data-raw/benchmark_transition_oom.R: self-checking assertions for the terra ops the rewrite rests on (factor-strip via *1L with no input mutation, NA propagation, %in% FALSE-at-NA, freq value==code, empty-RHS %in% guard, classify/subst, set.cats on arithmetic raster) -- all pass on terra 1.9.11. Synthetic profiling confirms the two-driver diagnosis: on a 4.0M-cell coherent field, 89% of transition patches are stable X->X (polygonized then discarded by the caller), motivating changes_only. Relates to NewGraphEnvironment/sred#16 Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
terra writes a <raster>.tif.aux.xml stats sidecar whenever the transition tests or the OOM benchmark read the bundled fixtures; ignore so it stops surfacing as an untracked stray. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Snapshot the full behavior (summary tibble + raster factor levels/frequencies + removed raster) across 8 parameter combos as a canonicalized, order-independent digest via expect_snapshot_value(style=serialize). This freezes the pre-rewrite output so the terra-native rewrite in Phase 3 can be proven byte-identical. Empty case (impossible filter) is normalized: cats()[[1]] is NULL and freq() errors on an all-NA raster. Relates to NewGraphEnvironment/sred#16 Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
…ctors) Rewrite the producer to eliminate the OOM class (#28): strip factor to code rasters via `* 1L`, encode from*1000+to by raster arithmetic, filter from/to classes by integer code-set membership (terra::subst), remove small patches with patches/subst/ifel, and derive codes + total_valid + summary from a single native terra::freq. No terra::values(), no rep(NA, ncell) -- peak R memory is now O(#codes + #patches), not O(ncell). Behavior-identical: the Phase-2 golden snapshot stays byte-for-byte; full suite 303 pass / 4 skip; /code-check round 1 clean (independent old-vs-new comparison byte-identical across 14 param combos incl. ESA WorldCover 3-digit codes). Producer-only peak RSS at 16M cells (patch_area_min=500): 2.66 GB -> 1.63 GB. Two terra gotchas handled: SpatRaster `%in%` is not dispatched when terra is imported (not attached) -> use terra::subst; terra::freq() errors on an all-NA raster -> guard with tryCatch. Relates to NewGraphEnvironment/sred#16 Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
…tors (#34) Cap the vectorizer working set for large floodplains. changes_only=TRUE drops stable (from==to) patches at the raster level (subst+mask) before as.polygons, so only actual change patches are polygonized -- the field OOM lever, since the producer's transition raster is non-NA over the whole floodplain (stable mosaic) and the caller discards stable patches anyway. NA-ing stable cells can't merge or split any change patch (stable neighbours were already boundaries), so the result equals the default filtered to non-stable rows -- proven by test. Also pre-filter small patches (freq+subst small pids -> NA) before as.polygons when patch_area_min is set, keeping the trailing st_area filter so output stays byte-identical (pinned 185 / 123.11 ha / 57 unchanged). Fixed a pre-existing empty-return schema gap surfaced by /code-check: the early return dropped the zone column, which changes_only makes reachable (an all-stable sub-basin) -> per-sub-basin bind_rows mismatch. Empty return now carries zone_col. Fragmented 9M-cell benchmark (as.polygons-dominated, NECR-like geometry): default 415,705 patches / 3.83 GB -> changes_only 44,537 patches / 1.71 GB (55% peak cut). Full suite 314 pass / 4 skip; lint clean. Relates to NewGraphEnvironment/sred#16 Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
NEWS entry for the OOM fixes (producer terra-native rewrite; vectorizer changes_only + small-patch pre-filter). Document the drift-only decision's follow-up: the floodplains caller passes changes_only=TRUE and drops its post-hoc from!=to filter so NECR completes end-to-end (separate floodplains PR). Relates to NewGraphEnvironment/sred#16 Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Memory-scalable categorical land-cover change detection (#34, #28): terra-native dft_rast_transition (no full-grid R vectors) + dft_transition_vectors changes_only and small-patch pre-filter. Behavior-preserving; R CMD check clean (0/0/0). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
NewGraphEnvironment
deleted the
34-lulc-transition-classify-ooms-on-large-f
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the remaining OOM class in the categorical land-cover change pipeline for large-floodplain AOIs (#34, closing #28). #27 fixed the vectorizer's per-class loop; this fixes the two drivers that remained. Both fixes are behavior-preserving:
dft_rast_transition()rewritten to stream entirely through terra — transitions are encoded/filtered with raster arithmetic,terra::subst(),patches(), and a singleterra::freq(), with noterra::values()pull and no full-grid R vectors (previously 6+, incl. two full-grid character vectors). Peak memory is now O(#codes + #patches), not O(ncell). Byte-identical output, verified by a golden snapshot across the full parameter matrix plus an independent old-vs-new comparison (incl. ESA WorldCover 3-digit codes). Producer-only peak at 16M cells: 2.66 GB → 1.63 GB.dft_transition_vectors()gains opt-inchanges_only(defaultFALSE): drop stable (from == to) patches at the raster level beforeas.polygons, so only actual change patches are polygonized. Plus a small-patch raster pre-filter whenpatch_area_minis set (trailingst_areafilter kept → identical output). On a fragmented NECR-like synthetic (9M cells / 415k patches, whereas.polygonsdominates): 3.83 GB → 1.71 GB (55% cut).changes_only=TRUEequals the default output filtered to change patches (proven by test).Released as v0.4.0.
Two terra gotchas handled
%in%is not dispatched when terra is imported (not attached) → useterra::subst().terra::freq()errors on an all-NA raster (does not return 0 rows) → guarded withtryCatch.Follow-up (not in this PR — drift-only decision)
NECR completes end-to-end only once the
floodplainscaller opts in: passchanges_only = TRUEand drop the post-hocfrom != tofilter inscripts/floodplain_lcc/03_lulc_classify.R. Left as a separate floodplains change.Related Issues
Test plan
devtools::test()— 314 pass / 4 skip (network/bfast tests skip)R CMD check— 0 errors / 0 warnings / 0 noteslintr::lint_package()— cleandft_rast_transitionbyte-identical across the parameter matrixpatch_area_min=1000) unchanged/code-checkround 1 clean (one pre-existing empty-return fragility found + fixed)🤖 Generated with Claude Code
https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz