From e7a238c9830edc460455e9dee96902e9ad87776c Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 10:43:35 -0700 Subject: [PATCH 01/10] Initialize PWF baseline for #34 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- planning/active/findings.md | 121 +++++++++++++++++++++++++++++++++++ planning/active/progress.md | 18 ++++++ planning/active/task_plan.md | 62 ++++++++++++++++++ 3 files changed, 201 insertions(+) create mode 100644 planning/active/findings.md create mode 100644 planning/active/progress.md create mode 100644 planning/active/task_plan.md diff --git a/planning/active/findings.md b/planning/active/findings.md new file mode 100644 index 0000000..b14d7bb --- /dev/null +++ b/planning/active/findings.md @@ -0,0 +1,121 @@ +# Findings — LULC transition/classify OOMs on large-floodplain AOIs (#34, closes #28) + +## Issue context (#34) + +The land-cover step (`dft_rast_classify` → `dft_rast_transition` → +`dft_transition_vectors`) OOMs on a large-floodplain AOI even solo with ~30 GB free. +#27 fixed the per-class loop in the vectorizer, but the pass still materializes +full-grid rasters into R vectors and builds one polygon per transition patch. + +Decisive field clue: **UFRA has a *larger* bounding box (1.8×0.8°, 102.6M-cell grid) +than NECR (1.4×0.6°) but a *smaller* floodplain (188 vs 551 km²), and UFRA completes +while NECR is SIGKILLed.** So the field driver is floodplain-area / transition-patch +count, not raster grid extent. (UFRA peaked ~13 GB at 188 km²; NECR at 3× the +floodplain exceeds ~30 GB.) #28 is the precise producer diagnosis; #34 is the field +report — same OOM class, resolved together here. + +## Two independent memory drivers (exploration 2026-07-09) + +Resolves the apparent contradiction (NECR OOMs on a *smaller* grid than UFRA): + +1. **Producer, ncell-driven (~13 GB floor at 102.6M cells) = #28.** + `R/dft_rast_transition.R:89–109` builds 6 persistent full-grid R vectors — + `v_from`, `v_to` (double), **`name_from`, `name_to` (CHARACTER, ~800 MB each)**, + `trans_code` (double), `keep` (logical) — plus transient `as.character()` spikes. + The two character vectors are consumed ONLY by the optional `from_class`/`to_class` + filters, so they are **pure waste when both are NULL (the default)**. The + `patch_area_min` path adds `rep(NA_integer_, ncell)` (`:122`, `:191`) + full-grid + `terra::values()` reads (`:131`) + a full-grid `terra::patches()` (`:125`). Scales + with grid cells → matches UFRA's ~13 GB. Present every call, survivable at 30 GB. + +2. **Vectorizer, floodplain-area-driven = the field OOM that kills NECR.** + The producer's transition raster is **non-NA over the *entire* floodplain** — it + encodes stable `from==to` transitions too (never filtered; the `keep` mask only + filters by from/to-class and NA, `:102–109`). The field caller + `floodplains/scripts/floodplain_lcc/03_lulc_classify.R:107` calls + `dft_transition_vectors(trans_all$raster, zones=…)` **with no `patch_area_min`**, so + `terra::as.polygons()` polygonizes the whole floodplain's stable mosaic, then the + caller discards stable patches (`from_class != to_class`, `:115`) *after* paying the + cost. `as.polygons` scales with patch/vertex count = floodplain area → NECR (551 km², + 3× UFRA) blows past 30 GB. + +**`dft_rast_classify` and `dft_rast_summarize` are already terra-native** (no +`terra::values()`; `terra::unique`/`classify`/`freq` only). #34's claim that classify +participates in the OOM is **not founded** at the R level. + +## Current producer structure (R/dft_rast_transition.R, v0.3.0) + +- Signature: `dft_rast_transition(x, from, to, class_table=NULL, source="io-lulc", + from_class=NULL, to_class=NULL, unit="ha", patch_area_min=NULL)`. +- Encoding: `trans_code = from_code * 1000L + to_code`. +- Returns `list(raster, summary, removed)`; summary cols `from_class, to_class, + n_cells, area, pct`; raster factor levels `"from -> to"`; `removed` is NULL unless + `patch_area_min` filtered. +- Summary already uses `terra::freq(r_trans)` (native); only full-grid R op there is + `total_valid <- sum(!is.na(trans_code))` (`:171`) + `valid` (`:140`). + +## Vectorizer (R/dft_transition_vectors.R, post-#27) + +- Signature: `dft_transition_vectors(x, zones=NULL, zone_col=NULL, patch_area_min=NULL)`. +- Single pass `terra::patches(x, directions=8, values=TRUE)` → `as.polygons` → sparse + pid→label via `terra::cells()`/`terra::extract()`; filters `patch_area_min` by + `st_area` AFTER polygonizing. +- Returns sf: `patch_id` (scan-order), `transition`, `area_ha` (+ zone col). Does NOT + filter stable transitions — caller does. + +## Terra semantics — empirically verified (Plan agent, terra 1.9.11 in-memory) + +Load-bearing facts the rewrite rests on: +- `r_from * 1L` **strips the factor** → new non-factor raster of raw codes, NA + preserved, and **does not mutate `r_from`** (stays factor) — critical: inputs are + references into the caller's list; no `deepcopy` needed. +- NA propagates through `* / + / != / & / ifel / %in%`. +- `code_r %in% c(...)` returns a boolean raster, **FALSE (not NA) at NA cells** → + reproduces base-R `name %in% sel` NA behavior exactly. +- `terra::freq(integer_raster)`: `value == code`, NA excluded, `sum(count) == + global(!is.na)` → both `unique_codes` and `total_valid` come free from one `freq`. +- `set.cats()` on a double-valued arithmetic raster (ids like 2005) → `is.factor==TRUE`. +- **terra SpatRaster `%in%` ERRORS on an empty RHS** (`[%in%] no matches supplied`), + unlike base R → every `p %in% ids` / `code %in% keep` needs a `length()>0` guard or + all-NA shortcut. + +**Gate before trusting:** re-verify facts 1–5 at the DESCRIPTION floor **terra 1.8-10** +(bump floor if any differ). Verified on installed 1.9.11 only. + +## Correctness contract (existing tests the rewrite must keep green) + +- `test-dft_rast_transition.R`: return shape `c("raster","summary","removed")`; summary + cols; `pct` sums 100 (±0.1; ±0.5 with `patch_area_min`); sorted by `n_cells` desc; + labels contain `" -> "`; ha = 100×km2; `patch_area_min` NULL≡0≡default; raster non-NA + count == `sum(summary$n_cells)`; validation errors "non-negative"; `$removed` + conservation law `n_kept + n_removed == n_changed_unfiltered`. +- `test-dft_transition_vectors.R`: **pinned regression 185 patches / 123.11 ha / 57 at + `patch_area_min=1000`**; `sum(area_ha)==sum(summary$area)`; every `transition` in + `cats(raster)`; `patch_id` unique; all-NA → empty sf. +- Fixtures: no precomputed goldens; tests re-read `inst/extdata/example_2017|2020|2023.tif` + (326×314, EPSG:32609, raw io-lulc codes) and re-run classify/transition inline. Reusable + synthetic builder `transition_test_rast(m)` at `test-dft_transition_vectors.R:113`. + +## Vectorizer mitigation design (Plan agent) + +- **`changes_only`** (NECR-saving lever): stable ids are `id %/% 1000L == id %% 1000L`; + NA-mask at raster level (`classify`/`subst` on `x*1L`) before `patches`/`as.polygons`. + Opt-in default FALSE → pinned 185/123.11/57 untouched. +- **Small-patch raster pre-filter**: `classify` small pids→NA before `as.polygons`, KEEP + the trailing `st_area` filter. The raster pre-filter's `count*cell_area < min` drops + are a **strict subset** of the `st_area` `< min` drops → output provably byte-identical + (57 preserved). `patch_id` densens under filtering (not test-pinned; NEWS note). + +## Cross-repo follow-up (drift-only decision) + +After the drift 0.4.0 release, wire the field caller: +`floodplains/scripts/floodplain_lcc/03_lulc_classify.R` → pass `changes_only = TRUE` to +`dft_transition_vectors()` (`:107`) and drop the post-hoc `from_class != to_class` +filter (`:115`). Only then does NECR complete end-to-end. Separate floodplains PR/issue. + +## Git base note + +Branch `34-lulc-transition-classify-ooms-on-large-f` is off `origin/main` (6ba10bb), NOT +local main — local main carries an unpushed memory-audit doc commit (`a5ef052`, +CLAUDE.md + inst/notes/) kept out of this PR's diff. User pushes `a5ef052` to main +separately. diff --git a/planning/active/progress.md b/planning/active/progress.md new file mode 100644 index 0000000..17fac9d --- /dev/null +++ b/planning/active/progress.md @@ -0,0 +1,18 @@ +# Progress — LULC transition/classify OOMs on large-floodplain AOIs (#34, closes #28) + +## Session 2026-07-09 + +- Committed pending memory-audit docs to `main` first (`a5ef052`: CLAUDE.md + + inst/notes/gdalcubes-pc-gotchas.md, soul#47) so they stay out of this PR. +- Plan-mode exploration: 2 Explore agents (producers + vectorizer/test harness) + + 1 Plan agent (terra-native rewrite design, terra-1.9.11 semantics verified) + read + the `floodplains` field caller. Diagnosed **two independent memory drivers** + (producer ncell-driven = #28; vectorizer floodplain-area-driven stable-mosaic + polygonize = the NECR field OOM). Reconciled #34 ≡ #28's OOM class. +- User decisions: both fixes on one branch (close #34 + #28), single 0.4.0 release; + drift-only with documented `floodplains` caller follow-up. +- Created branch `34-lulc-transition-classify-ooms-on-large-f` off `origin/main` + (6ba10bb), keeping `a5ef052` on local main only. +- Scaffolded PWF baseline with approved phases. +- Next: Phase 1 — `data-raw/benchmark_transition_oom.R` (terra semantics gate at the + 1.8-10 floor + synthetic profiling attributing the two drivers). diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md new file mode 100644 index 0000000..e71949a --- /dev/null +++ b/planning/active/task_plan.md @@ -0,0 +1,62 @@ +# Task: LULC transition/classify OOMs on large-floodplain AOIs (#34, closes #28) + +`dft_rast_transition()` → `dft_transition_vectors()` OOM on large-floodplain AOIs +(NECR, 551 km²) even solo with ~30 GB free. Two independent memory drivers: +(1) the **producer** builds 6+ full-grid R vectors incl. two full-grid character +vectors (ncell-driven ~13 GB floor — this is #28); (2) the **vectorizer** +polygonizes the whole floodplain's *stable* mosaic before the caller discards it +(floodplain-area-driven — the field OOM that kills NECR). Fix both on one branch, +close #34 + #28, single 0.4.0 release. drift-only; `floodplains` caller wiring is a +documented follow-up. + +Full design + verified terra semantics in `findings.md`. + +## Phase 1: Empirical gate + profiling harness (data-raw/, zero DESCRIPTION footprint) +- [ ] `data-raw/benchmark_transition_oom.R`: (a) semantics smoke — assert `*1L` + factor-strip/NA/no-mutation, NA propagation through `*/+/!=/&/ifel/%in%`, + `freq(integer)` value==code + NA-excluded + `sum(count)==n_nonNA`, empty-RHS + `%in%`/`classify`/`subst` guarded, `p %in% small_ids` FALSE at `p==NA`, `set.cats` + on double-arith raster → factor (run on installed terra; bump floor if 1.8-10 + differs). (b) profiling — synthetic classified pair with independently-tunable + ncell and transition-density; profile `classify → transition(±patch_area_min) → + vectors(±changes_only)` peak RSS via `/usr/bin/time -l`; before/after table + attributing the two drivers. Proceed only if semantics gates pass. + +## Phase 2: Golden-output correctness harness (capture current behavior first) +- [ ] Extend `tests/testthat/test-dft_rast_transition.R` with a byte-identical golden + capture on the packaged fixture across: default; `from_class="Trees"`; both filters; + `patch_area_min ∈ {NULL,0,500,1000,1e9}`; impossible-filter. Snapshot `summary`, + `raster` cats + `freq(value,count)`, `removed` cats. This is the rewrite's contract. + +## Phase 3: Producer terra-native rewrite (#28) +- [ ] Rewrite `R/dft_rast_transition.R`: `code_from/code_to <- r_from/r_to * 1L`; + `r_trans <- code_from*1000L + code_to`; `apply_codeset()` `@noRd` for from/to filters + (empty-set guard); streamed `patch_area_min` via `ifel`/`patches`/`freq`/`ifel` + (empty-id guard); freq-derived codes + `total_valid`; removed raster via `ifel` + + freq cats; preserve empty-return `removed=NULL`. No `terra::values()`, no `rep(NA,ncell)`. +- [ ] Green: full `test-dft_rast_transition.R` + Phase-2 golden identical. + +## Phase 4: Vectorizer working-set cap (#34 core) +- [ ] `R/dft_transition_vectors.R`: add `changes_only = FALSE` (raster-level stable + drop before `as.polygons`); add small-patch raster pre-filter (keep trailing + `st_area` filter). Roxygen: document `changes_only`; note `patch_id` densening + under filtering. +- [ ] New tests: `changes_only=TRUE` drops stable / keeps changes / survivor `area_ha` + unchanged; existing 185/123.11/57 stay green on the default path; `as.polygons` + peak drops on the Phase-1 synthetic high-patch grid. + +## Phase 5: Docs, NEWS, release, close #34 + #28 +- [ ] `devtools::document()`; `lintr::lint_package()` clean; full `devtools::test()`. +- [ ] `NEWS.md` 0.4.0: producer terra-native rewrite (no full-grid R vectors, OOM); + vectorizer `changes_only` + small-patch pre-filter; behavior-preserving defaults; + `patch_id` densening note. +- [ ] Document `floodplains` follow-up (caller passes `changes_only=TRUE`, drops the + `:115` `from!=to` post-filter) in progress.md / a floodplains issue. +- [ ] Bump `DESCRIPTION` 0.3.0 → 0.4.0 as the **final** commit; terra floor bump only + if Phase 1 required it. + +## Validation +- [ ] Tests pass (`devtools::test()`); network/bfast tests skip cleanly +- [ ] `/code-check` clean on each commit +- [ ] PWF checkboxes match landed work +- [ ] `/planning-archive` on completion From 845ff117eb7be6024bbb12fac4022780400ba907 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 10:49:43 -0700 Subject: [PATCH 02/10] Phase 1: terra-semantics gate + OOM profiling harness (#34) data-raw/benchmark_transition_oom.R: self-checking assertions for the terra ops the rewrite rests on (factor-strip via *1L with no input mutation, NA propagation, %in% FALSE-at-NA, freq value==code, empty-RHS %in% guard, classify/subst, set.cats on arithmetic raster) -- all pass on terra 1.9.11. Synthetic profiling confirms the two-driver diagnosis: on a 4.0M-cell coherent field, 89% of transition patches are stable X->X (polygonized then discarded by the caller), motivating changes_only. Relates to NewGraphEnvironment/sred-2025-2026#16 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- data-raw/benchmark_transition_oom.R | 183 ++++++++++++++++++++++++++++ planning/active/findings.md | 18 +++ planning/active/progress.md | 9 +- planning/active/task_plan.md | 2 +- 4 files changed, 209 insertions(+), 3 deletions(-) create mode 100644 data-raw/benchmark_transition_oom.R diff --git a/data-raw/benchmark_transition_oom.R b/data-raw/benchmark_transition_oom.R new file mode 100644 index 0000000..ba120b0 --- /dev/null +++ b/data-raw/benchmark_transition_oom.R @@ -0,0 +1,183 @@ +# Benchmark + terra-semantics gate for the #34 / #28 OOM fix. +# +# Two independent memory drivers in the land-cover change pipeline: +# (1) dft_rast_transition() (producer) builds 6+ full-grid R vectors incl. two +# full-grid CHARACTER vectors -> ncell-driven ~13 GB floor (issue #28). +# (2) dft_transition_vectors() polygonizes the whole floodplain's *stable* +# mosaic before the caller discards it -> floodplain-area-driven (the field +# OOM that kills NECR, 551 km^2). +# +# This script is data-raw only (NOT sourced by the package, zero DESCRIPTION +# footprint). It (A) asserts the terra semantics the rewrite rests on, and (B) +# profiles R-side peak memory of the pipeline on a synthetic classified pair with +# independently-tunable grid size and transition density. +# +# Usage: +# Rscript data-raw/benchmark_transition_oom.R semantics +# Rscript data-raw/benchmark_transition_oom.R profile +# +# Verified on terra 1.9.11 / R 4.5.2. The ops used (factor-strip via `*1L`, ifel, +# freq, classify, subst, set.cats, SpatRaster `%in%`) are stable terra features +# predating the DESCRIPTION floor 1.8-10; the only floor-relevant fix is the +# patches(values=TRUE) edge-wraparound bug (rspatial/terra#1675), unrelated here. + +suppressPackageStartupMessages(library(terra)) + +# ---- helpers --------------------------------------------------------------- + +# small projected factor raster mimicking io-lulc codes, from an integer matrix +mk_factor_rast <- function(m, codes = sort(unique(as.vector(m[!is.na(m)])))) { + r <- terra::rast(nrows = nrow(m), ncols = ncol(m), + xmin = 0, xmax = ncol(m) * 10, ymin = 0, ymax = nrow(m) * 10, + crs = "EPSG:32609") + terra::values(r) <- as.vector(t(m)) + terra::set.cats(r, layer = 1, + value = data.frame(id = codes, + class_name = paste0("class_", codes))) + r +} + +vals <- function(r) terra::values(r)[, 1] + +# ---- (A) terra-semantics gate ---------------------------------------------- + +run_semantics <- function() { + ok <- function(cond, msg) if (!isTRUE(cond)) stop("SEMANTICS FAIL: ", msg, call. = FALSE) + + # 3x3 with an out-of-AOI NA cell; codes mimic io-lulc (2 Trees, 5 Crops, 7 Built...) + mf <- matrix(c(2, 5, 7, + 5, NA, 2, + 7, 2, 5), nrow = 3, byrow = TRUE) + mt <- matrix(c(2, 2, 7, # (1,1) stable 2->2; others change + 5, NA, 5, + 7, 5, 2), nrow = 3, byrow = TRUE) + r_from <- mk_factor_rast(mf) + r_to <- mk_factor_rast(mt) + + # 1. `*1L` strips factor -> raw codes, NA preserved, DOES NOT mutate input + code_from <- r_from * 1L + ok(!terra::is.factor(code_from), "`r_from * 1L` should be non-factor") + ok(terra::is.factor(r_from), "`r_from * 1L` must NOT mutate r_from (still factor)") + ok(identical(which(is.na(vals(code_from))), which(is.na(vals(r_from)))), + "`*1L` must preserve NA positions") + ok(all(vals(code_from) == vals(r_from) | is.na(vals(code_from))), + "`*1L` values must equal raw codes") + code_to <- r_to * 1L + + # 2. NA propagates through `* / +` + r_trans <- code_from * 1000L + code_to + na_expected <- is.na(vals(code_from)) | is.na(vals(code_to)) + ok(identical(which(is.na(vals(r_trans))), which(na_expected)), + "trans_code arithmetic must be NA where either input is NA") + ok(vals(r_trans)[1] == 2002, "encoding from*1000+to wrong") # (1,1): 2->2 + + # 3. SpatRaster `%in%` is FALSE (not NA) at NA cells -> matches base `name %in% sel` + sel <- code_from %in% c(2, 5) + ok(vals(sel)[which(na_expected)[1]] == 0, + "`code %in% set` must be FALSE (0), not NA, at NA cells") + ok(vals(sel)[3] == 0, "code 7 should be FALSE for %in% c(2,5)") + + # 4. freq(integer raster): value==code, NA excluded, sum(count)==n_nonNA + ft <- terra::freq(r_trans) + ok(!any(is.na(ft$value)), "freq() must exclude NA by default") + ok(sum(ft$count) == sum(!is.na(vals(r_trans))), + "sum(freq$count) must equal non-NA cell count (total_valid)") + ok(2002 %in% ft$value, "freq value must equal the actual code") + + # 5. empty-RHS `%in%` ERRORS (guard needed); classify/subst with a set work + err <- tryCatch({ code_from %in% integer(0); FALSE }, error = function(e) TRUE) + ok(err, "SpatRaster `%in%` on empty RHS must error (so we guard length>0)") + cl <- terra::classify(code_from, cbind(7, NA)) + ok(all(is.na(vals(cl)[vals(code_from) == 7 & !is.na(vals(code_from))])), + "classify(cbind(id, NA)) must NA-out the id") + sb <- terra::subst(code_from, 7, NA) + ok(all(is.na(vals(sb)[vals(code_from) == 7 & !is.na(vals(code_from))])), + "subst(id, NA) must NA-out the id") + + # 6. patches + `p %in% small_ids` FALSE at p==NA + r_changed <- terra::ifel(!is.na(r_trans) & (code_from != code_to), 1L, NA) + p <- terra::patches(r_changed, directions = 8) + small <- terra::freq(p)$value[1] # pick one patch id + sm <- p %in% small + ok(all(vals(sm)[is.na(vals(p))] == 0), + "`p %in% ids` must be FALSE at p==NA cells") + + # 7. set.cats on the double-valued arithmetic raster -> factor, correct labels + codes <- sort(ft$value) + terra::set.cats(r_trans, layer = 1, + value = data.frame(id = codes, + transition = paste0(codes %/% 1000L, "->", + codes %% 1000L))) + ok(terra::is.factor(r_trans), "set.cats on arithmetic raster must yield a factor") + ok(any(grepl("->", terra::freq(r_trans)$value)), + "factor freq must return the transition labels") + + cat("SEMANTICS GATE: ALL PASS (terra", as.character(packageVersion("terra")), ")\n") +} + +# ---- (B) synthetic profiling ------------------------------------------------ + +# build a classified pair with COHERENT patches (a coarse seeded random field +# disaggregated to full res, ~floodplain-like block sizes), NOT salt-and-pepper. +# `change_frac` of coarse cells change class between years (drives transition-patch +# count); the rest are stable. ~10% NA (out-of-AOI mask). Seeded for reproducibility. +mk_synthetic_pair <- function(ncol, nrow, change_frac = 0.05, block = 20L) { + codes <- c(2L, 5L, 7L, 8L, 11L) # io-lulc-ish subset + set.seed(42) + cn <- ceiling(ncol / block); rn <- ceiling(nrow / block) + coarse_from <- sample(codes, cn * rn, replace = TRUE) + coarse_to <- coarse_from + chg <- sample(cn * rn, size = round(change_frac * cn * rn)) + coarse_to[chg] <- vapply(coarse_from[chg], + function(cf) sample(setdiff(codes, cf), 1), integer(1)) + # ~10% NA blocks (out-of-AOI); same mask both years + na_blk <- sample(cn * rn, size = round(0.10 * cn * rn)) + coarse_from[na_blk] <- NA; coarse_to[na_blk] <- NA + + mk <- function(v) { + rc <- terra::rast(nrows = rn, ncols = cn, xmin = 0, xmax = cn * block * 10, + ymin = 0, ymax = rn * block * 10, crs = "EPSG:32609") + terra::values(rc) <- v + r <- terra::disagg(rc, fact = block) # blow up to full res + r <- r[[1]] + terra::set.cats(r, layer = 1, + value = data.frame(id = codes, class_name = paste0("class_", codes))) + r + } + list("2017" = mk(coarse_from), "2023" = mk(coarse_to)) +} + +run_profile <- function(ncol, nrow, change_frac) { + devtools::load_all(quiet = TRUE) + pair <- mk_synthetic_pair(ncol, nrow, change_frac) + cat(sprintf("Synthetic %d x %d = %.1fM cells, change_frac=%.3f\n", + terra::ncol(pair[[1]]), terra::nrow(pair[[1]]), + terra::ncell(pair[[1]]) / 1e6, change_frac)) + # Whole-process peak RSS is the honest meter (run under `/usr/bin/time -l`). + # Patch count is the unambiguous O(patches) proxy: dft_transition_vectors + # polygonizes ALL patches (stable mosaic + changes); `changes_only` will drop + # the stable ones. Report both so before/after (this branch) is comparable. + trans <- drift::dft_rast_transition(pair, from = "2017", to = "2023", + patch_area_min = 500) + v_all <- drift::dft_transition_vectors(trans$raster) # current behavior (all) + n_stable <- sum(vapply(strsplit(v_all$transition, " -> ", fixed = TRUE), + function(p) identical(p[1], p[2]), logical(1))) + cat(sprintf("dft_transition_vectors (current): %d patches total, %d stable (%.0f%%), %d change\n", + nrow(v_all), n_stable, 100 * n_stable / nrow(v_all), + nrow(v_all) - n_stable)) +} + +# ---- dispatch --------------------------------------------------------------- + +arg_or <- function(x, default) if (length(x) == 0 || is.na(x)) default else x + +args <- commandArgs(trailingOnly = TRUE) +if (length(args) == 0 || args[1] == "semantics") { + run_semantics() +} else if (args[1] == "profile") { + run_profile(as.integer(arg_or(args[2], 3000)), + as.integer(arg_or(args[3], 3000)), + as.numeric(arg_or(args[4], 0.05))) +} else { + stop("unknown mode: ", args[1]) +} diff --git a/planning/active/findings.md b/planning/active/findings.md index b14d7bb..ed50a11 100644 --- a/planning/active/findings.md +++ b/planning/active/findings.md @@ -113,6 +113,24 @@ After the drift 0.4.0 release, wire the field caller: `dft_transition_vectors()` (`:107`) and drop the post-hoc `from_class != to_class` filter (`:115`). Only then does NECR complete end-to-end. Separate floodplains PR/issue. +## Phase 1 results (2026-07-09, `data-raw/benchmark_transition_oom.R`) + +- **Semantics gate: ALL PASS** on terra 1.9.11 — all 7 op-groups the rewrite rests on + hold (`*1L` factor-strip + no-mutation + NA-preserve; NA propagation through + `*/+/!=/&/ifel/%in%`; `%in%` FALSE-at-NA; `freq(int)` value==code / NA-excluded / + `sum(count)==n_nonNA`; empty-RHS `%in%` errors → guard confirmed; `classify`/`subst` + NA-out; `set.cats` on double-arith raster → factor). Ops used predate the 1.8-10 + floor (stable terra features) → floor NOT bumped. +- **Profiling (synthetic 2000×2000 = 4.0M cells, coherent 20-cell blocks, change_frac + 0.05):** the transition raster has **4200 patches, of which 3748 (89%) are stable + `X -> X`** and only 452 are real changes. The current `dft_transition_vectors` + polygonizes ALL 4200; the field caller then discards the 3748 stable. → `changes_only` + cuts the `as.polygons` working set ~9× on this shape (and more as floodplain area + grows). Whole-process peak RSS ≈ 1.24 GB at 4M cells (current code); the R-side + full-grid vectors + stable-mosaic polygonize are what scale to NECR's OOM. +- Confirms the two-driver diagnosis and the fix targets. Semantics-gate assertions are + the pre-implementation gate for Phase 3; they passed, so the rewrite proceeds. + ## Git base note Branch `34-lulc-transition-classify-ooms-on-large-f` is off `origin/main` (6ba10bb), NOT diff --git a/planning/active/progress.md b/planning/active/progress.md index 17fac9d..e54a910 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -14,5 +14,10 @@ - Created branch `34-lulc-transition-classify-ooms-on-large-f` off `origin/main` (6ba10bb), keeping `a5ef052` on local main only. - Scaffolded PWF baseline with approved phases. -- Next: Phase 1 — `data-raw/benchmark_transition_oom.R` (terra semantics gate at the - 1.8-10 floor + synthetic profiling attributing the two drivers). +- **Phase 1 done** (commit pending): `data-raw/benchmark_transition_oom.R`. Semantics + gate ALL PASS on terra 1.9.11 (floor not bumped — ops predate 1.8-10). Profiling on a + 4.0M-cell coherent synthetic: 4200 transition patches, **89% stable** (discarded by + the caller) → `changes_only` ~9× working-set cut; ~1.24 GB peak RSS (current code). + Two-driver diagnosis confirmed; rewrite cleared to proceed. +- Next: Phase 2 — golden-output correctness harness (capture current + `dft_rast_transition` outputs across the param matrix before the rewrite). diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index e71949a..43c883f 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -12,7 +12,7 @@ documented follow-up. Full design + verified terra semantics in `findings.md`. ## Phase 1: Empirical gate + profiling harness (data-raw/, zero DESCRIPTION footprint) -- [ ] `data-raw/benchmark_transition_oom.R`: (a) semantics smoke — assert `*1L` +- [x] `data-raw/benchmark_transition_oom.R`: (a) semantics smoke — assert `*1L` factor-strip/NA/no-mutation, NA propagation through `*/+/!=/&/ifel/%in%`, `freq(integer)` value==code + NA-excluded + `sum(count)==n_nonNA`, empty-RHS `%in%`/`classify`/`subst` guarded, `p %in% small_ids` FALSE at `p==NA`, `set.cats` From 7e512cd9d82ba68afc6c012b1b63a522e5790c25 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 10:50:28 -0700 Subject: [PATCH 03/10] Ignore terra .aux.xml sidecars in inst/extdata terra writes a .tif.aux.xml stats sidecar whenever the transition tests or the OOM benchmark read the bundled fixtures; ignore so it stops surfacing as an untracked stray. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index e3e6e2c..0e4183a 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,6 @@ vignettes/*.html # local STAC cube cache (generated by data-raw/vignette_data_break.R) data-raw/.break_cache/ + +# terra sidecars regenerated when tests/benchmarks read inst/extdata rasters +inst/extdata/*.aux.xml From 364f4680c1f9764901745c7e40cfe08dc90b0200 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 10:55:37 -0700 Subject: [PATCH 04/10] Phase 2: golden-output contract for dft_rast_transition (#34) Snapshot the full behavior (summary tibble + raster factor levels/frequencies + removed raster) across 8 parameter combos as a canonicalized, order-independent digest via expect_snapshot_value(style=serialize). This freezes the pre-rewrite output so the terra-native rewrite in Phase 3 can be proven byte-identical. Empty case (impossible filter) is normalized: cats()[[1]] is NULL and freq() errors on an all-NA raster. Relates to NewGraphEnvironment/sred-2025-2026#16 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- planning/active/progress.md | 9 +- planning/active/task_plan.md | 2 +- tests/testthat/_snaps/dft_rast_transition.md | 326 +++++++++++++++++++ tests/testthat/test-dft_rast_transition.R | 66 ++++ 4 files changed, 400 insertions(+), 3 deletions(-) create mode 100644 tests/testthat/_snaps/dft_rast_transition.md diff --git a/planning/active/progress.md b/planning/active/progress.md index e54a910..8433a96 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -19,5 +19,10 @@ 4.0M-cell coherent synthetic: 4200 transition patches, **89% stable** (discarded by the caller) → `changes_only` ~9× working-set cut; ~1.24 GB peak RSS (current code). Two-driver diagnosis confirmed; rewrite cleared to proceed. -- Next: Phase 2 — golden-output correctness harness (capture current - `dft_rast_transition` outputs across the param matrix before the rewrite). +- **Phase 2 done** (commit pending): added a golden-snapshot test to + `test-dft_rast_transition.R` capturing summary + raster cats/freq + removed across + 8 param combos (default, from_class, both filters, patch_area_min ∈ {0,500,1000,1e9}, + impossible). Canonicalized (sorted) digest → content-strict, order-independent. + `expect_snapshot_value(style="serialize")` golden at `_snaps/dft_rast_transition.md`. + Fixed empty-case handling (cats()[[1]] NULL + freq() errors on all-NA). 44 pass, 0 skip. +- Next: Phase 3 — terra-native rewrite of `dft_rast_transition`; the golden must stay green. diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index 43c883f..a07eaaa 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -23,7 +23,7 @@ Full design + verified terra semantics in `findings.md`. attributing the two drivers. Proceed only if semantics gates pass. ## Phase 2: Golden-output correctness harness (capture current behavior first) -- [ ] Extend `tests/testthat/test-dft_rast_transition.R` with a byte-identical golden +- [x] Extend `tests/testthat/test-dft_rast_transition.R` with a byte-identical golden capture on the packaged fixture across: default; `from_class="Trees"`; both filters; `patch_area_min ∈ {NULL,0,500,1000,1e9}`; impossible-filter. Snapshot `summary`, `raster` cats + `freq(value,count)`, `removed` cats. This is the rewrite's contract. diff --git a/tests/testthat/_snaps/dft_rast_transition.md b/tests/testthat/_snaps/dft_rast_transition.md new file mode 100644 index 0000000..4c8a461 --- /dev/null +++ b/tests/testthat/_snaps/dft_rast_transition.md @@ -0,0 +1,326 @@ +# dft_rast_transition output is stable across the terra-native rewrite (#34) + + WAoAAAACAAQFAgACAwAAAAITAAAACAAAAhMAAAAEAAADEwAAAAUAAAAQAAAAEwAEAAkAAAAK + QnVpbHQgQXJlYQAEAAkAAAAKQnVpbHQgQXJlYQAEAAkAAAAKQnVpbHQgQXJlYQAEAAkAAAAF + Q3JvcHMABAAJAAAAEkZsb29kZWQgVmVnZXRhdGlvbgAEAAkAAAAJUmFuZ2VsYW5kAAQACQAA + AAlSYW5nZWxhbmQABAAJAAAACVJhbmdlbGFuZAAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAlS + YW5nZWxhbmQABAAJAAAACFNub3cvSWNlAAQACQAAAAhTbm93L0ljZQAEAAkAAAAFVHJlZXMA + BAAJAAAABVRyZWVzAAQACQAAAAVUcmVlcwAEAAkAAAAFVHJlZXMABAAJAAAABVRyZWVzAAQA + CQAAAAVXYXRlcgAEAAkAAAAFV2F0ZXIAAAAQAAAAEwAEAAkAAAAKQnVpbHQgQXJlYQAEAAkA + AAAIU25vdy9JY2UABAAJAAAABVRyZWVzAAQACQAAAAlSYW5nZWxhbmQABAAJAAAACVJhbmdl + bGFuZAAEAAkAAAAFQ3JvcHMABAAJAAAACVJhbmdlbGFuZAAEAAkAAAAIU25vdy9JY2UABAAJ + AAAABVRyZWVzAAQACQAAAAVXYXRlcgAEAAkAAAAIU25vdy9JY2UABAAJAAAABVRyZWVzAAQA + CQAAAAVDcm9wcwAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAhTbm93L0ljZQAEAAkAAAAFVHJl + ZXMABAAJAAAABVdhdGVyAAQACQAAAAlSYW5nZWxhbmQABAAJAAAABVdhdGVyAAAADQAAABMA + AAAKAAAALAAAAAEAAAPmAAAAAgAAAJIAAAtcAAAAEwAAAD4AAAAzAAAAAQAAAAEAAAABAAAF + lQAAAHkAABVmAAAAYgAAAAEAAAOsAAAADgAAABM/uZmZmZmZmj/cKPXCj1wpP4R64UeuFHtA + I/XCj1wo9j+UeuFHrhR7P/dcKPXCj1xAPRR64UeuFT/IUeuFHrhSP+PXCj1wo9c/4FHrhR64 + Uj+EeuFHrhR7P4R64UeuFHs/hHrhR64Ue0AslHrhR64VP/NcKPXCj1xAS2PXCj1wpD/vXCj1 + wo9cP4R64UeuFHtAIszMzMzMzQAAAA4AAAATP7R64UeuFHs/1wo9cKPXCj+EeuFHrhR7QCA4 + UeuFHrg/lHrhR64Uez/zCj1wo9cKQDeeuFHrhR8/wzMzMzMzMz/gAAAAAAAAP9o9cKPXCj0/ + hHrhR64Uez+EeuFHrhR7P4R64UeuFHtAJzhR64UeuD/vXCj1wo9cQEZAAAAAAAA/6ZmZmZmZ + mj+EeuFHrhR7QB6PXCj1wo8AAAQCAAAAAQAEAAkAAAAFbmFtZXMAAAAQAAAABQAEAAkAAAAK + ZnJvbV9jbGFzcwAEAAkAAAAIdG9fY2xhc3MABAAJAAAAB25fY2VsbHMABAAJAAAABGFyZWEA + BAAJAAAAA3BjdAAABAIAAAABAAQACQAAAAlyb3cubmFtZXMAAAANAAAAAoAAAAD////tAAAE + AgAAAAEABAAJAAAABWNsYXNzAAAAEAAAAAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAADEwAA + AAIAAAANAAAAEwAAA+kAAAPzAAAH0QAAB9IAAAfVAAAH2QAAB9sAAA+rAAATkwAAG1oAABtf + AAAbYQAAIyoAACMxAAAq+QAAKvoAACr9AAArAQAAKwMAAAAQAAAAEwAEAAkAAAAOV2F0ZXIg + LT4gV2F0ZXIABAAJAAAAEldhdGVyIC0+IFJhbmdlbGFuZAAEAAkAAAAOVHJlZXMgLT4gV2F0 + ZXIABAAJAAAADlRyZWVzIC0+IFRyZWVzAAQACQAAAA5UcmVlcyAtPiBDcm9wcwAEAAkAAAAR + VHJlZXMgLT4gU25vdy9JY2UABAAJAAAAElRyZWVzIC0+IFJhbmdlbGFuZAAEAAkAAAAfRmxv + b2RlZCBWZWdldGF0aW9uIC0+IFJhbmdlbGFuZAAEAAkAAAASQ3JvcHMgLT4gUmFuZ2VsYW5k + AAQACQAAABNCdWlsdCBBcmVhIC0+IFRyZWVzAAQACQAAABhCdWlsdCBBcmVhIC0+IEJ1aWx0 + IEFyZWEABAAJAAAAFkJ1aWx0IEFyZWEgLT4gU25vdy9JY2UABAAJAAAAEVNub3cvSWNlIC0+ + IFRyZWVzAAQACQAAABRTbm93L0ljZSAtPiBTbm93L0ljZQAEAAkAAAASUmFuZ2VsYW5kIC0+ + IFdhdGVyAAQACQAAABJSYW5nZWxhbmQgLT4gVHJlZXMABAAJAAAAElJhbmdlbGFuZCAtPiBD + cm9wcwAEAAkAAAAVUmFuZ2VsYW5kIC0+IFNub3cvSWNlAAQACQAAABZSYW5nZWxhbmQgLT4g + UmFuZ2VsYW5kAAAEAgAAAf8AAAAQAAAAAgAEAAkAAAACaWQABAAJAAAACnRyYW5zaXRpb24A + AAQCAAAC/wAAAA0AAAACgAAAAP///+0AAAQCAAAD/wAAABAAAAABAAQACQAAAApkYXRhLmZy + YW1lAAAA/gAAAxMAAAACAAAAEAAAABMABAAJAAAAGEJ1aWx0IEFyZWEgLT4gQnVpbHQgQXJl + YQAEAAkAAAAWQnVpbHQgQXJlYSAtPiBTbm93L0ljZQAEAAkAAAATQnVpbHQgQXJlYSAtPiBU + cmVlcwAEAAkAAAASQ3JvcHMgLT4gUmFuZ2VsYW5kAAQACQAAAB9GbG9vZGVkIFZlZ2V0YXRp + b24gLT4gUmFuZ2VsYW5kAAQACQAAABJSYW5nZWxhbmQgLT4gQ3JvcHMABAAJAAAAFlJhbmdl + bGFuZCAtPiBSYW5nZWxhbmQABAAJAAAAFVJhbmdlbGFuZCAtPiBTbm93L0ljZQAEAAkAAAAS + UmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAAABJSYW5nZWxhbmQgLT4gV2F0ZXIABAAJAAAAFFNu + b3cvSWNlIC0+IFNub3cvSWNlAAQACQAAABFTbm93L0ljZSAtPiBUcmVlcwAEAAkAAAAOVHJl + ZXMgLT4gQ3JvcHMABAAJAAAAElRyZWVzIC0+IFJhbmdlbGFuZAAEAAkAAAARVHJlZXMgLT4g + U25vdy9JY2UABAAJAAAADlRyZWVzIC0+IFRyZWVzAAQACQAAAA5UcmVlcyAtPiBXYXRlcgAE + AAkAAAASV2F0ZXIgLT4gUmFuZ2VsYW5kAAQACQAAAA5XYXRlciAtPiBXYXRlcgAAAA4AAAAT + QCQAAAAAAABARgAAAAAAAD/wAAAAAAAAQI8wAAAAAABAAAAAAAAAAEBiQAAAAAAAQKa4AAAA + AABAMwAAAAAAAEBPAAAAAAAAQEmAAAAAAAA/8AAAAAAAAD/wAAAAAAAAP/AAAAAAAABAllQA + AAAAAEBeQAAAAAAAQLVmAAAAAABAWIAAAAAAAD/wAAAAAAAAQI1gAAAAAAAAAAQCAAAB/wAA + ABAAAAACAAQACQAAAAV2YWx1ZQAEAAkAAAAFY291bnQAAAQCAAAC/wAAAA0AAAACgAAAAP// + /+0AAAQCAAAD/wAAABAAAAABAAQACQAAAApkYXRhLmZyYW1lAAAA/gAAABAAAAABAAQACQAA + AAROVUxMAAAEAgAAAf8AAAAQAAAABAAEAAkAAAAHc3VtbWFyeQAEAAkAAAALcmFzdGVyX2Nh + dHMABAAJAAAAC3Jhc3Rlcl9mcmVxAAQACQAAAAdyZW1vdmVkAAAA/gAAAhMAAAAEAAADEwAA + AAUAAAAQAAAABQAEAAkAAAAFVHJlZXMABAAJAAAABVRyZWVzAAQACQAAAAVUcmVlcwAEAAkA + AAAFVHJlZXMABAAJAAAABVRyZWVzAAAAEAAAAAUABAAJAAAABUNyb3BzAAQACQAAAAlSYW5n + ZWxhbmQABAAJAAAACFNub3cvSWNlAAQACQAAAAVUcmVlcwAEAAkAAAAFV2F0ZXIAAAANAAAA + BQAAAAEAAAWVAAAAeQAAFWYAAABiAAAADgAAAAU/hHrhR64Ue0AslHrhR64VP/NcKPXCj1xA + S2PXCj1wpD/vXCj1wo9cAAAADgAAAAU/hHrhR64Ue0A0DMzMzMzNP/szMzMzMzNAUzcKPXCj + 1z/2FHrhR64UAAAEAgAAAf8AAAAQAAAABQAEAAkAAAAKZnJvbV9jbGFzcwAEAAkAAAAIdG9f + Y2xhc3MABAAJAAAAB25fY2VsbHMABAAJAAAABGFyZWEABAAJAAAAA3BjdAAABAIAAAL/AAAA + DQAAAAKAAAAA////+wAABAIAAAP/AAAAEAAAAAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAAD + EwAAAAIAAAANAAAABQAAB9EAAAfSAAAH1QAAB9kAAAfbAAAAEAAAAAUABAAJAAAADlRyZWVz + IC0+IFdhdGVyAAQACQAAAA5UcmVlcyAtPiBUcmVlcwAEAAkAAAAOVHJlZXMgLT4gQ3JvcHMA + BAAJAAAAEVRyZWVzIC0+IFNub3cvSWNlAAQACQAAABJUcmVlcyAtPiBSYW5nZWxhbmQAAAQC + AAAB/wAAABAAAAACAAQACQAAAAJpZAAEAAkAAAAKdHJhbnNpdGlvbgAABAIAAAL/AAAADQAA + AAKAAAAA////+wAABAIAAAP/AAAAEAAAAAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAADEwAA + AAIAAAAQAAAABQAEAAkAAAAOVHJlZXMgLT4gQ3JvcHMABAAJAAAAElRyZWVzIC0+IFJhbmdl + bGFuZAAEAAkAAAARVHJlZXMgLT4gU25vdy9JY2UABAAJAAAADlRyZWVzIC0+IFRyZWVzAAQA + CQAAAA5UcmVlcyAtPiBXYXRlcgAAAA4AAAAFP/AAAAAAAABAllQAAAAAAEBeQAAAAAAAQLVm + AAAAAABAWIAAAAAAAAAABAIAAAH/AAAAEAAAAAIABAAJAAAABXZhbHVlAAQACQAAAAVjb3Vu + dAAABAIAAAL/AAAADQAAAAKAAAAA////+wAABAIAAAP/AAAAEAAAAAEABAAJAAAACmRhdGEu + ZnJhbWUAAAD+AAAAEAAAAAEABAAJAAAABE5VTEwAAAQCAAAB/wAAABAAAAAEAAQACQAAAAdz + dW1tYXJ5AAQACQAAAAtyYXN0ZXJfY2F0cwAEAAkAAAALcmFzdGVyX2ZyZXEABAAJAAAAB3Jl + bW92ZWQAAAD+AAACEwAAAAQAAAMTAAAABQAAABAAAAACAAQACQAAAAVUcmVlcwAEAAkAAAAF + VHJlZXMAAAAQAAAAAgAEAAkAAAAFQ3JvcHMABAAJAAAACVJhbmdlbGFuZAAAAA0AAAACAAAA + AQAABZUAAAAOAAAAAj+EeuFHrhR7QCyUeuFHrhUAAAAOAAAAAj+x64UeuFHsQFj7hR64UewA + AAQCAAAB/wAAABAAAAAFAAQACQAAAApmcm9tX2NsYXNzAAQACQAAAAh0b19jbGFzcwAEAAkA + AAAHbl9jZWxscwAEAAkAAAAEYXJlYQAEAAkAAAADcGN0AAAEAgAAAv8AAAANAAAAAoAAAAD/ + ///+AAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMTAAAAAgAAAA0A + AAACAAAH1QAAB9sAAAAQAAAAAgAEAAkAAAAOVHJlZXMgLT4gQ3JvcHMABAAJAAAAElRyZWVz + IC0+IFJhbmdlbGFuZAAABAIAAAH/AAAAEAAAAAIABAAJAAAAAmlkAAQACQAAAAp0cmFuc2l0 + aW9uAAAEAgAAAv8AAAANAAAAAoAAAAD////+AAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0 + YS5mcmFtZQAAAP4AAAMTAAAAAgAAABAAAAACAAQACQAAAA5UcmVlcyAtPiBDcm9wcwAEAAkA + AAASVHJlZXMgLT4gUmFuZ2VsYW5kAAAADgAAAAI/8AAAAAAAAECWVAAAAAAAAAAEAgAAAf8A + AAAQAAAAAgAEAAkAAAAFdmFsdWUABAAJAAAABWNvdW50AAAEAgAAAv8AAAANAAAAAoAAAAD/ + ///+AAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAAQAAAAAQAEAAkA + AAAETlVMTAAABAIAAAH/AAAAEAAAAAQABAAJAAAAB3N1bW1hcnkABAAJAAAAC3Jhc3Rlcl9j + YXRzAAQACQAAAAtyYXN0ZXJfZnJlcQAEAAkAAAAHcmVtb3ZlZAAAAP4AAAITAAAABAAAAxMA + AAAFAAAAEAAAABMABAAJAAAACkJ1aWx0IEFyZWEABAAJAAAACkJ1aWx0IEFyZWEABAAJAAAA + CkJ1aWx0IEFyZWEABAAJAAAABUNyb3BzAAQACQAAABJGbG9vZGVkIFZlZ2V0YXRpb24ABAAJ + AAAACVJhbmdlbGFuZAAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAlSYW5nZWxhbmQABAAJAAAA + CVJhbmdlbGFuZAAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAhTbm93L0ljZQAEAAkAAAAIU25v + dy9JY2UABAAJAAAABVRyZWVzAAQACQAAAAVUcmVlcwAEAAkAAAAFVHJlZXMABAAJAAAABVRy + ZWVzAAQACQAAAAVUcmVlcwAEAAkAAAAFV2F0ZXIABAAJAAAABVdhdGVyAAAAEAAAABMABAAJ + AAAACkJ1aWx0IEFyZWEABAAJAAAACFNub3cvSWNlAAQACQAAAAVUcmVlcwAEAAkAAAAJUmFu + Z2VsYW5kAAQACQAAAAlSYW5nZWxhbmQABAAJAAAABUNyb3BzAAQACQAAAAlSYW5nZWxhbmQA + BAAJAAAACFNub3cvSWNlAAQACQAAAAVUcmVlcwAEAAkAAAAFV2F0ZXIABAAJAAAACFNub3cv + SWNlAAQACQAAAAVUcmVlcwAEAAkAAAAFQ3JvcHMABAAJAAAACVJhbmdlbGFuZAAEAAkAAAAI + U25vdy9JY2UABAAJAAAABVRyZWVzAAQACQAAAAVXYXRlcgAEAAkAAAAJUmFuZ2VsYW5kAAQA + CQAAAAVXYXRlcgAAAA0AAAATAAAACgAAACwAAAABAAAD5gAAAAIAAACSAAALXAAAABMAAAA+ + AAAAMwAAAAEAAAABAAAAAQAABZUAAAB5AAAVZgAAAGIAAAABAAADrAAAAA4AAAATP7mZmZmZ + mZo/3Cj1wo9cKT+EeuFHrhR7QCP1wo9cKPY/lHrhR64Uez/3XCj1wo9cQD0UeuFHrhU/yFHr + hR64Uj/j1wo9cKPXP+BR64UeuFI/hHrhR64Uez+EeuFHrhR7P4R64UeuFHtALJR64UeuFT/z + XCj1wo9cQEtj1wo9cKQ/71wo9cKPXD+EeuFHrhR7QCLMzMzMzM0AAAAOAAAAEz+0euFHrhR7 + P9cKPXCj1wo/hHrhR64Ue0AgOFHrhR64P5R64UeuFHs/8wo9cKPXCkA3nrhR64UfP8MzMzMz + MzM/4AAAAAAAAD/aPXCj1wo9P4R64UeuFHs/hHrhR64Uez+EeuFHrhR7QCc4UeuFHrg/71wo + 9cKPXEBGQAAAAAAAP+mZmZmZmZo/hHrhR64Ue0Aej1wo9cKPAAAEAgAAAf8AAAAQAAAABQAE + AAkAAAAKZnJvbV9jbGFzcwAEAAkAAAAIdG9fY2xhc3MABAAJAAAAB25fY2VsbHMABAAJAAAA + BGFyZWEABAAJAAAAA3BjdAAABAIAAAL/AAAADQAAAAKAAAAA////7QAABAIAAAP/AAAAEAAA + AAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAADEwAAAAIAAAANAAAAEwAAA+kAAAPzAAAH0QAA + B9IAAAfVAAAH2QAAB9sAAA+rAAATkwAAG1oAABtfAAAbYQAAIyoAACMxAAAq+QAAKvoAACr9 + AAArAQAAKwMAAAAQAAAAEwAEAAkAAAAOV2F0ZXIgLT4gV2F0ZXIABAAJAAAAEldhdGVyIC0+ + IFJhbmdlbGFuZAAEAAkAAAAOVHJlZXMgLT4gV2F0ZXIABAAJAAAADlRyZWVzIC0+IFRyZWVz + AAQACQAAAA5UcmVlcyAtPiBDcm9wcwAEAAkAAAARVHJlZXMgLT4gU25vdy9JY2UABAAJAAAA + ElRyZWVzIC0+IFJhbmdlbGFuZAAEAAkAAAAfRmxvb2RlZCBWZWdldGF0aW9uIC0+IFJhbmdl + bGFuZAAEAAkAAAASQ3JvcHMgLT4gUmFuZ2VsYW5kAAQACQAAABNCdWlsdCBBcmVhIC0+IFRy + ZWVzAAQACQAAABhCdWlsdCBBcmVhIC0+IEJ1aWx0IEFyZWEABAAJAAAAFkJ1aWx0IEFyZWEg + LT4gU25vdy9JY2UABAAJAAAAEVNub3cvSWNlIC0+IFRyZWVzAAQACQAAABRTbm93L0ljZSAt + PiBTbm93L0ljZQAEAAkAAAASUmFuZ2VsYW5kIC0+IFdhdGVyAAQACQAAABJSYW5nZWxhbmQg + LT4gVHJlZXMABAAJAAAAElJhbmdlbGFuZCAtPiBDcm9wcwAEAAkAAAAVUmFuZ2VsYW5kIC0+ + IFNub3cvSWNlAAQACQAAABZSYW5nZWxhbmQgLT4gUmFuZ2VsYW5kAAAEAgAAAf8AAAAQAAAA + AgAEAAkAAAACaWQABAAJAAAACnRyYW5zaXRpb24AAAQCAAAC/wAAAA0AAAACgAAAAP///+0A + AAQCAAAD/wAAABAAAAABAAQACQAAAApkYXRhLmZyYW1lAAAA/gAAAxMAAAACAAAAEAAAABMA + BAAJAAAAGEJ1aWx0IEFyZWEgLT4gQnVpbHQgQXJlYQAEAAkAAAAWQnVpbHQgQXJlYSAtPiBT + bm93L0ljZQAEAAkAAAATQnVpbHQgQXJlYSAtPiBUcmVlcwAEAAkAAAASQ3JvcHMgLT4gUmFu + Z2VsYW5kAAQACQAAAB9GbG9vZGVkIFZlZ2V0YXRpb24gLT4gUmFuZ2VsYW5kAAQACQAAABJS + YW5nZWxhbmQgLT4gQ3JvcHMABAAJAAAAFlJhbmdlbGFuZCAtPiBSYW5nZWxhbmQABAAJAAAA + FVJhbmdlbGFuZCAtPiBTbm93L0ljZQAEAAkAAAASUmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAA + ABJSYW5nZWxhbmQgLT4gV2F0ZXIABAAJAAAAFFNub3cvSWNlIC0+IFNub3cvSWNlAAQACQAA + ABFTbm93L0ljZSAtPiBUcmVlcwAEAAkAAAAOVHJlZXMgLT4gQ3JvcHMABAAJAAAAElRyZWVz + IC0+IFJhbmdlbGFuZAAEAAkAAAARVHJlZXMgLT4gU25vdy9JY2UABAAJAAAADlRyZWVzIC0+ + IFRyZWVzAAQACQAAAA5UcmVlcyAtPiBXYXRlcgAEAAkAAAASV2F0ZXIgLT4gUmFuZ2VsYW5k + AAQACQAAAA5XYXRlciAtPiBXYXRlcgAAAA4AAAATQCQAAAAAAABARgAAAAAAAD/wAAAAAAAA + QI8wAAAAAABAAAAAAAAAAEBiQAAAAAAAQKa4AAAAAABAMwAAAAAAAEBPAAAAAAAAQEmAAAAA + AAA/8AAAAAAAAD/wAAAAAAAAP/AAAAAAAABAllQAAAAAAEBeQAAAAAAAQLVmAAAAAABAWIAA + AAAAAD/wAAAAAAAAQI1gAAAAAAAAAAQCAAAB/wAAABAAAAACAAQACQAAAAV2YWx1ZQAEAAkA + AAAFY291bnQAAAQCAAAC/wAAAA0AAAACgAAAAP///+0AAAQCAAAD/wAAABAAAAABAAQACQAA + AApkYXRhLmZyYW1lAAAA/gAAABAAAAABAAQACQAAAAROVUxMAAAEAgAAAf8AAAAQAAAABAAE + AAkAAAAHc3VtbWFyeQAEAAkAAAALcmFzdGVyX2NhdHMABAAJAAAAC3Jhc3Rlcl9mcmVxAAQA + CQAAAAdyZW1vdmVkAAAA/gAAAhMAAAAEAAADEwAAAAUAAAAQAAAAEgAEAAkAAAAKQnVpbHQg + QXJlYQAEAAkAAAAKQnVpbHQgQXJlYQAEAAkAAAAKQnVpbHQgQXJlYQAEAAkAAAAFQ3JvcHMA + BAAJAAAACVJhbmdlbGFuZAAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAlSYW5nZWxhbmQABAAJ + AAAACVJhbmdlbGFuZAAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAhTbm93L0ljZQAEAAkAAAAI + U25vdy9JY2UABAAJAAAABVRyZWVzAAQACQAAAAVUcmVlcwAEAAkAAAAFVHJlZXMABAAJAAAA + BVRyZWVzAAQACQAAAAVUcmVlcwAEAAkAAAAFV2F0ZXIABAAJAAAABVdhdGVyAAAAEAAAABIA + BAAJAAAACkJ1aWx0IEFyZWEABAAJAAAACFNub3cvSWNlAAQACQAAAAVUcmVlcwAEAAkAAAAJ + UmFuZ2VsYW5kAAQACQAAAAVDcm9wcwAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAhTbm93L0lj + ZQAEAAkAAAAFVHJlZXMABAAJAAAABVdhdGVyAAQACQAAAAhTbm93L0ljZQAEAAkAAAAFVHJl + ZXMABAAJAAAABUNyb3BzAAQACQAAAAlSYW5nZWxhbmQABAAJAAAACFNub3cvSWNlAAQACQAA + AAVUcmVlcwAEAAkAAAAFV2F0ZXIABAAJAAAACVJhbmdlbGFuZAAEAAkAAAAFV2F0ZXIAAAAN + AAAAEgAAAAoAAAAsAAAAAQAAA+YAAACPAAALXAAAABMAAAAvAAAAKgAAAAEAAAABAAAAAQAA + BXUAAAB4AAAVZgAAAFsAAAABAAADrAAAAA4AAAASP7mZmZmZmZo/3Cj1wo9cKT+EeuFHrhR7 + QCP1wo9cKPY/9uFHrhR64UA9FHrhR64VP8hR64UeuFI/3hR64UeuFT/a4UeuFHrhP4R64Ueu + FHs/hHrhR64Uez+EeuFHrhR7QCvwo9cKPXE/8zMzMzMzM0BLY9cKPXCkP+0euFHrhR8/hHrh + R64Ue0AizMzMzMzNAAAADgAAABI/tHrhR64Uez/XCj1wo9cKP4R64UeuFHtAIEzMzMzMzT/y + uFHrhR64QDfAAAAAAAA/xHrhR64Uez/YUeuFHrhSP9XCj1wo9cM/hHrhR64Uez+EeuFHrhR7 + P4R64UeuFHtAJtHrhR64Uj/vXCj1wo9cQEZgAAAAAAA/564UeuFHrj+EeuFHrhR7QB64UeuF + HrgAAAQCAAAB/wAAABAAAAAFAAQACQAAAApmcm9tX2NsYXNzAAQACQAAAAh0b19jbGFzcwAE + AAkAAAAHbl9jZWxscwAEAAkAAAAEYXJlYQAEAAkAAAADcGN0AAAEAgAAAv8AAAANAAAAAoAA + AAD////uAAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMTAAAAAgAA + AA0AAAASAAAD6QAAA/MAAAfRAAAH0gAAB9UAAAfZAAAH2wAAE5MAABtaAAAbXwAAG2EAACMq + AAAjMQAAKvkAACr6AAAq/QAAKwEAACsDAAAAEAAAABIABAAJAAAADldhdGVyIC0+IFdhdGVy + AAQACQAAABJXYXRlciAtPiBSYW5nZWxhbmQABAAJAAAADlRyZWVzIC0+IFdhdGVyAAQACQAA + AA5UcmVlcyAtPiBUcmVlcwAEAAkAAAAOVHJlZXMgLT4gQ3JvcHMABAAJAAAAEVRyZWVzIC0+ + IFNub3cvSWNlAAQACQAAABJUcmVlcyAtPiBSYW5nZWxhbmQABAAJAAAAEkNyb3BzIC0+IFJh + bmdlbGFuZAAEAAkAAAATQnVpbHQgQXJlYSAtPiBUcmVlcwAEAAkAAAAYQnVpbHQgQXJlYSAt + PiBCdWlsdCBBcmVhAAQACQAAABZCdWlsdCBBcmVhIC0+IFNub3cvSWNlAAQACQAAABFTbm93 + L0ljZSAtPiBUcmVlcwAEAAkAAAAUU25vdy9JY2UgLT4gU25vdy9JY2UABAAJAAAAElJhbmdl + bGFuZCAtPiBXYXRlcgAEAAkAAAASUmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAAABJSYW5nZWxh + bmQgLT4gQ3JvcHMABAAJAAAAFVJhbmdlbGFuZCAtPiBTbm93L0ljZQAEAAkAAAAWUmFuZ2Vs + YW5kIC0+IFJhbmdlbGFuZAAABAIAAAH/AAAAEAAAAAIABAAJAAAAAmlkAAQACQAAAAp0cmFu + c2l0aW9uAAAEAgAAAv8AAAANAAAAAoAAAAD////uAAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAK + ZGF0YS5mcmFtZQAAAP4AAAMTAAAAAgAAABAAAAASAAQACQAAABhCdWlsdCBBcmVhIC0+IEJ1 + aWx0IEFyZWEABAAJAAAAFkJ1aWx0IEFyZWEgLT4gU25vdy9JY2UABAAJAAAAE0J1aWx0IEFy + ZWEgLT4gVHJlZXMABAAJAAAAEkNyb3BzIC0+IFJhbmdlbGFuZAAEAAkAAAASUmFuZ2VsYW5k + IC0+IENyb3BzAAQACQAAABZSYW5nZWxhbmQgLT4gUmFuZ2VsYW5kAAQACQAAABVSYW5nZWxh + bmQgLT4gU25vdy9JY2UABAAJAAAAElJhbmdlbGFuZCAtPiBUcmVlcwAEAAkAAAASUmFuZ2Vs + YW5kIC0+IFdhdGVyAAQACQAAABRTbm93L0ljZSAtPiBTbm93L0ljZQAEAAkAAAARU25vdy9J + Y2UgLT4gVHJlZXMABAAJAAAADlRyZWVzIC0+IENyb3BzAAQACQAAABJUcmVlcyAtPiBSYW5n + ZWxhbmQABAAJAAAAEVRyZWVzIC0+IFNub3cvSWNlAAQACQAAAA5UcmVlcyAtPiBUcmVlcwAE + AAkAAAAOVHJlZXMgLT4gV2F0ZXIABAAJAAAAEldhdGVyIC0+IFJhbmdlbGFuZAAEAAkAAAAO + V2F0ZXIgLT4gV2F0ZXIAAAAOAAAAEkAkAAAAAAAAQEYAAAAAAAA/8AAAAAAAAECPMAAAAAAA + QGHgAAAAAABAprgAAAAAAEAzAAAAAAAAQEeAAAAAAABARQAAAAAAAD/wAAAAAAAAP/AAAAAA + AAA/8AAAAAAAAECV1AAAAAAAQF4AAAAAAABAtWYAAAAAAEBWwAAAAAAAP/AAAAAAAABAjWAA + AAAAAAAABAIAAAH/AAAAEAAAAAIABAAJAAAABXZhbHVlAAQACQAAAAVjb3VudAAABAIAAAL/ + AAAADQAAAAKAAAAA////7gAABAIAAAP/AAAAEAAAAAEABAAJAAAACmRhdGEuZnJhbWUAAAD+ + AAACEwAAAAIAAAMTAAAAAgAAAA0AAAAHAAAH0QAAB9kAAAfbAAAPqwAAKvkAACr6AAAq/QAA + ABAAAAAHAAQACQAAAA5UcmVlcyAtPiBXYXRlcgAEAAkAAAARVHJlZXMgLT4gU25vdy9JY2UA + BAAJAAAAElRyZWVzIC0+IFJhbmdlbGFuZAAEAAkAAAAfRmxvb2RlZCBWZWdldGF0aW9uIC0+ + IFJhbmdlbGFuZAAEAAkAAAASUmFuZ2VsYW5kIC0+IFdhdGVyAAQACQAAABJSYW5nZWxhbmQg + LT4gVHJlZXMABAAJAAAAElJhbmdlbGFuZCAtPiBDcm9wcwAABAIAAAH/AAAAEAAAAAIABAAJ + AAAAAmlkAAQACQAAAAp0cmFuc2l0aW9uAAAEAgAAAv8AAAANAAAAAoAAAAD////5AAAEAgAA + A/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMTAAAAAgAAABAAAAAHAAQACQAA + AB9GbG9vZGVkIFZlZ2V0YXRpb24gLT4gUmFuZ2VsYW5kAAQACQAAABJSYW5nZWxhbmQgLT4g + Q3JvcHMABAAJAAAAElJhbmdlbGFuZCAtPiBUcmVlcwAEAAkAAAASUmFuZ2VsYW5kIC0+IFdh + dGVyAAQACQAAABJUcmVlcyAtPiBSYW5nZWxhbmQABAAJAAAAEVRyZWVzIC0+IFNub3cvSWNl + AAQACQAAAA5UcmVlcyAtPiBXYXRlcgAAAA4AAAAHQAAAAAAAAABACAAAAAAAAEAuAAAAAAAA + QCIAAAAAAABAQAAAAAAAAD/wAAAAAAAAQBwAAAAAAAAAAAQCAAAB/wAAABAAAAACAAQACQAA + AAV2YWx1ZQAEAAkAAAAFY291bnQAAAQCAAAC/wAAAA0AAAACgAAAAP////kAAAQCAAAD/wAA + ABAAAAABAAQACQAAAApkYXRhLmZyYW1lAAAA/gAABAIAAAH/AAAAEAAAAAIABAAJAAAABGNh + dHMABAAJAAAABGZyZXEAAAD+AAAEAgAAAf8AAAAQAAAABAAEAAkAAAAHc3VtbWFyeQAEAAkA + AAALcmFzdGVyX2NhdHMABAAJAAAAC3Jhc3Rlcl9mcmVxAAQACQAAAAdyZW1vdmVkAAAA/gAA + AhMAAAAEAAADEwAAAAUAAAAQAAAAEgAEAAkAAAAKQnVpbHQgQXJlYQAEAAkAAAAKQnVpbHQg + QXJlYQAEAAkAAAAKQnVpbHQgQXJlYQAEAAkAAAAFQ3JvcHMABAAJAAAACVJhbmdlbGFuZAAE + AAkAAAAJUmFuZ2VsYW5kAAQACQAAAAlSYW5nZWxhbmQABAAJAAAACVJhbmdlbGFuZAAEAAkA + AAAJUmFuZ2VsYW5kAAQACQAAAAhTbm93L0ljZQAEAAkAAAAIU25vdy9JY2UABAAJAAAABVRy + ZWVzAAQACQAAAAVUcmVlcwAEAAkAAAAFVHJlZXMABAAJAAAABVRyZWVzAAQACQAAAAVUcmVl + cwAEAAkAAAAFV2F0ZXIABAAJAAAABVdhdGVyAAAAEAAAABIABAAJAAAACkJ1aWx0IEFyZWEA + BAAJAAAACFNub3cvSWNlAAQACQAAAAVUcmVlcwAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAVD + cm9wcwAEAAkAAAAJUmFuZ2VsYW5kAAQACQAAAAhTbm93L0ljZQAEAAkAAAAFVHJlZXMABAAJ + AAAABVdhdGVyAAQACQAAAAhTbm93L0ljZQAEAAkAAAAFVHJlZXMABAAJAAAABUNyb3BzAAQA + CQAAAAlSYW5nZWxhbmQABAAJAAAACFNub3cvSWNlAAQACQAAAAVUcmVlcwAEAAkAAAAFV2F0 + ZXIABAAJAAAACVJhbmdlbGFuZAAEAAkAAAAFV2F0ZXIAAAANAAAAEgAAAAoAAAAsAAAAAQAA + A+YAAACPAAALXAAAABEAAAAuAAAAJQAAAAEAAAABAAAAAQAABTsAAAB3AAAVZgAAAFYAAAAB + AAADrAAAAA4AAAASP7mZmZmZmZo/3Cj1wo9cKT+EeuFHrhR7QCP1wo9cKPY/9uFHrhR64UA9 + FHrhR64VP8XCj1wo9cM/3XCj1wo9cT/XrhR64UeuP4R64UeuFHs/hHrhR64Uez+EeuFHrhR7 + QCrHrhR64Ug/8wo9cKPXCkBLY9cKPXCkP+uFHrhR64U/hHrhR64Ue0AizMzMzMzNAAAADgAA + ABI/tHrhR64Uez/XCj1wo9cKP4R64UeuFHtAIGZmZmZmZj/y4UeuFHrhQDfj1wo9cKQ/weuF + HrhR7D/YUeuFHrhSP9MzMzMzMzM/hHrhR64Uez+EeuFHrhR7P4R64UeuFHtAJgAAAAAAAD/v + XCj1wo9cQEaBR64UeuE/5rhR64UeuD+EeuFHrhR7QB7hR64UeuEAAAQCAAAB/wAAABAAAAAF + AAQACQAAAApmcm9tX2NsYXNzAAQACQAAAAh0b19jbGFzcwAEAAkAAAAHbl9jZWxscwAEAAkA + AAAEYXJlYQAEAAkAAAADcGN0AAAEAgAAAv8AAAANAAAAAoAAAAD////uAAAEAgAAA/8AAAAQ + AAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMTAAAAAgAAAA0AAAASAAAD6QAAA/MAAAfR + AAAH0gAAB9UAAAfZAAAH2wAAE5MAABtaAAAbXwAAG2EAACMqAAAjMQAAKvkAACr6AAAq/QAA + KwEAACsDAAAAEAAAABIABAAJAAAADldhdGVyIC0+IFdhdGVyAAQACQAAABJXYXRlciAtPiBS + YW5nZWxhbmQABAAJAAAADlRyZWVzIC0+IFdhdGVyAAQACQAAAA5UcmVlcyAtPiBUcmVlcwAE + AAkAAAAOVHJlZXMgLT4gQ3JvcHMABAAJAAAAEVRyZWVzIC0+IFNub3cvSWNlAAQACQAAABJU + cmVlcyAtPiBSYW5nZWxhbmQABAAJAAAAEkNyb3BzIC0+IFJhbmdlbGFuZAAEAAkAAAATQnVp + bHQgQXJlYSAtPiBUcmVlcwAEAAkAAAAYQnVpbHQgQXJlYSAtPiBCdWlsdCBBcmVhAAQACQAA + ABZCdWlsdCBBcmVhIC0+IFNub3cvSWNlAAQACQAAABFTbm93L0ljZSAtPiBUcmVlcwAEAAkA + AAAUU25vdy9JY2UgLT4gU25vdy9JY2UABAAJAAAAElJhbmdlbGFuZCAtPiBXYXRlcgAEAAkA + AAASUmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAAABJSYW5nZWxhbmQgLT4gQ3JvcHMABAAJAAAA + FVJhbmdlbGFuZCAtPiBTbm93L0ljZQAEAAkAAAAWUmFuZ2VsYW5kIC0+IFJhbmdlbGFuZAAA + BAIAAAH/AAAAEAAAAAIABAAJAAAAAmlkAAQACQAAAAp0cmFuc2l0aW9uAAAEAgAAAv8AAAAN + AAAAAoAAAAD////uAAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMT + AAAAAgAAABAAAAASAAQACQAAABhCdWlsdCBBcmVhIC0+IEJ1aWx0IEFyZWEABAAJAAAAFkJ1 + aWx0IEFyZWEgLT4gU25vdy9JY2UABAAJAAAAE0J1aWx0IEFyZWEgLT4gVHJlZXMABAAJAAAA + EkNyb3BzIC0+IFJhbmdlbGFuZAAEAAkAAAASUmFuZ2VsYW5kIC0+IENyb3BzAAQACQAAABZS + YW5nZWxhbmQgLT4gUmFuZ2VsYW5kAAQACQAAABVSYW5nZWxhbmQgLT4gU25vdy9JY2UABAAJ + AAAAElJhbmdlbGFuZCAtPiBUcmVlcwAEAAkAAAASUmFuZ2VsYW5kIC0+IFdhdGVyAAQACQAA + ABRTbm93L0ljZSAtPiBTbm93L0ljZQAEAAkAAAARU25vdy9JY2UgLT4gVHJlZXMABAAJAAAA + DlRyZWVzIC0+IENyb3BzAAQACQAAABJUcmVlcyAtPiBSYW5nZWxhbmQABAAJAAAAEVRyZWVz + IC0+IFNub3cvSWNlAAQACQAAAA5UcmVlcyAtPiBUcmVlcwAEAAkAAAAOVHJlZXMgLT4gV2F0 + ZXIABAAJAAAAEldhdGVyIC0+IFJhbmdlbGFuZAAEAAkAAAAOV2F0ZXIgLT4gV2F0ZXIAAAAO + AAAAEkAkAAAAAAAAQEYAAAAAAAA/8AAAAAAAAECPMAAAAAAAQGHgAAAAAABAprgAAAAAAEAx + AAAAAAAAQEcAAAAAAABAQoAAAAAAAD/wAAAAAAAAP/AAAAAAAAA/8AAAAAAAAECU7AAAAAAA + QF3AAAAAAABAtWYAAAAAAEBVgAAAAAAAP/AAAAAAAABAjWAAAAAAAAAABAIAAAH/AAAAEAAA + AAIABAAJAAAABXZhbHVlAAQACQAAAAVjb3VudAAABAIAAAL/AAAADQAAAAKAAAAA////7gAA + BAIAAAP/AAAAEAAAAAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAACEwAAAAIAAAMTAAAAAgAA + AA0AAAAIAAAH0QAAB9kAAAfbAAAPqwAAKvkAACr6AAAq/QAAKwEAAAAQAAAACAAEAAkAAAAO + VHJlZXMgLT4gV2F0ZXIABAAJAAAAEVRyZWVzIC0+IFNub3cvSWNlAAQACQAAABJUcmVlcyAt + PiBSYW5nZWxhbmQABAAJAAAAH0Zsb29kZWQgVmVnZXRhdGlvbiAtPiBSYW5nZWxhbmQABAAJ + AAAAElJhbmdlbGFuZCAtPiBXYXRlcgAEAAkAAAASUmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAA + ABJSYW5nZWxhbmQgLT4gQ3JvcHMABAAJAAAAFVJhbmdlbGFuZCAtPiBTbm93L0ljZQAABAIA + AAH/AAAAEAAAAAIABAAJAAAAAmlkAAQACQAAAAp0cmFuc2l0aW9uAAAEAgAAAv8AAAANAAAA + AoAAAAD////4AAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMTAAAA + AgAAABAAAAAIAAQACQAAAB9GbG9vZGVkIFZlZ2V0YXRpb24gLT4gUmFuZ2VsYW5kAAQACQAA + ABJSYW5nZWxhbmQgLT4gQ3JvcHMABAAJAAAAFVJhbmdlbGFuZCAtPiBTbm93L0ljZQAEAAkA + AAASUmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAAABJSYW5nZWxhbmQgLT4gV2F0ZXIABAAJAAAA + ElRyZWVzIC0+IFJhbmdlbGFuZAAEAAkAAAARVHJlZXMgLT4gU25vdy9JY2UABAAJAAAADlRy + ZWVzIC0+IFdhdGVyAAAADgAAAAhAAAAAAAAAAEAIAAAAAAAAQAAAAAAAAABAMAAAAAAAAEAs + AAAAAAAAQFaAAAAAAABAAAAAAAAAAEAoAAAAAAAAAAAEAgAAAf8AAAAQAAAAAgAEAAkAAAAF + dmFsdWUABAAJAAAABWNvdW50AAAEAgAAAv8AAAANAAAAAoAAAAD////4AAAEAgAAA/8AAAAQ + AAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAQCAAAB/wAAABAAAAACAAQACQAAAARjYXRz + AAQACQAAAARmcmVxAAAA/gAABAIAAAH/AAAAEAAAAAQABAAJAAAAB3N1bW1hcnkABAAJAAAA + C3Jhc3Rlcl9jYXRzAAQACQAAAAtyYXN0ZXJfZnJlcQAEAAkAAAAHcmVtb3ZlZAAAAP4AAAIT + AAAABAAAAxMAAAAFAAAAEAAAAAUABAAJAAAACkJ1aWx0IEFyZWEABAAJAAAACVJhbmdlbGFu + ZAAEAAkAAAAIU25vdy9JY2UABAAJAAAABVRyZWVzAAQACQAAAAVXYXRlcgAAABAAAAAFAAQA + CQAAAApCdWlsdCBBcmVhAAQACQAAAAlSYW5nZWxhbmQABAAJAAAACFNub3cvSWNlAAQACQAA + AAVUcmVlcwAEAAkAAAAFV2F0ZXIAAAANAAAABQAAAAoAAAtcAAAAAQAAFWYAAAOsAAAADgAA + AAU/uZmZmZmZmkA9FHrhR64VP4R64UeuFHtAS2PXCj1wpEAizMzMzMzNAAAADgAAAAU/vCj1 + wo9cKUA/I9cKPXCkP4R64UeuFHtATVXCj1wo9kAkI9cKPXCkAAAEAgAAAf8AAAAQAAAABQAE + AAkAAAAKZnJvbV9jbGFzcwAEAAkAAAAIdG9fY2xhc3MABAAJAAAAB25fY2VsbHMABAAJAAAA + BGFyZWEABAAJAAAAA3BjdAAABAIAAAL/AAAADQAAAAKAAAAA////+wAABAIAAAP/AAAAEAAA + AAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAADEwAAAAIAAAANAAAABQAAA+kAAAfSAAAbXwAA + IzEAACsDAAAAEAAAAAUABAAJAAAADldhdGVyIC0+IFdhdGVyAAQACQAAAA5UcmVlcyAtPiBU + cmVlcwAEAAkAAAAYQnVpbHQgQXJlYSAtPiBCdWlsdCBBcmVhAAQACQAAABRTbm93L0ljZSAt + PiBTbm93L0ljZQAEAAkAAAAWUmFuZ2VsYW5kIC0+IFJhbmdlbGFuZAAABAIAAAH/AAAAEAAA + AAIABAAJAAAAAmlkAAQACQAAAAp0cmFuc2l0aW9uAAAEAgAAAv8AAAANAAAAAoAAAAD////7 + AAAEAgAAA/8AAAAQAAAAAQAEAAkAAAAKZGF0YS5mcmFtZQAAAP4AAAMTAAAAAgAAABAAAAAF + AAQACQAAABhCdWlsdCBBcmVhIC0+IEJ1aWx0IEFyZWEABAAJAAAAFlJhbmdlbGFuZCAtPiBS + YW5nZWxhbmQABAAJAAAAFFNub3cvSWNlIC0+IFNub3cvSWNlAAQACQAAAA5UcmVlcyAtPiBU + cmVlcwAEAAkAAAAOV2F0ZXIgLT4gV2F0ZXIAAAAOAAAABUAkAAAAAAAAQKa4AAAAAAA/8AAA + AAAAAEC1ZgAAAAAAQI1gAAAAAAAAAAQCAAAB/wAAABAAAAACAAQACQAAAAV2YWx1ZQAEAAkA + AAAFY291bnQAAAQCAAAC/wAAAA0AAAACgAAAAP////sAAAQCAAAD/wAAABAAAAABAAQACQAA + AApkYXRhLmZyYW1lAAAA/gAAAhMAAAACAAADEwAAAAIAAAANAAAADgAAA/MAAAfRAAAH1QAA + B9kAAAfbAAAPqwAAE5MAABtaAAAbYQAAIyoAACr5AAAq+gAAKv0AACsBAAAAEAAAAA4ABAAJ + AAAAEldhdGVyIC0+IFJhbmdlbGFuZAAEAAkAAAAOVHJlZXMgLT4gV2F0ZXIABAAJAAAADlRy + ZWVzIC0+IENyb3BzAAQACQAAABFUcmVlcyAtPiBTbm93L0ljZQAEAAkAAAASVHJlZXMgLT4g + UmFuZ2VsYW5kAAQACQAAAB9GbG9vZGVkIFZlZ2V0YXRpb24gLT4gUmFuZ2VsYW5kAAQACQAA + ABJDcm9wcyAtPiBSYW5nZWxhbmQABAAJAAAAE0J1aWx0IEFyZWEgLT4gVHJlZXMABAAJAAAA + FkJ1aWx0IEFyZWEgLT4gU25vdy9JY2UABAAJAAAAEVNub3cvSWNlIC0+IFRyZWVzAAQACQAA + ABJSYW5nZWxhbmQgLT4gV2F0ZXIABAAJAAAAElJhbmdlbGFuZCAtPiBUcmVlcwAEAAkAAAAS + UmFuZ2VsYW5kIC0+IENyb3BzAAQACQAAABVSYW5nZWxhbmQgLT4gU25vdy9JY2UAAAQCAAAB + /wAAABAAAAACAAQACQAAAAJpZAAEAAkAAAAKdHJhbnNpdGlvbgAABAIAAAL/AAAADQAAAAKA + AAAA////8gAABAIAAAP/AAAAEAAAAAEABAAJAAAACmRhdGEuZnJhbWUAAAD+AAADEwAAAAIA + AAAQAAAADgAEAAkAAAAWQnVpbHQgQXJlYSAtPiBTbm93L0ljZQAEAAkAAAATQnVpbHQgQXJl + YSAtPiBUcmVlcwAEAAkAAAASQ3JvcHMgLT4gUmFuZ2VsYW5kAAQACQAAAB9GbG9vZGVkIFZl + Z2V0YXRpb24gLT4gUmFuZ2VsYW5kAAQACQAAABJSYW5nZWxhbmQgLT4gQ3JvcHMABAAJAAAA + FVJhbmdlbGFuZCAtPiBTbm93L0ljZQAEAAkAAAASUmFuZ2VsYW5kIC0+IFRyZWVzAAQACQAA + ABJSYW5nZWxhbmQgLT4gV2F0ZXIABAAJAAAAEVNub3cvSWNlIC0+IFRyZWVzAAQACQAAAA5U + cmVlcyAtPiBDcm9wcwAEAAkAAAASVHJlZXMgLT4gUmFuZ2VsYW5kAAQACQAAABFUcmVlcyAt + PiBTbm93L0ljZQAEAAkAAAAOVHJlZXMgLT4gV2F0ZXIABAAJAAAAEldhdGVyIC0+IFJhbmdl + bGFuZAAAAA4AAAAOQEYAAAAAAAA/8AAAAAAAAECPMAAAAAAAQAAAAAAAAABAYkAAAAAAAEAz + AAAAAAAAQE8AAAAAAABASYAAAAAAAD/wAAAAAAAAP/AAAAAAAABAllQAAAAAAEBeQAAAAAAA + QFiAAAAAAAA/8AAAAAAAAAAABAIAAAH/AAAAEAAAAAIABAAJAAAABXZhbHVlAAQACQAAAAVj + b3VudAAABAIAAAL/AAAADQAAAAKAAAAA////8gAABAIAAAP/AAAAEAAAAAEABAAJAAAACmRh + dGEuZnJhbWUAAAD+AAAEAgAAAf8AAAAQAAAAAgAEAAkAAAAEY2F0cwAEAAkAAAAEZnJlcQAA + AP4AAAQCAAAB/wAAABAAAAAEAAQACQAAAAdzdW1tYXJ5AAQACQAAAAtyYXN0ZXJfY2F0cwAE + AAkAAAALcmFzdGVyX2ZyZXEABAAJAAAAB3JlbW92ZWQAAAD+AAACEwAAAAQAAAMTAAAABQAA + ABAAAAAAAAAAEAAAAAAAAAANAAAAAAAAAA4AAAAAAAAADgAAAAAAAAQCAAAB/wAAABAAAAAF + AAQACQAAAApmcm9tX2NsYXNzAAQACQAAAAh0b19jbGFzcwAEAAkAAAAHbl9jZWxscwAEAAkA + AAAEYXJlYQAEAAkAAAADcGN0AAAEAgAAAv8AAAANAAAAAAAABAIAAAP/AAAAEAAAAAEABAAJ + AAAACmRhdGEuZnJhbWUAAAD+AAADEwAAAAIAAAANAAAAAAAAABAAAAAAAAAEAgAAAf8AAAAQ + AAAAAgAEAAkAAAACaWQABAAJAAAACnRyYW5zaXRpb24AAAQCAAAD/wAAABAAAAABAAQACQAA + AApkYXRhLmZyYW1lAAAEAgAAAv8AAAANAAAAAAAAAP4AAAMTAAAAAgAAABAAAAAAAAAADQAA + AAAAAAQCAAAB/wAAABAAAAACAAQACQAAAAV2YWx1ZQAEAAkAAAAFY291bnQAAAQCAAAD/wAA + ABAAAAABAAQACQAAAApkYXRhLmZyYW1lAAAEAgAAAv8AAAANAAAAAAAAAP4AAAAQAAAAAQAE + AAkAAAAETlVMTAAABAIAAAH/AAAAEAAAAAQABAAJAAAAB3N1bW1hcnkABAAJAAAAC3Jhc3Rl + cl9jYXRzAAQACQAAAAtyYXN0ZXJfZnJlcQAEAAkAAAAHcmVtb3ZlZAAAAP4AAAQCAAAB/wAA + ABAAAAAIAAQACQAAAAdkZWZhdWx0AAQACQAAAApmcm9tX3RyZWVzAAQACQAAAAxib3RoX2Zp + bHRlcnMABAAJAAAABnBtaW5fMAAEAAkAAAAIcG1pbl81MDAABAAJAAAACXBtaW5fMTAwMAAE + AAkAAAAJcG1pbl9odWdlAAQACQAAAAppbXBvc3NpYmxlAAAA/g== + diff --git a/tests/testthat/test-dft_rast_transition.R b/tests/testthat/test-dft_rast_transition.R index 5bd6e1e..cbcbe6e 100644 --- a/tests/testthat/test-dft_rast_transition.R +++ b/tests/testthat/test-dft_rast_transition.R @@ -331,3 +331,69 @@ test_that("removed + raster account for all transition pixels", { expect_equal(n_kept + n_removed, n_changed_unfiltered) }) + +# Golden-output contract for the #34 terra-native rewrite. Captures the full +# behavior (summary tibble + raster factor levels/frequencies + removed raster) +# across the parameter matrix as a snapshot, so the memory rewrite can be proven +# byte-identical. Digest is canonicalized (sorted) so it pins content, not the +# internal cell/tie order. +test_that("dft_rast_transition output is stable across the terra-native rewrite (#34)", { + r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) + r20 <- terra::rast(system.file("extdata", "example_2020.tif", package = "drift")) + classified <- dft_rast_classify(list("2017" = r17, "2020" = r20), source = "io-lulc") + + # both helpers normalize the empty/all-NA raster (cats()[[1]] is NULL and + # freq() errors there) to a canonical empty frame so old and new agree. + freq_df <- function(r) { + f <- tryCatch(terra::freq(r), error = function(e) NULL) + if (is.null(f) || nrow(f) == 0) { + return(data.frame(value = character(0), count = integer(0))) + } + f <- f[!is.na(f$value), c("value", "count"), drop = FALSE] + f <- f[order(as.character(f$value)), , drop = FALSE] + rownames(f) <- NULL + f + } + cats_df <- function(r) { + ct <- terra::cats(r)[[1]] + if (is.null(ct) || nrow(ct) == 0) { + return(data.frame(id = integer(0), transition = character(0))) + } + ct <- ct[order(ct$id), , drop = FALSE] + rownames(ct) <- NULL + ct + } + digest <- function(res) { + summ <- as.data.frame( + res$summary[order(res$summary$from_class, res$summary$to_class), , drop = FALSE] + ) + rownames(summ) <- NULL + list( + summary = summ, + raster_cats = cats_df(res$raster), + raster_freq = freq_df(res$raster), + removed = if (is.null(res$removed)) "NULL" + else list(cats = cats_df(res$removed), freq = freq_df(res$removed)) + ) + } + + cases <- list( + default = list(), + from_trees = list(from_class = "Trees"), + both_filters = list(from_class = "Trees", + to_class = c("Crops", "Rangeland", "Bare Ground")), + pmin_0 = list(patch_area_min = 0), + pmin_500 = list(patch_area_min = 500), + pmin_1000 = list(patch_area_min = 1000), + pmin_huge = list(patch_area_min = 1e9), + impossible = list(from_class = "NoSuchClass") + ) + + digests <- lapply(cases, function(args) { + res <- do.call(dft_rast_transition, + c(list(classified, from = "2017", to = "2020"), args)) + digest(res) + }) + + expect_snapshot_value(digests, style = "serialize", tolerance = 1e-8) +}) From a9bc274974ce564ce50289c56c028c17d2c6ef5c Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 11:09:09 -0700 Subject: [PATCH 05/10] Phase 3: stream dft_rast_transition with terra ops (no full-grid R vectors) Rewrite the producer to eliminate the OOM class (#28): strip factor to code rasters via `* 1L`, encode from*1000+to by raster arithmetic, filter from/to classes by integer code-set membership (terra::subst), remove small patches with patches/subst/ifel, and derive codes + total_valid + summary from a single native terra::freq. No terra::values(), no rep(NA, ncell) -- peak R memory is now O(#codes + #patches), not O(ncell). Behavior-identical: the Phase-2 golden snapshot stays byte-for-byte; full suite 303 pass / 4 skip; /code-check round 1 clean (independent old-vs-new comparison byte-identical across 14 param combos incl. ESA WorldCover 3-digit codes). Producer-only peak RSS at 16M cells (patch_area_min=500): 2.66 GB -> 1.63 GB. Two terra gotchas handled: SpatRaster `%in%` is not dispatched when terra is imported (not attached) -> use terra::subst; terra::freq() errors on an all-NA raster -> guard with tryCatch. Relates to NewGraphEnvironment/sred-2025-2026#16 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- R/dft_rast_transition.R | 167 +++++++++++++++++------------------ man/dft_rast_transition.Rd | 6 ++ planning/active/findings.md | 19 ++++ planning/active/progress.md | 6 +- planning/active/task_plan.md | 10 ++- 5 files changed, 118 insertions(+), 90 deletions(-) diff --git a/R/dft_rast_transition.R b/R/dft_rast_transition.R index feaad0b..7d1c955 100644 --- a/R/dft_rast_transition.R +++ b/R/dft_rast_transition.R @@ -30,6 +30,12 @@ #' - `removed`: A `SpatRaster` of transitions removed by `patch_area_min` #' filtering, or `NULL` when no filtering is applied. Same factor #' encoding as `raster`. +#' +#' @details +#' The transition raster, filters, and patch removal are computed with streamed +#' `terra` operations (`ifel`, `patches`, `freq`) on the underlying class codes — +#' no full-grid vectors are pulled into R — so peak memory scales with the number +#' of distinct transitions and patches, not the grid size. #' @export #' @examples #' r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) @@ -83,65 +89,55 @@ dft_rast_transition <- function(x, dft_check_crs(r_from, "dft_rast_transition") dft_check_crs(r_to, "dft_rast_transition") - # Resolve class names to codes + # code -> class name lookup (small; used only for factor labels + summary) code_lookup <- stats::setNames(class_table$class_name, class_table$code) - # Get raw integer values (strip factor) - v_from <- terra::values(r_from)[, 1] - v_to <- terra::values(r_to)[, 1] - - # Map codes to class names - name_from <- code_lookup[as.character(v_from)] - name_to <- code_lookup[as.character(v_to)] - - # Encode transitions: from_code * 1000 + to_code (supports up to 999 classes) - trans_code <- v_from * 1000L + v_to - - # Build mask for filters - - keep <- rep(TRUE, length(trans_code)) - if (!is.null(from_class)) keep <- keep & (name_from %in% from_class) - if (!is.null(to_class)) keep <- keep & (name_to %in% to_class) - - # Also mask where either raster is NA - keep <- keep & !is.na(v_from) & !is.na(v_to) + # Strip factor to raw integer codes as streamed rasters. `* 1L` returns a new + # non-factor raster of codes, preserves NA, and does NOT mutate the inputs + # (they are references into the caller's list). Encode each transition as + # from_code * 1000 + to_code; NA propagates wherever either input is NA. No + # full-grid R vector is materialized. + code_from <- r_from * 1L + code_to <- r_to * 1L + r_trans <- code_from * 1000L + code_to - trans_code[!keep] <- NA_integer_ + # from_class / to_class filters as integer code-set membership (streamed) + r_trans <- apply_codeset(r_trans, code_from, from_class, class_table) + r_trans <- apply_codeset(r_trans, code_to, to_class, class_table) + cell_area_m2 <- prod(terra::res(r_from)) - # Build transition raster - r_trans <- terra::rast(r_from) - terra::values(r_trans) <- trans_code - - # Filter small patches of changed pixels - removed_codes <- NULL + # Filter small patches of *changed* pixels (streamed; no rep()/values()) + r_removed <- NULL if (!is.null(patch_area_min) && patch_area_min > 0) { - # Binary raster of actual changes only (from != to), not same-class pixels - changed <- !is.na(trans_code) & (v_from != v_to) - r_changed <- terra::rast(r_from) - r_changed_vals <- rep(NA_integer_, length(trans_code)) - r_changed_vals[changed] <- 1L - terra::values(r_changed) <- r_changed_vals + r_changed <- terra::ifel(!is.na(r_trans) & (code_from != code_to), 1L, NA) p <- terra::patches(r_changed, directions = 8) - cell_area_m2 <- prod(terra::res(r_from)) - f <- terra::freq(p) - f <- f[!is.na(f$value), ] - small_ids <- f$value[f$count * cell_area_m2 < patch_area_min] - if (length(small_ids) > 0) { - p_vals <- terra::values(p)[, 1] - mask_small <- !is.na(p_vals) & (p_vals %in% small_ids) - removed_codes <- trans_code[mask_small] - trans_code[mask_small] <- NA_integer_ - terra::values(r_trans) <- trans_code + f <- tryCatch(terra::freq(p), error = function(e) NULL) # NULL when no changes + if (!is.null(f)) { + f <- f[!is.na(f$value), , drop = FALSE] + small_ids <- f$value[f$count * cell_area_m2 < patch_area_min] + if (length(small_ids) > 0) { + # 1 at small-patch cells, NA elsewhere (incl. p == NA / stable cells). + # subst() is exact-match and scales; SpatRaster `%in%` is not dispatched + # when terra is imported (not attached). + sm <- terra::subst(p, small_ids, 1L, others = NA) + r_removed <- terra::ifel(!is.na(sm), r_trans, NA) # capture removed codes first + r_trans <- terra::ifel(!is.na(sm), NA, r_trans) # then drop them + } } } - # Build factor table from observed transitions - valid <- !is.na(trans_code) - unique_codes <- sort(unique(trans_code[valid])) + # Observed transition codes + counts from a single native freq (value == code, + # NA excluded). freq() errors on an all-NA raster -> treat as no transitions. + freq_tbl <- tryCatch(terra::freq(r_trans), error = function(e) NULL) + if (!is.null(freq_tbl)) { + freq_tbl <- freq_tbl[!is.na(freq_tbl$value), , drop = FALSE] + } + + m2_to_unit <- switch(unit, "m2" = 1, "ha" = 1e-4, "km2" = 1e-6) + cell_area <- cell_area_m2 * m2_to_unit - if (length(unique_codes) == 0) { - # No transitions found — return empty + if (is.null(freq_tbl) || nrow(freq_tbl) == 0) { terra::set.cats(r_trans, layer = 1, value = data.frame(id = integer(0), transition = character(0))) summary_tbl <- tibble::tibble( @@ -151,58 +147,57 @@ dft_rast_transition <- function(x, return(list(raster = r_trans, summary = summary_tbl, removed = NULL)) } - from_codes <- unique_codes %/% 1000L - to_codes <- unique_codes %% 1000L + freq_tbl <- freq_tbl[order(freq_tbl$value), , drop = FALSE] + codes <- freq_tbl$value + from_codes <- codes %/% 1000L + to_codes <- codes %% 1000L labels <- paste0( code_lookup[as.character(from_codes)], " -> ", code_lookup[as.character(to_codes)] ) - - lvl_df <- data.frame(id = unique_codes, transition = labels) - terra::set.cats(r_trans, layer = 1, value = lvl_df) + terra::set.cats(r_trans, layer = 1, + value = data.frame(id = codes, transition = labels)) # Summary table - m2_to_unit <- switch(unit, "m2" = 1, "ha" = 1e-4, "km2" = 1e-6) - res <- terra::res(r_from) - cell_area <- res[1] * res[2] * m2_to_unit - - freq_tbl <- terra::freq(r_trans) - freq_tbl <- freq_tbl[!is.na(freq_tbl$value), ] - total_valid <- sum(!is.na(trans_code)) - - # freq on a factor raster returns labels in $value — map back to codes - label_to_code <- stats::setNames(lvl_df$id, lvl_df$transition) - freq_codes <- label_to_code[as.character(freq_tbl$value)] - + total_valid <- sum(freq_tbl$count) summary_tbl <- tibble::tibble( - from_class = code_lookup[as.character(freq_codes %/% 1000L)], - to_class = code_lookup[as.character(freq_codes %% 1000L)], + from_class = code_lookup[as.character(from_codes)], + to_class = code_lookup[as.character(to_codes)], n_cells = as.integer(freq_tbl$count), area = freq_tbl$count * cell_area, pct = round(freq_tbl$count / total_valid * 100, 2) ) - summary_tbl <- summary_tbl[order(summary_tbl$n_cells, decreasing = TRUE), ] - # Build removed raster when patch filtering was applied - r_removed <- NULL - if (!is.null(removed_codes)) { - r_removed <- terra::rast(r_from) - rem_vals <- rep(NA_integer_, terra::ncell(r_from)) - rem_positions <- which(!is.na(p_vals) & (p_vals %in% small_ids)) - rem_vals[rem_positions] <- removed_codes - terra::values(r_removed) <- rem_vals - # Build factor table from removed codes - rem_unique <- sort(unique(removed_codes[!is.na(removed_codes)])) - rem_from <- rem_unique %/% 1000L - rem_to <- rem_unique %% 1000L - rem_labels <- paste0( - code_lookup[as.character(rem_from)], " -> ", - code_lookup[as.character(rem_to)] - ) - terra::set.cats(r_removed, layer = 1, - value = data.frame(id = rem_unique, transition = rem_labels)) + # Factor levels for the removed raster (only when patch filtering removed cells) + if (!is.null(r_removed)) { + rf <- terra::freq(r_removed) + rf <- rf[!is.na(rf$value), , drop = FALSE] + rf <- rf[order(rf$value), , drop = FALSE] + terra::set.cats(r_removed, layer = 1, value = data.frame( + id = rf$value, + transition = paste0(code_lookup[as.character(rf$value %/% 1000L)], " -> ", + code_lookup[as.character(rf$value %% 1000L)]) + )) } list(raster = r_trans, summary = summary_tbl, removed = r_removed) } + +#' Mask a transition raster to a from/to class set (streamed) +#' +#' Reproduces the old full-grid `name %in% from_class` filter using integer +#' code-set membership on a code raster, so no full-grid character vector is +#' built. `code_r %in% keep_codes` is FALSE (not NA) at NA cells, matching the +#' base-R string filter's NA handling. `NULL` selection is a no-op; an empty +#' selection (no class name matched) masks everything to NA. +#' @noRd +apply_codeset <- function(r_trans, code_r, names_sel, class_table) { + if (is.null(names_sel)) return(r_trans) + keep_codes <- class_table$code[class_table$class_name %in% names_sel] + if (length(keep_codes) == 0L) return(r_trans * NA) # impossible filter -> all NA + # subst(): 1 at in-set codes, NA at out-of-set and NA cells. Avoids SpatRaster + # `%in%`, which is not dispatched when terra is imported (not attached). + keep_mask <- terra::subst(code_r, keep_codes, 1L, others = NA) + terra::ifel(!is.na(keep_mask), r_trans, NA) +} diff --git a/man/dft_rast_transition.Rd b/man/dft_rast_transition.Rd index ff0eb09..e1b6b05 100644 --- a/man/dft_rast_transition.Rd +++ b/man/dft_rast_transition.Rd @@ -60,6 +60,12 @@ encoding as \code{raster}. Compare two classified rasters cell-by-cell and return a transition raster and summary table. Each pixel is encoded with its from→to class pair. } +\details{ +The transition raster, filters, and patch removal are computed with streamed +\code{terra} operations (\code{ifel}, \code{patches}, \code{freq}) on the underlying class codes — +no full-grid vectors are pulled into R — so peak memory scales with the number +of distinct transitions and patches, not the grid size. +} \examples{ r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) r20 <- terra::rast(system.file("extdata", "example_2020.tif", package = "drift")) diff --git a/planning/active/findings.md b/planning/active/findings.md index ed50a11..4b83065 100644 --- a/planning/active/findings.md +++ b/planning/active/findings.md @@ -131,6 +131,25 @@ filter (`:115`). Only then does NECR complete end-to-end. Separate floodplains P - Confirms the two-driver diagnosis and the fix targets. Semantics-gate assertions are the pre-implementation gate for Phase 3; they passed, so the rewrite proceeds. +## Phase 3 result (2026-07-09) + +- Rewrote `dft_rast_transition` to stream everything (`* 1L` factor-strip → code + rasters → `code_from*1000L+code_to`; `apply_codeset()` filters; `patches`/`subst` + for `patch_area_min`; `freq` for codes + `total_valid` + summary). Zero + `terra::values()`, zero `rep(NA, ncell)`. +- **Discovery mid-implementation:** SpatRaster `%in%` is NOT dispatched when terra is + *imported* (package context) — only when *attached* (`library(terra)`). `code_r %in% + keep_codes` in package code fell through to base `match()` → "match requires vector + arguments". The semantics-gate benchmark used `library(terra)` so it passed there; + package context differs. Fix: `terra::subst(x, from, 1L, others = NA)` (exact-match, + scales, dispatches when imported). Codified this as a gotcha. +- `terra::freq()` **errors** on an all-NA raster (does NOT return 0 rows) — guarded the + two producer freq calls with `tryCatch(..., error = \(e) NULL)` → empty-return path. +- Golden (Phase 2) stays byte-identical; full suite 303 pass / 4 skip; lint clean; + `/code-check` round 1 Clean (independent old-vs-new byte-identical across 14 cases + + ESA WorldCover 3-digit codes). Producer-only peak RSS at 16M cells (patch_area_min=500): + **2.66 GB (old) → 1.63 GB (new)**, gap widening with grid size. + ## Git base note Branch `34-lulc-transition-classify-ooms-on-large-f` is off `origin/main` (6ba10bb), NOT diff --git a/planning/active/progress.md b/planning/active/progress.md index 8433a96..d245637 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -25,4 +25,8 @@ impossible). Canonicalized (sorted) digest → content-strict, order-independent. `expect_snapshot_value(style="serialize")` golden at `_snaps/dft_rast_transition.md`. Fixed empty-case handling (cats()[[1]] NULL + freq() errors on all-NA). 44 pass, 0 skip. -- Next: Phase 3 — terra-native rewrite of `dft_rast_transition`; the golden must stay green. +- **Phase 3 done** (commit pending): terra-native `dft_rast_transition` rewrite. Golden + byte-identical; 303 pass / 4 skip; lint clean; /code-check round 1 Clean. Hit the + imported-terra `%in%` non-dispatch gotcha (→ `subst`) and the freq-on-all-NA error + (→ tryCatch). Producer peak RSS at 16M cells: 2.66 → 1.63 GB. +- Next: Phase 4 — vectorizer `changes_only` + small-patch raster pre-filter. diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index a07eaaa..4754a91 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -29,12 +29,16 @@ Full design + verified terra semantics in `findings.md`. `raster` cats + `freq(value,count)`, `removed` cats. This is the rewrite's contract. ## Phase 3: Producer terra-native rewrite (#28) -- [ ] Rewrite `R/dft_rast_transition.R`: `code_from/code_to <- r_from/r_to * 1L`; +- [x] Rewrite `R/dft_rast_transition.R`: `code_from/code_to <- r_from/r_to * 1L`; `r_trans <- code_from*1000L + code_to`; `apply_codeset()` `@noRd` for from/to filters - (empty-set guard); streamed `patch_area_min` via `ifel`/`patches`/`freq`/`ifel` + (empty-set guard); streamed `patch_area_min` via `ifel`/`patches`/`subst`/`ifel` (empty-id guard); freq-derived codes + `total_valid`; removed raster via `ifel` + freq cats; preserve empty-return `removed=NULL`. No `terra::values()`, no `rep(NA,ncell)`. -- [ ] Green: full `test-dft_rast_transition.R` + Phase-2 golden identical. + (subst, not `%in%`: SpatRaster `%in%` isn't dispatched when terra is imported.) +- [x] Green: full `test-dft_rast_transition.R` + Phase-2 golden byte-identical; full + suite 303 pass / 4 skip; lint clean; `/code-check` round 1 Clean (old-vs-new + byte-identical across 14 cases incl. ESA 3-digit codes). Producer-only peak RSS at + 16M cells: 2.66 GB (old) → 1.63 GB (new). ## Phase 4: Vectorizer working-set cap (#34 core) - [ ] `R/dft_transition_vectors.R`: add `changes_only = FALSE` (raster-level stable From c62783464fa8b8295bc046a43cb15725405e4a85 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 11:25:04 -0700 Subject: [PATCH 06/10] Phase 4: changes_only + small-patch pre-filter for dft_transition_vectors (#34) Cap the vectorizer working set for large floodplains. changes_only=TRUE drops stable (from==to) patches at the raster level (subst+mask) before as.polygons, so only actual change patches are polygonized -- the field OOM lever, since the producer's transition raster is non-NA over the whole floodplain (stable mosaic) and the caller discards stable patches anyway. NA-ing stable cells can't merge or split any change patch (stable neighbours were already boundaries), so the result equals the default filtered to non-stable rows -- proven by test. Also pre-filter small patches (freq+subst small pids -> NA) before as.polygons when patch_area_min is set, keeping the trailing st_area filter so output stays byte-identical (pinned 185 / 123.11 ha / 57 unchanged). Fixed a pre-existing empty-return schema gap surfaced by /code-check: the early return dropped the zone column, which changes_only makes reachable (an all-stable sub-basin) -> per-sub-basin bind_rows mismatch. Empty return now carries zone_col. Fragmented 9M-cell benchmark (as.polygons-dominated, NECR-like geometry): default 415,705 patches / 3.83 GB -> changes_only 44,537 patches / 1.71 GB (55% peak cut). Full suite 314 pass / 4 skip; lint clean. Relates to NewGraphEnvironment/sred-2025-2026#16 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- R/dft_transition_vectors.R | 59 +++++++++++++++-- data-raw/benchmark_transition_oom.R | 26 +++++--- man/dft_transition_vectors.Rd | 23 ++++++- planning/active/findings.md | 28 ++++++++ planning/active/progress.md | 8 ++- planning/active/task_plan.md | 17 +++-- tests/testthat/test-dft_transition_vectors.R | 67 ++++++++++++++++++++ 7 files changed, 205 insertions(+), 23 deletions(-) diff --git a/R/dft_transition_vectors.R b/R/dft_transition_vectors.R index 3fc5c92..a484b9f 100644 --- a/R/dft_transition_vectors.R +++ b/R/dft_transition_vectors.R @@ -17,14 +17,22 @@ #' Required when `zones` is supplied. #' @param patch_area_min Numeric or `NULL`. Minimum patch area in m². Patches #' smaller than this are dropped before returning. `NULL` (default) keeps all. +#' @param changes_only Logical. When `TRUE`, drop stable (`from == to`) +#' transitions before polygonizing, so only actual change patches are +#' vectorized. On a floodplain the stable mosaic is most of the grid, so this +#' is the main memory lever for large AOIs. Default `FALSE` keeps every patch +#' (including stable ones). #' #' @return An `sf` data frame (polygon geometry) with columns: #' - `patch_id` (integer) — connected component ID, numbered in raster -#' scan order +#' scan order over the returned patches #' - `transition` (character) — transition label (e.g. "Trees -> Rangeland") #' - `area_ha` (numeric) — patch area in hectares #' - Zone column (if `zones` supplied) — from spatial intersection #' +#' When `patch_area_min` or `changes_only` drop patches, `patch_id` is numbered +#' over the surviving patches (dense `1..n`), not the pre-filter grid. +#' #' @export #' @examples #' r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) @@ -39,10 +47,15 @@ #' # Filter to large patches only #' patches_large <- dft_transition_vectors(result$raster, patch_area_min = 1000) #' head(patches_large) +#' +#' # Only actual changes (drop stable from == to patches before polygonizing) +#' changes <- dft_transition_vectors(result$raster, changes_only = TRUE) +#' head(changes) dft_transition_vectors <- function(x, zones = NULL, zone_col = NULL, - patch_area_min = NULL) { + patch_area_min = NULL, + changes_only = FALSE) { if (!inherits(x, "SpatRaster")) { stop("`x` must be a SpatRaster (e.g. from dft_rast_transition()$raster).", call. = FALSE) @@ -70,18 +83,56 @@ dft_transition_vectors <- function(x, } } + if (!is.logical(changes_only) || length(changes_only) != 1 || + is.na(changes_only)) { + stop("`changes_only` must be a single logical (TRUE or FALSE).", call. = FALSE) + } + + # Optionally drop stable (from == to) transitions before polygonizing, so + # as.polygons() only builds geometry for actual change patches. Stable codes + # are those where from_code == to_code (id %/% 1000 == id %% 1000). On a + # floodplain the stable mosaic is most of the grid, so this caps the working + # set to what callers actually keep. + if (isTRUE(changes_only)) { + ct <- terra::cats(x)[[1]] + stable_ids <- ct$id[(ct$id %/% 1000L) == (ct$id %% 1000L)] + if (length(stable_ids) > 0) { + # NA at stable cells, 1 at change cells; mask keeps x's factor levels + change_mask <- terra::subst(x * 1L, stable_ids, NA, others = 1L) + x <- terra::mask(x, change_mask) + } + } + # Single pass: 8-connected components of same-valued cells. # Requires terra >= 1.8-10 (earlier versions falsely connect patches # across the left/right raster edges with values = TRUE). p <- terra::patches(x, directions = 8, values = TRUE) names(p) <- "pid" + + # When filtering by area, drop small patches at the raster level BEFORE + # polygonizing (as.polygons is the patch-count-driven hotspot). For axis- + # aligned raster polygons st_area == count * cell_area, so this drops a strict + # subset of what the trailing st_area filter removes -> identical output. + if (!is.null(patch_area_min) && patch_area_min > 0) { + fp <- tryCatch(terra::freq(p), error = function(e) NULL) + if (!is.null(fp)) { + cell_area_m2 <- prod(terra::res(p)) + small <- fp$value[!is.na(fp$value) & fp$count * cell_area_m2 < patch_area_min] + if (length(small) > 0) p <- terra::subst(p, small, NA) + } + } + polys_sf <- sf::st_as_sf(terra::as.polygons(p)) if (nrow(polys_sf) == 0) { - return(sf::st_sf( + out <- sf::st_sf( patch_id = integer(0), transition = character(0), area_ha = numeric(0), geometry = sf::st_sfc(crs = sf::st_crs(x)) - )) + ) + # match the zone-attributed schema so per-zone results bind cleanly when a + # zone has no (or, under changes_only, no change) patches + if (!is.null(zones)) out[[zone_col]] <- zones[[zone_col]][0] + return(out) } # Map each patch to its transition label, touching only the non-NA cells diff --git a/data-raw/benchmark_transition_oom.R b/data-raw/benchmark_transition_oom.R index ba120b0..81756f3 100644 --- a/data-raw/benchmark_transition_oom.R +++ b/data-raw/benchmark_transition_oom.R @@ -147,9 +147,9 @@ mk_synthetic_pair <- function(ncol, nrow, change_frac = 0.05, block = 20L) { list("2017" = mk(coarse_from), "2023" = mk(coarse_to)) } -run_profile <- function(ncol, nrow, change_frac) { +run_profile <- function(ncol, nrow, change_frac, block = 20L) { devtools::load_all(quiet = TRUE) - pair <- mk_synthetic_pair(ncol, nrow, change_frac) + pair <- mk_synthetic_pair(ncol, nrow, change_frac, block = block) cat(sprintf("Synthetic %d x %d = %.1fM cells, change_frac=%.3f\n", terra::ncol(pair[[1]]), terra::nrow(pair[[1]]), terra::ncell(pair[[1]]) / 1e6, change_frac)) @@ -159,12 +159,19 @@ run_profile <- function(ncol, nrow, change_frac) { # the stable ones. Report both so before/after (this branch) is comparable. trans <- drift::dft_rast_transition(pair, from = "2017", to = "2023", patch_area_min = 500) - v_all <- drift::dft_transition_vectors(trans$raster) # current behavior (all) - n_stable <- sum(vapply(strsplit(v_all$transition, " -> ", fixed = TRUE), - function(p) identical(p[1], p[2]), logical(1))) - cat(sprintf("dft_transition_vectors (current): %d patches total, %d stable (%.0f%%), %d change\n", - nrow(v_all), n_stable, 100 * n_stable / nrow(v_all), - nrow(v_all) - n_stable)) + mode <- if (length(commandArgs(TRUE)) >= 5) commandArgs(TRUE)[5] else "all" + if (identical(mode, "changes")) { + v <- drift::dft_transition_vectors(trans$raster, changes_only = TRUE) + cat(sprintf("dft_transition_vectors(changes_only=TRUE): %d patches polygonized\n", + nrow(v))) + } else { + v_all <- drift::dft_transition_vectors(trans$raster) # default (all patches) + n_stable <- sum(vapply(strsplit(v_all$transition, " -> ", fixed = TRUE), + function(p) identical(p[1], p[2]), logical(1))) + cat(sprintf("dft_transition_vectors(default): %d patches total, %d stable (%.0f%%), %d change\n", + nrow(v_all), n_stable, 100 * n_stable / nrow(v_all), + nrow(v_all) - n_stable)) + } } # ---- dispatch --------------------------------------------------------------- @@ -177,7 +184,8 @@ if (length(args) == 0 || args[1] == "semantics") { } else if (args[1] == "profile") { run_profile(as.integer(arg_or(args[2], 3000)), as.integer(arg_or(args[3], 3000)), - as.numeric(arg_or(args[4], 0.05))) + as.numeric(arg_or(args[4], 0.05)), + block = as.integer(arg_or(args[6], 20))) } else { stop("unknown mode: ", args[1]) } diff --git a/man/dft_transition_vectors.Rd b/man/dft_transition_vectors.Rd index d20c0d3..77379ab 100644 --- a/man/dft_transition_vectors.Rd +++ b/man/dft_transition_vectors.Rd @@ -4,7 +4,13 @@ \alias{dft_transition_vectors} \title{Vectorize transition raster into individual change patches} \usage{ -dft_transition_vectors(x, zones = NULL, zone_col = NULL, patch_area_min = NULL) +dft_transition_vectors( + x, + zones = NULL, + zone_col = NULL, + patch_area_min = NULL, + changes_only = FALSE +) } \arguments{ \item{x}{A factor \code{SpatRaster} from \code{\link[=dft_rast_transition]{dft_rast_transition()}} (the \verb{$raster} @@ -18,16 +24,25 @@ Required when \code{zones} is supplied.} \item{patch_area_min}{Numeric or \code{NULL}. Minimum patch area in m². Patches smaller than this are dropped before returning. \code{NULL} (default) keeps all.} + +\item{changes_only}{Logical. When \code{TRUE}, drop stable (\code{from == to}) +transitions before polygonizing, so only actual change patches are +vectorized. On a floodplain the stable mosaic is most of the grid, so this +is the main memory lever for large AOIs. Default \code{FALSE} keeps every patch +(including stable ones).} } \value{ An \code{sf} data frame (polygon geometry) with columns: \itemize{ \item \code{patch_id} (integer) — connected component ID, numbered in raster -scan order +scan order over the returned patches \item \code{transition} (character) — transition label (e.g. "Trees -> Rangeland") \item \code{area_ha} (numeric) — patch area in hectares \item Zone column (if \code{zones} supplied) — from spatial intersection } + +When \code{patch_area_min} or \code{changes_only} drop patches, \code{patch_id} is numbered +over the surviving patches (dense \verb{1..n}), not the pre-filter grid. } \description{ Convert a transition \code{SpatRaster} (from \code{\link[=dft_rast_transition]{dft_rast_transition()}}) into \code{sf} @@ -53,4 +68,8 @@ head(patches) # Filter to large patches only patches_large <- dft_transition_vectors(result$raster, patch_area_min = 1000) head(patches_large) + +# Only actual changes (drop stable from == to patches before polygonizing) +changes <- dft_transition_vectors(result$raster, changes_only = TRUE) +head(changes) } diff --git a/planning/active/findings.md b/planning/active/findings.md index 4b83065..e272919 100644 --- a/planning/active/findings.md +++ b/planning/active/findings.md @@ -150,6 +150,34 @@ filter (`:115`). Only then does NECR complete end-to-end. Separate floodplains P ESA WorldCover 3-digit codes). Producer-only peak RSS at 16M cells (patch_area_min=500): **2.66 GB (old) → 1.63 GB (new)**, gap widening with grid size. +## Phase 4 result (2026-07-09) + +- Added `changes_only = FALSE` to `dft_transition_vectors`: derive stable ids from + `cats(x)` (`id %/% 1000 == id %% 1000`), `subst`+`mask` to NA them before + `patches`/`as.polygons`. **Correctness property (proven):** NA-ing stable cells + cannot merge/split any change patch (stable neighbours were already different-valued + boundaries; 8-diagonal adjacency is direct regardless of the two orthogonal corners), + so `changes_only=TRUE` == the default result filtered to non-stable rows — same change + patches, same `area_ha`. Unit-tested. +- Small-patch raster pre-filter: `freq(p)` → `subst` small pids → NA before + `as.polygons`; trailing `st_area` filter kept → byte-identical (pre-filter's + `count*cell_area < min` drops are a strict subset; at 10 m cells the threshold lands + on multiples of 100, no float boundary flip). Pinned 185/123.11/57 unchanged. +- **The blocky-vs-fragmented lesson:** on a blocky synthetic (20-cell coherent blocks, + 9M cells) `changes_only` cuts patches 89% but peak RSS barely moves (1.71 vs 1.73 GB) + because `as.polygons` isn't the peak there — the raster ops are. On a **fragmented** + synthetic (block=3, 9M cells, 415k patches — NECR-like braided floodplain geometry + where `as.polygons` DOES dominate): default 415,705 patches / **3.83 GB** → + `changes_only=TRUE` 44,537 patches / **1.71 GB** — a **55% peak cut**. So the field + OOM lever is realized precisely when the floodplain is fragmented, matching the NECR + report. +- `/code-check` round 1: additions correct; found a pre-existing fragility that + `changes_only` amplifies — the empty-return early exit dropped the zone column, so an + all-stable (under `changes_only`) sub-basin returned a 0-row sf without `zone_col` → + downstream `bind_rows` schema mismatch in the per-sub-basin field loop. Fixed + (empty return now carries `zone_col`); verified rbind + bind_rows across empty + + non-empty per-zone results. + ## Git base note Branch `34-lulc-transition-classify-ooms-on-large-f` is off `origin/main` (6ba10bb), NOT diff --git a/planning/active/progress.md b/planning/active/progress.md index d245637..7b47c0c 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -29,4 +29,10 @@ byte-identical; 303 pass / 4 skip; lint clean; /code-check round 1 Clean. Hit the imported-terra `%in%` non-dispatch gotcha (→ `subst`) and the freq-on-all-NA error (→ tryCatch). Producer peak RSS at 16M cells: 2.66 → 1.63 GB. -- Next: Phase 4 — vectorizer `changes_only` + small-patch raster pre-filter. +- **Phase 4 done** (commit pending): `dft_transition_vectors` gained `changes_only` + (raster-level stable drop before as.polygons) + small-patch pre-filter. Proven + equivalence (changes_only == default-filtered-to-non-stable); 185/123.11/57 green; + 314 pass. /code-check round 1 found + I fixed a pre-existing empty-return zone-col gap. + Fragmented 9M-cell benchmark (NECR-like): 3.83 GB → 1.71 GB (55% cut) with changes_only. +- Next: Phase 5 — docs/NEWS/version bump 0.3.0→0.4.0, floodplains follow-up note, + close #34 + #28. diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index 4754a91..bfa16aa 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -41,13 +41,16 @@ Full design + verified terra semantics in `findings.md`. 16M cells: 2.66 GB (old) → 1.63 GB (new). ## Phase 4: Vectorizer working-set cap (#34 core) -- [ ] `R/dft_transition_vectors.R`: add `changes_only = FALSE` (raster-level stable - drop before `as.polygons`); add small-patch raster pre-filter (keep trailing - `st_area` filter). Roxygen: document `changes_only`; note `patch_id` densening - under filtering. -- [ ] New tests: `changes_only=TRUE` drops stable / keeps changes / survivor `area_ha` - unchanged; existing 185/123.11/57 stay green on the default path; `as.polygons` - peak drops on the Phase-1 synthetic high-patch grid. +- [x] `R/dft_transition_vectors.R`: added `changes_only = FALSE` (raster-level stable + drop via `subst`+`mask` before `as.polygons`); small-patch raster pre-filter (keep + trailing `st_area` filter). Roxygen documents `changes_only` + `patch_id` densening. + Also fixed a pre-existing empty-return schema gap (zone col dropped) that + `changes_only` amplifies — /code-check round-1 finding. +- [x] New tests: `changes_only=TRUE` == default filtered to non-stable (same change + patches + `area_ha`); composes with `patch_area_min`; validates input; empty result + carries the zone col (binds cleanly). 185/123.11/57 stay green. Full suite 314 pass. + Fragmented 9M-cell benchmark (as.polygons-dominated, NECR-like): 415k patches / 3.83 GB + (default) → 44k / 1.71 GB (`changes_only`) — 55% peak cut. ## Phase 5: Docs, NEWS, release, close #34 + #28 - [ ] `devtools::document()`; `lintr::lint_package()` clean; full `devtools::test()`. diff --git a/tests/testthat/test-dft_transition_vectors.R b/tests/testthat/test-dft_transition_vectors.R index 84ed8e8..5b382aa 100644 --- a/tests/testthat/test-dft_transition_vectors.R +++ b/tests/testthat/test-dft_transition_vectors.R @@ -191,3 +191,70 @@ test_that("fixture decomposition is stable (regression guard)", { expect_equal(nrow(dft_transition_vectors(result$raster, patch_area_min = 1000)), 57L) }) + +# changes_only drops stable (from == to) patches before polygonizing. Dropping +# stable cells to NA cannot merge or split any change patch (stable neighbours +# were already different-valued boundaries), so the result must equal the default +# result filtered to non-stable rows -- same change patches, same area_ha. +test_that("changes_only == default filtered to non-stable rows", { + r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) + r20 <- terra::rast(system.file("extdata", "example_2020.tif", package = "drift")) + classified <- dft_rast_classify(list("2017" = r17, "2020" = r20), source = "io-lulc") + result <- dft_rast_transition(classified, from = "2017", to = "2020") + + is_stable <- function(labels) { + vapply(strsplit(labels, " -> ", fixed = TRUE), + function(p) identical(p[1], p[2]), logical(1)) + } + + all_p <- dft_transition_vectors(result$raster) + chg_p <- dft_transition_vectors(result$raster, changes_only = TRUE) + + # no stable rows survive + expect_false(any(is_stable(chg_p$transition))) + expect_gt(nrow(chg_p), 0) + + # same set of change patches as the default result filtered to non-stable + all_chg <- all_p[!is_stable(all_p$transition), ] + expect_equal(nrow(chg_p), nrow(all_chg)) + expect_equal(sort(round(chg_p$area_ha, 6)), sort(round(all_chg$area_ha, 6))) + expect_setequal(unique(chg_p$transition), unique(all_chg$transition)) +}) + +test_that("changes_only composes with patch_area_min and validates input", { + r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) + r20 <- terra::rast(system.file("extdata", "example_2020.tif", package = "drift")) + classified <- dft_rast_classify(list("2017" = r17, "2020" = r20), source = "io-lulc") + result <- dft_rast_transition(classified, from = "2017", to = "2020") + + # changes_only + area filter: still change-only, all areas >= threshold + chg_big <- dft_transition_vectors(result$raster, changes_only = TRUE, + patch_area_min = 500) + expect_true(all(chg_big$area_ha >= 500 / 1e4)) + expect_true(nrow(chg_big) <= nrow(dft_transition_vectors(result$raster, + changes_only = TRUE))) + + # validation + expect_error(dft_transition_vectors(result$raster, changes_only = "yes"), + "logical") + expect_error(dft_transition_vectors(result$raster, changes_only = c(TRUE, FALSE)), + "logical") +}) + +test_that("empty result carries the zone column so per-zone results bind", { + r17 <- terra::rast(system.file("extdata", "example_2017.tif", package = "drift")) + r20 <- terra::rast(system.file("extdata", "example_2020.tif", package = "drift")) + classified <- dft_rast_classify(list("2017" = r17, "2020" = r20), source = "io-lulc") + result <- dft_rast_transition(classified, from = "2017", to = "2020") + + zv <- terra::as.polygons(terra::ext(result$raster)) + terra::crs(zv) <- terra::crs(result$raster) + zones <- sf::st_as_sf(zv) + zones$zone_name <- "z1" + + # patch_area_min above any real patch -> empty early return (path 1) + empty <- dft_transition_vectors(result$raster, zones = zones, + zone_col = "zone_name", patch_area_min = 1e12) + expect_equal(nrow(empty), 0) + expect_true("zone_name" %in% names(empty)) +}) From a4d3da5429a846246b74bb6fcd912dcf3975c42b Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 11:26:54 -0700 Subject: [PATCH 07/10] Phase 5: NEWS 0.4.0 + floodplains follow-up note (#34) NEWS entry for the OOM fixes (producer terra-native rewrite; vectorizer changes_only + small-patch pre-filter). Document the drift-only decision's follow-up: the floodplains caller passes changes_only=TRUE and drops its post-hoc from!=to filter so NECR completes end-to-end (separate floodplains PR). Relates to NewGraphEnvironment/sred-2025-2026#16 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- NEWS.md | 6 ++++++ planning/active/progress.md | 10 ++++++++-- planning/active/task_plan.md | 13 +++++++------ 3 files changed, 21 insertions(+), 8 deletions(-) diff --git a/NEWS.md b/NEWS.md index a834e27..6d1cbfa 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,9 @@ +# drift 0.4.0 + +- Categorical land-cover change detection no longer exhausts memory on large-floodplain AOIs (#34, #28). `dft_rast_transition()` was rewritten to stream entirely through `terra` — transitions are encoded and filtered with raster arithmetic, `terra::subst()`, `patches()`, and a single `terra::freq()`, with no `terra::values()` pull and no full-grid R vectors — so peak memory scales with the number of distinct transitions and patches, not the grid size (producer-only peak at 16M cells dropped from 2.66 GB to 1.63 GB). Output is byte-identical to the previous version, verified by a golden snapshot across the full parameter matrix. +- `dft_transition_vectors()` gains `changes_only` (default `FALSE`): when `TRUE`, stable (`from == to`) transitions are dropped at the raster level before polygonizing, so `terra::as.polygons()` only builds geometry for actual change patches. On a fragmented floodplain — where the stable mosaic is most of the grid and polygonization dominates memory — this roughly halves peak use (a 9M-cell, 415k-patch benchmark went from 3.83 GB to 1.71 GB). The result equals the default output filtered to change patches. When `patch_area_min` is set, small patches are also dropped before polygonizing, with identical output. +- `patch_id` in `dft_transition_vectors()` is numbered over the surviving patches when filtering drops any, and an empty result now carries the zone column so per-zone results bind cleanly. + # drift 0.3.0 - Continuous index-trajectory change detection for floodplain reaches (#30). A new fetch-and-reduce pipeline complements the categorical `dft_stac_fetch()` path. `dft_stac_cube()` builds a cloud-masked monthly spectral-index stack from Sentinel-2 (via a new `"sentinel-2-l2a"` source); `dft_rast_break()` reduces it per pixel with `bfast::bfastmonitor()` into a two-band raster of *abrupt* break date and magnitude; and `dft_rast_trend()` reduces it to a per-pixel *gradual* trend — a robust Theil-Sen slope (index change per year) with Mann-Kendall significance — for degradation/recovery monitoring the annual labels cannot show. Together they let a continuous trajectory validate categorical land-cover transitions (confirming which mapped losses carry a real spectral decline) and detect gradual change. See the "Trajectories as a Check on Land-Cover Change" vignette. diff --git a/planning/active/progress.md b/planning/active/progress.md index 7b47c0c..2836157 100644 --- a/planning/active/progress.md +++ b/planning/active/progress.md @@ -34,5 +34,11 @@ equivalence (changes_only == default-filtered-to-non-stable); 185/123.11/57 green; 314 pass. /code-check round 1 found + I fixed a pre-existing empty-return zone-col gap. Fragmented 9M-cell benchmark (NECR-like): 3.83 GB → 1.71 GB (55% cut) with changes_only. -- Next: Phase 5 — docs/NEWS/version bump 0.3.0→0.4.0, floodplains follow-up note, - close #34 + #28. +- **Phase 5 (docs) done** (commit pending): NEWS.md 0.4.0 section; floodplains + follow-up documented in findings.md. Version bump to follow as the final commit. +- Floodplains follow-up (user to wire/file): in + `floodplains/scripts/floodplain_lcc/03_lulc_classify.R`, pass `changes_only = TRUE` + to `dft_transition_vectors()` (:107) and drop the post-hoc `from!=to` filter (:115). + Only then does NECR complete end-to-end. Separate floodplains PR/issue (drift-only + decision this branch). +- Next: Release v0.4.0 (DESCRIPTION bump) → /planning-archive → /gh-pr-push. diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index bfa16aa..e38f82e 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -53,14 +53,15 @@ Full design + verified terra semantics in `findings.md`. (default) → 44k / 1.71 GB (`changes_only`) — 55% peak cut. ## Phase 5: Docs, NEWS, release, close #34 + #28 -- [ ] `devtools::document()`; `lintr::lint_package()` clean; full `devtools::test()`. -- [ ] `NEWS.md` 0.4.0: producer terra-native rewrite (no full-grid R vectors, OOM); +- [x] `devtools::document()`; `lintr::lint_package()` clean; full `devtools::test()` + (314 pass / 4 skip). +- [x] `NEWS.md` 0.4.0: producer terra-native rewrite (no full-grid R vectors, OOM); vectorizer `changes_only` + small-patch pre-filter; behavior-preserving defaults; `patch_id` densening note. -- [ ] Document `floodplains` follow-up (caller passes `changes_only=TRUE`, drops the - `:115` `from!=to` post-filter) in progress.md / a floodplains issue. -- [ ] Bump `DESCRIPTION` 0.3.0 → 0.4.0 as the **final** commit; terra floor bump only - if Phase 1 required it. +- [x] Document `floodplains` follow-up (caller passes `changes_only=TRUE`, drops the + `:115` `from!=to` post-filter) — findings.md "Cross-repo follow-up"; user to file/wire. +- [ ] Bump `DESCRIPTION` 0.3.0 → 0.4.0 as the **final** commit; terra floor unchanged + (1.8-10 sufficient — Phase 1 semantics gate passed). ## Validation - [ ] Tests pass (`devtools::test()`); network/bfast tests skip cleanly From a7d92e26215e754f468a894fbe6636747b6bb4fe Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 11:28:47 -0700 Subject: [PATCH 08/10] Release v0.4.0 Memory-scalable categorical land-cover change detection (#34, #28): terra-native dft_rast_transition (no full-grid R vectors) + dft_transition_vectors changes_only and small-patch pre-filter. Behavior-preserving; R CMD check clean (0/0/0). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- DESCRIPTION | 4 ++-- planning/active/task_plan.md | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index b2af226..e536cd9 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,7 +1,7 @@ Package: drift Title: Detecting Riparian and Inland Floodplain Transitions -Version: 0.3.0 -Date: 2026-07-08 +Version: 0.4.0 +Date: 2026-07-09 Authors@R: c( person("Allan", "Irvine", , "al@newgraphenvironment.com", role = c("aut", "cre"), comment = c(ORCID = "0000-0002-3495-2128")), diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index e38f82e..e5bb37a 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -60,8 +60,9 @@ Full design + verified terra semantics in `findings.md`. `patch_id` densening note. - [x] Document `floodplains` follow-up (caller passes `changes_only=TRUE`, drops the `:115` `from!=to` post-filter) — findings.md "Cross-repo follow-up"; user to file/wire. -- [ ] Bump `DESCRIPTION` 0.3.0 → 0.4.0 as the **final** commit; terra floor unchanged - (1.8-10 sufficient — Phase 1 semantics gate passed). +- [x] Bump `DESCRIPTION` 0.3.0 → 0.4.0 as the **final** commit; terra floor unchanged + (1.8-10 sufficient — Phase 1 semantics gate passed). `devtools::check(vignettes=FALSE)`: + 0 errors / 0 warnings / 0 notes. ## Validation - [ ] Tests pass (`devtools::test()`); network/bfast tests skip cleanly From 8d804f4e34ed8f2e55b894c25606b790f3ba217f Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 11:29:37 -0700 Subject: [PATCH 09/10] Sync PWF validation checkboxes (#34) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- planning/active/task_plan.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md index e5bb37a..38011c3 100644 --- a/planning/active/task_plan.md +++ b/planning/active/task_plan.md @@ -65,7 +65,8 @@ Full design + verified terra semantics in `findings.md`. 0 errors / 0 warnings / 0 notes. ## Validation -- [ ] Tests pass (`devtools::test()`); network/bfast tests skip cleanly -- [ ] `/code-check` clean on each commit -- [ ] PWF checkboxes match landed work +- [x] Tests pass (`devtools::test()` 314 pass / 4 skip); network/bfast tests skip cleanly +- [x] `/code-check` clean on each commit (Phase 3 round-1 clean; Phase 4 round-1 found + + fixed one pre-existing empty-return zone-col gap) +- [x] PWF checkboxes match landed work - [ ] `/planning-archive` on completion From 3e42e282e5daeb6cc9a48f75c135f5d32d7a37f9 Mon Sep 17 00:00:00 2001 From: almac2022 Date: Thu, 9 Jul 2026 11:41:10 -0700 Subject: [PATCH 10/10] Archive planning files for issue #34 Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz --- .../.gitkeep | 0 .../README.md | 45 +++++++++++++++++++ .../findings.md | 0 .../progress.md | 0 .../task_plan.md | 0 5 files changed, 45 insertions(+) create mode 100644 planning/archive/2026-07-issue-34-lulc-transition-oom/.gitkeep create mode 100644 planning/archive/2026-07-issue-34-lulc-transition-oom/README.md rename planning/{active => archive/2026-07-issue-34-lulc-transition-oom}/findings.md (100%) rename planning/{active => archive/2026-07-issue-34-lulc-transition-oom}/progress.md (100%) rename planning/{active => archive/2026-07-issue-34-lulc-transition-oom}/task_plan.md (100%) diff --git a/planning/archive/2026-07-issue-34-lulc-transition-oom/.gitkeep b/planning/archive/2026-07-issue-34-lulc-transition-oom/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/planning/archive/2026-07-issue-34-lulc-transition-oom/README.md b/planning/archive/2026-07-issue-34-lulc-transition-oom/README.md new file mode 100644 index 0000000..034c4c7 --- /dev/null +++ b/planning/archive/2026-07-issue-34-lulc-transition-oom/README.md @@ -0,0 +1,45 @@ +# Issue #34 (closes #28) — LULC transition/classify OOMs on large-floodplain AOIs + +## Outcome + +Fixed the remaining OOM class in the categorical pipeline (#27 fixed the vectorizer's +per-class loop; this fixed the two drivers left). Exploration found **two independent +memory drivers**, resolving the issue's apparent contradiction (NECR OOMs on a *smaller* +grid than UFRA, which completes): (1) the **producer** `dft_rast_transition()` built 6+ +full-grid R vectors incl. two full-grid *character* vectors — ncell-driven, this was #28; +(2) the **vectorizer** polygonized the whole floodplain's *stable* (`from==to`) mosaic +before the caller discarded it — floodplain-area-driven, the field OOM that kills NECR. + +**Fixes, both behavior-preserving:** +- `dft_rast_transition()` rewritten to stream through terra (`* 1L` factor-strip → + `code_from*1000L+code_to` arithmetic → `subst` code-set filters → `patches`/`freq`/`ifel` + for `patch_area_min`). Zero full-grid R vectors; peak memory is O(#codes + #patches). + Byte-identical to the old version (golden snapshot across 8 param combos, plus an + independent old-vs-new comparison across 14 cases incl. ESA WorldCover 3-digit codes). + Producer-only peak at 16M cells: 2.66 GB → 1.63 GB. +- `dft_transition_vectors()` gained opt-in `changes_only` (drop stable patches at the + raster level before `as.polygons`) + a small-patch raster pre-filter (keeps the trailing + `st_area` filter → byte-identical). NA-ing stable cells can't merge/split a change patch, + so `changes_only=TRUE` == default filtered to non-stable rows (proven by test). On a + fragmented NECR-like synthetic (9M cells, 415k patches, as.polygons-dominated): default + 3.83 GB → changes_only 1.71 GB (55% peak cut). + +Released as **v0.4.0**. Full suite 314 pass / 4 skip; `R CMD check` 0/0/0; lint clean. + +**Key learnings (durable):** +- SpatRaster `%in%` is NOT dispatched when terra is *imported* (package context) — only + when *attached* (`library(terra)`); use `terra::subst()` for code-set masks. +- `terra::freq()` **errors** on an all-NA raster (does not return 0 rows) → guard with + `tryCatch`. +- The changes_only memory win only appears when `as.polygons` is the peak driver — + fragmented/braided floodplain geometry (NECR), not blocky synthetics. +- `/code-check` round 1 caught a pre-existing empty-return schema gap (zone column dropped) + that `changes_only` amplifies in the per-sub-basin field loop; fixed. + +**Follow-up (drift-only decision, NOT done here):** the `floodplains` caller must opt in +for NECR to complete end-to-end — pass `changes_only = TRUE` in +`scripts/floodplain_lcc/03_lulc_classify.R:107` and drop the `from!=to` post-filter at +`:115`. Left to the user (separate floodplains PR). + +Closed by: PR for branch `34-lulc-transition-classify-ooms-on-large-f` (commits +845ff11..8d804f4), release v0.4.0. diff --git a/planning/active/findings.md b/planning/archive/2026-07-issue-34-lulc-transition-oom/findings.md similarity index 100% rename from planning/active/findings.md rename to planning/archive/2026-07-issue-34-lulc-transition-oom/findings.md diff --git a/planning/active/progress.md b/planning/archive/2026-07-issue-34-lulc-transition-oom/progress.md similarity index 100% rename from planning/active/progress.md rename to planning/archive/2026-07-issue-34-lulc-transition-oom/progress.md diff --git a/planning/active/task_plan.md b/planning/archive/2026-07-issue-34-lulc-transition-oom/task_plan.md similarity index 100% rename from planning/active/task_plan.md rename to planning/archive/2026-07-issue-34-lulc-transition-oom/task_plan.md