Skip to content

#221 + #223: fix access-segmentation over-credit + accessible_km parity roll-up#228

Merged
NewGraphEnvironment merged 17 commits into
mainfrom
223-access-segmentation-frontier
Jul 4, 2026
Merged

#221 + #223: fix access-segmentation over-credit + accessible_km parity roll-up#228
NewGraphEnvironment merged 17 commits into
mainfrom
223-access-segmentation-frontier

Conversation

@NewGraphEnvironment

@NewGraphEnvironment NewGraphEnvironment commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Combined PR closing #223 (access-segmentation-frontier fix) and #221
(per-WSG accessible_km roll-up). They are inseparable: #221's accessible column
surfaced the #223 over-credit, and #223's fix makes #221's accessible parity pass.

#223 — the fix

gradient_barriers_minimal (the stream break source) was fed the
fresh::frs_barriers_minimal() downstream-most reduction as a segmentation
source, so a single stream segment straddled the accessibility frontier and its whole
reach — including the blocked part above the barrier — was credited accessible. BT/ST
accessible_km over-credited +40.4% (PCEA), +23.6% (FINA).

One-file fix (R/lnk_pipeline_prepare.R): union the raw per-model gradient + falls
positions into gradient_barriers_minimal so streams break at every frontier —
mirroring the orphan path already in the code, and matching bcfishpass (which segments
at every gradient barrier). frs_barriers_minimal is now unused in link.

#221 — the roll-up

Adds the accessible_km column to lnk_compare_rollup + a reusable lnk_rollup_wsg()
(tunnel-free vs fresh.streams_vw_bcfp).

Proof

data-raw/parity_crosssection.R + research/parity_accessible_habitat_2026_07_03.md
— across 11 WSGs × 8 species (Peace / Fraser / Skeena / Columbia):

metric pairs max |diff| within tol
accessible 44 0.05% 44/44
spawning 43 10.99% 42/43
rearing 35 35.20% 34/35

FINA/PCEA went from +23.6% / +40.4% pre-fix to −0.02% / −0.01%. The only 2 habitat
exceptions are BULK SK — the documented parked fresh#190 dual-rearing-lake topology
(research/bcfishpass_methodology.md:124), not a regression. Suite green (FAIL 1 | PASS 1298; the 1 FAIL is the environmental public.wsg_outlet missing table, #227).

Cost / scope

Segment count grows 2–3.5× (FINA 26k→70k, PCEA 34k→106k) — inherent to bcfp-matching
(streams now break at every gradient barrier, as bcfp does); intersects perf issue #205.
Re-runs stay 2–3 min/WSG.

Follow-ups (their own issues)

Closes #221. Closes #223. Relates to NewGraphEnvironment/sred#24.

🤖 Generated with Claude Code

https://claude.ai/code/session_014kSEJoad4Bc553qU8eFiE6

NewGraphEnvironment and others added 17 commits July 1, 2026 16:55
@NewGraphEnvironment @claude
Initialize PWF baseline for #221
d024b2c 
Per-WSG accessible_km roll-up + parity compare. Coho proof validated live
(MORR 0.09%, BULK 0.27% vs tunnel-free fresh.streams_vw_bcfp). Snapshot
barrier columns are comma-joined text, so accessible predicate is = '',
not = array[]::text[].

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#221 Phase 1: coho accessible_km parity proof (tunnel-free bcfp)
510ec08 
Add data-raw/accessible_km_proof_co.R: reproducible +/-5% assertion that
link's per-WSG coho accessible_km (streams_access.access_co IN (1,2),
joined to streams on full PK for length) matches the tunnel-free bcfp
snapshot (fresh.streams_vw_bcfp, barriers_ch_cm_co_pk_sk_dnstr = '').
19/20 local WSGs within +/-5% (most <1%); SETN allowlisted as the
documented bcfp-side stale-subsurfaceflow divergence. Comparison universe
is the coho-present lnk CTE (per-species vs salmon-group semantics), with
a null_ref guard for the link-built/bcfp-empty case.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#221 Phase 2 (1/4): add lnk_rollup_wsg reusable per-WSG roll-up
883c0e3 
New exported lnk_rollup_wsg: predicate-driven per-(WSG, species) length
roll-up over persisted streams + streams_access + streams_habitat_<sp>,
joined on full PK (#203). Aliases the species-varying columns to generic
access/spawning/rearing so the metrics SQL stays species-agnostic --
mirrors fresh::frs_aggregate's metrics/where shape. Default metrics emit
accessible_km + spawning_km + rearing_km.

accessible_km sums streams_access.access_<sp> IN (1,2) (the Phase-1
proof-validated source), NOT the streams_habitat_<sp>.accessible bool,
which carries different pre-gating semantics and diverges (MORR coho
3424 vs the validated 3330 km).

27 tests (arg validation + injection guards + offline SQL build via
DBI::ANSI()). Live-verified MORR coho accessible_km = 3330.25.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#221 Phase 2: doc-clarify lnk_rollup_wsg vs lnk_aggregate grain
f8ce11c 
Add @Seealso [lnk_aggregate()] and a note that lnk_rollup_wsg is a flat
per-WSG GROUP BY (whole-WSG length by species), distinct from
lnk_aggregate() / fresh::frs_aggregate() which roll habitat up the
network upstream of individual crossings. Makes the rollup-vs-aggregate
grain split intentional rather than an accidental shared-token clash.

Regenerated Rd family-index back-references so the compare family lists
lnk_rollup_wsg on each member (missed in the prior commit).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#221 Phase 2: fold habitat km rollup into lnk_rollup_wsg
4be5b87 
.lnk_compare_rollup_link's km block now delegates to lnk_rollup_wsg via a
COALESCE'd 5-metric vector, preserving the historical CASE-WHEN measured-0
semantics (FILTER over an empty set yields NULL, not 0), then renames
species->species_code and reselects columns to keep the
list(km, lake_ha, wetland_ha) contract. lake_ha/wetland_ha (DISTINCT-waterbody
polygon joins) stay as-is.

lnk_rollup_wsg's streams_access join changed JOIN->LEFT JOIN: access is optional
metadata for a length rollup, so habitat length is never dropped when access is
unbuilt (absent access -> NULL -> accessible_km 0). Safe even for
mapping_code=FALSE persists since lnk_pipeline_access runs unconditionally.

Byte-identical vs the old CASE-WHEN SQL on MORR BT+CO across all 5 metrics;
lnk_rollup_wsg accessible_km still 3330.25.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#221 Phase 2: emit accessible_km as 8th habitat_type in the rollup
0ced991 
.lnk_compare_rollup_link appends accessible_km to km_metrics:
round(COALESCE(sum(length_metre) FILTER (WHERE access IN (1,2)),0)/1000,2).
It sources link's per-species access model via lnk_rollup_wsg's LEFT-joined
`access` alias (streams_access.access_<sp>), NOT the divergent
streams_habitat.accessible bool. streams_access's PK
(id_segment, watershed_group_code) matches the join key, so the LEFT JOIN is
1:1 and cannot inflate any length sum.

.lnk_compare_wsg_assemble_rollup gains `accessible` (unit km) in its four
parallel named vectors. The bcfp habitat reference has no accessible column,
so accessible's ref_value/diff_pct are NA by design until the tunnel-free
reference path lands (Phase 2 4/4).

Row-count assertions bumped 7->8 / 14->16 in test-lnk_compare_wsg.R and 7->8
in test-lnk_compare_rollup.R, each with a new accessible-row check (link value
populated, ref NA). Live MORR coho accessible_km 3330.25 (= Phase-1 proof).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#221 Phase 2: tunnel-free accessible_km reference (salmon group)
aa65d68 
.lnk_compare_wsg_accessible_ref(conn, aoi, species) sums
fresh.streams_vw_bcfp.length_metre where a species' bcfp barrier-group
column = '' (empty = no barrier downstream = accessible). Sourced from
the local province-wide snapshot (conn), NOT the dead M1 :63333 tunnel.
Only the Phase-1-proven salmon group (CH/CM/CO/PK/SK ->
barriers_ch_cm_co_pk_sk_dnstr) is wired; other species short-circuit to
NA without a query (each needs its own proof, Phase 3). barrier_group
values are a trusted constant whitelist; aoi goes through
DBI::dbQuoteLiteral.

.lnk_compare_wsg_rollup_reference gains a local conn param and merges
accessible_km onto the tunnel-based habitat-ref data.frame. The two
reference sources stay intentionally decoupled: habitat stays
tunnel-based on conn_ref; accessible is tunnel-free on conn.
lnk_compare_rollup passes conn = conn.

Live: MORR CO ref 3327.38 vs link 3330.25 -> diff_pct 0.1% (= Phase-1
proof); BT ref NA. SK ref = CO ref (bcfp models the salmon group with
one shared column; per-species vs per-group coarsening surfaces in the
diff, documented for Phase 3).

4 new helper tests (mock DBI::dbGetQuery: salmon -> query with the
barriers_ch_cm_co_pk_sk_dnstr predicate; BT/ST/WCT -> short-circuit NA,
no query). m_ref mock signature gained conn. 65+20 tests green in the
two touched files; lint no new indent notes; docs regenerated.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
research: add blk 359209845 BT accessible_km segmentation proof figure
1a26f7d 
Visual proof of the accessible_km BT over-credit: link's segment
[3391,7998] spans the 25% gradient barrier at 3835 and stays accessible
whole, while bcfp breaks at 3835 and blocks [3835,7998] (4163 m).
Referenced by research/accessible_km_divergence.md and the tracking issue.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
research: root-cause the BT/ST accessible_km over-credit (#223)
4549713 
The accessible_km parity gap for BT/ST is a segmentation defect, not an
access-decision defect. The per-species gradient/falls break source is built
by running frs_barriers_minimal() (a downstream-most reduction) on each
barrier set, which strips the interior break points; a link segment then
straddles the accessibility frontier and the whole reach — including the
blocked part above the barrier — is labelled accessible. At co-located
positions link and bcfp agree on BT accessibility at 99.99%.

Documents the code path, the blk 359209845 segment-level proof (4163 m
over-credit on one blue line; aggregate FINA +1438 km / PCEA +2021 km), why
it stayed hidden at 99%+ parity, and the fix plan: break at the full
per-species barrier positions and reclassify like bcfp — no length clipping,
no barrier moved. Also captures the accessible_km parity findings from the
#221 investigation.

Filed as #223. Relates to #221, #203.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
Initialize PWF baseline for #223
43866a8 
Segmentation-frontier fix: gradient_barriers_minimal is fed the
frs_barriers_minimal downstream-most reduction as a segmentation source,
stripping interior breaks so a segment straddles the accessibility frontier
and the whole reach is credited accessible. Phases: reproduce (tests-first),
fix prepare.R, validate (no-regression), ship + rename follow-up.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#223: record live pre-fix fingerprint + session handoff
b32320d 
Confirmed on local fwapg: working_fina.gradient_barriers_minimal empty on blk
359209845; fresh.streams id_segment 4218 = [3391,7998] 4607 m one segment
straddling 3835; BT accessible_km FINA link 7520.7 / bcfp 6085.2 = +23.59%.
Enriched #223 with the full code trace, the exact fix, and this fingerprint.
Handoff note added for the next session to implement Phases 1-4.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@NewGraphEnvironment @claude
#223 Phase 1: accessible_km fix-validation script (fails pre-fix)
3ad1829 
Read-only DB validator that reproduces the BT/ST access-segmentation-frontier
over-credit. Parity sweep over {FINA,PARS,PCEA,LKEL} x species (link
fresh.streams JOIN fresh.streams_access on the full PK) vs the tunnel-free
reference fresh.streams_vw_bcfp, plus 4 structural checks on the canonical
evidence segment (blk 359209845, BT frontier 3834.78). Exits non-zero while the
bug is present; passes once the Phase 2 fix + re-run land.

- Validation set adds LKEL (smallest persisted WSG with steelhead + salmon) as
  the no-regression sentinel: the 3 Peace WSGs carry no salmon/ST, so the plan's
  item (d) salmon check would be vacuous there.
- One-sided-presence gate FAILs a link collapse-to-zero on a bcfp-present
  species, allowlisting the one known #189 residence case (LKEL:sk).
- Confirmed pre-fix: 7 checks fail (FINA/PARS/PCEA BT parity +23.6/+3.4/+40.4%;
  all 4 FINA structural, incl. the 4163 m above-frontier over-credit).
- Code-check (fresh-eyes agents): hardened S3 to isTRUE(all(...)) and closed the
  one-sided-presence exit-code blind spot.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_014kSEJoad4Bc553qU8eFiE6
@NewGraphEnvironment @claude
#223 Phase 2+3: break streams at every gradient frontier (fix + valid…
c86f103 
…ate)

Fix the BT/ST accessible_km over-credit. gradient_barriers_minimal (the stream
break source) was fed the frs_barriers_minimal() downstream-most reduction as a
SEGMENTATION source, so a single segment straddled the accessibility frontier and
its whole reach — including the blocked part above the barrier — was credited
accessible.

lnk_pipeline_prepare.R: stop minimal-reducing the per-model barrier set; union the
RAW per-model (gradient class-filtered + falls) positions into
gradient_barriers_minimal so streams break at every frontier — mirroring the orphan
path already used for sub-threshold classes, and matching bcfishpass (which segments
at every gradient barrier). frs_barriers_minimal is now unused in link; the per-model
_min tables are no longer built. Table name kept pending a rename issue; prepare.R +
break.R roxygen updated.

Validation (FINA/PARS/PCEA/LKEL re-run, mapping_code = TRUE):
- BT accessible_km converges to bcfp: FINA +23.59% -> -0.02%, PARS +3.43% -> -0.01%,
  PCEA +40.36% -> -0.01%, LKEL exact. data-raw/accessible_km_fix_validate.R ALL PASS.
- No downstream regression: BT spawning/rearing within ~1% on all 4 WSGs; LKEL ST/CO
  (salmon + steelhead sentinel) accessible exact, spawning/rearing hold; mapping_code
  improves. Habitat km is byte-identical pre/post fix — the fix is neutral on habitat.
- Cost: segments 2-3.5x (FINA 26k->70k, PCEA 34k->106k); runs still 2-3 min each.
  Inherent to bcfp-matching; intersects perf issue #205.

Also: validator S2 respects integer break rounding (breaks land at round(3834.78) =
3835, as bcfp does); the 4 .lnk_pipeline_prep_minimal unit tests updated to the new
contract (raw barriers_<sp> in the union, no frs_barriers_minimal call). devtools::test()
green (FAIL 1 = pre-existing environmental public.wsg_outlet, unrelated).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_014kSEJoad4Bc553qU8eFiE6
@NewGraphEnvironment
Merge remote-tracking branch 'origin/221-per-wsg-habitat-access-km-ro…
b53ad46 
…llup' into 223-access-segmentation-frontier

* origin/221-per-wsg-habitat-access-km-rollup:
  research: root-cause the BT/ST accessible_km over-credit (#223)
  research: add blk 359209845 BT accessible_km segmentation proof figure
  #221 Phase 2: tunnel-free accessible_km reference (salmon group)
  #221 Phase 2: emit accessible_km as 8th habitat_type in the rollup
  #221 Phase 2: fold habitat km rollup into lnk_rollup_wsg
  #221 Phase 2: doc-clarify lnk_rollup_wsg vs lnk_aggregate grain
  #221 Phase 2 (1/4): add lnk_rollup_wsg reusable per-WSG roll-up
  #221 Phase 1: coho accessible_km parity proof (tunnel-free bcfp)
  Initialize PWF baseline for #221

# Conflicts:
#	planning/active/findings.md
#	planning/active/progress.md
#	planning/active/task_plan.md
@NewGraphEnvironment @claude
#221+#223 Phase 5: provincial accessible+spawn+rear parity proof (11 …
20f815a 
…WSGs, 8 species)

Add data-raw/parity_crosssection.R — a tunnel-free harness that extends the habitat
parity rollup with the accessible_km column (#221): link via lnk_rollup_wsg vs bcfp
fresh.streams_vw_bcfp, IN(1,2) predicate (bcfp codes access/spawning/rearing 0/1/2/3;
a bare = 1 under-counts).

Ran across 11 WSGs re-run with the #223 fix (FINA/PARS/PCEA/LKEL + BULK/MORR/KISP/
LFRA/USKE + ELKR/KOTR — Peace/Fraser/Skeena/Columbia, all 8 species):

- accessible_km: 44/44 within 1%, max |diff| 0.05% — the fix converges exactly across
  every species and WSG (FINA/PCEA were +23.6% / +40.4% pre-fix).
- spawning 42/43, rearing 34/35 within 5% — the ONLY two exceptions are BULK SK
  (spawn +11.0%, rear +35.2%) = the documented parked fresh#190 dual-rearing-lake
  topology (research/bcfishpass_methodology.md:124). Pre-existing, not a #223 regression.
- #189 residence species (LKEL SK, BULK/MORR/USKE CM, USKE PK) correctly excluded.

research/parity_accessible_habitat_2026_07_03.md records the proof. Combined branch
test() green (FAIL 1 = pre-existing environmental wsg_outlet).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_014kSEJoad4Bc553qU8eFiE6
@NewGraphEnvironment @claude
Consolidate 3 accessible-km scripts into one parity_crosssection.R
d74702e 
Reduce data-raw script sprawl: fold accessible_km_fix_validate.R (#223 fix gate +
structural checks) and accessible_km_proof_co.R (#221 coho-only accessible proof)
into the single parity_crosssection.R, which already did the broader accessible +
spawn + rear × 8-species × N-WSG sweep.

parity_crosssection.R now also runs the #223 structural checks (FINA blk 359209845
frontier: breaks at frontier, above-frontier BT-blocked, accessible reach tops out)
and exits non-zero on any accessible-over-tol, structural failure, or non-parked
habitat divergence. Habitat parked cases (fresh#190 BULK SK) and #189 residence
species (link-absent) are allowlisted. Full 11-WSG run PASSes (exit 0).

Repoint lnk_rollup_wsg roxygen + the PWF Validation section to the single script.
Net: -344 lines, 3 scripts -> 1.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_014kSEJoad4Bc553qU8eFiE6
@NewGraphEnvironment
Archive planning files for #221 + #223 (accessible_km parity)
51f07a4
@NewGraphEnvironment NewGraphEnvironment merged commit d2de10f into main Jul 4, 2026
1 check passed
@NewGraphEnvironment NewGraphEnvironment deleted the 223-access-segmentation-frontier branch July 4, 2026 04:28
This was referenced Jul 5, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

1 participant

@NewGraphEnvironment