Problem
The pipeline reads crossings from fresh's bundled crossings.csv (lnk_pipeline_load.R:100) — a hand-exported bcfishpass.crossings snapshot, currently ~3 months stale — while link already syncs the override CSVs weekly. Base and overrides refresh on different clocks, breaking the same-build guarantee.
Verified bcfishpass.crossings_vw is a clean superset of the current source (all 10 modelling columns present, 99.4% population, not barrier-filtered), so a trimmed weekly export is a safe drop-in. The export also carries WGS84 longitude/latitude (12 columns total) — coordinates the pipeline itself doesn't use (it breaks by blue_line_key + downstream_route_measure) but which the upcoming crossing-rollup vignette needs to map crossings.
Change
Once db_newgraph emits a trimmed crossings.csv to s3://newgraph/ (NewGraphEnvironment/db_newgraph#15):
- Pull it in the weekly sync directly from the
newgraph bucket (public) — overrides stay on fresh-bc, so the sync reads both. Avoids a fresh-bc copy step and its cross-job ordering dependency.
- Assert the crossings build matches the overrides' build before accepting a bundle — crossings comes from the live DB (
jobs/dump_weekly), overrides from GitHub pinned to the ng-prod SHA; a mismatch reintroduces the base/override desync.
- Add a crate
canonical_schema: gate on the columns (halts the auto-PR on column drift). Note: aggregated_crossings_id is a UUID — confirm nothing downstream does integer math on it.
- Repoint
lnk_pipeline_load() to link's bundled copy; fresh sheds crossings.csv (its docker/load.sh pulls from S3 — fresh's R runtime never reads it).
Depends on NewGraphEnvironment/db_newgraph#15.
Problem
The pipeline reads crossings from fresh's bundled
crossings.csv(lnk_pipeline_load.R:100) — a hand-exportedbcfishpass.crossingssnapshot, currently ~3 months stale — while link already syncs the override CSVs weekly. Base and overrides refresh on different clocks, breaking the same-build guarantee.Verified
bcfishpass.crossings_vwis a clean superset of the current source (all 10 modelling columns present, 99.4% population, not barrier-filtered), so a trimmed weekly export is a safe drop-in. The export also carries WGS84longitude/latitude(12 columns total) — coordinates the pipeline itself doesn't use (it breaks byblue_line_key+downstream_route_measure) but which the upcoming crossing-rollup vignette needs to map crossings.Change
Once db_newgraph emits a trimmed
crossings.csvtos3://newgraph/(NewGraphEnvironment/db_newgraph#15):newgraphbucket (public) — overrides stay onfresh-bc, so the sync reads both. Avoids a fresh-bc copy step and its cross-job ordering dependency.jobs/dump_weekly), overrides from GitHub pinned to the ng-prod SHA; a mismatch reintroduces the base/override desync.canonical_schema:gate on the columns (halts the auto-PR on column drift). Note:aggregated_crossings_idis a UUID — confirm nothing downstream does integer math on it.lnk_pipeline_load()to link's bundled copy; fresh shedscrossings.csv(itsdocker/load.shpulls from S3 — fresh's R runtime never reads it).Depends on NewGraphEnvironment/db_newgraph#15.