feat(weather-recon): Phase 2c — archived NWS forecasts, 15.3k segments loaded (#184)#195
Merged
Conversation
…mpat) Add wfo and nws_zone fields to weather_stations schema and loader. Maintains backward-compat: loader accepts both 8-column (legacy) and 10-column headers, returning empty strings for missing fields on old CSVs until Task 3 regenerates. - stations.py: OPTIONAL_COLUMNS, flexible header check, wfo/nws_zone in rows - directus.py: weather_stations gains two string fields - test_stations.py: tests for 10-column rows, empty values for CA/MX, compat Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c
Implement zone resolution scoring to resolve station names to 2001 NWS forecast zones. Supports exact hint matching and name-based scoring. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c
Adds scripts/enrich_stations.py: per-US-station WFO lookup via
api.weather.gov points, then 2001 forecast-zone resolution against
archived ZFP segments (weather_recon.zone_resolve). Regenerates
data/stations.csv with wfo/nws_zone columns for all 153 US stations
(CA/MX left blank by design).
153 US stations: 100 exact + 41 name-matched automatically, 11 curated
overrides, 1 accepted gap (KHSV/HUN -- IEM's archive has zero ZFP
products for that office in 2001 under any pil).
Overrides (evidence read from archived ZFP segments, never guessed):
KJFK/KLGA->OKX/NYZ076, KEWR->OKX/NJZ005 (NYC boroughs bundled with
alphabetical-vs-numeric ordering mismatch in combined segments;
confirmed via unambiguous single/double-zone segments elsewhere in the
2001 stream), KOAK->MTR/CAZ007, KSJC->MTR/CAZ008, KGTF->TFX/MTZ012,
KHLN->TFX/MTZ014, KIDA->PIH/IDZ020, KPIH->PIH/IDZ021,
KFMY->TBW/FLZ065, KBFL->HNX/CAZ092.
Two minimal patches beyond the brief:
- enrich_stations.py: PIL_OVERRIDES for 16 WFOs whose modern cwa
differs from their 2001 AFOS product id (e.g. FFC->ATL, FWD->FTW,
MFL->MIA, HFO->HI) -- confirmed per-office via IEM's product-list
API. This alone resolved 39 of the original 40 unresolved stations,
including majors KATL/KDFW/KDAL/KMIA/KPBI, via the existing
exact/name logic once given the right archive.
- weather_recon/afos.py: WMO_RE only matched K-prefixed WMO station
ids, so Alaska/Hawaii/Pacific offices (P-prefixed: PAFC, PHFO, ...)
never parsed at all. Widened to [KP][A-Z0-9]{3}; resolved
PANC/PAFA/PAJN/PHNL/PHLI/PHOG/PHTO automatically.
- points_lookup now catches request/parse failures per-station
(treated as unresolved) instead of crashing the sweep.
data/enrich-cache/ added to .gitignore (dev-run cache, not committed).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c
Reviewer-confirmed all ZONE_OVERRIDES zone ids are factually correct, but 8 of 11 override comments (KJFK, KLGA, KEWR, KGTF, KHLN, KIDA, KPIH, and KBFL's Dec single-zone citation) cited segments outside the committed data/enrich-cache/ 2001-09-08..13 window -- they were verified out-of-band via full-2001 IEM retrieve.py fetches for the same office. Comments now state the real provenance per entry; in-cache wording kept for KOAK/KSJC/KFMY, whose evidence is in the cached window. Also dropped the docstring's unimplemented --refresh flag mention. No functional change: script still exits 0, 76 tests pass, ruff clean. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c
fetch_afos.fetch_wfo_products caches raw ZFP text per (wfo, window), mirroring fetch_ncei's contract. AFOS_PIL_OVERRIDES (moved from enrich_stations.py) + afos_pil() resolve the 2001-era AFOS PIL for the 16 offices whose modern cwa differs from their archived product id, so the flow doesn't silently fetch empty product sets for those WFOs. enrich_stations.py now imports afos_pil instead of duplicating the override table; its own cache/behavior is unchanged (verified: re-run exits 0 from cache, zero stations.csv diff). load_weather_forecasts (flow_forecasts.py) parses cached ZFP products into weather_forecasts rows (one per segment) and reloads Directus. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c
Co-Authored-By: Claude Fable 5 <[email protected]>
Comma-joined expanded zone lists reach 307 chars; the live load failed on string type. Directus field live-patched to match during 2c-5 verification. Co-Authored-By: Claude Fable 5 <[email protected]>
Phase 2c final review found that UGC_LINE_RE requires an SSZ/SSC prefix on every line, but real 2001 ZFP products (ILN/IWX/JAN/LMK/PBZ) wrap a segment's UGC onto a continuation line that starts with a bare 3-digit zone number inheriting the previous line's prefix. split_segments stopped collecting at the first line, silently dropping the continuation zones into area_names. Measured impact: 31/15,304 weather_forecasts rows incomplete, 96 zone ids dropped; KDAY (OHZ061) matched only 19/21 issuances. Add a continuation-line loop in split_segments (UGC_CONTINUATION_RE) that only fires after at least one UGC_LINE_RE line has matched, so a segment still can't start with a bare number. expand_ugc is unchanged since the joined UGC string still starts with a prefixed token. Reloaded weather_forecasts against the live cluster Directus: 15304 -> 15304 rows (defect corrected zones within existing rows, not row count). KDAY OHZ061 zone-contains count now 21 (was 19); ILN row 57179 now includes the continuation zones OHZ051/052/053/060/061/062. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c
|
✅ Playwright E2E — 1 passed · 0 failed · 0 flaky · 0 skipped |
This was referenced Jul 13, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2c of the Weather app (#184): archived NWS forecasts. 15,304 real Zone Forecast Product segments (2001-09-08..12, 103 forecast offices) are now in production Directus, and all 152 forecast-covered US stations resolve to their actual 2001 NWS zone.
What's in here
weather_recon/afos.py— AFOS archive parsers: \x01/\x03 product splitting, WMO-header issuance times (UTC, the anachronism boundary), UGC zone expansion (ranges, prefix inheritance, multi-line + digit-starting continuation lines),$$segment splitting. Fixtures are real captured KOKX/ILN product text.weather_recon/zone_resolve.py— station→2001-zone resolution. Key finding: api.weather.gov zone ids are modern (KJFK's NYZ178 didn't exist in 2001) — so the modern id is only a hint, trusted when present in the archived products, else name-matched against real segment headers.scripts/enrich_stations.py—stations.csvnow carrieswfo+nws_zone(10 columns): 100 exact + 41 name-matched + 11 evidence-cited manual overrides + 1 accepted gap (KHSV — the HUN office has no 2001 ZFP products in the archive at all). Also discovered 16 offices whose 2001 AFOS product ids differ from their modern cwa (Atlanta's forecasts live underZFPATL, notZFPFFC) — mapping lives infetch_afos.AFOS_PIL_OVERRIDES, shared by script and flow.weather_recon/fetch_afos.py+flow_forecasts.py— PIL-aware cached fetch → parse → one row per zone segment (zone= comma-joined expanded ids,texttype after a live-found varchar(255) overflow) → idempotent reload.Verification
issued_atbounds clean; idempotent reload; 0 of 152 stations without a matching forecast row; Dayton's UGC-continuation gap found by the final review's full-corpus differential and fixed + reloaded (19→21 issuances).Follow-ups
Phase 2d: almanac (GHCN-Daily). Later: AFD products (spec lists them;
product_typecolumn ready), month-boundary guard if the window ever widens.🤖 Generated with Claude Code
https://claude.ai/code/session_01W179bqEPFP92a3JrsPkD5c