fix(wmo): make high-volume sources fit inside the poll timeout - #75
Merged
Conversation
The shared cache held 256 entries. cn-cma-xx needs 501 distinct CAP URLs per poll, so a single poll evicted its own earliest entries before it finished and every subsequent poll re-fetched all 501 from cold — ~32 s against a 30 s default timeout. The entry never completed an update, so coordinator.data stayed frozen at the last success; _sync_alert_entities therefore saw stale ids still present and never removed their entities, which surfaced as an alert entity stuck reading "unknown" after its area filter changed. An entry count cannot bound this. CAP body size spans two orders of magnitude across sources: sampled 2026-08-04, cn-cma-xx bodies average 88.9 KiB and peak at 520 KiB, where a typical NWS or MeteoAlarm body is a few KiB. So 256 entries was simultaneously ~22 MiB of CMA XML — for a cache shared by every config entry, which also evicted the other entries' bodies — and less than half of what one poll of that source needs. Bound by memory instead: 64 MiB, evicting LRU by measured size, which holds cn-cma-xx's ~44 MiB working set with room for the rest and caps worst-case growth however large a body gets. Sizes come from sys.getsizeof rather than a character count, since these bodies are largely non-ASCII and a Python str there costs well over a byte per character. A body exceeding the whole budget is not cached at all — storing it would evict everything else and then be evicted itself, which is strictly worse than a miss. Assisted-by: Claude:claude-opus-5
The geocode-prefix filter (#73) runs post-fetch, so a narrowed cn-cma-xx entry still fetched all 501 CAP bodies and then threw away 94% of them. Correct, but it means the filter cannot help the one problem that makes that source unusable: the fetch itself does not fit in the poll timeout. Every one of the source's 500 RSS items carries a guid of the shape <6-digit area code><serial>_<timestamp> — "52272741600000_2026080410351" is Pingtang County (GB/T 2260 522727), whose CAP body geocode is "522727000000". Where that shape holds, a prefix decision on the guid is identical to one on the geocode, so the body need never be fetched. Measured A/B against the live feed, prefix 13: with pre-filter 141 docs fetched -> 71 alerts 10.1 s without pre-filter 478 docs fetched -> 71 alerts 23.7 s Same 71 alerts either way, which is the property that matters: this is an optimization under an authoritative filter, never a second opinion about what the user asked for. Losslessness rests on disengaging entirely — fetch everything, decide after parsing — whenever the guid cannot answer the question. A prefix longer than the embedded area code reaches into the serial digits, where guid and geocode provably diverge: the body's "130709000000" matches a full 12-digit code but the guid's "13070941600000" does not. A non-numeric prefix says nothing about a numeric guid. A single item whose guid does not match the shape means this is not such a feed, and applying the gate to the rest would silently drop the odd one out. And a filter that would keep nothing is far more likely to mean the convention changed than that the user has no alerts, so that fails open too. Each guard has a test, since the cost of getting one wrong is an alert that never appears. No effect on any other source or on an entry with no prefixes set: the gate is skipped outright, and every guard above returns the full item list, so link extraction is unchanged. Assisted-by: Claude:claude-opus-5
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #74, from a bug found while running it against a live
cn-cma-xxentry: after changing the area prefix from11to13, the old alert entity persisted showing "unknown".Root cause
The entry had not completed a successful poll since the change:
When the fetch raises,
coordinator.datakeeps the last successful result but listeners still fire, so_sync_alert_entitiessaw the stale id still incurrent_idsand never put it into_remove.AlertEntity.native_valuethen returnedNonefor an alert it could no longer resolve — "unknown". The sync logic itself was fine; it was being fed frozen data.The timeout was not incidental. The shared
CAPContentCacheheld 256 entries;cn-cma-xxneeds 501 distinct CAP URLs per poll. One poll evicted its own earliest entries before finishing, so every poll re-fetched all 501 cold — ~32 s against a 30 s default timeout, forever. Being shared domain-wide, it also evicted every other entry's bodies.1. Bound the cache by bytes, not entries
An entry count cannot bound memory here. Sampled 2026-08-04:
So 256 entries was both ~22 MiB of CMA XML and less than half of one poll's working set. Naively raising it to 2048 would have meant ~178 MiB.
Now a 64 MiB LRU budget measured with
sys.getsizeof— real memory, not a character count, since these bodies are largely non-ASCII. That holds CMA's ~44 MiB working set with room for other entries and caps worst-case growth whatever a body weighs. A body larger than the whole budget is not cached at all, since storing it would evict everything then be evicted itself.2. Skip CAP bodies the geocode filter would discard
The #73 filter is post-fetch, so a narrowed entry still fetched all 501 bodies and discarded 94%. Correct, but useless against the timeout.
Every one of the source's 500 RSS items carries a guid shaped
<6-digit area code><serial>_<timestamp>—52272741600000_20260804103516is Pingtang County (GB/T 2260522727), whose CAP body geocode is522727000000. Where that holds, a prefix decision on the guid equals one on the geocode, so the body need never be fetched.Measured A/B against the live feed, prefix
13:Same 71 alerts either way — the property that matters. This is an optimization under an authoritative filter, never a second opinion about what the user asked for.
Losslessness
It disengages entirely — fetch everything, decide after parsing — whenever the guid cannot answer the question:
130709000000matches a full 12-digit code, the guid's13070941600000does not. Filtering there would drop the exact alert requested.Each guard has a test, because the cost of getting one wrong is an alert that never appears. Sources without prefixes set, and every non-CMA source, skip the gate entirely and extract links exactly as before.
Verification
659 tests pass (19 new), ruff/format/mypy clean. Deployed to a dev HA instance: the CMA entry now polls without timing out, where it previously failed every cycle.
One existing test moved from
max_entries=2to an equivalent byte budget.Assisted-by: Claude:claude-opus-5