Sentinel-verified redeploy detection in price-backfill#14
Merged
Conversation
A run of 404s is normally just items with no ChromieCraft data, not a site redeploy. The old heuristic re-resolved the buildId after N consecutive 404s, so long no-data streaks caused a re-resolve storm + huge recovery re-fetch that rate-limited the site (a real 20k run hit gen=181 and 3175 false fetch-failures). Now, on crossing the threshold, verify with a known-good sentinel item (--sentinel-id, default 4389): only re-resolve if the sentinel also 404s under the current buildId, and bump the generation only if the buildId actually changed. Adds --sentinel-id, documents behavior, updates tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The redeploy heuristic re-resolved the wowauctions.net
buildIdafter N consecutive 404s. But ~half of WotLK items legitimately have no ChromieCraft data (a normal 404), so long no-data streaks looked like redeploys. The real 20,721-item run hitgen=181, triggered a 9,005-item recovery re-fetch, and rate-limited the site into 3,175 falsefetch-failedresults (~1,400 of which actually had prices). Those had to be salvaged with a manual gentle re-fetch.Fix
A genuine redeploy makes the current
buildId404 for every item, including ones that always have data. So after--redeploy-thresholdconsecutive 404s, verify with a known-good sentinel item (--sentinel-id, default 4389 = Gyrochronatom):Cost of a false alarm is now one cheap sentinel fetch per streak instead of a re-resolve storm.
Tests
29 passing.
NoteResultTestrewritten to cover: sentinel-ok (no re-resolve), sentinel-404 with changed buildId (re-resolve + gen bump), sentinel-404 with unchanged buildId (no gen bump), and counter reset.