fix(ingest): trust CADC over _report.txt when deciding tile status#9
Merged
Merged
Conversation
do_ingest() was gating the 'Ingested' status on _report.txt showing 24/24 successes AND CADC having all expected products. When a transient minoc HEAD/PUT timeout dropped one file in the current run but the file was already in CADC from a prior partial run, the tile got marked IngestFailed even though every expected product was visible at CADC. Seen on tiles 11720, 11532, 11846 — each reported '_report.txt reports that ingestion failed' and 'CADC contains all products' in the same run. Flip the priority: CADC is the source of truth. If all 17 expected products are in CADC, the tile is Ingested regardless of _report.txt's per-run accounting. Only mark IngestFailed when CADC is actually missing products. This self-heals false-negative status updates without changing real-failure detection (e.g. tile 11519 has a genuinely missing RMSF_real_p3d_v1 product from a separate caom2utils blueprint bug and will still correctly mark IngestFailed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
do_ingestis marking tilesIngestFailedeven when CADC contains every expected product. Observed today on tiles 11720 and 11532 (both 943MHz):A previous run uploaded all 17 expected 3D-pipeline products. A re-run hit a transient minoc HEAD/PUT timeout on one or two files, so the run's
_report.txtshowedNumber of Successes: 23 (or less) / 24. The status decision logic gatesIngestedon both_report.txtstrict 24/24 and CADC having all products, so the tile got markedIngestFailedeven though the data is fully present.Cause
ingest3Dpipeline.py:319-326 before this PR:
_report.txtonly records the per-file outcomes of the current run. It's not the source of truth for whether the tile is actually fully at CADC — CADC's holdings are.Fix
Flip the priority: CADC is the source of truth. If all expected products are visible at CADC, the tile is
Ingested, regardless of what this particular run's_report.txtsays. Only markIngestFailedwhen CADC is actually missing products.What this fixes (and doesn't)
_report.txt= 24/24_report.txt< 24/24 (transient timeouts; data was uploaded by an earlier run)_report.txt< 24/24_report.txt= 24/24 (very unusual, would indicate something off in caom2pipe)Tiles affected by the false-negative case so far on v3.0.0: 11720, 11532, 11846, and likely earlier 11293.
Out of scope: tile 11519 has a genuinely missing product (
RMSF_real_p3d_v1) from a separate caom2utils blueprint double-configuration bug in theOutput3DMappingchain — that's a real data-loss case and will still correctly markIngestFailedafter this change. That bug needs a fix on thepossum2caom2/caom2utilsside (raising separately).Test plan
do_ingeston tile 11720 (943MHz). With this fix, it should markIngestedbecause CADC already has all 17 products.IngestFailed(correct) —RMSF_real_p3d_v1is genuinely missing.IngestFailedthat is actually fully at CADC will need a manual status correction (this PR only fixes future runs).