Skip to content

fix(ingest): trust CADC over _report.txt when deciding tile status#9

Merged
Sebokolodi merged 1 commit into
mainfrom
fix/ingest-status-prioritize-cadc
Jun 11, 2026
Merged

fix(ingest): trust CADC over _report.txt when deciding tile status#9
Sebokolodi merged 1 commit into
mainfrom
fix/ingest-status-prioritize-cadc

Conversation

@Sebokolodi

Copy link
Copy Markdown
Collaborator

Symptom

do_ingest is marking tiles IngestFailed even when CADC contains every expected product. Observed today on tiles 11720 and 11532 (both 943MHz):

check_CADC-0fa  →  CADC contains all products.
prefect.flow_runs →  _report.txt reports that ingestion failed
update_tile_database-3b7  →  Updated tile 11720 status to IngestFailed

A previous run uploaded all 17 expected 3D-pipeline products. A re-run hit a transient minoc HEAD/PUT timeout on one or two files, so the run's _report.txt showed Number of Successes: 23 (or less) / 24. The status decision logic gates Ingested on both _report.txt strict 24/24 and CADC having all products, so the tile got marked IngestFailed even though the data is fully present.

Cause

ingest3Dpipeline.py:319-326 before this PR:

status = "IngestFailed"
if success:                # _report.txt strict 24/24
    if CADCsuccess:
        status = "Ingested"
    else:
        print("_report.txt reports succesful ingest, but CADC ingest failed")
else:
    print("_report.txt reports that ingestion failed")

_report.txt only records the per-file outcomes of the current run. It's not the source of truth for whether the tile is actually fully at CADC — CADC's holdings are.

Fix

Flip the priority: CADC is the source of truth. If all expected products are visible at CADC, the tile is Ingested, regardless of what this particular run's _report.txt says. Only mark IngestFailed when CADC is actually missing products.

status = "IngestFailed"
if CADCsuccess:
    status = "Ingested"
    if not success:
        print("_report.txt reports per-file errors but CADC contains all expected products; treating as Ingested.")
elif success:
    print("_report.txt reports successful ingest but CADC is missing products.")
else:
    print("_report.txt reports that ingestion failed and CADC is missing products.")

What this fixes (and doesn't)

Scenario Before After
CADC has all 17 products, _report.txt = 24/24 Ingested Ingested
CADC has all 17 products, _report.txt < 24/24 (transient timeouts; data was uploaded by an earlier run) IngestFailed (false negative) Ingested ✅
CADC missing products, _report.txt < 24/24 IngestFailed IngestFailed
CADC missing products, _report.txt = 24/24 (very unusual, would indicate something off in caom2pipe) Ingested (was wrong) IngestFailed ✅

Tiles affected by the false-negative case so far on v3.0.0: 11720, 11532, 11846, and likely earlier 11293.

Out of scope: tile 11519 has a genuinely missing product (RMSF_real_p3d_v1) from a separate caom2utils blueprint double-configuration bug in the Output3DMapping chain — that's a real data-loss case and will still correctly mark IngestFailed after this change. That bug needs a fix on the possum2caom2 / caom2utils side (raising separately).

Test plan

  • Re-run do_ingest on tile 11720 (943MHz). With this fix, it should mark Ingested because CADC already has all 17 products.
  • Re-run on tile 11519. Should still mark IngestFailed (correct) — RMSF_real_p3d_v1 is genuinely missing.
  • Spot-check the spreadsheet / DB: any tile previously stuck on IngestFailed that is actually fully at CADC will need a manual status correction (this PR only fixes future runs).

do_ingest() was gating the 'Ingested' status on _report.txt showing 24/24 successes AND CADC having all expected products. When a transient minoc HEAD/PUT timeout dropped one file in the current run but the file was already in CADC from a prior partial run, the tile got marked IngestFailed even though every expected product was visible at CADC. Seen on tiles 11720, 11532, 11846 — each reported '_report.txt reports that ingestion failed' and 'CADC contains all products' in the same run.

Flip the priority: CADC is the source of truth. If all 17 expected products are in CADC, the tile is Ingested regardless of _report.txt's per-run accounting. Only mark IngestFailed when CADC is actually missing products. This self-heals false-negative status updates without changing real-failure detection (e.g. tile 11519 has a genuinely missing RMSF_real_p3d_v1 product from a separate caom2utils blueprint bug and will still correctly mark IngestFailed).
@Sebokolodi Sebokolodi merged commit af1475a into main Jun 11, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant