Skip to content

Decouple versions file from PCC catalog commits (main) - #29531

Open
crackcodecamp wants to merge 1 commit into
red-hat-data-services:mainfrom
crackcodecamp:fix-pcc-partial-update-main
Open

Decouple versions file from PCC catalog commits (main)#29531
crackcodecamp wants to merge 1 commit into
red-hat-data-services:mainfrom
crackcodecamp:fix-pcc-partial-update-main

Conversation

@crackcodecamp

Copy link
Copy Markdown
Member

Problem

When component bundles are published to production, the Red Hat operator indexes for each OCP version don't all update at the same time. Some indexes get the new version quickly, others lag. The PCC regeneration commits whatever catalogs changed, along with the versions file — even if only 2 of 9 catalogs were updated. This locks out future regeneration for the remaining 7 catalogs (PCC_CACHE_VALID=YES since the versions file was committed).

This happened twice:

  • 2.25.9: only v4.20 updated, v4.16/v4.19/v4.21/v4.22 stayed stale
  • 2.25.10/3.3.6/3.4.3 embargo: v4.16 and v4.19 missed entirely

Both required manual reverts to unblock.

Fix

Decouple catalog commits from the versions file:

  1. Stage only pcc/catalog-*.yaml files (not git add -A)
  2. For each new version in the diff, check if it exists in ALL catalog files
  3. If ALL catalogs are consistent → commit catalogs + versions file (full update)
  4. If some catalogs are missing new versions → commit only changed catalogs, NOT the versions file (partial update)
  5. If no catalogs changed → revert everything (fully stale)

Partial updates commit what's ready without locking out future regeneration. The next run re-detects the gap and retries.

Test results (local simulation)

Scenario Catalogs committed Versions file Result
1/4 indexes caught up 1 catalog NOT committed Partial — next run retries
All 4 indexes caught up All 4 catalogs Committed Full update — done
No indexes caught up None None Reverted — retry
3/4 caught up 3 catalogs NOT committed Partial — retry for last
Last one catches up 1 catalog + versions Committed Full — consistent now
Intentional skip (v4.22) 3 catalogs NOT committed Keeps retrying (use skip-bundles)

Fixes: RHOAIENG-74801

When operator indexes update partially (some OCP versions get the new
bundle, others don't yet), the PCC commit was including the versions
file alongside whichever catalogs changed. This locked out future
regeneration for the remaining catalogs since PCC_CACHE_VALID=YES.

Now the commit logic checks if ALL new versions are present in ALL
catalog files. Only commits the versions file when fully consistent.
Partial updates commit changed catalogs without the versions file,
so the next run retries for the remaining catalogs.

Fixes: RHOAIENG-74801
@crackcodecamp crackcodecamp changed the title Fix partial PCC update in regen-pcc-cache and multi-push-to-stage (main) Decouple versions file from PCC catalog commits (main) Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant