OCPBUGS-86842: [release-4.21] catalogd: clean up orphaned temp dirs in catalog storage by rashmigottipati · Pull Request #742 · openshift/operator-framework-operator-controller

rashmigottipati · 2026-06-01T23:29:19Z

Backport of OCPBUGS-76453 to release-4.21.

LocalDirV1.Store creates a temp dir (.{catalog}-{random}) and renames it into place atomically. If the process is interrupted before the deferred RemoveAll runs, the temp dir persists. Each restart adds another, eventually filling the disk.

Primary fix: remove any matching temp dirs at the start of Store before creating a new one. Since Store holds the write mutex, any dirs found are guaranteed to be from a previous run.

Defense-in-depth: add /var/cache/catalogs to GarbageCollector's list of scanned paths (CachePath string -> CachePaths []string), so orphaned temp dirs are also removed by the periodic GC cycle.

Original PR: operator-framework/operator-controller#2537

Jira: https://redhat.atlassian.net/browse/OCPBUGS-86842
Original bug: https://redhat.atlassian.net/browse/OCPBUGS-76453

openshift-ci-robot · 2026-06-01T23:29:26Z

@rashmigottipati: This pull request references Jira Issue OCPBUGS-86842, which is invalid:

expected Jira Issue OCPBUGS-86842 to depend on a bug targeting a version in 4.22.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Backport of OCPBUGS-76453 to release-4.21.

LocalDirV1.Store creates a temp dir (.{catalog}-{random}) and renames it into place atomically. If the process is interrupted before the deferred RemoveAll runs, the temp dir persists. Each restart adds another, eventually filling the disk.

Primary fix: remove any matching temp dirs at the start of Store before creating a new one. Since Store holds the write mutex, any dirs found are guaranteed to be from a previous run.

Defense-in-depth: add /var/cache/catalogs to GarbageCollector's list of scanned paths (CachePath string -> CachePaths []string), so orphaned temp dirs are also removed by the periodic GC cycle.

Original PR: operator-framework/operator-controller#2537

Jira: https://redhat.atlassian.net/browse/OCPBUGS-86842
Original bug: https://redhat.atlassian.net/browse/OCPBUGS-76453

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-01T23:29:27Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 835cdfa5-4eb5-4f35-bdaf-fea21680ba1f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

rashmigottipati · 2026-06-01T23:33:52Z

/jira refresh

openshift-ci-robot · 2026-06-01T23:33:56Z

@rashmigottipati: This pull request references Jira Issue OCPBUGS-86842, which is invalid:

expected dependent Jira Issue OCPBUGS-76453 to target a version in 4.22.0, but it targets "4.18.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rashmigottipati · 2026-06-01T23:36:37Z

/jira refresh

openshift-ci-robot · 2026-06-01T23:36:44Z

@rashmigottipati: This pull request references Jira Issue OCPBUGS-86842, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.z) matches configured target version for branch (4.21.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-76453 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-76453 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
bug has dependents

Requesting review from QA contact:
/cc @kuiwang02

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-06-01T23:36:47Z

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: kuiwang02.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

@rashmigottipati: This pull request references Jira Issue OCPBUGS-86842, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)

bug target version (4.21.z) matches configured target version for branch (4.21.z)

bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

release note text is set and does not match the template

dependent bug Jira Issue OCPBUGS-76453 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))

dependent Jira Issue OCPBUGS-76453 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0

bug has dependents

Requesting review from QA contact:
/cc @kuiwang02

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rashmigottipati · 2026-06-01T23:38:11Z

cc @tmshort

rashmigottipati · 2026-06-02T02:36:38Z

/retest-required

tmshort · 2026-06-02T14:07:43Z

Need to update the commit headline to include UPSTREAM: <carry>:

…log storage LocalDirV1.Store creates a temp dir (.{catalog}-{random}) and renames it into place atomically. If the process is interrupted before the deferred RemoveAll runs, the temp dir persists. Each restart adds another, eventually filling the disk. Primary fix: remove any matching temp dirs at the start of Store before creating a new one. Since Store holds the write mutex, any dirs found are guaranteed to be from a previous run. Defense-in-depth: add /var/cache/catalogs to GarbageCollector's list of scanned paths (CachePath string -> CachePaths []string), so orphaned temp dirs are also removed by the periodic GC cycle. Backport of OCPBUGS-76453 to release-4.21. Signed-off-by: Rashmi Gottipati <[email protected]>

rashmigottipati · 2026-06-02T14:31:30Z

@tmshort thanks, just updated the commit headline.

tmshort · 2026-06-02T16:05:51Z

/test e2e-gcp-ovn-upgrade

ankitathomas · 2026-06-02T16:52:10Z

/verified by @ankitathomas

verified orphaned cache directories get cleaned up by checking whether orphaned catalog directories created by failed unpack attempts of invalid clustercatalog images get removed on reconciliation

openshift-ci-robot · 2026-06-02T16:52:23Z

@ankitathomas: This PR has been marked as verified by @ankitathomas.

Details

In response to this:

/verified by @ankitathomas

verified orphaned cache directories get cleaned up by checking whether orphaned catalog directories created by failed unpack attempts of invalid clustercatalog images get removed on reconciliation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-06-02T18:09:24Z

@rashmigottipati: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

tmshort

/approve
/lgtm

Looks like a clean backport

openshift-ci · 2026-06-02T19:53:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rashmigottipati, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~DOWNSTREAM_OWNERS~~ [tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tmshort · 2026-06-02T19:53:54Z

/label backport-risk-assessed

openshift-ci · 2026-06-02T19:54:23Z

@tmshort: The label(s) backport-risk-assessed cannot be applied or removed, because you are not in one of the allowed teams and are not an allowed user. Must be a member of one of these teams: openshift-patch-managers, openshift-release-oversight, openshift-staff-engineers, openshift-sustaining-engineers

Details

In response to this:

/label backport-risk-assessed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

oceanc80

/label backport-risk-assessed

openshift-ci-robot · 2026-06-03T11:11:50Z

@rashmigottipati: Jira Issue Verification Checks: Jira Issue OCPBUGS-86842
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-86842 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Backport of OCPBUGS-76453 to release-4.21.

LocalDirV1.Store creates a temp dir (.{catalog}-{random}) and renames it into place atomically. If the process is interrupted before the deferred RemoveAll runs, the temp dir persists. Each restart adds another, eventually filling the disk.

Primary fix: remove any matching temp dirs at the start of Store before creating a new one. Since Store holds the write mutex, any dirs found are guaranteed to be from a previous run.

Defense-in-depth: add /var/cache/catalogs to GarbageCollector's list of scanned paths (CachePath string -> CachePaths []string), so orphaned temp dirs are also removed by the periodic GC cycle.

Original PR: operator-framework/operator-controller#2537

Jira: https://redhat.atlassian.net/browse/OCPBUGS-86842
Original bug: https://redhat.atlassian.net/browse/OCPBUGS-76453

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rashmigottipati · 2026-06-03T11:56:29Z

/cherrypick release-4.20

rashmigottipati · 2026-06-03T11:58:46Z

/cherry-pick release-4.20

openshift-cherrypick-robot · 2026-06-03T11:59:42Z

@rashmigottipati: new pull request created: #744

Details

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-merge-robot · 2026-06-04T02:24:34Z

Fix included in release 4.21.0-0.nightly-2026-06-03-195808

openshift-ci Bot requested review from anik120 and pedjak June 1, 2026 23:29

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 1, 2026

rashmigottipati force-pushed the OCPBUGS-86842-backport-4.21 branch from 2c477cc to 6b9588f Compare June 2, 2026 14:29

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 2, 2026

tmshort approved these changes Jun 2, 2026

View reviewed changes

openshift-ci Bot assigned tmshort Jun 2, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 2, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 2, 2026

oceanc80 reviewed Jun 3, 2026

View reviewed changes

openshift-ci Bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jun 3, 2026

openshift-ci Bot assigned bandrade Jun 3, 2026

openshift-merge-bot Bot merged commit 834ed2c into openshift:release-4.21 Jun 3, 2026
13 checks passed

openshift-cherrypick-robot mentioned this pull request Jun 3, 2026

[release-4.20] OCPBUGS-86968: catalogd: clean up orphaned temp dirs in catalog storage #744

Merged

rashmigottipati mentioned this pull request Jun 5, 2026

OCPBUGS-87180: [release-4.19] catalogd: clean up orphaned temp dirs in catalog storage #748

Open

Conversation

rashmigottipati commented Jun 1, 2026

Uh oh!

openshift-ci-robot commented Jun 1, 2026

Uh oh!

coderabbitai Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

rashmigottipati commented Jun 1, 2026

Uh oh!

openshift-ci-robot commented Jun 1, 2026

Uh oh!

rashmigottipati commented Jun 1, 2026

Uh oh!

openshift-ci-robot commented Jun 1, 2026

Uh oh!

openshift-ci Bot commented Jun 1, 2026

Uh oh!

rashmigottipati commented Jun 1, 2026

Uh oh!

rashmigottipati commented Jun 2, 2026

Uh oh!

tmshort commented Jun 2, 2026

Uh oh!

rashmigottipati commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmshort commented Jun 2, 2026

Uh oh!

ankitathomas commented Jun 2, 2026

Uh oh!

openshift-ci-robot commented Jun 2, 2026

Uh oh!

openshift-ci Bot commented Jun 2, 2026

Uh oh!

tmshort left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jun 2, 2026

Uh oh!

tmshort commented Jun 2, 2026

Uh oh!

openshift-ci Bot commented Jun 2, 2026

Uh oh!

oceanc80 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 3, 2026

Uh oh!

rashmigottipati commented Jun 3, 2026

Uh oh!

rashmigottipati commented Jun 3, 2026

Uh oh!

openshift-cherrypick-robot commented Jun 3, 2026

Uh oh!

openshift-merge-robot commented Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

coderabbitai Bot commented Jun 1, 2026 •

edited

Loading

rashmigottipati commented Jun 2, 2026 •

edited

Loading