OCPBUGS-89231: version-gate TestPullSecretUnavailable to 4.23+

[bryan-cox]

What this PR does / why we need it:

Adds e2eutil.AtLeast(t, e2eutil.Version423) to TestPullSecretUnavailable so it skips on release branches older than 4.23.

The fix for OCPBUGS-77268 (PR #8352) is on main and release-4.23 but not on release-4.22 or older. Since hypershift-tests:latest is built from main and used by release branch CI jobs, the test runs against operators that don't have the fix — causing permafailures on release-4.22 periodic jobs.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-89231

Special notes for your reviewer:

The test was introduced in #8352 alongside the operator fix. Since the fix was not backported to release-4.22 (and no backport is planned per OCPBUGS-77268), the test needs to be gated to avoid running against operators that lack the fix.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
  • Updated test infrastructure requirements to ensure compatibility with newer e2e utilities.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@bryan-cox: This pull request references Jira Issue OCPBUGS-89231, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Adds e2eutil.AtLeast(t, e2eutil.Version422) to TestPullSecretUnavailable so it skips on release branches older than 4.22.

This test was introduced in #8352 alongside the operator fix for OCPBUGS-77268. Since hypershift-tests:latest is built from main and used by release branch CI jobs, the test runs against operators that don't have the fix — causing permafailures on release-4.22 (and older) periodic jobs.

The cherry-pick of the operator fix to release-4.22 is in #8743.

Bug: https://redhat.atlassian.net/browse/OCPBUGS-89231

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 42260c39-7b10-485b-9b05-e4427319ceb5

📥 Commits

Reviewing files that changed from the base of the PR and between 926d030 and 1ceddbc.

📒 Files selected for processing (1)

• test/e2e/chaos_test.go

✅ Files skipped from review due to trivial changes (1)

• test/e2e/chaos_test.go

---

📝 Walkthrough

Walkthrough

A single version prerequisite check is added to the TestPullSecretUnavailable e2e test in test/e2e/chaos_test.go. The call e2eutil.AtLeast(t, e2eutil.Version423) is inserted at the beginning of the test function, gating its execution on the minimum supported version Version423.

Suggested reviewers

• sjenning
• sdminonne

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding a version gate to TestPullSecretUnavailable test for version 4.23+, which matches the file change and the PR's primary objective.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The test file uses standard Go testing (not Ginkgo). TestPullSecretUnavailable uses only static test names and string values; no dynamic information appears in test titles or naming patterns.
Test Structure And Quality	✅ Passed	TestPullSecretUnavailable demonstrates all required quality standards: single responsibility (validates pull secret unavailability doesn't break field propagation), proper cleanup via t.Cleanup(),...
Topology-Aware Scheduling Compatibility	✅ Passed	The PR modifies only test code (test/e2e/chaos_test.go) by adding a version gate. The check applies to deployment manifests, operator code, or controllers—none of which are modified here. No schedu...
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Custom check targets new Ginkgo e2e tests being added; this PR only adds a version gate to an existing test, so the check is not applicable.
No-Weak-Crypto	✅ Passed	The PR modifies only test/e2e/chaos_test.go to add a version gate. No weak cryptography (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or insecure secret comparisons are...
Container-Privileges	✅ Passed	This PR modifies test/e2e/chaos_test.go (a Go test file), not container/K8s manifests. The container-privileges check is not applicable.
No-Sensitive-Data-In-Logs	✅ Passed	The PR only adds a version gate e2eutil.AtLeast(t, e2eutil.Version423) to skip the test on older versions. No new logging is introduced. Existing logging statements in the test do not expose pass...

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.75%. Comparing base (392fd5a) to head (1ceddbc).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8744   +/-   ##
=======================================
  Coverage   41.75%   41.75%           
=======================================
  Files         758      758           
  Lines       93981    93981           
=======================================
  Hits        39240    39240           
  Misses      51988    51988           
  Partials     2753     2753           

Flag	Coverage Δ
cmd-support	35.02% <ø> (ø)
cpo-hostedcontrolplane	44.10% <ø> (ø)
cpo-other	43.45% <ø> (ø)
hypershift-operator	51.82% <ø> (ø)
other	31.56% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bryan-cox]

/jira refresh

[openshift-ci-robot]

@bryan-cox: This pull request references Jira Issue OCPBUGS-89231, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[cblecker]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[bryan-cox]

/verified by e2e

[openshift-ci-robot]

@bryan-cox: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2066954719347085312 | Cost: $2.82267525 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[cwbotbot]

Test Results

e2e-aws

• Status: ✅ PASS
• Started: 2026-06-16T23:23:27Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-06-16T21:14:37Z
• View Job
• View Job History

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2066954719288365056 | Cost: $2.666706750000001 | Failed step: hypershift-azure-run-e2e

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[bryan-cox]

/retest

[hypershift-jira-solve-ci]

I now have all the evidence needed. Both failures are identical CI infrastructure issues. Here is the report:

Test Failure Analysis Complete

Job Information

• Prow Job 1: pull-ci-openshift-hypershift-main-e2e-v2-aws
• Build ID 1: 2066992828596097024
• Prow Job 2: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID 2: 2066992828474462208
• Build Cluster: build01
• Namespace: ci-op-1kl8yl2q (shared by both jobs)
• Start Time: 2026-06-16T21:14:27Z
• Failure Time: 2026-06-16T21:16:42Z
• PR: OCPBUGS-89231: version-gate TestPullSecretUnavailable to 4.23+ #8744 (OCPBUGS-89231: version-gate TestPullSecretUnavailable to 4.23+)

Test Failure Analysis

Error

e2e-v2-aws: could not run steps: step e2e-v2-aws failed: import secret cluster-secrets-hypershift-aws
  for cluster profile hypershift-aws: could not update secret cluster-secrets-hypershift-aws:
  secrets "cluster-secrets-hypershift-aws" not found

e2e-aws: could not run steps: step e2e-aws failed: import secret cluster-secrets-hypershift-aws
  for cluster profile hypershift-aws: could not update secret cluster-secrets-hypershift-aws:
  delete failed: secrets "cluster-secrets-hypershift-aws" not found

Summary

Both jobs failed due to a CI infrastructure issue — not a product or test code defect. The cluster-secrets-hypershift-aws Kubernetes Secret, which provides AWS credentials for the hypershift-aws cluster profile, was not found on the build01 cluster when ci-operator attempted to import it into the test namespace ci-op-1kl8yl2q. The failure occurred immediately upon attempting to set up the multi-stage test (within ~0.1 seconds of the step starting), before any test code could execute. Both jobs ran in the same ci-operator namespace at the same time, confirming a shared infrastructure dependency failure. The PR changes (version-gating TestPullSecretUnavailable to 4.23+) are completely unrelated to this failure.

Root Cause

The CI infrastructure secret cluster-secrets-hypershift-aws was temporarily unavailable on the build01 cluster. This secret is managed by the ci-secret-bootstrap controller (as indicated by the label dptp.openshift.io/requester: ci-secret-bootstrap in the step-graph manifest metadata) and provides AWS credentials needed for the hypershift-aws cluster profile.

When ci-operator starts a multi-stage test, it imports the cluster profile secret from the ci namespace into the test namespace. In this case, the secret did not exist at all in the target namespace — one job got a "not found" error on update, the other got a "delete failed: not found" (slightly different code paths, same root cause).

This is a transient CI infrastructure failure. The secret is centrally managed and periodically synced; a temporary gap in availability caused both jobs to fail simultaneously. This is not caused by the PR under test.

Recommendations

1. Retest the PR — simply re-trigger both jobs with /retest or /test e2e-aws and /test e2e-v2-aws. The infrastructure issue is transient and the PR changes (version-gating a test) cannot have caused this failure.
2. If retests continue to fail with the same error, escalate to the DPTP/CI team (#forum-ocp-testplatform on Slack) as the ci-secret-bootstrap controller may have a persistent issue syncing the cluster-secrets-hypershift-aws secret to build01.
3. No code changes needed — the PR (OCPBUGS-89231) only modifies test version-gating logic and has no relationship to AWS credential provisioning.

Evidence

Evidence	Detail
Error (e2e-v2-aws)	secrets "cluster-secrets-hypershift-aws" not found during secret import
Error (e2e-aws)	delete failed: secrets "cluster-secrets-hypershift-aws" not found during secret import
Step duration	~0.1 seconds — failed immediately, no test code executed
Build cluster	build01
Shared namespace	Both jobs ran in ci-op-1kl8yl2q simultaneously
Job start time	Both started at 2026-06-16T21:14:27Z
Failure reason	executing_graph:step_failed:utilizing_lease (infra-level failure during lease utilization)
Secret manager	dptp.openshift.io/requester: ci-secret-bootstrap (centrally managed)
Cluster profile	hypershift-aws — requires cluster-secrets-hypershift-aws secret
PR relevance	PR #8744 only version-gates TestPullSecretUnavailable — no infra/credential changes

---

[bryan-cox]

/retest

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 03f89c3 and 2 for PR HEAD 1ceddbc in total

[bryan-cox]

/override ci/prow/e2e-kubevirt-aws-ovn-reduced

This is known to be permafailing.

[openshift-ci]

@bryan-cox: Overrode contexts on behalf of bryan-cox: ci/prow/e2e-kubevirt-aws-ovn-reduced

Details

In response to this:

/override ci/prow/e2e-kubevirt-aws-ovn-reduced

This is known to be permafailing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@bryan-cox: Jira Issue Verification Checks: Jira Issue OCPBUGS-89231
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-89231 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

What this PR does / why we need it:

Adds e2eutil.AtLeast(t, e2eutil.Version423) to TestPullSecretUnavailable so it skips on release branches older than 4.23.

The fix for OCPBUGS-77268 (PR #8352) is on main and release-4.23 but not on release-4.22 or older. Since hypershift-tests:latest is built from main and used by release branch CI jobs, the test runs against operators that don't have the fix — causing permafailures on release-4.22 periodic jobs.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-89231

Special notes for your reviewer:

The test was introduced in #8352 alongside the operator fix. Since the fix was not backported to release-4.22 (and no backport is planned per OCPBUGS-77268), the test needs to be gated to avoid running against operators that lack the fix.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
• Updated test infrastructure requirements to ensure compatibility with newer e2e utilities.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.