capi dump 1.11 (2)

[LiangquanLi930]

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• New Features

  • Added an optional Initialization status field to hosted control plane status to expose control-plane initialization state.

• Chores

  • Adjusted lint rules to ignore a specific deprecated package pattern and one pointer/bool lint for a status field.
  • Narrowed scope for generated build artifacts to reduce excess outputs.
  • Bumped several indirect module dependencies to newer compatible versions.
  • Minor file formatting fix to add a trailing newline.

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

• do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 37b8ea69-577b-45fe-9534-3503139c2294

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This pull request makes several small changes: adds a staticcheck exclusion (SA1019) for sigs.k8s.io/cluster-api/(.*) in the repo .golangci.yml; adds a terminating newline to a LABEL in the Dockerfile; narrows the cluster-api Makefile target to generate CRDs from .../api/core/... and .../api/ipam/... (retaining .../api/addons/...); updates indirect module versions and removes some unused indirect deps in api/go.mod; adds an Initialization field to HostedControlPlaneStatus and a new HostedControlPlaneInitializationStatus type in api/hypershift/v1beta1/hosted_controlplane.go; and adds a kubeapilinter nobools exclusion for that file in api/.golangci.yml.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: LiangquanLi930
Once this PR has been reviewed and has the lgtm label, please assign csrwng for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• api/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[LiangquanLi930]

/test e2e-aws e2e-aks

[cwbotbot]

Test Results

e2e-aks

• Status: ❌ FAIL
• Started: 2026-03-31T18:38:20Z
• View Job
• View Job History

Failed Tests

Total failed tests: 5

• TestAutoscaling
• TestAutoscaling/Main
• TestAutoscaling/Main/TestAutoscaling
• TestAutoscaling/Main/TestAutoscalingBalancing
• TestAutoscaling/Teardown

e2e-aws

• Status: ✅ PASS
• Started: 2026-03-31T08:49:45Z
• View Job
• View Job History

[LiangquanLi930]

/test e2e-aws e2e-aks

[LiangquanLi930]

/test e2e-aws e2e-aks

[LiangquanLi930]

/test e2e-aws e2e-aks

[LiangquanLi930]

/test e2e-aws e2e-aks

[LiangquanLi930]

/test e2e-aws e2e-aks

[LiangquanLi930]

/test e2e-aws e2e-aks

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 26.83%. Comparing base (9fda149) to head (bf282c1).
⚠️ Report is 54 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8105      +/-   ##
==========================================
+ Coverage   26.72%   26.83%   +0.11%     
==========================================
  Files        1090     1091       +1     
  Lines      105340   105306      -34     
==========================================
+ Hits        28155    28264     +109     
+ Misses      74760    74611     -149     
- Partials     2425     2431       +6     

Files with missing lines	Coverage Δ
.../v1beta1/hostedcontrolplaneinitializationstatus.go	0.00% <ø> (ø)
...ion/hypershift/v1beta1/hostedcontrolplanestatus.go	0.00% <ø> (ø)
client/applyconfiguration/utils.go	0.00% <ø> (ø)
cmd/cluster/core/dump.go	4.27% <ø> (ø)

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LiangquanLi930]

https://github.com/coderabbitai can you help analyze why this job failed?

[LiangquanLi930]

@coderabbitai can you help analyze why this job failed?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Get PR comments for more details about the failure
gh pr view 8105 --repo openshift/hypershift --json comments --jq '.comments[-10:] | .[] | {author: .author.login, body: .body[0:500]}'

Length of output: 1677

---

🏁 Script executed:

#!/bin/bash
# Try to fetch the failed job details from the testgrid URL
curl -s "https://testgrid-ci-testgrid.apps.rosa.hypershift-ci-2.1xls.p3.openshiftapps.com/?job=2038689692886503424&testName=e2e-aks" | head -200 || echo "Could not access URL"

# Try to get the Prow job details via GitHub checks API
gh api /repos/openshift/hypershift/commits/$(gh pr view 8105 --repo openshift/hypershift --json headRefOid --jq '.headRefOid')/check-runs --jq '.check_runs[] | select(.name | contains("aks")) | {name, status, conclusion, html_url, details_url}' 2>/dev/null || echo "No check runs found via API"

Length of output: 6147

---

🏁 Script executed:

#!/bin/bash
# Get the head SHA of the PR
SHA=$(gh pr view 8105 --repo openshift/hypershift --json headRefOid --jq '.headRefOid')
echo "Head SHA: $SHA"

# Try to get check runs
gh api "/repos/openshift/hypershift/commits/$SHA/check-runs" --jq '.check_runs[] | {name, status, conclusion, details_url}' 2>/dev/null || echo "Failed to get check runs"

# Try to get PR statuses
gh api "/repos/openshift/hypershift/commits/$SHA/statuses" --jq '.[] | select(.context | contains("aks")) | {context, state, description, target_url}' 2>/dev/null | head -50 || echo "No statuses found"

Length of output: 6956

---

🏁 Script executed:

#!/bin/bash
# Search for TestAutoscaling in the codebase to understand what it tests
rg -n "TestAutoscaling" --type go -l | head -20

Length of output: 50

---

🏁 Script executed:

#!/bin/bash
# Find the TestAutoscaling test implementation
rg -n "func TestAutoscaling" --type go -A 5

Length of output: 308

---

🏁 Script executed:

#!/bin/bash
# Look for TestAutoscalingBalancing as well
rg -n "TestAutoscalingBalancing" --type go -A 10

Length of output: 1946

---

🏁 Script executed:

#!/bin/bash
# Check if there are any AKS-specific changes in this PR that could affect autoscaling
git diff origin/main...HEAD --name-only | grep -i "aks\|azure\|autoscal" | head -30

# Also check what files changed
git diff origin/main...HEAD --name-only | head -60

Length of output: 6562

---

🏁 Script executed:

#!/bin/bash
# Fetch the Prow job logs for the failed e2e-aks job
curl -s "https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/8105/pull-ci-openshift-hypershift-main-e2e-aks/2038901436632272896" | grep -i "fail\|error\|panic\|autoscal" | head -50 || echo "Could not parse page"

# Try to get the raw build log from GCS
curl -s "https://storage.googleapis.com/test-platform-results/pr-logs/pull/openshift_hypershift/8105/pull-ci-openshift-hypershift-main-e2e-aks/2038901436632272896/build-log.txt" 2>/dev/null | grep -A 10 -B 2 "autoscal\|FAIL\|panic\|Error\|timeout" | head -200 || echo "Could not access build log"

Length of output: 40461

---

🏁 Script executed:

#!/bin/bash
# Look at the autoscaling_test.go changes in this PR
git diff origin/main...HEAD -- test/e2e/autoscaling_test.go

Length of output: 6455

---

🏁 Script executed:

#!/bin/bash
# Look at the full autoscaling test file
cat test/e2e/autoscaling_test.go

Length of output: 29422

---

🏁 Script executed:

#!/bin/bash
# Check nodepool controller changes related to autoscaling and MachineDeployment readiness
git diff origin/main...HEAD -- hypershift-operator/controllers/nodepool/ | grep -A 10 -B 5 "autoscal\|UpToDateReplicas\|MachinesReady\|ReadyReplicas\|MachineDeploymentComplete" | head -200

Length of output: 8316

---

@LiangquanLi930 Here's my analysis of the e2e-aks failure:

---

e2e-aks Failure Analysis

Failed Tests

All 13 failures are within the TestAutoscaling tree (including TestAutoscalingBalancing, subtests, and teardown), pointing to a single root cause rather than multiple independent failures.

What the Logs Show

The build log shows transient TLS issues at startup (net/http: TLS handshake timeout, EOF) for multiple clusters, but TestAutoscaling did successfully connect to the guest API server:

util.go:375: Successfully waited for a successful connection to the guest API server in 48.475s

So the failure happened after cluster creation, during the actual autoscaling validation.

Most Likely Root Causes (CAPI v1beta2 Changes)

This PR introduces several CAPI v1beta2 behavioral changes that interact with the AKS platform:

1. 🔴 ReadyCondition → MachinesReadyCondition

// Before (v1beta1):
if c.Type == capiv1.ReadyCondition {

// After (v1beta2):
if c.Type == capiv1.MachinesReadyCondition {

If the Azure CAPI provider does not set MachinesReadyCondition in its v1beta2 implementation (or sets it differently from AWS), the NodePoolReadyConditionType will never become True. This would cause testAutoscaling to stall waiting for nodes to be ready.

2. 🔴 AvailableReplicas is now a Pointer (*int32)

// Before:
nodePool.Status.Replicas = machineDeployment.Status.AvailableReplicas

// After:
nodePool.Status.Replicas = ptr.Deref(machineDeployment.Status.AvailableReplicas, 0)

If the Azure provider does not populate AvailableReplicas in v1beta2 (leaving it nil), NodePool.Status.Replicas stays at 0. This would break the balancing check in testAutoscalingBalancing:

if nps[0].Status.Replicas + nps[1].Status.Replicas != 6 { // would always be 0+0=0 }

3. 🟡 MachineDeploymentComplete() — MachineSet-based Check

// New signature - now inspects owned MachineSets for spec comparison
func MachineDeploymentComplete(deployment *capiv1.MachineDeployment, machineSets []*capiv1.MachineSet) bool {

The new logic compares Bootstrap.DataSecretName and Version on owned MachineSets. If Azure's CAPI MachineSet reconciliation is slower or behaves differently with v1beta2, this check may timeout.

Recommendation

The suggested investigation steps:

1. Check Azure CAPI provider v1beta2 support — verify that MachinesReadyCondition is actually being set on MachineDeployment and MachineSet objects in the AKS environment.
2. Check AvailableReplicas population — confirm the Azure provider sets Status.AvailableReplicas (non-nil) in v1beta2.
3. You can add a diagnostic log in reconcileMachineDeployment to dump the full machineDeployment.Status.Conditions slice on AKS to confirm which conditions are present.

The e2e-aws test passing while e2e-aks fails strongly suggests an Azure-specific CAPI provider gap with the new v1beta2 API surface.

[LiangquanLi930]

/test e2e-aks

[LiangquanLi930]

/test e2e-azure-self-managed

[LiangquanLi930]

/test e2e-aws-4-21

[LiangquanLi930]

/test e2e-v2-aws e2e-aws-upgrade-hypershift-operator

[LiangquanLi930]

/test e2e-aws e2e-aks

[LiangquanLi930]

/test e2e-aws-4-21

[openshift-ci]

@LiangquanLi930: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-upgrade-hypershift-operator	bf282c1	link	true	/test e2e-aws-upgrade-hypershift-operator
ci/prow/e2e-aws	03b67b5	link	true	/test e2e-aws
ci/prow/e2e-aws-4-21	03b67b5	link	true	/test e2e-aws-4-21
ci/prow/e2e-aks	03b67b5	link	true	/test e2e-aks

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

Stale PRs are closed after 21d of inactivity.

If this PR is still relevant, comment to refresh it or remove the stale label.
Mark the PR as fresh by commenting /remove-lifecycle stale.

If this PR is safe to close now please do so with /close.

/lifecycle stale

[hypershift-jira-solve-ci]

Now I have all the evidence. Let me compile the final report.

Test Failure Analysis Complete

Job Information

• Prow Jobs: pull-ci-openshift-hypershift-main-e2e-aws, pull-ci-openshift-hypershift-main-e2e-aks, Unit Tests (GitHub Actions)
• Build IDs: 2040122503912755200 (e2e-aws), 2040122503866617856 (e2e-aks), 23953116421 (Unit Tests)
• PR: capi dump 1.11 (2) #8105 — "capi dump 1.11 (2)" by LiangquanLi930 (branch: pr-7590)
• PR HEAD: 03b67b5bc94bbff7d6e35bd38a89b40567e768b4

Test Failure Analysis

Error

--- FAIL: TestSetupCRDs/When_is_TechPreviewNoUpgrade_it_should_have_a_single_nodepool_CRD_with_the_TechPreviewNoUpgrade_annotation (0.01s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x264e6d0]

goroutine 115 [running]:
github.com/openshift/hypershift/cmd/install.setupCRDs.func2(0xc000df8008)
    cmd/install/install.go:891 +0x4f0

Summary

All five CI failures on PR #8105 stem from the same root cause: the PR introduces CAPI v1beta1→v1beta2 conversion webhook support in setupCRDs() (cmd/install/install.go), adding a new CAPICRDOverrides map that triggers conversion webhook setup for CAPI CRDs. This new code path dereferences operatorService.Name (line 891) without a nil guard. The TestSetupCRDs unit test calls setupCRDs() with operatorService=nil (line 317 of install_test.go), causing a nil pointer panic that fails Unit Tests directly and blocks the Konflux build. The e2e-aws job fails because the hosted-cluster-config-operator pod crash-loops with failed to wait for hcpstatus caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.HostedControlPlane, and the e2e-aks job times out at 2 hours with nodes failing to become ready and multiple control-plane operators (cluster-network-operator, hosted-cluster-config-operator, ingress-operator) showing unavailable replicas — both consistent with CAPI API version transition issues introduced by this PR.

Root Cause

The PR adds CAPI v1beta1→v1beta2 conversion webhook logic in cmd/install/install.go. Specifically:

1. New code path in setupCRDs.func2 (lines 863-888): A new else if branch checks assets.CAPICRDOverrides[crd.Name] for CAPI CRDs (e.g., machinedeployments.cluster.x-k8s.io, clusters.cluster.x-k8s.io). When a match is found with NeedsConversion: true, the code falls through to set up the conversion webhook at line 891:

   Name: operatorService.Name,  // line 891 — nil pointer dereference

2. Missing nil guard: The operatorService parameter can be nil (as passed by the test at install_test.go:317), but the new CAPI conversion path does not check for nil before accessing operatorService.Name. The existing HyperShift conversion path (line 857) had the same issue but was gated by opts.EnableConversionWebhook which defaults to false in tests.

3. Why this breaks now: Previously, only HyperShift CRDs (hypershift.openshift.io group) triggered the conversion webhook code, and that path was gated by opts.EnableConversionWebhook=true. The new CAPI override path (CAPICRDOverrides) is always active — it runs whenever a CAPI CRD name matches the map, regardless of the EnableConversionWebhook flag. Since the test doesn't set operatorService, the nil dereference occurs.

4. E2e failures: The e2e-aws failure shows hosted-cluster-config-operator crash-looping with timed out waiting for cache to be synced for Kind *v1beta1.HostedControlPlane, indicating the CAPI v1beta1→v1beta2 transition has runtime implications beyond the install path. The e2e-aks failure shows similar symptoms with multiple operators (CNO, ingress-operator, hosted-cluster-config-operator) failing to become available and nodes not joining the cluster.

Recommendations

1. Fix the nil pointer dereference: Add a nil guard for operatorService before the webhook setup block in setupCRDs.func2:

   if !needsConversion || operatorService == nil {
       return
   }

   Or alternatively, pass a non-nil operatorService in the test.

2. Update the test: Modify install_test.go:317 to pass a non-nil *corev1.Service object:

   crds, err := setupCRDs(t.Context(), nil, tc.inputOptions, &corev1.Namespace{}, &corev1.Service{
       ObjectMeta: metav1.ObjectMeta{Name: "test-operator"},
   })

3. Investigate the runtime CAPI v1beta1→v1beta2 cache sync failure: The hosted-cluster-config-operator crash-loop (timed out waiting for cache to be synced for Kind *v1beta1.HostedControlPlane) in the e2e-aws job suggests a deeper compatibility issue with the CAPI version transition that needs investigation beyond the install code fix.

4. Add test coverage for the CAPI conversion path: Ensure tests exercise the CAPICRDOverrides code path with both nil and non-nil operatorService to catch regressions.

Evidence

Evidence	Detail
Unit Test panic	install.go:891 — operatorService.Name dereferences nil; called from install_test.go:317 with setupCRDs(ctx, nil, opts, &Namespace{}, nil)
Stack trace	setupCRDs.func2 → assets.CustomResourceDefinitions.func1 → fs.WalkDir — crashes while iterating CRD manifests
New code path	CAPICRDOverrides map in assets/assets.go:37-52 lists 12 CAPI CRDs with NeedsConversion: true, always active (not gated by EnableConversionWebhook)
e2e-aws failure	TestUpgradeControlPlane/ValidateHostedCluster — hosted-cluster-config-operator CrashLoopBackOff: failed to wait for hcpstatus caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.HostedControlPlane
e2e-aks failure	7 test failures, 2h timeout — nodes not ready (0/2, 1/3, 2/6), operators unavailable: CNO, ingress-operator, hosted-cluster-config-operator, cluster-autoscaler
e2e-aws degraded conditions	Degraded=True (hosted-cluster-config-operator 1 unavailable replica), ClusterVersionAvailable=False, upgrade stuck at 81% (564/693 operators)
Konflux failure	control-plane-operator-main-on-pull-request — build failure, blocked by same code defect
Tide error	Merge blocked due to failing required checks
PR context	Title: "capi dump 1.11 (2)" — CAPI v1beta1→v1beta2 transition, modifies hosted_controlplane.go, generated CRDs, vendor deps

---

[openshift-ci]

Stale PRs rot after 14d of inactivity.

Mark the PR as fresh by commenting /remove-lifecycle rotten.
Rotten PRs close after an additional 7d of inactivity.

If this PR is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-ci]

Rotten PRs close after 7d of inactivity.

Reopen the PR by commenting /reopen.
Mark the PR as fresh by commenting /remove-lifecycle rotten.

/close

[openshift-ci]

@openshift-ci[bot]: Closed this PR.

Details

In response to this:

Rotten PRs close after 7d of inactivity.

Reopen the PR by commenting /reopen.
Mark the PR as fresh by commenting /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[clebs]

Superseded by #8717