Skip to content

OCPBUGS-87957: Declare KubeStateMetricsTimezonePanic risk for 4.20.z#10074

Open
shahsahil264 wants to merge 1 commit into
openshift:masterfrom
shahsahil264:OCPBUGS-87957-kube-state-metrics-timezone-panic
Open

OCPBUGS-87957: Declare KubeStateMetricsTimezonePanic risk for 4.20.z#10074
shahsahil264 wants to merge 1 commit into
openshift:masterfrom
shahsahil264:OCPBUGS-87957-kube-state-metrics-timezone-panic

Conversation

@shahsahil264

Copy link
Copy Markdown

Summary

Adds conditional update risk for all 4.20.z GA releases (4.20.0 through 4.20.30) to warn clusters upgrading from 4.19.z that kube-state-metrics may crash loop if CronJobs with .spec.timeZone are present.

Root cause: /usr/share/zoneinfo/ is missing from the kube-state-metrics container image. The statically-compiled Go binary panics when time.LoadLocation() fails for timezone-aware CronJobs.

  • Affected: 4.20.z, 4.21.z, 4.22.z (confirmed by Benjamin Affolter and Simon Pasquier)
  • Not affected: 4.19.z
  • Blocked-edges for 4.20.z only: upgrading from 4.20→4.21 or 4.21→4.22 does not increase exposure (affected→affected)
  • No fixedIn: fix PRs openshift/kube-state-metrics#147 and #150 still open
  • type: Always: no PromQL metric for CronJob timezone usage

Bug

Test plan

  • ./hack/validate-blocked-edges.py passes

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 22, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@shahsahil264: This pull request references Jira Issue OCPBUGS-87957, which is invalid:

  • expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.22.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Adds conditional update risk for all 4.20.z GA releases (4.20.0 through 4.20.30) to warn clusters upgrading from 4.19.z that kube-state-metrics may crash loop if CronJobs with .spec.timeZone are present.

Root cause: /usr/share/zoneinfo/ is missing from the kube-state-metrics container image. The statically-compiled Go binary panics when time.LoadLocation() fails for timezone-aware CronJobs.

  • Affected: 4.20.z, 4.21.z, 4.22.z (confirmed by Benjamin Affolter and Simon Pasquier)
  • Not affected: 4.19.z
  • Blocked-edges for 4.20.z only: upgrading from 4.20→4.21 or 4.21→4.22 does not increase exposure (affected→affected)
  • No fixedIn: fix PRs openshift/kube-state-metrics#147 and #150 still open
  • type: Always: no PromQL metric for CronJob timezone usage

Bug

Test plan

  • ./hack/validate-blocked-edges.py passes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 247bc3db-5f39-4efc-af12-998f6404b4b1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@shahsahil264

Copy link
Copy Markdown
Author

@wking Can you PTAL!

Thanks

@openshift-ci
openshift-ci Bot requested review from DavidHurta and hongkailiu July 22, 2026 16:56
@openshift-ci

openshift-ci Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shahsahil264
Once this PR has been reviewed and has the lgtm label, please assign davidhurta for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sdodson

sdodson commented Jul 22, 2026

Copy link
Copy Markdown
Member

/hold

I don't want us to block all updates to 4.20 until such a time that we fix this. Please look closer at 4.20.28 and 4.20.29, /usr/share/zoneinfo exists in the former but not the latter. Why?

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 22, 2026
@sdodson
sdodson self-requested a review July 22, 2026 21:17

@sdodson sdodson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…d versions

Adds conditional update risk for versions where /usr/share/zoneinfo is
missing from the kube-state-metrics container image: 4.20.29-30,
4.21.24-25, and 4.22.0-6. Clusters upgrading from unaffected versions
with CronJobs using .spec.timeZone will experience kube-state-metrics
crash loops, breaking monitoring and potentially stalling upgrades.

First affected versions verified by Scott Dodson via rpm -Va tzdata:
4.20.29, 4.21.24, 4.22.0. Earlier versions (4.20.28, 4.21.23) have
zoneinfo present and are not affected.

Bug: https://redhat.atlassian.net/browse/OCPBUGS-87957
Impact: https://redhat.atlassian.net/browse/MON-4619

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Signed-off-by: Sahil Shah <[email protected]>
@shahsahil264
shahsahil264 force-pushed the OCPBUGS-87957-kube-state-metrics-timezone-panic branch from fe6078e to 890e4f3 Compare July 23, 2026 14:23
@shahsahil264

Copy link
Copy Markdown
Author

Updated, narrowed to 4.20.29-30, 4.21.24-25, 4.22.0-6 per your findings
Thanks!

@openshift-ci

openshift-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@shahsahil264: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@shahsahil264
shahsahil264 requested a review from sdodson July 23, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants