feat(upgrade): add v1.9 upgrade known issue - #1120
Open
starbops wants to merge 1 commit into
Open
Conversation
Signed-off-by: Zespre Chang <[email protected]>
|
|
|
||
| ### 1. Upgrade Is Stuck in "Images preloaded" | ||
|
|
||
| In rare cases, an upgrade from v1.8.x to v1.9.0 may remain in the **Images preloaded** state after the operating system is upgraded on a node. The upgrade controller marks the node `Succeeded` before removing the `harvesterhci.io/pendingOSImage` annotation. |
Contributor
There was a problem hiding this comment.
Suggested change
| In rare cases, an upgrade from v1.8.x to v1.9.0 may remain in the **Images preloaded** state after the operating system is upgraded on a node. The upgrade controller marks the node `Succeeded` before removing the `harvesterhci.io/pendingOSImage` annotation. | |
| In rare cases, an upgrade from v1.8.x to v1.9.0 may remain in the `Images preloaded` state after a node reboots into the new operating system. This occurs when the upgrade controller changes the node's upgrade state to `Succeeded` before removing the `harvesterhci.io/pendingOSImage` annotation. |
Comment on lines
+29
to
+37
| Check the latest Upgrade CR: | ||
|
|
||
| ```bash | ||
| kubectl -n harvester-system get upgrades.harvesterhci.io \ | ||
| -l harvesterhci.io/latestUpgrade=true \ | ||
| -o yaml | ||
| ``` | ||
|
|
||
| When the upgrade has stopped progressing, this issue is present if the output shows an upgraded node with `state: Succeeded` while the remaining nodes are still in the `Images preloaded` state. |
Contributor
There was a problem hiding this comment.
Suggested change
| Check the latest Upgrade CR: | |
| ```bash | |
| kubectl -n harvester-system get upgrades.harvesterhci.io \ | |
| -l harvesterhci.io/latestUpgrade=true \ | |
| -o yaml | |
| ``` | |
| When the upgrade has stopped progressing, this issue is present if the output shows an upgraded node with `state: Succeeded` while the remaining nodes are still in the `Images preloaded` state. | |
| 1. Check the latest `Upgrade` CR: | |
| ```bash | |
| kubectl -n harvester-system get upgrades.harvesterhci.io \ | |
| -l harvesterhci.io/latestUpgrade=true \ | |
| -o yaml | |
| ``` | |
| When the upgrade stops progressing, check if the output shows an upgraded node with `state: Succeeded` while the remaining nodes are stuck in the `Images preloaded` state. |
Comment on lines
+39
to
+46
| Check the upgraded node's current and pending operating system images: | ||
|
|
||
| ```bash | ||
| kubectl get node <node-name> \ | ||
| -o custom-columns='NAME:.metadata.name,CURRENT-OS-IMAGE:.status.nodeInfo.osImage,PENDING-OS-IMAGE:.metadata.annotations.harvesterhci\.io/pendingOSImage' | ||
| ``` | ||
|
|
||
| Use the workaround only when the status pattern described above is present, `PENDING-OS-IMAGE` is not empty, and it matches `CURRENT-OS-IMAGE`. |
Contributor
There was a problem hiding this comment.
Suggested change
| Check the upgraded node's current and pending operating system images: | |
| ```bash | |
| kubectl get node <node-name> \ | |
| -o custom-columns='NAME:.metadata.name,CURRENT-OS-IMAGE:.status.nodeInfo.osImage,PENDING-OS-IMAGE:.metadata.annotations.harvesterhci\.io/pendingOSImage' | |
| ``` | |
| Use the workaround only when the status pattern described above is present, `PENDING-OS-IMAGE` is not empty, and it matches `CURRENT-OS-IMAGE`. | |
| 1. Check the upgraded node's current and pending operating system images: | |
| ```bash | |
| kubectl get node <node-name> \ | |
| -o custom-columns='NAME:.metadata.name,CURRENT-OS-IMAGE:.status.nodeInfo.osImage,PENDING-OS-IMAGE:.metadata.annotations.harvesterhci\.io/pendingOSImage' | |
| ``` | |
| Check if the value of `PENDING-OS-IMAGE` matches `CURRENT-OS-IMAGE`. |
Comment on lines
+50
to
+58
| Change the affected node's state in the Upgrade CR back to `Waiting Reboot`: | ||
|
|
||
| ```bash | ||
| kubectl -n harvester-system patch upgrades.harvesterhci.io <upgrade-name> \ | ||
| --type=json \ | ||
| -p '[{"op":"replace","path":"/status/nodeStatuses/<node-name>/state","value":"Waiting Reboot"}]' | ||
| ``` | ||
|
|
||
| Replace `<upgrade-name>` and `<node-name>` with the names of the Upgrade CR and affected node. The upgrade resumes after the upgrade controller reconciles the node. |
Contributor
There was a problem hiding this comment.
Suggested change
| Change the affected node's state in the Upgrade CR back to `Waiting Reboot`: | |
| ```bash | |
| kubectl -n harvester-system patch upgrades.harvesterhci.io <upgrade-name> \ | |
| --type=json \ | |
| -p '[{"op":"replace","path":"/status/nodeStatuses/<node-name>/state","value":"Waiting Reboot"}]' | |
| ``` | |
| Replace `<upgrade-name>` and `<node-name>` with the names of the Upgrade CR and affected node. The upgrade resumes after the upgrade controller reconciles the node. | |
| Apply this workaround only if all conditions described in the previous section are met. | |
| 1. Change the affected node's state in the Upgrade CR back to `Waiting Reboot`: | |
| Replace `<upgrade-name>` with the name of the `Upgrade` CR, and `<node-name>` with the name of the affected node. | |
| ```bash | |
| kubectl -n harvester-system patch upgrades.harvesterhci.io <upgrade-name> \ | |
| --type=json \ | |
| -p '[{"op":"replace","path":"/status/nodeStatuses/<node-name>/state","value":"Waiting Reboot"}]' | |
| ``` | |
| 1. Verify that the upgrade resumes once the upgrade controller reconciles the node state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
There are currently no upgrade documents available for v1.9.
In addition, we have identified a known issue where an upgrade from v1.8.2 to v1.9.0 may get stuck in a state where some nodes have been fully upgraded, with their node upgrade state marked as
Succeeded, while the remaining nodes stay atImage preloadedand make no further progress.Solution:
Create a new upgrade documentation page for clusters upgrading from v1.8.x to v1.9.x.
Add a section describing how to identify the upgrade issue and the available workaround.
Related Issue(s):
harvester/harvester#11543
Test plan:
Additional documentation or context