fix(upgrade): patch rke2-multus failurePolicy to abort (backport #59)

[mergify]

Problem:

rke2-multus upgrade can fail during the node-upgrade phase. According to the Klipper logic, it will try to uninstall before upgrading a failed chart release. This is considered disastrous because the NAD CRD will be deleted, and all the user-created NADs will be purged as well.

Solution:

Disabling the rke2-multus chart upgrade during the node-upgrade phase by dropping .skip files on each node. After the node-upgrade phase is passed, remove those .skip files and let the Helm controller do the upgrade for rke2-multus.

Related Issue(s):

harvester/harvester#10359

Test plan:

1. Prepare a 3-node cluster in v1.7.1
2. Prepare the upgrade toolkit image (either build it yourself or use mine, starbops/harvester-upgrade-toolkit:feat-patch-rke2-multus-failure-policy-head)
3. Install and enable the harvester-upgrade-manager add-on

   apiVersion: harvesterhci.io/v1beta1
   kind: Addon
   metadata:
     name: harvester-upgrade-manager
     namespace: harvester-system
     labels:
       addon.harvesterhci.io/experimental: "true"
   spec:
     enabled: true
     repo: https://charts.harvesterhci.io
     version: 1.8.0-rc6
     chart: harvester-upgrade-manager
     valuesContent: |
       manager:
         image:
           repository: starbops/harvester-upgrade-toolkit
           tag: feat-patch-rke2-multus-failure-policy-head

4. Create the Version CR

   apiVersion: harvesterhci.io/v1beta1
   kind: Version
   metadata:
     name: v1.8.0-rc6
     namespace: harvester-system
   spec:
     isoURL: https://releases.rancher.com/harvester/v1.8.0/harvester-v1.8.0-rc6-amd64.iso
     isoChecksum: "daf8e33f402a80640df5f4b8de22eb496e6c29f98ab29a9ee1adaef9dd5fc355e2fef547f5c2a5a1f60d541c09e6930a220a73f013d2368eed51f9ea174f6d96"

5. Create the UpgradePlan CR

   apiVersion: management.harvesterhci.io/v1beta1
   kind: UpgradePlan
   metadata:
     annotations:
       management.harvesterhci.io/allow-deletion: "true"
       management.harvesterhci.io/upgrade-toolkit-image: starbops/harvester-upgrade-toolkit
     generateName: hvst-upgrade-
   spec:
     version: v1.8.0-rc6
     upgrade: feat-patch-rke2-multus-failure-policy-head

6. The upgrade should complete successfully
7. The failurePolicy should be set to abort for the rke2-multus HelmChartConfig

Additional documentation or context

Porting changes from harvester/harvester#10366

---

This is an automatic backport of pull request #59 done by Mergify.