[TEST] v1.8.2 Release Testing - Four node upgrade with bonded nics

[asc5543]

What's the test to develop? Please describe

Perform Four node upgrade with bonded nics for v1.8.2 release testing

Prerequisite and dependency of test

1. Apply Harvester Storage network settings in the env.
2. 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped
3. 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped
4. A VM created with Longhorn V2 volume
5. Create a new storage class apart from default one. Use the new storage class for some basic operations.

Test Steps before upgrade

1. Import to Rancher. Verify the UI-extension version is correct. Use Rancher UI to do all the operation now onwads.
2. Create multiple RKE2 guest clusters (1 CP node, 2 worker nodes) provisioned on Harvester VM. Keep the RKE2 versions as different.
3. Create Load balancer and PVC in the guest clusters.
4. Follow the upgrade path as mentioned in the test plan.

Upgrade Test Path

Harvester v1.7.2 & Rancher v2.13.7 & Guest RKE2 v1.34
-> Harvester v1.7.2 & Rancher v2.14.3 & Guest RKE2 v1.34
-> Harvester v1.8.2-rc1 & Rancher v2.14.3 & Guest RKE2 v1.35

OS:

• SLE 15 SP7
• SLE 16

Upgrade Test Check

1. Can correctly upgrade from v1.7.2 to v1.8.2-rc1
2. Create another one node guest cluster.
3. Scale down up the guest cluster (Only the worker nodes).

Post Upgrade Check

1. Dependencies Check
   • dependency_charts
   • Versions of all the crds
2. RKE2 guest clusters are running fine and all the workloads/pods are running.
3. Virtual machines are in same state as before and accessible.
4. Restore the backups, check the data
5. Image and volume status
6. Monitoring chart status
7. VM operations are highlighted and working fine.
8. Restart a node of Harvester.
9. Add a node after the upgrade
10. Upgrade guest clusters.
11. Create a new Harvester PVC and loadbalancers.
12. Shutting off VM and restarting VM of guest clusters.
13. Migrate regular VMs created in Harvester.

[asc5543]

Prerequisite and dependency setup

1. ✔ Apply Harvester Storage network settings in the env. (VLAN 1 network on mgmt and 1 network on bonded NICs)

   
2. ✔ 4 Virtual machines with data and md5sum computed- 2 running, 2 stopped (2 VM created with Longhorn V2 volume)

   

   sles@sle16-running:~> cat running.txt 
   sle16-running
   sles@sle16-running:~> sha512sum running.txt 
   115e965eecfc2369a6a6d2a4753ce1be563bf556450d41bc6304fa10c58d5c5cffad22e7eb2141d7914e062bd1637c38193a4bfc3dcba354cd6c1e3e66765774  running.txt
   

   sles@sle16-stopped:~> cat stopped.txt 
   sle16-stopped
   sles@sle16-stopped:~> sha512sum stopped.txt 
   6591dbda4cf81292878e7f9adbaac28ba8ccc9e1b766806ad312bb1a95b660509312bd2767e58a2bc8046b9255fa3ffd73c8ee237ee6a6b225390d0c46239744  stopped.txt
   

   sles@sle15-sp7-running:~> cat running.txt
   sle15-sp7-running
   sles@sle15-sp7-running:~> sha512sum running.txt
   75d586cf311aaf8e11f4efd5766f044bda2aa18fc19cb9bef70b173b6d0e493bea94213ea1412f9f32a947d0243877e0c5c2591a4489f2aea8387b8a28cdb0b9  running.txt
   

   sles@sle15-sp7-stopped:~> cat stopped.txt
   sle15-sp7-stopped
   sles@sle15-sp7-stopped:~> sha512sum stopped.txt 
   38f78c38139e6c645b9200c27c45016e1ae0f6ccb06c310fab4115608f89df2ad3d49f43689c56b33d2e32dca2b8a7b6fc66f8582d481a7ef2300a586be94d0c  stopped.txt
   

3. ✔ 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stopped

   
4. ✔ Create a new storage class apart from default one. Use the new storage class for some basic operations.

   

[asc5543]

Test Steps before upgrade

1. ✔ Import to Rancher. Verify the UI-extension version is correct. Use Rancher UI to do all the operation now onward.

   
2. ✔ Create multiple RKE2 guest clusters (1 CP node, 2 worker nodes) provisioned on Harvester VM. Keep the RKE2 versions as different.

   
3. ✔ Create nginx and PVC in the guest clusters.

   
4. ✔ Create Load balancer in the guest clusters.

   

[asc5543]

Test Steps of upgrade

1. ✔ Rancher 2.13.7 -> 2.14.3 and Harvester UI Extension 1.7.2 -> 1.8.2

   
2. ✔ Harvester v1.7.2 -> v1.8.2

   Run https://github.com/harvester/upgrade-helpers/blob/main/pre-check/v1.x/check.sh first

   # ./check.sh 
   Upgrading from version v1.7.2
   ==============================
   
   Starting Host check... 
   Host Test: Pass
   
   ==============================
   
   Starting Certificates check...
   Certificates Test: Pass
   
   ==============================
   
   Starting Node Free Space check...
   Node-Free-Space Test: Pass
   
   ==============================
   
   Starting Helm Bundle status check... 
   Helm-Bundles Test: Pass
   
   ==============================
   
   Starting Harvester Bundle status check...
   Harvester-Bundles Test: Pass
   
   ==============================
   
   Starting Node Status check... 
   Node-Status Test: Pass
   
   ==============================
   
   Starting CAPI Cluster State check...
   CAPI-Cluster-State Test: Pass
   
   ==============================
   
   Starting CAPI Cluster Pause check...
   CAPI-Cluster-Pause Test: Pass
   
   ==============================
   
   Starting CAPI Machine Count check...
   CAPI-Machine-Count Test: Pass
   
   ==============================
   
   Starting CAPI Machine State check...
   CAPI-Machine-State Test: Pass
   
   ==============================
   
   Starting Longhorn Volume Health Status check...
   Longhorn-Volume-Health-Status Test: Pass
   
   ==============================
   
   Starting Stale Longhorn Volumes check...
   Stale-Longhorn-Volumes Test: Pass
   
   ==============================
   
   Starting Longhorn Backing Images check...
   Longhorn-Backing-Images Test: Pass
   
   ==============================
   
   Starting Virtual Machines Live Migration check...
   Virtual Machines Test: Pass
   
   ==============================
   
   Starting Pod Status check...
   Pod-Status Test: Pass
   
   ==============================
   
   Starting Kubeconfig Secret check...
   Kubeconfig Secret Test: Pass
   
   ==============================
   
   Starting Backup Target check...
   The current version (v1.7.2) is not v1.4.1 or v1.4.2. Skip the Backup Target check.
   Backup Target Test: Skipped
   
   ==============================
   
   All checks have passed.
   

   
3. ✔ Create another one node guest cluster.

   
4. ✔ Scale down up the guest cluster (Only the worker nodes).

   

[asc5543]

Post Upgrade Check

1. ✗ Dependencies Check

   Chart: https://github.com/harvester/harvester/tree/v1.8.2-rc1/deploy/charts/harvester/dependency_charts
   Values: https://github.com/harvester/harvester/tree/v1.8.2-rc1/deploy/charts/harvester/values.yaml

   • cdi

     $\color{red}{\textsf{FAIL}}$ $~~$

     • tag: 1.64.0-150700.9.6.1
       https://github.com/harvester/harvester/blob/v1.8.2-rc1/deploy/charts/harvester/values.yaml
     • Harvester: cdi
       

   • csi-snapshotter

     $\color{green}{\textsf{PASS}}$ $~~$

     • tag: v8.5.0
       https://github.com/harvester/harvester/blob/v1.8.2-rc1/deploy/charts/harvester/dependency_charts/csi-snapshotter/values.yaml
     • Harvester: csi-snapshotter
       

   • kubevirt-operator

     $\color{green}{\textsf{PASS}}$ $~~$

     • tag: 1.7.4-150700.3.24.2
       https://github.com/harvester/harvester/blob/v1.8.2-rc1/deploy/charts/harvester/values.yaml
     • Harvester: virt-operator (deployment)
       

   • kubevirt

     $\color{green}{\textsf{PASS}}$ $~~$

     • tag: 1.7.4-150700.3.24.2
       https://github.com/harvester/harvester/blob/v1.8.2-rc1/deploy/charts/harvester/values.yaml
     • Harvester: kubevirt (crd)
       

   • whereabouts

     $\color{green}{\textsf{PASS}}$ $~~$

     • tag: 0.9.3
       https://github.com/harvester/harvester/blob/v1.8.2-rc1/deploy/charts/harvester/values.yaml
     • Harvester: harvester-whereabouts- (pod)
       
2. ✔ Versions of all the crds

   # ./checker.sh v1.8.2-rc1
   Fetching crd list from: https://api.github.com/repos/harvester/harvester/contents/deploy/charts/harvester-crd/templates?ref=v1.8.2-rc1
   CRD LIST:
   ---
   crd-kubevirt-1.7.0.yaml
   harvesterhci.io_addons.yaml
   harvesterhci.io_keypairs.yaml
   harvesterhci.io_preferences.yaml
   harvesterhci.io_resourcequotas.yaml
   harvesterhci.io_schedulevmbackups.yaml
   harvesterhci.io_settings.yaml
   harvesterhci.io_supportbundles.yaml
   harvesterhci.io_upgradelogs.yaml
   harvesterhci.io_upgrades.yaml
   harvesterhci.io_versions.yaml
   harvesterhci.io_virtualmachinebackups.yaml
   harvesterhci.io_virtualmachineimagedownloaders.yaml
   harvesterhci.io_virtualmachineimages.yaml
   harvesterhci.io_virtualmachinerestores.yaml
   harvesterhci.io_virtualmachinetemplates.yaml
   harvesterhci.io_virtualmachinetemplateversions.yaml
   harvesterhci.io_volumeremotebackups.yaml
   harvesterhci.io_volumeremoterestores.yaml
   volumesnapshotclasses-8.3.0.yaml
   volumesnapshotcontents-8.3.0.yaml
   volumesnapshots-8.3.0.yaml
   whereabouts.cni.cncf.io_ippools-1.16.0.yaml
   whereabouts.cni.cncf.io_nodeslicepools.yaml
   whereabouts.cni.cncf.io_overlappingrangeipreservations-1.16.0.yaml
   ---
   Check crd-kubevirt-1.7.0.yaml version
   Observation: v1 v1alpha3 
   Expectation: v1 v1alpha3 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_addons.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_keypairs.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_preferences.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_resourcequotas.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_schedulevmbackups.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_settings.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_supportbundles.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_upgradelogs.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_upgrades.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_versions.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_virtualmachinebackups.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_virtualmachineimagedownloaders.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_virtualmachineimages.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_virtualmachinerestores.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_virtualmachinetemplates.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_virtualmachinetemplateversions.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_volumeremotebackups.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check harvesterhci.io_volumeremoterestores.yaml version
   Observation: v1beta1 
   Expectation: v1beta1 
   Status: True (PASS)
   
   ---
   Check volumesnapshotclasses-8.3.0.yaml version
   Observation: v1 v1beta1 
   Expectation: v1 v1beta1 
   Status: True (PASS)
   
   ---
   Check volumesnapshotcontents-8.3.0.yaml version
   Observation: v1 v1beta1 
   Expectation: v1 v1beta1 
   Status: True (PASS)
   
   ---
   Check volumesnapshots-8.3.0.yaml version
   Observation: v1 v1beta1 
   Expectation: v1 v1beta1 
   Status: True (PASS)
   
   ---
   Check whereabouts.cni.cncf.io_ippools-1.16.0.yaml version
   Observation: v1alpha1 
   Expectation: v1alpha1 
   Status: True (PASS)
   
   ---
   Check whereabouts.cni.cncf.io_nodeslicepools.yaml version
   Observation: v1alpha1 
   Expectation: v1alpha1 
   Status: True (PASS)
   
   ---
   Check whereabouts.cni.cncf.io_overlappingrangeipreservations-1.16.0.yaml version
   Observation: v1alpha1 
   Expectation: v1alpha1 
   Status: True (PASS)
   
   ---
   ALL PASSED!
   

3. ✔ RKE2 guest clusters are running fine and all the workloads/pods are running.

   
4. ✔ Virtual machines are in same state as before and accessible.

   
5. ✔ Check the VM status and the file sha512sum

   sles@sle16-running:~> cat running.txt 
   sle16-running
   sles@sle16-running:~> sha512sum running.txt 
   115e965eecfc2369a6a6d2a4753ce1be563bf556450d41bc6304fa10c58d5c5cffad22e7eb2141d7914e062bd1637c38193a4bfc3dcba354cd6c1e3e66765774  running.txt
   

   sles@sle15-sp7-running:~> cat running.txt
   sle15-sp7-running
   sles@sle15-sp7-running:~> sha512sum running.txt 
   75d586cf311aaf8e11f4efd5766f044bda2aa18fc19cb9bef70b173b6d0e493bea94213ea1412f9f32a947d0243877e0c5c2591a4489f2aea8387b8a28cdb0b9  running.txt
   

   sles@sle16-stopped:~> cat stopped.txt 
   sle16-stopped
   sles@sle16-stopped:~> sha512sum stopped.txt 
   6591dbda4cf81292878e7f9adbaac28ba8ccc9e1b766806ad312bb1a95b660509312bd2767e58a2bc8046b9255fa3ffd73c8ee237ee6a6b225390d0c46239744  stopped.txt
   

   sles@sle15-sp7-stopped:~> cat stopped.txt
   sle15-sp7-stopped
   sles@sle15-sp7-stopped:~> sha512sum stopped.txt 
   38f78c38139e6c645b9200c27c45016e1ae0f6ccb06c310fab4115608f89df2ad3d49f43689c56b33d2e32dca2b8a7b6fc66f8582d481a7ef2300a586be94d0c  stopped.txt
   

6. ✔ Restore the backups, check the data with the same sha512sum as before

   sles@sle16-running:~> cat running.txt 
   sle16-running
   sles@sle16-running:~> sha512sum running.txt 
   115e965eecfc2369a6a6d2a4753ce1be563bf556450d41bc6304fa10c58d5c5cffad22e7eb2141d7914e062bd1637c38193a4bfc3dcba354cd6c1e3e66765774  running.txt
   

   sles@sle15-sp7-stopped:~> cat stopped.txt
   sle15-sp7-stopped
   sles@sle15-sp7-stopped:~> sha512sum stopped.txt 
   38f78c38139e6c645b9200c27c45016e1ae0f6ccb06c310fab4115608f89df2ad3d49f43689c56b33d2e32dca2b8a7b6fc66f8582d481a7ef2300a586be94d0c  stopped.txt
   

7. ✔ Image and volume status

   
8. ✔ Monitoring chart status

   
9. ✔ VM operations are highlighted and working fine.

   • Start:
   • Stop
   • Restart
   • Soft reboot
   • Delete
   • Clone
   • Migrate
   • Add volume
10. ✔ Restart a node of Harvester.

    
11. ✔ Add a node after the upgrade

    
12. ✔ Create a new Harvester PVC and loadbalancers.

    • ✔ PVC

      
    • ✔ Load Balancers

      
13. ✔ Shutting off VM and restarting VM of guest clusters.