[TEST] v1.8.2 Release Testing - Automation - E2E automation

[asc5543]

Ref: https://app.qase.io/plan/HV/19

[asc5543]

There is the test result of v1.8.2-rc1
https://ci.suse-virtualization.suse.de/job/harvester-install-and-test/106/TestReport/

No new issue found in harvester

Environment

• Harvester: v1.8.2-rc1
• Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): 3 node Baremetal

Failed/Skipped test case analysis

There are 0 new issues found for harvester and 1 new issues for tests in this E2E automation test:

• Tests
  • [TEST] test_4_vm_template.py::TestVMTemplate::test_create_template_with_data fails due to AssertionError: (500, 'unexpected end of JSON input') #2756

And there are known issues which are already found before:

• Harvester
  • [BUG] Create an image which name length over the limitaion via YAML, the image still can be created harvester#9515
  • [BUG] Creates volume with size "999999999Ti" via "Edit as YAML", the error message doesn't show harvester#9268
  • [BUG] When creating VM with oversized volume cuz it sutck at Starting instead of Unschedulable harvester#9850
  • [BUG] Restoring the backup of Live Migrated VM , starts VM back on the original node where the backup was taken  harvester#1473
• Tests
  • [TEST] All test_z_terraform.py cases are failed due to "Failed to install provider" #2389

Analysis Details

Result	Test	Reason or Links	Comment
Skipped	harvester_e2e_tests/apis/test_networks.py::TestNetworksNegative::test_create_with_invalid_id_103[0]::setup	byVersion
Skipped	harvester_e2e_tests/apis/test_networks.py::TestNetworksNegative::test_create_with_invalid_id_103[4095]::setup	byVersion
Skipped	harvester_e2e_tests/apis/test_networks.py::TestNetworks::test_create_103::setup	byVersion
Failed	harvester_e2e_tests/apis/test_settings.py::TestAdditionalGuestMemOverhead::test_invalid_values	E AssertionError: Setting additional-guest-memory-overhead-ratio be updated to the invalid value 100E assert 422 == 409	Rerun Pass
Skipped	harvester_e2e_tests/apis/test_volumes.py::TestVolumesNegative::test_create_without_name::setup	byVersion
XFailed	harvester_e2e_tests/integrations/test_1_images.py::TestImageEnhancements::test_image_name_length_limits	harvester/issues/9515🔗
XFailed	harvester_e2e_tests/integrations/test_1_volumes.py::test_create_volume_invalid_specifications[too_large_size]	harvester/issues/9268🔗
Failed	harvester_e2e_tests/integrations/test_3_vm_functions.py::TestVMResource::test_create_schedule_on_maximum[memory]	E AssertionError: (200, {'apiVersion': 'kubevirt.io/v1', 'id': 'default/vm-07h18m15s821303-07-07', 'kind': 'VirtualMachineInstance', 'li... 'https://10.115.252.80/v1/harvester/kubevirt.io.virtualmachineinstances/default/vm-07h18m15s821303-07-07', ...}, ...})E assert False	Rerun Pass
XFailed	harvester_e2e_tests/integrations/test_3_vm_functions.py::test_create_vm_no_available_resources[disk]::setup	harvester/issues/9850🔗
Skipped	harvester_e2e_tests/integrations/test_3_vm_functions.py::test_update_vm_machine_type[q35_to_pc]::setup	byVersion
Skipped	harvester_e2e_tests/integrations/test_3_vm_functions.py::test_update_vm_machine_type[pc_to_q35]::setup	byVersion
Error	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_connection[S3]::setup		Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::tests_backup_vm[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_update_backup_by_yaml[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_with_new_vm[S3]::setup	byPriorCase
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_replace_with_delete_vols[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_replace_vm_not_stop[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_with_invalid_name[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_replace_with_vm_shutdown_command[S3]::setup	byPriorCase	Rerun Pass
Error	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::tests_backup_vm[NFS]::setup		Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_update_backup_by_yaml[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_with_new_vm[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_replace_with_delete_vols[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_replace_vm_not_stop[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_with_invalid_name[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestore::test_restore_replace_with_vm_shutdown_command[NFS]::setup	byPriorCase	Rerun Pass
XFailed	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestoreOnMigration::test_backup_migrated_vm[S3]::setup	/issues/1473🔗
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestoreOnMigration::test_restore_replace_migrated_vm[S3]::setup	byPriorCase
XFailed	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestoreOnMigration::test_backup_migrated_vm[NFS]::setup	/issues/1473🔗
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestBackupRestoreOnMigration::test_restore_replace_migrated_vm[NFS]::setup	byPriorCase
Error	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_backup_multiple[S3]::setup		Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_delete_first_backup[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_delete_last_backup[S3]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_delete_middle_backup[S3]::setup	byPriorCase	Rerun Pass
Failed	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_backup_multiple[NFS]	E AssertionError: Timed-out waiting for the backup '0-0734425622-08h27m13s931339-07-07' to be ready.	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_delete_first_backup[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_delete_last_backup[NFS]::setup	byPriorCase	Rerun Pass
Skipped	harvester_e2e_tests/integrations/test_4_vm_backup_restore.py::TestMultipleBackupRestore::test_delete_middle_backup[NFS]::setup	byPriorCase	Rerun Pass
Failed	harvester_e2e_tests/integrations/test_4_vm_overcommit.py::TestVMOverCommit::test_update_vm_overcommit	E assert (18.242 - 17.242) < 1	Rerun Pass
Failed	harvester_e2e_tests/integrations/test_4_vm_template.py::TestVMTemplate::test_create_template_with_data	#2756	Manual test pass
Failed	harvester_e2e_tests/integrations/test_9_addons.py::TestVMDHCPControllerAddon::test_verify_vm_dhcp_controller	E AssertionError: Invalid IP address format: fe80::dc14:3dff:fefd:b3d8E assert 1 == 4E + where 1 = len(['fe80::dc14:3dff:fefd:b3d8'])	Manual test pass
Error	harvester_e2e_tests/integrations/test_z_terraform.py::test_create_ssh_key::setup	#2389
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::test_delete_ssh_key::setup	byPriorCase
Error	harvester_e2e_tests/integrations/test_z_terraform.py::test_create_volume::setup
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::test_delete_volume::setup	byPriorCase
Error	harvester_e2e_tests/integrations/test_z_terraform.py::test_create_image::setup
Error	harvester_e2e_tests/integrations/test_z_terraform.py::test_create_volume_from_image::setup
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::test_delete_image::setup	byPriorCase
Error	harvester_e2e_tests/integrations/test_z_terraform.py::TestNetworking::test_create_clusternetwork::setup
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::TestNetworking::test_create_vlanconfig::setup	byPriorCase
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::TestNetworking::test_create_vm_network::setup	byPriorCase
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::TestNetworking::test_delete_vm_network::setup	byPriorCase
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::TestNetworking::test_delete_vlanconfig::setup	byPriorCase
Skipped	harvester_e2e_tests/integrations/test_z_terraform.py::TestNetworking::test_delete_clusternetwork::setup	byPriorCase