From 16651752752e4a448fd0410b87001148e0de2494 Mon Sep 17 00:00:00 2001 From: Philipp Ahmann Date: Wed, 24 Jun 2026 15:56:50 +0000 Subject: [PATCH 1/3] Verification: Update test specification details Enhance verification process to include pending parts of 8-9.4.2.2 and 8-9.4.3.3 Closes #566 Signed-off-by: Philipp Ahmann --- .../guidance/verification_guideline.rst | 38 ++++++++++++++++--- .../guidance/verification_process_reqs.rst | 7 ++-- .../guidance/verification_specification.rst | 11 +++++- .../verification/verification_concept.rst | 10 +++-- .../verification/verification_getstrt.rst | 4 +- 5 files changed, 55 insertions(+), 15 deletions(-) diff --git a/process/process_areas/verification/guidance/verification_guideline.rst b/process/process_areas/verification/guidance/verification_guideline.rst index 67175f5f6d..54d9d3c31a 100644 --- a/process/process_areas/verification/guidance/verification_guideline.rst +++ b/process/process_areas/verification/guidance/verification_guideline.rst @@ -99,9 +99,14 @@ In order to check the test results for the impact of a change or addition, it is execute affected test cases locally upfront using the execution framework of the build tooling following basically the steps the CI does locally. +Tests have to be identifiable by a unique identifier, as described in :need:`gd_guidl__verification_specification`. +They need need to have a clear passed or failed result and a documented configuration to enable proper evaluation of the result. + Automated tests can also be executed locally, as the sources and binaries are available for re-execution. Failing test cases during re-execution can be reported following the guide :need:`gd_temp__problem_template`. +During a relase, for any non executed test case, the reason for non-execution has to be documented in the +:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level (unit to platform) of the test case. Execution of manual test cases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -112,20 +117,43 @@ test cases and reports the result in the same logging format as automated tests This enables parsable execution logs which can be used in automated test result collection. Failing test cases during manual execution are reported by following the guide :need:`gd_temp__problem_template`. +Also manual test cases are written following the :need:`gd_guidl__verification_specification` and have a unique identifier. + +A tag `manual` is suggested to mark the test case as manual, so that it can be filtered out from automated test execution. + +In case a manual test case is not executed for a release, the reason for non-execution has to be documented in the +:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level of the test case. + Reporting of failing test cases ------------------------------- Any failing test case requires an ISSUE. + The passing rate of safety-critical test cases need to be 100% in order to release the affected component. -In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead` and -:need:`rl__project_lead` can decide, if the platform is in a releasable state. The accepted minimal -path rate is defined in the :need:`wp__verification_plan`. Due to the high degree of automation, a -it is recommended that a path rate lower 95% is not acceptable. +In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead` +can decide, if the platform is in a releasable state. The accepted minimal pass rate is defined +in the :need:`wp__verification_plan`. Due to the high degree of automation, it is recommended +that a pass rate lower 95% is not acceptable. This percentage may increase with the maturity of +the overall platform and the test coverage. -In case an existing test case is failing due to regression in the CI, the respective issuer of the +In case an existing test case is failing due to regression in the CI, the respective issue of the PR in their role as :need:`rl__contributor` is responsible for fixing the test case as part of respective PR. +Test case results are also documented in the :need:`wp__verification_platform_ver_report` and +:need:`wp__verification_module_ver_report`. + +Reporting of not executed or skipped test cases +----------------------------------------------- + +In case a test case is not executed or skipped, a rational has to be provided in the release notes +in case they are link against a requirement. A skipped or not executed test case is not counting into +the value for the requirements coverage, as only executed and passed test cases can be counted for the +coverage of a requirement. + +Skipped or not executed test cases are also documented in the :need:`wp__verification_platform_ver_report` and +:need:`wp__verification_module_ver_report`. + Reuse of existing test cases ---------------------------- diff --git a/process/process_areas/verification/guidance/verification_process_reqs.rst b/process/process_areas/verification/guidance/verification_process_reqs.rst index 806cddf9e9..cf84309001 100644 --- a/process/process_areas/verification/guidance/verification_process_reqs.rst +++ b/process/process_areas/verification/guidance/verification_process_reqs.rst @@ -146,6 +146,7 @@ Process Requirements The tool automation shall automatically generate the Verification reports. These may be independent documents (i.e. not integrated into docs-as-code based repositories). The content of the reports is specified in :need:`gd_temp__platform_ver_report` and :need:`gd_temp__mod_ver_report`. + The execution results of a test cases are marked with a clear passed of failed result. .. gd_req:: Verification Report Archiving :id: gd_req__verification_report_archiving @@ -174,9 +175,9 @@ Process Requirements - TestType and DerivationTechnique shall be set - Description shall not be empty - In a Platform Integration Test Partially/FullyVerifies shall be set to a Platform Requirement - - If Partially/FullyVerifies are set in Feature Integration Test these shall link to Feature Requirements - - If Partially/FullyVerifies are set in Component Integration Test these shall link to Component Requirements - - If Partially/FullyVerifies are set in Unit Test these shall link to Component Requirements + - If Partially/FullyVerifies are set in Feature Integration Test these shall link to at least one Feature Requirement + - If Partially/FullyVerifies are set in Component Integration Test these shall link to at least one Component Requirement + - If Partially/FullyVerifies are set in Unit Test these shall link to at least one Component Requirement .. gd_req:: Verification Documentation Checks Extended :id: gd_req__verification_checks_extended diff --git a/process/process_areas/verification/guidance/verification_specification.rst b/process/process_areas/verification/guidance/verification_specification.rst index 37b5415c68..9d65eb2497 100644 --- a/process/process_areas/verification/guidance/verification_specification.rst +++ b/process/process_areas/verification/guidance/verification_specification.rst @@ -55,8 +55,9 @@ A test specification contains the following attributes. - The objective of the test. - Inputs - - Expected outcome (e.g. "A success message is displayed.") + - Expected outcome/output/result (e.g. "A success message is displayed." or "Result should be 42.") - Test environment (e.g. network configuration, clean system state) + - Expected time sequence of events and behavior (where applicable and an expectation) - * - TestType - Examples are: @@ -82,6 +83,14 @@ A test specification contains the following attributes. The implementation of :need:`wp__verification_plan` defines the full list of allowed types and methods. +It is assumed that tests will be written as code (also for manual tests, which are script based) +and each test case will have a unique identifier, by its script, execution call, or function name. +The call used to execute the test marks the uniqueness of the test case and its identification, +e.g. guaranteeing proper traceability and reproducibility. + +As the tests are stored in a repository close to the implementation code, versioning is done by the versioning of the repository. + +Any specification and resulting implementation ends with a clear passed or failed result. Test description ---------------- diff --git a/process/process_areas/verification/verification_concept.rst b/process/process_areas/verification/verification_concept.rst index 938a9a76ff..82287023fe 100644 --- a/process/process_areas/verification/verification_concept.rst +++ b/process/process_areas/verification/verification_concept.rst @@ -124,7 +124,6 @@ Usually the defined methods are not applied on each verification level between u Also their execution may differ whether it is a QM or ASIL rated test case. The rigor is described in the implementation of :need:`wp__verification_plan`. - Automated test cases (as well as manual, where applicable) shall contain further information about which methods have been applied. The corresponding guidance is given here: :need:`gd_guidl__verification_guide`. The identifier of the respective method is to be used as meta data (*TestType* and *DerivationTechnique*). @@ -161,17 +160,20 @@ tests or component tests. This linking is done using metatags. This is also true integration tests linking to the component requirements and architecture. Traceability of feature integration tests shall be established through linking those test cases to -feature requirements and architecture as features describe the integrated behaviour of all components. +feature requirements and architecture as features describe the integrated behavior of all components. Traceability of platform integration tests shall be established through linking those test cases to -stakeholder requirements as stakeholder requirements describe the platform behaviour. +stakeholder requirements as stakeholder requirements describe the platform behavior. Note that all the above tests shall only link to requirements of type "Functional" and "Interface". The verification of requirements of types "Process" and "Non-Functional" will be done via Analysis, which is part of the requirement inspection :need:`doc__feature_name_req_inspection` and `Component Requirements Inspection Checklist `__. Requirements always include Assumptions Of Use. -A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests` +A more detailed description of how to link code to requirements is available by :need:`gd_req__verification_link_tests`. + +Another element of traceability for a proper backlink is the unique identification of test cases as described in the +:need:`gd_guidl__verification_specification`. Workflow for Verification Guidance ---------------------------------- diff --git a/process/process_areas/verification/verification_getstrt.rst b/process/process_areas/verification/verification_getstrt.rst index a3ba474b9a..bd4cbb16ee 100644 --- a/process/process_areas/verification/verification_getstrt.rst +++ b/process/process_areas/verification/verification_getstrt.rst @@ -41,8 +41,8 @@ General Workflow The workflows can be split into 4 major parts: -* Test planning filling the template :need:`gd_temp__verification_plan`. -* Test specification and implementation for the respective testing level +* Test planning by filling the template :need:`gd_temp__verification_plan`. +* Test specification and implementation for the respective testing level (unit to platform). * Test execution by the CI. (Manual test cases are treated as automated test with user interaction and timeouts.) * Test reports are created when all verification artifacts on a module and platform level are From b243e61bf8739618f917b9e357bc52627f56c1c0 Mon Sep 17 00:00:00 2001 From: Philipp Ahmann Date: Wed, 24 Jun 2026 16:30:47 +0000 Subject: [PATCH 2/3] Incorparating CoPilot review findings Signed-off-by: Philipp Ahmann --- .../guidance/verification_guideline.rst | 16 +++++++--------- .../guidance/verification_process_reqs.rst | 2 +- .../guidance/verification_specification.rst | 12 ++++++------ .../verification/verification_concept.rst | 2 +- 4 files changed, 15 insertions(+), 17 deletions(-) diff --git a/process/process_areas/verification/guidance/verification_guideline.rst b/process/process_areas/verification/guidance/verification_guideline.rst index 54d9d3c31a..4e245afec9 100644 --- a/process/process_areas/verification/guidance/verification_guideline.rst +++ b/process/process_areas/verification/guidance/verification_guideline.rst @@ -100,12 +100,12 @@ execute affected test cases locally upfront using the execution framework of the following basically the steps the CI does locally. Tests have to be identifiable by a unique identifier, as described in :need:`gd_guidl__verification_specification`. -They need need to have a clear passed or failed result and a documented configuration to enable proper evaluation of the result. +They need to have a clear pass/fail result and a documented configuration to enable proper evaluation of the result. Automated tests can also be executed locally, as the sources and binaries are available for re-execution. Failing test cases during re-execution can be reported following the guide :need:`gd_temp__problem_template`. -During a relase, for any non executed test case, the reason for non-execution has to be documented in the +During a release, for any non-executed test case, the reason for non-execution must be documented in the :need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level (unit to platform) of the test case. Execution of manual test cases @@ -136,9 +136,8 @@ in the :need:`wp__verification_plan`. Due to the high degree of automation, it i that a pass rate lower 95% is not acceptable. This percentage may increase with the maturity of the overall platform and the test coverage. -In case an existing test case is failing due to regression in the CI, the respective issue of the -PR in their role as :need:`rl__contributor` is responsible for fixing the test case as part of -respective PR. +In case an existing test case is failing due to regression in the CI, the PR author in their role as +:need:`rl__contributor` is responsible for fixing the test case as part of the respective PR. Test case results are also documented in the :need:`wp__verification_platform_ver_report` and :need:`wp__verification_module_ver_report`. @@ -146,10 +145,9 @@ Test case results are also documented in the :need:`wp__verification_platform_ve Reporting of not executed or skipped test cases ----------------------------------------------- -In case a test case is not executed or skipped, a rational has to be provided in the release notes -in case they are link against a requirement. A skipped or not executed test case is not counting into -the value for the requirements coverage, as only executed and passed test cases can be counted for the -coverage of a requirement. +In case a test case is not executed or skipped, a rationale has to be provided in the release notes +if the test case is linked to a requirement. A skipped or non-executed test case does not count towards +requirement coverage, as only executed and passed test cases can be counted for the coverage of a requirement. Skipped or not executed test cases are also documented in the :need:`wp__verification_platform_ver_report` and :need:`wp__verification_module_ver_report`. diff --git a/process/process_areas/verification/guidance/verification_process_reqs.rst b/process/process_areas/verification/guidance/verification_process_reqs.rst index cf84309001..3f5f60b21d 100644 --- a/process/process_areas/verification/guidance/verification_process_reqs.rst +++ b/process/process_areas/verification/guidance/verification_process_reqs.rst @@ -146,7 +146,7 @@ Process Requirements The tool automation shall automatically generate the Verification reports. These may be independent documents (i.e. not integrated into docs-as-code based repositories). The content of the reports is specified in :need:`gd_temp__platform_ver_report` and :need:`gd_temp__mod_ver_report`. - The execution results of a test cases are marked with a clear passed of failed result. + The execution results of test cases are marked with a clear pass/fail result. .. gd_req:: Verification Report Archiving :id: gd_req__verification_report_archiving diff --git a/process/process_areas/verification/guidance/verification_specification.rst b/process/process_areas/verification/guidance/verification_specification.rst index 9d65eb2497..f774c85a3c 100644 --- a/process/process_areas/verification/guidance/verification_specification.rst +++ b/process/process_areas/verification/guidance/verification_specification.rst @@ -83,14 +83,14 @@ A test specification contains the following attributes. The implementation of :need:`wp__verification_plan` defines the full list of allowed types and methods. -It is assumed that tests will be written as code (also for manual tests, which are script based) -and each test case will have a unique identifier, by its script, execution call, or function name. -The call used to execute the test marks the uniqueness of the test case and its identification, -e.g. guaranteeing proper traceability and reproducibility. +It is assumed that tests will be written as code (also for manual tests, which are script-based) +and that each test case will have a unique identifier (e.g. its script name, execution call, or function name). +The invocation used to execute the test defines the test case identification, thereby guaranteeing +proper traceability and reproducibility. -As the tests are stored in a repository close to the implementation code, versioning is done by the versioning of the repository. +As the tests are stored in a repository close to the implementation code, versioning is handled by the repository's version control. -Any specification and resulting implementation ends with a clear passed or failed result. +Any specification and resulting implementation ends with a clear pass/fail result. Test description ---------------- diff --git a/process/process_areas/verification/verification_concept.rst b/process/process_areas/verification/verification_concept.rst index 82287023fe..a29dd68449 100644 --- a/process/process_areas/verification/verification_concept.rst +++ b/process/process_areas/verification/verification_concept.rst @@ -170,7 +170,7 @@ The verification of requirements of types "Process" and "Non-Functional" will be which is part of the requirement inspection :need:`doc__feature_name_req_inspection` and `Component Requirements Inspection Checklist `__. Requirements always include Assumptions Of Use. -A more detailed description of how to link code to requirements is available by :need:`gd_req__verification_link_tests`. +A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests`. Another element of traceability for a proper backlink is the unique identification of test cases as described in the :need:`gd_guidl__verification_specification`. From d5097d2d0ccb4fe1d94d9aa8a7cf3a92383c663f Mon Sep 17 00:00:00 2001 From: Philipp Ahmann Date: Wed, 24 Jun 2026 16:56:47 +0000 Subject: [PATCH 3/3] 2nd round of addressing CoPilot findings Signed-off-by: Philipp Ahmann --- .../verification/guidance/verification_guideline.rst | 10 +++++----- .../guidance/verification_process_reqs.rst | 2 +- .../guidance/verification_specification.rst | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/process/process_areas/verification/guidance/verification_guideline.rst b/process/process_areas/verification/guidance/verification_guideline.rst index 4e245afec9..9cc6298b09 100644 --- a/process/process_areas/verification/guidance/verification_guideline.rst +++ b/process/process_areas/verification/guidance/verification_guideline.rst @@ -106,7 +106,7 @@ Automated tests can also be executed locally, as the sources and binaries are av Failing test cases during re-execution can be reported following the guide :need:`gd_temp__problem_template`. During a release, for any non-executed test case, the reason for non-execution must be documented in the -:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level (unit to platform) of the test case. +:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_note` depending on the level (unit to platform) of the test case. Execution of manual test cases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -122,18 +122,18 @@ Also manual test cases are written following the :need:`gd_guidl__verification_s A tag `manual` is suggested to mark the test case as manual, so that it can be filtered out from automated test execution. In case a manual test case is not executed for a release, the reason for non-execution has to be documented in the -:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level of the test case. +:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_note` depending on the level of the test case. Reporting of failing test cases ------------------------------- Any failing test case requires an ISSUE. -The passing rate of safety-critical test cases need to be 100% in order to release the affected component. +The passing rate of safety-critical test cases needs to be 100% in order to release the affected component. In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead` -can decide, if the platform is in a releasable state. The accepted minimal pass rate is defined +can decide if the platform is in a releasable state. The accepted minimal pass rate is defined in the :need:`wp__verification_plan`. Due to the high degree of automation, it is recommended -that a pass rate lower 95% is not acceptable. This percentage may increase with the maturity of +that a pass rate lower than 95% is not acceptable. This percentage may increase with the maturity of the overall platform and the test coverage. In case an existing test case is failing due to regression in the CI, the PR author in their role as diff --git a/process/process_areas/verification/guidance/verification_process_reqs.rst b/process/process_areas/verification/guidance/verification_process_reqs.rst index 3f5f60b21d..0821cbeb44 100644 --- a/process/process_areas/verification/guidance/verification_process_reqs.rst +++ b/process/process_areas/verification/guidance/verification_process_reqs.rst @@ -174,7 +174,7 @@ Process Requirements - TestType and DerivationTechnique shall be set - Description shall not be empty - - In a Platform Integration Test Partially/FullyVerifies shall be set to a Platform Requirement + - In a Platform Integration Test Partially/FullyVerifies shall be set to at least one Platform Requirement - If Partially/FullyVerifies are set in Feature Integration Test these shall link to at least one Feature Requirement - If Partially/FullyVerifies are set in Component Integration Test these shall link to at least one Component Requirement - If Partially/FullyVerifies are set in Unit Test these shall link to at least one Component Requirement diff --git a/process/process_areas/verification/guidance/verification_specification.rst b/process/process_areas/verification/guidance/verification_specification.rst index f774c85a3c..b2ec494905 100644 --- a/process/process_areas/verification/guidance/verification_specification.rst +++ b/process/process_areas/verification/guidance/verification_specification.rst @@ -57,7 +57,7 @@ A test specification contains the following attributes. - Inputs - Expected outcome/output/result (e.g. "A success message is displayed." or "Result should be 42.") - Test environment (e.g. network configuration, clean system state) - - Expected time sequence of events and behavior (where applicable and an expectation) + - Expected time sequence of events and behavior (where applicable and where such an expectation exists) - * - TestType - Examples are: @@ -85,7 +85,7 @@ The implementation of :need:`wp__verification_plan` defines the full list of all It is assumed that tests will be written as code (also for manual tests, which are script-based) and that each test case will have a unique identifier (e.g. its script name, execution call, or function name). -The invocation used to execute the test defines the test case identification, thereby guaranteeing +The invocation used to execute the test defines the test case identification, thereby supporting proper traceability and reproducibility. As the tests are stored in a repository close to the implementation code, versioning is handled by the repository's version control.