Verification: Update test specification details

process/process_areas/verification/guidance/verification_guideline.rst

@@ -99,9 +99,14 @@ In order to check the test results for the impact of a change or addition, it is
 execute affected test cases locally upfront using the execution framework of the build tooling
 following basically the steps the CI does locally.
 
+Tests have to be identifiable by a unique identifier, as described in :need:`gd_guidl__verification_specification`.
+They need to have a clear pass/fail result and a documented configuration to enable proper evaluation of the result.
+
 Automated tests can also be executed locally, as the sources and binaries are available for re-execution.
 Failing test cases during re-execution can be reported following the guide :need:`gd_temp__problem_template`.
 
+During a release, for any non-executed test case, the reason for non-execution must be documented in the
+:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_note` depending on the level (unit to platform) of the test case.
 
 Execution of manual test cases
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -112,19 +117,40 @@ test cases and reports the result in the same logging format as automated tests
 This enables parsable execution logs which can be used in automated test result collection.
 Failing test cases during manual execution are reported by following the guide :need:`gd_temp__problem_template`.
 
+Also manual test cases are written following the :need:`gd_guidl__verification_specification` and have a unique identifier.
+
+A tag `manual` is suggested to mark the test case as manual, so that it can be filtered out from automated test execution.
+
+In case a manual test case is not executed for a release, the reason for non-execution has to be documented in the
+:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_note` depending on the level of the test case.
+
 Reporting of failing test cases
 -------------------------------
 
 Any failing test case requires an ISSUE.
-The passing rate of safety-critical test cases need to be 100% in order to release the affected component.
-In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead` and
-:need:`rl__project_lead` can decide, if the platform is in a releasable state. The accepted minimal
-path rate is defined in the :need:`wp__verification_plan`. Due to the high degree of automation, a
-it is recommended that a path rate lower 95% is not acceptable.
-
-In case an existing test case is failing due to regression in the CI, the respective issuer of the
-PR in their role as :need:`rl__contributor` is responsible for fixing the test case as part of
-respective PR.
+
+The passing rate of safety-critical test cases needs to be 100% in order to release the affected component.
+In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead`
+can decide if the platform is in a releasable state. The accepted minimal pass rate is defined
+in the :need:`wp__verification_plan`. Due to the high degree of automation, it is recommended
+that a pass rate lower than 95% is not acceptable. This percentage may increase with the maturity of
+the overall platform and the test coverage.
+
+In case an existing test case is failing due to regression in the CI, the PR author in their role as
+:need:`rl__contributor` is responsible for fixing the test case as part of the respective PR.
+
+Test case results are also documented in the :need:`wp__verification_platform_ver_report` and
+:need:`wp__verification_module_ver_report`.
+
+Reporting of not executed or skipped test cases
+-----------------------------------------------
+
+In case a test case is not executed or skipped, a rationale has to be provided in the release notes
+if the test case is linked to a requirement. A skipped or non-executed test case does not count towards
+requirement coverage, as only executed and passed test cases can be counted for the coverage of a requirement.
+
+Skipped or not executed test cases are also documented in the :need:`wp__verification_platform_ver_report` and
+:need:`wp__verification_module_ver_report`.
 
 Reuse of existing test cases
 ----------------------------

process/process_areas/verification/guidance/verification_process_reqs.rst

@@ -146,6 +146,7 @@ Process Requirements
     The tool automation shall automatically generate the Verification reports.
     These may be independent documents (i.e. not integrated into docs-as-code based repositories).
     The content of the reports is specified in :need:`gd_temp__platform_ver_report` and :need:`gd_temp__mod_ver_report`.
+    The execution results of test cases are marked with a clear pass/fail result.
 
 .. gd_req:: Verification Report Archiving
     :id: gd_req__verification_report_archiving
@@ -173,10 +174,10 @@ Process Requirements
 
     - TestType and DerivationTechnique shall be set
     - Description shall not be empty
-    - In a Platform Integration Test Partially/FullyVerifies shall be set to a Platform Requirement
-    - If Partially/FullyVerifies are set in Feature Integration Test these shall link to Feature Requirements
-    - If Partially/FullyVerifies are set in Component Integration Test these shall link to Component Requirements
-    - If Partially/FullyVerifies are set in Unit Test these shall link to Component Requirements
+    - In a Platform Integration Test Partially/FullyVerifies shall be set to at least one Platform Requirement
+    - If Partially/FullyVerifies are set in Feature Integration Test these shall link to at least one Feature Requirement
+    - If Partially/FullyVerifies are set in Component Integration Test these shall link to at least one Component Requirement
+    - If Partially/FullyVerifies are set in Unit Test these shall link to at least one Component Requirement
 
 .. gd_req:: Verification Documentation Checks Extended
     :id: gd_req__verification_checks_extended

process/process_areas/verification/guidance/verification_specification.rst

@@ -55,8 +55,9 @@ A test specification contains the following attributes.
 
        - The objective of the test.
        - Inputs
-       - Expected outcome (e.g. "A success message is displayed.")
+       - Expected outcome/output/result (e.g. "A success message is displayed." or "Result should be 42.")
        - Test environment (e.g. network configuration, clean system state)
+       - Expected time sequence of events and behavior (where applicable and where such an expectation exists)
      -
    * - TestType
      - Examples are:
@@ -82,6 +83,14 @@ A test specification contains the following attributes.
 
 The implementation of :need:`wp__verification_plan` defines the full list of allowed types and methods.
 
+It is assumed that tests will be written as code (also for manual tests, which are script-based)
+and that each test case will have a unique identifier (e.g. its script name, execution call, or function name).
+The invocation used to execute the test defines the test case identification, thereby supporting
+proper traceability and reproducibility.
+
+As the tests are stored in a repository close to the implementation code, versioning is handled by the repository's version control.
+
+Any specification and resulting implementation ends with a clear pass/fail result.
 
 Test description
 ----------------

process/process_areas/verification/verification_concept.rst

@@ -124,7 +124,6 @@ Usually the defined methods are not applied on each verification level between u
 Also their execution may differ whether it is a QM or ASIL rated test case.
 The rigor is described in the implementation of :need:`wp__verification_plan`.
 
-
 Automated test cases (as well as manual, where applicable) shall contain further information about which methods have been applied.
 The corresponding guidance is given here: :need:`gd_guidl__verification_guide`.
 The identifier of the respective method is to be used as meta data (*TestType* and *DerivationTechnique*).
@@ -161,17 +160,20 @@ tests or component tests. This linking is done using metatags. This is also true
 integration tests linking to the component requirements and architecture.
 
 Traceability of feature integration tests shall be established through linking those test cases to
-feature requirements and architecture as features describe the integrated behaviour of all components.
+feature requirements and architecture as features describe the integrated behavior of all components.
 
 Traceability of platform integration tests shall be established through linking those test cases to
-stakeholder requirements as stakeholder requirements describe the platform behaviour.
+stakeholder requirements as stakeholder requirements describe the platform behavior.
 
 Note that all the above tests shall only link to requirements of type "Functional" and "Interface".
 The verification of requirements of types "Process" and "Non-Functional" will be done via Analysis,
 which is part of the requirement inspection :need:`doc__feature_name_req_inspection` and `Component Requirements Inspection Checklist <https://eclipse-score.github.io/module_template/main/score/component_example/docs/requirements/chklst_req_inspection.html>`__.
 Requirements always include Assumptions Of Use.
 
-A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests`
+A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests`.
+
+Another element of traceability for a proper backlink is the unique identification of test cases as described in the
+:need:`gd_guidl__verification_specification`.
 
 Workflow for Verification Guidance
 ----------------------------------

process/process_areas/verification/verification_getstrt.rst

@@ -41,8 +41,8 @@ General Workflow
 
 The workflows can be split into 4 major parts:
 
-* Test planning filling the template :need:`gd_temp__verification_plan`.
-* Test specification and implementation for the respective testing level
+* Test planning by filling the template :need:`gd_temp__verification_plan`.
+* Test specification and implementation for the respective testing level (unit to platform).
 * Test execution by the CI.
   (Manual test cases are treated as automated test with user interaction and timeouts.)
 * Test reports are created when all verification artifacts on a module and platform level are