Handling exceptions when applications do not report errors but time out in stateful transitions

[PawelPlesniak]

Description

Fixes issue #803
If a segment does not reach the target state, it is marked as in error, and the timeout is logged in the relevant server.
Also defines a set of configurations constructed to fail, and defines a set of unit tests to demonstrate this behaviour.
Error recovery with the supervisor will address what happens if an application completes this outside of the designated window. This is defined in #840

Type of change

• New feature / enhancement
• Optimization
• Bug fix
• Breaking change
• Documentation

List of required branches from other repositories

Requires DUNE-DAQ/druncschema#87

Change log

Defines a set of intentionally failing configurations in config/tests/failure-mode-testing.data.xml, which contain configurations with a set of fake-daq-apps that fail at pre-defined points. These configurations are (note the checkboxes are for final testing whether the intended behaviour is as desired, and will be removed prior to marking as ready for review.

• ft-reference - this is a reference configuration without the small changes required to simulate failures.
• ft-death-on-boot-nest-app - this kills a nested application (2+ segments deep) on boot.
• ft-death-on-boot-top-app - this kills the top application on boot.
• ft-death-post-boot-nest-app - this kills a nested application (2+ segments deep) after boot, before applications are marked as ready.
• ft-death-post-boot-top-app - this kills the top application after boot, before applications are marked as ready.
• ft-fsm-cmd-timeout-nest-app - this times out an FSM transition on a nested application.
• ft-fsm-cmd-timeout-top-app - this times out an FSM transition on the top application.
• ft-fsm-cmd-death-nest-app - this kills a nested fake daq app during a FSM transition.
• ft-fsm-cmd-death-top-app - this kills the top fake daq app during a FSM transition.

These tests have been integrated into the unit test framework.

Suggested manual testing checklist

Run each of the commands, and validate that the intended behaviour is as expected. Prior to running each of these configurations, the following script will need to be run

<DRUNC_ROOT>/config/setup_drunc_config_path.sh

These are the following commands to run each of the tests manually. The checkboxes are left for the reviewer to keep track of their testing progress.

• Reference test

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-reference pr868 start-run --run-number 1 wait 10 shutdown

This run should complete without any error conditions.

• Nest app death on boot

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-death-on-boot-nest-app pr868 boot

bottom-segment-2-application should die on boot, and its logs should contain

Simulating death of bottom-segment-2-application on boot

the top-segment-controller should be in error, and the following line should be in the tty:

ERROR      commands.py:119                          drunc.unified_shell.boot                           Booted, but the top controller is in error

• Top app death on boot

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-death-on-boot-top-app pr868 boot

nested-segment-application should die on boot, and its logs should contain

Simulating death of nested-segment-application on boot

the top-segment-controller should be in error, and the following line should be in the tty:

ERROR      commands.py:119                          drunc.unified_shell.boot                           Booted, but the top controller is in error

• Nest app death post boot

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-death-post-boot-nest-app pr868 boot

bottom-segment-2-application should die at the end of boot, and its logs should contain

Simulating death of bottom-segment-2-application post boot

the top-segment-controller should be in error, and the following line should be in the tty:

ERROR      commands.py:119                          drunc.unified_shell.boot                           Booted, but the top controller is in error

• Top app death post boot

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-death-post-boot-top-app pr868 boot

nested-segment-application should die at the end of boot, and its logs should contain

Simulating death of nested-segment-application post boot

the top-segment-controller should be in error, and the following line should be in the tty:

ERROR      commands.py:119                          drunc.unified_shell.boot                           Booted, but the top controller is in error

• Nested app timeout on FSM transition

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-fsm-cmd-timeout-nest-app pr868 boot

bottom-segment-2-application should time out on conf, and its logs should contain

Delaying execution of bottom-segment-2-application by 100 seconds

the top-segment-controller should be in error, and the following line should be in the tty:

ERROR      shell_utils.py:640                       drunc.controller.iface.shell_utils                 The command timed out, unfortunately this means the server is in undefined state, and your best option at this stage is to terminate and boot.
ERROR      shell_utils.py:657                       drunc.controller.iface.shell_utils                 The session did not complete the stateful transition in the specified time of 60 seconds. To investigate the cause, please check the controller and application logs with the 'logs' command.```
ERROR      commands.py:119                          drunc.unified_shell.boot                           Booted, but the top controller is in error

• Top app timeout on FSM transition

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-fsm-cmd-timeout-top-app pr868 boot

nested-segment-application should time out on conf, and its logs should contain

Delaying execution of nested-segment-application by 100 seconds

the top-segment-controller should be in error, and the following line should be in the tty:

ERROR      shell_utils.py:640                       drunc.controller.iface.shell_utils                 The command timed out, unfortunately this means the server is in undefined state, and your best option at this stage is to terminate and boot.
ERROR      shell_utils.py:657                       drunc.controller.iface.shell_utils                 The session did not complete the stateful transition in the specified time of 60 seconds. To investigate the cause, please check the controller and application logs with the 'logs' command.```
ERROR      commands.py:119                          drunc.unified_shell.boot                           Booted, but the top controller is in error

• Nested application death during FSM transition

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-fsm-cmd-death-nest-app pr868 boot conf

bottom-segment-2-application should die on conf, and its logs should contain.

Simulating death of bottom-segment-2-application during FSM cmd execution

TTY TBC

• Top application death during FSM transition

drunc-unified-shell ssh-standalone config/tests/failure-mode-testing.data.xml ft-fsm-cmd-death-top-app pr868 boot conf

nested-segment-application should die on conf, and its logs should contain.

Simulating death of nested-segment-application during FSM cmd execution

TTY TBC

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed
  • Only concern yourself if failures related to drunc are in the log files
  • If non-drunc failure appears:
    • Validate failure in fresh working area
    • Contact Pawel if unsure

Once the features are validated and both the unit and integration tests pass, the PRs is ready to be merged.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #daq-sw-librarians Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[PawelPlesniak]

In the case where a second application also fails to complete a transition in time, the same error gets thrown. This is likely caused by the nested structure, and the fact that there are multiple layers to this configuration. A robust solution to this problem will take longer to achieve, but I will continue working on it.

[PawelPlesniak]

• Log files do not contain wierd characters from redirecting rich ASCII output to a file with colors
• Top app config failure not working?

[PawelPlesniak]

• Stream handler has no logging time zone, file name, line number, etc?
• Duplicate logs of apps failing
• Remove old logs used for debugging
• Add druncschema dependency to this PR log
• Integrate the failure mode testing into the unit tests