Fix #191: Add producer-consumer model

[tomschr]

This PR plays around with the producer-consumer model requested in issue #191. It maybe be integrated or it goes nowhere.

To test it, use these commands:

source devel/activate-aliases.sh
upython -m docbuild.utils.concurrency

Questions:

• The current implementation uses TaskGroups, which is considered as the go-to way. Semaphores seems not always that reliable or appropriate for this use case.
  => We use TaskGroups
• What do we want to "return" inside the consumer() loop?
  The consumer calls the worker function. Depending on if it's successful or not, we add either a Success object or a Failure object.
  I'm not sure if this is a bit too much and can be simplified.
  => We return the respective object and if something fails, a TaskFailedError exception.
• I'm not sure about the naming: process_unordered. Possible better names:
  • execute_concurrently
  • run_in_parallel

[github-actions]

Coverage Report

For commit 691014b

Click to expand Coverage Report

  Name                                           Stmts   Miss Branch BrPart  Cover
  --------------------------------------------------------------------------------
+ src/docbuild/models/deliverable.py               180      1     22      0  99.5%
+ src/docbuild/cli/cmd_check/process.py             58      0     22      1  98.8%
+ src/docbuild/models/manifest.py                  111      1     12      1  98.4%
+ src/docbuild/utils/pidlock.py                     79      1     14      1  97.8%
+ src/docbuild/cli/cmd_cli.py                       96      1     10      2  97.2%
+ src/docbuild/cli/cmd_validate/process.py         178      5     52      4  96.1%
+ src/docbuild/cli/callback.py                      35      0     10      2  95.6%
+ src/docbuild/utils/concurrency.py                 69      3     18      1  95.4%
- src/docbuild/cli/cmd_config/__init__.py            9      1      0      0  88.9%
- src/docbuild/config/xml/stitch.py                 47      5     12      0  88.1%
- src/docbuild/cli/cmd_metadata/metaprocess.py     215     26     66     13  82.6%
- src/docbuild/cli/cmd_check/__init__.py            18      5      2      0  65.0%
- src/docbuild/cli/cmd_build/__init__.py            13      5      0      0  61.5%
- src/docbuild/cli/cmd_metadata/__init__.py         27     10      2      0  58.6%
- src/docbuild/cli/cmd_config/environment.py        11      6      2      0  38.5%
  --------------------------------------------------------------------------------
+ TOTAL                                           2992     70    706     25  96.9%
  
  46 files skipped due to complete coverage.

[sushant-suse]

This PR plays around with the producer-consumer model requested in issue #191. It maybe be integrated or it goes nowhere.

To test it, use these commands:

source devel/activate-aliases.sh
upython -m docbuild.utils.concurrency

Questions:

• The current implementation uses TaskGroups, which is considered as the go-to way. Semaphores seems not always that reliable or appropriate for this use case.
  => We use TaskGroups

• What do we want to "return" inside the consumer() loop?
  The consumer calls the worker function. Depending on if it's successful or not, we add either a Success object or a Failure object.
  I'm not sure if this is a bit too much and can be simplified.
  => We return the respective object and if something fails, a TaskFailedError exception.

• I'm not sure about the naming: process_unordered. Possible better names:

  • execute_concurrently
  • run_in_parallel

Thanks for the awesome work Toms. It looks great. I have also added my suggestions.

[tomschr]

Ready to be squash-merged! 👍 🎉