Commit 6d0a973
fix: remove bottleneck caused by waiting for the result of the next task
# Description
ScheduledTasksIterator contained a bottleneck in its scheduling that hat a severe effect on the runtime performance, if some tasks take considerably longer than others. Before the fix, the iterator would only schedule at most `nThreads` tasks in its internal thread pool. Upon consumption of the iterator, a new task was only scheduled, if the next result became available (i.e., a call to `next()` returned a result). This creates a bottleneck, if the iterator has to block upon a `next()` call for a long task. This potentially leads to a high number of idle threads.
With the fix, all tasks are scheduled upon creation of the iterator and their execution is managed by the thread pool with an internal queue. This way, all threads are busy until all scheduled tasks are done. The order of results is still managed by the list of futures, which guarantees that order of results is the same as the order of the given tasks.
# Example
Consider an iterator using a thread pool with 3 threads and 6 tasks to complete. Upon instantiation, the iterator would schedule the firsts 3 tasks for execution. This results in the following state for the tasks:
```
- task1: ONGOING
- task2: ONGOING
- task3: ONGOING
- task4: NOT SCHEDULED
- task5: NOT SCHEDULED
- task6: NOT SCHEDULED
```
and for the threads:
```
- thread1: PROCESSING task1
- thread2: PROCESSING task2
- thread3: PROCESSING task3
```
Now, assume that the iterator is being consumed and a bit of time passed. `task2` and `task3` are done; `task1` is ongoing. The iterator blocks upon the first `next()` call until the result is ready. No new tasks are scheduled. The state would now look as follows:
```
- task1: ONGOING
- task2: DONE
- task3: DONE
- task4: NOT SCHEDULED
- task5: NOT SCHEDULED
- task6: NOT SCHEDULED
```
and for the threads:
```
- thread1: PROCESSING task1
- thread2: IDLE
- thread3: IDLE
```
With the fix, all tasks are scheduled immediately and sequentially submitted to free threads by the thread pool. The initial state now looks like this:
```
- task1: ONGOING
- task2: ONGOING
- task3: ONGOING
- task4: SCHEDULED
- task5: SCHEDULED
- task6: SCHEDULED
```
and for the threads:
```
- thread1: PROCESSING task1
- thread2: PROCESSING task2
- thread3: PROCESSING task3
```
And the second state with the same conditions as above looks like this:
```
- task1: ONGOING
- task2: DONE
- task3: DONE
- task4: ONGOING
- task5: ONGOING
- task6: NOT SCHEDULED
```
and for the threads:
```
- thread1: PROCESSING task1
- thread2: PROCESSING task4
- thread3: PROCESSING task5
```1 parent cec1747 commit 6d0a973
1 file changed
Lines changed: 6 additions & 16 deletions
Lines changed: 6 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | | - | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
32 | | - | |
33 | | - | |
| 33 | + | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | 48 | | |
58 | 49 | | |
59 | 50 | | |
| |||
70 | 61 | | |
71 | 62 | | |
72 | 63 | | |
73 | | - | |
74 | 64 | | |
75 | 65 | | |
76 | 66 | | |
| |||
0 commit comments